I do the same... sometimes I pre-wash, sometimes I don't. I couldn't say exactly WHY I make that decision. Maybe it's the feel of the fabric, maybe because I am using parts that I can't pre-wash like jelly rolls. I do keep them consistent, all washed or all not.
Today I got some flannel for backing on two baby quilts and I will definately wash that. I washed the fabrics in those quilts too because I wanted them softer and I didn't use any starch in the construction.