I understand exactly where you're coming from.
I myself have to have a certain "orderliness" to what few scrap quilts I have done. If you look at my avatar, you will see what I consider to be my most "successful" scrap quilt (a Bonnie Hunter pattern). I HAD to have a certain amount of "order" or "continuity" to it, otherwise it would make me nuts.
Just willy-nilly is completely beyond me.
I think for me, it partly due to vision problems - certain amount of color-blindness as well as poor vision in general - and me being a little bit OCD.
So, don't feel bad, you're not alone!