I don't think you're crazy at all. In fact I much prefer the look of CQs without any identifiable regular layout. Much more like the original ones were made.
Make 'fabric' of random shapes using whatever method you prefer (I don't use foundation) to any size that feels comfortable, then start combining them as if they were a single piece of fabric, trimming as necessary. You have to truly 'think outside the block', but the results are well worth it. The only thing that needs watching is how the seams meet when joining sections...looks best if they don't actually.
Have fun!