It depends very much on your style of quilting. If you don't like scrappy quilts, you probably aren't going to get a lot of value out of saving every little offcut.
I make foundation-pieced crazy quilts (sewing fabric scraps down to squares of thin muslin or salvaged sheets), so I can use virtually ANY size of scrap. If a scrap is really small but also very precious I'll go ahead and just topstitch it down and let it be a little frayed or go over the edges with decorative stitches - for my scrappy style that works just fine...it's just a little added texture!
So I save pretty much everything. The really itty-bitties (like trimmings) and all my batting scraps end up in pet beds - it takes me roughly a year to fill one up and I have 3 dogs and 2 cats so that works out well.