From repeated washings, in my experience it is the binding that takes the beating.
But... from lying on, sitting, and "spending time with"... the quilt I made for my kids' bed when they were small... there are squares completely worn through. No worries though, it was a quilt that was made to be used up.