Absolutely. One of my 2011 personal challenges is to make a Man Quilt out of only mens shirts that I find on garage sales and thrift stores. You can get fabric from the yokes, cuffs, sleeves, front and back---lots of yardage in 30 shirts. I'm going to design something so that I can have a rather large block every now and then. I want to put in a square here and there that still has the pocket attached. There was a picture posted here once that had clothing--uncut and sewn together to make a quilt. It looked like a really messy room but it was unique. I'm probably going to go for more of a traditional look.
Be sure and grab a baggie for the buttons--you never know when you'll want to use them for embellishing something or PIF to someone needing buttons for their projects.