You sure can! If the fabric is a knit, I would stabilize it by ironing fusible interfacing to the back. If it's a woven cotton, treat it like you would any quilt fabric.
A lot of old scrap quilts were made using fabric salvaged from clothing. I just quilted an "oldie"made by the deceased mother of a friend of mine that was made using denim, corduroy, flannel shirting and old fashioned polyester made from clothing scraps.
Have fun with it!