I make a fair number of memory quilts and I have included all kinds of fabric in them. I even made on that consisted enitirely of ribbons won at horse shows. This was to be used as a wall hanging only, because there was no way to wash those ribbons, no matter what did them.
I just completed a quilt for someone that had some swimsuit fabric in it, along with some work-out gear. I used a lightweight stabilizer on those fabrics (and any other strechy ones) and then had no problem. I have found that I have to be careful how and where I place these fabrics when they are mixed with lots of other fabric types. I often try not to put too many "stabilized" fabrics together and then have a bunch of other thinner fabrics together. In other words, I try to distribute the thickness throughout the quilt.
I will agree that this project will not be an easy or quick one to do, but it will be special. I do enjoy making memory quilts simply because I know how much they mean to the recipient. Of course, I do try to make sure I am compensated for the extra time and work.
Good luck and I look forward to seeing pictures of your progress and finished quilt, if you decide to do it.