It should be fine!! Once you fuse the fleece onto it, it's going to be more stable if it's stretchy...just be careful not to tug on it too much.
I have some fabric that was my mom's that is a jacket lining type fabric, that I'm going to use for the inside of a couple of bowtucks...