Welcome to the world of fleece! It can be such a challenge to work with. I sell hundreds of yards of fleece to folks making the tie blankets and the first thing I tell them is to not try to make the 2 pieces match. Lay one down and gently lay the other on top of it - do not stretch to make it fit. If you do, it will pull back and make everything wobbly. Once you have gently laid the 2 pieces and pinned them, then you can trim them to make them even. Same thing applies to a quilt. Lay down the fleece and lay your top on it, then pin and trim. In fact, with the quilt, I would probably do a basting stitch all around before I would trim.