Maybe cut the lining for the bag AFTER the outer layer has been quilted?
The quilting draws up the fabric some.
I had that problem, too, when I was making some bags. So after I waited to cut the lining to the size of the outsde AFTER I had the outer layer quilted, that did help quite a bit with the interior fluffiness.