It sounds like the fabrics are stretching. You can add a row of stay tape to the block and then stitch together. It will help prevent that stretching.
Are you pinning the blocks together before stitching? Are the blocks fused with a backing stabilizer? There are light weight ones that do not add bulk to the project.
There are some tips here that might help. I have used this tape to shorten some knit work shirts. It is nice to work with on these types of fabrics.
https://emmaseabrooke.com/