1. From here, it seems to me that some of you green squares, on the left side, are simply too small.
2. Are your seam allowances uniform at 1/4"? If they are a little bit too large, your 4 patches will be 2 little bits too small. I used to measure them incorrectly. The "scant 1/4" that people talk about means measure your 1/4" from the center of the seam to the outermost shred of the seam allowance.
3. I found that where seams cross, the allowance on the top should be placed so it reaches the needle before the seam does. This keeps the seam line from pushing apart. As Always quilting says, you can get the seam allowance back to going the 'right' way by clipping it near the cross seam.
4. Sometimes just pressing corrects the apparent problem. If it does not, what I woudl do is start over on the 4 patch area and use THESE 4 patches in another project. There is no way I would do that much frog stitching.
5. In the future, remember that it is a good idea to do some measurement checks as you go along, so you don't get this far into a problem.
You have a nice design here. The color placement is very attractive.
The top, also called a 'flimsy', needs to be flat regardless of how you are using it. A 2 layer quilt, also called a "summer quilt" shows any irrgularity more than a bulkier quilt might.