The button loop gets sewn on the back side, not the front with the pocket. Sew the button loop and the handles to the right side of the fabric with a scant 1/4 inch seam.
Does this make sense? You'll want to put your button above the pocket on the front.