I agree that starch is our friend here.
I've just done two samples of these lovely little geese. The first one I did without spray starch, and it went veering over to the right... very messy, and definitely no longer on the straight of grain.
When I did the second set, I starched the square and triangle together lightly (I use a homemade spray starch, of 250 ml water and 1.5 teaspoons of potato flour) before started any sewing. The result was a lot better, straighter, and on a nice flight path, but the danger of veering off is always there.
How you press and starch them seems to me to be very important.