I don't know of a formula, but the initial rectangle has to be big enough to allow for the 6" dimensions. Maybe use some scrap fabric or paper to experiment with sizes of rectangles to have enough fabric for the 6 inch width, and then add on enough paper/scrap fabric to achieve the height, then you can take it apart to get the right overall rectangle size.Does that make sense? The 10" width is the easy part!
I sort of assumed you are using the technique where you make a triangle at the corners to achieve the box end.