I don't know if this idea might work, I have never tried it, but if I were in your shoes I think I would give it a go.
Could you iron the panel onto a piece of lightweight interfacing so that it holds the shape you want and then it shouldn't matter if the fabric is off-grain.