I have a pattern for making sweatshirts out of quilted fabric. The best sandwich stuff would be prewashed flannel and backed with muslin. Batting would be too much, unless it was as thin as flannel. Sweatshirt fleece does have some give and being a knit, it does breathe.
Even the wrong interfacing on sweatshirt knit can make it too stiff.