Poly felt is just fibers pressed together. I think it would fall apart with use, washing, and abuse.
Wool felt is also not woven, but felt fibers have little "hooks" on them that attach to each other so felted wool is very stable. I think wool felt is extremely expensive, though.
I have never heard of using felt as a batting, probably because it would either ball up or separate (poly) or be outrageously expensive (wool).