Neither of those products would work to hold together a sweatshirt & velour. I'm allergic to basting spray so I'd go with safety pins or washable glue if I was going to quilt it.
I'm not sure why it would be a problem is the layers shift around a bit. That's typically how ponchos are constructed. If you really want that -- though I'm not sure I'd personally like how heavy & clingy it would be -- you could always use Fabric Fusion. It's a glue-like adhesive that's sold in most craft stores (Hobby Lobby, Joann...). I've used it on fabric & it holds well even through multiple washings (warm water/medium heat). Just make sure that's really the route you want to go because once it's fused, it's not coming apart. It's typically used in small amounts by people who want a no-sew option for hemming or fixing a seam, but I've used it with very young children to create faux-quilts without any stitching & so I know it will hold larger pieces.