Well I got the pulley loose with some penetrating oil.
When I sprayed it on there I covered the vents in the motor housing with a rag and the thing slid right off.
I turned the shaft and I still hear the noise.
I was really hoping it was just the set screw.
Maybe there's a tiny little rough spot or carbon buildup on that spinning doohicky or a brush.
It's not loud or anything it just wasn't there before so it's bugging me.
It still works so I guess I can live with it.