My design wall is black fleece .... one layer! Everything sticks to it without any problems.
Once I had my OBW all put together and pressed, I hung it there for a good drying time .... no pins!
BuzzinBumble ... Maybe it's the weight of the fleece you used? I didn't buy any specific type, just black of what was in stock.