This is an entertaining thread. Several of the comments made me lol.
I think your coworker is a bit rude, actually, rolling his eyes at what you like to do. Rolling his eyes at your passion is just like making a disparaging remark to you about it. Who needs that? Who needs him? I think I'd search out nicer, more understanding friends at work. And definitely join a quilt guild!