You're not weird at all. By all means GO. You said it yourself - the ONLY thing you'll have in common with the others is quilting (which may or may not be true). Well quilting is the purpose of the retreat and that common ground will open up conversations in which you'll take part, maybe slowly at first, but after a little time you (and maybe other first-timers) will all be part of the group. The hardest part is to take a deep breath and walk thru the door, after that you'll be fine. Let yourself enjoy the experience and you will. Years ago in the middle of a bad romance I decided "I'll show him, I'm going to Europe". As I was on my way to London I thought what on earth am I doing. But I took that deep breath.....turned out to be a wonderful trip, something I'd never have thought I could do. You may surprise yourself. Plus the practical side - you'll be getting a lot of new quilting ideas and instructions.
If you go, please write and tell us all about it. You have a big cheering section here.