I'm 34 and started quilting when I was 21.
I suspect that some of the 'old ladies' label comes from the fact that people have more time to do what they want (whatever that is) once they retire. Plus women have 24/7 occupation with house and kids, so for much of their lives they don't get to have hobbies in the same way that men do (yes, I am making vast generalisations here, but as a general rule...). So OK, the kids grow up and leave. The house is still there, but the retirement-age women I know seem to decide that there is more to life than having a perfect house (I agree!) and/or insist that hubby take on his fair share of house-chores when he retires which allows them to have more free time to quilt. Or paint. Or dance. Or scuba dive. Or whatever.