I think it sounds great and that you should go for it. Maybe consider selling online as well as in your store to increase sales opportunities?
I met a young woman once who lived in a tourist town, and she rented bicycles to tourists and sold fabric from her shop. She only had lines from companies that didn't require too big a minimum purchase, and she mostly sold the styles she liked – mostly modern, cute stuff. Later, she shut it down and started selling used books. I thought it was great the way she put her passion to work in a business. I think if you are creative you can make anything work.