If you don't have a lot of visitor's I'd say go for it, besides like others have said its your house. I have a couple of friends that their living/family room is downstairs and the space upstairs is used for other things.
Right now my dining room table is my work space for cutting, we don't do alot with company either. Were in the process of renovating the office area upstairs into a quilting room for me. That area gets the best sunlight all day and it has a big window to let the light in.