It would suddenly go from something you enjoy doing to something you have to do. Working from home requires alot of self dicipline. Any business venture is a risk. If it dosen't work out can you get another job? To earn a decent income would you have to work more hours than you can realistically do? Do you have enough savings to get by on until the workload becomes steady enough to replace your current salary?
I've been a business owner for many years. Alot of self examination is in order, a good solid business plan, and write down the pros and cons and be honest about both.