While I waited 8 months for our house to be built I thought of many drawbacks and suggested we look at other options. DH was worried about losing the down payment, but I ended up spending that to put a sidewalk down the side of the house and a patio in, etc. I still regret buying this house. We paid for many "extras" that I thought would suit our needs.
At the time, during the recession, I could have bought a lot more house for the money. Hubby was thinking this was our dream house and it sure turned out not to be.
So, bottom line, if you don't think it will work the way you want to -- back out and build one that will.
Forgot to add, at the time we built I didn't know I'd be a quilter, I sure would have designed the office/sewing room differently with table height outlets, more storage, etc.