Sounds like you are looking at your decision together very seriously an taking the $$$$ in to account...sleep on it and agree on what to do that is the best thing for you to do for your peace of mind. No one else can tell you all what to do, just you'all have to agree. It is very serious move either way. Maybe get your house valued at current market price and see what it is worth now...