I didn't read all of the posts, but I would just finish it "as is" and bind it in brown. It seems exactly right to me the way it is. If you take off the brown, it will stretch the edge and you'll have to deal with that.
(disclosure: olive green is one of my least favorite colors)
bkay