I hate when that happens - going along as nice as can be - and then the stitches get wonky!
What you do will depend on how much the wonky stitches bother you. I have had a tough time lining up a "fill-in" - so I would probably take out the whole side and try again. I think the holes will fill up - unless you used a huge needle.