As Angela Walters says, you can only worry about one side of the quilt. Decide which one is most important to you and quilt for that side. That being said, if you use a fine thread in a closely matching color for the panel side it shouldn’t be too noticeable.