jill, this is wonderful! I would seam the end panels as you said and then "birth the backing". Leave an opening in the backing end panels seam for turning and stitch the inner circle and the outer edge. Trim and clip the curves and turn the backing right side out.
Then to secure the backing to the rest you could stitch in the ditch on the wedge seams.