Being able to smoothly make "O"s is how many people test a frame. How easily the wheels or bearings turn on the carriage, how smooth the tracks, rails or pipes are and the frame must be level for it all to function correctly.
Even some of the big name machine quilting frame companies went through several upgrades & re-designs before they came up with a frame & carriage that could make smooth "O"s.
Look at other frames that people rave about--Hinterberg Original, Stretch & Summit; SuperQuilter and SuperQuilter ProFlex; Grace Pro & Pinnacle and several others to see how they are made and see if there are any modifications that you can make to yours to make it more like them.
I know Hinterberg sold the plans to make your own frame at one time and you can buy just the carriage from several different manufacturers. Your rail system will determine which kind of carriage you can use.
I have a B-Line Studio frame and a SuperQuilter ProFlex. They both can make round "O"'s. Because of the way the B-Line is designed--it's fussier about being & staying level than the ProFlex