Solution = Pockets!
........ and one reason why I have stayed with a flip phone.
If it does not fit in the pocket (aka smartphones tend not to fit), I don't want it.
Most of my clothes have pockets, and when they don't then yes, I have problems.
You could make a pocket to attach to a belt, if you wear belts.
Or make a belt-like sash that you can tie around your middle.