My daughter injured her pinky several years ago while playing soccer. The pinky wasn't broken. It took months for it to heal, and it healed in a bent position. Tendons (or is it ligaments)were torn. That doesn't show up on an x-ray. The pinky should have been braced to prevent the bending, and exercised (like PT) to prevent it from 'freezing'. Similar thing might have happened to you.
There are other conditions that can cause pain in fingers besides arthritis, although arthritis is the one we hear most about.
I'll send a few prayers your way!