So I finally took my joystick apart last night. I had cleaned up the panel years ago, but this was before new grommets were available. I disassembled the stick, it had a sloppiness to it, like the grommet was torn. Surprisingly it wasn't, but the shaft had a lot of play in the grommet. (pic below). I figured the shaft was worn, but when I tried a newer/clean grommet out of a newer microswitch stick, it fit tight. The metal in the grommet was worn out.
Will scrub all the parts, polish the stick again, use the nice grommet, and add a little grease where metal touches metal.
Below, nice grommet on left, worn one on right
Wow, it's amazing what those sticks went through. Now get that white balltop back to pristine white!

