I doubt very seriously you are hearing any leaf switch noise unless they aren't screwed in tightly underneath. Usually you can hear a sort of "rebound vibration" noise from the rubber grommet snapping the joystick back to center. It's easy to silence that by keeping your hand on the joystick and not letting it just snap back to center.
Any other noise may be caused by a misassembled joystick. If you are missing any of the pieces between the c-clip and the grommet, or between the grommet and the balltop, then you may hear some noise because the pieces that are there have too much room to wiggle around....