If you're really picky about them, the only way to roll is with a new trackball. probably cost 80 bucks or so with the translucent red ball like original.
You can replace the rollers and bearings, but all it takes is 1 bearing to hang a little bit and the thing won't act right. Also the ball is worn.
Are your rollers straight, or do they have a groove worn into them? That's solid steel, and the ball just wore through it... imagine what has happened to the ball itself. Basically it's smaller now than it used to be, but it wears evenly so you can't tell by looking at it. If you put new rollers and bearings in it, it will work pretty good but the ball won't quite fill the case like it should, and it'll hang in certain directions just slightly, or wobble a little bit to the side instead of rolling smoothly.
So it just depends on how 'nice' you want it to be. New rollers and bearings would help. New rollers, bearings, and ball would be pretty good, and new assembly would be perfect.
BTW, what do you mean by you lubed the rollers? do you mean the bearings (the circles on the end, with balls inside?)