Usually you grab the base of the button and give it a twist and pull, and it pops out so you can replace the bulb. Some have little clips that must be pushed in first....
In order to check those bulbs you need to FIRMLY grab the white plastic part thats in the button (from the back) and wiggle it until it pops out. However, that plastic gets brittle so it can break if your not careful.