In case anyone else can't figure this out, here's what I did:
I tried what everyone has said to do with the actual switch, which is apply heat. I don't have a heat gun yet so I used a hair dryer. I wouldn't recommend that. It worked but the whole hilt got pretty warm. Not to hot to touch but almost. With a heat gun I imagine I'd be able to direct the heat a little more precisely, whereas the blow dryer can't really do that.
I was able to get the AV switch out no problem at all. Heated that sucker up, stuck a rat-tail file in the end of the hilt and used it to press the button out. Popped right out, VERY little force required. The dummy button was another matter entirely I didn't realize how thick the wall of the hilt is and the dummy button is very flat. It's about the same height as a stack of two US quarters and it's in a hole that's clearly drilled for a tall 12mm switch/button.
I tried all kinds of things to get it out. What finally did it was the allen wrench that came with my hilt (the one to tighten the blade retention screw). I stuck the wrench into the hole I had just pulled the latching switch out of, so the short end was inside the hilt. Then I turned it so the end was tapping on the button of the dummy button and pushed it out.
The dummy button was significantly better-secured than the real switch, which seemed to be held on by a healthy amount of Loctite (or something similar). While the switch has what looks like the expected white residue of that sort of adhesive, the dummy button has a bit of a crust on it. Almost looks like a very thin application of superglue or even Gorilla Glue.