I realized I wasn't really using the vim shortcut layer on my keyboard (there were only four shortcuts anyway), so I cut that entirely. I also realized that, most of the time, I'm only going to the numeric layer for one key at a time, and I sometimes forget I'm on that layer.
So, I replaced my persisting layer toggle key with a one-shot layer switch.So far, it's working well, though I do have moments of disorientation, like when you're playing Zelda Second Quest, and the dungeons aren't where you think they are.