With this tool running on hardware (You can run it on an emulator, but none of the bits will have any effect) I was able to decode what the different bits meant and did on my Model 1 VA5 NTSC Mega Drive with a Sega CD attached (No difference was observed with it removed):
Bit 14 set = Sprites hidden (perhaps part of a 'debug mode' register)
Bit 13 set = Sprites hidden (perhaps part of a 'debug mode' register)
Bit 12 set = Sprites hidden (perhaps part of a 'debug mode' register)
Bit 9 set = PSG disabled
Bit 8 set = BG layer hidden, dots at top of screen, line at right and sprites hidden (besides artefact of one all the way to the right.)
Bit 7 set = FG starts using an odd tile for everything, BG uses odd palette - possibly reading corrupt data. There's small black lines on the side of the rainbow bars (BG has been disabled for clarity)
Bit 6 set = Display is disabled
Bit 5 set = Some sort of grid pattern, all planes seemingly hidden (Causes some brokenness in A plane at $C000)
Bits 7 and 8 set = Garbles FG and sprites.
Bits 15, 14 and 8 set = BG hidden as well as sprites disappear.
Bit 10 or 11 toggled = Causes a click in the audio (Thanks to Chilly Willy for pointing this out — I didn't listen carefully when I ran this test)
Here is a video of the programme in action on hardware, with various bits being set. (The audio is slightly out of sync)
I'm interested in seeing if other models of the MD (and other VDP chips) deal with these modes differently — so if you've got a flashcart, please try setting the bits on your hardware and report back what happened, as well as your hardware version and VA revision. You can download the ROM here.
Lastly, I'd like to thank Jorge and Tiido for giving me the idea to toy around with this by suggesting I figure out if the VDP has more access cycles in one line if a specific plane was disabled. And I also apologise for the shitty quality of the video — my composite cable disappeared and my capture card likes being an asshat.