Utility clownmdemu - The Greatest Mega Drive Emulator Ever (Someday)

[Crosspost from the blog]

I was hoping to release this update after I finished adding support for the Mega CD, but progress on that has ground to a halt. Instead, the focus of this update will be the frontend! From features to refactoring to an entire port, here's what's new in clownmdemu!

Frame Advance
Frame Advance is a fairly mundane feature for typical users but a very handy tool for developers: it allows for advancing the game by only a single frame, which can be useful for examining a bug or verifying that animations occur correctly. It is activated by pressing the fast-forward hotkey while the emulator is paused, and can be combined with the rewind key to advance to the previous frame instead of the next.

At first, I figured that this would be an awkward feature to add, for code-flow reasons. However, the opposite turned out to be true: it couldn't have been more convenient to add!

Debug Logging Window

Previously, the frontend would output debug information to the terminal (known to you Windows users as the command prompt), but when emulation goes catastrophically wrong, so much debug information is output to the terminal that it causes the entire emulator to slow to a crawl and becomes unresponsive. To avoid this, debug information is now output to a Dear ImGui window. By default, logging is disabled, minimising the performance impact of the emulator issuing debug information in the background. Even when enabled, however, performance is still far better than before, with instances where the window is flooded with debug information still resulting in a responsive emulator. If, for whatever reason, information absolutely must be output to the console, then an option is provided to reenable it.

There is one downside to this: the logging window is very RAM hungry. However, to avoid crashes, the logging window will detect when RAM has ran out and clear the log, freeing much of the RAM and allowing the emulator to continue running normally.

Disassembler
In a Mega Drive cartridge or Mega CD disc, code is stored as machine code, which is essentially just a bunch of hexadecimal numbers.

This is Sonic 1's boot-code in machine code form:

This is very hard for a person to read, so the more practical way of viewing code is in assembly form. The process of converting machine code into assembly is called 'disassembly'. My emulator's frontend is now capable of automatically performing disassembly, allowing the code of the emulated game to be viewed with ease!

This is Sonic 1's boot-code in assembly form:

The 68000 disassembler is actually an off-shoot of my CPU interpreter library - clown68000. This is because, long ago, I made an effort to split the interpreter into two parts: the interpreter itself, and a kind of database library that describes the functionality of the 68000. The idea behind this was to minimise the work that would have to be redundantly repeated every time the interpreter was rewritten, or a new 68000-related tool was made. This plan ended up paying off, as the database library was not only used to automatically generate part of the interpreter, but now components of it are used by the disassembler. In particular, the instruction and operand decoding were made trivial by the library, allowing a quick-and-dirty prototype to be made in just one day. Being closely related to each other, the two now share a Git repository.

While this will surely be useful for the developers of ROM-hacks and homebrew, I'm hoping to use this for reverse-engineering games that don't work correctly in my emulator. In particular, I'm hoping that this will be the key to getting Sonic CD working.

Frontend Refactor
Originally, the frontend was written in ANSI C, but it became C++98 when Dear ImGui was integrated, as that was the language that it (and, more importantly, its API) were written in. Later, Dear ImGui switched to C++11 and the frontend switched with it. Throughout all of this, however, the frontend never made much use of C++'s features, remaining very C-style. This has finally changed, as now the frontend has been converted to proper C++: 'NULL' has been replaced with 'nullptr', references are used instead of pointers, casts are now C++-style, and many components has been split off into their own classes. The migration to modern C++ features should improve the frontend's code hygiene, such as through the more-explicit nature of C++-style casting.

The frontend has also been refactored to split much of the code from 'main.cpp' to separate files. 'main.cpp' was getting excessively large, which was making it hard to navigate, so I began splitting whatever I could to other files. This was done long ago with the debug menus, but poor encapsulation made them a bit of a mess. In this new wave, the audio stream was split to its own self-contained class, as was the window subsystem, the emulator wrapper, and the file dialogs.

68000 Interpreter Refactor
It is possible for an emulator to be excessively accurate: an emulator could recreate the exact behaviour of a console on the transistor level, but the game running on the emulator would behave exactly the same as it would on an emulator that recreates the console's behaviour on a logic-gate level. The only difference is that one emulator is far more performance-intensive than the other. There comes a point where additional accuracy does not benefit emulation, as the external behaviour is already perfectly accurate, even if the internal behaviour is not.

This same issue applies to CPU emulation. You may have heard the term 'cycle-accurate' before; this refers to the accuracy of the emulator's timing: on a given cycle (a fixed time-step that things like CPUs operate on), a cycle-accurate emulator's state (RAM, CPU registers, etc.) will be the exact same as the original hardware on the same cycle. Neither of the CPU emulators in my Mega Drive emulator are cycle-accurate: the closest one is my Z80 emulator, which is accurate only during the cycles on which an instruction ends. This is because instructions are executed over the course of multiple cycles on a real Z80, while my emulator completes every instruction instantly in a single cycle, and simply does nothing for the remaining cycles. This may sound like an unimportant technical detail, but this does affect when certain operations (such as accesses to the VDP's ports) occur. This creates a difference in behaviour that could cause the emulated game to behave differently. This is imperfect emulation.

To correct this, I could rework my CPU emulators to be 'truly' cycle-accurate. That is, on each cycle, the emulator does the exact same tasks as a real CPU. However, like described earlier, this would be overkill: many of the tasks done per cycle are strictly internal to the CPU, and are completely invisible to anything outside of it. Since a game exists outside of the CPU running it, this means that these internal behaviours are completely irrelevant to a game running exactly the same as it does on a real console. One example of such an invisible task is adding two numbers together using the CPU's registers. In contrast, a task that would not be invisible is sending the result of that addition to RAM, since the RAM exists outside of the CPU.

Since cycle-accuracy is too accurate, and being accurate only on the cycles that instructions end is not accurate enough, I have settled on the perfect in-between: being accurate only on the cycles that a bus event occurs on. A bus event is a transfer of data between the CPU and the rest of the system, making it the fundamental external task of a CPU. Hopefully you can tell where I am going with this: by making bus event behaviour accurate, I make all external behaviour accurate, and therefore make the behaviour of software running on the CPU emulator 100% identical to a real console (at least when putting aside other emulated components of the console that can influence code execution such as the bus and VDP).

With that massive preamble out of the way, I have been refactoring the 68000 emulator in preparation for making it accurate on a bus-event level. This involved splitting the instructions into multiple steps, so that they can be completed over the course of multiple bus events instead of all at once as they did before. Each step is made into a unique function, then pointers to these functions are grouped into lists, one per instruction, and then, when an instruction needs to be ran, it is done by calling each function in the instruction's list, with bus events occurring between each function call.

In hindsight, by doing all of this, I have implemented microcode into my 68000 emulator. While the emulator is still not yet accurate on a bus event level, all of the preparatory refactoring is now out of the way, leaving me with far less work to do to finally make the switch.

Improved High-DPI Support
On Windows, the emulator's frontend already enjoys high-DPI support, making its fonts as crisp as possible! I assumed that the code for this would work on other platforms too, but I heard from a Mac user that it was horribly broken!

What it should have looked like:


What it looked like:


Without access to a Mac, and no understanding of what could possibly be going wrong, I was unable to fix this. Around the same time, I noticed that the DPI scaling was overkill on Linux, causing the font to be far larger than it needed to be. With all of these issues in mind, I disabled high-DPI on any platform that wasn't Windows. This left Mac and Linux with blurry fonts (and blurry rendering in general), which I wasn't happy with.

High-DPI is one of the SDL2 library's shortcomings: while its support for the feature on Linux and Mac is good, Windows support was entirely missing for the longest time. More recently, it gained some support for Windows, but it had to be activated in a different manner to the other platforms and came with a bunch of caveats and edge-cases, defeating the whole point of SDL2 being a platform abstraction library.

I wanted to try again to implement high-DPI support on Linux, so I enabled the dormant original high-DPI code to see what would go wrong. When the emulator window appeared, I was surprised to see that it exhibited the exact same bugs as on macOS. As it turns out, on Linux (with Wayland, at least), SDL2 uses the same high-DPI mechanism as it does on macOS. This meant that, if I could fix high-DPI on Linux, then high-DPI on macOS would be fixed too!

At first, I tried supporting high-DPI the same way that Dear ImGui's SDL2 example does, but it was terrible: all it does is force SDL2 to convert the rendering coordinates from pixel coordinates to 'device coordinates', to match the mouse movement events and window size, which are also measured in device coordinates. Device coordinates remain the same across DPIs, whereas pixel coordinates get smaller and smaller, which is why using device coordinates universally is one way to support high-DPI. The problem with making Dear ImGui render using device coordinates manifests when using a DPI that is not a multiple of 100%, such as 150%: in this situation, the font and other native-resolution graphics will not be aligned with the screen's pixels as they are rendered, causing them to appear blurry, and also aliasing is introduced to the clip rectangle logic, causing letters to occasionally be cut off at an edge. The end result is extremely ugly rendering.


(Left: Good Rendering - Right: Garbage Rendering)

Appalled by this, I instead tried the opposite approach: rather than make the coordinates consistent by converting the renderer to device coordinates, I converted the mouse movement events and window size to pixel coordinates. This did result in awkward aliasing in mouse positions because SDL2 brilliantly only exposes mouse coordinates as integers instead of floating-point numbers, but otherwise this works perfectly. This does create more work for me though, as all of the calculations for rendering the frontend must be upscaled in accordance with the DPI because a higher DPI means more pixels. However, all of the work for this was already done when high-DPI support was added for the Windows port, since everything was already measured in pixel coordinates on that platform. So, in the end, all I did was make Linux and macOS behave like Windows, and everything magically worked! Gee, if only there was a library called SDL2 that did that... that would be great...

Web Browser Port
One of the main features of my emulator is its portability: the core emulator is written in portable ANSI C, the frontend is written in C++11 and only dependant on cross-platform libraries such as SDL2 and Dear ImGui, and the build system is CMake. In addition, while the build script supports linking system libraries, it is able to build these libraries manually if they are absent. All of this combined made the emulator almost immediately compatible with Emscripten!

For those that do not know, Emscripten is a toolchain that allows C and C++ to be compiled to either JavaScript or WebAssembly, instead of the usual CPU-specific assembly. JavaScript and WebAssembly can be embedded into a website, allowing it to be ran in a web browser. Emscripten also exposes various APIs to C and C++, allowing code to interact with the web browser to do things like render graphics and receive keyboard and mouse input. SDL2, which serves as a platform abstraction layer, supports Emscripten's APIs, granting software that is built upon it compatibility with Emscripten.

The 'main' Function
There is, however, one incompatibility between typical C/C++ code and Emscripten: the 'main' function. Normally, the 'main' function only returns when the program closes. For a constantly-running program like an emulator, this means that there needs to be an infinite loop inside the function. But, with Emscripten, infinite loops are not allowed. Because of this, software that targets Emscripten must instead use the function to register a callback, which Emscripten will then run on the function's behalf a specified number of times per second. This callback can be used to iterate the program in the same way that the function's infinite loop would have.

Because of this incompatibility, I did need to refactor my emulator's frontend slightly. It resulted in some nice decoupling, however: now, the emulator frontend and the 'main' function are entirely separate, with the frontend occupying its own source file and exposing an interface that allows for initialising, updating, and deinitialising the frontend. The 'main' function is the sole occupant of the 'main.cpp' source file, calling the frontend's iteration function in an infinite loop in standard builds, and registering it as a callback in the Emscripten build.

File IO
Another one of SDL2's shortcomings as a platform abstraction library is that it does not provide any way to create a file dialog to allow the user to select a file for the program to load or save. Because of this, my emulator's frontend manually implements these file dialogs for each platform, using the Win32 API for Windows, and Zenity and kdialog on Linux and the BSDs. Since Emscripten is compatible with none of these, it requires its own platform-specific code.

I was able to find a handy MIT-licensed single-header-file library that implements Emscripten file dialogs, however, its API was much higher-level than what the frontend was designed for. The reason for this is that, due to running in a web browser, software suffers from strict limitations, with one such limitation being that files on the user's computer cannot be directly accessed. Instead, software must ask the browser to access the file on its behalf.

Because of this, the library does not simply return the path of the file that the user selected: instead, for loading a file, the library makes the browser read the whole file into a memory buffer and then returns a pointer to it, and, for saving a file, the library takes a pointer to a memory buffer and makes the browser write its contents to the file.

While, previously, the tasks of reading and writing data to and from a file were the responsibility of the frontend, they now had to become the responsibility of a library. To achieve this, file IO had to be abstracted to operate purely on sending and receiving buffers instead of file paths. This abstraction layer would be implemented as a C++ class - the 'FileUtilities' class - which acts as a wrapper around the 'emscripten-browser-file' library in Emscripten builds, and standard file IO in non-Emscripten builds.

The Result
With all of this work complete, my browser is able to load and run games within the confines of a web browser!


What is great about this is that I can link to it directly from my blog's Projects page, allowing anyone that is interested in the emulator to try it out! It would be nice to do this with some of my other software too, such as ClownMapEd. With some work, I could even make this auto-load my various ROM-hacks, allowing them to be tried-out in the browser too!

If you're interested in trying this port out, you can find it at clownmdemu.clownacy.com.

Vastly-Improved Audio Playback
I have never been happy with the frontend's audio delivery system: due to emulation being synchronous, the frontend is not simply able to render audio on command, instead needing to wait for audio to be generated by the emulator before it is able to pass it on to the operating system to be played through the user's speakers. In order for there to not be gaps in the audio, audio must be generated as fast as it is played. However, the audio-generation process is extremely complicated (involving generating the audio output of two different sound chips - FM and PSG - at two entirely different sample rates - 53kHz and 224kHz - and then resampling them both to a common sample rate - typically 48kHz - and then mixing them together), meaning that audio is never exactly the correct size. Because of this, it's possible for audio to be generated slightly faster than it is played. When this happens, a gradual build-up of audio occurs, making the audio more and more delayed. For the longest time, this was worked-around by detecting when too much audio was being generated and simply throwing away some of the excess. This kept the audio from ever exceeding a certain amount and becoming latent, but also caused the audio to audibly 'skip', producing irritating popping and crackling sounds.

This problem came to a head when adding support for 50Hz (PAL emulation) to the Emscripten port: the popping was so annoying that I simply had to do something about it. I decided that I would make the frontend resample (stretch or squash) its audio in accordance with how much audio build-up there was. This way, there would be no skips in the audio, making playback as smooth as could be. I spent the next few days experimenting with different methods of making the audio playback speed self-adjusting. It was a demoralising few days, as everything I tried failed in one way or another: either the self-adjustment would be too weak and it would fail to respond to fluctuations in framerate, causing audio skips or latency, or it would be too aggressive, causing the audio to constantly warble as the pitch was greatly increased, only to be greatly decreased to prevent too much audio being generated, only to be greatly increased again to prevent too little audio being generated, and so on and so forth.

On the verge of giving up, I remembered that somewhere in RetroArch's documentation was a paper that described a method of 'dynamic rate control' that was just what I needed. I found it, implemented its formula, and was happy to hear it work perfectly! The frontend now adjusts the speed of its audio to always maintain a certain amount of generated audio at all times, preventing both skipping and latency. One of the RetroArch formula's features is that the speed adjustment is kept to a minimum, preventing the change in pitch from being audible. At last, the frontend's audio system is something that I can be proud of!

Unicode File Path Support on Windows
To end things off, here is a feature that I implemented just now! The frontend no longer has any problem with accessing files and folders whose names contain non-ANSI characters. This means that if you have a file whose name is written in Japanese in a folder whose name contains emojis, then the frontend will be able to load it! Previously this would just fail, as the internal way of encoding strings was incompatible.

Unfortunately, the font system doesn't support the entirety of Unicode, so many characters will appear as question marks when viewed in the frontend itself, like in the 'Recent Software' list.

Closing
So here it is - one big update! How typical of me to want to implement Mega CD support, only to end up doing a whole bunch of other things instead. Hopefully these new additions will be enough to tide people over until then!

Source code: https://github.com/Clownacy/clownmdemu-frontend
Windows executable: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.5
Try it in your web browser: clownmdemu.clownacy.com

 

Oh no... it turns out that the crackling you're hearing isn't related to the audio delivery system - it's the FM-PSG mixer. The FM and PSG emulators output separate streams of audio, and these streams need to be resampled and mixed. The trouble is, after resampling, the FM and PSG audio tend to be slightly different lengths, and the emulator will discard the difference. During certain sounds (such as the 'twinkling' during Sonic 2's title screen intro), dropping these samples produces a crackling sound. I'll have to figure out a proper fix for this.

 

v0.5.1
Try it in your web browser: clownmdemu.clownacy.com
Windows EXE: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.5.1
Source code: https://github.com/Clownacy/clownmdemu-frontend

Even-Further-Improved Audio Playback
It was pointed out to me that my Mega Drive emulator has another source of audio distortion: while v0.5 had managed to fix the audio skipping that was caused by audio being generated too quickly, there was still crackling that was caused by the FM or PSG audio being truncated by 1-3 samples every frame.

The reason for this is that the audio resampling process is prone to rounding error, meaning that if 400 samples were desired, 401 samples may be produced instead. Because there are two resamplers running simultaneously (one for the FM audio and one for the PSG audio), it's possible for each of them to output a different number of samples. When this happens, the number of samples which are output is capped to the smaller of the two. This causes the audio with the larger number of samples to skip slightly, creating a crackling noise.

At first, I tried fixing this by making the recently-added dynamic audio sizing system apply to the FM and PSG audio separately, but, while this did eliminate the crackling noise, it caused the two audio sources to frequently desynchronise with each other, causing music to play with disjointed timing.

A proper fix was achieved by reworking the resamplers to always generate an exact number of samples. By doing this, the FM and PSG audio are always perfectly synchronised, and no excess audio frames are produced. At last, perfect audio delivery!

Frontend Debugging
While working on v0.5's dynamic audio sizing system, I became interested in monitoring the audio buffer. I had added some code to print statistics to the debug log, but this was messy and so it was never included in the released version. Even once the system was complete, I wanted to be able to keep on eye on how it was operating to ensure that it was working correctly. Instead of keeping the hacked-together debug logging code around forever, I made it into a proper debugging menu!



Various useful little bits of information are displayed here, particularly the target and average numbers of audio frames. The closer these numbers are; the better: if the average becomes too low, then audio drop-out occurs, and if the average becomes too high, then latency occurs. Additionally, there are statistics for the framebuffer upscaling, which is nice for ensuring that the non-integer scaling is working optimally. The list of SDL2 drivers has been moved here from the About menu too, making it easy to see which APIs SDL2 is taking use of. Here you can see my painfully contrarian OpenGLES2, Wayland, and PipeWire configuration. Because OpenGL 2.1, X11, and PulseAudio are so last year.

C++ Refactoring
The frontend has been receiving more C++ refactoring to make it less C-like and more exception-safe. Particularly, instead of using 'fake' constructors and destructors that are just regular methods, the frontend's classes have been converted to use real constructors and destructors. This had many implications for field lifetime and initialisation, requiring even further refactoring. I've also eliminated the need for manually-defined destructors from a number of classes, leveraging smart pointers and the like to make the automatically-generated ones sufficient.

The strictness of object-oriented code has helped expose a number of design deficiencies already, such as the configuration loader calling methods of the audio output object before it had even been initialised. It has also encouraged better factoring of the code by nudging me towards splitting the audio device logic from the audio queuing system, which coincidentally makes it simpler to port to other APIs. Overall, I've enjoyed getting to know modern C++ better, and think that it benefits the codebase quite a bit.

Hopefully, this shake-up to the codebase has not introduced any bugs nor instability. If it has, expect to see some small hotfix updates in the coming days.

 

That was pointed out back in April too. The EXE on GitHub hasn't been meddled with, as proven by its file hash being the same as when I first uploaded it (SHA256: bceec93b869615b6c078d8abf1bd57eca923a1e613089fd857227d50452c36fe). Unless Visual Studio 2019 is inserting malware into the EXE as it's being compiled and linked, or one of the program's dependencies (SDL2, Dear ImGui, inih) has malware bundled into its source code, then your antivirus is reporting a false positive. Perhaps the antivirus doesn't like that the EXE immediately loads two files when it's started (it loads the emulator settings as well as the cached Dear ImGui window positions).

 

Well I think it's suspicious that Chrome knows what files people do and don't often download. :U

 

[Cross-post from the blog]

Recently, I fixed a couple of inaccuracies, and I thought they would be interesting to read about:

Fixed the Title Cards in Sonic Crackers
I've tested many Sonic games and ROM-hacks in my emulator, but I've never tried the famous Chaotix prototype - Sonic Crackers. If I did, I would have noticed an obvious bug that affects its title cards:


(It may be hard to believe, but this is not what it's supposed to look like).

This bug had flown under the radar for over two years until it was helpfully pointed out to me in a GitHub Issue. I found this bug very interesting because it appeared to stem from a mature and battle-tested part of the VDP emulator, so I was surprised that there was still an inaccuracy within it.

Upon investigation, I discovered that the bug in my emulator was being triggered by a bug in Sonic Crackers! The tiles for the title card are uploaded using DMA transfers, however the game's DMA transfer routine is bugged: immediately after performing the DMA transfer, the routine attempts to reupload the first word of tile data (this is presumably to work around a known bug in early Mega Drives that causes the first word of a DMA transfer to be corrupt), however, it accidentally only writes half of the address command. This is important, as, when half of an address command is written to the VDP, a 'latch' flag is set to signal that the next word written will be the command's other half. The DMA transfer routine will then loop and begin to prepare another DMA transfer, starting by writing a register command for the DMA length. Because the latch flag is set, however, this register command is instead interpreted as the second half of an address command, completely breaking the DMA transfer, causing the bug that is seen above.

This is not the case on a real Mega Drive: there, the register command is processed correctly, allowing the DMA transfer to successfully occur. This shows that the latch flag only applies to words that are not register commands, whereas my emulator was making it apply to all words, register command or otherwise.

But that is not the only issue that this bug exposes: later in the routine, an address command is written to the VDP in order to trigger the DMA transfer. With the latch flag still set, the emulator would receive the first half of the command, notice that the latch flag is set, and use it as the previous, incomplete command's second half. As with before, this breaks the DMA transfer.

Once again, this is not the case on a real Mega Drive. The most likely reason being that the latch flag is cleared whenever the VDP receives any word, even if it isn't part of an address command, preventing the DMA transfer from being broken by the incomplete address command.

Addressing both of these inaccuracies finally allows the title card to work properly:


(It may be hard to believe, but this is what it's supposed to look like).

This issue illustrates why accuracy is so important to emulators: while well-behaved code will obediently follow the rules to the letter, bugs will invoke all kinds of edge-cases and undocumented behaviour that no sane code ever would. In this case, Sonic Crackers' bugged code relied on quirks of the VDP's address command latching behaviour that would never be tested in normal circumstances.

Fixed the Data Select Background in Sonic 3
A longstanding bug has finally been fixed! Ever since Sonic 3 first booted in this emulator, the background of the Data Select menu has looked like this:



With the help of the emulator's debugging menus, I could see that the background's rows had been split into two groups:



This struck me as very, very odd, as there is no way that this could ever work on a real Mega Drive: what's happening is that the game has set the VDP to render in 128x32 mode, but it has loaded the background in 64x32 mode. There is no bug here: this is what the game actually does, and the VDP is doing exactly what it should in response.

So why is it that this works on a real console? The foreground needs to be in 128x32 mode to fit its graphics, and it's not possible for Plane A and Plane B to be different sizes. So how could Plane A possibly be correctly displaying 64x32 data in 128x32 mode?!

That is a trick question - did you figure it out? Here's the answer: the background is not displayed by Plane A - it's displayed by the Window Plane. Yep, that obscure feature that's never used by Sonic 1 and 2 is used for a menu background of all things in Sonic 3.

But what's special about the Window Plane? Well, one of its "features" is that, unlike Plane A, it cannot scroll. Because of this, being 128x32 would make no sense for the Window Plane, as most of it would be forever off-screen. As a result, the Window Plane is hardcoded to either 64x32 and 32x32, depending on what resolution the game is rendering at. I never knew about this hidden gimmick, so when I added Window Plane emulation almost a year ago, I made it match the size of Plane A and Plane B.

Fixing this makes the background display correctly at long last:

 

Thanks for the report! I have no idea why Thunder Force IV is using the IN instruction, considering that it shouldn't do anything at all on the Mega Drive.

It seems that the game needed the STOP instruction to be implemented in order to work properly. My guess is that it relies on that instruction to set the interrupt mask properly, otherwise interrupts fire at inopportune times, causing the graphical corruption.



It's looking good now!

 

Ooooh, so that's why the security code doesn't work in my emulator! Thanks, that's another mystery solved.

They rely on being able to read the bus request register to tell when the bus request is complete. Without it implemented, they were waiting endlessly for it. Both games now boot.


This also gets Combat Cars and Micro Machines to boot too.

 

(Cross-post from the blog).

v0.5.2
Here is a small update that fixes a few inaccuracies in order to get more games working.

Try it in your web browser: clownmdemu.clownacy.com
Download: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.5.2

Fixes for Sonic 3's Data Select and Sonic Crackers' Title Cards
I have already covered this in another post, but for those not in the loop, the VDP emulation logic contained a few inaccuracies that caused visual corruption in a couple of Sonic games. These issues have now been addressed, allowing these games to work as they do on real Mega Drives.




Fix for Thunder Force IV's Gradual Corruption
A user, Pinkerton, recently pointed out a very strange bug: the game Thunder Force IV would start off working fine, but gradually corrupt over time.



This is unlike any bug that I have ever seen before in this emulator: games usually either work or they break completely - they do not slowly corrupt little by little whilst otherwise remaining completely stable.

An invaluable tip that Pinkerton provided was that the debug log kept mentioning the use of the 'STOP' instruction, which is a 68000 CPU instruction that was not yet emulated. I noticed that the Sega splash screen was playing at too fast a speed, so it was apparent that the screen was using the STOP instruction for controlling its framerate. The rest of the game played at the correct speed, and the debug log stopping complaining about the STOP instruction after this screen was over, so it was clear that only the Sega screen used the STOP instruction. Still, I figured it would be worth implementing the STOP instruction to get the screen working properly, then afterwards I could focus on the graphical corruption.

The STOP instruction is not used by the main Sonic games, so I am not very familiar with it, hence why it was not emulated. Upon researching it, I found that the instruction essentially causes the 68000 to cease processing instructions until an interrupt occurs. Given that there is an interrupt - Vertical Interrupt - that occurs once per frame, this instruction becomes a useful way of idling the CPU until the next frame. It would be unfortunate if no interrupts were set to occur, as the CPU would end up waiting forever, so the STOP instruction is also able to set which interrupts are enabled and disabled.

Implementing the STOP instruction only took a couple of minutes and a few lines of code, and, with that done, the Sega screen was running at the correct speed. Surprisingly, this also fixed the gradual graphical corruption in the game itself!

But how? The main game does not even use the STOP instruction! My guess is that the game relies on the STOP instructions in the Sega screen to set the interrupts to a sane configuration, and that, without it, the game would receive interrupts whilst performing sensitive operations like communicating with the VDP, causing said communication to be corrupted and leading to the graphics being broken.

Regardless, I am glad to have fixed the bug. Thunder Force IV was a game that I was meaning to try out sometime, so it's nice that I can play it in my emulator.

Get Combat Cars, Micro Machines, Dynamite Heady, and Gunstar Heroes to Boot
For as long as my emulator has existed, it was unable to boot Combat Cars and Micro Machines, even after massive improvements in accuracy to both the 68000 and Z80 interpreters. This has always stumped me.

After informing me of the bug in Thunder Force IV, Pinkerton also noted that two Treasure titles - Dynamite Heady and Gunstar Heroes - failed to boot. With the new 68000 disassembler, I set about diagnosing the issue.

I was surprised to see that the games were stuck reading from the Z80 bus request register. To my knowledge, reading that register will provide a flag that indicates whether the bus request is in-progress or not. The Sonic games frequently wait for this flag to be clear. However, these Treasure games were different: they were waiting for the flag to be set instead. Because my emulator does not work like an actual Mega Drive, bus requests complete instantaneously, so the flag is never set. Because the flag is never set, the games wait indefinitely, resulting in their failure to boot.

Upon looking into the details of the bus request register, I discovered that I was wrong about what reading the register does: the flag provided does not indicate whether the request is in-progress, but rather whether the bus is held or released. What this means is that, if you're releasing the bus, you wait for the flag to be set, and, if you're requesting the bus, you wait for it to be clear.

With this inaccuracy corrected, Dynamite Heady and Gunstar Heroes now boot!



Not only that, but I later noticed that Combat Cars and Micro Machines finally boot too!



According to Pinkerton, this change gets Alien Soldier to boot as well!

Bonus: Struggling with Windows XP Compatibility
Going back to at least the second release of this emulator, I have tried to maintain compatibility with Windows XP. Why? Because I can, and because it was pretty convenient: the frontend's main dependency, SDL2, supports Windows XP, and Visual Studio provides a (deprecated) toolchain for targeting Windows XP. I also figured that maintaining XP support was a good way of benchmarking the frontend's portability: if it doesn't run on Windows XP, then what are the odds of it running on niche homebrew platforms?

Recently, I set up a Windows XP virtual machine just to test this emulator, so I could enjoy the satisfaction of seeing it running. I double-clicked the EXE and... it failed. No error messages at all; it just failed.

I did some research, and found that Visual Studio broke its Windows XP compatibility after version 16.4. This affects the runtime library, which I normally link statically. Switching to dynamic linkage and installing an older runtime library on Windows XP should bypass this issue, so I did just that. The emulator still crashed.

At this point, I considered Visual Studio to be a dead-end, and decided to switch to an open-source toolchain instead. MSYS2 would have been a good option, but it not only dropped support for Windows XP, but also Windows 7, so that would not work here.

That left one more option: Arch Linux's MinGW-w64. Years ago, I used it to produce Windows XP-compatible builds of the Sonic 1 and 2 remasters. I rebooted into Linux, installed the appropriate packages, cross-compiled the emulator, copied the binary over to my virtual machine, double-clicked it, and... it failed.

This time was slightly different: the error message mentioned a missing DLL import, that being the function GetTickCount64, which was added in Windows Vista. This was a known issue with MinGW-w64's libwinpthread. A recent commit from the end of 2023 should have fixed this, but it appears that this has not yet been released in an update. Fortunately, an Arch User Repository package is available for compiling the latest libwinpthread sources. With that done, I built a new binary and ran it. It still crashed.

Well, damn. Now what? At this point I was well and truly defeated, and gave up. Windows XP support with a modern toolchain is infeasible in 2024... or so I thought. Just this morning an idea occurred to me: what if SDL2 was failing to initialise because hardware acceleration was not available? VirtualBox no longer provides hardware acceleration to Windows XP guests, and SDL2 tries to create a hardware accelerated renderer before falling-back on its software renderer, so perhaps the emulator would work if I just forced SDL2 to skip hardware acceleration. To do this, I had to set the 'SDL_RENDER_DRIVER' environment variable to 'Software'. After that was done, I ran the EXE and...



It works!

Unfortunately, this is not as convenient as compiling with Visual Studio on Windows, but I think that it is acceptable given that I already have to switch to Linux to compile the Emscripten port. This may be the very first release of the emulator whose pre-built EXE actually works on Windows XP! Though, that does not mean that the EXE will run badly on modern versions of Windows either: it will still support high-DPI displays and leverage newer Windows APIs like Direct3D 11 and WASAPI! That is the magic of SDL2, dynamically bridging the gap between the program and the operating system's underlying APIs!