don't click here

Utility clownmdemu - The Greatest Mega Drive Emulator Ever (Someday)

Discussion in 'Technical Discussion' started by Clownacy, Jun 23, 2022.

  1. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    [Cross-post from my groovy blog ;)]

    It's been too long, but finally my emulator has an update!

    Since the first release, the emulator has been greatly optimised, some inaccuracies in the 68000 interpreter have been addressed, and the occasional missing CPU instruction has been added. Compatibility with games should be a bit better than before, but still not great as many essential features of the Mega Drive are not emulated.

    The standalone frontend has had some extra debug menus added, which allow you to view the registers of the YM2612, 68000, and Z80:

    [​IMG]

    New to the emulator is a libretro core frontend, allowing the emulator to be used by libretro implementations such as RetroArch. It lacks the debug menus of the standalone frontend, but makes up for it with features that libretro cores get for free, like customisable controllers and shaders:

    [​IMG]

    In theory, the libretro core should provide a simple way of getting this emulator running on a variety of platforms: just compile the core into a library (static or shared), and use it in tandem with a libretro frontend such as RetroArch.

    During the development of this update, I have set up a test suite for the 68000 interpreter which allows me to check that each instruction does as it is supposed to. It was this test suite that notified me of how the word-size ADDA, SUBA, and CMPA instructions were pitifully broken. I'm surprised that this didn't break Sonic 1, 2, or 3&K, but it did break Linux.

    Yes, Linux: to test my 68000 interpreter, I extracted it from this emulator and used it to create an emulator for the 68 Katy, a hand-built computer that runs Linux. You can read more about it here.

    I also made a small benchmarking tool which measures the speed of the core emulation logic. This is useful for measuring the impact of optimisations and the difference in speed between platforms.

    Overall, this has been a rather incremental update. Rather than being focussed on optimisation and refactoring, I hope that the next update will be focussed on improving compatibility and emulating more features of the Mega Drive.

    You can download the standalone frontend here, and the libretro frontend here.
     
    Last edited: Feb 10, 2023
  2. I had to do a lot of re configuring to go along with the components being separate, took a good while but no too bad. The only problem now is that for some reason it won't compile, it seems to be something with imgui that keeps it from compiling successfully, but doesn't get really specific, will go ahead and give what showed up in my terminal (I'm on Linux and great work btw!):

    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clownmdemu.h:7,
    from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:7:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon.h:19: warning: "CC_SIGN_EXTEND" redefined
    19 | #define CC_SIGN_EXTEND(type, bit_index, value) (((value) & (((type)1u << (bit_index)) - (type)1u)) - ((value) & ((type)1u << (bit_index))))
    |
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:6:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon/clowncommon.h:171: note: this is the location of the previous definition
    171 | #define CC_SIGN_EXTEND(type, bit_index, value) (((value) & (((type)1 << (bit_index)) - 1)) - ((value) & ((type)1 << (bit_index))))
    |
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clownmdemu.h:7,
    from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:7:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon.h:8:20: error: ‘cc_false’ conflicts with a previous declaration
    8 | cc_false = 0,
    | ^
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:6:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon/clowncommon.h:139:9: note: previous declaration ‘<unnamed enum> cc_false’
    139 | cc_false = 0,
    | ^~~~~~~~
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clownmdemu.h:7,
    from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:7:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon.h:9:19: error: ‘cc_true’ conflicts with a previous declaration
    9 | cc_true = 1
    | ^
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:6:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/clownmdemu-frontend-common/clownmdemu/clowncommon/clowncommon.h:140:9: note: previous declaration ‘<unnamed enum> cc_true’
    140 | cc_true = 1
    | ^~~~~~~
    /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp: In function ‘void SetSoftware(unsigned char*, size_t, const ClownMDEmu_Callbacks*)’:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:487:9: error: ‘ClownMDEmu_Parameters_Initialise’ was not declared in this scope; did you mean ‘ClownMDEmu_State_Initialise’?
    487 | ClownMDEmu_Parameters_Initialise(&clownmdemu, &clownmdemu_configuration, &clownmdemu_constant, &emulation_state->clownmdemu);
    | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    | ClownMDEmu_State_Initialise
    /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp: In function ‘int main(int, char**)’:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:636:66: error: too many arguments to function ‘bool ImGui_ImplSDL2_InitForSDLRenderer(SDL_Window*)’
    636 | ImGui_ImplSDL2_InitForSDLRenderer(window, renderer);
    | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
    In file included from /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:12:
    /home/segamaniac/Documents/genem/clownmdemu-frontend/libraries/imgui/backends/imgui_impl_sdl.h:28:25: note: declared here
    28 | IMGUI_IMPL_API bool ImGui_ImplSDL2_InitForSDLRenderer(SDL_Window* window);
    | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /home/segamaniac/Documents/genem/clownmdemu-frontend/main.cpp:753:49: error: ‘ClownMDEmu_Parameters_Initialise’ was not declared in this scope; did you mean ‘ClownMDEmu_State_Initialise’?
    753 | ClownMDEmu_Parameters_Initialise(&clownmdemu, &clownmdemu_configuration, &clownmdemu_constant, &emulation_state->clownmdemu);
    | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    | ClownMDEmu_State_Initialise
    make[2]: *** [CMakeFiles/clownmdemu-frontend.dir/build.make:82: CMakeFiles/clownmdemu-frontend.dir/main.cpp.o] Error 1
    make[1]: *** [CMakeFiles/Makefile2:143: CMakeFiles/clownmdemu-frontend.dir/all] Error 2
    make: *** [Makefile:103: all] Error 2
     
  3. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    It sounds like you checked-out the wrong commit of the 'clownmdemu' submodule: the file 'clownmdemu/clowncommon.h' shouldn't exist anymore, and instead there should only be 'clownmdemu/clowncommon/clowncommon.h'. Somehow you have both.

    The separation of the various components shouldn't require much work to get working, as long as you're using Git to clone the main repo: the components are included as submodules, so all you should have to do is run the command 'git submodule update --init --recursive' within clownmdemu-frontend's folder, and the correct versions of the submodules will be downloaded.
     
  4. Brainulator

    Brainulator

    Regular garden-variety member Member
    For me, GitHub Desktop downloads all submodule files when I clone a repository to my own PC.

    By the way, because I've been using Visual Studio 2019 to build several of your projects, could you please add these to .gitignore wherever feasible?

    • /out
    • /.vs
    • CMakeSettings.json
     
  5. Thanks yo, it worked!! Btw the rom hack here is Sonic 1 Definitive by Inferno.
    Screenshot_20230212_135510.png
     
  6. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    I have mixed feelings about that: my projects only officially support CMake, not Visual Studio's CMake-compatibility layer. In-source builds are discouraged in modern CMake, which is why there's a directory called 'build' listed in the '.gitignore' file: junk build files are supposed to go in there, not the source directory. If Visual Studio truly supported CMake, then it wouldn't break such basic rules and require special treatment. When setting up my emulator to compile with Visual Studio, I just use CMake on the command line to generate a solution file in the 'build' directory, then the source directory remains clean.
     
  7. Brainulator

    Brainulator

    Regular garden-variety member Member
    I see. I guess I'm just a noob with this sort of thing, but then again, I've managed to build clownmdemu successfully using Visual Studio in the past. Feel free to not put those in, then.
     
  8. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    [Cross-posted from my blog.]

    Keyboard Rebinding

    One shortcoming of the standalone frontend is that it lacks keyboard rebinding: the W, A, S, and D keys will always control the Control Pad's D-Pad, and so on.

    But not anymore!

    [​IMG]

    New to the frontend is full keyboard rebinding! In addition, the default key bindings have been switched to the more common arrow keys and Z/X/C keys combination.

    Unlike some other emulators, this system allows the user to bind multiple keys to the same action: for instance, if the user wanted to bind both the 'Z' key and the 'space' key to the Control Pad's 'A' button, then they can do so!

    It would be pretty frustrating for binding customisations to be lost whenever the program is closed, so support has been added for persistent configuration: settings are saved to a file called 'clownmdemu-frontend.ini', allowing settings such as the keyboard bindings, console region, and V-sync to be remembered by the emulator.

    Options Menu

    Previously, the options would all be managed through the menu bar, but this is quite clunky as the menu bar would close after each option is toggled. To improve the user experience, the options have now been moved to a dedicated menu:

    [​IMG]

    This menu provides a much more intuitive way to change options! Additionally, each option shows a tooltip when hovered over with the mouse, allowing unfamiliar users to understand what they do!

    [​IMG]

    Default Window Sizes

    Another improvement to the user experience is that windows are now given a sane default size, meaning that they will now have a proper size when opened for the first time.

    [​IMG]
    This is dumb.

    [​IMG]
    Much better.

    Recent Software

    Opening the same ROM file over and over again is tedious, so now the emulator keeps a list of the 10 most recent files used:

    [​IMG]

    FM and PSG Debugging Toggles

    The standalone frontend has had the ability to disable individual VDP planes for ages, but now it can also toggle FM and PSG channels. A dedicated menu has been added for this:



    This feature is also available in the libretro core:

    [​IMG]

    PSG Debugger Overhaul

    The PSG debugging menu was butt-ugly before, and has been given a makeover:

    Before:
    [​IMG]

    After:
    [​IMG]

    Support for Alternate PAL Detection Method

    Previously, when playing Sonic the Hedgehog 2 with the emulated Mega Drive in PAL mode, the music would play at slightly slower speed, just like it does in the first game. This shouldn't happen.

    The reason that this was occurring was that the game relies on an alternative method of detecting the PAL video mode: by checking bit 0 of the VDP's control port. This bit should reflect whether PAL mode is enabled or not. Now that this is the case, the game properly detects and accounts for the speed difference in its music, allowing it to play at the proper speed.

    Download
    v0.3 of the standalone frontend can be downloaded here: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.3

    v0.1.1 of the libretro core can be downloaded here: https://github.com/Clownacy/clownmdemu-libretro/releases/tag/v0.1.1
     
  9. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    [Cross-post from my blog.]

    This is just a quick update to address some issues in the previous v0.3 release.

    Make FM Debugger More Compact

    The FM debugger was a bit ‘verbose’ in v0.3…

    [​IMG]

    As you can see, each channel was given its own window, which meant that it was a lot of effort to simply switch from one channel to another without just having all windows open at the same time, which would take up a lot of the screen.

    Since it's unlikely that a user would ever need to see more than one FM channel's registers at a time, these windows have all been merged into a single tabbed window:

    [​IMG]

    Fix DPI Support

    Unfortunately, after hyping it up so much in v0.3's release, the default window sizes were broken on DPIs that weren't 150% the standard. I was expecting Dear ImGui to handle DPI differences like this automatically like it usually does, but that's not the case here.

    I'll have to remember to test this frontend at alternate DPIs before each release to prevent a repeat of this mistake.

    Add a Horizontal Scrollbar to the Plane Debugger

    As the result of yet another strange quirk of Dear ImGui, horizontal scrollbars do not exist by default, even in windows that need them. This affected the VDP's Plane A/B debuggers, which only had a vertical scrollbar. By explicitly telling Dear ImGui to create a horizontal scrollbar, this issue is no more:

    [​IMG]

    User-Friendliness Improvements to Keyboard Rebinding

    Sometimes it's the small things that matter most.

    When the user is repeatedly adding key bindings, the newly-extended binding list would push the 'Add Binding' button off-screen, requiring the user to scroll down to be able to press it again. This is a small annoyance, but an annoyance nonetheless, so it has been corrected by automatically scrolling the window down after a new binding is added.

    Additionally, when selecting an action to bind to a selected key, the selected key is displayed to the user. This extra feedback allows the user to verify that they selected the correct key, instead of them being left in the dark.

    [​IMG]

    Fix Phantom Keyboard Inputs After Rebinding

    Sometimes, after rebinding the keyboard inputs, the emulated Control Pad would behave as if certain buttons were held when they are not. This was due to edge-cases in how the key-binding system works. For instance, if a key's binding were changed after it has been pressed but before it is released, then the emulator would 'forget' which Control Pad button to release when the key is released. This should no longer be the case.

    Fix Ugly Seams Around Tiles in VRAM Debugger

    Depending on the display's DPI, odd artifacts could appear around the tiles in the VRAM viewer:

    [​IMG]

    This was the result of some accidental fractional image scaling. This has been corrected to use the proper integer image scaling, eliminating the seams.

    Closing

    With this much-needed polishing complete, hopefully the next update will include some improvements to the core emulation: Window Plane, SRAM, LFO, SSG-EG, YM2612 Timers - there are plenty of things left to add.

    Download here: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.3.1
     
    Last edited: Feb 20, 2023
  10. Brainulator

    Brainulator

    Regular garden-variety member Member
    Personally, I see value in being able to see multiple FM channel debuggers at once, particularly with multi-monitor setups, though I think the tabbed view is a good default.
     
  11. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    (Cross-post from my blog.)

    frontend v0.4 & libretro v0.2

    Custom Dear ImGui Style
    Dear ImGui's default style looked a bit tacky and 'programmer art'-like to me, so I've made my own:

    Before
    [​IMG]

    After
    [​IMG]

    I’ve tried to create a typical ‘dark mode’ theme while still keeping a high degree of contrast in order to maintain good legibility. Another focus of the style is technical and artistic minimalism, hence the removal of borders, tab/scrollbar rounding, and colour (colour introduces visual noise, and rounding greatly increases the number of rendered polygons).

    Optimised VRAM Viewer
    The VRAM viewer wasn’t very efficient: it would try to render every single tile even when only a fraction of them are actually visible to the user. This is something that can be seen with Dear ImGui’s built-in debugger:

    [​IMG]

    As the debugger shows, hidden tiles are rendered above and below the section of the viewable region. This causes 4096 polygons to be drawn in total, when far fewer are actually needed.

    I discovered that Dear ImGui has a feature specifically intended to address this kind of problem: the List Clipper! The List Clipper automates the process of selecting only the elements in a list that are visible to be rendered. Applying this to the VRAM viewer only required the smallest bit of refactoring (changing a single for-loop that iterated over each tile into two for-loops that iterate over each row and then each tile in each row) as well as the removal of a small hack, and the problem was solved!

    [​IMG]

    In this example, the number of polygons has been reduced to just over 500!

    Support for the YM2612’s Timers
    Huzzah – an improvement to the actual Mega Drive emulation!

    Background
    The Mega Drive’s primary sound chip – the YM2612 – has a nifty little feature that went tragically underused by Sega: a pair of timers that are capable of raising CPU interrupts. This is notable because the only other interrupts of its kind are the VDP’s V-blank and H-blank interrupts, but those interrupts’ timings vary based on whether the console is a PAL or NTSC model. Additionally, these timers are fully configurable, allowing for the possibility of arbitrarily-timed interrupts!

    …And Sega didn’t think to connect these timers to either of the Mega Drive’s CPUs. What a waste.

    These timers are still usable, but they must be manually polled by the CPU to check if they’ve expired yet. This wastes precious CPU time, and squanders the timers’ potential for certain uses.

    Games typically use these timers for controlling the timing of their sound engines. There are two alternative ways to achieve this, but they both have their own downsides:

    One way to control the speed of music and sound effects is to use the V-blank interrupt, however, this interrupt occurs less often on PAL consoles, in turn causing the sound engine to update less often. This causes music and sound effect to play slower on PAL consoles, which is perhaps most well known in Sonic the Hedgehog (1991). The game’s sequels avoid this by detecting PAL consoles and forcing every fifth V-blank interrupt to update the music twice, resulting in it having roughly the correct speed, albeit with some minor distortion. Using the YM2612’s timers instead would have avoided this issue entirely as they are almost exactly the same on PAL consoles as they are on NTSC consoles.

    Another method of controlling the speed of audio is to manually time code execution by writing the code so that it uses a certain number of CPU cycles. This approach is quite extreme, but it does see heavy use in Z80 code that is responsible for feeding PCM samples to the YM2612’s DAC channel (such code is called a ‘DAC driver’). Most DAC drivers use idle loops to waste CPU cycles until it is time to send the next sample. Were the YM2612’s timers capable of raising CPU interrupts, then this technique would be largely unnecessary except for in the most advanced DAC drivers.

    Implementation
    The YM2612 timers are quite rudimentary: every time the YM2612 outputs a full frame of audio, Timer A is decremented, and every 16 times a full frame is output, Timer B is decremented.

    Unfortunately, I ended up not noticing this, and redundantly refactoring my YM2612 emulator to operate in cycles instead of audio frames (there are 144 cycles to 1 audio frame). The reason that I made this mistake was that the (unofficially translated) official documentation for the YM2608 (the chip that the YM2612 is derived from) states that Timer A decrements every 72 cycles, not 144. And yet, my own testing and the documentation found here both suggest that the timers are twice as slow as the YM2608 manual claims that they are. This may have something to do with how the YM2612 differs from the YM2608 in how it “mixes” all six channels togethers (it doesn’t: it just cycles between outputting each one hundreds of thousands of times per second).

    Results
    Since these timers are typically used to time sound engines, the lack of emulating this feature causes many games to output no audio. Such games include Vectorman and Castlevania: Bloodlines. With support for the timers added, these games now produce audio.

    Another game affected by this is an old ROM-hack of mine – ‘Sonic 2 except the music goes as fast as you do’:



    Unlike the vanilla Sonic the Hedgehog 2, this ROM-hack uses Timer A to control the speed of the sound engine. By adjusting the timer, the speed of the music and sound effects is changed!

    About Menu
    clownmdemu’s frontend is a rather complex bit of software, so, to do the things that it does, it leverages a number of open-source libraries (and fonts). These libraries are made available under certain conditions: for example, a library may require that its authors are credited in the documentation of any software that uses it, while other libraries go a step further and require that an entire copy of the library’s licence is provided in said documentation. The libraries used by my frontend tend to require the latter.

    Being a minimalist, I don’t like the idea of every release of my frontend bundling the executable with a dozen text files, so instead I had the idea of embedding the licences into the frontend itself. To this end, there is now an ‘About’ menu which gives a brief overview of what the program is and provides a list of open-source licences.

    This makes it a lot easier for myself and anyone else who uses my emulator to abide by the various licences, as there no longer has to be any worry about forgetting to reproduce licences with every binary distribution.

    [​IMG]

    Personally, I think the requirement to reproduce a big blob of legalese with every binary distribution of non-copyleft software is stupid, which is why my libraries are usually zlib- or 0BSD-licensed instead, as they don’t have that requirement.

    Support for the Window Plane
    Another big feature for Mega Drive emulation!

    The window plane is an oddity to me: the first two Sonic games (whose codebases I am very familiar with due to spending over a decade reverse-engineering them) never use it, so it’s completely alien to me. That’s why it took me so long to add support for the Window Plane: I never had the need nor knowledge to.

    What the Window Plane is is a bizarre override for Plane A. Unlike Plane A, the Window Plane cannot be scrolled, but it is otherwise capable of everything that Plane A is. The Window Plane is not rendered on top of Plane A, like Plane A is to Plane B, but rather the Window Plane renders instead of Plane A. The VDP specifies two boundaries – one vertical and one horizontal – that determine where Plane A stops being drawn and the Window Plane starts being drawn instead (or vice versa).

    This feature tends to be used for drawing a HUD that does not scroll with the game’s foreground or background. An example of a game that does this is Castlevania: Bloodlines:

    [​IMG]

    This feature has a glaring bug on real Mega Drives: if the Window Plane is drawn to the left of Plane A, and Plane A is scrolled horizontally by a number of pixels that is not a multiple of 16, then the two columns of Plane A tiles that are next to the Window Plane will be “disfigured”. This bug is noted in Sega’s official Mega Drive developer documentation (the “Genesis Software Manual”, page 50). My emulator does not yet reproduce this bug, but I plan to add it in the future.

    One other bit of software that I have on-hand to test the Window Plane is a little bit of homebrew that I found here, which was made by someone called ‘Fonzie’. It’s useful for illustrating the aforementioned bug on real Mega Drives, but it’s also handy for testing that emulators support the Window Plane. Here’s a screenshot of it running in my emulator:

    [​IMG]

    Like the Sprite Plane, Plane A, and Plane B, the Window Plane can be disabled in my frontend’s debugging toggles menu. As a novelty, disabling the Window Plane will cause Plane A to be drawn in its place. This allows the user to see the part of Plane A that is “hidden” by the Window Plane.

    Support for the 68000’s BCD Instructions
    The 68000 has three instructions for performing binary-coded-decimal arithmetic. The long and short of it is that performing a BCD addition between 0x1 and 0x9 results in 0x10 instead 0f 0xA. This is useful in situations where you need to extract individual decimal digits from a number but don’t want to resort to repeatedly dividing by 10 to do so, as, if the number is in BCD format, you can instead just use bit-shifts and bit-masks, which are way faster than divisions. The two games that I know of which use BCD instructions use them for HUD elements, which makes sense since each digit needs to be extracted so that it can be used to determine which number graphic to display on the HUD.

    The reason for these instructions taking so long to be implemented, besides them being quite niche, is that they have an absurd number of undocumented behaviours and edge-cases. For instance, the BCD instructions are some of the only instructions to have their overflow condition code behaviour officially specified by Motorola themselves as ‘undefined’, meaning that the official documentation is of no help in understanding how it works. The documentation also fails to explain what is supposed to happen when a BCD operation is performed on non-BCD numbers. For instance, what’s supposed to happen when 2 is added to 0xC? Is the result 0xE? 0x12? 0x10?

    Luckily, other emulator developers encountered this problem too, and exhaustively documented how a real 68000 performs its BCD operations, figuring out every quirk and feature. This information has been collected in this SpritesMind thread. Most notably, Flamewing provided some homebrew that you can run on an emulator or a real Mega Drive to verify that its BCD instructions do as they should. Safe to say, my emulator now passes all of this homebrew’s tests!

    With this, three of the 68000’s quirkiest instructions are now fully emulated!

    The lack of BCD instruction emulation broke Castlevania: Bloodlines in a couple of humorous ways: it was impossible to use items because the ‘gem’ counter would never go above 0, and it was also not possible to get a Game Over because the lives counter would never go below 0, which effectively gave the player infinite lives.

    Vastly-Improved Z80 Emulation Accuracy
    Some people have told me that certain Sonic ROM-hacks were missing their DAC audio output when ran in my emulator. One such hack that I was able to verify this with was Sonic 1 Megahack: Ultra Edition, which lacked the music and sound effects during the “Sonic have pased” popup after completing a level.

    Unlike my 68000 emulator, my Z80 emulator had not been verified against any test suites, so it was likely that my Z80 emulator had a number of bugs and inaccuracies that these ROM-hacks were invoking, causing them to misbehave and not output audio through the DAC channel.

    Back when I initially developed my Z80 emulator, I had read a blog post that detailed something called ‘ZEXDOC’ which is a program that is written in Z80 assembly and can be ran on both real and emulated Z80 CPUs to verify that each instruction performs properly, which it does by analysing RAM, registers, and flags before and after each instruction. The catch was that this program is intended to be ran on CP/M, which is an old operating system for the Z80. With that said, the dependency on CP/M was quite minimal: ZEXDOC only used two of CP/M’s console-printing system calls, and expected itself to be placed at address 0x100.

    I was able to quickly rig-up a small program that implemented just enough of CP/M to get ZEXDOC running on my Z80 emulator, and it signalled that many instructions were not working as intend:

    [​IMG]

    Unfortunately, ZEXDOC gave little feedback as to exactly which instructions went wrong and how they went wrong. Not to mention, if my emulator is performing Z80 instructions incorrectly, and ZEXDOC uses Z80 instructions to perform the calculations that determine whether other Z80 instructions work correctly, then how can I be sure that its calculations are being performed correctly? What if some of these failed tests were false-positives?

    So I couldn’t trust ZEXDOC’s output, nor make attempts to narrow-down exactly what inaccuracies my Z80 emulator had.

    This problem didn’t exist with my 68000 emulator’s test suite, as the actual tests are not performed by the emulated 68000 itself but rather the host computer’s CPU. If I could find a test suite like that for the Z80, then that would be a great help. Not to mention that a test suite like the one used for my 68000 emulator would do a much better job of helping me verify exactly which instructions are incorrect and why, since it lists the exact contents of each register and memory address before and after the tested instruction’s executions.

    I was in luck, because I stumbled across a Z80 test suite that was almost identical to the test suite that I used for my 68000 validator!

    Due to its similarly to the 68000 test suite, I was able to make a modified version of my 68000 validator read the Z80 test suite’s data, and begin performing said tests on my Z80 emulator. It immediately uncovered numerous bugs, of which some were minor (such as undocumented flag behaviours) and others were severe (such as entire swaths of instructions using the wrong operands).

    Soon, I had my Z80 emulator passing every test, with the exception of instructions which I had yet to implement in the first place such as IN, OUT, CPI, CPIR, and DAA. Now confident that my emulator could run ZEXDOC at least somewhat properly, I tried it again and, this time, most of the tests passed:

    [​IMG]

    The only failed tests were for instructions which I had not yet implemented in my emulator. I have never seen these instructions used in any Z80 code for the Mega Drive, be it code from official games or homebrew. Because of this, implementing these instructions was never a priority for me, and I had no test-cases for them either. However, because both the Z80 test suite and ZEXDOC provide exhaustive tests for these instructions, I decided that I would finally add them to my emulator. Eventually, these new instructions passed both validators, meaning that every ZEXDOC test passed:

    [​IMG]

    There are still three instructions that are not yet implemented: HALT, IN, and OUT. The reason that the latter two are not implemented is because they are special instructions that write to “IO ports”, which do not exist on the Mega Drive.

    The instructions that I did implement are CPI, CPD, CPIR, CPDR, and DAA. The first four all are variants of each other, and are used for searching for a particular byte in a block of memory (much like C’s 'memchr' function). The DAA instruction is for performing “BCD correction” to the accumulator register. It essentially performs the same task as the second half of the 68000’s ABCD, NBCD, and SBCD instructions: it computes a ‘correction factor’ based on the output of the previous arithmetic instruction, and adds it to the output to make it into a valid BCD number.

    Now that my Z80 emulator is much closer to behaving like a real Z80, those Sonic ROM-hacks should all have working DAC audio output, right?

    Well, the joke’s on me, because it turns out that Sonic 1 Megahack: Ultra Edition still has missing DAC audio. But why? Why does the audio still not work despite the Z80 emulation being so much more accurate? Well… it’s because the audio doesn’t work on a real Mega Drive!

    Yep, you read that right: that ROM-hack only works properly on inaccurate emulators, because that’s all that the developer used to test the game during its development. This is actually a fairly common occurrence when it comes to ROM-hacks, since running a ROM-hack on a real Mega Drive requires an expensive flash-cartridge, so most developers just stick to testing exclusively with emulators.

    So, in the end, the audio in Sonic 1 Megahack: Ultra Edition was bugged, not because my emulator was inaccurate, but because it was too accurate. I suppose I should be proud of that.

    Improved FM Debug Menu
    [​IMG]

    While debugging Sonic 1 Megahack: Ultra Edition‘s missing audio, I realised that the FM debugger could be improved, so I’ve moved the per-FM-channel data out of the tabbed section and into a shared table, and exposed the timers, latched address and port, and channel panning.

    Fix 1-Cell Horizontal Scrolling Mode
    This is another bug that I didn’t notice because no Sonic game or ROM-hack that I know of uses it. The Mega Drive’s VDP has three ways of scrolling the screen horizontally:
    • Scrolling the entire screen.
    • Scrolling each row of pixels individually.
    • Scrolling each row of pixels in groups of 8 (the size of a tile, or, as Sega’s official documentation calls it, a “cell”).

    This bug involves that last one: it just didn’t work at all. This can be seen in Earthworm Jim‘s “What the heck?” level:

    [​IMG]

    So what’s going on? When I first wrote my VDP emulator, I assumed that 1-cell mode kept all of its scroll values next to each other in memory, just like in 1-line mode. However, that is not the case: strangely, each cell’s scroll value is actually spaced 8 values apart. Correcting this behaviour fixes Earthworm Jim, and presumably everything else that uses 1-cell mode.

    [​IMG]

    Add General VDP Debugger
    To diagnose the above bug, I needed information on what the VDP was doing, so I added a menu to show the VDP’s various settings. It’s not all that pretty, but it gets the job done for now:

    [​IMG]

    It was with this that I figured out that Earthworm Jim‘s scrolling was only broken when the VDP was in 1-cell horizontal scrolling mode, so this menu should prove useful in the future too.

    Abandon ‘tiny file dialogs’ Library
    Dear ImGui and SDL2 are great and all, but they don’t provide a cross-platform way to browse files. This is a problem since it’s important for the user to be able to select a ROM image or a save-state file to load or save. To this end, the frontend made use of the ‘tiny file dialogs’ library, which enables the use of the operating system’s standard file dialogs.

    Unfortunately, the library is as much a help as it is a burden: its code is of questionable quality, producing various compiler warnings for such novice mistakes as returning pointers to local arrays. In addition, despite emphasising portability and supporting POSIX, the library is incompatible with the BSDs. Finally, its support for symbolic links is allegedly completely broken.

    With the library being riddled with code hygiene issues and actively limiting the portability of my frontend, I’ve decided to ditch it. In its place, I’ve added a barebones file input prompt that leverages Dear ImGui.

    [​IMG]

    Neither C++11 nor SDL2 provide a way of querying directories for their contents, so this is the only universally-compatible solution. I could have used C++17’s file-system API, but I worry that such a new API is not very ubiquitous yet.

    Of course, using such a limited, clunky way of opening files would harm the frontend’s usability, so I intend to add platform-specific logic to use the native file dialogs whenever possible. Right now, this has been done for Windows, and I intend to do the same for Linux (GTK and Qt) soon. Users of other operating systems will have to get comfy with the barebones dialog, but it’s still an improvement over the frontend not compiling at all.

    The barebones file dialog supports drag-and-drop: just drag the desired file onto the window and its path will automatically be entered into the text box. The file dialog doesn’t even have to be open: ROMs and save states can be dragged onto the window at any time, and the frontend will apply them appropriately. Intuitiveness is great.

    Fix VRAM Fill
    The Mega Drive’s Video Display Processor has three Direct Memory Access modes: 68000 to VDP, VRAM Fill, and VRAM to VRAM. The first is used quite often and implemented in my emulator, the second is much less common and supported by my emulator but not well tested, and the third is rare and not yet in my emulator at all.

    Across a large number of games (including Sonic, Castlevania, Vectorman, and Earthworm Jim), VRAM Fill is seemingly only used for setting large swarths of VRAM to 0. This basic usage is easy to emulate, but does not serve well to ensure that said emulation is entirely accurate to the behaviour of a real Mega Drive.

    Cue Mega Man: The Wily Wars, which makes some interesting use of VRAM FIll: at the start of VRAM, it stores a handful of blank tiles, each of a different colour – these tiles are all generated with VRAM Fill.

    [​IMG]

    This behaviour was enough to expose issues with my emulator’s implementation of VRAM Fill:

    [​IMG]

    Looking at the VRAM debugger shows exactly what’s going wrong:

    [​IMG]

    Every other two pixels is not being filled with the correct colour. I immediately had my suspicions about the cause of this: back when I was first writing the VDP emulator in September 2021, I learnt that VRAM is apparently 8-bit, while CRAM and VSRAM are 16-bit. This caught me by surprise, as I’d always assumed that all three of them are 16-bit, so I was not sure what the proper way to implement this quirk was. Instead, I implemented VRAM as 16-bit, as I was originally going to. I figured that this might cause certain edge-cases – such as uploading data to an odd VRAM address – to behave incorrectly, but otherwise it would not be an issue. However, seeing this bug in Wily Wars convinced me that this workaround had to go.

    It turns out that many aspects of my emulator’s implementation of VRAM Fill were incorrect: for instance, it mistakenly assumed that the length was measured in 16-bit words, when in reality it was measured in 8-bit bytes minus one. Additionally, the 16-bit word that specifies the value to fill the VRAM with is not entirely used: only the upper 8 bits of it are actually written to VRAM.

    Addressing these issues fixed the bug, and Wily Wars now looks much better:

    [​IMG]

    Wily Wars still has one small bug: those four coloured dots near the start of VRAM. The game appears to accidentally write a VDP command ($8F02) to the VDP’s data port instead of its command port, causing it to be uploaded to VRAM as graphics. While the game certainly makes the same mistake on a real Mega Drive, it doesn’t seem to result in those pixels being written to that particular place in VRAM, as it causes artifacts to be visible in the level which aren’t present on a real Mega Drive:

    [​IMG]

    See the pixels in that pit? Those aren’t there on a real Mega Drive. Just another inaccuracy to fix in the future, I suppose.

    Implement YM2612’s ‘BUSY’ Flag
    This is a feature that wasn’t too important to emulate, but it’s so simple that I figured that I’d might as well.

    The YM2612 takes time to process the data that it is given. To signal to the CPU that it can’t accept more data yet, the YM2612 sets the high bit of its status byte. The CPU obtains this byte by reading from one of the YM2612’s address ports.

    On the YM2612, the busy flag is extremely basic: it is set whenever either data port is written to, and always lasts for 32 YM2612 cycles (192 68000 cycles), regardless of how long the submitted data actually takes to process. I’ve heard that the longest YM2612 operation is only 24 cycles, and that the YM3438 actually does set the busy flag for lengths that match each operation’s duration.

    With this feature implemented, any software that explicitly relies on the busy flag for timing should work correctly now. An Earthworm Jim game and a game called Hellfire apparently rely on this.

    Add ‘Other’ Debugging Menu
    To expose yet more of the Mega Drive’s internal state to the user, a menu has been added to show information about the general console, rather than specific components. Right now, this menu revolves around the bus arbiter, but it will be expanded with other information in the future, as the need arises.

    [​IMG]

    Closing
    This has been a pretty massive update (and a massive blog post), so I’m feeling a bit burnt-out on working on this emulator. Progress may be slow for a while after this. I’m pretty happy to see how far this emulator has come, though! It wasn’t long ago that this emulator couldn’t even boot anything, and now look at what it can do: play the classic Sonic trilogy, run Linux, and even run ROM-hacks and homebrew!

    Standalone frontend: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.4

    libretro core: https://github.com/Clownacy/clownmdemu-libretro/releases/tag/v0.2
     
    Last edited: Mar 1, 2023
  12. Chimes

    Chimes

    NO! NO! NO! NO! NO! NO! NO! NO! NO! NO! NO! NO! NO Member
    713
    512
    93
    Finally, I can listen to Hellfire's music b u t i n a d i f f e r e n t s p e e d !
     
  13. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    Last edited: Mar 2, 2023
  14. JGMR

    JGMR

    Member
    5
    0
    1
    Wowee! It’s been over a year since the PSG is slightly off-key as in Regen, and I’m happy to see that it was finally fixed! This emulator is awesome!

    + - And hey, this was my very first post that I made here on the forums. Hello to all!  
     
    Last edited: Mar 2, 2023
  15. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    Thanks! The reason that it took so long was that I wanted to verify that the PSG was incorrect before trying to fix it, but that would require digging one of my Mega Drives out of storage and listening to how it sounds, so I ended up constantly putting it off. Even when I finally did get around to it today, the difference was so subtle that I still had a hard time telling whether my emulator sounded closer to my Mega Drive before or after I attempted to fix its PSG. Maybe I'm just tone-deaf. Eventually I noticed that Sonic 1's Special Stage music makes a pretty good test, as it uses some very high-pitched notes. The bug affects high notes a lot more severely than it affects low notes, so that song sounded different enough for me to notice.
     
    Last edited: Mar 10, 2023
  16. Devon

    Devon

    A̸ ̴S̴ ̵C̵ ̷E̶ ̸N̸ ̴D̶ ̵E̶ ̸D̶ Tech Member
    1,289
    1,492
    93
    your mom
    A while ago I did report the PSG was slightly off key, but I can report that it now sounds correct to me. It makes sense that the higher frequencies were affected more, since the distances between each note gets shorter and shorter as you raise the pitch (I believe it's logarithmic), hence decreasing the overall wiggle room for error for higher frequencies.
     
    Last edited: Mar 2, 2023
  17. Brainulator

    Brainulator

    Regular garden-variety member Member
    For some reason, on my system, FM sounds come out crackly; as far as Sonic 1 is concerned, this at least affects the ring, rolling, and "score tally complete" sounds. This didn't happen prior to version 0.4, it seems...

    EDIT: Apparently, yes, it did, at least for the ring sound. It's just that the latest releases made it more apparent somehow.
     
    Last edited: Mar 3, 2023
  18. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    The frontend's audio code is garbage. It's held together with duct tape and wishful thinking; it needs completely redoing. I'm still trying to think of a good way to do it, but my current best guess is leveraging a 'pulling' audio API and using audio-stretching to always make the available audio data fit the callback's buffer. That way there are no drops or skips in the audio, at the cost of some hopefully-microscopic changes in pitch and tempo as the audio is slightly stretched and squashed.
     
  19. nineko

    nineko

    I am the Holy Cat Tech Member
    6,321
    490
    63
    italy
    What you said isn't true per se, it's actually the other way around entirely, as a musician let me clarify.

    Yes, note frequencies are logarithmic, so the distances between each note become larger and larger if the pitch increases.

    Here is an example: you know the middle note "A" is usually tuned to 440Hz nowadays, right? Go one octave lower, and that frequency becomes 220Hz; go one octave higher, and that frequency becomes 880Hz. If you are out of tune by 1 hertz, 221 against 220 is relatively worse than 881 against 880.

    The lack of precision of the PSG with the higher notes comes from a formula which puts the register value at the denominator: https://www.smspower.org/Development/SN76489?from=Development.PSG#ToneChannels.

    Also relevant: https://www.smspower.org/forums/11185-TuningThePSGToItself.

    With all that said, I have no idea if the inaccuracy you were talking about is related to this at all, but I thought to provide some musical insight.
     
    • Agree Agree x 1
    • Informative Informative x 1
    • List
  20. Clownacy

    Clownacy

    Tech Member
    1,066
    617
    93
    A PSG tone channel is merely a counter that flips the speaker back and forth from positive to negative when it expires, creating a square wave. The lower the counter, the more frequently the square wave alternates, the higher the pitch. Since the bug was an off-by-one error in this counter, it would affect higher notes more because the error becomes proportionally larger compared to the value of the counter. For instance, the counter being off-by-one is a pretty big problem when the counter is set to 3 because it makes the counter 33% slower, but it's not so much a problem when the counter is set to 100 as the error only makes for a difference of 1%.
     
    Last edited: Mar 3, 2023
    • Like Like x 1
    • Informative Informative x 1
    • List