Sonic CD decompilation

Sorry, I was under the impression that the evidence was inconclusive, given that we don't even have an object ID.

 

All the source code related to R4, Tidal Tempest, has been committed!

Each zone has more files to decompile. R1 has 66 files. R3 has 76 files. R4 has a whopping 101 files! In the case of Tidal Tempest, it's not only because of more objects, but also because it has water.

The zones usually share a GAME.C file which contains all of the initialisation code, and a PLAYSUB.C file that contains code for common zone-related objects like sign posts and flowers. R4, however, has its own copy of these to have custom code for water.

R4 also has its own version of SCRCHK.C, but only for the first act's present.

Now, as for my nemesis, scrolling code...

R3's files for scrolling management were largely identical. It's mostly the scroll() function that differed each time. Thankfully, whoever did R4 did some optimisation.

The scroll() function used to be a 200-line monster. Now, it's 70 lines for the first act's present, 50 lines for the rest, and most of the time it's identical. Some files were even merged; R42A and R42B, and R42C and R42D, share the same scrolling file. They could have gone even further, as the files are still largely identical, but for some reason they didn't.

Wait, you mean the position in the index, not its "actno"? I took a look, and it's indeed in the index at position 43 (0x2B)!

 

I've whined mentioned before that the ASM produced by the PS2 compiler has a lot of small differences with the original ASM. Instead of being a configuration issue, it is possible that I have the wrong version of the compiler.

The disassembled ELF files's debug info mentions that it was compiled with Metrowerks CodeWarrior version 2.4.1.01. If I use the version found on archive.org to compile the decompiled code, that same version string is inserted into the result. However, if you launch it on the command line without any arguments and actually look at the compiler info, it says... 3.0.1:

This suggests there's a discrepancy between the version inserted into the debug info and the actual version of the compiler used.

I hope to find the actual compiler used in the near future.

 

Today's the project's first anniversary! The first commit to my local SVN repository was exactly one year ago.

The following has been decompiled the past year:

• core/shared code
• Palmtree Panic
• Collision Chaos
• Tidal Tempest
• Special Stage
• AVIGOOD (ending movie code)
• AVIOPEN (opening movie code)
• Best Time
• Title screen
• D.A. Garden
• Save Data
• Sound Test
• Stage Select
• Time Attack
• THANKS (post-credits scene)
• Visual Mode

The following is what remains to be decompiled:

• Quartz Quadrant
• Wacky Workbench
• Stardust Speedway
• Metallic Madness

There's something missing, though: the control program that ties all of this together. That doesn't have debug info, as far as I know, so it can't easily be decompiled. I'll have to look deeper into this so I can have something playable to show.

As for the compiler issue I mentioned last month: it turns out that several versions of CodeWarrior output 2.4.1.01 as the version string in the ELF file. The most recent one to do so is CodeWarrior 3.0.4. A preview version of this one is available, and it seems to fix most ASM discrepancies. The main discrepancy still present is the ASM generated for the initialisation of local arrays, which I don't know yet how to solve. A bug-fixed version of the compiler shipped with 3.0.4 is part of the CodeWarrior 3.5 package, but that also has this discrepancy, so it's most likely due to the decompiled code.

 

I focused on my C port of Sonic & Knuckles Collection in March and April, hoping in the meantime that I'd find the version of the compiler that was used. I did try to find the control program used for the game, but haven't found it.

When I got fired near the end of April I figured I'd use the extra time to decompile another zone. It's almost done, but, as always, the copy of the scrolling code for each act version takes time.

Now, let's move on to the reason I'm writing this update.

I mentioned that there seems to be a discrepancy between the version inserted into the debug info and the actual version of the compiler used.

The version number given on the command line by the compiler seems to vary on the environment, and it's not clear where it gets it from. The only thing you can rely on is the build date.

A fellow reverse engineer, who's also decompiling a game compiled with Metrowerks CodeWarriors, created a list of CodeWarrior releases to aid in finding the right version. As the debug info contained version 2.4.1.01 in the comment section, I narrowed my search to compilers inserting that version string. According to the list, this meant a compiler as old as the one shipped with CodeWarrior R2.5, and not newer than the one shipped with CodeWarrior R3.04.

Like I said before, the output from the version shipped with CodeWarrior R3.04 is closer to the original, but it's not there yet.

Then, yesterday evening, it was revealed to me that, while the compilers that ship with CodeWarrior R3.5 and newer do emit 3.0.0 to the debug info's comment section, the linker overrides this with 2.4.1.01! In other words, one of the newer compilers could also have been the one used to compile the original ASM!

I tested the newer compilers. Their output is almost identical to the original. The only remaining difference is that in 1% of the output they use a different register. Most of the time this means it uses $v0 or $v1 instead of $at.

I went back to WARP.ELF, and managed to make my version's main (code) section 99% identical (if you disregard the memory addresses). Next, I got the data section identical save for a couple numeric identifiers. Memory addresses are still off because in memory the global data seems to start at another location. I suspect it requires tweaking the linker command file, which I know little about, so I'm putting that off.

I'm already reviewing the core files used by R5, and will resume my R5 work after that. You can expect a release of R5's code soon!

 

All the source code related to R5, Quartz Quadrant, has been committed!

I've found the cause of the remaining 1% of differences, which means that this is the first zone to compile to identical ASM! Except for data offsets, of course.

This zone had less code than previous ones, which is why I was able to decompile it in a month despite also working on other things for a part of that. The size of the boss code, however, is second only to the first zone's, as that easy running challenge consists of more than a couple parts.

 

I've gone back through the code of the previous zones I've decompiled, and corrected those until they also matched the original ASM. This was mostly successful.

I say "mostly", because there's one file of R4 where the decompiled C code generates one less instruction than the original. It's not an important instruction (it's actually superfluous), but ideally I'd also like to match it.

Then there's some good news and bad news.

The good news: I've found the control program that ties all of the ELF files together. It's S1.DAT in the root of the disc.

The bad news: it has no symbols whatsoever, and I know nothing about PS2 programming that would enable me to reverse engineer it. It does have some debug messages, which enabled me to identify some functions, but that's pretty much it.

 

Everything under TITLE (title screen, Sound Test, D.A. Garden, etc.) has been checked again and corrected until it matched 99% of the original ASM. Not 100%, because the main source file of Time Attack doesn't match one superfluous instruction. Yes, it's the same case as for R4...

Also checked WARP again, corrected it, and it matches 100%.

Now that that's out of the way, I can get back to decompiling zones. Or rounds, as the game calls them.

 

All the source code related to R6, Wacky Workbench, has been committed!

This only took two weeks due to two factors:

• I could copy/paste most of the scrolling code from R5, and then between R6 versions
• I made maximum use of my jobless free time to decompile like a madman. On my most productive day I processed 18 files.

This zone uses its own version of the PLAYER object that seems to remove some sound-related code, and adds some functions. One of those functions I can't match 100% because one variable is put into a register instead of on the stack, and I can't figure out why.

Two more zones to go.

 

All the source code related to R7, Stardust Speedway, has been committed! Took me less than two weeks!

In other news, I recently revamped my ASM differences wiki page, detailing the few differences that are left, along with links to decomp.me scratches. If you're feeling brave, feel free to play around with those to try to match the original assembly.

 

All the source code related to R8, Metallic Madness, has been committed! This means that everything we have debug symbols for has been decompiled.

Now we need to somehow recreate the game's engine that is responsible for loading files, drawing to the screen, processing input, etc. We could decompile the Gems version, but without debug symbols, it won't be easy. I wonder if the PC version's executable would be more manageable to dissect.

I'll do a technical explanation on the authenticity of the decompiled code later.

 

I'm still working on the technical explanation referred to earlier, but I need more time as it's going to be pretty in-depth and I want to explain things clearly.

In the meantime, I've been looking into the engine.

The PS2 version of the engine is pretty impenetrable without debug symbols for someone not familiar with PS2 programming, so I gave up on that and turned my attention to the PC version's executable.

Being familiar with Windows programming, and Ghidra being well-suited to dissecting Windows executables, I'm pretty far into decompiling it. While I'm still far from done, I got most of the initialisation code covered, and would like to share my findings.

The Dino survives

It is said that the 1995 OEM release of the game used Intel's Dino libraries to abstract multimedia features, and that the 1996 retail release replaced Dino in favor of DirectX. However, this doesn't seem to be entirely correct.

Even without diving into the executable, this should be apparent. The game disc indeed has a folder named DIRECTX, but it also has a folder named RDX. This is short for "Realistic Display miXer", the official name of the graphics library that's part of Dino. In this folder, you can find the files DINO2D.DLL and DMIX.DLL.

Decompiling the executable, I noticed pretty quickly that all function calls related for sound were done to MCI (Multimedia Control Interface), a high-level multimedia API created by Microsoft and IBM and implemented into Windows 95. The original release used Dino's RSX (Realistic Sound Experience).

However, graphics operations are still handled by RDX. For example, the function to create a bitmap surface is srfCreate, and this function lives in... DMIX.DLL.

At this point it's not clear what part of the program uses DirectX.

A Windows 3.x program?

While decompiling the PS2 version, I noticed that the functions LibMain and WEP were stubbed, so I had to go looking for their signature. They weren't easy to find, because it turns out that these are 16-bit DLL functions. Both of these were replaced by DllMain in 32-bit Windows.

But that's not all.

Decompiling the PC executable reveals that it uses file handling API functions that were deprecated with the introduction of Windows 95. They continued to exist in 32-bit Windows for compatibility with 16-bit programs. They are: _lopen (instead of CreateFile), _hread (instead of ReadFile), and _lclose (instead of CloseHandle).

Initialisation check oddities

Those of us who used to play the PC release are no doubt familiar with the program checking that our display setting is 256 colours. But there's more:

• It checks the computer's Windows version. If it's too low, a message pops up that says: "You must have a Windows® 95 inside your system to run Sonic the Hedgehog CD.".
• At one point the plan was to check that your computer had a Pentium processor. However, this function is stubbed and always returns TRUE. If the check failed, it'd have said: "You must have a Pentium® Processer (or better) inside your system to run Sonic the Hedgehog CD.".
• There's a function that checks if your A drive is not a floppy drive, and if your system's system.drv file contains the value "NEC" for key "system.drv" in section "boot.description". If either of these is true, the function returns TRUE. From what I've seen so far this only affects whether one submenu is enabled or not.

 

Going to put all this info on the wiki soon, but I'll be posting about it here first.

I was talking about this with a friend yesterday, and he could actually explain this.

Back when we were using IBM PCs with DOS and Windows 3.x, people in Japan were using NEC's PC88 and PC98 computers, which mapped drive letters differently. We're used to always having floppy drives mapped to drive letters A and B, with the HDD being mapped to C. But NEC's computers would (when the OS is booted from the HDD) map drive letter A to the HDD, and the floppy drive to B.

In other words, as suspected, those two checks were checking if Sonic CD was running on a NEC PC98.

I've since reverse-engineered the menu resources, and can now say that what was being disabled is the "Full Screen" menu item.

Good and Better sound quality

The retail release contains evidence of a removed option for better sound quality.

During initialisation, the program queries the configuration file SONIC.INI's "SOUND" section for the value of a key named "QUALITY". The default return value is "GOOD ". If the first character of this return value is 'B' (presumably the first letter of "BETTER"), a global boolean variable (from this point on called gbBetterSoundQuality) is set to 1. The program uses gbBetterSoundQuality to choose from two strings, which it passes to the function that loads the game's PCM (sound) file. However, both strings have the same text: "pcm8.cmp".

That's not all that gbBetterSoundQuality is used for, however. When setting up the sound output, it uses the value to select from two values for the number of samples per second: 11025 and 22050.

The most important piece of evidence, however, is a function I've found that bears similarities to the above initialisation logic. It toggles gbBetterSoundQuality between 0 and 1, then changes the text of a menu item based on the new value. The texts? "Good Sound Quality" and "Better Sound Quality". Next, just like during initialisation, it chooses from two strings based on the value of gbBetterSoundQuality. But this time, they are different: "pcm.cmp" and "pcm8.cmp".

But here comes the kicker. This menu item does not exist in the program's menu resource! Furthermore, this function that toggles between both sound qualities is only called when a certain global variable has a value different from zero. And the initial value of this global variable is zero, and never changes outside of this function.

At this point you could argue that it was planned during development, but they ended up not including the higher quality sound. Then I wondered: could the higher quality PCM file exist in the OEM release? I downloaded it, mounted the disc, and...



Holy shit.

I loaded this version's main executable into Ghidra to verify if it actually uses this file. Unfortunately, this executable seems to be very different from the retail one, and as a result Ghidra's analysis is lacking here. I wasn't able to find WinMain, the starting point for all Windows executables. However, I found that it contained two sets of the strings "pcm.cmp" and "pcm8.cmp", hinting that both the initialisation and the sound quality toggle logic choose between the two files. More importantly, the menu resource contains the menu item "Better Sound Quality" that's missing from the retail version!

So there you have it. The OEM Dino version actually has better PCM sound quality than the retail version that released a year later.