don't click here

Sonic 1 J2ME General Hacking Thread

Discussion in 'Engineering & Reverse Engineering' started by Kilo, Jun 13, 2020.

  1. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    In 2005 and 2006, a mobile port of Sonic the Hedgehog was made by iFone and Glu. It came in 2 parts containing the first and second half of the zones. Up until recently, hacks and mods for it have been pretty underground. However, as of April, I and a few others have started a small community dedicated to researching Sonic 1 J2ME and other flip-phone classic mobile games.
    Sonic 1 J2ME is notable for 2 reasons, mostly relating back to the Mega Drive version: It uses prototype assets, and names for files in the original source code. As well, the game is super easy to mod in both art and music, because they literally just use PNGs and MIDIs.
    Levels have been easy to edit too as of recent, because of one of our leading researchers; Leia Ivon Flame, found that the level files are practically just Sonic 1 MD's (Albeit tiles are 16x16 PNG sheets) Leia's also been working on a tool called SON_ME to make level editing user friendly. Here's a look;
    [​IMG]
    Please note graphics are corrupted for an unknown reason, it does not appear this way ingame.
    If you'd like to get into Sonic 1 J2ME hacking (Or the various other games, there's also games like Sonic Golf and Sonic Runners Adventure if that's your thing) here are a few resources!
    Sonic 1 J2ME's Decompilation Github Page
    Sonic Java ME hacking Discord Server
    KEmulator 1.0.3 (J2ME Emulator)
    FernFlower Java Decompiler
    We love new faces that can help us with research, so don't be shy to join the community! We still have a lot to uncover, so you're not missing out. Feel free to use the thread for questions, or even showcasing what you've done.
     
  2. Esrael

    Esrael

    Neto Tech Member
    304
    257
    63
    Brazil, São Paulo, Guarulhos
    Neto Assembler Editor / Sonic 2 Delta / Neto MD-DOS
    Look likes the tool is misinterpreting the bit flags as VRAM Art data pointers.
     
  3. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Thanks to @MarkeyJester providing the original assets from Pana Der Hejhog, Sonic 1 J2ME has it's first custom level!~
    Background is still using GHZ's layout though because we don't know where exactly background data is.
    [​IMG]
     
  4. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Alright time to revive this thread since I've gotten back into the grove of researching J2ME.
    First order of business here, is a new decompilation. This version has a depadding tool for level files, and I've extracted the Japanese exclusive P902i version's files (Unfortunately I cannot find the graphics or MIDIs in the scratchpad... Yet. And the latter half of the game had to be downloaded from the Sonic Cafe's servers... Which are 17 years past expiration.)

    Because I had mostly made postings about my research on SSRG back in the day, I never posted tech info here. So here's a recap on what is known so far.

    Graphics and music are the easiest thing to work with. Graphics are just PNGs, in order to have transparency they need to have color indexing... Or so I thought, some emulators, and even some versions of KEmulator aren't consistent with this, but I'd recommend just having a 256 color index just to be safe. Unfortunately, the sprite frame width and heights are defined in code, seemingly in Actor.java. And until I can get the rat's nest that is the code fully worked out and even compilable, you'll just have to make sprites fit within their prexisting definitions. Same with animations, done in code so no modifying those yet.

    Music are just MIDIs. There's seemingly no limitation on them at all, I've thrown black MIDIs at this thing and they play fine. Though that's just on emulator, no guarantee on how hardware will take it.

    Things start getting interesting in the level formats. They're taken straight from the Mega Drive version and use their names from the actual source code of Sonic 1.

    zonex.bmd are chunk files, the format is the exact same to the Mega Drive version, although uncompressed from Kosinski.

    zonex.blt are the zone collision files. With scdtblwk.scd being the GitHub disassembly equivalent of collide/Collision Array (Normal).bin and Collision Array (Rotated).bin both combined into 1 file. scddirtbl.blt is Angle Map.bin, and has a mysterious extra $100 bytes after where the Mega Drive version ends. blkcol.bct Seems to be in the infamous bitmap collision format, although with all the collision files accounted for with the final Mega Drive version here, I'm not sure what this actually is.

    MapLzoneX.blt are tile priority files, with each nybble representing a tile, 0 being low and 1 being high. There's been a misconception that this may have been how Sonic 1 stored priority in an earlier version of the level engine, but it's just a matter that the Mega Drive version stored priority in the block mappings... But those are PNG now so they can't hold priority data.

    mc_xx_map_data.bin are the level layouts. They have 3 bytes of padding between each byte of data, which is why I wrote the depadder tool. It merges all 3 acts into 1 file, and has a strange header I still haven't figured out, as it doesn't match the width or height properly, and it's not an offset to point to the next level either. But other than that, it seems to be the same format as the Mega Drive. Any help on researching level headers would be awesome.

    And as a side note, backgrounds aren't stored in the level layout as one might expect. But rather, they're in a massive nested array in Background_Renderer.java. Think of it as a kind of tree going from zone > parallax strip > rows. With the rows containing the tile IDs it should display. It's kind of a pain in the ass to keep track of the arrays, but in theory this should make parallax editing easier than the Mega Drive version.
    upload_2024-2-24_9-47-50.png

    ZONEXACT.act are the object layouts. They have an 8 byte header, 2 bytes per act (with act 4 support) denoting the act's data length. And the rest I believe is the same as the Mega Drive version? Might be wrong on that. But as all the object IDs in J2ME are swapped around, and positional data is another weird mess, the data doesn't really look the same.

    Now these 2 files have been bugging me since I started researching J2ME 4 years ago.
    framedata.bin, and mc_obj_size_table.bin. They're padded like the main level files. But their purpose is unclear. According to the code they're only loaded for Green Hill, but I have a feeling that that is a decompiler inaccuracy (This obfuscated mess really makes any decompiler crap itself, JADX just provides the best and most usable version of it), as their if cases that load them are also present for Marble and Spring Yard but empty. And I think my first goal is going to be figuring out what these are.
     
    Last edited: Feb 24, 2024
    • Like Like x 1
    • Informative Informative x 1
    • List
  5. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    So nevermind, levels weren't as complicated as I had thought. The header is just $04, height, and width. Instead of the Mega Drive just being width and height. Simple enough.
    Here's J2ME GHZ1 loaded into SonLVL.
    upload_2024-2-24_14-3-3.png
    I wonder if the $4 in the header, has anything to do with the also unused $4 in the start of entries in LevelSizeArray in the Mega Drive version.
    Edit- the width and the height don't add +1. I had thought this because there are some garbage chunks at the end of the level, but I guess that's actually part of the data, for whatever reason.
    upload_2024-2-24_14-15-39.png
    2nd edit
    I'm stupid and copy pasted the act 2 data into act 1. This is not garbage data.

    3rd edit, well this is interesting, while browsing around in Green Hill 3, I actually found the ending layout below the stage. How odd!
    upload_2024-2-24_14-45-14.png
     
    Last edited: Feb 24, 2024
  6. Code (Text):
    1.     private void am() {
    2.         try {
    3.             int[][][] var3;
    4.             label42: {
    5.                 this.b = null;
    6.                 this.i = null;
    7.                 this.h = null;
    8.                 this.c = (byte[][])null;
    9.                 this.k = null;
    10.                 this.c = null;
    11.                 this.j = null;
    12.                 System.gc();
    13.                 var3 = (int[][][])null;
    14.                 String var10000;
    15.                 switch(this.zoneID) {   // Get zone ID
    16.                 case 0:
    17.                     var10000 = "/mc_gh_map_data.bin";   // Load GHZ level maps
    18.                     break;
    19.                 case 1: // LZ
    20.                 case 3: // SLZ
    21.                 default:
    22.                     break label42;
    23.                 case 2:
    24.                     var10000 = "/mc_ma_map_data.bin";   // Load MZ level maps
    25.                     break;
    26.                 case 4:
    27.                     var10000 = "/mc_sy_map_data.bin";   // Load SYZ level maps
    28.                 }
    29.  
    30.                 var3 = a(a(var10000));
    31.             }
    32.             q();
    33.             this.bu = var3[this.actID][0].length;
    34.             this.bv = var3[this.actID].length;
    35.             this.d = new byte[this.bv][this.bu];
    36.  
    37.             for(int var4 = 0; var4 < this.bv; ++var4) {
    38.                 for(int var5 = 0; var5 < this.bu; ++var5) {
    39.                     this.d[var4][var5] = (byte)var3[this.actID][var4][var5];
    40.                 }
    41.             }
    Tracked down the code that handles layout loading; while I don't know Java at all, the actual layout code appears to be inline with the Genesis game, so I'm not sure why they'd add an extra byte at the start.
     
  7. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    I'm now starting to believe that it's actually an act count. It's actually only 4 in GHZ, but 3 in Marble in Spring Yard. And as my edit notes, the ending sequence is actually in the data here, making the Green Hill map in effect, have 4 acts.
    So the level file formats are as follows
    #acts, act 1 height, act 1 width, act 1 data, act 2 height, width, data, etc.
    Now that that's been cracked I'll work on a Python script to split the act data and flip the height and width bytes.

    And as I do that, here's something neat. J2ME actually has custom chunks for the spike pit section in Green Hill act 2, to make it appear as though the spikes are sitting on something. This was done due to the game being ported to all sorts of different phones with resolutions so it'd be uncertain if the boundary code would 100% make sure you couldn't see below the spikes. 2013 and Mania also have this change. In fact, on a technical scale, J2ME shares a surprising amount of similarities to 2013.
    upload_2024-2-24_15-18-38.png
    There's also this unused chunk which accidentally has a sky and grass tile.
    upload_2024-2-24_16-18-58.png
     
    Last edited: Feb 24, 2024
  8. Merging all layout data into one is pretty unusual, but I'm guessing it was easier to store that way.
    Ironically, if it were Kosinski compressed, it would be very small given the repeated backgrounds could be stored once instead of three-times.
     
  9. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Huh? Backgrounds in the Mega Drive version only use 1 file anyways and not 3. The only exception is Marble Zone due to how the interiors are set up. And J2ME doesn't even store backgrounds as part of the layout they just use 1 repeating chunk that's defined in the nested array in my previous post.
     
  10. Oh, sorry.
     
  11. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Note on chunks. The Mega Drive version by default knows chunk 0 is a blank chunk and so it is not defined in the chunk mappings to save on ROM space. This is not the case with J2ME, and there is a chunk mapping for chunk 0. So when importing J2ME assets into SonLVL you need to delete the mappings for that extra blank chunk or the level data gets offset by 1 chunk.
    And I have now said chunk so much that it looks weird to me. :V
     
  12. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,742
    338
    63
    SonLVL
    So are you gonna make a fork of SonLVL for the J2ME games?
     
  13. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Perhaps? To me the easiest route is just writing Python tools to convert between J2ME and MD since the formats are similar enough. And since I'm really the only person interested in hacking this port that seems like a sufficient method of doing things.
     
  14. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    So I've been trying to import a custom level for the past couple days, with no luck. I think there may be some sort of checksum or an upper limit on the size levels can be? Maybe the clue lies in framedata.bin or mc_obj_size_table.bin (Those files really do drive me crazy)
    On another note though, I've determined int[][] f186b to be the player's start positions, it's defined just below Player_Info. And while the decomp doesn't build anything usable yet you can hex edit it at offset 0x1C5A0 in c.class from a clean ROM.
    upload_2024-2-28_17-59-16.png
     
  15. TwoSpaces

    TwoSpaces

    Member
    Hi all, this is gonna be my first post here I guess. I've been doing surface-level research on Sonic J2ME games because I sadly can't do any better as of yet, I have had some nice finds though.

    In regards to Sonic 1, first I should mention that there are two versions for Motorola 176x220 phones that reveal the official names of the classes: MainCanvas, MainCanvasDraw, Sonic, SoundManager. I believe the more powerful versions had one additional class? In either case, this might also mean that these versions (as well as possibly some Samsung versions with a similar trait) are not obfuscated, but I don't know all that much about Java so this might not be the case after all. I do know that three of the Sonic 2 Dash versions which reveal the original class names are not obfuscated.

    The DoJa version of the game (as opposed to the MIDP versions of the Part 1 and 2 game for the Western release) does not use MIDI music, it uses the format .MLD (i-Melody, not to be confused with the other completely unrelated phone ringtone format of the same name, go figure). The MIDP versions use .MID music (standard MIDI file) probably played through ordinary j2me MMAPI on the phones, which I think probably had like 128 simultaneous voices at most. Some device versions were different: some Samsung versions use .MMF music (Yamaha SMAF) and the Nokia low-end versions on the different "8-bit-esque" engine use .OTT (Nokia Tone).

    By the way, yes, some very low-end versions do use a different engine, with the midlet startup class being called "b" or "Main". This is mainly about versions for very low-end Nokia phones, but amusingly there's a 240x320 version for Samsung with original-sized graphics, because some Samsung phones really couldn't get the proper thing going I guess.

    Also, it's really weird that there are chunks for the ground under the spikes (since I very distinctly remember the spikes being suspended in the air in this version) and the entire ending scroll being there, even though only about one screen worth of the ending is actually shown in the game.
     
    • Informative Informative x 1
    • List
  16. Kilo

    Kilo

    That inbetween sprite from S&K's title screen Member
    309
    309
    63
    Canada
    Upon decompilation, unfortunately these alternate versions are still obfuscated despite their file names. Some of these do have unique files though like sindata.dat, and sonic.dat. Which is useful in terms of research on Sonic 1 Mega Drive's source code.
     
  17. TwoSpaces

    TwoSpaces

    Member
    I know this is more on the topic of Sonic 2 J2ME, but, one of the reasons it's not as well-liked sometimes as the Sonic 1 port is the 9 fps framerate cap (in some versions it's very slightly higher but still low). This is something you can easily observe on emulators, and here I have to stress that none of the J2ME implementations (emulators) are actual device emulators, so they run the game on your CPU although they have limiting options for games that need it (such as, incidentally, various Sonic 1 versions). Therefore this is something that is entirely on the game developers who built this FPS cap in.

    One evening I sat down in voice chat with RibShark and showed him these J2ME ports of classic Sonic games. He must've been a bit perplexed regarding the 9 fps cap, so he found a way to increment the framerate by modifying one of the classes. While it might look somewhat fine on emulators, you can start seeing the issue once you sideload this modified version on an actual phone: on a Nokia N86 8MP (released in 2009, a year after this game), going past 18 fps results in the game going in slow motion as opposed to the intended speed. 2008 did see the release of two slightly slower but still good Nokia flagship series phones, which were the N79 and N85, but out of the dumped English versions, the most complete Nokia 240x320 version (the only one to have the CPU-controlled Tails) is the Nokia N95 version, for an older (but evidently more popular at the time) flagship model with noticeably slower Java performance. Now you can probably see why the devs picked an FPS cap like that.

    But that's not the end of it! Running any Nokia version on these Symbian 9.3 phones (such as N79, N85, N86) will expose one of the many compatibility issues of this operating system's JVM with various existing dumped games. As can be seen in the video, Sonic's sprite likes to disappear and reappear in unknown circumstances. I also noticed that jumping on "foot pedal" springs in Chemical Plant and Aquatic Ruin will display only the first animation frame, which is possibly also related to this glitch. When Sonic J2ME hacking ever takes off, I would be interested to see a fix to this glitch in hacks based on Sonic 2 J2ME.

    Hopefully I won't forget to attach the modified class (for the Nokia N95 version, goes into the STH2P1 directory), and here's a video:
     

    Attached Files:

    • Informative Informative x 1
    • List