don't click here

Issues with Nemesis compression in Fatal Labyrinth

Discussion in 'Engineering & Reverse Engineering' started by virgobit, Feb 15, 2020.

  1. virgobit

    virgobit

    Member
    6
    0
    1
    Hi everyone :)

    Recently I've been trying to hack Fatal Layrinth, namely sprites and palletes. I understand it uses Nemesis compression, so after extraction I proceeded to test it out. I redrew a standard Jelly, edited it in the decompressed file, recompressed it and it worked.

    But things never go smooth all the way with romhacking, of course :) I then tried editing the Ice Bar. After several repeated attempts, the recompressed file is always bigger than the original compressed code. Which doesn't make much sense to me at this point, because the redrawn Jelly was also bigger (in terms of sprite size) but was recompressed without a hitch.

    Even more troubling is that if I decompress the Ice Bar and recompress it as, without any edit, the recompressed code is also bigger than the original. To explain how I'm going about this, I ran Nemesis Searcher, which extracted all the compressed artwork and named each file according to the respective offset. I then used the Sega Data Compressor and since it doesn't seem these programs can compress directly to ROM, I opened the game with a hex editor, and finally copy/pasted the code (it worked for the Jelly, in this case).

    When it misfired with the Ice Bar I tried used the KENS libraries but the result is the same. Additionally, KENS and SDC will output different code and different code length. To wit, the Ice Bar starts at offset 13AC4 while the Magician starts at offset 13B42 ($7E bytes worth of compression). Recompressed, they exceed this length up to $8F.

    It makes sense that I would have to cut down in detail on my edits, but it doesn't add up that an unedited file would have the same issue.

    Can anyone provide any insight into this? Is it a problem on my end? Thansk in advance :)
     
  2. Rika Chou

    Rika Chou

    Tech Member
    5,276
    169
    43
    It's been a while but this was a common issue when editing ROMs was still popular. The most common solution was to relocate the compressed art to the end of the ROM when it was too big to fit in the original location, then change the pointers accordingly. It's kind of a sloppy solution but it worked.

    I think this happens because the nemesis compression tools we have are not the same as the originals, and can be less efficient at times.
     
    • Like Like x 1
    • Agree Agree x 1
    • List
  3. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,742
    338
    63
    SonLVL
    Have you tried the latest FW-KENSC tools?
     
    • Like Like x 1
    • Agree Agree x 1
    • List
  4. AURORA☆FIELDS

    AURORA☆FIELDS

    The cute one here Tech Member
    216
    24
    18
    Finland
    AMPS
    FW-KENSC should, with any sense, always produce equivalent or smaller files because it is far better at its job than the original compressors. The Sega Data Compressor is horribly outdated and slightly buggy tool that not many people use nowadays, indeed because of FW-KENSC. Additionally, there is a really handy shell extension version of it that makes recompressing files much quicker.
     
    • Agree Agree x 2
    • Like Like x 1
    • List
  5. virgobit

    virgobit

    Member
    6
    0
    1
    Thank you for the replies, everyone :)

    @Rika Chou: I did thought about relocating, but this would require expanding the ROM: there's almost no space left in it.

    @MainMemory & Natsumi: I wasn't aware the SDC was that outdated... I'll try FW-KENSC then, and report back as soon as I can :)
     
  6. virgobit

    virgobit

    Member
    6
    0
    1
    Report: working just fine now :) I might show it off once I have made substantial edits (for now still going about the general feel of the enemies). Edit: just a couple done for now, but need revision.

    [​IMG]

    However, not being very technically inclined, I keep thinking there's something strange about Nemesis. I got it working at the 10th or so attempt because more often than not, removing pixels to save space actually outputed a larger amount of code. Is this a common thing with compression?
     
    Last edited: Feb 16, 2020
  7. AURORA☆FIELDS

    AURORA☆FIELDS

    The cute one here Tech Member
    216
    24
    18
    Finland
    AMPS
    It is very difficult to explain how compression works because it usually depends on very unintuitive ideas and things that make things easier to compress, and depends largely on the algorithm used. Usually, compression favours repeated section, such as a simple repeating pattern over using less data, or very predictable easy patterns. This means, a solid tile of single colour will compress the best, followed by a short repeating pattern (for example just 2 pixels of different colours) horizontally. Of course, this is not very interesting, but compression can work over several tiles, or several similar patterns, depending on algorithm. Removing pixels won't do a lot, instead removing complexity will, but it also results in less interesting art.
     
    • Informative Informative x 2
    • List
  8. virgobit

    virgobit

    Member
    6
    0
    1
    @Natsumi: I understand it better now, thank you for the explanation - and everyone for the heads up about FW-KENSC :)

    Hopefully, I can show it off when it's done a couple of months from now.
     
  9. RadioTails

    RadioTails

    Member
    29
    16
    3
    Yay, I can finally post this!

    So use Nemesis Searcher (https://www.romhacking.net/utilities/766/) to separate all the nemesis graphics. It seems there are 37 for Fatal Labyrinth.

    Use this to decompress and re-compress the graphics (not sure why Natsumi is saying not to use it): https://www.romhacking.net/utilities/757/
    EDIT: Use FW-KENSC. I can confirm it works much better: https://forums.sonicretro.org/index.php?threads/fw-kensc-windows-shell-extension.36252/

    Having a quick look, 00019EDC.smd contains the icon menu graphics. So uncompress it, make edits and compress it.

    I would recommend you double the ROM from 128K to 256K. Just add 20000 bytes of FF. Change data at 1A4 to 0003FFFF (the rom header will say the rom ends at 3FFFF).

    Now insert you need compressed data at 20000 location. It's best to place the data in the spare rom data so you don't interfere with the code. Plus the compressed data can vary in size. Then search for 00019EDC in the rom. You should find two results. The one at 0C4A does appear in the trace I did. The 41F9 before it means it's a LEA instruction. Replace the offset with the one where the new graphic is located.

    As a bonus, the Nemesis Compressor starts at: 00019280 :)

    The other one 3EE0 I'm not sure if this is related to the graphic or not. If it's anything like how Mean Bean Machine sometimes lays it out:

    000B = Tells the game to use Nemesis Compression
    0000 = VRAM Location
    00019EDC = Graphic Offset

    You would need to investigate that. Also, the font doesn't seem to be in the compressed (although I would need to double check). I suspect it might be stored as 1bpp like in Alex Kidd. I can provide my notes if interested.
    EDIT: Yes, it is 1bpp. I sadly didn't write the offset down. Easy to find, just convert the bits into hex. As an example, if the tile was laid out like so:

    00000000 = 00
    00011000 = 18
    00111100 = 3C
    00100100 = 24
    00100100 = 24
    00111100 = 3C
    00011000 = 18

    So search: 00183C24243C18

    Hope this helps.
     
    Last edited: Feb 27, 2020
  10. BinBowie

    BinBowie

    Na Zdrowie! Member
    73
    24
    8
    USA
    Holy cow. I know this isn't adding much to the conversation but I'm loving the sprites you did. It's cool to see someone else hacking other Mega Drive games from my childhood besides Sonic. Definitely hook me up with a ROM of this when you're finished.

    Also if you need a musician for custom music. Please let me know. I can hook you up with my own tunes or remixes of the songs from in game if you'd like.
     
  11. virgobit

    virgobit

    Member
    6
    0
    1
    @RadioTails:

    Hi RadioTails, thank you for the reply!

    Using Nemesis searcher, I already have the vast majority of graphics indexed; there are only a couple which I'm not sure about, whether in nature or use (for instance, I noticed that some tiles are used in two different files but am unsure about why since one file already aggregates tiles, walls and interface icons and message boxes).

    The font isn't compressed, by the way. You're right that it is in 1bpp format and starts at offset 198E0 (it includes the japanese characters as well; actually the game also includes the japanese logo too. Maybe they figured it wouldn't be worth the trouble to replace it and just added the localized one).

    Initially I wasn't planning on expanding the ROM but I might just do that, if only because it's less time consuming to keep adjusting the artwork to the alloted space.

    I'm also thinking about the Sega CD version, released in the Game no Kandume Vol. 2 compilation, which uses a rearranged soundtrack (new intro and end songs but no music ingame). I'm wondering how difficult it might be to import the CD soundtrack to the expanded ROM? Will have to check the sizes.

    I will humbly and gladly accept any notes you can provide in the game. Thank you once again :)

    EDIT: The main hurdle so far has been tracking down the player character's palette for some reason. I already found several, from enemies to even the starting town, but his palettes have eluded me so far.

    @BinBowie:

    Thanks, glad you like them! The main objective so far has been reworking the visuals but since I have been curious about the Sega CD's version, I might just take you up on that :) As soon as this hits a more advanced state, I'll report back - expanding the ROM might just be enough for a reworked soundtrack, too :)
     
    Last edited by a moderator: Feb 29, 2020
  12. BinBowie

    BinBowie

    Na Zdrowie! Member
    73
    24
    8
    USA
    I should note that I own a copy of "Sega Games Can Vol. 2" which contains the Sega CD version of the game. Sadly that version only has (CD Quality) music for the Title Screen and main town area. Yet when you start up the game into the dungeon there is no music to be found. It's just silence with SFX.
     
    Last edited: Mar 4, 2020