[FraGag]

KENSSharp is a set of libraries written in C# that contain compressors and decompressors for the Kosinski, Enigma, Nemesis and Saxman compression formats. It is a port of flamewing's port, which is included in Sonic 2 Special Stage Editor. This port was motivated by the desire to make S2LVL work on other implementations of the Common Language Infrastructure, such as Mono, in which P/Invoke is not available.

sonicblur independently ported the compressors and decompressors from the original KENS library to C# in a project called KensNET; see next post.

KENSSharp is licensed under the GNU Lesser General Public License, version 3 or later.

This post has been edited by FraGag: 21 March 2011 - 06:49 PM

[sonicblur]

Sorry I didn't notice this sooner. My port you mentioned was finished that weekend as promised:
http://www.sappharad.com/junk/KensNETv1.zip

All 4 compressors and decompressors are giving identical results for me. Feel free to integrate anything into your codebase if you want.
However, as you know I took a different strategy (line-by-line, as close to the original code as possible but with some bugfixes to the original code) so it's not following the same structure. Cleanup shouldn't be difficult now that it compiles and works. If needed, I'd be glad to add some additional signatures to each for working with Memory streams instead of files.

This post has been edited by sonicblur: 04 March 2011 - 08:07 PM

[FraGag]

Sorry I didn't notice this sooner. My port you mentioned was finished that weekend as promised:
http://www.sappharad.com/junk/KensNETv1.zip

All 4 compressors and decompressors are giving identical results for me. Feel free to integrate anything into your codebase if you want.
However, as you know I took a different strategy (line-by-line, as close to the original code as possible but with some bugfixes to the original code) so it's not following the same structure. Cleanup shouldn't be difficult now that it compiles and works.

I've looked at it briefly, and I find that the code is barely comprehensible; the comments explain what's being done but not why. Of course, it's a problem with the original code, so you're not to blame here. flamewing's code, on the other hand, is more structured and I was able to better relate to the descriptions I read on Sega Retro. (However, flamewing's code for Nemesis decompression didn't work right, so I had to change some things to make it work correctly.) Starting from your code to make it more structured would be pointless when I can just start from more structured code like flamewing's. Furthermore, flamewing improved some compressors to give smaller results (in particular, revision 31 in s2-ssedit improved Nemesis compression), so that's another reason why I prefer using his code as a base.

However, flamewing didn't make an Enigma a Saxman compressor and decompressor, so I'll probably use your code as a base for that. I feel a bit hypocritical for encouraging you to port the original code but not using it though...

This post has been edited by FraGag: 06 March 2011 - 11:36 AM

[sonicblur]

I've looked at it briefly, and I find that the code is barely comprehensible; the comments explain what's being done but not why. Of course, it's a problem with the original code, so you're not to blame here. flamewing's code, on the other hand, is more structured and I was able to better relate to the descriptions I read on Sega Retro. (However, flamewing's code for Nemesis decompression didn't work right, so I had to change some things to make it work correctly.) Starting from your code to make it more structured would be pointless when I can just start from more structured code like flamewing's. Furthermore, flamewing improved some compressors to give smaller results (in particular, revision 31 in s2-ssedit improved Nemesis compression), so that's another reason why I prefer using his code as a base.

All of the comments were also from the original, but I believe you're aware of that. I literally just pasted the original code into a .cs class and fixed each line one-by-one.

I agree that in the end, your implementation is probably a better thing to do. When I was struggling to figure out why the Saxman decompressor wasn't working, comments or even documentation on the format (which I wasn't able to find) would've helped. In the process of fixing that code though, I eventually started to understand how the format worked. But a nice structure and comments would have helped.

So keep up the good work. I just wanted to push something out quickly in the event you didn't continue and since I could finish it in a day. In the end, someone using the library doesn't need to care how awful the code inside is as long as it's working. I only briefly looked at yours, but the fact that it's broken up nicely is already a big improvement.

[flamewing]

However, flamewing didn't make an Enigma compressor and decompressor,

I think you mean saxman compressor/decompressor, as I have indeed done the Enigma compressor and decompressor (click here to go directly to it).

I agree that in the end, your implementation is probably a better thing to do. When I was struggling to figure out why the Saxman decompressor wasn't working, comments or even documentation on the format (which I wasn't able to find) would've helped. In the process of fixing that code though, I eventually started to understand how the format worked. But a nice structure and comments would have helped.

Yeah, this is the single biggest hurdle I am facing on rewriting the saxman compressor and decompressor -- no documentation at all (it is also why I am delaying working on it).

[FraGag]

Nemesis compression and decompression are now implemented.

However, flamewing didn't make an Enigma compressor and decompressor,

I think you mean saxman compressor/decompressor, as I have indeed done the Enigma compressor and decompressor (click here to go directly to it).

Indeed, I meant Saxman (de)compressor, sorry. And for the record, I'm keeping an eye on the s2-ssedit repository with Commit Monitor, so if you happen to find a way to improve the existing compressors or fix bugs, I'll know about it. :P Speaking of which...

(However, flamewing's code for Nemesis decompression didn't work right, so I had to change some things to make it work correctly.)

The way you're checking for codes (at "Find out if the data so far is a nibble code") doesn't make sense. During my tests, I use Green Hill Zone's first pattern set. Its header defines codes 000 and 001 (these are the shortest codes in the header). In your decompressor, as soon as one bit is read, it will match one of those codes, because you only consider the numeric value of the code. I used a binary tree instead, similar to the node class you use for encoding (and in fact, KENSSharp has 2 such classes, one used for encoding and the other used for decoding).

This post has been edited by FraGag: 12 March 2011 - 04:31 AM

[flamewing]

The way you're checking for codes (at "Find out if the data so far is a nibble code") doesn't make sense. During my tests, I use Green Hill Zone's first pattern set. Its header defines codes 000 and 001 (these are the shortest codes in the header). In your decompressor, as soon as one bit is read, it will match one of those codes, because you only consider the numeric value of the code. I used a binary tree instead, similar to the node class you use for encoding (and in fact, KENSSharp has 2 such classes, one used for encoding and the other used for decoding).

Oops, you are absolutely correct; I was extending too far the prefix-free property; I should also have been checking the code size, not just its value. I have fixed it in SVN now, so thanks for the report.

[FraGag]

I've just committed the code for Enigma compression and decompression.

flamewing, guess what... I found another problem! In your Enigma compressor, you have this:

CODE

            unsigned short next = unpack[pos+1];
            int delta = int(next) - int(v);
            if (delta == -1 || delta == 0 || delta == 1)
            {
                flush_buffer(buf, bits, mask, packet_length);
                size_t cnt = 1;
                unsigned short prev = next;
                next += delta;
                for (size_t I = pos + 2; I < unpack.size() && cnt < 0xf; I++)
                {
                    if (next == unpack[I])
                    {
                        if (delta == 1 && prev == incrementing_value)
                            break;
                        next += delta;
                        cnt++;
                    }
                    else
                        break;
                }

I'm not too sure what you're trying to do with prev, but you're comparing it to incrementing_value too late (and neither prev not incrementing_value changes in the inner loop, so testing this in the loop is pointless). Right after initializing next, you should check if it's equal to incrementing_value and output an inline copy of a single word. In the inner loop, I compare unpack[I] to incrementing_value and leave the loop if they're equal. I used tilemaps\Title Screen.bin from Sonic 1 for my tests, and my code now outputs a much smaller file when recompressed (222 bytes instead of 272 bytes with your implementation, and 269 bytes with The Sega Data Compressor), because the incrementing word is actually used correctly.

[flamewing]

I'm not too sure what you're trying to do with prev, but you're comparing it to incrementing_value too late (and neither prev not incrementing_value changes in the inner loop, so testing this in the loop is pointless). Right after initializing next, you should check if it's equal to incrementing_value and output an inline copy of a single word. In the inner loop, I compare unpack[I] to incrementing_value and leave the loop if they're equal. I used tilemaps\Title Screen.bin from Sonic 1 for my tests, and my code now outputs a much smaller file when recompressed (222 bytes instead of 272 bytes with your implementation, and 269 bytes with The Sega Data Compressor), because the incrementing word is actually used correctly.

You are correct, of course; this happened because I gave much less love to the Enigma compressor than I did for the Nemesis compressor. My intention was to do what you did, but -- as you noted -- I botched the logic in my implementation; at least, this isn't an error that caused compression errors. This misuse of the incrementing word was the first mistake I saw in the original implementation -- it was computed as the word that gave the longest incremental run, but failed to take full advantage of it because of the inline incrementing runs. I back-ported your changes. Thanks again.

[FraGag]

Saxman compression and decompression is done. KENSSharp is now complete!

At MainMemory's request, I've put an option on Enigma and moduled Kosinski to choose between big endian and little endian, in order to support Sonic CD PC and Sonic & Knuckles Collection.

This post has been edited by FraGag: 21 March 2011 - 07:59 PM

[MainMemory]

AFAIK Sonic CD PC doesn't use any of the MD compression formats.

But S&KC does, so thanks.

This post has been edited by MainMemory: 21 March 2011 - 06:53 PM

[MainMemory]

I seem to have found a bug: Kosinski decompress "mappings/16x16/EHZ.bin" from the Sonic 2 disassembly, compress as Enigma, decompress. The result doesn't match the original decompressed file.

[FraGag]

I seem to have found a bug: Kosinski decompress "mappings/16x16/EHZ.bin" from the Sonic 2 disassembly, compress as Enigma, decompress. The result doesn't match the original decompressed file.

Right, there was a little bug in the Enigma decompressor. It's now fixed.

[MainMemory]

I've written a command-line interface for KensSharp, the source code to which will be available as soon as the repository works again.

Usage: kenssharp [options] input output

Arguments:

    -h, --help              Shows this help screen.
    -c, --compress=FORMAT   Compresses a file with the specified FORMAT.
    -d, --decompress=FORMAT Decompresses a file with the specified FORMAT.
    -r, --recompress=FORMAT Decompresses and recompresses a file with the
                            specified FORMAT. If output file is not given,
                            input file will be recompressed in place.
    -s, --same-filename     The output file name will be the same as the
                            input, with an extension indicating the type of
                            compression: .kos, .eni, .nem, .sax, .kosm or
                            .unc.
    -l, --little-endian     Uses little endian (Intel) byte order
                            for Enigma and Moduled Kosinski formats.
    -n, --no-size           Do not include size in Saxman compressed file.

Formats:

    Kosinski, kos, k    The general-purpose Kosinski compression
                        format.
    Enigma, eni, e      The Enigma compression format for plane
                        mappings.
    Nemesis, nem, n     The Nemesis compression format for art tiles.
    Saxman, sax, s      The Saxman compression format used by Sonic the
                        Hedgehog 2's sound driver and music files.
    ModuledKosinski,    The general-purpose Moduled Kosinski
    KosinskiModuled,    compression format used by Sonic 3 & Knuckles.
    mkos, kosm, mk, km

Download binaries.

This post has been edited by MainMemory: 12 October 2013 - 06:33 PM

[MainMemory]

I've just finished writing a shell extension for Windows explorer that adds a submenu to the context menu for all files, allowing you to compress and decompress data. It looks like this:

You should be aware that the kenssharp program is invoked with the -s option, so the output file will have the same name as the input file, with the extension indicating the type of compression.
You can download the installer here. I have not tested the installer or the extension itself on a 32-bit platform, so let me know if it doesn't work.