KENSSharp is a set of libraries written in C# that contain compressors and decompressors for the Kosinski, Enigma, Nemesis and Saxman compression formats. It is a port of flamewing's port, which is included in Sonic 2 Special Stage Editor. This port was motivated by the desire to make <a href='index.php?showtopic=24572'>S2LVL</a> work on other implementations of the Common Language Infrastructure, such as Mono, in which P/Invoke is not available. sonicblur independently ported the compressors and decompressors from the original KENS library to C# in a project called KensNET; see next post. KENSSharp is licensed under the GNU Lesser General Public License, version 3 or later.
Sorry I didn't notice this sooner. My port you mentioned was finished that weekend as promised: http://www.sappharad.com/junk/KensNETv1.zip All 4 compressors and decompressors are giving identical results for me. Feel free to integrate anything into your codebase if you want. However, as you know I took a different strategy (line-by-line, as close to the original code as possible but with some bugfixes to the original code) so it's not following the same structure. Cleanup shouldn't be difficult now that it compiles and works. If needed, I'd be glad to add some additional signatures to each for working with Memory streams instead of files.
I've looked at it briefly, and I find that the code is barely comprehensible; the comments explain what's being done but not why. Of course, it's a problem with the original code, so you're not to blame here. flamewing's code, on the other hand, is more structured and I was able to better relate to the descriptions I read on Sega Retro. (However, flamewing's code for Nemesis decompression didn't work right, so I had to change some things to make it work correctly.) Starting from your code to make it more structured would be pointless when I can just start from more structured code like flamewing's. Furthermore, flamewing improved some compressors to give smaller results (in particular, revision 31 in s2-ssedit improved Nemesis compression), so that's another reason why I prefer using his code as a base. However, flamewing didn't make an Enigma a Saxman compressor and decompressor, so I'll probably use your code as a base for that. I feel a bit hypocritical for encouraging you to port the original code but not using it though...
All of the comments were also from the original, but I believe you're aware of that. I literally just pasted the original code into a .cs class and fixed each line one-by-one. I agree that in the end, your implementation is probably a better thing to do. When I was struggling to figure out why the Saxman decompressor wasn't working, comments or even documentation on the format (which I wasn't able to find) would've helped. In the process of fixing that code though, I eventually started to understand how the format worked. But a nice structure and comments would have helped. So keep up the good work. I just wanted to push something out quickly in the event you didn't continue and since I could finish it in a day. In the end, someone using the library doesn't need to care how awful the code inside is as long as it's working. I only briefly looked at yours, but the fact that it's broken up nicely is already a big improvement.
I think you mean saxman compressor/decompressor, as I have indeed done the Enigma compressor and decompressor (click here to go directly to it). Yeah, this is the single biggest hurdle I am facing on rewriting the saxman compressor and decompressor -- no documentation at all (it is also why I am delaying working on it).
Nemesis compression and decompression are now implemented. Indeed, I meant Saxman (de)compressor, sorry. And for the record, I'm keeping an eye on the s2-ssedit repository with Commit Monitor, so if you happen to find a way to improve the existing compressors or fix bugs, I'll know about it. :P Speaking of which... The way you're checking for codes (at "Find out if the data so far is a nibble code") doesn't make sense. During my tests, I use Green Hill Zone's first pattern set. Its header defines codes 000 and 001 (these are the shortest codes in the header). In your decompressor, as soon as one bit is read, it will match one of those codes, because you only consider the numeric value of the code. I used a binary tree instead, similar to the node class you use for encoding (and in fact, KENSSharp has 2 such classes, one used for encoding and the other used for decoding).
Oops, you are absolutely correct; I was extending too far the prefix-free property; I should also have been checking the code size, not just its value. I have fixed it in SVN now, so thanks for the report.
I've just committed the code for Enigma compression and decompression. flamewing, guess what... I found another problem! In your Enigma compressor, you have this: Code (Text): unsigned short next = unpack[pos+1]; int delta = int(next) - int(v); if (delta == -1 || delta == 0 || delta == 1) { flush_buffer(buf, bits, mask, packet_length); size_t cnt = 1; unsigned short prev = next; next += delta; for (size_t I = pos + 2; I < unpack.size() && cnt < 0xf; I++) { if (next == unpack[I]) { if (delta == 1 && prev == incrementing_value) break; next += delta; cnt++; } else break; } I'm not too sure what you're trying to do with prev, but you're comparing it to incrementing_value too late (and neither prev not incrementing_value changes in the inner loop, so testing this in the loop is pointless). Right after initializing next, you should check if it's equal to incrementing_value and output an inline copy of a single word. In the inner loop, I compare unpack to incrementing_value and leave the loop if they're equal. I used tilemaps\Title Screen.bin from Sonic 1 for my tests, and my code now outputs a much smaller file when recompressed (222 bytes instead of 272 bytes with your implementation, and 269 bytes with The Sega Data Compressor), because the incrementing word is actually used correctly.
You are correct, of course; this happened because I gave much less love to the Enigma compressor than I did for the Nemesis compressor. My intention was to do what you did, but -- as you noted -- I botched the logic in my implementation; at least, this isn't an error that caused compression errors. This misuse of the incrementing word was the first mistake I saw in the original implementation -- it was computed as the word that gave the longest incremental run, but failed to take full advantage of it because of the inline incrementing runs. I back-ported your changes. Thanks again.
Saxman compression and decompression is done. KENSSharp is now complete! At MainMemory's request, I've put an option on Enigma and moduled Kosinski to choose between big endian and little endian, in order to support Sonic CD PC and Sonic & Knuckles Collection.
I seem to have found a bug: Kosinski decompress "mappings/16x16/EHZ.bin" from the Sonic 2 disassembly, compress as Enigma, decompress. The result doesn't match the original decompressed file.
I've written a command-line interface for KensSharp, the source code to which will be available as soon as the repository works again. Code (Text): Usage: kenssharp [options] input output Arguments: -h, --help Shows this help screen. -c, --compress=FORMAT Compresses a file with the specified FORMAT. -d, --decompress=FORMAT Decompresses a file with the specified FORMAT. -r, --recompress=FORMAT Decompresses and recompresses a file with the specified FORMAT. If output file is not given, input file will be recompressed in place. -s, --same-filename The output file name will be the same as the input, with an extension indicating the type of compression: .kos, .eni, .nem, .sax, .kosm or .unc. -l, --little-endian Uses little endian (Intel) byte order for Enigma and Moduled Kosinski formats. -n, --no-size Do not include size in Saxman compressed file. Formats: Kosinski, kos, k The general-purpose Kosinski compression format. Enigma, eni, e The Enigma compression format for plane mappings. Nemesis, nem, n The Nemesis compression format for art tiles. Saxman, sax, s The Saxman compression format used by Sonic the Hedgehog 2's sound driver and music files. ModuledKosinski, The general-purpose Moduled Kosinski KosinskiModuled, compression format used by Sonic 3 & Knuckles. mkos, kosm, mk, km Download binaries.
I've just finished writing a shell extension for Windows explorer that adds a submenu to the context menu for all files, allowing you to compress and decompress data. It looks like this: You should be aware that the kenssharp program is invoked with the -s option, so the output file will have the same name as the input file, with the extension indicating the type of compression. You can download the installer here. I have not tested the installer or the extension itself on a 32-bit platform, so let me know if it doesn't work.
HOLY GODDAMN SHIT!!! ... Because this is going be pretty useful for when compressing a lot of files at once (I'm talking about more than 500 files) without needing to copy derecmp, entering the DOS shell, and all that
https://github.com/sonicretro/KENSSharp/releases/tag/v1.1 A new release is out, now with Comper compression added.
https://github.com/sonicretro/KENSSharp/releases/tag/v1.2 Here's another release. It fixes a fair few bugs, and adds support for Kosinski+ compression.
https://github.com/sonicretro/KENSSharp/releases/tag/v1.3 Kosinski+ has been updated to its newer format. This improves decompression time (on the Mega Drive, that is).