[Up: 10/16/04] TSDC V2.2, KENS V1.4 + Source

Discussion in 'Engineering & Reverse Engineering' started by Magus, Aug 26, 2004.

  1. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    -----------------------------------------------------------------

    LATEST UPDATES

    KENS V1.4 RC3
    * Corrected a bug that prevented the functions that output data to a buffer to create the buffer (could only use already initialized buffers). Thanks a LOT to Korama for helping me fixing this.
    * Fixed all the compression functions so they produce files with an even size
    * Included the Delphi unit corrected by Korama (my changes to this unit were not really good - I'm not a Delphi programmer)
    * Headers and units are now in a sub-directory named "Include"
    * Updated the info in the README.HTML file

    KENS V1.4 RC2
    * Turned all functions to local functions, so they all use temporary files
    * Removed the temporary local Kosinski functions (as the original functions are now local)
    * Added functions that output data to a buffer rather than to a file (thanks to Stealth for asking for that improvement)
    * Edited the headers and units to reflect the changes
    * Converted README.TXT, VERSION.TXT and LICENSE.TXT to HTML, because it's easier to read
    * Updated the info in the README.HTML file

    KENS V1.4 RC1
    * Corrected a bug in the Enigma Decompression algorithm. Thanks to Stealth for reporting and Nemesis for helping in fixing it.
    * Added "Local" Kosinski functions, which copy the file to the Windows temporary directory before compressing or decompressing. This fixes a speed issue when the source or destination files are on a remote computer. Thanks to Stealth for finding out this issue (we spent hours trying to find out why it was so slow) and helping in fixing it (although I used a different method from the one used in the unofficial version that he released).
    * Edited the headers and units to reflect the changes in the Kosinski functions

    TSDC V2.2 RC3
    - Repacked with version 1.4 RC3 of the KENS DLLs

    -----------------------------------------------------------------

    Hello my friends !

    Here's the last release of The Sega Data Compressor. It fully supports both compression and decompression of Kosinski, Engima, Nemesis and Saxman formats. TSDC should no longer be updated (but could be repacked with newer versions of the DLLs), unless there's a new compression format.

    As for KENS, it is a pack that contains 4 DLLs, each of them featuring compression and decompression of a format. They are distributed under LGPL, so you have access to their source code. These DLLs can be used by both standard (AKA C/C++ :D), Visual Basic and Game Maker applications. Using these DLLs instead of having the code written in the application allows to correct bugs in the algorithm without having to recompile the application. I made these DLLs because I had several applications using the algorithms, and it was a pain to update all of them each time I changed something in an algorithm. Though, the very first time I compiled them, it was for Beta, but I then realized they would be extremely useful for everybody, me included. TSDC uses these DLLs.

    If you need help reusing these DLLs in your applications, don't hesitate to ask me for help.

    TSDC-V2.2-RC3 (ACE - 91 KB)
    TSDC-V2.2-RC3 (SFX ACE - 153 KB)
    TSDC-V2.2-RC3 (ZIP - 228 KB)

    KENS-V1.4-RC3 (ACE - 71 KB)
    KENS-V1.4-RC3 (ZIP - 125 KB)

    KENS-V1.4-RC3-Source (ACE - 43 KB)
    KENS-V1.4-RC3-Source (ZIP - 73 KB)

    Also available on SourceForge (KENS 1.4 RC3 and TSDC 2.2 RC3 not uploaded yet).

    As I heard that some people were asking how that tool works, here are some basic explanations:

    - In order to decompress a file, open one of the decompression tool, and choose the ROM from which you want to extract data, using the first "Set" button. Then, choose a file that will receive the decompressed data, using the second "Set" button. Then, you can either choose a location from the list or typing one yourself (in that case, you MUST write it in hex, and without the 0x prefix). Finally, press the "Decompress" button. The status line should say "Processing..." for a very short time (usually 2 or 3 ms, unless it's a big archive), and then "Done !".

    - Now, compression: run one of the compression tools, and choose the file containing the data which you want to compress, using the first "Set" button. Then, choose a file that will receive the compressed data, using the second "Set" button. All you have to do now is clicking the "Compress" button. The status line should say "Processing..." for a few seconds, and then "Done !".

    - Specific options:
    * Enigma: when compressing or decompressing, you can choose to add and remove padding. This option should only be used when compressing or decompressing a S1 Special Stage. If you added padding when decompressing the Special Stage, then you will have to remove it when recompressing.
    * Kosinski: when compressing or decompressing, you can choose whether the archive is moduled or not. Be very careful when using this. Moduled archives can't be used in place of standard archive and vice versa. In the decompressor, this option is semi-automatic: whenever you click a location line ending by (M), the checkbox will automatically be checked. Do remember to recompress these archives using the moduled option.
    * Kosinski (2): when compressing, you can change the sliding window and maximum recurrence length. This does not have any real use for end-users, but it might be interresting for hackers. If you do not know what I'm talking about, then you will not need these options.

    - Warning: Enigma and Nemesis compression formats are data-type specific. You CAN'T use the Enigma and Nemesis compression / decompression tools for archiving. You could use the Kosinski tools however, but this is pointless, as the compression process is quite long, and the compression ratio not interresting enough. As for the Saxman compression format, it can only compress very small files, and the maximum size of a compressed file is 64 K (I think).

    - Difficulties downloading: if you can't download the files, try opening the FTP folder instead of clicking that link, then drag-drop the file you want to download from your IE window to the desktop or on any folder.
     
  2. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Okay, I've worked on it three more days, and made several improvements. The new key feature is the ability to output data to buffers instead of a new file. Enjoy it !
     
  3. Korama

    Korama

    Tech Member
    272
    0
    0
    Good job, as usual. :D

    I see that you also updated the Delphi unit, nice. :(
    However, I don't think that your CompToBuf/DecompToBuf work, at least not the way they are declared in Delphi.

    The parameter "DstBuffer" is passed by value. That's fine as long as the caller initialized the buffer, because the DLL will copy the data to the specified address. However, if DstBuffer is NULL, the DLL will change its value (by allocating memory), but that changed value will not be returned to the caller! That would only work if DstBuffer was passed by reference, as pointer to the pointer to a char. ;)
    That's "char **DstBuffer", I think.

    In Delphi declarations you use "var" to pass parameters by reference instead of value (or "out", in case the parameter is used for output only).

    Example:
    Code (Text):
    1. function NCompToBuf(SrcFile: PChar; var DstBuffer: PByte): Longint; cdecl;
    That would pass the address of the DstBuffer variable, not its value. Note that I also changed the type from "PChar" to "PByte", because in Delphi PChar is normally only used for real characters, not for binary data. Another option would be to use the basic typeless "Pointer".
     
  4. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Thanks for the info. Well, I'm not a Delphi programmer at all, so, could you please correct the unit and test it to check if it works correctly ?
     
  5. Korama

    Korama

    Tech Member
    272
    0
    0
    I'll gladly correct the Delphi unit.
    But first you should fix your C++ sources, because as I said, a problem will occur if you let one of the CompToBuf functions allocate the buffer for you.

    Anyway, let's better continue talking about that in PMs or on MSN. :(
     
  6. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Yep, I replyed to your PM. I'll be fixing them right now. Thanks a lot.
     
  7. LOst

    LOst

    Tech Member
    4,889
    1
    16
    I'm using your data compressor for the first time. However, compressing in Enigma will often end in a filesize not dividable by 2. That will end up in Address Error for any data inserted after the data generated by your program.

    You must fix that. The Genesis can't handle data starting an odd address. Applies to all compression format! Not just Enigma.
     
  8. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    I'll consider you just asked "Could you please fix this ?". :D

    That'll be done for the next release.
     
  9. LOst

    LOst

    Tech Member
    4,889
    1
    16
    Thanks!

    It's very simple. You just check the output size just before you close the file, and if it is odd, then write one more byte:

    if ((output_size & 1) == 1)
    fwrite (&zero_byte, 1, 1, file_pointer);
     
  10. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Yep, that's pretty much the way I wanted to do it in my algorithms.

    However, the algorithms provided by Nemesis do not use the ANSI-C I/O functions, so I'll have to be tricky. :D
     
  11. LOst

    LOst

    Tech Member
    4,889
    1
    16
    I know. And even worse. I tried to port Nemesis' compressor into Watcom C for DOS for use with LOst Library, but something in that code is crashing... Have you experience that? The decompressor is fine! But the compressor is older code, and it is unstable
     
  12. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Are you talking about Nemesis' compressor ? Well, I made a DOS port some time ago, and it worked well if I remember correctly. This was under DJGPP.

    Anyway, I'm studying this format in order to make my own algorithm, so I can fix bugs more easily. Let me tell you this is a real pain. But I will end up understanding this format. Even if I have to stay awake 24/24, 7/7. :D
     
  13. LOst

    LOst

    Tech Member
    4,889
    1
    16
    No! I mean the Decompressor!
     
  14. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Well, you said compressor earlier. Anyway I was asking if it was the Nemesis or Enigma ones. Seems you're talking about the Nemesis ones. :D However, both the compressor and decompressor worked. No worries though, I'll be making new algorithms for these formats. It might take some time, but I'll do it anyway.
     
  15. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Hello there !

    KENS 2.2 RC3 has been uploaded and is ready for download !
    It mainly includes fixes. Check the original post for more info.
    As always, feedback is appreciated. ;)
     
  16. Hivebrain

    Hivebrain

    Administrator
    2,787
    0
    16
    53.4N, 1.5W
    HivePal 2.0
    Even filesize. That'll save me a lot of time - no more checking compressed files.
     
  17. Magus

    Magus

    Fallen Hacker From The Past Tech Member
    157
    0
    0
    Paris, France
    Translating Grandia 2 PC to French
    Yep, you have to thank LOst for asking for that. :D

    BTW, be sure to update it on your site, as there are critical changes in this release.
     
  18. fuzzbuzz

    fuzzbuzz

    Tech Member
    625
    0
    0
    Here
    Many Many Programming/hacking tools...
    Move me to Archive, plz. :) k thanks. (I was looking for this over there. :D)