don't click here

HiveRLE compression

Discussion in 'Engineering & Reverse Engineering' started by Hivebrain, Jan 27, 2024.

  1. Hivebrain

    Hivebrain

    Administrator
    3,046
    154
    43
    53.4N, 1.5W
    Github
    I wrote my own high speed compression format:

    https://github.com/cvghivebrain/HiveRLE

    Compression ratios aren't great, but they generally seem to be slightly better than SNKRLE. For data with lots of repeating bytes it'll actually compress better than Kosinski. I'm using it to compress level layouts.

    The HiveDec (to RAM) subroutine can be found in the asm folder. Decompressing to VRAM is untested and probably needs rewriting.
     
    • Like Like x 6
    • Useful Useful x 1
    • List
  2. rata

    rata

    Member
    684
    71
    28
    Argentina
    Trying to be useful somehow.
    Even though I decided that game editing is not for me (well, sorta), I'm always amazed with this forum and their many new compression methods.

    Do you have some comparission examples vs Kosinski, Comper, the improved Kosinski and Twizzler? It could help people on making a decision on where to pick which.
     
  3. OrionNavattan

    OrionNavattan

    Tech Member
    152
    150
    43
    Oregon
    Please trash, posted prematurely.
     
    • Informative Informative x 1
    • List
  4. Hivebrain

    Hivebrain

    Administrator
    3,046
    154
    43
    53.4N, 1.5W
    Github
    I also happen to have written a program for doing just that:

    https://github.com/cvghivebrain/CmpCompare

    Code (Text):
    1.  
    2. GHZ main level graphics:
    3. Uncompressed                  14752  
    4. compcmp.exe                   6648      45%
    5. comperx.exe                   6648      45%
    6. enicmp.exe                    12654     86%
    7. HiveRLE.exe                   13536     92%
    8. kosinski_compress.exe         4864      33%
    9. nemcmp.exe                    5615      38%
    10. snkcmp.exe                    15392     104%
    11. kosplus.exe                   4632      31%
    12.  
    13. Sonic on the title screen:
    14. Uncompressed                  16800  
    15. compcmp.exe                   10510     63%
    16. comperx.exe                   10510     63%
    17. enicmp.exe                    12716     76%
    18. HiveRLE.exe                   13533     81%
    19. kosinski_compress.exe         7424      44%
    20. nemcmp.exe                    6518      39%
    21. snkcmp.exe                    13886     83%
    22. kosplus.exe                   7024      42%
    23.  
    24. GHZ 256x256 mappings:
    25. Uncompressed                  32256  
    26. compcmp.exe                   11384     35%
    27. comperx.exe                   11384     35%
    28. enicmp.exe                    14702     46%
    29. HiveRLE.exe                   27161     84%
    30. kosinski_compress.exe         8336      26%
    31. nemcmp.exe                    17040     53%
    32. snkcmp.exe                    27072     84%
    33. kosplus.exe                   7710      24%
    34.  
    35. GHZ1 level layout:
    36. Uncompressed                  1024    
    37. compcmp.exe                   114       11%
    38. comperx.exe                   114       11%
    39. enicmp.exe                    112       11%
    40. HiveRLE.exe                   108       11%
    41. kosinski_compress.exe         128       12%
    42. nemcmp.exe                    170       17%
    43. snkcmp.exe                    110       11%
    44. kosplus.exe                   110       11%
    45.  
    46. A file containing the same byte repeated:
    47. Uncompressed                  512    
    48. compcmp.exe                   8         2%
    49. comperx.exe                   8         2%
    50. enicmp.exe                    20        4%
    51. HiveRLE.exe                   9         2%
    52. kosinski_compress.exe         16        3%
    53. nemcmp.exe                    22        4%
    54. snkcmp.exe                    8         2%
    55. kosplus.exe                   12        2%
    56.  
    57. A file containing random bytes:
    58. Uncompressed                  512    
    59. compcmp.exe                   548       107%
    60. comperx.exe                   548       107%
    61. enicmp.exe                    532       104%
    62. HiveRLE.exe                   518       101%
    63. kosinski_compress.exe         592       116%
    64. nemcmp.exe                    569       111%
    65. snkcmp.exe                    514       100%
    66. kosplus.exe                   578       113%
    67.  
    Twizzler isn't a normal command line program so it won't work with cmpcompare.
     
    Last edited: Jan 27, 2024
  5. OrionNavattan

    OrionNavattan

    Tech Member
    152
    150
    43
    Oregon
    I forgot you have an app to automate those comparisons, lol. I'll still share the results of my manual comparison, since one of the files I tested is quite massive. I used ARZ 1's level layout, OOZ's 16x16 maps (both from Sonic 2), the animated menu background maps from Sonic 2, and the PPZ 1 Present 256x256 maps from Sonic CD. Sizes are taken from the macOS file info dialogue. (And for the record, Twizzler does seem to work from the command line, at least in PowerShell, although it creates a copy of the input file with a .bin extension if it's invoked that way.)


    ARZ 1 Layout:
    Uncompressed: 4096 bytes
    SNKRLE: 1668 bytes
    HiveRLE: 1626 bytes
    Enigma: 1562 bytes
    Saxman: 1242 bytes
    Comper and ComperX: 1138 bytes
    Kosinski: 930 bytes
    Twizzler: 900 bytes

    OOZ 16x16:
    HiveRLE: 5648 bytes
    SNKRLE: 5610 bytes
    Uncompressed: 5600 bytes
    Comper and ComperX: 4648 bytes
    Enigma: 4132 bytes
    Saxman: 4102 bytes
    Kosinski: 3686 bytes
    Twizzler: 3653 bytes

    Sonic and Miles Animated Menu Background Map:
    HiveRLE & SNKRLE: 2242 bytes
    Uncompressed: 2240 bytes
    Enigma: 432 bytes
    Saxman: 294 bytes
    Kosinski: 782 bytes
    Twizzler: 53 bytes
    Comper and ComperX: 50 bytes

    PPZ 1 Present 256x256 Maps:
    Uncompressed: 61952 bytes
    HiveRLE: 50078 bytes
    SNKRLE: 49860 bytes
    Enigma: 32456 bytes
    Comperx: 25562 bytes
    Comper: 25560 bytes
    Saxman: 20964 bytes
    Kosinski: 18074 bytes
    Twizzler: 16721 bytes
     
  6. Sonic Hachelle-Bee

    Sonic Hachelle-Bee

    Taking a Sand Shower Tech Member
    805
    200
    43
    Lyon, France
    Sonic 2 Long Version
    Would be nice if all of them are referenced somewhere with download links / code samples and these comparisons with CmpCompare. A Wiki SCHG page maybe?
     
  7. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,709
    300
    63
    SonLVL
    You could implement support for this directly in SonLVL via a custom layout format class. This won't help if you want to use it for other things however. Perhaps someday compression types will be made extensible.