don't click here

Sonic Boom: Rise of Lyric File Formats

Discussion in 'Engineering & Reverse Engineering' started by NeKit, Dec 20, 2014.

  1. NeKit

    NeKit

    Member
    57
    0
    0
    Russia
    Sonic SCANF
    Most of Sonic Boom (Wii U version) data (except movies and sounds) is contained in .wiiu.stream. Movies are Bink Video and can be easily viewed by BIK Player.

    As for .wiiu.stream, QuickBMS script can be used to extract files from them in compressed format. I've written a small Python 3 script to decompress them: https://github.com/NotKit/sonic-boom-tools . It may have yet unknown bugs, but so far it's able to decompress things like .dds textures, .chr models (can be viewed by Noesis with cryengine_cgf.dll plugin) and level geometry. Compression itself seems to be LZSS variation. The format is as follows:
    Code (Text):
    1.  
    2. Size data
    3.     First byte
    4.         Bit 0-5 size bits
    5.         Bit 6   If set, copy size+3 bytes from unpacked stream - offset, otherwise read size bytes into unpacked stream
    6.         Bit 7   If set, read one more byte and add bits 0-6 from it to the end of size value
    7.     Next bytes
    8.         Bit 0-6 size bits
    9.         Bit 7   If set, read one more byte and add bits 0-6 from it to the end of size value
    10. Offset data (only if Bit 6 of first size byte is set)
    11.     Bit 0-6 offset bits
    12.         Bit 7   If set, read one more byte and add bits 0-6 from it to the end of offset value
    13.  
    For example, C1 BD 60 01 tell us to copy last 1+3 bytes (0xC1 & 0x3F) << (7 * 2) | (0xBD & 0x7F) << 7 | 0x60 = 24288 times (that's it, back offset is constant when copying).

    That's all for now. I'm probably going to write another script to extract .wiiu.stream files, since it's more convenient to have it both extracted and decompressed at once, and QuickBMS script seems to fail in some cases. Many thanks to TwilightZoney and Paraxade for buying the disc and getting the files ripped.
     
  2. Paraxade

    Paraxade

    Member
    186
    0
    0
    I wrote a tool for .stream files last week that can be downloaded here. It handles extraction + decompression simultaneously and comes with some bat files to make it simple to extract everything.

    also, here's some C++ code for decompression:

    Code (Text):
    1. u32 getSize(u8 *&src, bool seek)
    2. {
    3.     u32 size;
    4.     u8 byte = *src++;
    5.  
    6.     if (seek) size = (byte & 0x7F);
    7.     else      size = (byte & 0x3F);
    8.  
    9.     while (byte & 0x80)
    10.     {
    11.         byte = *src++;
    12.         size = (size << 7) | (byte & 0x7F);
    13.     }
    14.  
    15.     return size;
    16. }
    17.  
    18. bool decompress(u8 *src, u32 src_len, u8 *dst, u32 dst_len)
    19. {
    20.     u8 *src_end = src + src_len;
    21.     u8 *dst_end = dst + dst_len;
    22.  
    23.     while ((src < src_end) && (dst < dst_end))
    24.     {
    25.         u8 byte = *src;
    26.         u32 size = getSize(src, false);
    27.  
    28.         // repeat data: copy back from decompressed stream
    29.         if (byte & 0x40)
    30.         {
    31.             size += 3;
    32.             u32 seekSize = getSize(src, true);
    33.             u8 *seekStart = dst - seekSize;
    34.             u8 *seekEnd = dst;
    35.  
    36.             u8 *dstCopy = seekStart;
    37.             for (u32 b = 0; b < size; b++)
    38.             {
    39.                 *dst++ = *dstCopy++;
    40.                 if (dstCopy >= seekEnd) dstCopy = seekStart;
    41.             }
    42.         }
    43.  
    44.         // new data: read directly from compressed stream
    45.         else
    46.         {
    47.             memcpy(dst, src, size);
    48.             dst += size;
    49.             src += size;
    50.         }
    51.     }
    52.  
    53.     if ((src == src_end) && (dst == dst_end)) return true;
    54.     else return false;
    55. }
    .stream format is pretty simple:

    Code (Text):
    1. u32 strm_magic;
    2.  
    3. // read files until EOF
    4. struct file {
    5.   u32 compressed_size; // if 0, file is not compressed; use decompressed size
    6.   u32 decompressed_size;
    7.   u32 hash; // the game uses this to verify the integrity of the file data
    8.   u32 unknown;
    9.   string filename;
    10.   u8 data[];
    11. };
    I'm guessing the reason that QuickBMS script fails sometimes is because it doesn't account for uncompressed files.