CPKREDIR - loading and combining mods for Sonic Generations

Discussion in 'Engineering & Reverse Engineering' started by Korama, May 9, 2012.

  1. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    I haven't been able to get Sound redirection to work. I tried placing the following folder in the "Sound" folder(using CPKREDIR simple mode), and when I play the stage I just hear nothing at all. If I remove the folder, stock green hill zone plays out.
    http://fbe.am/6pt, I know that these files work since I can just use the regular repacked CPK with these and it works ingame.
     
  2. iGamr

    iGamr

    Member
    591
    10
    18
    What mod are you trying to use, exactly? I know you posted a link but it's not very clear to me how you'd use those files or what they'd do... Is it just a music swap?
     
  3. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    It's just the AAX sound files that are usually inside the green hill zone music CPK, but using Windmill Isle's instead. It was done by JoeTE back in January. The fact that the game plays no music whatsoever on the stage when placing the files there kind of indicates the redirection is doing something, but it's not working correctly for streaming it yet.
     
  4. Korama

    Korama

    Tech Member
    272
    0
    0
    Thanks. Without you, I'd barely have any test cases. ;)
    I've never tested redirecting those sound CPKs, I just assumed (hoped) it would work the same. But apparently it doesn't. Here's the relevant part of the log:

    [​IMG]

    After reading the TOC of SNG_01GHZ.cpk, the game tries to use the regular OPEN function to open those AAX files instead of just reading from the CPK. Very weird...
    I'll look into it.
     
  5. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    Does the TOC store the absolute address of the file in the CPK? I'd assume they stream the file instead of opening it via the regular CPK functions and store it in memory. Given the game crashes when the AAX files are on a different sampling sometimes, perhaps they use some math with that to find the exact position at which they should be reading. I would've given you a log but it doesn't seem to get created when I try to use it. I'm stupid, I didn't know there was a mode 2 for logging, although I doubt it's necessary now.
     
  6. Korama

    Korama

    Tech Member
    272
    0
    0
    Yes, LogType=2 is for logging to a file.
    The TOC of a CPK stores file offsets in a quasi-absolute way. You have to add 2048 (=size of a CPK header block) to the offset given by the TOC to get the absolute address. Unless the CPK version is 8 or higher, then you have to add the "ContentOffset" value given by the CPK header. ;)

    I just found a bug in my TOC routine, the directory name got corrupted somehow, that's why the game can't find the AAX files in the CPK. It's interesting that it actually tries to fall back to external files then! Don't tell me I could have spared myself all the work on CPKREDIR and just pass the game an empty TOC to make it look for external files...
    Oh well, even if that were true, you wouldn't have the arbitrary file location and cascading features. It was a good programming exercise anyway. :specialed:

    I was thinking the same at first, but now I don't think that's the case.
    Anyway, I suspect the CSB files might be responsible for those crashes and other issues. They obviously contain control information for the respective sound CPKs.
    I think CSB stands for "cue sheet binary". It's just another use of CRI's UTF (universal table format?).
    Here's a screenshot of SNG01_GHZ.csb in an experimental UTF reader that I made, a byproduct of CPKREDIR:
    [​IMG]
    If I find the time, I might release a full-blown UTF/CSB editor one day.
     
  7. Korama

    Korama

    Tech Member
    272
    0
    0
  8. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    Thanks for the quick support! It works perfectly now music-wise, no crashes, could quit the level fine. Will report if I run into any more stability issues later on.
     
  9. iGamr

    iGamr

    Member
    591
    10
    18
    If we already have CPKREDIR installed, which files need to be replaced?
     
  10. Korama

    Korama

    Tech Member
    272
    0
    0
    The DLL, of course. And the txt file, maybe.
    But replacing the other files shouldn't do any harm either. Just be aware that overwriting any ini file will naturally reset any settings you might have customized.
     
  11. iGamr

    iGamr

    Member
    591
    10
    18
    I was thinking it was along those lines, but I just figured I'd check. Thanks for the confirmation, though!

    (Also, damn... I'm getting close to the 20-post limit. Does anybody know if they generally wait until you hit 20 to decide whether or not you become a full member?)
     
  12. Josh

    Josh

    Oldbie
    1,763
    433
    63
    USA
    Could anyone put together a little tutorial on how to customize the music with this tool? I've always been interested in doing that, and I think this'll make it worthwhile.
     
  13. Guess Who

    Guess Who

    It's a miracle! Oldbie
    4,273
    39
    28
    Texas
    lol
    Oh hello there.
     
  14. Josh

    Josh

    Oldbie
    1,763
    433
    63
    USA
    Wow, I dunno how I missed that one, haha. Thanks a ton!
     
  15. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    I believe this tool's logging mode has been the cause of many stability issues, since I've disabled it and I got rid of:
    - Endless loading when quitting the stages
    - Varying performance when playing the same stage again(increased pop-in and loading times)

    It's widely known that printing to the console can be a source of lag for repeated operations(even more for a game that streams from the disk constantly), so my question is, is CPKREDIR's process in the same thread? How does the logging work that it could be causing these issues? It could just be luck and I'm drawing false conclusions, but since I've disabled it the game's been running much better for me.

    In any case, could you provide the default option in the INI as not-logging if it turns out to be the problem?
     
  16. Korama

    Korama

    Tech Member
    272
    0
    0
    Weird, I've never had any of those issues. But my system is fairly fast and very stable.

    Do you have any sources for that? Never heard anything of the sort.
    But yeah, if you print a ton of text to the console window, I guess it could slow things down, depending on how fast those console text output functions by Windows are.

    Sonic Generations has a bunch of threads, over a dozen, I think. CPKREDIR gets called by a CRI file loader thread that handles read requests by other threads, which will wait until the requested data has been retrieved.
    If you do grab the scrollbar thumb of the console window, or select text in it, then you WILL stall CPKREDIR's thread, which is the mentioned loader thread, and this will of course stall the thread that is waiting for the read request to finish. But if you leave the console window alone, it shouldn't be a problem, or so I thought at least.

    Funny thing is, even if you disable logging (LogType=0), CPKREDIR still performs exactly the same steps, that is preparing the logging messages, except for the final write to console or write to file calls. Yeah, I could obviously optimize that (don't waste time on building strings if they aren't output in the end), but I haven't gotten around to it yet. So, if disabling the console actually helps with your issues, then the problem aren't my logging routines but rather the output. Maybe console writes really are too slow. Have you checked if logging to file (LogType=2) has the same issues as the console? That'd be interesting to know. There's also an undocumented LogType=3 setting, which uses the OutputDebugString WinAPI function.

    Sure, that was my intention for the "finished" version anyway. But for these early releases, I thought it would be better if people had some immediate feedback whether CPKREDIR was actually running or not, via the console window.
     
  17. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    Not that I can find instantly, but you can try writing something simple in C to check. I think it's related to how Windows handles the I/O of the console in its own, it just seems to stall your program until it's done updating that console window.

    What you describe pretty much sounds like printing to the console is indeed the culprit.
     
  18. Yarharhar

    Yarharhar

    Oldbie
    54
    0
    0
    Dario is correct; printing to the windows console is really slow. That's why a lot of console apps offer a 'silent' mode. Windows console IO is simple/easy to work with, but man does it muck up performance.
     
  19. Korama

    Korama

    Tech Member
    272
    0
    0
    You're right. I did some research and found a couple of sources that confirm that console output is inherently slow.
    Thanks for making me aware of it. I'll disable the console by default in the next release.
     
  20. Bareirito

    Bareirito

    Member
    102
    0
    0
    Argentina
    Sonic - Mystery of the Chaos Emeralds
    I just don't know if it's related about the situation Dario and Korama mentioned, but I have a problem. I installed this program and installed the Unleashed mod, everything fine (save for some random crashes), but when I restored the original file and start a new save file, the game for some reason crashes after Modern GHZ when loading the main HUB world, not allowing me to continue any further. The worst thing is when I tried with another save file, I do pass through Modern GHZ and the game saves just fine! I tried reinstalling it, deleted the mod, CPKREDIR and I still have this problem, considering it worked fine before. :(

    I hope someone gets the solution of this problem.