Sonic Lost World (PC / Wii U) Hacking & Resources

Discussion in 'Engineering & Reverse Engineering' started by Dario FF, Jan 1, 2015.

  1. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    So there's a PC version of this game apparently.

    Tools
    CPKREDIR: To be able to load and modify files without touching the CPK files.

    LibGens: Tools to work with formats from Unleashed / Generations / Lost World.

    SonicGLvl-lostworld: A modified branch of SonicGLvl 0.9 by Paraxade to work with Lost World. Heavily unfinished & experimental (like regular GLVL is), but it's possible to work with it.

    A lot of the research from Generations applies to Lost World, so you might want to look at the SCHG page for it.

    Older thread below spoiler:
    Development and File Structure

    The speculation behind Sonic Team still being organized into two teams for the mainline games and the Storybook/Colors team developing Lost World becomes even more apparent with a brief examination of the files. The folder/file structure and file formats use share a LOT of similarities with Colors. The project's name internally is "Sonic2013" (Colors was "Sonic2010"), and almost all the directories they exported their stuff from was paths in the style of "Sonic2010/Sonic2013", heavily hinting at which team developed this.

    The files themselves are the most reliable proof as they share a lot of formats with Colors, but the biggest change is that instead of relying on Nintendo's Wii model formats, they now used a modified version of the Hedgehog Engine's formats.

    Criware Pack (.cpk): Compatible with current tools, there's only one CPK used for the entirety of the game's assets (not movies). About 1.8 GB. (Generations totals to around 5.2 GB)

    Layout Object Files (.orc): Just like in Colors, there's a dedicated "set" folder with .orc files that are used for the object level layouts. While I haven't looked much into the format, besides a small file header upgrade it seems both Colors .orc and Lost World .orc are very similar. Nowhere near as easy to deal with as Generations XML formats sadly, but not impossible to crack either.

    Lots and lots of lua files: Again like in Colors, lua files are used for defining a lot of the stage's properties (in Generations XMLs are used instead), and are also very similarly coded. The new things are a lot of graphical parameters that have the same names as in the Hedgehog Engine games (stuff like SceneEffect.prm.xml's properties).

    Animation Scripts (.anm): Not sure what their purpose is but it seems like they were used in Colors as well.

    Pac Pack Files (.pac): Seems like a new package format in place of .arc or .ar. This format is more complicated than it looks at first glance compared to any package format I've ever seen. The main problem that comes from it is that some certain file formats that are packed inside (.effect files, etc.), end up sharing their strings and offset table with the entire pac file. This makes it a huge pain to make a reliable extractor and repacker as the files have to be basically rebuilt. Luckily they don't need to be cracked, as the offset table in the pac file is all the information needed to fix this. I now have a working PAC extractor/repacker (that I hope to be able to test in the future in the console) that automatically appends some metadata to the affected file formats inside and allows them to be properly repacked.

    Updated Hedgehog Engine Formats (.terrain-model, .terrain-instanceinfo, .material, .model): These should be familiar to anyone who's modded Generations, and to my surprise they were mostly compatible with the existing tools already. The main difference seems to be that the header got slightly updated, and the root node offset got changed to 16 from 24 (0x10 from 0x18). I don't really understand the purpose behind the new header yet (probably some context information as they need it), but it shouldn't be a big problem. It seems the game also uses regular files without the header that are pretty much the same thing from Generations, and I'll theorize they can run on real hardware as well.

    Images:
    [​IMG]
    [​IMG]


    Lightfield (.lft): Compatible with Generations, no changes whatsoever. Determines how Sonic and objects are shaded depending on the section of the level he's in, basically simulating the light bounces from the terrain.

    Havok 2012 Formats (anm.hkx, skl.hkx, phy.hkx): Also familiar to anyone who worked with Generations, the Hedgehog Engine seems to rely on Havok's implementation of animation, and ends up appearing on Wii U, a format that does not actually exist as a target platform in the tools. The version of the files is hk_2012.2.0-r1, and lack any metadata of any kind that describes the formats. They're also in big-endian format. HkxConverter (developed for the Unleashed 360 to PC conversion) needed an update to read the metadata out of other files (that I exported myself with the Havok 2012 Content Tools and used similar metadata to Lost World) to be able to properly convert the Wii U HKX to PC HKX. The result is usable, and with some luck, can actually be converted back to 2010 if the formats aren't very different. Here's a lot of TwilightZoney's work regarding the animations, along with the converted Sonic Lost World .model to Generations:

    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]

    Comparison of Sonic Lost World Model (yellow) to Sonic Generations Model (blue):
    [​IMG]

    Gameplay Videos of Sonic Lost World Model with Lost World Animations back-ported to Generations:
    http://youtu.be/fuK6bkhYAwo
    http://youtu.be/BHZln9k_cF0


    Hedgehog Engine Changes and Terrain

    Ever since its reveal it was very apparent that Lost World was a Hedgehog Engine game that was heavily nerfed graphically, and its files tell the same story as well. The two reasons are pretty obvious, a reduced budget (exclusive deal) and a 60 FPS target frame-rate instead of the usual 30 on similar hardware. The nerf affects pretty much every aspect of the engine: the quality of the shaders, the overall size and detail of the geometry, the GI quality, the elaborate streaming engine, etc.

    Shader Quality: Most of the shaders from Generations made the transition to Wii U as well as several new ones, so that means that almost all Generations materials are compatible with Lost World. However, the overall quality of the shading seems to have gotten a significant downgrade and it's very apparent with the SonicGLvl screenshots that will come later (that use Generations shaders). Certain shader parameters aren't used the same way as in Generations and several new ones were introduced. This causes huge consistency problems when trying to preview these models with the Generations shaders.

    Geometry Size & Detail: The numbers tell the entire story here, as I was really surprised when I was told the size of the Sonic Lost World dump. It was to be expected but the difference in size when compared to Generations is still stunning.

    All Lost World terrain size (uncompressed): 1.953.601.752 bytes (1.9 GB)
    All Generations terrain size (ACTUALLY COMPRESSED): 4.604.474.273 bytes (4.6 GB)

    The terrain in Lost World is not compressed and is simply dumped inside Pac files. Generations compresses all terrain inside PFDs and heavily benefits from it, with an almost 70% smaller compression ratio for each Stage.pfd. The terrain from 4 stages of Unleashed UNCOMPRESSED is as big as the entirety of the terrain in Lost World. We already know the reasons but it's still insane to think how much Sonic Team reduced the workload for the stages.

    Global Illumination Quality: The GI Streaming and the GI Atlas texture system are gone, in comes individual GI textures for each instance file in the stage. There's no longer any level of variable quality to the GI, and only the lowest quality ones as possible were rendered and used in the game. The shading algorithm however luckily remains the same, and there's no need to come up with new rendering techniques to preview it or create our own.

    Streaming Engine: The Hedgehog Engine in both Generations and Unleashed uses a lot of files to determine Terrain Groups and Terrain Blocks. These control when to load and unload chunks of terrain, when to cull it from view with a Visibility Tree, etc. That entire system is gone and instead the game will auto-load anything that is in the terrain folder (I assume since I haven't seen anything that describes what terrain to load). While this makes terrain repacking extremely easy in comparison to Generations, it also limits a lot what can be done with the game. The game simply streams terrain by just using the bounding spheres of each terrain node and loading it if it's close to the camera.


    Terrain Previewing

    As expected, since a lot of the formats are similar, it's pretty easy to add some compatibility into my current branch of SonicGLvl and preview it with the Generations shaders. However as mentioned before, the changes means a lot of materials are incompatible and not supposed to be previewed like this, leading to a lot of red materials (used when the shader does not exist in my database), broken uvs, broken lighting, etc.

    Bring on the picture dump!
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]
    [​IMG]

    Conclusions for now

    • The dual-Sonic Team theory gains new ground as the workflow is extremely similar to the Colors' one. Whoever's in charge of this team seems content with what they have and would rather not adapt to the other team's workflow despite adapting the Hedgehog Engine to Wii U. The middle-ware inconsistency between the two teams is even more apparent now.
    • Sadly there's barely any cut content or interesting areas from what I found, but I've not had a very thorough check yet. Hopefully when the files are public other people will give it a better look. I'm still awaiting the DLC files to inspect those as well.
    • The small differences between the formats means a back-port of the terrain to Generations is easily possible with some coding. However, I'm not interested in the idea nor think it would work well due to the massive gameplay differences between the two games. There's a lot to be discussed as well if it's actually beneficial for us to do that to a new game that is still being sold as the latest title. We don't know what reaction Sega could have to seeing a botched copy of their newest game in an older one, and the situation gets even more dodgy considering the low amount of Wii U owners and the implications of that. I will not prevent anyone from trying when the files are public, but I would rather not take the risk. That and I don't really like the level design anyways. :v:/>/>/>/>/>/>
    • As a side-note to the previous point, it could be interesting to discuss the reuse of assets instead rather than ports.
    • What I'm more interested about as soon as an exploit appears is to test how easy it would be to import brand new levels into Lost World. It should be much simpler than Generations levels in theory (except for the .orc file cracking), and the gameplay could easily lend itself to more regular 3D platformer stages than the kind of racing stages Generations usually does. I'm sure a lot of people would be interested to see the Parkour being used in more interesting levels as well rather than Straight Extruded Tube Zone Act 1.
    • I've not seen much in the way of physics configurations files yet, which is pretty disappointing considering how many tweaks this game would benefit from. Seems like a lot of it is hard-coded like Colors.
    • All of Sky Road's levels are named in a way that they imply are the final acts of each world instead (w1a01 is Windy Hill Act 1, w1a06 is Sky Road Act 1). The sharp spike in difficulty would suggest otherwise, so it's probably just an organization thing rather than an old level order.
      The actual level order is laid out in lua files. Notice how a lot of the "x" levels were stuck into the worlds, explaining the big inconsistency in themes for each world:
      Code (Text):
      1.  
      2. ---------------------------------------------------------------
      3. -- ゾーン、ミッション、ステージの対応表
      4.  
      5. zone_table = {
      6.     -- world1
      7.     {
      8.         index = 0,
      9.         missions = { "w1a01", "w1a03", "xind01","w1a06"  },
      10.     },
      11.     -- world2
      12.     {
      13.         index = 1,
      14.         missions = { "w2a01", "xbee01","xswe01","w2a06"  },
      15.     },
      16.     -- world3
      17.     {
      18.         index = 2,
      19.         missions = { "w3a01", "w3a03", "xsea01","w3a06", "w6a03" },
      20.     },
      21.     -- world4
      22.     {
      23.         index = 3,
      24.         missions = { "w4a01", "w4a04", "xcas01","w4a06"  },
      25.     },
      26.     -- world5
      27.     {
      28.         index = 4,
      29.         missions = { "w5a01", "w5a02", "xind02","xmoa01" },
      30.     },
      31.     -- world6
      32.     {
      33.         index = 5,
      34.         missions = { "w1a04", "w5a03", "w6a05", "w2c01", "xnim02" },
      35.     },
      36.     -- world7
      37.     {
      38.         index = 6,
      39.         missions = { "w7a01", "xsea02","w7a02", "w7a06"  },
      40.     },
      41.     -- world8
      42.     {
      43.         index = 7,
      44.         missions = { "xcub01","xsky01","xnim01","xbak01"  },
      45.     },
      46.     -- minigame (WorldMap, SaveData)
      47.     --world1
      48.     {
      49.         index = 8,
      50.         missions = { "w1m0a","w1m0b","w1m0c","w1m0d"  },
      51.     },
      52.     --world2
      53.     {
      54.         index = 9,
      55.         missions = { "w2m0a","w2m0b","w2m0c","w2m0d"  },
      56.     },
      57.     --world3
      58.     {
      59.         index = 10,
      60.         missions = { "w3m0a","w3m0b","w3m0c","w3m0d"  },
      61.     },
      62.     --world4
      63.     {
      64.         index = 11,
      65.         missions = { "w4m0a","w4m0b","w4m0c","w4m0d"  },
      66.     },
      67.     --world5
      68.     {
      69.         index = 12,
      70.         missions = { "w5m0a","w5m0b","w5m0c","w5m0d"  },
      71.     },
      72.     --world6
      73.     {
      74.         index = 13,
      75.         missions = { "w6m0a","w6m0b","w6m0c","w6m0d"  },
      76.     },
      77.     --world7
      78.     {
      79.         index = 14,
      80.         missions = { "w7m0a","w7m0b","w7m0c","w7m0d"  },
      81.     },
      82.  
      83.     -- battle
      84.     -- speed race
      85.     {
      86.         index = 15,
      87.         missions = { "batl01","batl02","batl03","batl04","batl05"  },
      88.     },
      89.     -- expart race
      90.     {
      91.         index = 16,
      92.         missions = { "batl06","batl07","batl08","batl09","batl10"  },
      93.     },
      94.     -- ring race
      95.     {
      96.         index = 17,
      97.         missions = { "batl11","batl12","batl13","batl14","batl15"  },
      98.     },
      99.  
    • I hope none of the horrible nerfing to the engine that took place in this game happens for Generations' Sonic Team's next game. :v:/>/>/>/>/>/>

    My focus remains on working on SonicGLvl and a Generations mod as well (that is not related to this), but hopefully we get to talk more about this once the files are public and the Wii U exploits are possible. So far it's looking very promising and possible to make level mods for anyone interested without much research. 2015 looking pretty interesting so far. :)/>/>/>/>/>/>
     
  2. TimmiT

    TimmiT

    ¯\_(ツ)_/¯ Member
    Just read through it all, very interesting stuff. Hopefully this all will be useful when it's possible to mess around more with the Wii U.
     
  3. Candescence

    Candescence

    Member
    1,979
    2
    18
    Sydney, Australia
    3D Indie Stuff
    I'd actually be interested in seeing Prince of Persia (Sands of Time quadrilogy or the reboot) or Darksiders II levels ported over for funsies, due to the parkour stuff. Granted, Lost World's parkour is somewhat more simplistic, but still.
     
  4. Overlord

    Overlord

    ...did we just switch realities? Moderator
    17,583
    68
    28
    Berkshire, England
    Learning Cymraeg
    Absolutely fascinating stuff. The fact it's already so compatible with existing extraction tools (which does make a degree of sense) should help lots down the road when this game's at a point of being modded for new levels.
     
  5. Joe T.E.

    Joe T.E.

    All life form data, successfully copied! Member
    142
    0
    0
    At my home in Florida
    Big the Cat in Sonic Generations, High-Res HUD for Sonic Lost World, Metal Sonic Lost World (Version 1.0,) Sonic Generations: Episode Metal (Version 3.0.)
    Hmm...

    Super Smash Bros. Brawl also had a .pac file format, which has been fully cracked for quite a few years (I'd know since I've dealt with that format numerous times while making mods for Brawl.)

    The reason I'm mentioning this is because Brawl (if I recall correctly) was the only Wii game to use the .pac format, AND was also (if I recall correctly) the only Wii game to use the Havok physics engine. There may be a connection between the .pac format and Havok (at least on Nintendo's consoles.)

    Because of this, I'm wondering if Lost World's .pac format might be the same one used in Brawl (or, at least, an updated variant of it.) I wouldn't be too surprised since I've heard that some of the other common Wii U formats (like bfstm music files) are just updated Wii formats.

    Also, everything else mentioned isn't surprising me too much, but is still good to hear. I'd definitely be interested in modding the game sometime in the future.
     
  6. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    It's just a coincidence in the extension, the formats are completely different, so there's not many conclusions to draw from it. Further evidence is the .pac files use the offset table/header that other Sega games have used in the past (The ones that say 1BBINA).
     
  7. Dark Sonic

    Dark Sonic

    I'll find the old avatar eventually Member
    12,404
    114
    43
    Working on my art!
    While I understand that we shouldn't back port levels for many reasons, any chance of a Lost World model mod? In particular I just like Modern Sonic's classic like standing pose. This is all very cool stuff though!

    Also this gives me some hope because it probably means that the Generations team is working on something, and I'm excited to find out what it is (as long as it's not that mobile game...). Despite it's flaws, I thought Unleashed was a fun game, and Generations was just great.
     
  8. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    Sure, I believe Zoney intends to release the mod from the two videos up above. The rigs are almost the exact same in both games so the animations work pretty well.

    That's what gives me hope for the next title as well.
     
  9. Joe T.E.

    Joe T.E.

    All life form data, successfully copied! Member
    142
    0
    0
    At my home in Florida
    Big the Cat in Sonic Generations, High-Res HUD for Sonic Lost World, Metal Sonic Lost World (Version 1.0,) Sonic Generations: Episode Metal (Version 3.0.)
    Oh well. It's still a really odd coincidence, but it's always good to get these things cleared up as soon as possible. Thanks for clearing that up, Dario.

    Also, you didn't mention anything about the game's audio formats in your opening post. Am I right to assume that the game is using the same audio formats as Unleashed/Colors/Generations? (assuming the audio was in the early dump you received, of course.)
     
  10. TheKazeblade

    TheKazeblade

    "Our Life is More than a Side-Effect" Member
    On top of that, provided they haven't been called upon to work on other projects in the interim, it means that they've been developing for almost four years at this point. Unless there are massive setbacks and problems, I'm excited for what that could mean.

    These discoveries are quite fascinating, more for the modding possibilities than the actual game itself. I'm excited to see the possibilities of new levels taking advantage of the mechanics of Lost World without the round tube gimmick applied to everything.
     
  11. Dario FF

    Dario FF

    Tech Support Hotline Tech Member
    It's .acb and .awb files, which AFAIK are possible to rip without much trouble. I will ask if I can upload them. (Stuff that isn't the soundtrack that is already out)
     
  12. Azookara

    Azookara

    take it from me, little buddy Member
    2,997
    30
    28
    Meowth's party
    hoboagogo?
    I wonder if Sonic's movement in the game really is as binary as it feels. If it's not, it'd be neat if (when we get to the point of being able to mess with physics variables and junk) we can make Sonic accelerate naturally to top speed without needing the run button. Or better yet, make it so that Sonic doesn't stop on the ground or air on a dime. That'd be nicer than porting levels, to be honest, but I'll gladly take both.

    Can't wait til the SLW model gets a public release, either. Gonna have a lot of fun with that, especially if the jump ball model and the figure eight thing for his feet comes with it. :J
     
  13. Paraxade

    Paraxade

    Member
    186
    0
    0
    Something silly Dario missed is that textures are labeled with the extension ".dds"; however, they're not actually DDS files. They're Nintendo's GTX texture format, which is commonly used in a lot of Wii U games. Apparently Sega just couldn't be bothered to change the file extension.
     
  14. Twilightzoney

    Twilightzoney

    Tech Member
    353
    0
    16
    Elgin, IL And Hampshire
    Unleashed and Generations Stuff and Custom Works
    But Dario did see in the game files that there were regular DDS files that get loaded in the game. So we think that we can be able to load normal DDS files without them having to be GTX file format at all. We'll find out soon hopefully.

    And there is no current way to repack the sound files for the awb/acb files yet. And not all the acb files are extractable for some reason in Sonic Lost World compared to that of Sonic Booms.

    Regarding the Sonic Lost World Model Mod, I'll be releasing it once I get the rest of the models ported over for it so he can run with his figure 8 stuff and get the sound effects ported over as well for it.
     
  15. Shakidna

    Shakidna

    Rehash Rampage Member
    Can you get clean rips of the music? Hopefully the Yoshi's Island Music doesn't have the sfx baked in.
     
  16. Possibly some stupid questions, but please bear with me...

    First off, because the formats are so similar, would you be able to potentially port the shaders from Generations (Either the vanilla shader or the Unleashed shader) to Lost World? Would it make it easier to do if you were to reduce the frame rate down to 30 or 45 instead of a full 60 FPS? If not the shaders, what about other things like the models and animations? Could we possibly get Classic Sonic in Lost World?

    Second, even though you had said that you had no interest in back-porting the Lost World levels to Generations, I'd like to ask something anyways. Lost World has the spinning gravity for long tube levels, so you don't just run off the side and fall to your death. Would that be something that could be ported to Generations for the benefit of people actually trying to back-port levels? What about other gimmicks? Even if we don't back-port the full levels, the gimmicks for those levels could be interesting to work with.
     
  17. Pexs

    Pexs

    Otherwise known as Spex Member
    That's very interesting.

    With it being somewhat similar to Colors, I half expected it to be using the updated Wii BRRES system, BFRES.

    The idea that it's kind of a Colors game ported to a simplified Hedgehog Engine is incredibly interesting. I can't wait until the true dump gets released (by the original dumper) so I can dig into this thing myself.
     
  18. Paraxade

    Paraxade

    Member
    186
    0
    0
    Shaders aren't feasible to port because shaders are a very hardware-dependent thing, which means the shader formats between the two games are completely different. Generations uses HLSL shaders for DirectX while Lost World uses Nintendo's shader formats (language is similar to GLSL) for GX2. The shaders are pre-compiled in both games, so the source can't be easily modified. As far as assets are concerned, like Dario said the formats are extremely similar to Generations, so it should be relatively easy to port them over between games.

    Also, the gimmicks aren't portable. That's the exact reason why backporting is a bad idea; gimmicks rely on a lot of hardcoded behavior that can't easily be transferred from game to game.
     
  19. Twilightzoney

    Twilightzoney

    Tech Member
    353
    0
    16
    Elgin, IL And Hampshire
    Unleashed and Generations Stuff and Custom Works
    We don't have the DLC data yet, but we'll upload the music to that when we get ahold of it.

    Well converting shaders to Wii U specific shaders don't see to be possible. They seem to have converted the PS3 Shaders of Generations and added more from them and converted them to Wii U shaders. They have the same extensions as the PS3 Version FPO and VPO and so on. Being that the PS3 and Wii U graphical wise are similar so I assume thats why they converted over their PS3 Shaders to the game.

    We could do Classic Sonic in Lost World, but I"m not really willing to animate that model but its a definite possibility to do in the future though when modding comes around for this game.

    I too would like to see the levels work in Generations only to do BullShots and such. But levels working is a no go. They handle the gravity different in this game, so you couldn't just use the gravity object in Generations to make it work.
    They have Gfield objects that refer to splines to define the gravity collision areas.
     
  20. Dark Sonic

    Dark Sonic

    I'll find the old avatar eventually Member
    12,404
    114
    43
    Working on my art!
    Feel free to not bother porting Lost World's double mouthed Super Sonic. I don't know if it was laziness or just them trying to hark back to the days of Sonic 2 but christ was it lazy.