[Dario FF]

Update 1/14/2015:
The files are already in the public, but you'll have to find them on your own. Adding a tools section. Maybe setting up an SCHG page will be more useful?

Tools
pacpack: Extracts the contents of the PAC files that hold mostly everything that Lost World has.

HKX Converter 2012 with Metadata: A modified version of the HKX Converter used for Unleashed 360 to Generations PC conversion. Since Lost World's HKX files don't have any metadata packed into them unlike Unleashed, the alternative is to make the converter read the metadata from another file that shares most of the classes it uses. Some HKX files may not convert if there's no matching metadata for them. Note that the resulting files are for Havok 2012, a newer version than the one in Generations.

modelfbx: A command-line tool that allows you to convert .model files, Havok skeletons and Havok animations into an FBX file. The Havok animation support is kinda dodgy at the moment due to not being able to figure out how to properly translate the Havok scaling yet, but the skeletons should work correctly. Supports both the updated .model format and .material format from Lost World, and expects Havok 2012 2.0 files (Lost World version).

Development and File Structure

The speculation behind Sonic Team still being organized into two teams for the mainline games and the Storybook/Colors team developing Lost World becomes even more apparent with a brief examination of the files. The folder/file structure and file formats use share a LOT of similarities with Colors. The project's name internally is "Sonic2013" (Colors was "Sonic2010"), and almost all the directories they exported their stuff from was paths in the style of "Sonic2010/Sonic2013", heavily hinting at which team developed this.

The files themselves are the most reliable proof as they share a lot of formats with Colors, but the biggest change is that instead of relying on Nintendo's Wii model formats, they now used a modified version of the Hedgehog Engine's formats.

Criware Pack (.cpk): Compatible with current tools, there's only one CPK used for the entirety of the game's assets (not movies). About 1.8 GB. (Generations totals to around 5.2 GB)

Layout Object Files (.orc): Just like in Colors, there's a dedicated "set" folder with .orc files that are used for the object level layouts. While I haven't looked much into the format, besides a small file header upgrade it seems both Colors .orc and Lost World .orc are very similar. Nowhere near as easy to deal with as Generations XML formats sadly, but not impossible to crack either.

Lots and lots of lua files: Again like in Colors, lua files are used for defining a lot of the stage's properties (in Generations XMLs are used instead), and are also very similarly coded. The new things are a lot of graphical parameters that have the same names as in the Hedgehog Engine games (stuff like SceneEffect.prm.xml's properties).

Animation Scripts (.anm): Not sure what their purpose is but it seems like they were used in Colors as well.

Pac Pack Files (.pac): Seems like a new package format in place of .arc or .ar. This format is more complicated than it looks at first glance compared to any package format I've ever seen. The main problem that comes from it is that some certain file formats that are packed inside (.effect files, etc.), end up sharing their strings and offset table with the entire pac file. This makes it a huge pain to make a reliable extractor and repacker as the files have to be basically rebuilt. Luckily they don't need to be cracked, as the offset table in the pac file is all the information needed to fix this. I now have a working PAC extractor/repacker (that I hope to be able to test in the future in the console) that automatically appends some metadata to the affected file formats inside and allows them to be properly repacked.

Updated Hedgehog Engine Formats (.terrain-model, .terrain-instanceinfo, .material, .model): These should be familiar to anyone who's modded Generations, and to my surprise they were mostly compatible with the existing tools already. The main difference seems to be that the header got slightly updated, and the root node offset got changed to 16 from 24 (0x10 from 0x18). I don't really understand the purpose behind the new header yet (probably some context information as they need it), but it shouldn't be a big problem. It seems the game also uses regular files without the header that are pretty much the same thing from Generations, and I'll theorize they can run on real hardware as well.

Images:

Spoiler

Lightfield (.lft): Compatible with Generations, no changes whatsoever. Determines how Sonic and objects are shaded depending on the section of the level he's in, basically simulating the light bounces from the terrain.

Havok 2012 Formats (anm.hkx, skl.hkx, phy.hkx): Also familiar to anyone who worked with Generations, the Hedgehog Engine seems to rely on Havok's implementation of animation, and ends up appearing on Wii U, a format that does not actually exist as a target platform in the tools. The version of the files is hk_2012.2.0-r1, and lack any metadata of any kind that describes the formats. They're also in big-endian format. HkxConverter (developed for the Unleashed 360 to PC conversion) needed an update to read the metadata out of other files (that I exported myself with the Havok 2012 Content Tools and used similar metadata to Lost World) to be able to properly convert the Wii U HKX to PC HKX. The result is usable, and with some luck, can actually be converted back to 2010 if the formats aren't very different. Here's a lot of TwilightZoney's work regarding the animations, along with the converted Sonic Lost World .model to Generations:

Spoiler

Comparison of Sonic Lost World Model (yellow) to Sonic Generations Model (blue):

Spoiler

Gameplay Videos of Sonic Lost World Model with Lost World Animations back-ported to Generations:




Hedgehog Engine Changes and Terrain

Ever since its reveal it was very apparent that Lost World was a Hedgehog Engine game that was heavily nerfed graphically, and its files tell the same story as well. The two reasons are pretty obvious, a reduced budget (exclusive deal) and a 60 FPS target frame-rate instead of the usual 30 on similar hardware. The nerf affects pretty much every aspect of the engine: the quality of the shaders, the overall size and detail of the geometry, the GI quality, the elaborate streaming engine, etc.

Shader Quality: Most of the shaders from Generations made the transition to Wii U as well as several new ones, so that means that almost all Generations materials are compatible with Lost World. However, the overall quality of the shading seems to have gotten a significant downgrade and it's very apparent with the SonicGLvl screenshots that will come later (that use Generations shaders). Certain shader parameters aren't used the same way as in Generations and several new ones were introduced. This causes huge consistency problems when trying to preview these models with the Generations shaders.

Geometry Size & Detail: The numbers tell the entire story here, as I was really surprised when I was told the size of the Sonic Lost World dump. It was to be expected but the difference in size when compared to Generations is still stunning.

All Lost World terrain size (uncompressed): 1.953.601.752 bytes (1.9 GB)
All Generations terrain size (ACTUALLY COMPRESSED): 4.604.474.273 bytes (4.6 GB)

The terrain in Lost World is not compressed and is simply dumped inside Pac files. Generations compresses all terrain inside PFDs and heavily benefits from it, with an almost 70% smaller compression ratio for each Stage.pfd. The terrain from 4 stages of Unleashed UNCOMPRESSED is as big as the entirety of the terrain in Lost World. We already know the reasons but it's still insane to think how much Sonic Team reduced the workload for the stages.

Global Illumination Quality: The GI Streaming and the GI Atlas texture system are gone, in comes individual GI textures for each instance file in the stage. There's no longer any level of variable quality to the GI, and only the lowest quality ones as possible were rendered and used in the game. The shading algorithm however luckily remains the same, and there's no need to come up with new rendering techniques to preview it or create our own.

Streaming Engine: The Hedgehog Engine in both Generations and Unleashed uses a lot of files to determine Terrain Groups and Terrain Blocks. These control when to load and unload chunks of terrain, when to cull it from view with a Visibility Tree, etc. That entire system is gone and instead the game will auto-load anything that is in the terrain folder (I assume since I haven't seen anything that describes what terrain to load). While this makes terrain repacking extremely easy in comparison to Generations, it also limits a lot what can be done with the game. The game simply streams terrain by just using the bounding spheres of each terrain node and loading it if it's close to the camera.


Terrain Previewing

As expected, since a lot of the formats are similar, it's pretty easy to add some compatibility into my current branch of SonicGLvl and preview it with the Generations shaders. However as mentioned before, the changes means a lot of materials are incompatible and not supposed to be previewed like this, leading to a lot of red materials (used when the shader does not exist in my database), broken uvs, broken lighting, etc.

Bring on the picture dump!

Spoiler

Spoiler

Spoiler

Conclusions for now

• The dual-Sonic Team theory gains new ground as the workflow is extremely similar to the Colors' one. Whoever's in charge of this team seems content with what they have and would rather not adapt to the other team's workflow despite adapting the Hedgehog Engine to Wii U. The middle-ware inconsistency between the two teams is even more apparent now.
  
  
• Sadly there's barely any cut content or interesting areas from what I found, but I've not had a very thorough check yet. Hopefully when the files are public other people will give it a better look. I'm still awaiting the DLC files to inspect those as well.
  
  
• The small differences between the formats means a back-port of the terrain to Generations is easily possible with some coding. However, I'm not interested in the idea nor think it would work well due to the massive gameplay differences between the two games. There's a lot to be discussed as well if it's actually beneficial for us to do that to a new game that is still being sold as the latest title. We don't know what reaction Sega could have to seeing a botched copy of their newest game in an older one, and the situation gets even more dodgy considering the low amount of Wii U owners and the implications of that. I will not prevent anyone from trying when the files are public, but I would rather not take the risk. That and I don't really like the level design anyways. />
  
  
• As a side-note to the previous point, it could be interesting to discuss the reuse of assets instead rather than ports.
  
  
• What I'm more interested about as soon as an exploit appears is to test how easy it would be to import brand new levels into Lost World. It should be much simpler than Generations levels in theory (except for the .orc file cracking), and the gameplay could easily lend itself to more regular 3D platformer stages than the kind of racing stages Generations usually does. I'm sure a lot of people would be interested to see the Parkour being used in more interesting levels as well rather than Straight Extruded Tube Zone Act 1.
  
  
• I've not seen much in the way of physics configurations files yet, which is pretty disappointing considering how many tweaks this game would benefit from. Seems like a lot of it is hard-coded like Colors.
  
  
• All of Sky Road's levels are named in a way that they imply are the final acts of each world instead (w1a01 is Windy Hill Act 1, w1a06 is Sky Road Act 1). The sharp spike in difficulty would suggest otherwise, so it's probably just an organization thing rather than an old level order.
  The actual level order is laid out in lua files. Notice how a lot of the "x" levels were stuck into the worlds, explaining the big inconsistency in themes for each world:
  
  Spoiler

  ---------------------------------------------------------------
  -- ゾーン、ミッション、ステージの対応表
  
  zone_table = {
      -- world1
      {
          index = 0,
          missions = { "w1a01", "w1a03", "xind01","w1a06"  },
      },
      -- world2
      {
          index = 1,
          missions = { "w2a01", "xbee01","xswe01","w2a06"  },
      },
      -- world3
      {
          index = 2,
          missions = { "w3a01", "w3a03", "xsea01","w3a06", "w6a03" },
      },
      -- world4
      {
          index = 3,
          missions = { "w4a01", "w4a04", "xcas01","w4a06"  },
      },
      -- world5
      {
          index = 4,
          missions = { "w5a01", "w5a02", "xind02","xmoa01" },
      },
      -- world6
      {
          index = 5,
          missions = { "w1a04", "w5a03", "w6a05", "w2c01", "xnim02" },
      },
      -- world7
      {
          index = 6,
          missions = { "w7a01", "xsea02","w7a02", "w7a06"  },
      },
      -- world8
      {
          index = 7,
          missions = { "xcub01","xsky01","xnim01","xbak01"  },
      },
      -- minigame (WorldMap, SaveData)
      --world1
      {
          index = 8,
          missions = { "w1m0a","w1m0b","w1m0c","w1m0d"  },
      },
      --world2
      {
          index = 9,
          missions = { "w2m0a","w2m0b","w2m0c","w2m0d"  },
      },
      --world3
      {
          index = 10,
          missions = { "w3m0a","w3m0b","w3m0c","w3m0d"  },
      },
      --world4
      {
          index = 11,
          missions = { "w4m0a","w4m0b","w4m0c","w4m0d"  },
      },
      --world5
      {
          index = 12,
          missions = { "w5m0a","w5m0b","w5m0c","w5m0d"  },
      },
      --world6
      {
          index = 13,
          missions = { "w6m0a","w6m0b","w6m0c","w6m0d"  },
      },
      --world7
      {
          index = 14,
          missions = { "w7m0a","w7m0b","w7m0c","w7m0d"  },
      },
  
      -- battle
      -- speed race
      {
          index = 15,
          missions = { "batl01","batl02","batl03","batl04","batl05"  },
      },
      -- expart race
      {
          index = 16,
          missions = { "batl06","batl07","batl08","batl09","batl10"  },
      },
      -- ring race
      {
          index = 17,
          missions = { "batl11","batl12","batl13","batl14","batl15"  },
      },
  
  

  
  
  
• I hope none of the horrible nerfing to the engine that took place in this game happens for Generations' Sonic Team's next game. />

My focus remains on working on SonicGLvl and a Generations mod as well (that is not related to this), but hopefully we get to talk more about this once the files are public and the Wii U exploits are possible. So far it's looking very promising and possible to make level mods for anyone interested without much research. 2015 looking pretty interesting so far. />

This post has been edited by Dario FF: 15 January 2015 - 01:20 PM

[TimmiT]

Just read through it all, very interesting stuff. Hopefully this all will be useful when it's possible to mess around more with the Wii U.

[Candescence]

I'd actually be interested in seeing Prince of Persia (Sands of Time quadrilogy or the reboot) or Darksiders II levels ported over for funsies, due to the parkour stuff. Granted, Lost World's parkour is somewhat more simplistic, but still.

[Overlord]

Absolutely fascinating stuff. The fact it's already so compatible with existing extraction tools (which does make a degree of sense) should help lots down the road when this game's at a point of being modded for new levels.

[Joe T.E.]

Dario FF, on 01 January 2015 - 01:10 PM, said:

Pac Pack Files (.pac): Seems like a new package format in place of .arc or .ar. This format is more complicated than it looks at first glance compared to any package format I've ever seen. The main problem that comes from it is that some certain file formats that are packed inside (.effect files, etc.), end up sharing their strings and offset table with the entire pac file. This makes it a huge pain to make a reliable extractor and repacker as the files have to be basically rebuilt. Luckily they don't need to be cracked, as the offset table in the pac file is all the information needed to fix this. I now have a working PAC extractor/repacker (that I hope to be able to test in the future in the console) that automatically appends some metadata to the affected file formats inside and allows them to be properly repacked.

Hmm...

Super Smash Bros. Brawl also had a .pac file format, which has been fully cracked for quite a few years (I'd know since I've dealt with that format numerous times while making mods for Brawl.)

The reason I'm mentioning this is because Brawl (if I recall correctly) was the only Wii game to use the .pac format, AND was also (if I recall correctly) the only Wii game to use the Havok physics engine. There may be a connection between the .pac format and Havok (at least on Nintendo's consoles.)

Because of this, I'm wondering if Lost World's .pac format might be the same one used in Brawl (or, at least, an updated variant of it.) I wouldn't be too surprised since I've heard that some of the other common Wii U formats (like bfstm music files) are just updated Wii formats.

Also, everything else mentioned isn't surprising me too much, but is still good to hear. I'd definitely be interested in modding the game sometime in the future.

[Dario FF]

Joe T.E., on 01 January 2015 - 02:05 PM, said:

Super Smash Bros. Brawl also had a .pac file format, which has been fully cracked for quite a few years (I'd know since I've dealt with that format numerous times while making mods for Brawl.)

It's just a coincidence in the extension, the formats are completely different, so there's not many conclusions to draw from it. Further evidence is the .pac files use the offset table/header that other Sega games have used in the past (The ones that say 1BBINA).

[Dark Sonic]

While I understand that we shouldn't back port levels for many reasons, any chance of a Lost World model mod? In particular I just like Modern Sonic's classic like standing pose. This is all very cool stuff though!

Also this gives me some hope because it probably means that the Generations team is working on something, and I'm excited to find out what it is (as long as it's not that mobile game...). Despite it's flaws, I thought Unleashed was a fun game, and Generations was just great.

This post has been edited by Dark Sonic: 01 January 2015 - 02:21 PM

[Dario FF]

Dark Sonic, on 01 January 2015 - 02:19 PM, said:

While I understand that we shouldn't back port levels for many reasons, any chance of a Lost World model mod? In particular I just like Modern Sonic's classic like standing pose. This is all very cool stuff though!

Sure, I believe Zoney intends to release the mod from the two videos up above. The rigs are almost the exact same in both games so the animations work pretty well.

Dark Sonic, on 01 January 2015 - 02:19 PM, said:

Also this gives me some hope because it probably means that the Generations team is working on something, and I'm excited to find out what it is (as long as it's not that mobile game...). Despite it's flaws, I thought Unleashed was a fun game, and Generations was just great.

That's what gives me hope for the next title as well.

[Joe T.E.]

Dario FF, on 01 January 2015 - 02:11 PM, said:

Joe T.E., on 01 January 2015 - 02:05 PM, said:

Super Smash Bros. Brawl also had a .pac file format, which has been fully cracked for quite a few years (I'd know since I've dealt with that format numerous times while making mods for Brawl.)

It's just a coincidence in the extension, the formats are completely different, so there's not many conclusions to draw from it. Further evidence is the .pac files use the offset table/header that other Sega games have used in the past (The ones that say 1BBINA).

Oh well. It's still a really odd coincidence, but it's always good to get these things cleared up as soon as possible. Thanks for clearing that up, Dario.

Also, you didn't mention anything about the game's audio formats in your opening post. Am I right to assume that the game is using the same audio formats as Unleashed/Colors/Generations? (assuming the audio was in the early dump you received, of course.)

[TheKazeblade]

Dark Sonic, on 01 January 2015 - 02:19 PM, said:

Also this gives me some hope because it probably means that the Generations team is working on something, and I'm excited to find out what it is (as long as it's not that mobile game...). Despite it's flaws, I thought Unleashed was a fun game, and Generations was just great.

On top of that, provided they haven't been called upon to work on other projects in the interim, it means that they've been developing for almost four years at this point. Unless there are massive setbacks and problems, I'm excited for what that could mean.

These discoveries are quite fascinating, more for the modding possibilities than the actual game itself. I'm excited to see the possibilities of new levels taking advantage of the mechanics of Lost World without the round tube gimmick applied to everything.

[Dario FF]

Joe T.E., on 01 January 2015 - 02:35 PM, said:

Also, you didn't mention anything about the game's audio formats in your opening post. Am I right to assume that the game is using the same audio formats as Unleashed/Colors/Generations? (assuming the audio was in the early dump you received, of course.)

It's .acb and .awb files, which AFAIK are possible to rip without much trouble. I will ask if I can upload them. (Stuff that isn't the soundtrack that is already out)

This post has been edited by Dario FF: 01 January 2015 - 02:46 PM

[Azukara]

I wonder if Sonic's movement in the game really is as binary as it feels. If it's not, it'd be neat if (when we get to the point of being able to mess with physics variables and junk) we can make Sonic accelerate naturally to top speed without needing the run button. Or better yet, make it so that Sonic doesn't stop on the ground or air on a dime. That'd be nicer than porting levels, to be honest, but I'll gladly take both.

Can't wait til the SLW model gets a public release, either. Gonna have a lot of fun with that, especially if the jump ball model and the figure eight thing for his feet comes with it. :J

[Paraxade]

Something silly Dario missed is that textures are labeled with the extension ".dds"; however, they're not actually DDS files. They're Nintendo's GTX texture format, which is commonly used in a lot of Wii U games. Apparently Sega just couldn't be bothered to change the file extension.

[Twilightzoney]

But Dario did see in the game files that there were regular DDS files that get loaded in the game. So we think that we can be able to load normal DDS files without them having to be GTX file format at all. We'll find out soon hopefully.

And there is no current way to repack the sound files for the awb/acb files yet. And not all the acb files are extractable for some reason in Sonic Lost World compared to that of Sonic Booms.

Regarding the Sonic Lost World Model Mod, I'll be releasing it once I get the rest of the models ported over for it so he can run with his figure 8 stuff and get the sound effects ported over as well for it.

This post has been edited by Twilightzoney: 01 January 2015 - 05:18 PM

[Shakidna]

Can you get clean rips of the music? Hopefully the Yoshi's Island Music doesn't have the sfx baked in.