- Tech Support Hotline
-
- Posts: 1030
- Joined: 03-April 10
- Gender:Male
- Location:Mar Del Plata
- Project:SonicGLvl
So there's a PC version of this game apparently.
Tools
CPKREDIR: To be able to load and modify files without touching the CPK files.
LibGens: Tools to work with formats from Unleashed / Generations / Lost World.
SonicGLvl-lostworld: A modified branch of SonicGLvl 0.9 by Paraxade to work with Lost World. Heavily unfinished & experimental (like regular GLVL is), but it's possible to work with it.
A lot of the research from Generations applies to Lost World, so you might want to look at the SCHG page for it.
Older thread below spoiler:
Tools
CPKREDIR: To be able to load and modify files without touching the CPK files.
LibGens: Tools to work with formats from Unleashed / Generations / Lost World.
SonicGLvl-lostworld: A modified branch of SonicGLvl 0.9 by Paraxade to work with Lost World. Heavily unfinished & experimental (like regular GLVL is), but it's possible to work with it.
A lot of the research from Generations applies to Lost World, so you might want to look at the SCHG page for it.
Older thread below spoiler:
Spoiler
Development and File Structure
The speculation behind Sonic Team still being organized into two teams for the mainline games and the Storybook/Colors team developing Lost World becomes even more apparent with a brief examination of the files. The folder/file structure and file formats use share a LOT of similarities with Colors. The project's name internally is "Sonic2013" (Colors was "Sonic2010"), and almost all the directories they exported their stuff from was paths in the style of "Sonic2010/Sonic2013", heavily hinting at which team developed this.
The files themselves are the most reliable proof as they share a lot of formats with Colors, but the biggest change is that instead of relying on Nintendo's Wii model formats, they now used a modified version of the Hedgehog Engine's formats.
Criware Pack (.cpk): Compatible with current tools, there's only one CPK used for the entirety of the game's assets (not movies). About 1.8 GB. (Generations totals to around 5.2 GB)
Layout Object Files (.orc): Just like in Colors, there's a dedicated "set" folder with .orc files that are used for the object level layouts. While I haven't looked much into the format, besides a small file header upgrade it seems both Colors .orc and Lost World .orc are very similar. Nowhere near as easy to deal with as Generations XML formats sadly, but not impossible to crack either.
Lots and lots of lua files: Again like in Colors, lua files are used for defining a lot of the stage's properties (in Generations XMLs are used instead), and are also very similarly coded. The new things are a lot of graphical parameters that have the same names as in the Hedgehog Engine games (stuff like SceneEffect.prm.xml's properties).
Animation Scripts (.anm): Not sure what their purpose is but it seems like they were used in Colors as well.
Pac Pack Files (.pac): Seems like a new package format in place of .arc or .ar. This format is more complicated than it looks at first glance compared to any package format I've ever seen. The main problem that comes from it is that some certain file formats that are packed inside (.effect files, etc.), end up sharing their strings and offset table with the entire pac file. This makes it a huge pain to make a reliable extractor and repacker as the files have to be basically rebuilt. Luckily they don't need to be cracked, as the offset table in the pac file is all the information needed to fix this. I now have a working PAC extractor/repacker (that I hope to be able to test in the future in the console) that automatically appends some metadata to the affected file formats inside and allows them to be properly repacked.
Updated Hedgehog Engine Formats (.terrain-model, .terrain-instanceinfo, .material, .model): These should be familiar to anyone who's modded Generations, and to my surprise they were mostly compatible with the existing tools already. The main difference seems to be that the header got slightly updated, and the root node offset got changed to 16 from 24 (0x10 from 0x18). I don't really understand the purpose behind the new header yet (probably some context information as they need it), but it shouldn't be a big problem. It seems the game also uses regular files without the header that are pretty much the same thing from Generations, and I'll theorize they can run on real hardware as well.
Images:
Lightfield (.lft): Compatible with Generations, no changes whatsoever. Determines how Sonic and objects are shaded depending on the section of the level he's in, basically simulating the light bounces from the terrain.
Havok 2012 Formats (anm.hkx, skl.hkx, phy.hkx): Also familiar to anyone who worked with Generations, the Hedgehog Engine seems to rely on Havok's implementation of animation, and ends up appearing on Wii U, a format that does not actually exist as a target platform in the tools. The version of the files is hk_2012.2.0-r1, and lack any metadata of any kind that describes the formats. They're also in big-endian format. HkxConverter (developed for the Unleashed 360 to PC conversion) needed an update to read the metadata out of other files (that I exported myself with the Havok 2012 Content Tools and used similar metadata to Lost World) to be able to properly convert the Wii U HKX to PC HKX. The result is usable, and with some luck, can actually be converted back to 2010 if the formats aren't very different. Here's a lot of TwilightZoney's work regarding the animations, along with the converted Sonic Lost World .model to Generations:
Comparison of Sonic Lost World Model (yellow) to Sonic Generations Model (blue):
Gameplay Videos of Sonic Lost World Model with Lost World Animations back-ported to Generations:
Hedgehog Engine Changes and Terrain
Ever since its reveal it was very apparent that Lost World was a Hedgehog Engine game that was heavily nerfed graphically, and its files tell the same story as well. The two reasons are pretty obvious, a reduced budget (exclusive deal) and a 60 FPS target frame-rate instead of the usual 30 on similar hardware. The nerf affects pretty much every aspect of the engine: the quality of the shaders, the overall size and detail of the geometry, the GI quality, the elaborate streaming engine, etc.
Shader Quality: Most of the shaders from Generations made the transition to Wii U as well as several new ones, so that means that almost all Generations materials are compatible with Lost World. However, the overall quality of the shading seems to have gotten a significant downgrade and it's very apparent with the SonicGLvl screenshots that will come later (that use Generations shaders). Certain shader parameters aren't used the same way as in Generations and several new ones were introduced. This causes huge consistency problems when trying to preview these models with the Generations shaders.
Geometry Size & Detail: The numbers tell the entire story here, as I was really surprised when I was told the size of the Sonic Lost World dump. It was to be expected but the difference in size when compared to Generations is still stunning.
All Lost World terrain size (uncompressed): 1.953.601.752 bytes (1.9 GB)
All Generations terrain size (ACTUALLY COMPRESSED): 4.604.474.273 bytes (4.6 GB)
The terrain in Lost World is not compressed and is simply dumped inside Pac files. Generations compresses all terrain inside PFDs and heavily benefits from it, with an almost 70% smaller compression ratio for each Stage.pfd. The terrain from 4 stages of Unleashed UNCOMPRESSED is as big as the entirety of the terrain in Lost World. We already know the reasons but it's still insane to think how much Sonic Team reduced the workload for the stages.
Global Illumination Quality: The GI Streaming and the GI Atlas texture system are gone, in comes individual GI textures for each instance file in the stage. There's no longer any level of variable quality to the GI, and only the lowest quality ones as possible were rendered and used in the game. The shading algorithm however luckily remains the same, and there's no need to come up with new rendering techniques to preview it or create our own.
Streaming Engine: The Hedgehog Engine in both Generations and Unleashed uses a lot of files to determine Terrain Groups and Terrain Blocks. These control when to load and unload chunks of terrain, when to cull it from view with a Visibility Tree, etc. That entire system is gone and instead the game will auto-load anything that is in the terrain folder (I assume since I haven't seen anything that describes what terrain to load). While this makes terrain repacking extremely easy in comparison to Generations, it also limits a lot what can be done with the game. The game simply streams terrain by just using the bounding spheres of each terrain node and loading it if it's close to the camera.
Terrain Previewing
As expected, since a lot of the formats are similar, it's pretty easy to add some compatibility into my current branch of SonicGLvl and preview it with the Generations shaders. However as mentioned before, the changes means a lot of materials are incompatible and not supposed to be previewed like this, leading to a lot of red materials (used when the shader does not exist in my database), broken uvs, broken lighting, etc.
Bring on the picture dump!
Conclusions for now
My focus remains on working on SonicGLvl and a Generations mod as well (that is not related to this), but hopefully we get to talk more about this once the files are public and the Wii U exploits are possible. So far it's looking very promising and possible to make level mods for anyone interested without much research. 2015 looking pretty interesting so far.
/>/>/>/>/>/>
Development and File Structure
The speculation behind Sonic Team still being organized into two teams for the mainline games and the Storybook/Colors team developing Lost World becomes even more apparent with a brief examination of the files. The folder/file structure and file formats use share a LOT of similarities with Colors. The project's name internally is "Sonic2013" (Colors was "Sonic2010"), and almost all the directories they exported their stuff from was paths in the style of "Sonic2010/Sonic2013", heavily hinting at which team developed this.
The files themselves are the most reliable proof as they share a lot of formats with Colors, but the biggest change is that instead of relying on Nintendo's Wii model formats, they now used a modified version of the Hedgehog Engine's formats.
Criware Pack (.cpk): Compatible with current tools, there's only one CPK used for the entirety of the game's assets (not movies). About 1.8 GB. (Generations totals to around 5.2 GB)
Layout Object Files (.orc): Just like in Colors, there's a dedicated "set" folder with .orc files that are used for the object level layouts. While I haven't looked much into the format, besides a small file header upgrade it seems both Colors .orc and Lost World .orc are very similar. Nowhere near as easy to deal with as Generations XML formats sadly, but not impossible to crack either.
Lots and lots of lua files: Again like in Colors, lua files are used for defining a lot of the stage's properties (in Generations XMLs are used instead), and are also very similarly coded. The new things are a lot of graphical parameters that have the same names as in the Hedgehog Engine games (stuff like SceneEffect.prm.xml's properties).
Animation Scripts (.anm): Not sure what their purpose is but it seems like they were used in Colors as well.
Pac Pack Files (.pac): Seems like a new package format in place of .arc or .ar. This format is more complicated than it looks at first glance compared to any package format I've ever seen. The main problem that comes from it is that some certain file formats that are packed inside (.effect files, etc.), end up sharing their strings and offset table with the entire pac file. This makes it a huge pain to make a reliable extractor and repacker as the files have to be basically rebuilt. Luckily they don't need to be cracked, as the offset table in the pac file is all the information needed to fix this. I now have a working PAC extractor/repacker (that I hope to be able to test in the future in the console) that automatically appends some metadata to the affected file formats inside and allows them to be properly repacked.
Updated Hedgehog Engine Formats (.terrain-model, .terrain-instanceinfo, .material, .model): These should be familiar to anyone who's modded Generations, and to my surprise they were mostly compatible with the existing tools already. The main difference seems to be that the header got slightly updated, and the root node offset got changed to 16 from 24 (0x10 from 0x18). I don't really understand the purpose behind the new header yet (probably some context information as they need it), but it shouldn't be a big problem. It seems the game also uses regular files without the header that are pretty much the same thing from Generations, and I'll theorize they can run on real hardware as well.
Images:
Spoiler
Lightfield (.lft): Compatible with Generations, no changes whatsoever. Determines how Sonic and objects are shaded depending on the section of the level he's in, basically simulating the light bounces from the terrain.
Havok 2012 Formats (anm.hkx, skl.hkx, phy.hkx): Also familiar to anyone who worked with Generations, the Hedgehog Engine seems to rely on Havok's implementation of animation, and ends up appearing on Wii U, a format that does not actually exist as a target platform in the tools. The version of the files is hk_2012.2.0-r1, and lack any metadata of any kind that describes the formats. They're also in big-endian format. HkxConverter (developed for the Unleashed 360 to PC conversion) needed an update to read the metadata out of other files (that I exported myself with the Havok 2012 Content Tools and used similar metadata to Lost World) to be able to properly convert the Wii U HKX to PC HKX. The result is usable, and with some luck, can actually be converted back to 2010 if the formats aren't very different. Here's a lot of TwilightZoney's work regarding the animations, along with the converted Sonic Lost World .model to Generations:
Spoiler
Comparison of Sonic Lost World Model (yellow) to Sonic Generations Model (blue):
Spoiler
Gameplay Videos of Sonic Lost World Model with Lost World Animations back-ported to Generations:
Hedgehog Engine Changes and Terrain
Ever since its reveal it was very apparent that Lost World was a Hedgehog Engine game that was heavily nerfed graphically, and its files tell the same story as well. The two reasons are pretty obvious, a reduced budget (exclusive deal) and a 60 FPS target frame-rate instead of the usual 30 on similar hardware. The nerf affects pretty much every aspect of the engine: the quality of the shaders, the overall size and detail of the geometry, the GI quality, the elaborate streaming engine, etc.
Shader Quality: Most of the shaders from Generations made the transition to Wii U as well as several new ones, so that means that almost all Generations materials are compatible with Lost World. However, the overall quality of the shading seems to have gotten a significant downgrade and it's very apparent with the SonicGLvl screenshots that will come later (that use Generations shaders). Certain shader parameters aren't used the same way as in Generations and several new ones were introduced. This causes huge consistency problems when trying to preview these models with the Generations shaders.
Geometry Size & Detail: The numbers tell the entire story here, as I was really surprised when I was told the size of the Sonic Lost World dump. It was to be expected but the difference in size when compared to Generations is still stunning.
All Lost World terrain size (uncompressed): 1.953.601.752 bytes (1.9 GB)
All Generations terrain size (ACTUALLY COMPRESSED): 4.604.474.273 bytes (4.6 GB)
The terrain in Lost World is not compressed and is simply dumped inside Pac files. Generations compresses all terrain inside PFDs and heavily benefits from it, with an almost 70% smaller compression ratio for each Stage.pfd. The terrain from 4 stages of Unleashed UNCOMPRESSED is as big as the entirety of the terrain in Lost World. We already know the reasons but it's still insane to think how much Sonic Team reduced the workload for the stages.
Global Illumination Quality: The GI Streaming and the GI Atlas texture system are gone, in comes individual GI textures for each instance file in the stage. There's no longer any level of variable quality to the GI, and only the lowest quality ones as possible were rendered and used in the game. The shading algorithm however luckily remains the same, and there's no need to come up with new rendering techniques to preview it or create our own.
Streaming Engine: The Hedgehog Engine in both Generations and Unleashed uses a lot of files to determine Terrain Groups and Terrain Blocks. These control when to load and unload chunks of terrain, when to cull it from view with a Visibility Tree, etc. That entire system is gone and instead the game will auto-load anything that is in the terrain folder (I assume since I haven't seen anything that describes what terrain to load). While this makes terrain repacking extremely easy in comparison to Generations, it also limits a lot what can be done with the game. The game simply streams terrain by just using the bounding spheres of each terrain node and loading it if it's close to the camera.
Terrain Previewing
As expected, since a lot of the formats are similar, it's pretty easy to add some compatibility into my current branch of SonicGLvl and preview it with the Generations shaders. However as mentioned before, the changes means a lot of materials are incompatible and not supposed to be previewed like this, leading to a lot of red materials (used when the shader does not exist in my database), broken uvs, broken lighting, etc.
Bring on the picture dump!
Spoiler
Spoiler
Spoiler
Conclusions for now
- The dual-Sonic Team theory gains new ground as the workflow is extremely similar to the Colors' one. Whoever's in charge of this team seems content with what they have and would rather not adapt to the other team's workflow despite adapting the Hedgehog Engine to Wii U. The middle-ware inconsistency between the two teams is even more apparent now.
- Sadly there's barely any cut content or interesting areas from what I found, but I've not had a very thorough check yet. Hopefully when the files are public other people will give it a better look. I'm still awaiting the DLC files to inspect those as well.
- The small differences between the formats means a back-port of the terrain to Generations is easily possible with some coding. However, I'm not interested in the idea nor think it would work well due to the massive gameplay differences between the two games. There's a lot to be discussed as well if it's actually beneficial for us to do that to a new game that is still being sold as the latest title. We don't know what reaction Sega could have to seeing a botched copy of their newest game in an older one, and the situation gets even more dodgy considering the low amount of Wii U owners and the implications of that. I will not prevent anyone from trying when the files are public, but I would rather not take the risk. That and I don't really like the level design anyways.
/>/>/>/>/>/>
- As a side-note to the previous point, it could be interesting to discuss the reuse of assets instead rather than ports.
- What I'm more interested about as soon as an exploit appears is to test how easy it would be to import brand new levels into Lost World. It should be much simpler than Generations levels in theory (except for the .orc file cracking), and the gameplay could easily lend itself to more regular 3D platformer stages than the kind of racing stages Generations usually does. I'm sure a lot of people would be interested to see the Parkour being used in more interesting levels as well rather than Straight Extruded Tube Zone Act 1.
- I've not seen much in the way of physics configurations files yet, which is pretty disappointing considering how many tweaks this game would benefit from. Seems like a lot of it is hard-coded like Colors.
- All of Sky Road's levels are named in a way that they imply are the final acts of each world instead (w1a01 is Windy Hill Act 1, w1a06 is Sky Road Act 1). The sharp spike in difficulty would suggest otherwise, so it's probably just an organization thing rather than an old level order.
The actual level order is laid out in lua files. Notice how a lot of the "x" levels were stuck into the worlds, explaining the big inconsistency in themes for each world:
Spoiler--------------------------------------------------------------- -- ゾーン、ミッション、ステージの対応表 zone_table = { -- world1 { index = 0, missions = { "w1a01", "w1a03", "xind01","w1a06" }, }, -- world2 { index = 1, missions = { "w2a01", "xbee01","xswe01","w2a06" }, }, -- world3 { index = 2, missions = { "w3a01", "w3a03", "xsea01","w3a06", "w6a03" }, }, -- world4 { index = 3, missions = { "w4a01", "w4a04", "xcas01","w4a06" }, }, -- world5 { index = 4, missions = { "w5a01", "w5a02", "xind02","xmoa01" }, }, -- world6 { index = 5, missions = { "w1a04", "w5a03", "w6a05", "w2c01", "xnim02" }, }, -- world7 { index = 6, missions = { "w7a01", "xsea02","w7a02", "w7a06" }, }, -- world8 { index = 7, missions = { "xcub01","xsky01","xnim01","xbak01" }, }, -- minigame (WorldMap, SaveData) --world1 { index = 8, missions = { "w1m0a","w1m0b","w1m0c","w1m0d" }, }, --world2 { index = 9, missions = { "w2m0a","w2m0b","w2m0c","w2m0d" }, }, --world3 { index = 10, missions = { "w3m0a","w3m0b","w3m0c","w3m0d" }, }, --world4 { index = 11, missions = { "w4m0a","w4m0b","w4m0c","w4m0d" }, }, --world5 { index = 12, missions = { "w5m0a","w5m0b","w5m0c","w5m0d" }, }, --world6 { index = 13, missions = { "w6m0a","w6m0b","w6m0c","w6m0d" }, }, --world7 { index = 14, missions = { "w7m0a","w7m0b","w7m0c","w7m0d" }, }, -- battle -- speed race { index = 15, missions = { "batl01","batl02","batl03","batl04","batl05" }, }, -- expart race { index = 16, missions = { "batl06","batl07","batl08","batl09","batl10" }, }, -- ring race { index = 17, missions = { "batl11","batl12","batl13","batl14","batl15" }, },
- I hope none of the horrible nerfing to the engine that took place in this game happens for Generations' Sonic Team's next game. />/>/>/>/>/>
/>/>/>/>/>/>My focus remains on working on SonicGLvl and a Generations mod as well (that is not related to this), but hopefully we get to talk more about this once the files are public and the Wii U exploits are possible. So far it's looking very promising and possible to make level mods for anyone interested without much research. 2015 looking pretty interesting so far.
/>/>/>/>/>/>
This post has been edited by Dario FF: 01 February 2016 - 01:12 PM
