SonicGLvl v0.9 - Now on GitHub

SonicGLvl 0.9 - Now on GitHub

NEWS 8/27/2016

This version is not meant for public use and is merely uploaded as the work that I've done so far. It's very much in need of work to be considered a stable release.

GitHub repository licensed under GPLv3
Building on Windows

Development Roadmap

SonicGLvl Wiki

Builds:
Stable: None yet.
The bin folder in the repository at the moment has the last compiled version of the code, but ideally no binaries should be hosted there.

Code Status:

• In need of some cleaner refactoring, especially how file opening is handled at the moment. Loading the files should be an external method that returns a value, not a constructor per type of object.
• Almost all of the ugly WinAPI code can and should be ignored since a port to Qt5 is inevitable.
• Switch all headers to #pragma once and try to forward declare mostly everything.
• Needs to be documented a lot more, although the names are pretty obvious as to what the methods do most of the time.

Last Preview Video:
https://www.youtube.com/watch?v=2fAkw7kcQPw

FAQ

Q: Can I use this over the original SonicGLvl?
A: No, and you shouldn't. The current new branch does not have feature parity with the original SonicGLvl yet. At most it's fairly usable for some simple object layout editing.

Q: Can I use the database/models from the original SonicGLvl?
A: No, the databases are no longer compatible for the purpose of extending the functionality a lot more. However, on the plus side, dumping and detecting new object templates is pretty much unnecessary unless you want bigger control over how the objects are displayed. The new branch of SonicGLvl no longer relies on having objects on the database to be able to load/display them at all.

Q: How similar is it to use this branch of SonicGLvl compared to the original one?
A: For the purposes of optimizing the work pipeline and making it easier to understand, a lot of old tutorials will be pretty much useless, so a new knowledge database will need to be built on the wiki.

Q: Summary of new features planned/implemented compared to original SonicGLvl?

• Smoother and much easier to use Viewport Controls.
• Undo/Redo.
• Smarter Object Database with extended previewing features.
• Smarter Stage/Terrain cache, verifies hashes and cleans up automatically.
• Game shader support, able to preview materials in the editor very similarly to the game.
• FBX Importing/Exporting instead of using Ogre formats, which gives us plenty of possibilities for a smoother pipeline between 3DSMax and SonicGLvl.
• Generalized Formats Library: You can do other applications that use LibGens for reading files from the game without having to integrate them into SonicGLvl. Also allows the possibility to do batching command-line tools.
• Terrain Lightfield Generation: Generate a lighting map based on rendered maps of the stage so objects and the player are affected by them.
• Better Terrain Group/GI generation workflow. More details on this can be seen on the roadmap.
• Ghost Playback/Camera Previewing for easier iteration on editing cameras.
• New material editor with real-time previewing, using the real game shaders, simplifying the iteration on polishing graphics a lot.

Original thread:

Spoiler

Current Version 0.4: Check the last post for update information

For the sake of not pestering the Generations resource thread for the ones who actually CARE about resources there, I'm splitting off the discussion of my editor in here, and also sharing all the information I researched so far in case anyone wants to try it on their own. Also, the editor is pretty much feature complete at the moment when it comes to level data format and templates so far. What's needed at the moment is raw models, templates tweaking, lots and lots of game content that needs ripping and being distributed with the editor.

SonicGLvl - Visual 3D editor for level objects layouts


Features:

• Loading XML or AR.00 files directly as levels.
• 3D camera controls and edition.
• Customizable model previews of in-game objects, can be fully extended later using a template database.
• MultiSetParam array support for creating copies of a single object easily.
• Ogre3D engine ensures a pretty good performance all around.
• Saving back directly to the AR.00 or XML files, ready to import straight to the game.

Hosting and Downloads

I will host SonicGLvl on Google Code, although I haven't opened the source code yet. SVN will allow me to keep a common database of the objects so far ripped out. Also, this would help people to not have to download a whole new version of the editor, just for adding a simple template. Using SVN update, you can just download the latest changes.

SonicGLvl on Google Code.

Checking out the latest SVN(for use for anyone who was the tools):

Code (Text):

1. svn checkout http://sonic-glvl.googlecode.com/svn/trunk/ sonic-glvl-read-only

Also, for the users who obviously ain't gonna download SVN tools just to check out the program, I will upload stable "snapshots" of the repository periodically. In other words, you will be able to download the latest uploaded version on the Downloads page.

Important notes:

• Do not use fullscreen mode or you'll be locked out of any "File Open" dialogue Windows pops up.
• You can limit the FPS turning VSync on
• If it looks like crap, you can put some FSAA to improve the looks
• Look over at the help menu for a "Quick Overview" so you're not completely lost with the controls.
• There's some focus issues with the windows. So make sure of what you click.

Who can contribute to SonicGLvl
The program needs lots of content ripping at the moment. Obviously I will work on it when more time frees up sooner or later, but all help is appreciated. Here's what can be contributted by simply posting here, or even comitting to the repository. Note that I'll need your google account name for adding project members. You can create a google account with any e-mail you currently have.

What you can contribute:
- Models and textures(models preferrably compatible with 3DS Max 2012)
- Converted models: Processed models with materials ready to be imported into the editor. This includes the .mesh files exported and .dds textures, plus the .material script.
- Templates (.xml): Refer to the technical info later on.
- Ideas & feature suggestions are welcome.

How to contribute?
- Post here.
- Request here to be added to the repository's project members so you can contribute using SVN tools.
- Or just PM me anything.

Tools needed for ripping
Currently SonicGLvl is using Ogre3D engine to provide display previews. Ogre3D provides enough features and a pretty fast performance for the needs of the editor, and there's no plans of changing the engine. Plus, it's cross platform in the event that SonicGLvl goes that path in the future.

- OgreMax Exporter is the plugin I'm currently using to export all models to the .mesh format Ogre uses. Do note that OgreMax supports 3DS Max, Maya, Softimage/XSI, and covers a big array of the versions of them. There's a commercial version, but the non-commercial version works fine for our purposes even with the restrictions. Non-commercial version of the plugin for 3DS Max.

- Darkspines' script with a slight modification to change the naming of the materials created is the one I'm currently using to import game's models. The name modification is because OgreMax exports the materials named as the maps in 3DS Max.

- Any 3D editor that's compatible with the above should work for exporting models into the editor. At the moment you can just use 3DS Max 2012 for free with a student 3 year license.

Screens & Videos



Some simple modifications of levels I made in the past with the editor:

Future plans:
- Add needed features like Undo/Redo.
- Support editting additional vectors from the object with another 3D point in space.
- Support loading level geometry when we got the chance to export it. Editting in blank space ain't fun.
- Add support for editting easily stuff like Lines, spline paths, etc.

Tech Info dump
So for the ones currently researching into the game's objects as well, here's all the info I got so far.

AR.00 Files
Sonic generations seems to use two different kinds of .ar.00 files for loading an stage, appart from the geometry in the /Packed folder.

- The file with a '#' (e.g. #ghz100.ar.00) is likely to be the "info" file of the level. The file size is small most of the time, and it holds all the data regarding object placement, as well as some physic files. While "Stage.xml" seems to refer to which files are the ones holding level data at the moment, it's safe to assume that all xml files with the prefix setdata_ are object level data. Mostly because "Stage.xml" seems to have a broken format sometimes, so it isn't really reliable.

- The file without the # holds the geometry, textures, and other stuff of the custom objects from the level. You can look into these files if you want to rip any custom models.

- These setdata_ xmls are completely human readable. While they can be editted by hand to add or move any objects you want, it's better to rely on the editor for the task. Mainly because using a 3D preview is way more intuitive.

Objects
- All objects of the level will be placed under the <SetObject> node in all these xmls. Each entry added to the file will appear as a whole new object in-game. Do note that most of these objects have a unique ID, set by a <SetObjectID> node inside them. Clashing IDs ingame are not something you'd like to see, so no objects should EVER have the same ID.

- The level files seem to have a special option built into the game to generate copies of itself across different positions. These are called <MultiSetParam> nodes, and each of these hold various <Element> nodes with their own <Position>, <Rotation>, and <Index>. There's also a <Count> node for the total ammount of copies. This <MultiSetParam> makes it easy to generate say, a line of 40 rings in a perfect pattern. While there's other parameters such as <LineBase>, or <Interval>, this only seem to be relevant to whatever editor SonicTeam used to make the game, because they have no relevance whatsoever apparently if modified ingame. What matters is the individual positions and rotations in each <Element> node, and this is handled by the editor already.

- It's considered pretty much globally for ALL objects to have their own <Position> and <Rotation> nodes. Coordinates between the editor and in-game match perfectly. You can consider the X-Z plane as the "floor" plane of the game, while the Y axis represents the Height.

- The nodes inside each object never specify what kind of variable they are. That's where the concept of templates comes in. SonicGLvl HAS to have a way to know what variables an object normally has, and what kind of variable they are.

Templates are simple XML files located in the editor's folder. They are normally built in the following way:

Code (Text):

1. <?xml version="1.0" encoding="utf-8" standalone="no" ?>
2. <MyObject>
3.     <MyVariable1>bool:true</MyVariable1>
4.  
5.     <Position>
6.       <x>double</x>
7.       <y>double</y>
8.       <z>double</z>
9.     </Position>
10.     <Rotation>
11.       <w>double</w>
12.       <x>double</x>
13.       <y>double</y>
14.       <z>double</z>
15.     </Rotation>
16.     <SetObjectID>integer</SetObjectID>
17.     <Mesh>object.mesh</Mesh>
18. </MyObject>

Each node represents the name of the object's attributes. Instead of writing a normal value inside though, you are supposed to write the type of variable it is. The editor currently supports the following:
string: a name
double: decimal number
integer: self-explanatory I guess
bool: boolean value, will write either "true" or "false"

Any value after ':' will be considered the "default" value with which the object is created from the editor's menu. For example, <CastShadow>bool:true</CastShadow> means that true is the default value for that attribute.

Note that it supports attributes inside other nodes, like <Position> and <Rotation>.
Also of notice, <Mesh> isn't an actual attribute, but a reference for the editor to know what model to use in the preview display.

Here's an example for a game ring:

Code (Text):

1. <?xml version="1.0" encoding="utf-8" standalone="no" ?>
2. <Ring>
3.     <GroundOffset>integer</GroundOffset>
4.     <InitDisp>bool</InitDisp>
5.     <IsCastShadow>bool:true</IsCastShadow>
6.     <IsLightSpeedDashTarget>bool</IsLightSpeedDashTarget>
7.     <IsReset>bool</IsReset>
8.     <Position>
9.       <x>double</x>
10.       <y>double</y>
11.       <z>double</z>
12.     </Position>
13.     <Range>double</Range>
14.     <ResetTime>double</ResetTime>
15.     <Rotation>
16.       <w>double</w>
17.       <x>double</x>
18.       <y>double</y>
19.       <z>double</z>
20.     </Rotation>
21.     <SetObjectID>integer</SetObjectID>
22.     <TreasureSearchHideType>integer</TreasureSearchHideType>
23.     <Mesh>ring.mesh</Mesh>
24. </Ring>

- Another option provided is the <AltMesh> reference. Some objects will change depending on the attributes they have, so the best way to represent that inside the editor is just changing the model displayed.

For example, if you would want a red spring to be displayed instead of a yellow one when the power of the spring is above 30.

Code (Text):

1.    
2. <AltMesh>
3.        <Condition>FirstSpeed >= 30</Condition>
4.        <Mesh>spring_classic_r_01.mesh</Mesh>
5.        <Mesh>spring_classic_r_02.mesh</Mesh>
6. </AltMesh>

This is purely for the editor's reference, it doesn't change anything in the game.

<Conditions> are single line statements. You can add as many conditions as you want. There's also a custom string value, LevelPrefix to check what level you're currently working on. For example, ghz for Green Hill zone, cpz for Chemical Plant. Same way as the game's files are named.

- Yet another option provided in the case it's needed, <MeshScaleX> (or Y, or Z) will allow you to change the preview model's scale using an attribute of the object(for example, <Width>). It also has a custom multiplier attribute. Here's an example for the object <OnewayFloor>, which depends on <Width>, <Height> and <Depth> to be represented accurately inside the editor.

Code (Text):

1.  
2.     <MeshScaleX multiplier="2.0">Width</MeshScaleX>
3.     <MeshScaleY multiplier="2.0">Depth</MeshScaleY>
4.     <MeshScaleZ multiplier="2.0">Height</MeshScaleZ>

- The editor will search for any matching objects with the templates it has, and will write any changes made. It will also add any new objects, as well as erase any deleted ones automatically. It's totally safe and compatible so far to open and save a level and import it into the game directly so far, given that you haven't tampered too much with object's attributes that you don't know about and make the game crash.

- The editor supports reading an XML and dumping all types of detected objects manually, using the default "unknown.mesh" as display model. It's recommended to tweak these files dumped manually.

TL;DR; Editting the levels is way easier than you ever thought.

 

Update
+ Fixed the XML dumper for reading as much attributes as possible from the XML with objects of the same type.
+ Nearly 30+ objects added from Act 2 Common like springs, dash/rainbow rings, jump/trick boards, etc. and some common elements from GHZ.
+ Added some more models for stuff like sounds, collisions, etc.

You can get the latest snapshot from the Downloads page, or use SVN update.

My intention right now is to get GHZ Act 2 with a pretty accurate visual representation of some objects at least.


Also, Falk reminded me of those Unleashed DLC Hard acts on the other thread, so for kicks, I did a lame and short edit of the first section of GHZ Act 2 to make it harder to speedrun through.


I like my troll dash rings after the spike mini-tunnel. This helped me find a nasty bug in a template though, which I fixed.

I can't wait til Stage geometry rips are released. I'll be so happy to do edits later with that instead of a grey background.

 

Yeah, I'm not really sure what's the deal with that. Maybe I'm not passing yet EXACTLY the rotations, but I thought they were fine. Or maybe it's just that the model exporter has some problems? I will have to look into it.

Could be added I guess.

Yeah I considered that. That should be separate files for the types of filters though. It would also help sort the main object menu. Adding something like "Category:Camera" to templatelist.txt should do the trick.

I didn't add that because there was no template option for it yet. I will add it though, something like

Code (Text):

1. <CopyScaleX>
2.   <Count>NumWood</Count>
3.   <Interval>2</Interval>
4. </CopyScaleX>

The X axis would be local and relative to the rotation.

That can be achieved already on the templates. Just gotta add:

Code (Text):

1.     <MeshScaleX multiplier="2.0">Width</MeshScaleX>
2.     <MeshScaleY multiplier="2.0">Depth</MeshScaleY>
3.     <MeshScaleZ multiplier="2.0">Height</MeshScaleZ>

I would love that idea, but I just have no idea what the camera attributes do yet. If someone could play around with them and document them, it'd be easier. Right now I'm quite busy, but I'm already adding all of these into the TODO list since they shouldn't be hard to pull off.

Thanks for the feedback.

Also, it seems I need to add support for another kind of attribute. Objects like <EventCollision> have attribute lists like these:

Code (Text):

1.     <TargetList0>
2.         <SetObjectID>23435</SetObjectID>
3.         <SetObjectID>23356</SetObjectID>
4.     </TargetList0>

My templates don't support lists with attributes of the same name yet, so some stuff might be broken when saving. I noticed bugs in the swing at GHZ Act 1, which is managed by an event that calls two of these. Support for lists shouldn't be hard to add anyway, just thought I should point out that don't panic if some stuff in the levels seem broken yet.

 

Update
+ List support for templates. This adds support and will fix A LOT of glitches that might occurr on levels, such as non-moving swings, or any other stuff. Lists are mostly used by objects such as <EventCollision>, and they normally have 2 or more object IDs to call for triggers. For example:

Code (Text):

1. <TargetList0>list:SetObjectID</TargetList0>
2.  

will produce in the object

Code (Text):

1. <TargetList0>
2. <SetObjectID>3213213</SetObjectID>
3. <SetObjectID>4326213</SetObjectID>
4. <SetObjectID>543543</SetObjectID>
5. ...
6. </TargetList0>

As many that you want. These lists are string values by default. I think that pretty much closes adding more parameters for templates, but we'll see what happens when editting other stages.

Inside the editor GUI, adding or deleting to a list is a single string, where each ';' symbol will indicate a new element for the list. So for example, for <EffectNameList>:
Editor: particle1;lightparticle;darkparticle;

Outputs to the XML:

Code (Text):

1. <EffectNameList>
2.    <Name>particle1</Name>
3.    <Name>lightparticle</Name>
4.    <Name>darkparticle</Name>
5. </EffectNameList>
6.  

* Scaled the axis node 3 times as bigger. A proper solution for that is under research. :v:

GUI improvements are planned next for anyone who cares. Although nothing really serious can be done until we got a way to import the level geometry yet.

A new snapshot of this version is available at the Downloads page as always. Happy editting!



Also, as a bonus, here's my experimentation with DustArma's idea, AND me adding the CTE trick boards to spice things up in the first part of GHZ act 2.

Link

I had to add the CTE trick board resources to the ghz_cmn file for this to work. And also the custom entry to StageObject.xml in ghz200.ar.00. Other relevant modifications have been stated on the Resource thread already.

 

Update v0.4

+ Dumped QuickGUI, using native GUI now. There's still some issues with window focus, but it's still far, far better than my older implementation of QuickGUI.



+ MultiSetParam now supports using a vector as base for its array copy instead of the world 3 axes. There's still buttons for setting the 6 presets though, as well as using an angle with a degree across the XZ plane(floor).

+ Categories of objects are now available, allowing the user to find much easier the object he's looking for, as well as applying viewing filters to the current categories. This allows for hiding stuff like cameras, particles, so the main view doesn't get cluttered with stuff you don't care about. Categories aren't hardcoded, and they can be added from templatelist.txt.



+ The moving and rotation axis will now scale according to the distance to the camera. This will make the axis' size stay constant on the screen regardless of how far or how close the object is.

+ Templates have a new feature implemented, and it's called <CopyScaleX>(or Y, or Z), which has the following format:

Code (Text):

1.     <CopyScaleX multiplier="value">Attribute's Name to use</CopyScaleX>

This feature allows the editor to generate a number of copies of an object along its local X, Y, or Z axis. The purpose of this feature is for objects like the Green Hill Wood bridge, which depending on <NumWood>, is the relative ammount of wood logs created.

For example, for GhWoodBridge.xml:

Code (Text):

1.     <CopyScaleX multiplier="0.6">NumWood</CopyScaleX>

It will copy itself every 0.6 points for each value of NumWood. NumWood should be an integer of course. For example:



+ Added lots of resources and templates from Chemical Plant Zone and its missions. Also some new common objects. Special thanks to lagstarr for contributing all these models to the repository, and more are about to follow soon as soon as I implement them!(City Escape, Speed Highway, etc.)

Here's a screenshot of Chemical Plant Zone Tails Mission. You can see the custom mission objects here.



* Resizing the editor's window will no longer fuck everything up. The toolbox windows will also be moved to the appropiate spot when resizing.


As always, you can get the latest snapshot from the Downloads page, or do an SVN checkout. Happy editting!

P.S: Any old custom templates you created are backwards compatible.

 

I'd rather not at its current state. It's usable, but it's missing crucial stuff like having SOME way to see the level geometry/collision(which is being researched by 2 other people ATM). I'll work on researching the path format now.

Also, we're barely 2 months from the game's first release. :v:

 

Aren't the levels just a bunch of instanced geometry across different positions of the level? Ogre has support for world geometry if needed, as well as the SceneManager supporting scenes like it. I wouldn't doubt the power of the engine before we even try. It would be quite useless to have that much geometry inside the editor, but I think it'd be interesting to try.

Damizean is working on the collisions meshes though. Hopefully he'll be lucky, since Havok tools are open on PC and there's plugins for Max.

 

SonicGLvl is now open source under the terms of the GPLv3. I did an extensive verification of the code base that all external code has its proper licenses/copyright notices marked as such, but I might've missed something. If that's the case please let me know. Only all original code written for the project is subject under the license.

The code design has changed a lot since I've also learned more about application design over the 2-3 years since I started with this, so I'm sorry for any weird inconsistencies. There's no restrictions to how much we should modify it if it helps in the long run.

You can check the first post for more details and the current development Roadmap.

 

I just pushed a couple of new tools to the repository. All these tools are currently using the Qt Framework for its UI and the Open Asset Import Library "Assimp" for supporting a wide range of popular 3D formats with one consistent programming interface. This allows us to slowly move the workflow for creating custom Generations stages outside of Autodesk's tools. I'd appreciate if you could test them out.

Hedgehog Converter: A fully-fledged terrain importer. While being much more capable of dealing with large amounts of geometry and node counts (it will take a while to get used to this, but you'll want to replicate the node distribution of a Generations stage), it also supports the creation of custom Terrain groups that can be manually loaded or unloaded. It also has a much more powerful system for importing to certain mesh layers. All details are in the application.




GIAtlas Converter: A very powerful Global Illumination Atlas importer for all stages, whether custom or even the originals. It contains two operation modes. One mode, Pre-Render, allows you to create the GI Atlas group distribution and test it out in-game with debugging colors. This includes auto-generating the texture sizes depending on the node sizes, using reference paths for quality settings and multiple variables to configure. As the name denotes, this mode is meant to test out your GI distribution before proceeding with an expensive process like rendering the stage. The Post-Render mode allows you to import individual lightmaps or shadowmaps on top of existing GI groups. This goes for both regular stages and custom stages. This makes working with GI much easier, as it detaches the group generation process from the painting process, which can require a lot of trial & error depending on the result ingame.




Havok Converter: A pretty bare-bones tool that allows you to import a model in a common 3D format and convert to use as the collision for a stage. Supports adding properties via custom tags.




The patron for the development of these tools was TwilightZoney, so be sure to send him your thanks if you find these useful. The development of these started around March of this year.

Couple of pending issues that should be resolved if possible:

• Using a good and open source DDS encoder. The only one capable of good DXT5 textures I found was Nvidia's tool so far.
• A good Compression/Decompression CAB LZX library. Relying on Microsoft's tools is very unreliable, and they're currently broken in the Windows 10 anniversary update.

Any documentation on how to use these is best kept to the Sonic Community Hacking Guide for Generations. All the tools include links to this page.

A couple of experiments done with these:

Spoiler

Importing higher resolution shadowmaps on top of existing GI Atlas Groups.



Altering the color of Generations' lightmaps.




TwilightZoney's Green Forest early large geometry imports and a small GI import.

This functionality being moved away from GLVL should help lighten the amount of features that need to be done for the rewrite in the future.

Possible future features:

• Support for Skeletons and Rigging on Hedgehog Converter.
• Support for Lost World and Unleashed. Lost World is almost done, it just needs to be verified to be working as intended.
• Skeleton, Animation and Dynamic Rigid Bodies with Convex Hulls support for HavokConverter.
• Some way to create and edit lights that doesn't rely on a 3D model format. These tools do not currently support light editing, so you'll just have to grab those from the original GLVL converter that uses the Ogre formats or write a custom tool using LibGens.