Sonic and Sega Retro Message Board: The SX Engine - Sonic and Sega Retro Message Board

Jump to content

Hey there, Guest!  (Log In · Register) Help
  • 7 Pages +
  • 1
  • 2
  • 3
  • 4
  • 5
  • Last ►
    Locked
    Locked Forum

The SX Engine Recreating the Sonic Xtreme executable (now open source!)

#31 User is offline SANiK 

Posted 10 January 2011 - 09:47 PM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
QUOTE (Uhyve @ Jan 10 2011, 01:07 PM)
QUOTE (SANiK @ Jan 10 2011, 10:30 AM)
The scale and offset are stored as a combined 32bit number.
The top 16 bits are the relative offset and the bottom 16 bits are the relative scale.

The scale is relative to 1 which you've figured out already and it uses 128.0f accuracy which you've also figured out.
The offset is 256.0f accuracy.

Ah man, that is alot simpler, feel like I should've thought of that.

QUOTE (SANiK @ Jan 10 2011, 11:02 AM)
Ofer made up as he went along. There are various functional differences in the DEF format, even though the DEF files show the same file version number.
You might also want to invest time in a nice database to manage the resources.

Yeah, I noticed some stuff in the JG_0* levels that suggested something like that happening, some sort of database probably would be the best idea. My code feels really hacky at this point...

QUOTE (SANiK @ Jan 10 2011, 11:02 AM)
Code

... Especially after seeing that. That is really clean code, I'm kind of embarrassed to have released my source now. Gonna have to spend some time looking through all that, I'm not too proud to admit that I'm alittle giddy right now. My code has so many glitches, I've constantly been wondering how you managed to get some of the stuff working.


Well even my code is hacky, but that is expected when reverse engineering. Furthermore I could have used a loop for certain parts of the above code, but I did not and instead I wrote things out in case there was an unexpected exception to how something might be handled. All part of the learning process.

I would suggest you focus more on coding than on cleansiness.
Do this so you can understand the file format better and write a file format specification.
After that is done, you can finally start coding a neater map viewer since you will have the ability to preplan your code from your specification notes.

Trying to reverse engineer and make a complete product at the same time will only exhaust you.

#32 User is offline SegaLoco 

Posted 10 January 2011 - 10:37 PM

  • W)(at did you say?
  • Posts: 999
  • Joined: 16-August 06
  • Gender:Male
  • Location:Corpus Christi, TX
  • Wiki edits:79
Dude, are you considering porting to OpenGL sometime, or will this stay a Direct3D thing?

#33 User is offline Uhyve 

Posted 11 January 2011 - 02:11 AM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
QUOTE (SANiK @ Jan 11 2011, 02:47 AM)
Well even my code is hacky, but that is expected when reverse engineering. Furthermore I could have used a loop for certain parts of the above code, but I did not and instead I wrote things out in case there was an unexpected exception to how something might be handled. All part of the learning process.

I would suggest you focus more on coding than on cleansiness.
Do this so you can understand the file format better and write a file format specification.
After that is done, you can finally start coding a neater map viewer since you will have the ability to preplan your code from your specification notes.

Trying to reverse engineer and make a complete product at the same time will only exhaust you.

Yeah, that sounds like a good idea, I've never been very patient when it comes to programming, but in this case, that sounds like the best way of going.

QUOTE (SegaLoco @ Jan 11 2011, 03:37 AM)
Dude, are you considering porting to OpenGL sometime, or will this stay a Direct3D thing?

I'll probably be leaving it as DirectX for now. But the project is open source, so if anyone thinks that they're up to the task, they can always get in contact. I assume that it's also pretty dependent on MSVS, so I dunno if it would compile for Linux, since I've barely ever used GCC. Would be nice though.
This post has been edited by Uhyve: 11 January 2011 - 02:12 AM

#34 User is offline kazade 

Posted 14 January 2011 - 04:15 AM

  • Posts: 64
  • Joined: 16-March 10
  • Project:A 2D Physics Engine
QUOTE
QUOTE (SegaLoco @ Jan 11 2011, 03:37 AM)
Dude, are you considering porting to OpenGL sometime, or will this stay a Direct3D thing?

I'll probably be leaving it as DirectX for now. But the project is open source, so if anyone thinks that they're up to the task, they can always get in contact. I assume that it's also pretty dependent on MSVS, so I dunno if it would compile for Linux, since I've barely ever used GCC. Would be nice though.


Just so you know, I'm taking a look at an SDL/OpenGL port (developing on Ubuntu). I'll let you know how it goes...

P.S. Thank you for open sourcing it. Respect++;
This post has been edited by kazade: 14 January 2011 - 05:34 AM

#35 User is offline SANiK 

Posted 14 January 2011 - 05:23 PM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
In Level.cpp there is a typo:
usedCubes[currentCube].zScale = (1+(signed short)combinedScaleOffset)/128.0f;

The 1+ should be outside the parenthesis.

#36 User is offline Uhyve 

Posted 14 January 2011 - 07:32 PM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
QUOTE (SANiK @ Jan 14 2011, 10:23 PM)
In Level.cpp there is a typo:
usedCubes[currentCube].zScale = (1+(signed short)combinedScaleOffset)/128.0f;

The 1+ should be outside the parenthesis.

Right, with the correct scaling, everything looked way too big, so that was me trying to fix it without really understanding the code. Got it figured out now, my problem was that I was already adding 1 to the scale somewhere else in the code, so it looked more right with the +1 inside the parenthesis because it was getting divided by 128. Now I've just gotta figure out where the offsets are going wrong, I'd put money on me overlooking some old bit of code.

QUOTE (kazade @ Jan 14 2011, 09:15 AM)
Just so you know, I'm taking a look at an SDL/OpenGL port (developing on Ubuntu). I'll let you know how it goes...

P.S. Thank you for open sourcing it. Respect++;

Nice. I'll try to keep away from using any more DirectX features to make it alittle bit less of a moving target. I can see the pointless shaders (HLSL) being annoying, they're really just there for fun.

Edit: Heh, got it. I was using my old bit of "offset" code, which worked by altering the axis of the scaling (seemed to work surprisingly well, even though it was obviously completely wrong), now I'm actually adding onto the block positions and it's fine.
This post has been edited by Uhyve: 14 January 2011 - 07:51 PM

#37 User is offline SANiK 

Posted 14 January 2011 - 08:11 PM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
I would get rid of the shaders totally as well as smoothing and keep it pixel filled.

I would split the engine up into the following classes:
CODE
//Sonic X-treme components
class sxValue //A growing linked list that takes an index an returns the string or value stored there
class sxHexDump //A linked list that contains raw byte data

class sxVariable
class sxPalette
class sxCamera
class sxPath
class sxTexture
class sxQubix
class sxCube
class sxActorType

class sxFilePcx //Loads a pcx file and returns a palette array and an index array
class sxFileDef //Loads a def file and creates the above classes in sxSystem. Remember multiple DEF files can be loaded and they merge variables and other settings.

class sxSystem //Keeps the merged DEF classes.

//Engine based components
class enCamera
class enInput
class enPoly      //Rendering 3D polygons with an enTexture
class enImage   //Software image manipulation code for generating some of the dynamic textures in X-treme.
class enTexture  //Abstraction layer to send texture to renderer
class enSprite    //Rendering on screen enTextures for UIs
class enBillboard  //Rendering 3D billboard sprites
class enMath     //Math function

class enSystem //Encompasses everything

This post has been edited by SANiK: 14 January 2011 - 08:19 PM

#38 User is offline Uhyve 

Posted 14 January 2011 - 08:42 PM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
QUOTE (SANiK @ Jan 15 2011, 01:11 AM)
I would get rid of the shaders totally as well as smoothing and keep it pixel filled.

I would split the engine up into the following classes:
CODE
//Sonic X-treme components
class sxValue //A growing linked list that takes an index an returns the string or value stored there
class sxHexDump //A linked list that contains raw byte data

class sxVariable
class sxPalette
class sxCamera
class sxPath
class sxTexture
class sxQubix
class sxCube
class sxActorType

class sxFilePcx //Loads a pcx file and returns a palette array and an index array
class sxFileDef //Loads a def file and creates the above classes in sxSystem. Remember multiple DEF files can be loaded and they merge variables and other settings.

class sxSystem //Keeps the merged DEF classes.

//Engine based components
class enCamera
class enInput
class enPoly      //Rendering 3D polygons with an enTexture
class enImage   //Software image manipulation code for generating some of the dynamic textures in X-treme.
class enTexture  //Abstraction layer to send texture to renderer
class enSprite    //Rendering on screen enTextures for UIs
class enBillboard  //Rendering 3D billboard sprites
class enMath     //Math function

class enSystem //Encompasses everything

That seems like a good way of organizing stuff, I'm already getting to a point where moving around the huge classes is getting annoying. But yeah, I think I can manage that, my code is a bit messy but I don't think it'll be too hard to modularize. Plus, a general rearrangement of code will probably help me get rid of any old useless stuff. In fact, yeah, I don't think that'll be too difficult actually.

Yeah, I'm gonna need to get rid of the shaders, it was becoming a bit of a hassle keeping them in really. The bloom is pretty useless, and shader based lighting seems to be quite slow.

PS Heh, I like open source, it's been up for less than a month and I've already got a ton of useful help. If anyone hadn't guessed, I'm not a professional coder, I've got a degree and everything, but not a job yet, so I'm still super inexperienced in certain areas.
This post has been edited by Uhyve: 14 January 2011 - 08:43 PM

#39 User is offline kazade 

Posted 15 January 2011 - 04:06 AM

  • Posts: 64
  • Joined: 16-March 10
  • Project:A 2D Physics Engine
QUOTE (Uhyve @ Jan 15 2011, 12:32 AM)
QUOTE (kazade @ Jan 14 2011, 09:15 AM)
Just so you know, I'm taking a look at an SDL/OpenGL port (developing on Ubuntu). I'll let you know how it goes...

P.S. Thank you for open sourcing it. Respect++;

Nice. I'll try to keep away from using any more DirectX features to make it alittle bit less of a moving target. I can see the pointless shaders (HLSL) being annoying, they're really just there for fun.



The main problem I'm having is the use of D3DXVector, D3DXMatrix and D3DXPlane (and probably others I haven't go to yet) in public interfaces. Those make it difficult to port, but I understand there's not much you can do about that aside from using a different math library which will just make more work for you. Fortunately, I've written a GL math library that's similar to the D3DX stuff: http://blog.kazade.co.uk/p/kazmath.html

So the way I've started implementing stuff is by duplicating files (e.g. Camera.h -> CameraGL.h) and replacing things with my math library. I'm currently writing a library for sprites ( https://launchpad.net/kazsprite ), when I've done that I can move on. So far I've ported the window init code and the camera.

#40 User is offline Uhyve 

Posted 15 January 2011 - 10:05 AM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
Oh yeah, it hadn't even occurred to me that the matrix math is using DirectX, that's a pain, it gets used quite a lot in the engine.

Does OpenGL have native support for any specific mesh types? Just wondering because the cubes that are used in the engine is an actual mesh file (.x file). That was more out of laziness though, I can make it create the mesh in memory at run time if need be.

#41 User is offline SANiK 

Posted 15 January 2011 - 11:10 AM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
OpenGL just draws, nothing else. If anything look up the .obj format. It is pretty quick and painless to implement.

Sonic X-treme has two occasions where renderflags occur. They occur at the Texture level and the Cube level.
At the texture level things like FlipH and Rotate get applied or baked to the actual texture image.
At the cube level they get applied to the UVs.

Playing around with texture scrolling, it seems to be that the format is: Color(?, Velocity, Angle)
The angle and velocity look to be both unsigned values from 0 to 256.
The Angle must be converted to degrees. I am still tinkering with this.

There is a couple of things to consider such as the order of operations:
UVFlipH then UVFlipY then UVRotate then UVScroll is what I am currently using, but I still see some bad UVs.
I will need to look into it further.

I was looking at the ACT_Path in JADE1 like Energy was in the previous topic:
HexDump "01 00 03 00 46 00 00 00 00 00 01 3a 80 0d 00 09 80 04 00 c0 b1 13 00 00 80 0e 00 09 80 04 00 c0 e4 46 00 00 80 10 00 0a 80 04 00 a0 ff ff 00 00 80 13 00 0e 80 01 00 40 ff ff 00 00 80 06 00 15 80 01 00 80 ff ff 00 00 80 17 00 09 80 00 00 80 ff ff"

I split it up into header and data. There are different versions of paths, but the later one follows mostly this scheme.

Header:
0100 0300 4600 0000 0000 013a

The value in red is the data size which is 70. Energy seems to have included the 013a as part of the data when it is part of the header.

Data:
80 0d 00 09 80 04 00 c0 b1 13 00 00
80 0e 00 09 80 04 00 c0 e4 46 00 00
80 10 00 0a 80 04 00 a0 ff ff 00 00
80 13 00 0e 80 01 00 40 ff ff 00 00
80 06 00 15 80 01 00 80 ff ff 00 00
80 17 00 09 80 00 00 80 ff ff

The above has a pattern, but there are a couple of things to consider:
1) Path data and other HexDump data may be in the format:
Instr Par0 Par1 Par2... and may not necessarily have a pattern.

2) 00 00 is missing at the end to complete the pattern, but it might be an implied 00 00 since if the system expects to read 12 byte slices anything out of bounds may be configured to read as 00?

3) If the data is X Y Z, it might be word word word or byte byte byte.
Furthermore it is highly likely there is an orientation vector to specify 'up' or as physicists call it, a normal.
This might be encoded as word word word or byte byte byte, or possibly an angle.
There may also be a velocity value encoded as well?

Another interesting HexDump is not a path but an action definition called "moveIO_1" in RED1
HexDump "01 00 03 00 38 00 00 00 00 04 00 04 10 00 00 00 17 10 01 01 20 00 00 00 17 04 01 01 20 00 00 00 03 21 18 10 00 00 00 00 00 01 10 01 00 00 00 00 00 20 10 01 00 00 00 00 00 01 10 01 00 00 00 00 00 20 10 01"

Header:
0100 0300 3800 0000 0004 0004

Data:
10 00 00 00 17 10 01 01
20 00 00 00 17 04 01 01
20 00 00 00 03 21 18 10
00 00 00 00 00 01 10 01 00 00 00 00 00 20 10 01
00 00 00 00 00 01 10 01 00 00 00 00 00 20 10 01

In the Red Sands video it is the platform going in and out at the start of the video. The format is noticeably different.
It might have to do with the 0004 0004, changing the type of HexDump format that follows.

Furthermore I would expect paths to be lists of connecting nodes and nothing more, whilst I expect action definitions having some type of code to them.
The code can be something like IfEnter CreateCube Type, X, Y, Z.
It might be possible the moveIO_1 HexDump references a path and the in/out motion may not necessarily be stored within the local HexDump.

-------------------------------------------------------------------------------------------------------------------

Looking at the JADE1, there is that switch that Sonic hits:
NEW_CUBE "SWITCH_1"
{
Value 026,41 //Swap
HexDump "0100030008000000000400341000000001000801" //Action Definitons
}

NEW_CUBE "SWITCH_1A!"
{
Value 026,40 //Swap
HexDump "010003001e000000000000220f0001010002000009000801000200000100100200000000100
001010002" //Action Definitons
}

"SWITCH_1" upon getting hit in the video changes to "SWITCH_1A!"
The Swap value is probably a parameter that gets passed to an instruction in the HexDump?

"SWITCH_1A!" being complicated most likely calls the path to be made at a certain location?
Or might "SWITCH_1" be what creates the path?

If we take the action definition for "SWITCH_1 and break it down:

Header:
0100 0300 0800 0000 0004 0034

Data:
10 00 00 00 01 00 08 01

Somehow the above encodes to "Change current cube to swap cube when sonic jumps into it"
I'm starting to think the header's 0004 0034 decides Sonic's action to activate the HexDump,
or Sonic's action might be encoded within the HexDump itself.

It also might be:
aa bb cc dd ee gg hh kk
10 00 00 00 01 00 08 01

aa = Function to do, 0x10 = swap?
bb = Parameter A (unused)
cc = Parameter B (unused)
dd = Parameter C (unused)

eegg = When Actor/Cube

hhkk = Jumps into front face?
This post has been edited by SANiK: 15 January 2011 - 11:55 AM

#42 User is offline SANiK 

Posted 15 January 2011 - 02:43 PM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
So I got the coordinate based stuff working for paths.


0100 0300 8100 0000 0001 00fe

xx xx yy yy zz zz
14 00 04 80 04 00 00 b1 13 00 00 00 //Segment 0
15 00 04 80 04 00 00 23 37 00 00 00 //Segment 1
18 00 04 80 04 e2 d7 85 5e 00 00 00 //Segment 2
1a 00 07 80 03 00 c0 07 7e 00 00 00 //Segment 3
1a 00 0b 80 03 0e 8d 7a a1 00 00 00 //Segment 4
17 00 0c 80 03 1e 68 fc c0 00 00 00 //Segment 5
14 00 0a 80 04 00 20 8d dc 00 00 00 //Segment 6
16 00 08 80 04 00 00 ff ff 00 00 00 //Segment 7
19 00 08 80 04 00 00 ff ff 00 00 00 //Segment 8
1c 00 08 80 04 1d ed ff ff 00 00 80 //Segment 9
1c 00 07 80 04 1d ed ff ff //Segment 10, 00 00 00 seems to be implied

The last 6 bytes are probably 3 normals in the form of bytes and 3 flags, or 3 normals in the form of fixed16.

Coordinates are unsigned absolute fixed16 positions on the map. Values are Motorola Big Endian and not little endian.
To convert from fixed16 to float divide by 256.0f.
This post has been edited by SANiK: 15 January 2011 - 02:45 PM

#43 User is offline Uhyve 

Posted 15 January 2011 - 05:00 PM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
psyduck.png
That... is amazing. You figured that out really quickly. There's not really much for me to say since it all makes perfect sense. Though I'm alittle clueless about what "swap" refers to, obviously it's in the cube definition aswell, but I don't really understand what is does there either.

QUOTE (SANiK @ Jan 15 2011, 04:10 PM)
OpenGL just draws, nothing else. If anything look up the .obj format. It is pretty quick and painless to implement.

Ah, didn't realize, I've never used OpenGL, figured it would be essentially the same as DirectX. I was thinking of writing a model importer/extractor at some point, Andrew75 suggested Collada because it's an open standard.

QUOTE (SANiK @ Jan 15 2011, 04:10 PM)
Sonic X-treme has two occasions where renderflags occur. They occur at the Texture level and the Cube level.
At the texture level things like FlipH and Rotate get applied or baked to the actual texture image.
At the cube level they get applied to the UVs.

I only just realised you mentioned that in your code. It probably should have occurred to me, I've just been adding flags for both blocks and textures as I figure them out, didn't realise that they were the same. Another place where some sort of database would come in handy really.

Gonna need to put some time in this week. Don't like that you've managed to get so much figured out and I've... managed to complete Mass Effect.

#44 User is offline SANiK 

Posted 15 January 2011 - 05:25 PM

  • Posts: 412
  • Joined: 21-September 04
  • Gender:Male
  • Wiki edits:6
QUOTE (Uhyve @ Jan 15 2011, 05:00 PM)
Though I'm alittle clueless about what "swap" refers to, obviously it's in the cube definition aswell, but I don't really understand what is does there either.


Well if each cube has an ID then you can use "swap" to make a cube change to another cube upon a certain event happening.
So if you have a ring, and an empty cube is '0', then you can set the ring's swap to 0 and have a HexDump event that triggers the swap once Sonic touches the ring cube.

QUOTE (Uhyve @ Jan 15 2011, 05:00 PM)
I only just realised you mentioned that in your code. It probably should have occurred to me, I've just been adding flags for both blocks and textures as I figure them out, didn't realise that they were the same. Another place where some sort of database would come in handy really.


I think the way you are doing it might be best if you want to make a game, which is to generate the internal format for the map.

Currently I load the DEF into a database and I draw directly from the database which is slow, but since the database is 1:1 with the DEF file format, I can easily test different variations when reverse engineering.
This post has been edited by SANiK: 15 January 2011 - 05:25 PM

#45 User is offline Uhyve 

Posted 15 January 2011 - 05:57 PM

  • Master Procrastinator
  • Posts: 46
  • Joined: 08-November 10
  • Gender:Male
  • Location:Huddersfield, England
  • Project:SX Engine
QUOTE (SANiK @ Jan 15 2011, 10:25 PM)
Well if each cube has an ID then you can use "swap" to make a cube change to another cube upon a certain event happening.
So if you have a ring, and an empty cube is '0', then you can set the ring's swap to 0 and have a HexDump event that triggers the swap once Sonic touches the ring cube.

Ohh, that makes sense. Yeah, figured it was probably something like that, just wanted to make sure.

QUOTE (SANiK @ Jan 15 2011, 10:25 PM)
I think the way you are doing it might be best if you want to make a game, which is to generate the internal format for the map.

Cool, no reason to do extra work, heh.

  • 7 Pages +
  • 1
  • 2
  • 3
  • 4
  • 5
  • Last ►
    Locked
    Locked Forum

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users