Dreamcast mods for SADX

Discussion in 'Engineering & Reverse Engineering' started by TheArcadeStriker, Jul 30, 2016.

  1. All this thing of the lightning seems somewhat curious now that I see this in the folder of the data extracted from splitSADX: Stage Lights.ini, which as it seems to indicate, it has the lightning or stage light data list for the stages. So I think that experimenting with this maybe could get to make stuff closer to DC. I will be trying for now with this if it works (Exporting with Struct Converter and trying with Mod Loader after exporting with Visual Studio).
     
  2. MainMemory

    MainMemory

    Have no fear...Amy Rose is here! Tech Member
    4,396
    64
    28
    SonLVL
    You have to either import a model, or copy an existing level piece.

    All that needs to be done is locating the data in the Dreamcast version, and feeding it into the PC version. Also, you don't have to build a DLL, the mod loader can load data from ini files.
     
  3. Well, that you can load the inis without building a DLL seems really convenient, though I don't know where I should place them anyways. I also wonder if you can actually do a kind of export of character mods (A.K.A modify CHRMODELS exported data and use it as a mod in Mod Loader), since one time I tried extracting the data for a Dreamcast Sonic mod for SADX, and tried making it into a DLL, but gave me errors in Visual Studio compilation, so it can be considered it couldn't work (I used DLLGenerator or something like that to get the cpp).
    I also checked around in Final Egg 2 in SALVL, both SADX and SADC and I didn't find anything that was like a invisible wall in both, so I don't know where I could get one to add it.

    By the way, thanks for all the help, and I wonder where the data of the lights could be found in the Dreamcast. Since as I mentioned, I would try changing some values for Ice Cap 2 and:
    [​IMG]
    I just changed some values with Blue and Red RGB values, and that was the result. I didn't intend it for it to look like the DC (And it doesn't indeed) but only a test. It looks overall nicer than DX though, and the change doesn't seem to affect characters or enemies (The snake enemy didn't have lightning like if it was in the modified stage, but in the DX normal stage lightning), so I wonder where the data could be in DX so that it either the lights that affect the characters, or something that makes the stage lights affect the characters too.
    EDIT: When I done that, I didn't touch AmbientRGB, so that may be what I needed to change?
     
  4. MainMemory

    MainMemory

    Have no fear...Amy Rose is here! Tech Member
    4,396
    64
    28
    SonLVL
    Use StructConverter to create the necessary INI files in the mod's folder, then include EXEData=data.ini in mod.ini to get the mod loader to load it.

    DLLModGenerator creating broken C++ is a known problem, using the INI option might work, but probably not, because character model swaps require extra stuff to not crash the game.

    Objects use different lighting data than the level geometry, I think the location of the data is known, but I'm not sure if the format is.
     
  5. E-122-Psi

    E-122-Psi

    Member
    1,897
    69
    28
    Looking at those comparisons, the distance lighting in DX was a nice effect on paper, but clashes badly with the backgrounds (unless there's a way to illuminate the windows separately from the buildings themselves to match the art).

    I think the character lighting is what does in a lot of the ambience in DX, they're always the same illumination, never matching the shadowing of their surrounding.
     
  6. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    Not sure if this is known already, but it seems that there are at least 3 types of lighting-related data in SADX. Have a look at this comparison:
    Dreamcast:
    [​IMG]

    SADX PC 2004:
    [​IMG]

    SADX PC 2004 with Dreamcast landtables and textures:
    [​IMG]


    As you can see, the shadow on the right disappears when you load the Dreamcast landtable in SADX PC. This leads me to believe there are the following types of lighting:
    1. Lighting baked into the landtable itself - apparently the Dreamcast version doesn't use it, which explains why Dreamcast landtables are often smaller in size and why the lighting is odd when you port Dreamcast landtables to SADX PC.
    2. "Stage lights" that you can edit in SADX thanks to MainMemory's tools. This may be the "Dreamcast lighting" we're looking for, which may restore the original DC lighting if it is found and ported from DC.
    3. Character lighting data, which is used a lot by the Dreamcast version and to a lesser extent the PC version.
     
  7. Dude

    Dude

    Tech Member
    3,138
    0
    16
    Southbridge, MA
    Random VR/AR trash
    The first one is the lighting in the vertex color channel. And yes, I believe that the dreamcast version of the game did little to none of this kind of lighting. As far as I can tell, your assessment of the different kinds of lighting is correct.
     
  8. SF94

    SF94

    Tech Member
    776
    2
    18
    Utah
    SA1/2 hax
    The shadow is literally just a transparent black model that they overlayed on top of the standard detail mesh, iirc. I wouldn't call that lighting per se. =P
     
  9. Dude

    Dude

    Tech Member
    3,138
    0
    16
    Southbridge, MA
    Random VR/AR trash
    It could also be extra slices made into the model with darker vertex colors in the cut-out segment. Both kinds of 'fake' shadows were used in the Emerald Coast upgrade, but you're right in that it's only faux-lighting. I actually like Emerald Coast much more in the GameCube version, but it's obvious that Emerald Coast got all the love and the other levels got a more 'meh' level of polish. Interestingly enough if they had wanted to they could have added vibrant lighting with lots of shadows like I did with the WV restoration, but if you go with high contrast then the character models look out of place in the shadows and stuff, so it makes sense that they didn't do it. They'd have to do an octree lightfield thing like they did in Generations / Unleashed or just live with weirdly bright character models chillin in the shade.
     
  10. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    So it appears that the lighting data in the Dreamcast version is stored in PL_ and SL_ .bin files. As evilhamwizard pointed out in the SA DC Prototype thread:

    Although the post says the LANTERN engine is only functional in the Dreamcast version, I found the same files in the Gamecube version and tried to load a modified ISO. After replacing all PL_ and SL_ files with their Dreamcast counterparts I couldn't see any differences. I tried replacing them on both the final version and the "Preview" prototype, which has some noticeable lighting differences from the final version. It stayed exactly the same, including character lighting and stage lights. I then tried to rebuild the ISO without any PL/SL files altogether. The game worked and looked just fine. So it seems the Gamecube version doesn't use these fles.

    Well, at least now we know where the Dreamcast version stores its lighting. Maybe this data is compatible with SADX and it's only a matter of reimporting it into the correct place, or maybe the LANTERN engine is just non-existent on the GC and PC versions at all. I still hope this helps us get a little closer to restoring Dreamcast lighting in SADX PC.

    EDIT: I removed all SL_ and PL_ files in the Dreamcast version and got some really interesting results:

    [​IMG]

    [​IMG]

    [​IMG]

    Pretty much all lighting disappeared from the game, including character gloss. This is interesting because it means these files could also contain character lighting data.

    EDIT2: Tried to boot Dreamcast SA with PL_ and SL_ files from the Gamecube version. Level lighting and character gloss are back, but everything is tinted blue for whatever reason. It does look like Dreamcast lighting though, so my guess is it's just a minor format incompatibility between DC and GC, and the files in the GC version contain DC lighting, which isn't used in the game.

    Here's another comparison. So the files in the Gamecube version indeed contain Dreamcast lighting but it's not used in the game.
     
  11. E-122-Psi

    E-122-Psi

    Member
    1,897
    69
    28
    Interesting find. I hope the tint issue can be solved. I'd love to see the Final Egg base in all it's original gloomy green glory.
     
  12. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    Sorry, I need to clarify that it wasn't the PC version that I managed to run with Gamecube PL and SL files, it was the Dreamcast version. So there's little point in resolving the tinting, since the data inside these files is likely to be the same in Dreamcast and Gamecube versions. Even though the PL and SL .bin files are on the disk, the Gamecube version doesn't seem to use them at all. The game runs just fine without them. I think they are leftovers that were converted into a Gamecube-friendly format for whatever reason.

    The PC version doesn't use PL and SL files either, but I suspect the data is inside sonic.exe. I hope MainMemory, Morph and Dude join in and share their insights.
     
  13. Essentially that could be the Stage Lights.ini, since the ini files are things extracted from the sonic.exe, like the landtables and stuff. Basically it may be possible if:

    1. There is away to extract the data/values of the PL/SL DC files.
    2. The values are converted or are compatible with similarities to StageLights stuff in the ini.
    3. Use the values there, and play!

    I wonder if it gets to be possible, either the way above or with another one. BTW I'm also working in Station Square with DC textures (Should start and finish Casinopolis though).
     
  14. MainMemory

    MainMemory

    Have no fear...Amy Rose is here! Tech Member
    4,396
    64
    28
    SonLVL
    Looking at the PL/SL files, I don't see any similarity between them and the PC version's lighting data. I can't tell what file is for what, or if there's multiple stages in each file, or anything about how they'd be used.
     
  15. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    I guess we need to crack the format first. I think I've found something about the way PL files work in the Dreamcast version.

    Have a look at PL_10B.bin (Emerald Coast Act 1, but apparently this affects all Emerald Coast acts):
    [​IMG]
    The highlighted area is an inverted RGB (BGR) code, which was originally 000000.
    So here's what it looks like when it's FF0000:
    [​IMG]
    Then FFFF00:
    [​IMG]
    Then 00FFFF:
    [​IMG]
    Hope this helps. The reason everything is tinted blue when you load the Dreamcast version with PL files from the Gamecube version is that the entire row of RGB values changes from 000000 to FF0000.

    EDIT: Okay, so PL files are literally palettes, at least in the first half. They are 4-byte patterns, with RGB codes separated with FF or 00 (could be light intensity or something). Apparently the second half of the PL file contains character lighting data.

    EDIT2: I got the Gamecube "Preview" prototype to work with PL files!

    [​IMG]

    Apparently the palette-based lighting engine is turned off by default, but you can make the game switch back to the lighting that uses PL files if you mess around with the debug menus a little and press Z when in a level (the Retro wiki says it toggles alpha blending). The game then picks up lighting information from the PL files that I modified. It still looks different from original SA, but the way lighting is applied seems very similar to the Dreamcast version. This leads me to the following conclusions:

    1. The palette-based lighting engine is disabled by default in the Gamecube version, which explains why the final game doesn't use PL/SL files.
    2. The engine might still be in the game, perhaps it just needs to be turned on. It would be interesting to check what happens in memory when you press Z in the Preview prototype. It could point us to a value that can be changed in the final Gamecube version, and maybe even the PC version.
     
  16. McAleeCh

    McAleeCh

    Oldbie
    995
    22
    18
    Amazing work! Guess the next step will be to try and discern whether this lighting system still exists unused in the final GC version, and if so whether it's possible to get the game to use it again. The fact that it seems to still exist in the preview gives me hope - guess whether or not it's in the final too will depend how lazy the developers were. Really hope they were lazy in this case, though - it'd be awesome to be able to restore the DC lighting to the GC version (and hopefully increase the chances of being able to do so for the PC version, too - if this lighting system isn't in the GC final then I doubt there's any chance of it having made its way into the PC port).

    Fingers crossed - will keep watching with interest for further developments...! = )
     
  17. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    Some more info on PL files...

    The only difference between Gamecube PL files and Dreamcast PL files is that the Gamecube files use a reverse format: Separator, R, G, B. Otherwise they are identical, at least the ones I've checked so far.

    Here is my attempt to decipher a PL file (Dreamcast version):
    [​IMG]
    It looks like two sequences of colors, which I think are two palettes. If you convert the BGR codes into colors, you can see that the first palette (highlighted red) is a white-to-black gradient. In Twinkle Park it's slightly more red, which is why many shadows in the stage have a subtle pinkish tint. The second palette is mostly black, but it has other colors in other sections of the file.

    As far as I can see, the PL files have the following sections:
    0x0 to 0x1000 - stage & object palettes, including badniks, sky etc. Some stages seem to have a separate set of palettes at 0x800-0x1000, I don't know where it is used (yet).
    0x1000 to 0x1800 - character lighting. This applies only to player models, though in some rare cases other objects may use this palette, e.g. the floating meteors in the circuit of Twinkle Park Act 1.
    0x1800 - 0x2000 and 0x2000 onwards - no idea yet
    Now even if you fill the entire PL file with 00 of FF, some areas will not be affected. For example, in Twinkle Park Act 1 most of the walls in the kart area and the floor in the area where you first get the kart - these won't change no matter what you do with the PL file.

    Now let's go back to SADX Preview prototype. While it can load lighting data from PL files, it seems to load character lighting data from elsewhere. You can still edit character lighting data in the "LS PALETTE EDIT" debug menu:
    [​IMG]
    Notice the two gradients that appear at the bottom when you switch to this menu. These gradients change as you play around with the parameters in the menu and they seem to be Sonic's "palette". Which makes me wonder if these two gradients are the same two palettes that are stored in PL files at offsets 0x1000-0x1800. Quite possibly the parameters in the menu are used to generate these two gradients, which are then applied to character models. Or, in case of PL files in the Dreamcast version, the gradients are loaded directly from the file. Now if the above is indeed true, this means that stage lighting uses a similar scheme and format.

    Another interesting thing. If you keep tapping Y during the SEGA screen, you will see this:
    [​IMG]
    Hint at the possible layout of a PL file perhaps?
     
  18. darkspines35

    darkspines35

    It's Easy Actually. No, seriously. Tech Member
    248
    0
    16
    .V.
    Sanik Adevnt Casters
    Let me begin by saying wow, this progress is pretty awesome to understanding how the SA1 light engine (named Lantern according to evilhamwizard) worked. I figured I should toss this in real quick. About two years ago I did some work on the character light table. I got a hold of where the data was a long time ago and figured out the format of it thanks to the SADX Preview. Here's my notes on it, I hope it helps out with the future of SADX mods.

    Code (Text):
    1. Character & Object Light Table (LS Palette Data)
    2.  
    3. Data Starts: 0x503E88
    4.  
    5. Pointer to Data: 0xAA4D
    6.  
    7. --Structure Setup--
    8.  
    9. 0x00    Byte    Stage ID
    10. 0x01    Byte    Act ID
    11. 0x02    Byte    00 = Characters, 06 = Unknown, 08 = Enemies
    12. 0x03    Byte    Light type (Or Use XYZ/Specular)
    13.  
    14. 0x04    Float3  Direction
    15. 0x10    Float   Diffuse     (Max 4)
    16. 0x14    Float3  Ambient RGB (Max 4)
    17. 0x20    Float   CO      (Max 99)
    18. 0x24    Float3  CO RGB      (Max 4)
    19. 0x30    Float   Specular    (Max 99)
    20. 0x34    Float3  Specular RGB    (Max 4)
    21. 0x40    Float   CO2     (Max 99)
    22. 0x44    Float3  CO2 RGB     (Max 4)
    23. 0x50    Float   Specular2   (Max 99)
    24. 0x54    Float3  Specular2 RGB   (Max 4)
    25.  
    26. -Notes-
    27.  
    28. CO Affects Diffuse Colouring
    29.  
    30. Specular Affects amount of Specularity.
    31.  
    32. Both have affect on the GC Preview, neither seem to have any affect on the PC version.
    Just to note, I did try modifying this data before, and for some reason, it never worked right. I could only modify it using Game Engine when the game was running.

    Oh, and to add to the byte indicating what's affected. The characters were also affected when it was set to 00. I don't remember what the unknown affected at all, and the enemies were not universally affected. Just some of them IIRC.
     
  19. PkR

    PkR

    Member
    186
    10
    18
    SADX Dreamcast Conversion, The Emeralds' Awakening
    Thank you very much for your input! I was able to locate character lighting in the PC .exe.

    Replacing it all with 00 makes the characters pitch black, like in the Dreamcast version when you replace the corresponding area in the PL file:
    [​IMG]
    So now we know how to change character lighting data in the PC version.

    This is pure speculation at this point, but I assume the following:
    The Dreamcast version uses character lighting data from PL files, and PL files are just color gradients. I couldn't find anything else inside them.
    The GC Preview prototype doesn't use PL files for character lighting (it does for the levels though, if you press Z when in a level). And there's a toggle between "THIS INFO" and "FILE" in the same menu where you edit character lighting. And those gradients at the bottom look very similar to the two sequences of colors inside the PL file. Could it be the same system that uses pre-made values on the DC and generates them on the fly on the Gamecube?

    Now if those gradients are the same palettes as in PL files, we could convert the data from PL files into actual color gradients and then use the debug mode in GC Preview prototype to recreate these gradients. This way we could get the values to be fed into the PC version to recreate Dreamcast character lighting.

    These are just my thoughts at the moment, now I'll be trying to recreate Dreamcast character lighting in Twinkle Park Act 1 in the PC version.

    Thanks again for the extremely valuable information!

    EDIT: I'm all over the place with random information again. So, about PL files and those gradients in SADX Preview. There are 256 rows (00 to FF) of R, G, and B values. Each series of RGB values corresponds to parts of the gradients you see in LS PALETTE EDIT mode, but that is probably just an inaccurate visualization. There are at least two tables that contain those RGB values: BASE0 and SPEC0, so I'm guessing the top gradient is "base" lighting and the bottom one is "specular" lighting.
    The character lighting section in the PL file is 2048 bytes long. This is exactly the number of bytes needed to store two 256-color palettes if each color is separated with 1 extra byte, like it is in PL files.

    EDIT2: Regarding this one:

    0x03 Byte Light type (Or Use XYZ/Specular)

    I have no idea how this is supposed to work, but this affects specular lighting. By setting this byte to various values from 00 to 0F I got like 7 different-looking results. Could it be some kind of a preset? Have a look:
    [​IMG]
    (actually A and F look the same to me now... disregard that)
    Setting it to SOME of these make the game read specular light setting at 0x30. However, it seems to be applied differently from the Gamecube version. I still haven't figured out how exactly it works. The byte above changes a lot in the way specular lighting looks. I suspect the Specular parameter needs to be multiplied or something like that.

    I'm starting over with everything zeroed to see which parameters match exactly as in the Gamecube Preview version. But it's good news that at least something is working...
     
  20. SF94

    SF94

    Tech Member
    776
    2
    18
    Utah
    SA1/2 hax
    I bet those FF separators are actually the alpha. I doubt they end up getting used in any meaningful way, but it's probably because they didn't want to make a new structure with just RGB in it.

    In any case, I was just messing with lighting myself trying to achieve a Dreamcast-esque effect, but I did it manually by copying the frame buffer to a texture and calculating the average color of the upper and lower halves of the screen to configure two specular lights, one for above and below the character. Here's the original, and here's my dynamic lighting test. It ended up not being a great solution, but it did look cool specifically in this section of Red Mountain (some cool things like the lava spouts lighting up the whole area was really interesting looking). This was done exclusively with DirectX though, so it probably doesn't help in your case.

    (an example of it not working so well: original -> dynamic)

    I'd love to help out though. If you can give me the address of the data you found in the PC version, I can look at how it's used in more detail in the disassembly. 19th post as trial member though... He has my vote to become a full member at least, moderators. :v: You're welcome to join #x-hax on IRC if you want to talk in real-time.