don't click here

Saxman's Sonic Boom engine

Discussion in 'Engineering & Reverse Engineering' started by saxman, Dec 4, 2014.

  1. saxman

    saxman

    Oldbie Tech Member
    So, a few of you have figured out that the driver behind Sonic 2 revision 3 is Sonic Boom. But what exactly IS it?


    [​IMG]


    BACKGROUND:
    It's called Saxman's Sonic Boom engine, but you can just call it Sonic Boom for short like I do (and did going all the way back to 2011). Why Sonic Boom? Because "boom" means to progress rapidly.

    The project started out as a series of bug fixes for Sonic 2 with the intent of being released as THE unofficial revision 3 of Sonic the Hedgehog 2. However, at some point along the way I decided this project should be much more than simple bug fixes, because who would really want to download that? I probably wouldn't bother myself. So, inspired by Team TNT's BOOM engine (Remember that? No? Well, it was a pretty big deal back in 1998), I decided to make this an enhancement of the Sonic 2 disassembly itself and focus on trying to turn it into a developer's crutch of sorts. The idea was that new features could be plugged in by setting certain flags in the code. That way, hackers can pick and choose what they want for their projects.

    The source code is intended to replace the need for any other disassemblies currently being used. The source code is based on the 2007 disassembly, partly because I did not really like the overall state of the SVN code. I decided to clean up the code myself and slowly but surely make it more maintainable, customizable, and readable. That effort is not done, but I did make a lot of progress on that front. If you want byte-by-byte perfection, look elsewhere. The Sonic Boom source code is intended to provide you with a plethora of bug fixes and enhancements designed to make your hacks better.

    I wrote a wiki entry on Sonic Boom -- http://info.sonicretro.org/Saxman%27s_Sonic_Boom_Engine. I will highlight the engine enhancements which I think are most important changes below.


    ENHANCEMENTS:
    ## A new level layout format was created to conserve RAM. This allows levels to be much larger than even those from Sonic 3. It also separates physical layout from visual layout, allowing one meta-block to use a different meta-block's physical properties.
    ## Levels can now wrap at points other than 0x800. These include 0x2000, 0x1000, 0x400, 0x200, and 0x100.
    ## The maximum number of pattern load requests has been increased from 15 to 24. No additional RAM is required to make this possible.
    ## Each act can now have its own set of art.
    ## Now any level can have water, and you can also control the movement of the water for each level.
    ## Sonic and Tails can now start in different locations of a level. The 2 player vs. mode also has its own start positions for both players.
    ## Game demos now have more data associated with them. Each demo has data for the start position, length of the demo, which player the demo was recorded with, the zone and act, and whether or not two player split-screen was used.
    ## Knuckles the Echidna has been imported from Knuckles in Sonic 2. He can be enabled using the cheat code U, U, U, D, D, D, L, R, L, R at the title screen.
    ## The "Saxman" cheat was added to allow many other cheats and options be enabled through one code. The code is 01, 09, 08, 04, 00, 07, 02, 07.
    ## Invulnerability mode from the original Sonic the Hedgehog can now be used. It is enabled when the debug cheat is entered. To activate, hold A and press the start button when starting the game.
    ## A demo recorder has been added to the game. Hold the B button when entering a level with the demo recorder flag enabled (via the Saxman cheat) to start recording gameplay. Press the start button to stop recording. The demo will be played during the normal game demo sequence following the title screen.
    ## The in-game debugger from the original Sonic the Hedgehog has been imported.
    ## Support for 6-button controllers has been added.
    ## Sega 32X support added.
    ## The 32X version includes a new version of the Kosinski decompressor that runs on the SH-2. It provides a modest speed increase.

    Additionally, the source code has been restructured in certain ways so data can be more easily maintained. For instance, the animals designated to each zone is now controlled by simplified tables that allow for easy maintanence and expandability. The same with water settings for each stage.


    SONIC BOOM LEVEL FORMAT (SBLF):
    I completely rewrote the code for the handling of level data. The source code still supports the original Sonic 2 level format. But now, optionally, hackers can choose to use the brand new Sonic Boom Level Format (SBLF). It does a couple things:

    First, it separates physical layout from visual layout. This allows collision properties to be placed independant of their original meta-block associations. So the same meta-block at two different locations in the same level could have different collision properties assigned to it.

    Second, a number of light-weight compression schemes are used so the levels take up less space in RAM.

    Third, as made possible by the point noted above, levels can now be much larger than they were in the original Sonic 2. The maximum theoretical size of a level is 0x8000 x 0x2000. That's freakin' huge.

    An earlier version of Sonic Boom was supported by an earlier version of S2LVL, although the current version does not fully support Sonic Boom (ring layout is, but not meta-block layout). I have provided plenty of documentation though, so interested individuals could add this support. Thanks to MainMemory for working with me on this.

    In the downloads below, I have included a link to some utilities I created to convert standard Sonic 2 level format files over to the new Sonic Boom format.


    DOWNLOADS:
    Get Saxman's Sonic Boom engine source code and other goodies below:

    Sonic 2 R3 (Genesis and 32X ROMs): https://onedrive.live.com/redir?resid=6C991B4888367A4D%211723

    Saxman's Sonic Boom engine source: https://onedrive.live.com/redir?resid=6C991B4888367A4D%211722

    Level layout format conversion tools: https://onedrive.live.com/redir?resid=6C991B4888367A4D%211716

    SonLVL INI files (legacy level format only): http://1drv.ms/1z4fOUV

    Sonic Boom engine documentation: https://onedrive.live.com/redir?resid=6C991B4888367A4D%211720

    Sonic Boom Level Format (SBLF) documentation: https://onedrive.live.com/redir?resid=6C991B4888367A4D%211719


    GIT HUB REPOSITORY
    Saxman's Sonic Boom Engine is now on GitHub: https://github.com/saxman727/sboom

    Please create a new branch with a unique name (I.e. NOT "sboom_#.##") if you wish to make modifications.


    THANKS:
    I didn't do a good job of keeping track of everyone who helped me in one way or another. If I forget anyone at all, then please accept my fullest appologies!

    - redhotsonic
    - Chilly Willy
    - FraGag
    - MainMemory
    - nesboy43
    - 87th
    - MarkeyJester
    - KingofHarts
    - Eric Wright
    - Overlord
    - sasuke
     
  2. Lilly

    Lilly

    Member
    2,419
    234
    43
    United States
    Shang Mu Architect
    It might actually be possible to cram the officially canon layout of Hidden Palace Zone in this thing. I'd love to see that, and would do it myself if I knew anything about using tools beyond replacing graphics in SonMapEd.

    Edit : Wow, I should use Preview more often.
     
  3. Varion Icaria

    Varion Icaria

    He's waiting.... Tech Member
    1,019
    11
    18
    S4: Cybernetic Outbreak
    Looking over this; you say you want to use the 32X to it's full potential, but from first glance it seems most of the game still runs in 68K memory mode with the RV bit to keep DMA from putzing up, and only executing the 32X code to Decompress Kos for level stuff before switching the RV bit back to 68K mode to keep the game 'running' normally. I don't see much of the banking actually being used here at all, except for when the RV bit is cleared for SH2 Map Access.

    This was at first glance so correct me if I'm wrong, but from the time I spent working with the 32X the RV bit usage to keep 68K mode is a pure waste because the game can't process 32X graphics or anything of the sort when the main game is running. What the hell is the point in that if all it ever does is decompress data 'slightly' faster? Hell, this would even cripple the usage of the PWM if you added it to the Sonic 2 Driver because the SH2's need ROM Access to the samples, unless you somehow waste a ton of space in SDRAM to fit just the Sonic 2 Samples. Even then it would be a waste because of how PWM would be size limited.

    From my time in porting S3K to the 32X I've changed the entire engine to stay locked to the SH2 Map so it runs accordingly, and from there in order for DMA to work effectively I only set the RV bit when I need to run the DMA to access the data running the code in ROM purely with setting up the whole method of not messing up the PC at all.

    I hope you can clarify exactly what you plan to use the 32X for, because right now it's potential is being quite wasted. I do like the regular enhancements to the original Sonic 2 Core, but if you're planning on using the 32X use it right and enable the banking appropriately and run the game in the proper mapping.
     
  4. RetroKoH

    RetroKoH

    Member
    1,662
    22
    18
    Project Sonic 8x16
    I'll continue to keep a close eye on the continued progress of this, as well as the above noted Varion project. I'd love to learn the ins and outs of all the advantages that the Sonic games can earn via 32x or CD.
     
  5. Overlord

    Overlord

    Now playable in Smash Bros Ultimate Moderator
    19,218
    965
    93
    Long-term happiness
    I saw this and o_O'd for a moment, then suddenly it clicked exactly wtf my name is in there for. Wow, you HAVE been at this a while =P
     
  6. MarkeyJester

    MarkeyJester

    Original, No substitute Resident Jester
    2,192
    406
    63
    Japan
    ^ You know what? Same here...

    Nice work there! Do you plan on expanding on this further?
     
  7. Clownacy

    Clownacy

    Tech Member
    1,053
    581
    93
    Can we expect any in depth guides from this?

    Speaking of, seeing the changelog reminded me: me and some fellas on IRC were discussing the CNZ bumpers bug, and we found the cause.

    The bumpers files contain entries that are 6 bytes in size. Two dummy entries are needed for the bumper manager to function properly: one at the beginning, and one at the end. I believe the first dummy needs an X-pos of $0000, while the last one needs $FFFF. Now, CNZ act 2 starts and ends with dummy entries, but act 1... it doesn't start with one. In fact, the jmp right before the binclude just happens to have $0000 in the same place as an entry's X-pos, so it works. This could have been a real goofy space optimisation by Sonic Team. Since the 32x version moves everything, the address that the jmp points to changes, losing the $0000, breaking the bumpers. To fix it, you just gotta shove 6 blank bytes in the beginning of the CNZ act 1 bumper layout file. Tada. But, that's the 'clean' way: as just a longword before the file, as you have done it, works too.
     
  8. Dracula

    Dracula

    Oldbie
    605
    0
    16
    I'm watching you!
    Converting NES Mappers to MMC5
    Thank you very much! Now I can use the extra buttons for other stuff.
     
  9. saxman

    saxman

    Oldbie Tech Member
    I think you misunderstand. I never claimed I wanted to use the 32X to its fullest potential. I wanted to get it working as a 32X ROM first. Then I wanted to slowly add to it. I think the way it is written meets the goals I put into place. I didn't want to be too ambitious, otherwise I would NEVER get it released! With that said, I admire your enthusiasm for doing an all-out 32X solution. And this could be leveraged to do that with the appropriate tweaking.

    The decompression scheme works as it is for now. If it causes problems for PWM down the road, the decompression routine can be scrapped. I mainly put it in to test it and make sure the Hitachi codebase was being executed correctly.


    Since sometime in 2011. I developed it for several months straight, took about a year off, started again, and then took another break. And then finally, I got around to releasing it. So, I have kept it under wraps for quite some time! I just hope it lives up to the expectations I have for it. Please, EVERYONE, start using this instead of the other disassemblies! ;)/>


    I will probably do some incremental changes to this. What I would really like to do is build a small team to sort of take over some of the responsibilities of this for me. There is much more I want to do with it, but I'm only one person. Anyone interested??


    I'm interested in knowing what kind of in-depth guides you or anyone else would like to see. I'll write anything if it'll aid people in their efforts to use my code.


    Yeah, that sounds familliar. I remember scratching my head and wondering why in the world they would do it that way. It could be an optimization, but it almost seems more like a convenient mistake than anything.
     
  10. Billy

    Billy

    RIP Oderus Urungus Member
    2,118
    178
    43
    Colorado, USA
    Indie games
    Put it on GitHub, that'd be the best way. I'd be interested, if I could just wrap my head around assembly programming.
     
  11. Cinossu

    Cinossu

    Administrator
    2,832
    44
    28
    London, UK
    Sonic the Hedgehog Extended Edition
    While this is an ambitious idea, similar to previous versions of things like this in the past, I don't think it will be used wholesale for anything more than a few projects here and there. It would be better released as a series of guides or implementations; each individual aspect of the features you have added and worked on yourself available for anyone to put into their existing hacks and allowing them to pick and choose exactly what they want.

    This was always something I disliked about "solutions" such as this; it's either an all-or-nothing thing. This way, it also allows for the incremental changes to not affect people using previous versions too much. With whole packages as this, any changes or improvements made will inevitably break everything done on it in the interim, usually leading to projects requiring a whole redo from scratch. As someone who has experience in restarting a project over and over (:v:) it really bogs you down.

    Another point, unfortunately, that comes across with complete packages such as this, is that it enables people to be lazy. With a guide, even the copy-paste-do-everything-for-me ones, it at least gets people into the code and looking around, and this small step alone is usually enough to give people the confidence to go "if I'm doing this, and this code controls this.. what happens if I tweak this instead?" With something like this, with everything done for you out of the pot, all of that is lost.

    While it's good work you've done tying all of these things together, and the level format with the breaking up of tile and collision certainly useful and impressive enough, altogether I think this isn't going to be used as much as you seem to think it will be. As a series of separate pieces, I feel it would succeed in a lot better fashion.
     
  12. Eduardo Knuckles

    Eduardo Knuckles

    Not a loved one, but the most hated person. Banned
    414
    10
    18
    Someplace somewhere
    Project S.A.M.G.
    Just a small question: Were the redhotsonic's Sonic 2 fixes (their guides themselves) applied to it?
     
  13. Clownacy

    Clownacy

    Tech Member
    1,053
    581
    93
    You know, the usual: the new bugfixes and maybe some of the features. Having it that way would greaten the chances of Git disasm users making use of your code, without actually appropriating the code for the Git disasm.

    "Replace your aging Sonic 2 disassembly with a better one!"

    "The source code is based on the 2007 disassembly"

    =P
     
  14. saxman

    saxman

    Oldbie Tech Member
    I missed part of what you said at first. The RV bit is used during the boot process. But the entire game runs in the regular 32X addressing mode. Most code runs in the 0x88000-0x8FFFF region.


    I might do that. And, assembly isn't too hard. Are you any good with a hex editor?


    I could be wrong, but I look at it a different way. By doing all these bug fixes for them up front and importing Knuckles, I think that will invite more people to use this. A vast majority of hacks will implement bug fixes and Knuckles. I did that for them, saving them a lot of time so they can focus on other things.

    Of course, a lot of this depends on how well I sell it. By the lack of activity on this forum, I can see that ROM hacking isn't quite as popular as I remember it being. So, my goal will be to try and get new people to do it. It'll be a challenge.

    But again, I could be wrong. I'm keeping a mental note of your suggestion.


    A vast majority of the bugs were done by myself. There were at least a few that redhotsonic helped with. I know the vertical wrapping of the rings was sent to me by redhotsonic, for example. I believe I also got the Aquatic Ruin Zone "walking in air" bug fix from another guide he wrote. He also made me aware of an optimization that S3K implements for the object manager which I then implemented. I don't remember if his guides were used anywhere else or not. But he was a big supporter of my efforts.


    I may. It's not high on my radar, but I understand the interest people have in that stuff. If I do, it would likely be on the SBLF implementation.

    Also, technically the SVN disassembly started from the 2007 disassembly. So it's not an unusual claim to make. I'm not going to say my stuff is better than the SVN stuff -- that would be like comparing apples and oranges. But it's definitely better than the 2007 disassembly.
     
  15. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,735
    334
    63
    SonLVL
    Importing Knuckles really isn't that hard. Speaking of which, did you fix his art loading so it's not horrendously wasteful, and add in his missing sound effects from S3K?

    I could write some code for SonLVL to handle the new level and ring formats, but obviously it wouldn't be able to do the separate graphics and collision, it would only be able to use one set of data and the other would be lost.
     
  16. saxman

    saxman

    Oldbie Tech Member
    I felt importing Knuckles was pretty challenging myself, but perhaps I did it differently. I haven't actually read the guide on porting him over. I did not import the sound effects. I'd like to. And if you mean did I change the art so it uses the correct palette right out of the box, then yes I did.

    Honestly, if you did what you propose there regarding the SBLF, then what I could do to accommodate the physical/visual separation is to simply "merge" two maps via a tool of some sort. That is, you have a copy of EHZ for the physical, and another copy for the visual. Then, run my tool and it will bind them together in a third copy. It would be a pretty easy solution.

    I'll probably start that tool sooner rather than later so I can put together some demonstrations, record them in a video, and upload the video so people can really see the advantages of the new format.
     
  17. Varion Icaria

    Varion Icaria

    He's waiting.... Tech Member
    1,019
    11
    18
    S4: Cybernetic Outbreak
    I'm still rather curious on the whole format and how it works, can you explain it better and how much more RAM it takes?
     
  18. saxman

    saxman

    Oldbie Tech Member
    If you look at Chemical Plant Zone Act 2, the whole meta-block layout takes up 1241 bytes. 4096 bytes are allocated for meta-block layout. So, CPZ2 is only using 30% of that. And obviously, because of the nature of data compression, different levels are going to compress at different ratios.

    The level format specification from my first post lays out the entire format, but I can summarize it for you:

    You have two bytes to start. One says how many rows there are for the front layout data, and the other says how many rows there are for the background layout data.

    Then you have a list of words, each one for a different row, with physical and visual each having one. So for 16 rows of visual and physical layout, you will have 32 words. Each word specifies the format of the row, as well as where the row data is located. The format is as follows:

    Code (Text):
    1. fffffppp pppppppp
    2.  
    3. f = format
    4. p = pointer
    The pointer is multiplied by 2, so data always has to start at even addresses.

    Now, there are 7 formats. They are as follows:

    Code (Text):
    1. 0x00 (00000) = AST: Always specified type
    2. 0x01 (00001) = BALE: Bitwise-AND at line-end
    3. 0x02 (00010) = WOLE: Wrap once at line-end
    4. 0x03 (00011) = ZLE: Zero at line-end
    5. 0x10 (10000) = (reserved -- acts as AST)
    6. 0x11 (10001) = BALE4: Bitwise-AND at line-end 4-bit
    7. 0x12 (10010) = WOLE4: Wrap once at line-end 4-bit
    8. 0x13 (10011) = ZLE4: Zero at line-end 4-bit
    AST is special, because it uses the pointer value to determine which meta-block to use. It will fill up the entire row with that value. It works great for levels like EHZ where the very top of the level uses 0 for every meta-block position.

    BALE data starts with a byte to say how long the data is. 1 is always added to that value. Then the list of meta-blocks starts. At the end of the data, Sonic Boom it will do a bitwise AND operation and repeat the meta-block data endlessly. This format works well with backgrounds.

    WOLE data is formatted just like BALE is. The difference is Sonic Boom will wrap this data once (meaning, it does NOT do a modulus operation, but rather does simple subtraction to repeat the data when needed). This is useful when the data is more than half the length of the level and needs repeating, even if only for a few meta-blocks at the very end.

    ZLE data is formatted like BALE and WOLE, but Sonic Boom uses 0 for every meta-block position after the data ends.

    4-bit formats start with a byte to specify how many nybbles will be used. Then is uses a series of nybbles (even numbered of course) to specify meta-block references. Then a list of meta-blocks for each reference is specified after the nybbles. So for example, 04 01 23 A0 B0 C0 D0 means that there are 4 nybbles, and nybbles 0 = A0, 1 = B0, 2 = C0, and 3 = D0. It cuts normal 8-bit data down to almost half the size. These formats are useful when 16 or fewer different meta-blocks are used in a row.


    As for ring layout, it uses pretty much the same general format as Sonic 2, but now all 32 bits of the positions are used with some bits being moved. Doing this allows rings to be placed anywhere in the 0x8000 x 0x2000 area supported by the engine. Here's the format used for each position:

    Code (Text):
    1. The 'xPos' word is broken up into this format:
    2.  
    3. oxxxxxxx xxxxxxxx
    4.  
    5. o = orientation
    6. x = horizontal position
    7.  
    8. The orientation determines if this is a row or column of rings.
    9.  
    10. The following format is used for the 'yPos' word:
    11.  
    12. lllyyyyy yyyyyyyy
    13.  
    14. l = length
    15. y = vertical position
    16.  
    17. The length determines the number of rings between 1 and 8.
     
  19. Varion Icaria

    Varion Icaria

    He's waiting.... Tech Member
    1,019
    11
    18
    S4: Cybernetic Outbreak
    That's really interesting but seems tedious to work with, without proper tools
     
  20. DigitalDuck

    DigitalDuck

    Arriving four years late. Member
    5,338
    412
    63
    Lincs, UK
    TurBoa, S1RL
    Fuck the haters.

    This would be perfect for one of the things I'd like to do with Sonic 2, and it seems to be just me because I'm finding Sonic 1 Two-Eight more useful than the original too...