don't click here

OSX support in S3&K dissassembly

Discussion in 'Engineering & Reverse Engineering' started by Orangy Tang, Jul 10, 2011.

  1. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    I'm trying to get a S3&K hacking environment setup on my macbook, so as a first step I've pulled down the SVN dissassembly and been tweaking it to build in OSX.

    I can now build what (appears to be) a working S3&K rom. I need to do some more testing to make sure it's byte-accurate, but a quick play through of Angel Island showed everything looking normal. Changes I had to make:

    - Fix and build an AS binary for OSX. Mostly resolving it's wonky strcpy replacement.
    - Builld an s3p2bin binary for OSX. Worked without changes, so kudos to whoever wrote that in the first place.
    - Knock up a replacement build script (works, but needs refinement).
    - Alter the .asm files to use forward slashes in file paths rather than backwards slashes.

    So, questions:
    1. Are people interested in having this added to the community SVN? What's the best way of going about this? Assuming I've got commit permissions I could add it in a branch and someone could give it a sanity check before moving it into trunk.

    2. Is it worth having my modified source for AS included in SVN as well? Not sure if this is helpful or not myself.

    3. While I'm making changes I'd like to rearrange the build scripts so they're separate from the source files, and have output files in a separate dir as well. Has anyone got any thoughts on this?

    4. Is anyone else hacking on OSX? Just wondering what kind of setup other people have got.

    Thanks.
     
  2. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,161
    65
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    I would definitely like to see it, especially what you have done for AS. While I am on Linux, it may be instructive to see how you managed to get a native build of AS that does not missassemble the input...
     
  3. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    I tried to make a branch for these changes, but I'm getting the same handshake error you were getting ( <a href="http://forums.sonicretro.org/index.php?showtopic=25094" target="_blank">http://forums.sonicretro.org/index.php?showtopic=25094</a> ), so I guess I can't do that until someone resolves that.

    Instead I've uploaded the changes here if you want to give it a try: <a href="http://www.triangularpixels.com/Junk/RomHacking/" target="_blank">http://www.triangularpixels.com/Junk/RomHacking/</a> (the build script is actually python, but I renamed it .txt to stop my webserver trying to execute the damn thing).

    The as-current.zip is basically the latest C code (the 'beta' version from here: <a href="http://john.ccac.rwth-aachen.de:8000/as/download.html" target="_blank">http://john.ccac.rwth-aachen.de:8000/as/download.html</a> ) with the custom strcpy replaced so it would build. And also a small fix in the parsing code where it was incorrectly using strcpy with overlapped strings.

    What misassembled output were you seeing? I didn't do anything particularly special but it seems to work. I guess the parsing bug might have been the problem - it really looks like the author doesn't really understand C strings, and frankly I'm suspicious of anyone's code who likes to drag Pascal's BEGIN/END into C via macro defines. >_<

    Give it a go and let me know how it goes. It looks like there's preprocessor defines for a couple of vanilla linux setups, so with any luck it should just build.
     
  4. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,649
    214
    43
    SonLVL
    I'm not getting any SSL errors here.
     
  5. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    Ah I figured it out - the url in the SVN post is wrong, it says <a href="https://sonicretro.org/asm_svn" target="_blank">https://sonicretro.org/asm_svn</a> when it should be <a href="https://svn.sonicretro.org/asm_svn" target="_blank">https://svn.sonicretro.org/asm_svn</a>.
     
  6. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,649
    214
    43
    SonLVL
    <a href="https://sonicretro.org/asm_svn" target="_blank">https://sonicretro.org/asm_svn</a> works fine for me...
     
  7. sasuke

    sasuke

    Member
    66
    0
    0
    <!--quoteo(post=604163:date=Jul 10 2011, 02:28 PM:name=Orangy Tang)--><div class='quotetop'>QUOTE (Orangy Tang @ Jul 10 2011, 02:28 PM) <a href="index.php?act=findpost&pid=604163">[​IMG]</a></div><div class='quotemain'><!--quotec-->- Fix and build an AS binary for OSX. Mostly resolving it's wonky strcpy replacement.<!--QuoteEnd--></div><!--QuoteEEnd-->

    Thanks very much. I actually attempted to compile AS a couple weeks ago, but, while it compiled, it kept giving me ENOTTY errors when I ran it.
     
  8. FraGag

    FraGag

    Tech Member
    <!--quoteo(post=604163:date=Jul 10 2011, 02:28 PM:name=Orangy Tang)--><div class='quotetop'>QUOTE (Orangy Tang @ Jul 10 2011, 02:28 PM) <a href="index.php?act=findpost&pid=604163">[​IMG]</a></div><div class='quotemain'><!--quotec-->1. Are people interested in having this added to the community SVN? What's the best way of going about this? Assuming I've got commit permissions I could add it in a branch and someone could give it a sanity check before moving it into trunk.<!--QuoteEnd--></div><!--QuoteEEnd-->
    Sure, go ahead.

    <!--quoteo(post=604163:date=Jul 10 2011, 02:28 PM:name=Orangy Tang)--><div class='quotetop'>QUOTE (Orangy Tang @ Jul 10 2011, 02:28 PM) <a href="index.php?act=findpost&pid=604163">[​IMG]</a></div><div class='quotemain'><!--quotec-->2. Is it worth having my modified source for AS included in SVN as well? Not sure if this is helpful or not myself.<!--QuoteEnd--></div><!--QuoteEEnd-->
    If AS's author is still maintaining AS, it might be better for everyone to send him your changes directly. If AS isn't being actively maintained anymore, then just releasing an archive with the updated sources and binaries should be enough. Since I'm working on my own assembler, I tend to consider updating the other assemblers we use a low priority. :P

    <!--quoteo(post=604163:date=Jul 10 2011, 02:28 PM:name=Orangy Tang)--><div class='quotetop'>QUOTE (Orangy Tang @ Jul 10 2011, 02:28 PM) <a href="index.php?act=findpost&pid=604163">[​IMG]</a></div><div class='quotemain'><!--quotec-->3. While I'm making changes I'd like to rearrange the build scripts so they're separate from the source files, and have output files in a separate dir as well. Has anyone got any thoughts on this?<!--QuoteEnd--></div><!--QuoteEEnd-->
    Considering the small number of files the build scripts and the output files represent, I think it's not necessary to do this. I have recently adopted out-of-source builds in my C and C++ projects and it's useful when you have one .o/.obj file for each .c/.cpp file, but in our disassemblies, we have only one output file (though AS generates other intermediate files that we don't use). Furthermore, most of the people who use the disassemblies aren't programmers, so giving them a simple build system makes it easier for them. Those who prefer to work with a more complex build system usually have the skills to make it themselves.

    <!--quoteo(post=604163:date=Jul 10 2011, 02:28 PM:name=Orangy Tang)--><div class='quotetop'>QUOTE (Orangy Tang @ Jul 10 2011, 02:28 PM) <a href="index.php?act=findpost&pid=604163">[​IMG]</a></div><div class='quotemain'><!--quotec-->4. Is anyone else hacking on OSX? Just wondering what kind of setup other people have got.<!--QuoteEnd--></div><!--QuoteEEnd-->
    I can't comment on this, but I can say that making an easy and consistent hacking experience across operating systems is one of the reasons I started working on my assembler.
     
  9. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    I've made a branch of S&K and checked my changes into it now, you can find it here: http://svn.sonicretro.org/listing.php?repname=CommunityDisassemblies&path=%2Fbranches%2FS%26K+OSX+port%2F&rev=408&peg=408#ab269d77d23954d99fa1eddfd7fcdfbde

    This contains Osx binaries for the build tools and new python build scripts. Here's the bit from the README about building now:

    Code (Text):
    1.  
    2. The 'BuildScripts' directory contains two sets of build scripts - the windows batch files (*.bat), and the python scripts (*.py).
    3. The .bat ones are the simplest to use but will only work on windows. The python scripts are cross platform (windows and mac) but
    4. require python to be installed.
    5.  
    6. Batch files: (double click to run)
    7. buildSK.bat          - builds a byte-perfect copy of the Sonic And Knuckles rom in the base dir.
    8. buildS3Complete.bat  - builds a combined Sonic 3 And Knuckles rom (using the SONIC3_COMPLETE macro detailed above).
    9.  
    10. Python files: (run with 'python nameOfScript')
    11. buildSK.py             - builds a byte-perfect copy of the Sonic And Knuckles rom in the base dir. Output and errors go in the 'Builds' sub dir.
    12. buildS3Complete.py     - builds a combined Sonic 3 And Knuckles rom (using the SONIC3_COMPLETE macro detailed above). Output and errors go in the 'Builds' sub dir.
    13. buildAndVerify.py      - builds both a S&K rom and a S3 Complete rom, then verifies that the S&K rom is byte identical to the original rom. Output and errors go in the 'Builds' dir
    In theory the new build scripts should work in Linux too, but you'll have to compile a version of AS and s3p2bin yourself.

    Unless there's any objections I'll promote this into the main branch in a few days.
     
  10. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,161
    65
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    Can you generate a diff from it? Or send me the modified code?
     
  11. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    The modified code is already uploaded here: http://www.triangularpixels.com/Junk/RomHacking/asl-current.zip

    Probably easier if you download that and do a local compare, there's lots of files with one line changes so a diff file would be pretty huge. It looks like it's got platform defs for a couple of flavours of linux, so with a bit of luck you'll just have to run make.
     
  12. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    1
    0
    Writing my own MD/Genesis sound driver :D
    My Mac is still stuck on an update so I can't check right now, but: does this build system generate bit-perfect ROMs?
     
  13. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,649
    214
    43
    SonLVL
    You made a commit with the description "Orangy Tang - S&K - merged OSX port changes into trunk", but all it did was revert my edit to "/Mega Drive Disassemblies/Sonic & Knuckles Split Disassembly/General/Sprites/Level Misc/Map - Path Swap.asm"...

    Edit: And now you've reverted all my other edits.
     
  14. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    Oh balls. Sorry about that. I'm using a different SVN client from what I'm used to and it handles merges oddly.

    I'll have a look at putting your changes back now.

    Andlabs: Yes.

    Edit: Oh I see you've already put your changes back. Sorry about that, hope it wasn't too much trouble. >_<
     
  15. MainMemory

    MainMemory

    Kate the Wolf Tech Member
    4,649
    214
    43
    SonLVL
    Also for some reason you commented out the line that defines Sonic3_Complete.
     
  16. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    That's deliberate - it's done in the build scripts now. There's separate scripts for S3Complete and S&K. The advantage being that the buildAndVerify script can build both off the same source and check they both build (and S&K is byte accurate). Theres a more complete description in the readme (or a few posts up).

    Would have been nice if you've have read the posts or asked me about this first, because by reverting it you broke the build scripts. :( I've fixed it up again now.
     
  17. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,161
    65
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    Did you notice that you have completely reverted my extensive modification of the sound driver? Damn it, man, next time update from SVN before you merge changes!

    On other news: I now remember hos AS was misassembling: it is because of its truly boneheaded handling of strings. In the S2 disassembly, several macros are used that automatically include/binclude parameter files. If AS is not in case-sensitive mode, it converts the parameters to uppercase and can't find the files in a case-sensitive file system API. If AS is in case sensitive mode, it can't correctly handle the macros anyway and fails spectacularly. As far as I am concerned, I will keep using AS through Wine until FraGag's assembler is finished, at which point I will ditch AS completely.
     
  18. Orangy Tang

    Orangy Tang

    Member
    15
    0
    0
    Damn. I'm very sorry about that - I *did* update before merging, honest. I'm gonna ditch this SVN client and find a different one I think. Although I don't know how it happened since I branched *after* those changes were checked in. Oh well. Again, sorry about that.

    That sounds like a nasty bug. Do you know of any specific cases where it fails in case-senstitive mode? Presumably this isn't an issue with the S3k disassembly since it's building fine, but it would be good to know/avoid.
     
  19. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,161
    65
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    The S&K disassembly is safe for now because it doesn't use macros to include/binclude files -- the moment it does, it will start to fail to build as described. Examples of the problem abound in the Sonic 2 disassembly: search for 'palette macro' and look at the table below it, for example. Zone ordered tables (done using zoneOrderedTable) with zoneTableBinEntry entries also fail. And so on.
     
  20. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,161
    65
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    Recently, I became unable to build due to changes in Wine again. When I went to do a regression testing, I found out that Canonical is rolling out a new method of distributing dependency libraries (multiarch) which basically prevented* me from building Wine (since multiarch is buggy and prevents installation dev packages for amd64 and x86 at the same time), which prevented* me from doing regression testing. So I decided to try this again, and tried harder to figure out what was causing assembly to fail.

    I have now built S2 successfully on Linux with native tools only. I have fixed SVNHg to build correctly. Here is how to build it:
    1. Install unzip and g++ through your favorite package manager.
    2. Get the asl source code, compile and install. I recommend this version I made based on Tang's code, applied to the latest AS source code.
    3. When you want to build your hack, run the build.sh script at the root of the disassembly. This script will do the following:
      1. If needed, it extracts the source files from build_source.zip and compiles all binaries needed;
      2. everything else done by the bat file, but natively.
    4. Profit.
    * Unless, of course, I want to install a virtual machine with 32-bit Ubuntu just to make regression testing...