Custom AS builds

Discussion in 'Engineering & Reverse Engineering' started by flamewing, Jun 27, 2021.

  1. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    So, I decided to fork and maintain AS. The project is here, with automated snapshots generated for Windows (32- and 64-bit builds), Linux (32- and 64-bit builds for Ubuntu), and OSX (64-bit only) daily if there was activity. I updated the disassemblies to match. If you know any annoying bugs from AS, please submit a bug report on the repository and I will see if I can fix it.

    Currently, it is tracking the latest beta version released by the original author (1.4.2 build 204), and it has a few patches of my own.

    Link for automated releases.
     
  2. MarkeyJester

    MarkeyJester

    A D V A N C E Resident Jester
    2,080
    211
    43
    Japan
    Ahhh nice! So we have more or less a developing assembler within the community (or at least as close as), excellent~

    I've never been a fan of AS, it's always felt as this majorly clunky and unnecessarily messy thing, but over the years I have been compelled to have to use it against my better judgement and simply grit my teeth. However, there is this one bug that infuriates me every single time with the current iteration, I'm not sure if this is resolved as part of your patches (or if you're even aware of it), but I've had it occur multiple times (on multiple of our disassemblies for example), and others I've worked with have also experienced this issue and can vouch for me.

    The rare and illogical nature of the bug make it considerably difficult to explain, so I don't want to just throw it on the bug report on the repository, I'd rather explain here in-case someone else might know, but the gist is, it'll throw errors... which aren't errors... particularly with 68k PC relative indexing ranges where it is definitely 100% certain the relative index reaches. In these cases the only solution we found was to intentionally insert junk code somewhere earlier on, and hope that it tricks AS into slipping past whatever internal mistake it keeps coming to. I'm not sure if this issue occurs with other CPU types, I've not noticed it with Z80 or x86.

    ...unfortunately, I don't have an example right now where it's happening, and it comes and goes as you add/remove large portions of assembly code. It ALWAYS seems to hit me when I'm really deep in thought and don't need distractions. If you have a fix I would utterly grateful!
     
  3. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    I am not sure if I understood what the issue is; I think I do, but I am not sure. I also never encountered it, so I can't say for sure if it has been fixed by the original author (the versions we had on the disassembly were seriously out of date); but my patches do not fix it, that is for sure.

    In any event, with someone in the community taking charge of fixing this, I am hoping that whoever comes into this (or any other) issue next will remember and make a bug report.
     
  4. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    @MarkeyJester I was thinking a bit more about the issue you describe, and a lightbulb went on; and I may or may not be onto something, but it is worth checking out anyway.

    Many people that mentioned this kind of issue mentioned the S3&K disassembly, which uses named temporary symbols (labels starting with $$). I myself don't use them because of several issues I half remember from years ago when I tried them; and maybe that is why I don't remember having encountered the issue you were describing.

    So do you recall if these kinds of labels were in the mix when you encountered the issue?
     
  5. MarkeyJester

    MarkeyJester

    A D V A N C E Resident Jester
    2,080
    211
    43
    Japan
    No $$ symbols were used in any of the projects as far as I know (not the only person working on these projects mind), but temporary labels such as those prefixed with a '.' (dot) have been used extensively.

    I would like to point out (which I assume is obvious to you as you haven't questioned it), no, it's not an issue somewhere else, like we've made an error in one place and it's shown a totally separate error, this is 100% no errors at all could be possible, and it throws it:

    [​IMG]

    This is a screenshot of someone else getting this very issue (not gonna share their name), but nothing was wrong, the indexing was more than close enough, and there were no other issues at all which would mislead AS. It says addressing mode not allowed, but as you well know; that's the usual (misnomer) message it throws when the indexing is too far.

    Myself (and probably others) really appreciate that you're at least *looking* into this, even if you don't find a solution, it is still extremely grateful :)

    EDIT: what I'll do, is if this issue occurs again, I'll see about leaving the code exactly as it is, and replacing the assembler with the most up to date version, and see if it fixes it, and then I'll monitor it over the months to see if the issue crops up again. For all we know it might already be fixed, I recall you mentioned the version supplied with the disassemblies are extremely out of date. I'll keep you posted, but I cannot promise when as the bug is unpredictable, we might not get it again for a long time....
     
    Last edited: Jun 28, 2021
  6. Brainulator

    Brainulator

    Regular garden-variety member Member
    I think in terms of things to fix, I wonder if this thread could be of any help (OP notwithstanding).
     
    • Agree Agree x 1
    • Useful Useful x 1
    • List
  7. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    @MarkeyJester : For what is worth (and this might be considered a heads up warning), upgrading the assembler has three main pain points due to a few internal changes in AS that are not backwards-compatible:

    1. moveq no longer silently sign-extends; it gives an error if you do something like "moveq #$80,d0". In the S2 Git disassembly, this was dealt with in this commit. I can roll back this change, but I am not sure I want to because it can prevent silly mistakes and forces people to be more clear about what they want.
    2. phase/dephase were changed to be a stack, and you need a matching dephase for every phase. In S2 Git disassembly, this was dealt with in this commit. This is more greyinsh and while I see the argument for it, I could live rolling back the change. But it is easy enough to fix on the disassembly, so meh.
    3. Adding a character constant to an integer results in a character constant, which can overflow the limits of a character and give an error. This was dealt with in this commit in the S2 Git disassembly. This change allows for manual upper/lowercasing strings in AS, but seems like a heavy price. This was the only instance in S2 Git; I could easily revert to having an integer + character spit out an integer.
    @Brainulator : That is an interesting thread. The expression parsing was something I had noted and was thinking about how to fix before (and I started doing so yesterday, in fact).

    The issue with phase is one I raised with the original author years ago; he said it would be too hard to fix. I will be taking a look at this and see if I can get it to work at least for the processors we care about (68k, z80, SH2).

    I will see about contacting Aurora Fields and FireRat and see if they can reproduce the addressing issue easily so I can try fixing it.
     
  8. biggestsonicfan

    biggestsonicfan

    Model2wannaB Tech Member
    1,324
    223
    43
    Formerly Sonic the Fighters
    ArE yOu KiDdInG mE?!?!! This is FANTASTIC news!

    EDIT: I don't particularly wish to "install" this, is there a way to make it portable? Like how GNU binutils has "tooldir=path"? Or do I just edit my def file (I am assuming I am Makefile.def-x86_64-unknown-linux) "*DIR" paths to something local like ./bin, ./include/asl, etc...
     
    Last edited: Jun 29, 2021
  9. Brainulator

    Brainulator

    Regular garden-variety member Member
    Thanks. I was wondering whether it would be better to have our disassemblies strip down AS to only support those three processors (or two, depending on whether SH2 support is needed for raw non-32X programs). I'm not sure of the specifics of how AS works and what makes AS slower than ASM68K, but if it is indeed having to worry about building for different processors with different instructions...

    As for contacting those two, I'm not sure I would bother with the latter. It's a long story and one I have no experience or interest in, but I'm pretty sure you'll get a lot of people saying it's dangerous to associate with him.
     
  10. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    You are correct, the current build system requires editing the def file to suit your needs. After I have fixed a few issues I will probably swap to a sane build system, maybe something like cmake.
     
    Last edited: Jun 29, 2021
  11. biggestsonicfan

    biggestsonicfan

    Model2wannaB Tech Member
    1,324
    223
    43
    Formerly Sonic the Fighters
    Alright, now I guess my next question is how to use it? How do I specify the processor, the processor variables, etc. The repo isn't very upfront and trying to parse the "doc_EN" folder before I build it isn't very easy.
     
  12. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    For the moment, the only real source of documentation is the original author's website: specifically, this page.

    I still have to figure out how to build the documentation myself (I still haven't found out all packages I need), in particular because I have already found a coulpe mistakes in it and also because I want to document my changes. Or maybe I should convert it all to markdown and just discard all the other options. I am not sure yet.

    On another front: changes I have made so far:

    Fixed effective addresses on 68k family to allow arbitrary expressions on most places, as well as not choking or dying on whitespace in the middle. The following are now all accepted:
    Code (Motorola 68000 Assembler):
    1.     move.l    d3 , d7
    2.     move.l    a3 , d7
    3.     move.l    ( a3 ) , d7
    4.     move.l    ( a3 ) + , d7
    5.     move.l    - ( a3 ) , d7
    6.     move.l    mul ( 20 , 50 ) ( a3 ) , d7        ; "68000 style"
    7.     move.l    ( mul ( 20 , 50 ) , a3 ) , d7        ; "68020 style"
    8.     move.l    mul ( 10 , 12 ) ( a3 , a4.w ) , d7        ; "68000 style"
    9.     move.l    ( mul ( 10 , 12 ) , a3 , a4.w ) , d7    ; "68020 style"
    10.     move.l    mul ( 10 , 12 ) ( a3 , a4.l ) , d7        ; "68000 style"
    11.     move.l    ( mul ( 10 , 12 ) , a3 , a4.l ) , d7    ; "68020 style"
    12.     move.l    mul ( 10 , 12 ) ( a3 , d4.w ) , d7        ; "68000 style"
    13.     move.l    ( mul ( 10 , 12 ) , a3 , d4.w ) , d7    ; "68020 style"
    14.     move.l    mul ( 10 , 12 ) ( a3 , d4.l ) , d7        ; "68000 style"
    15.     move.l    ( mul ( 10 , 12 ) , a3 , d4.l ) , d7    ; "68020 style"
    16.     move.l    10000 , d7        ; "68000 style"
    17.     ; move.l    mul ( 100 , 100 ) , d7        ; "68000 style"
    18.     move.l    mul ( 100 , 100 ) .l , d7        ; "68000 style"
    19.     ; move.l    mul ( 200 , 500 ) , d7        ; "68000 style"
    20.     move.l    100000 , d7        ; "68000 style"
    21.     move.l    ( mul ( 100 , 100 ) ) , d7        ; "68020 style"
    22.     move.l    ( mul ( 100 , 100 ) .l ) , d7        ; "68020 style"
    23.     move.l    ( mul ( 200 , 500 ) ) , d7        ; "68020 style"
    24.     move.l    * ( pc , a4.w ) , d7        ; "68000 style"
    25.     move.l    ( * , pc , a4.w ) , d7        ; "68020 style"
    26.     move.l    * ( pc , a4.l ) , d7        ; "68000 style"
    27.     move.l    ( * , pc , a4.l ) , d7        ; "68020 style"
    28.     move.l    * ( pc , d4.w ) , d7        ; "68000 style"
    29.     move.l    ( * , pc , d4.w ) , d7        ; "68020 style"
    30.     move.l    * ( pc , d4.l ) , d7        ; "68000 style"
    31.     move.l    ( * , pc , d4.l ) , d7        ; "68020 style"
    32.     move.l    # $aa554711 , d7


    ifdef/ifndef work with macros and functions.

    Added a couple common aliases to z80 undocumented instructions which AS did not accept. Specifically, the undocumented slia/sls/sll/sl1 instruction now accepts all of those forms.

    When replacing integers into symbol names, they are now turned to strings instead of requiring quoting. So instead of this:
    Code (Motorola 68000 Assembler):
    1.     cntstr equ "\{cnt}"
    2.     var{cntstr} equ    cnt
    or this:
    Code (Motorola 68000 Assembler):
    1.    var{"\{cnt}"} equ    cnt
    you can just write this:
    Code (Motorola 68000 Assembler):
    1.    var{cnt} equ    cnt

    Added some common aliases to macro stuff:
    • elsec as an alternate of elseif
    • endc as an alternate of endif
    • endr as an alternate to endm for rept, irp, irpc, and while

    Fixed phase, so you now don't have to do stupid stuff to assemble z80 code in ROM. You still need to do some stuff to compress the sound driver, and I will update the S2 and S&K disassemblies with an improved system.

    Made a few pseudo instructions (specifically, reg and label) support the alternate syntax that equ(=) and set (:=) also support:
    Code (Motorola 68000 Assembler):
    1.     label mylabel,*
    2.     reg myreg,d0
     
  13. biggestsonicfan

    biggestsonicfan

    Model2wannaB Tech Member
    1,324
    223
    43
    Formerly Sonic the Fighters
    Ah so I was parsing the information correctly, it's still non-descriptive and vague about how I would accomplish assembling for an 80960. I will just have to run through the source code to see if I can assemble a KB variant processor through it. It mentions the SB manual, so it should handle KB just fine.
     
  14. Techokami

    Techokami

    For use only on NTSC Genesis systems Researcher
    1,349
    42
    28
    HoleNet!
    Sonic Worlds Next
    This would be interesting and promising... IF it can be a perfect drop-in replacement for ASM68K. That is, no code editing required, and I can continue to utilize the ASM68K conventions.

    Either that or start commenting the heck out of code that uses nameless temporary symbols. [​IMG]
     
  15. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    There are things in AS which would require a complete rewrite of significant portions of the code to allow doing some things like asm68k does because they are different on a fundamental level; like macro expansion. In AS, macro parameters are replaced by special tokens in the unparsed macro body when it is read, and the macro is parsed and has the tokens replaced by the respective values when the macro is used -- that is, it works kind of like the C preprocessor. So while in asm68k you can process variadic macro arguments with rept and shift, you cannot in AS because the parameters will already have been substituted in when the rept is processed. There are other ways, though: you can either use recursive macros, or use the irp/irpn directives (the latter of which I added) to do the same.

    I actually agree that the nameless temporaries are generally bad. But 90% of the time, they would be just replaced by something like ".loop" or ".return" (or maybe with @ or $$ instead of the .) and the real explanation would not be in the label but in the comments nearby.
     
    Last edited: Jul 18, 2021
  16. AURORA☆FIELDS

    AURORA☆FIELDS

    The cute one here Tech Member
    216
    21
    18
    Finland
    AMPS
    True, however the big issue with nameless temporary labels is, if they are not used sparingly, they are easy to confuse with each other or break existing code by adding or removing them. It doesn't help that disassemblies and individual programmers seem to litter them everywhere possible. They also work in some unexpected ways with relation to scopes and temporary labels. Although using temporary labels may not at all aid in documentation, it provides the ability for the assembler to let you know when you have incorrectly edited the code no longer to be intended, in many many ways. nameless temporaries don't afford this possibility. For the confusing, error prone and unhelpful nature of nameless temporary labels, I deem them a counter-productive thrash feature. It promotes writing unmaintainable lazy code, poor documentation and difficult to follow logic. Although temporary labels have a similar issue, its far less prominent. I have never found nameless temporary labels have any upside that doesn't come down to laziness.
     
    Last edited: Jul 19, 2021
  17. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    Hear, hear. I have been slowly getting rid of them in SCH, and BFD has them completely eliminated because of all the points you make. I think that the biggest issue with nameless temporaries is that you can use things like --, ---, ++, or +++. If only + and - were ever used (single jump forward or backward), they would be much more tolerable. I guess that the S2 disassembly could be edited to remove these cases, and leave the single + and - cases for an enterprising soul (because there are MANY of them).

    That, at least, is one thing I fixed: in my builds, temporary ($$) and composite (.) labels completely ignore nameless temporaries and each other.
     
  18. flamewing

    flamewing

    Emerald Hunter Tech Member
    1,153
    37
    28
    France
    Sonic Classic Heroes; Sonic 2 Special Stage Editor; Sonic 3&K Heroes (on hold)
    I have been investigating the issue @MarkeyJester reported; and I have good news. @AURORA☆FIELDS managed to reproduce it, and I have determined that the issue was apparently fixed in 1.42 build 139, with improvements in build 142. The issue was in the way that AS was forward symbols in the first pass: before build 139, values of some forward symbols could try to be checked before their values were settled and end up being considered to be out of range because a -1 was returned by the evaluation function, which becomes out of range.

    I still want to try out more cases before declaring the bug fixed, but I am optimistic.
     
  19. biggestsonicfan

    biggestsonicfan

    Model2wannaB Tech Member
    1,324
    223
    43
    Formerly Sonic the Fighters
    Alright so I've built the binaries and used a custom directory. Incidentally, "make install" does not need to be run in that case. However, I assume not "installing" the compiled software causes problems. For instance, trying to run asl gives me the following error: "message catalog handling: cannot open msg file as.msg - program terminated". "as.msg" relative to the "asl" binary is "../lib/asl/", but it's obviously not seeing that.