[Sik]

Eh, I needed a place to store all my MD development tools so I decided to open a GitHub repository to store them (in before complaints about how I'm not using it properly and such). So whatever, here we go:
https://github.com/sikthehedgehog/mdtools

Currently stored there:

• SLZ (general purpose compression)
• UFTC (fast graphics compression)

I don't think anybody here cares, but whatever =P

[Bgvanbur]

How do these compare in terms of size and speed to the other compression methods used by MD games? And what is the limit of the uncompressed data for SLZ24 (is it ~16777216)?

[Sik]

SLZ24 has a limit of ~16MB (16,777,215 bytes, more exactly), yes, but there isn't any chunk of contiguous RAM that large in the MD =P And SLZ... is your usual decompression routine I think, in terms of speed. Anyways, average compressed size is 32.8% of original size if I recall correctly.

The title screen in Project MD takes up 3 frames to load. That includes 8KB that needs to be decompressed, and that's using an outdated code that was quite slower, and that's considering it was doing other stuff, including loading all that into VRAM and such, not to mention any unused CPU time in the last frame... Er, I guess you'll have to try, just use SLZ for load-time stuff, not in-game.

UFTC is the polar opposite, you're meant to use it in-game, since it's for compressing sprites that may be streamed. The assembly subroutine takes up ~3.81% of a NTSC frame to decompress 16 tiles. The compression ratio isn't that great but it's something. Quoting from Sega-16:

Also just now I bothered to do a stress test of UFTC, I had forgotten to do it >_> Tested against Stephany sprites in Project MD, which seems like a good candidate for this thing:

• Uncompressed size: 44,576 bytes (100%)
• Compressed size: 19,482 bytes (43.7%)

Well, it isn't much, but it's still some important saving =| This also goes to prove that UFTC works well when used in the domain it was designed for.

This post has been edited by Sik: 05 July 2011 - 09:44 AM

[Bgvanbur]

On my current project my sprites were getting to 26496 bytes (and I only have a few small ones implemented). Using UFTC, I was able to compress it to 9250 bytes (35% of the original size). Since I wanted to decompress from word RAM straight to the VRAM, it was very easy to modify the 68k routine to do this.

Thanks for UFTC!

Possible suggestions that aren't needed but may help:

1) I had to changed @Loop to DecompressUftcLoop and @LoopEnd to DecompressUftcLoopEnd so asmx can also assemble it.

2) I have all my sprites as separate files, so I had to make a file that combined these files into one so that UFTC could make one compressed data set. If "uftc -c" could handle multiple input files these would make this easier.

[Sik]

Bgvanbur, on 18 February 2012 - 07:06 PM, said:

1) I had to changed @Loop to DecompressUftcLoop and @LoopEnd to DecompressUftcLoopEnd so asmx can also assemble it.

@ is the local label symbol in asm68k, in some assemblers it's different (and in some cases they outright don't support named local variables, ugh). Though I swear asmx supported using @ as the local label symbol...

There is nothing I can do about it without cluttering the global namespace (which is what you did). Each assembler seems to have its own flavor of the 68000 syntax and it's pretty much impossible to write anything even remotely interesting that works in all assemblers, short of getting involved into bad programming practices (also don't even attempt to use GCC's assembler, it outright will not work without rewriting the entire code, even in MRI-compatibility mode).

Do you know how does the C function fare in comparison to the assembly version? (you will want to build it as -O3)

Bgvanbur, on 18 February 2012 - 07:06 PM, said:

2) I have all my sprites as separate files, so I had to make a file that combined these files into one so that UFTC could make one compressed data set. If "uftc -c" could handle multiple input files these would make this easier.

Eh, lazy and technically I should make it use an -o option for the output file, just like about every single *nix program.

I guess I don't care much about this since my graphics conversion tool is designed to output multiple sprites into a single file. Since I already need to specify the mappings explicitly, it made sense to provide it the ability to group multiple sprites into a single file (and vice versa: make multiple files out of a PNG file).

I seriously need to remove one (deprecated by now) dependency on it so I can upload it to the mdtools repo.

[Bgvanbur]

Sik, on 18 February 2012 - 08:02 PM, said:

There is nothing I can do about it without cluttering the global namespace (which is what you did).

Polluting the namespace doesn't bother me, I would rather have my assembly code work on more assemblers since assembling takes next to no time anyways.

Sik, on 18 February 2012 - 08:02 PM, said:

Each assembler seems to have its own flavor of the 68000 syntax and it's pretty much impossible to write anything even remotely interesting that works in all assemblers, short of getting involved into bad programming practices (also don't even attempt to use GCC's assembler, it outright will not work without rewriting the entire code, even in MRI-compatibility mode).

I have a script that will assemble a file on asmx, asm68k, and snasm68k and verify each version produces an indentical binary file. Some of the things I have noticed are to make addresses on or after 0xFF8000 to use 0xFFFF8000 as a base (since all the assemblers don't know that we are using a 24 bit address space). Another thing is to use immediate version for arithmetic instructions when using an immediate (SUBI #... instead of SUB #...) since some assemblers convert these into the immediate version. Another one is to avoid 0(ax) format since some assemblers convert it to (ax) and others don't.

I also have a version of asmx that adds warnings for these types of things but I can't release it since there is not stated license and the author of asmx did not respond to my email (and we don't need yet another assembler available anyways).

Sik, on 18 February 2012 - 08:02 PM, said:

Do you know how does the C function fare in comparison to the assembly version? (you will want to build it as -O3)

I don't have a C toolchain setup for the 68k so I will stick to the assembly version.

[Sik]

Bgvanbur, on 19 February 2012 - 08:30 AM, said:

I have a script that will assemble a file on asmx, asm68k, and snasm68k and verify each version produces an indentical binary file. Some of the things I have noticed are to make addresses on or after 0xFF8000 to use 0xFFFF8000 as a base (since all the assemblers don't know that we are using a 24 bit address space).

Don't forget AS then, which is waaaaaay more popular than all those assemblers these days. In particular, if I recall correctly, you actually need to use 0xFFFFFFFFFFFF8000 instead of 0xFFFF8000 on 64-bit builds because it does the arithmetic with 64-bit integers. Yes, that's stupid, and probably a bug since 68000 syntax would demand a 32-bit integer. Also AS doesn't use @ for local labels (besides nameless labels)... in fact it has two types of local labels. For this case I think $$ is the nearest replacement (not sure, didn't check).

Also assemblers not expanding 0xFF8000 to 0xFFFF8000 is correct behavior. Some programs actually make use of the highest byte for their own purposes (e.g. to store additional information besides the address).

[Bgvanbur]

The three assemblers I test have very similar syntax which makes it easy to make code that assembles the same for all three. From what I remember, a few of the important assembler directives of these three have no overlap with the AS assembler directives.

Sik, on 19 February 2012 - 08:47 AM, said:

Also assemblers not expanding 0xFF8000 to 0xFFFF8000 is correct behavior. Some programs actually make use of the highest byte for their own purposes (e.g. to store additional information besides the address).

The 0x??FF8000 issue is for when you use an address for an instruction (which make it use a word instead of a long and saves two bytes). You can always force an address to be treated as a long using (<address).L. For example, (0x00FF8000).L and (0xFFFF8000).L. I did this several times in disassemblies so that asmx/asm68k/snasm68k can all make byte perfect binaries of the original binary.

[Sik]

Bgvanbur, on 19 February 2012 - 09:36 AM, said:

The three assemblers I test have very similar syntax which makes it easy to make code that assembles the same for all three. From what I remember, a few of the important assembler directives of these three have no overlap with the AS assembler directives.

The reverse is true too. In particular, AS has functions, which are completely missing on the other assemblers. The disassemblies here made for AS abuse functions like crazy (including to work around that 64-bit bug I mentioned before). Then again even those assemblers are lacking important directives. IIRC asm68k doesn't have align (you have to use cnop), and I don't think asmx has the filesize() function (does snasm68k have it?), which is pretty much needed if you want to store uncompressed graphics without crossing a 128KB boundary and without doing alignment by hand.

[Chilly Willy]

The main issue I have will all these other assemblers is it's nigh-on impossible to use them with a C compiler. GAS syntax might be incompatible with them, but at least you could mix C and asm all you want when you stick to GAS. All my examples use GAS since my toolchain is based on gcc. GAS isn't too bad once you get used to it. I think if we had S1/2/3/K in gas syntax, more people would switch.

[Sik]

GAS syntax is still horrible since it clashes not just with the disassemblies, but also with about every single 68000 assembly tutorial out there x_x; And this is in MRI compatible mode, without that it gets worse. It really isn't meant to be used anywhere outside GCC itself.

The real issue here is that technically assemblers should be outputting object code to be passed to a linker... and actually several of them do =P Though not sure how many of them output in an object format GNU's ld understands.

[Chilly Willy]

Yeah, I wouldn't use gas syntax if I didn't have to. I'd love to see just one "standard" syntax assembler for all platforms (a nice posix command line app) that output ELF format object files. That would be perfection.

EDIT: Actually, vasm will output ELF files for the 68000. I'll have to try it with gcc to see if I can actually link vasm elf files with the rest of the gcc stuff. I could always use vbcc for the 68000 as well, but vbcc doesn't support the SH2, so I'm still stuck with gcc/gas for the SH2 side of things.

This post has been edited by Chilly Willy: 22 February 2012 - 03:32 AM

[Sik]

Ugh, seems like vasm's license explicitly forbids modification and commercial usage. I'm not sure many people will like that - especially since that'd explicitly forbid any cartridge releases unless cartridges are given away for free (also donations for homebrew projects wouldn't be allowed since legally they're considered commercial transactions too, go figure!).

I can imagine asmx not outputting ELF (since asmx in general is crap), but what about AS? Can it output ELF?

[Andlabs]

I was actually considering making my own assembler and I made a lexer and opcode table, but stopped for reasons too wordy to explain here. This was my syntax resolution: http://idisk.mac.com.../68000asm.4.pdf I wanted a syntax that could be parsed unambiguously — the standard parentheses-based syntax was overloaded so it quickly got difficult to talk about. It completely fell apart when I started considering the MC68020 =P

[Bgvanbur]

Andlabs, on 22 February 2012 - 07:25 AM, said:

I was actually considering making my own assembler...

We don't need yet another assembler. If it has any special features, they will get used and make code incompatible with other assemblers. Then when we have open source modules (like mdtools) that target different assemblers, you either need to use multiple assemblers or modify some of the code to use your flavor of assembly.

I think GAS hasn't taken off since it is a pain to install (whereas the other tools can be easily distributed since they are a single executable). And the differences in syntax help much either.

I tried using macro assembler (AS) and was able to get some of my projects to byte perfect assemble with it (as long as I added a macro to convert incbin to binclude). It has some quirks and I even found two bugs in it though.