XM to smps converter: released

Discussion in 'Engineering & Reverse Engineering' started by nineko, Sep 10, 2007.

Thread Status:
Not open for further replies.
  1. nineko

    nineko

    I am the Holy Cat Tech Member
    Updated the first post to say that I released it. The old message is still archived below for continuity purposes.

    -----
    In the past days I've been working on a program that allows you to convert songs from the XM format to the smps format used in Sonic games. Here is the thread in which I start to talk about it. Before you ask: no, no public betas yet. Now if you're interested keep reading.

    Background (aka don't read this section if you don't care about I had the idea)
    Some time ago I noticed that there are few hacks that feature original songs, most of the times they are music ports from other games. Sure, there are some guys from here that are skilled with smps programming (I can name: Tweaker, Puto, R.Solaris, Cinossu...), either in pure hex or aided by Sonic QX, but still, bringing a song from "the world outside Sonic" to a Sonic game is not something trivial and for everyone.
    Then the idea to program an smps tracker began to spin in my head. I also wondered if it is possible to convert vgm (or gym) files back to something usable, but this is another story.
    I soon scrapped the idea of an smps tracker because I realized that it's pointless to write something from scratch when I can use an existing tracker and an existing song format as a starting point. It's interesting how module files are so much similar to smps ones. From a certain point of view, smps songs are indeed like modules, so it shouldn't be so hard to make a converter, I thought. I then chose XM as starting format because it's the one I'm most familiar with; however, any tracker (like Mod Plug Tracker) can convert IT, MOD, S3M and many other formats to XM, so my choice isn't that important.
    However, the thing was started, and in a few days I managed to correctly import XM files.

    What my converter does
    It takes any XM file with any number of channels and instruments and correctly convert it to a song in smps format, that can be used in Sonic games. Right now the output format is Sonic 1's, but I will soon add options to export also to Sonic 2 and 3&K.

    Features so far
    1) Full support for both the YM2612 and the SN76496 sound chips. This means, up to 6 FM channels (or 5 FM channels + 1 DAC) and 3 PSG channels. One of the PSG channels can be used as noise, usually for hi-hats. While the SN76496 by itself is able to control 3 melodic PSGs and 1 noise channel, the Sonic driver seems unable to support it, and any attempt to do that results in broken sounds. No "official" song made by SEGA uses all the 4 PSG channels at the same time, so it's not a great loss, and not my fault;
    2) Full support for up to 256 voices, that can be chosen from a library. Right now this includes ALSO the Universal Voice Bank, but the library will be eventually bigger with voices from other songs or even custom made;
    3) Customizable initial volume and pitch setting for each channel. You can't change the volume nor the pitch during the song yet, but I'm definitely going to add at least the volume control (no promises for the pitch);
    4) The loops are 100% working, given that the loop begins at the start of the song. Right now you can't loop a specific section of the song (e.g. to have an intro and then the looping section), but it's going to be added soon;
    5) Some optimizations are in. This means that streaks of notes of the same duration or notes with the same pitch are saved with the fewest information possible. Some optimizations aren't in, there are no jumps right now (other than the main one). This means that SLZ's drum pattern would still be saved unrolled, but I'm soon going to add this optimization as well.

    Download section (expect this to be updated soon)
    Here you are three test songs if you want to try them in your Sonic 1. Remember: no notes have been coded by hand, everything here is the result of a conversion.
    Also, while my PSG support works, all the songs here only have FM channels (for no reason).

    1-1 theme from Super Mario Land. Uses GHZ's voices because it's one of my earliest conversions.
    Magical Sound Shower from Outrun. Still with GHZ's voices (another old track).
    Green Garden from Bomberman 64. My first attempt to use the Universal Voice Bank.

    End
    As you can see, there is still some work to do with this, but some results are already here, and I'm quite satisfied so far. I will soon add the other planned features, and I will be ready to give private betas to a few chosen people. I already have some people in my mind, chosen amongs hackers with a good knowledge of the Sonic sound driver, but I may need (and I will be glad) to have also someone familiar with the XM file format (or at least with module files in general).
    However, this isn't going to be a program for everyone. I mean, once it's done, don't expect that dropping an XM file on it is enough to get a perfect conversion, the user will need to know what he is doing, and he will be asked for several parameters, e.g. what voices to use, the initial volumes, and so on. While I can easily make the conversion even simpler, I don't want to go against this website's general guidelines, nothing should be too easy. Otherwise we would end up like the SCAA.
    I think it's enough for now. I fail at writing stuff, but I felt that a good presentation thread was needed. I think it was harder for me to write this text rather than programming the converter itself.

    Credits
    MAJOR CREDITS TO TWEAKER AND R.SOLARIS.
    Credits also to Puto.
    Thanks also to the SCHG guide.
    If I forgot someone, please tell me, I didn't mean to offend anyone.

    Waiting for comments plz.
     
  2. Chimpo

    Chimpo

    Member
    7,310
    0
    16
    Atomic Sonic Part II
    Just heard all three of them, with the Bomberman 64 being my favorite one. This is some awesome shit your convertor created here. Awesome job.
     
  3. nineko

    nineko

    I am the Holy Cat Tech Member
    Thanks. Since I just read this thread about your new Street Fury hack, and there is a Guile in the title screen, here you are the first notes of Guile's theme in smps format. Done in like, 10 minutes, just to show how simple it will be to work with my converter once it will be completely done. Of course you'll need more time to achieve a good result, but still :p
     
  4. Chimpo

    Chimpo

    Member
    7,310
    0
    16
    Atomic Sonic Part II
    eageaiopbjmeipaom#@()^($)#G#@^$&JKOK!K#$&JPBK!

    Tweaker has NO fucking excuse now.

    This is awesome. I can't exagerate enough.
     
  5. Aquaslash

    Aquaslash

    Emerald Dragoon Moderator
    2,818
    0
    16
    Hampton, VA
    The S Factor: Sonia and Silver
    This sounds interesting...at least it would be if I was familiar with the XM format.
     
  6. nineko

    nineko

    I am the Holy Cat Tech Member
    Yeah, I know, module files are quite obscure nowadays, not many people use them. During the Amiga era, songs in the MOD format were really popular, and during the late 90s a lot of cool songs in the XM format were released thanks to the introduction of Fast Tracker II (probably the best tracker ever). You have to be kinda oldschool to be familiar with module files, this is why I wasn't sure if I should take the effort to program this converter, I knew that the interest couldn't be very high, and by judging as the number of replies, I can easily see that I was right, probably no one is ever going to use this thing except for a very few people.

    The point is that if you want to add original music in your hack, the only other way to do that is to write song code in pure hex, either from scratch or "fine-tuning" the output from Sonic QX, but it's still a skill you have to master. So I'm still trying to release this converter because if one has to learn something, it can as well be how to use a tracker rather than how to write music in hex. It's much easier and allows for further edits with little effort. I'm not pretending that everyone has a complete collection of XM files ready to be put in various hacks, I'm saying that if one wants new music in his hack can open a tracker of choice and spend some hours composing them. And that would still be less time than what he would need by composing it in hex, imo. No bytes to count, no crazy shit. Just an user-friendly tracker where you actually see notes.

    I promise that I'm going to have some betas ready soon, I know that introduction threads without downloads aren't the best thing, but there are still a few features that need a finetune.

    Here, have another teaser track. This one with PSG. Super Mario 64 theme.

    edit: fixed link.
     
  7. SMTP

    SMTP

    Tech Member
    Using a tracker to create songs is really really easy.

    Also, the only reason your not getting very much replies is because people are too lazy to build a rom with your songs. :P
     
  8. SoullessSentinel

    SoullessSentinel

    Member
    257
    0
    16
    Grimsby, England
    Cxbx-Reloaded
    This tool would be great,
    I was about to start composing my own songs in hex, but, instead I think ill work on other aspects of my hack, until this tool comes out, and then, after the release, work on the music aspects. (not that I will rely on the tool completely however)

    Good work :(
     
  9. Hivebrain

    Hivebrain

    Administrator
    2,798
    0
    16
    53.4N, 1.5W
    HivePal 2.0
    This should increase the quality of a lot of hacks, especially if it's fairly user-friendly and works with most or all XM files. There's a huge library of XM/S3M/MOD/IT files on modarchive.org, so we wouldn't have to listen to old songs ported from Chaotix or whatever (ask the XM author for permission first though).
     
  10. nineko

    nineko

    I am the Holy Cat Tech Member
    I thought about uploading GYMs or VGMs, but my Gens is broken and can't output proper files. And mp3s aren't an option right now because I'm still on 56k, we're experiencing troubles with the ISP.

    About the second part: yes, it's going to work with every XM file (with possibly few exceptions). Of course the final outcome will depend on the Genesis's limitations, anyway, since it's impossible to replicate some of the effects allowed in a module. Examples of effects include vibrato, tone portamento (or pitch bends), arpeggio, volume slides. The volume column and the effect column in the XM file are simply being ignored right now, but all the notes and all the instrument changes are correctly recognized, and that's the important thing. I hope to include at least some kind of volume and panning control, but I can't make promises for now. The user will still be allowed to hand-edit the final outcome anyway, so final touches in hex are still completely possible.
    Now, the bad news, about the first part of your sentence. No, it's not really user-friendly, and in my plans it'll never be too easy to use, for the reasons stated above. Converting a song shouldn't be as easy as dropping an XM file on an icon, the final user must know what he is doing. This converter works in ms-dos text mode, there isn't a GUI and I don't plan to add one. The user is required to enter the parameters by hand while the program is running.

    Oh, good news now. Hi-hats finally work, it wasn't easy to add them. Here is another version of the Outrun song, now with Universal Voice Bank voices and hi-hats. Don't tell me that the voices suck, the fact that I'm able to extract them from the UVB doesn't mean that I'm also good at chosing them :P

    Also, the hi-hat support was the last "important" feature I wanted to add before releasing private betas. So we're getting closer, it's just a matter of few days now. Stay tuned. And thanks for this new feedback :D
     
  11. Armada

    Armada

    Sometimes I do things Member
    338
    0
    0
    All I can say is that this sounds simply brilliant.
     
  12. Hivebrain

    Hivebrain

    Administrator
    2,798
    0
    16
    53.4N, 1.5W
    HivePal 2.0
    I can't agree that it shouldn't be easy to convert a file. It should be as easy as it possibly can be without losing flexibility.
     
  13. ICEknight

    ICEknight

    Researcher Researcher
    Erm. Please do? I doubt anybody would be angry at having a user friendly program.
    Bingo. =(


    This might be asking for too much, but would it be possible to make a small ROM that can play these when attached to the end? I'm sure that would be very appreciated by many.
     
  14. nineko

    nineko

    I am the Holy Cat Tech Member
    Here, download this thing. Since I was lazy as well and I didn't want to build 83294205 test roms I wrote this nice utility that hard-replaces GHZ's music with a given binary file. Just place it in the same folder with a clean Sonic 1 rom (that must be named "Sonic.bin") and the binary file of the song you wish to use. Please note that the resulting rom will need the "fix checksum" option in Gens to run, and if the file you're going to test is longer than the original GHZ (1896 bytes) you're going to destroy other songs (most likely, Labyrinth, and Marble if the song is that long).
    This replacer tool isn't mean to be used to put songs in your hack, it's for testing purposes only!

    edit: fixed link
     
  15. Aquaslash

    Aquaslash

    Emerald Dragoon Moderator
    2,818
    0
    16
    Hampton, VA
    The S Factor: Sonia and Silver
    While this may potentially open the flood gates, I should mention that composing new song with current methods really isn't as difficult as its being made out to be. (I'll post my exact procedure later)

    Now, as for this, how much will this assist the hacker who...let's say wants to compose all of his game's music on his own (as in completely original songs) and import it into a game?
     
  16. ICEknight

    ICEknight

    Researcher Researcher
    Okay, this is awesome. Great job, dude.
     
  17. Tweaker

    Tweaker

    Misfit
    12,389
    1
    0
    This would be even more helpful if a decent MIDI to XM converter existed. :P
     
  18. SMTP

    SMTP

    Tech Member
    Should help alot considering there are lots of programs out there that can make .XM format songs easily.
     
  19. nineko

    nineko

    I am the Holy Cat Tech Member
    You know, fuck the private beta thing, I can as well release this to everyone, since it's kinda working right now.

    I'm probably going to release a public beta tomorrow if I have the chance to write some tutorial about it. I also need to remove all the text that is still in italian (lol), and I should make it idiot-proof. Right now all the inputs are unprotected, you can say you want 73 PSGs and 124 FM channels :p
     
  20. Chimpo

    Chimpo

    Member
    7,310
    0
    16
    Atomic Sonic Part II
    While I can understand not making this too easy so that any idiot can just put god awful songs in their shit (And believe me, I seriously understand when people use your shit to make their shit even shittier.) , just how much user-unfriendly is your program here? I know jack shit about music, my only talents is in the art department, but is this still understandable enough to let someone like me figure out how to port songs without needing a middle-man at all (lol Tweaker.) From what it sounds like, it's not very friendly for somebody like me.

    Sure, you don't to be too easy so idiots dump shit in, but I surely hope that you didn't make this so unfriendly that those with zero knowledge and are willing to learn that this just becomes a whole other obstacle they have to overcome.

    So yeah, just how unfriendly is this.
     
Thread Status:
Not open for further replies.