don't click here

Valley Bell's SMPS Research

Discussion in 'Technical Discussion' started by ValleyBell, Dec 31, 2013.

  1. ValleyBell

    ValleyBell

    Tech Member
    246
    25
    28
    researching PC-98/X68000 sound drivers
    Most recent releases
    SMPS Research Pack v5: direct link part 1 / part 2, release post
    SMPSPlay 2.20: Win32/64 binary, source code archive / git repo, release post


    Original post
    Over the last two years I did a lot of research on SMPS drivers, both 68k and Z80.
    I ripped songs from 21 SMPS 68k and 35 SMPS Z80 games, did some research on preSMPS, especially Rent-A-Hero, and disassembled some sound drivers.

    And today I want to show you the results: the SMPS Research Pack.
    The probably most interesting thing inside are rips from almost 60 games.
    Except for the preSMPS games, every ripped game has not only the extracted SMPS files, but DAC and PSG data, too. (PSG data is stored in mid2smps' PSG list format.)

    There is also a file called "Pointers.txt" in every folder that lists the locations of pointer lists and other data, as well as the format of the DAC sound list. Later rips are more complete and have more notes than earlier ones.
    For two games, OutRunners and Virtua Racing (SVP), there is also table that explains the custom coordination flags.

    Some folders have songs patched to work with SMPSPlay. The original songs are marked with .org in this case.

    The folder for Rent-A-Hero contains files in SMPS 68k format. I converted them from preSMPS 68k to SMPS 68k using a small tool I wrote. The conversion is not perfect (missing tempo changes and DAC on/off flags), but pretty close.


    The programmer guys may be interested in the folder with disassembled SMPS drivers. (Included are original .bin, the .idb and an exported .asm file.)

    There are disassemblies of a fair number of DAC drivers. I also compared them and noted the differences in a separate .txt file.
    I also collected some SMPS Z80 drivers, sorted them into categories and disassembled a few of them. Please note that some of the older disassemblies (2012 and early 2013) contain mistakes.
    Finally there are also a few disassembled preSMPS drivers. I didn't get far with Phantasy Star 2, but the other two are labelled well. PS2 with its "chord mode" and Space Harrier 2 with a 2x 2-Operator FM drum are both pretty interesting.


    Finally, there is also a folder with a few .txt files with tables that tell you the playback speed of a few DAC drivers. These are DAC rate -> frequency in Hz conversion tables and the values are reused by SMPSPlay to determinate the playback speed of the DAC samples.


    You think that's all? No, it's not.

    In order to play all those rips, you probably want to download an updated version of SMPSPlay, an SMPS player for Windows.
    It includes a Win32 binary, as well as a 64-bit one.
    If you want to look at the source code, you can get it here. Be warned though, it's pretty messy right now and I plan to make some larger changes that make it easier to support various SMPS variants.

    Important notes about SMPS Z80:
    - The offset autodetection is unstable and doesn't work on some games or songs. It's pretty much impossible to guess if there is garbage data between header and actual song data or not.
    - For some songs you will need instrument sets that are shared between songs. You can edit data\config.ini to use the respective InsSet.bin files.
    - Some SMPS Z80 games use FM and PSG drums by playing notes on the DAC/Rhythm channel. SMPSPlay currently supports only DAC sounds on this channel, so I tried to replace them with DAC drums in a few folders. The only game where it worked well was Dyna Brothers 2 though.

    Now, enjoy and happy hacking!

    P.S.: It's not ready yet, but I will release an updated version of my SMPS Extractor soon, too. I used it to extract all songs and DAC sounds.

    EDIT: You can download a slightly updated SMPSPlay here.
     
  2. amphobius

    amphobius

    doing more important things with my life Member
    2,120
    0
    16
    life
    Oh my god I fucking love you
     
  3. Mastered Realm

    Mastered Realm

    Member
    3,828
    553
    93
    -
    H

    O

    L

    Y


    S

    H

    I

    T

    Non-crappy PCM playback? Incredible I say. Please rip Sonic 3 when you have time. I bet the 'Go' and 'Woo' samples would sound much better this way!

    Awesome work :D Btw the tracks that have that fake reverse cymbal in 3D blast got a corrupted bass in your rip.

    The PSG also behaves differently from real hardware, dunno maybe the tremolo is too fast?
     
  4. ValleyBell

    ValleyBell

    Tech Member
    246
    25
    28
    researching PC-98/X68000 sound drivers
    Thanks for the nice feedback :)

    SMPSPlay itself comes with ripped DACs and PSG from Sonic 1, 2, 3, S&K and 3D already.
    You can get S3K's songs from the Retro Git. You need to rename them to .s3k, activate the S3K instrument set in config.ini, then most of them should play.

    For completeness I'll do my own rip of S3 and S&K sometime though.

    Oh, right. I remember that these tracks caused me trouble when writing smps2mid, too.
    The bass' transpose value is invalid actually, but the note/octave calculation in the original driver results in values that make correctly sounding notes.
    If you want to quickly solve the problem, change the byte at offset $000C in both songs from $D4/$D5 to $14/$15.

    It shouldn't, but there is a chance I broke something. It's also possible that it has something to do with the PSG emulation. Especially the noise volume might be a bit off.
     
  5. Mastered Realm

    Mastered Realm

    Member
    3,828
    553
    93
    -
    No problem :D

    Before I start, could I make a request? -w for WAV logging! I currently have my audio running through HDMI so my onboard Sound card disables the stereo mix as it's not outputting the audio itself (it's my GPU!) so I can't make recordings from the program, only if I export a VGM and then use Winamp's wav logger. I've used some virtual stereo mix solutions already but it was more a hassle than it's worth...

    The PSG is specially off on the Outrunners set. Ah, an observation, the PCM percussion plays apparently fine inside SMPSPlay but as soon as I export to VGM it doesn't play properly. I've chosen '0A Last Wave' from it and made a comparison:

    1) Real Hardware from Outrun 20th Anniversary OST - (seems to be an MD2 and with a tad bit of noise there),
    2) Dark Pulse's VGz RIP,
    3) WAV logged from the VGM export of SMPS Play (the PSG sounds the same as in the program, so it's valid for the PSG matter anyway; you can record yourself and compare, though).

    Download the ogg file!
     
  6. ValleyBell

    ValleyBell

    Tech Member
    246
    25
    28
    researching PC-98/X68000 sound drivers
    Here is an updated SMPSPlay. I fixed the OutRunners modulation bug, the VGM logging and -w for wave logging. (It writes to dumps\wavelog.wav.)
    Wave logging logs the raw output stream, so it begins logging when you open the program and stops when you close it.

    Ahh, so the vibrato was too fast. (Tremolo usually means volume modulation.)
    The problem was that the modulation was handled incorrectly. Most things (including modulation) in OutRunners work like SMPS 68k, but it emulated SMPS Z80 modulation due to a small oversight.
    I also included a fixed PSG list. OutRunners handles the PSG envelope commands a differently to other games.

    Anyway, SMPSPlay is far from perfect and there are still some features missing. Things like "frequency flutter" (see Music Hacking Guide, S3K flag $F1) and YM2612 based timing don't work yet. The latter one is really noticeable with some songs with .smy extention.
    SMPSPlay should play SMPS files from Sonic 1 and 2 almost 100% accurately though. (except that it doesn't emulate a few bugs from S1 that S2 has fixed)

    Indeed.
    This happened only with OutRunners and Virtua Racing, because I disabled the (heavily underdeveloped) SMPS parser that scans the files for DAC sounds in those cases. (I didn't bother to handle their custom coordination flags.)
    I added a workaround that puts all existing DAC samples into the VGMs for now. This makes them a lot larger than they would need to be, but it works.

    In case you didn't notice, the "Pit In" song in Virtua Racing (SVP) played a wrong DAC sound. I noticed that the game writes another DAC sample to the Z80 RAM if this song is played, so I included a fixed sound.
     
  7. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    1
    0
    Writing my own MD/Genesis sound driver :D
    boring technical and historical information no one other than ValleyBell (who I already talked to) will care about

    I will need to re-prepare a table of SMPS games sorted by date and categorize each; it should give us better insight as to the development history of this driver. However, true insight cannot be obtained without looking at Sega's arcade games, as SMPS has its roots as far back as Space Harrier, if not even further back (but before Space Harrier the only 68000-based games were Hang-On and Major League, so a bit more work will be needed to look at earlier games).

    I have a feeling variants of SMPS with hardcoded PCM panning values are properly SMPS/T's Music.
     
  8. Mastered Realm

    Mastered Realm

    Member
    3,828
    553
    93
    -
    Awesome, ValleyBell. Thanks for the fix and the WAV logging! :D

    New observations! The pitch on some drums is completely different!
    Download the OGG! In the order they appear on my file:

    SMPS version: it plays at 7550Hz

    Dark Soul's VGz rip: it plays at 5500Hz

    Real hardware (Outrun 20th Anniversary OST): it plays at 5250Hz
     
  9. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    I can at least start this off - I grabbed the list of sound drivers from SMSPower, dumped it into OpenOffice, then scraped Sega Retro by hand for release dates. This is the result.
     
  10. Mastered Realm

    Mastered Realm

    Member
    3,828
    553
    93
    -
    And that is only for the Mega Drive! There are Sega CD games that use SMPS, not to mention the GG and Master System (and I bet Sonic Pocket Adventure uses something similar!).
     
  11. ICEknight

    ICEknight

    Researcher Researcher
    Also, SEGA Pico?
     
  12. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    I've added the 32X section of the sound engine list in another sheet, but in the absence of lists of Mega CD/Master System/Game Gear/Pico games that use SMPS, I can't do anything more. I'm willing to give edit access to people who can fill in these gaps - PM me your info.
     
  13. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    1
    0
    Writing my own MD/Genesis sound driver :D
    All of the above, and some System C/C2 games (I think all except Thunder Force AC; not sure about Puyo Puyo/Puyo Puyo Tsuu). I think I actually do have a list on Sega Retro already, but I need to check again.
     
  14. Mastered Realm

    Mastered Realm

    Member
    3,828
    553
    93
    -
    Valleybell! I can't play any song from Chaotix 32x! Why?

    It just says 'invalid track offset' o.o
     
  15. ValleyBell

    ValleyBell

    Tech Member
    246
    25
    28
    researching PC-98/X68000 sound drivers
    You know, this is very obvious actually and I know this already.
    I just didn't care a lot about it, because it can be fixed by editing the DAC.ini files. (It takes some work to measure the sample rates and get correct values, because I need to hack the ROMs to inject custom sound data and do some realtime RAM editing.)
    You can try to mess around with the BaseRate and RateDiv values to get it right. I recommend to increase both until it fits more or less.
    It should fit for most SMPS 68k games, as well as Sonic 2, 3K and 3D and Rent-A-Hero. (I.e. all games that use a DAC driver that's listed in the DACFrequencies folder are verified.)

    You don't need to post ogg files of everything, btw, especially when it's obvious what's wrong.

    I can tell you that Phantasy Star 1 uses preSMPS. Sonic 2/Chaos/TT use usual SMPS, but that is already known.
    Sonic Gameworld on the Sega Pico uses SMPS according to Sonic Of 8! from vgmrips. (He was so nice and noted it in the VGM pack's description file.) Same for the Pocket Monsters game.

    SPA doesn't use SMPS. It uses SNK's sound driver and I'm still having a hard time disassembling it.

    Because of something you won't find in the readme file: the .skc extention.

    That means: There is no support for Chaotix yet.
    There are 2 reasons for that:
    1. I haven't added PWM support to SMPSPlay. (But I disassembled the PWM sound driver that mixes the samples some months ago.)
    2. Unlike all other SMPS files, the DAC/rhythm track is missing (the very first track in every SMPS file), so I'd need to add extra code to handle Chaotix' SMPS files correctly. And I don't want to add anything unless I reworked a majority of the SMPS playback code in SMPSPlay. (because of the huge mess it is right now)
     
  16. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    Master System or Game Gear, or both? What about Sonic 1?


    There's three, you'll have to be more specific.
     
  17. ValleyBell

    ValleyBell

    Tech Member
    246
    25
    28
    researching PC-98/X68000 sound drivers
    MS and GG versions are almost the same and use the same sound driver. Also I forgot to list Sonic Blast/G Sonic, whose sound driver code matches the one of STT exactly IIRC, and Sonic Labyrinth.
    Sonic 1 MS/GG and Sonic Spinball MS use different sound drivers each.


    Its full name is Pocket Monsters Suuji o Tsukamaeyou!.
     
  18. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    I wanted to find out how possible a SMPS 68k variant with SMPS Z80 coordination flags would be, since I know 68k assembly a lot better than Z80 assembly, but I'm aware that Sonic 1's SMPS 68k is lacking in coordination flags, and there's quite a few more games using SMPS Z80 than SMPS 68k. To that end I took a bunch of games using SMPS 68k (starting from the most recent, Ristar, and leaping backwards) and disassembled them to find out what coordination flags they used. Here's what I got.

    Code (Text):
    1.  
    2. Ristar
    3.     Andlabs already analysed this one
    4.     sound driver RAM at $FFFFF000
    5.     ten metacoordination flags
    6.     cfNoteFill has an extra instruction. Possibly a bugfix, possibly part of
    7.     Ristar's customisation.
    8.    
    9.     cfNoteFill:
    10.             move.b  (a4),$12(a5)
    11.             move.b  (a4)+,$13(a5)
    12.             move.b  #0,$2D(a5) ; <--
    13.             rts
    14.  
    15. Samurai Shodown
    16.     not sure where sound driver starts - code starts at $2F5B06
    17.     sound driver RAM at $FFFF0000
    18.     five metacoordination flags
    19.     cfJumpTo has an extra instruction. Looks like it corrects for the off-by-one
    20.     thing people noticed when porting between S1 and Ristar.
    21.    
    22.     cfJumpTo:
    23.             move.b  (a4)+,d0
    24.             lsl.w   #8,d0
    25.             move.b  (a4)+,d0
    26.             adda.w  d0,a4
    27.             subq.w  #1,a4 ; <--
    28.             rts
    29.  
    30. Mighty Morphin' Power Rangers: The Movie
    31.     sound driver starts at $30000
    32.     sound driver RAM at $FFFFFC00
    33.     three metacoordination flags
    34.     cfJumpTo has an extra instruction. Looks like it corrects for the off-by-one
    35.     thing people noticed when porting between S1 and Ristar.
    36.    
    37.     cfJumpTo:
    38.             move.b  (a4)+,d0
    39.             lsl.w   #8,d0
    40.             move.b  (a4)+,d0
    41.             adda.w  d0,a4
    42.             subq.w  #1,a4 ; <--
    43.             rts
    44.  
    45. Pulseman
    46.     not sure where sound driver starts - code starts at $73BC8
    47.     sound driver RAM at $FFFFF000
    48.     three metacoordination flags
    49.     cfJumpTo has an extra instruction. Looks like it corrects for the off-by-one
    50.     thing people noticed when porting between S1 and Ristar.
    51.    
    52.     cfJumpTo:
    53.             move.b  (a4)+,d0
    54.             lsl.w   #8,d0
    55.             move.b  (a4)+,d0
    56.             adda.w  d0,a4
    57.             subq.w  #1,a4 ; <--
    58.             rts
    59.  
    60. Streets of Rage
    61.     sound driver starts at $72000
    62.     two sets of coordination flags, depending on the value of 1(a3) bit 7 - none
    63.     are recognisable from comparing with S1 or Ristar (is this pre-SMPS?)
    64.  
    65. Golden Axe 2
    66.     sound driver starts at $76000
    67.     sound driver RAM at $FFFF0000
    68.     two metacoordination flags
    69.     flags F1/F4 are S1's cfEnableModulation/cfDisableModulation, not
    70.     Ristar's cfAlterModulation/cfSetModulation
    71.     flag FC is Ristar's cfSetTempoDividerMusic (flag FF02)
    72.     flag FD is new - it does something with priority
    73.     flag FE is new - it takes SEVEN parameter bytes and does something with the
    74.     current note
    75.  

    A digest for people who don't want to read the whole thing:
    • SMPS 68k and SMPS Z80 development seems to have happened in parallel. The coordination flags used in Ristar are similar to those used in Sonic and Knuckles, but there's a fair few differences.
    • As a rough approximation, the number of coordination flags correlates with the release date, and after a certain point the SMPS developers seem to have fixed the list of coordination flags between $E0 and $FE and opted instead to add new flags to the switch table for metaflag FF instead. Pulseman and MMPR:TM have the same flags, Samurai Shodown adds a couple, and Ristar adds a few more on top of that. There's no disagreement on order or what particular flags are other than one game having more/fewer than another.
    • I expected Streets of Rage to have roughly the same array of coordination flags as Sonic 1, but it seems to have a completely different setup - coordination flags start at $F0, and there's two sets of them which are switched between by flipping (what looks to be) bit 7 of the playback control byte. Sonic 1 is more like Golden Axe 2 in that respect. That said...
    • Sonic and Knuckles customised the coordination flags quite heavily. I suspect the "standard" set looks more like Ristar, but (as the saying goes) research is ongoing.
    My plan now is to look at World of Illusion to find out what other games at around the same time as Sonic 2 had for coordination flags, and keep working my way backwards through SMPS 68k games - Golden Axe 2 is right where things start getting interesting because the "core" flags ($E0-$FE) aren't yet quite set in stone.
     
  19. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    onos double post

    ValleyBell suggested in #techies that I look at a SMPS/Treasure game, so I picked Light Crusader because it's roughly contemporaneous with Ristar so it gives an idea what the two drivers looked like at the same point in time.

    Code (Text):
    1.  
    2. Light Crusader
    3.     sound driver doesn't seem to start like other SMPS variants (no "header")
    4.     sound driver RAM at $FFFFF800
    5.     five coordination flags
    6.     various coordination flags access sound memory directly rather than an
    7.     indirect reference through a6:
    8.         cfUnknown1, cfSetLFOData, cfQueueSound, cfSetVoice
    9.     cfRepeatAtPos is simpler than in Ristar (indicated lines Ristar-only)
    10.    
    11. cfRepeatAtPos:
    12.         moveq   #0,d0
    13.         move.b  (a4)+,d0
    14.         move.b  (a4)+,d1
    15.         movem.l d2,-(sp)    ; <--
    16.         move.b  $28(a5,d0.w),d2 ; <--
    17.         movem.l (sp)+,d2    ; <--
    18.         tst.b   $28(a5,d0.w)
    19.         bne.s   $$loopexists
    20.         move.b  d1,$28(a5,d0.w)
    21.  
    22.     $$loopexists:
    23.         subq.b  #1,$28(a5,d0.w)
    24.         movem.l d2,-(sp)    ; <--
    25.         move.b  $28(a5,d0.w),d2 ; <--
    26.         movem.l (sp)+,d2    ; <--
    27.         bne.s   cfJumpTo
    28.         addq.w  #2,a4
    29.         rts
    30.    
    31.     coord flags FF03 and FF04 are new - seem to alter tempo in some way
    32.  

    This is less drastic than I expected for something that gets its own designation in the SMSPower listing.
     
  20. Quickman

    Quickman

    be attitude for gains Tech Member
    5,595
    18
    18
    :x
    omg porjcet
    I'm getting into some serious antiquity now - Battle Golfer Yui predates Sonic 1.

    Code (Text):
    1.  
    2. Battle Golfer Yui
    3.     sound driver RAM at $FFFFE000
    4.     coord flag jump table routine looks somewhat like Streets of Rage's, but
    5.     coord flags seem unfinished - they're largely Ristar-like, but the
    6.     extras (FFxx) are completely different
    7.     E3 does nothing
    8.     cfSetLFOData does nothing if bit 7 of the voice control byte is set
    9.     EA increases tempo rather than setting it
    10.     more flags accessing RAM directly rather than through a6:
    11.         cfQueueSound, cfSetVoice
    12.     F1 is cfEnableModulation like Sonic 1, not cfAlterModulation like Ristar
    13.     F2 might be cfStopTrack but is very different to the routine in Sonic 1
    14.     and Ristar
    15.     cfSetPSGNoise is simpler than in Ristar
    16.  
    17. cfSetPSGNoise:
    18.         move.b  #$E0,1(a5)
    19.         move.b  (a4)+,$26(a5)       ; move.b (a4),...
    20.         btst    #2,0(a5)        ; <--
    21.         bne.s   $$locret        ; <--
    22.         move.b  -1(a4),($C00011).l  ; move.b (a4)+,...
    23.  
    24.     $$locret:               ; <--
    25.         rts
    26.    
    27.     F4 is cfDisableModulation like Sonic 1, not cfSetModulation like Ristar
    28.     F5 (cfSetPSGTone) has an extra check (to make it a noop in FM channels?)
    29.     which isn't present in either S1 or Ristar
    30.     FC is cfSetTempoMod like Sonic 1, not cfEnableModulation like Ristar
    31.     FD is cfSetSSGEG, not cfDisableModulation
    32.     FE is novel - similar to cfFM3SpecialMode, but uses a RAM value rather
    33.     than a lookup table
    34.     FF00 might stop the music, not sure
    35.     FF01 is a duplicate of cfPanningAMSFMS
    36.     FF02 no idea
    37.     FF03 is some sort of conditional jump; it takes four parameter bytes -
    38.     the first two are an offset from the start of sound RAM, the second two
    39.     are the destination (same rules as for cfJumpTo) jumped to if the value
    40.     indicated by the first two parameter bytes is nonzero
    41.     FF04 is weird - it takes two parameter bytes which are an offset from
    42.     the start of sound RAM (like FF03) - that byte is read into d0, then the
    43.     routine jumps to SMPS_FMSetFreq
    44.     FF05 does nothing but eat three parameter bytes
    45.     FF06 does nothing
    46.  
    I'm surprised by the number of coordination flags that outright do nothing. I didn't think of it at the time, but flag FE might be like Golden Axe II's flag FE (which, as ValleyBell informed me, enables FM3 special mode) - I need to go take a second look at that. e: yeah, it's cfFM3SpecialMode.