Sonic Chronicles Hacking

Hi guys.
NeKit and I have been working on exploring HERF-archives recently. Now we know this format much better, so we can enlarge and correct data we knew from softman's research.

Code (Text):

1. Offset 0x04 - files count (word or dword)
2. Starting from offset 0x08 there are dword's blocks for the amount of files, format:
3. -- Unknown, probably ID of something like this (dword)
4. -- File Size (dword)
5. -- File Offset (dword)

As for the file names, there is something weird with them. We could found them only for test.herf.
The most weird thing there is that they were in a file from the archive. That file had the format of HERF-archive with the same amount of files. Every file there was 132 bytes long and contained file names for the files from main archive, including the archive these files were in.
The format was:

Code (Text):

1. Unknown dword (4 bytes)
2. File Name (128 bytes)

Interesting, that every unknown dword from these files had the same value as unknown dwords from main archive.

The name of the archive containing file names is "erf.dict".
In test.herf itself, there were also lots of graphical files (.tga, .nftr, .ncgr), btw nftr-files are the fonts!

Other herf-archives seem to contain some language data.
test_e.herf - English
test_i.herf - Italian
test_s.herf - Spanish
test_g.herf - German
test_f.herf - French

They have far less files, but no file names.

Also, the knowledge we had was enough to make a small util to unpack HERF-archives. So I did it.
Meet the Stupid HERF-unpacker v.0.1!
<a href="http://vladikcomper.narod.ru/download/Stupid_HERF-Unpacker_v01.rar" target="_blank">Download</a>
It's stupid because it has no interface as well as any error exceptions, though it's easy to use. =P
Just run it and enter your archive name and it will be unpacked in the same folder where the util is.
Note that it doesn't look for file names.

 

Code (Text):

1. # test.herf from Sonic Chronicles EU
2. # script for QuickBMS http://aluigi.org/papers.htm#quickbms
3.  
4. goto 0x4
5. get FILES long
6.  
7. print "%FILES% files in archive"
8.  
9. goto 0xC
10.  
11. set CURNAME 45059704;
12.  
13. for I = 0 < FILES
14. &nbsp;&nbsp; savepos CURRENT
15. &nbsp;&nbsp; get SIZE long
16. &nbsp;&nbsp; get OFFSET long
17. &nbsp;&nbsp;
18. &nbsp;&nbsp; goto CURNAME
19. &nbsp;&nbsp; get NAME string
20. &nbsp;&nbsp; math CURNAME += 132
21.  
22. &nbsp;&nbsp; math CURRENT += 0xC
23. &nbsp;&nbsp; goto CURRENT
24.  
25. &nbsp;&nbsp; log NAME OFFSET SIZE
26.  
27. next I

This small script should be able to extract files with names from European test.herf. It just makes an assumption that filenames are in the same order as files (and, surprisingly, it is true, as later we found erf.dict).

 

While translating the text, we've found something interesting. Developers left cutscenes descriptions, which, I suppose, were used for internal purposes. But, many of them never made their way in game.

This explains why all thougt that Eggman was killed.

There is neither Tutoris Island nor captain in game.

 

What? No "replace the soundtrack" option?

 

Replacing the soundtrack... well, I don't know if anyone has successfully crafted and replaced music bank files in a DS ROM before.

Also, out of curiousity. Sonic Chronicles was made by Bioware, in North America. It came out in English first. Why, then, are you going through the effort to translate the Japanese text back to English?

 

Not sure if this is something that is actually relevant to this kind of game manipulation, but there's an oddity that has always bugged me on Voxai Colony Beta. On the minimap, the Overmart has a gold star over it implying that it's the source of a mission, but no mission is ever forthcoming from the little jellyfish. It seems to me like he might have been a stop on an expanded "Deliver the N'rrgal spit" merchant fetch quest. Don't suppose you've located any scraps of dialogue or instance markers around that area which might account for a half-programmed mission?

 

We are not translating Japanese text back to English, we are translating English text to Russian.

Nothing yet, but it might be there, as they've left even such strings as "Test Level" or "I didn't SPOILER SPOILER SPOILER. It looked as if SPOILER SPOILER SPOILER!".

Also, does anybody know anything about ActImagine Video Codec (.vx files), used in Sonic Chronicles? It's supposed to be in Nintendo GameBoy Advance and DS SDK, but either leaked on torrent one is incomplete, or I'm not not looking at the right place. There is text, embedded into video, so we had to edit it somehow.

 

I finally managed to extract files with names from Japanese version of test.herf using erf.dict of both versions as erf.dict from Japanese test.herf is incomplete. I did diff at directories, and, wow, they really made some changes:
<div class='codetop'>CODE</div><div class='codemain' style='height:200px;white-space:pre;overflow:auto'>
Only in test_jn: 00000163.GFF
Only in test_jn: 000001be.RTF
Only in test_jn: 0000035b.JBF
Only in test_jn: 0000036c.GFF
Only in test_jn: 00000532.GFF
Only in test_jn: 000005c4.GFF
Only in test_jn: 00000d02.dat
Only in test_jn: 000013be.GFF
Only in test_jn: 00001b61.GFF
Only in test_jn: 000021de.dat
Only in test_jn: 000021df.dat
Only in test: 000021f1.dat
Only in test: 000021f2.dat
Files test/Adv_Big.GDA and test_jn/Adv_Big.GDA differ
Files test/CFX_CHLp1.nsbtx.small and test_jn/CFX_CHLp1.nsbtx.small differ
Files test/CFX_HEAp1.nsbtx.small and test_jn/CFX_HEAp1.nsbtx.small differ
Files test/CameraPanNocturne.gff and test_jn/CameraPanNocturne.gff differ
Files test/ChaoGarden_LOWER.gui and test_jn/ChaoGarden_LOWER.gui differ
Only in test: Chap0Scr_Bot.tga
Only in test: Chap10Scr_Top.tga
Only in test: Chap1Scr_Top.tga
Only in test: Chap2Scr_Top.tga
Only in test: Chap3Scr_Top.tga
Only in test: Chap4Scr_Top.tga
Only in test: Chap5Scr_Top.tga
Only in test: Chap6Scr_Top.tga
Only in test: Chap7Scr_Top.tga
Only in test: Chap8Scr_Top.tga
Only in test: Chap9Scr_Top.tga
Files test/Chapter0.GDA and test_jn/Chapter0.GDA differ
Files test/FRONT_SelectASlot.gui and test_jn/FRONT_SelectASlot.gui differ
Files test/FRONT_SlotSelected.gui and test_jn/FRONT_SlotSelected.gui differ
Files test/FX_FRCFLD.nsbmd.small and test_jn/FX_FRCFLD.nsbmd.small differ
Files test/HowToJournal.GDA and test_jn/HowToJournal.GDA differ
Files test/ICN_L_Medkit.NCGR.small and test_jn/ICN_L_Medkit.NCGR.small differ
Files test/ICN_S_Medkit.NCGR.small and test_jn/ICN_S_Medkit.NCGR.small differ
Only in test: ICN_S_PAtk_TAI3.NCGR.small
Only in test: ITM_ACT2_chaos3_0.NCGR.small
Only in test: ITM_ACT2_chaos3_1.NCGR.small
Only in test: ITM_ACT2_chaos3_2.NCGR.small
Only in test: ITM_ACT2_chaos3_3.NCGR.small
Only in test: ITM_ACT2_chaos4_0.NCGR.small
Only in test: ITM_ACT2_chaos4_1.NCGR.small
Only in test: ITM_ACT2_chaos4_2.NCGR.small
Only in test: ITM_ACT2_chaos4_3.NCGR.small
Only in test: ITM_ACT2_chaos6_0.NCGR.small
Only in test: ITM_ACT2_chaos6_1.NCGR.small
Only in test: ITM_ACT2_chaos6_2.NCGR.small
Only in test: ITM_ACT2_chaos6_3.NCGR.small
Only in test: ITM_ACT2_chaos7_0.NCGR.small
Only in test: ITM_ACT2_chaos7_1.NCGR.small
Only in test: ITM_ACT2_chaos7_2.NCGR.small
Only in test: ITM_ACT2_chaos7_3.NCGR.small
Files test/IntroMovies.GDA and test_jn/IntroMovies.GDA differ
Files test/Items.GDA and test_jn/Items.GDA differ
Files test/Jillcan12ol.NFTR and test_jn/Jillcan12ol.NFTR differ
Files test/Jillcan12rg.NFTR and test_jn/Jillcan12rg.NFTR differ
Files test/Jillcan12sh.NFTR and test_jn/Jillcan12sh.NFTR differ
Files test/Jillcan13bdol.NFTR and test_jn/Jillcan13bdol.NFTR differ
Files test/Jillcan13bdrg.NFTR and test_jn/Jillcan13bdrg.NFTR differ
Files test/Jillcan13bdsh.NFTR and test_jn/Jillcan13bdsh.NFTR differ
Files test/Jillcan13ol.NFTR and test_jn/Jillcan13ol.NFTR differ
Files test/Jillcan13rg.NFTR and test_jn/Jillcan13rg.NFTR differ
Files test/Jillcan13sh.NFTR and test_jn/Jillcan13sh.NFTR differ
Files test/MovieList.GDA and test_jn/MovieList.GDA differ
Files test/ProfileCreaturePanel.gui and test_jn/ProfileCreaturePanel.gui differ
Files test/Prologue.GDA and test_jn/Prologue.GDA differ
Files test/RingsItems.gui and test_jn/RingsItems.gui differ
Files test/Spell_Antidote_All_2.spl and test_jn/Spell_Antidote_All_2.spl differ
Files test/Spell_Antidote_All_3.spl and test_jn/Spell_Antidote_All_3.spl differ
Files test/Spell_Flashbang.spl and test_jn/Spell_Flashbang.spl differ
Files test/Spell_Flashbang_2.spl and test_jn/Spell_Flashbang_2.spl differ
Files test/Spell_Flashbang_3.spl and test_jn/Spell_Flashbang_3.spl differ
Files test/WORLDMAP_top1.gui and test_jn/WORLDMAP_top1.gui differ
Files test/a1_s1_ex_slide3.gff and test_jn/a1_s1_ex_slide3.gff differ
Files test/a2_tails.dlg and test_jn/a2_tails.dlg differ
Files test/a2scy_shop.dlg and test_jn/a2scy_shop.dlg differ
Files test/a2sq_dragon.dlg and test_jn/a2sq_dragon.dlg differ
Files test/act1_world0.are and test_jn/act1_world0.are differ
Files test/amb_uppertextbubble.gui and test_jn/amb_uppertextbubble.gui differ
Files test/ambientlower.gui and test_jn/ambientlower.gui differ
Files test/bif_OME.nbfp and test_jn/bif_OME.nbfp differ
Files test/bif_OME.nbfs and test_jn/bif_OME.nbfs differ
Files test/bif_ROU.nbfp and test_jn/bif_ROU.nbfp differ
Files test/bif_ROU.nbfs and test_jn/bif_ROU.nbfs differ
Files test/bif_SHA.nbfp and test_jn/bif_SHA.nbfp differ
Files test/bif_SHA.nbfs and test_jn/bif_SHA.nbfs differ
Files test/bif_SON.nbfp and test_jn/bif_SON.nbfp differ
Files test/bif_SON.nbfs and test_jn/bif_SON.nbfs differ
Files test/bif_TAI.nbfp and test_jn/bif_TAI.nbfp differ
Files test/bif_TAI.nbfs and test_jn/bif_TAI.nbfs differ
Files test/combo.GDA and test_jn/combo.GDA differ
Files test/creatures.GDA and test_jn/creatures.GDA differ
Files test/credits.dlg and test_jn/credits.dlg differ
Files test/debug.GDA and test_jn/debug.GDA differ
Files test/debugCreateItems.GDA and test_jn/debugCreateItems.GDA differ
Files test/erf.dict and test_jn/erf.dict differ
Files test/journalcredits.GDA and test_jn/journalcredits.GDA differ
Files test/lowertextbubble.gui and test_jn/lowertextbubble.gui differ
Files test/plt_a2sq_nrrgalshipment.plo and test_jn/plt_a2sq_nrrgalshipment.plo differ
Files test/squads.GDA and test_jn/squads.GDA differ
Files test/uppertextbubble.gui and test_jn/uppertextbubble.gui differ
Files test/version.txt and test_jn/version.txt differ
Files test/worldmap.GDA and test_jn/worldmap.GDA differ
</div>
Also, different plt_a2sq_nrrgalshipment.plo may show that they finished the mission Frozen Nitrogen asked about, or at least fixed Overmart.

By the way, a lot of files inside can be changed with Dragon Age Toolset. It definitely opens some opportunities for hacking. *.small ones are LZ77 compressed. *.spl (POW moves) and *.itm (Items) are really 2DA.

 

Too bad there's no free version of this tool available. You have to buy and register the PC version of Dragon Age in order to even obtain this.

Also, I made a very quick and dirty PHP script to do the one thing your HERF unpacker didn't support.
<div class='codetop'>CODE</div><div class='codemain' style='height:200px;white-space:pre;overflow:auto'><?php
// Time to do what Stupid HERF Unpacker fails at: filenames.

//Stupid PHP time limits
set_time_limit(0);
// For people playing at home: names.bin is 8053.bin from test.herf
$dalist = fopen("names.bin", r);
//Skip 8 bytes.
fseek($dalist, 0x8);
//Target directory.
$targetdir = "herf";
//Target destination.
$dest = "output";
//8691 files to go through.
for($I = 1; $I <= 8691; $I++)
{
//Skip 4 bytes.
fread($dalist, 0x4);
//Read 128 bytes.
$filename = rtrim(fread($dalist, 0x80));
//Copy the file!
copy($targetdir."/".$I.".bin", $dest."/".$filename);
}
//And we're done!
?></div>

 

This is interesting!

 

Toolset can be found on torrent trackers without registration, but you still need the game installed.

Actually, I've written another script in Python not long ago, which also supports packing files back.
<div class='codetop'>CODE</div><div class='codemain' style='height:200px;white-space:pre;overflow:auto'>from struct import unpack, pack
from os.path import getsize
from sys import exit
import csv
import argparse

parser = argparse.ArgumentParser(description='Unpacks and packs *.herf files from Sonic Chronicles')
parser.add_argument('action', choices=['unpack', 'pack', 'dictionary'],
help='action')
parser.add_argument('herf', metavar='file', help='herf file to be unpacked/created')
parser.add_argument('dir', help='output/input dir (without slash)')
parser.add_argument('-e', metavar='erf.dict', help='can be unpacked from test.herf with "dictionary" action, contains file names for unpacking')

args = parser.parse_args()

herf = args.herf
out_dir = args.dir+'/'

if args.action == 'unpack':
names = {}
if 'e' in args:
# Get names
with open(args.e, 'rb') as file:
file.seek(4)
files = file.read(4)
files = unpack('<I', files)[0]
I = 0
names = {}
while I < files:
entry = file.read(132)
id, name = unpack('<i128s', entry)
names[id] = name.strip('\x00')
I += 1

# Extract files
with open(herf, 'rb') as file:
file.seek(4)
files = file.read(4)
files = unpack('<I', files)[0]
I = 0
info = []
while I < files:
entry = file.read(12)
id, size, offset = unpack('<iii', entry)
if id in names:
name = names[id]
else:
name = pack('<I', id).encode("hex") + '.dat'
current = file.tell()
file.seek(offset)

with open(out_dir + name, 'w') as out_file:
out_file.write(file.read(size))

file.seek(current)
info.append((pack('<I', id).encode("hex"),name))
I += 1
writer = csv.writer(open(herf + '.unpack', 'w'))
writer.writerows(info)
elif args.action == 'pack':
reader = csv.reader(open(herf + '.unpack', 'r'))
info = []
for row in reader:
id = row[0].decode("hex")
path = out_dir + row[1]
size = getsize(path)
info.append((id, path, size))
files = len(info)

with open(herf, 'wb') as file:
# Header
file.writelines(('\xC0\xA5\xF1\x00', pack('<I', files)))
offset = (files*12)+8
for item in info:
file.write(item[0])
file.write(pack('<I', item[2]))
file.write(pack('<I', offset))
offset += item[2]
# 4-byte padding
while offset % 4 != 0:
offset += 1

# Write data itself
for item in info:
with open(item[1], 'rb') as input_file:
file.write(input_file.read())
while file.tell() % 4 != 0:
file.write('\x00')
elif args.action == 'dictionary':
with open(herf, 'rb') as file:
file.seek(4)
files = file.read(4)
files = unpack('<I', files)[0]
I = 0
while I < files:
entry = file.read(12)
id, size, offset = unpack('<iii', entry)
if id == -360542764:
file.seek(offset)
with open(out_dir + 'erf.dict', 'w') as out_file:
out_file.write(file.read(size))
exit()
I += 1
print "Sorry, no erf.dict found inside {0}!".format(herf)</div>

 

I sorted all strings in name by id and... wow, they really cut down the first cutscene. In this order lost strings explain far more. Look youself. It's a great pity we didn't see it in game.

Also, what is it?

The fifth race? We first thought it could be prototype name for Voxai Colony Beta, but it's already mentioned earlier. And string with id 19135 is missing here, which could be "Voxai Beta Colony".