SIDin #3 — Music Engine pattern searching, part 1 (Stefano Tognon, 2003)

How to tell which music routine a .sid was made with: scan the file for a byte sequence that only that engine's code contains. Tognon gives the method (with source: pick characteristic code; without: compare disassemblies of tunes known to use the player), the wildcard rules, and Perl patterns for the galway-player, the matt-gray-player and JCH NewPlayer v1 — the working notes behind player-identification and, years later, sidid.

Key claims#

• The motive: "it is more interesting to know with witch routine the sound was made". The concept: "scan a file to found if it contains some bytes in sequence that we know that only a file made by a particular music engine can have". The hard part is finding a representative pattern; two cases — with a source (released by the author, or reverse-engineered) or without.
• With source: look at the code that handles the player's own instruments, patterns and note representation, or at how portamento, vibrato and arpeggio are made. Galway: pattern values above $C0 are flow instructions, so the fetch LDY #$00 / LDA ($xx),Y / CMP #$C0 / BCC — Perl '\xA0\x00'.'\xB1.'.'\xC9\xC0'.'\x90' — finds the engine with six fixed bytes and one wildcard; "As 7 bytes are very little, maybe some other music engines completely different can have the same code part with other meanings". Matt Gray (from the Driller source printed in #2, s-sidin02-matt-gray-driller): the code that reads the next pattern value and tests the portamento flag — INC $xxxx,X / BNE / CMP #$FB / BCC / CMP #$FB / BNE / LDA #$01 / STA $xxxx,X / INY / INC $xxxx,X / LDA ($xx),Y, i.e. '\xFE..'.'\xD0.'.'\xC9\xFB'.'\x90\x20'.'\xC9\xFB'.'\xD0\x18'.'\xA9\x01'.'\x9D..'.'\xC8'.'\xFE..'.'\xB1' — "a good point as the code is very characteristic".
• Without source: disassemble a tune you know is from the player ("the author say that or you have generate a test tune from the player editor just for this"), pick a stretch, and prefer one "not used by the other versions of the same music player". Worked example: JCH NewPlayer v1, v2 and v3 from Hoppin (v1), Brown Ice (v1), Cavern (v2) and 2Cvee (v3), disassembled with SIDedit, pasted into a spreadsheet and aligned row by row — easy for successive versions "as the new versions probably have little more features before the previous". The two v1 tunes differ slightly (the second "seems to be an improvement of the fist"); the v1 signature is code present in both but in neither later version; v2 and v3 get their own stretches. "What the selected code in the pattern means for the player we are not able to say: this require an accurate reverse engineering work, but we only need a pattern that find that player." Test against every tune known to use the engine; a miss means "a minor variant that need further investigation".
• Writing the pattern: take bytes and mnemonics from SIDedit — the v1 stretch is $12BC–$1302 in Hoppin: LDY $1677,X / LDA $16E9,Y / PHA / AND #$0F / STA $1722,X / PLA / AND #$F0 / LSR / LSR / STA $1702,X / STA $102C,X / LDA #$00 / STA $16EF,X / SEC / SBC $102C,X / … / LDA $16ED,Y / PHA / AND #$F0 / CMP #$10 / BNE $133E, 32 instructions. Rules: absolute addresses "are always relative to the point where the player was linking in memory, so they are not to be found" — wildcard them (BD ?? ?? B9 ?? ??); keep only addresses in I/O space "like D4xx (the sid chip)"; wildcard zero-page addresses too, since "maybe some player let the user to choose the zero-page locations"; drop a branch's offset when the target lies outside the pattern. In Perl . is any byte and \x a hex byte; the test is one line, if ($data =~ /$Search/s) { print "Found!!!\n"; } with $data the body of the tune.
• Next: Lada 'Ray' Lostak's engine database (www.unreal64.net), "over 450 engines"; "the main goal is to will be able to find the engines of more that 90% of HVSC tunes!" (hvsc).

Practical takeaways#

• A signature is opcodes plus immediates plus $D4xx; everything else is ... Relocation, not the song data, is what breaks naive byte matching.
• To fingerprint a player you do not understand: lay the same routine from two or three known tunes side by side and take what the version shares and its siblings lack.
• The idioms he picks are the ones the wiki's driver pages describe — Galway's ≥ $C0 command bytes (galway-player), Gray's $FB/$FC portamento bytes (matt-gray-player).

Notable quotes#

"We only need a pattern that find that player."

Relevance to the wiki#

First half of the method on player-identification; fingerprints for galway-player and matt-gray-player; the JCH NewPlayer case study for chordian; the ancestor of sidid's signature list; the relocation rule for sid-player-routine.

Pages touched#

player-identification · sidid · sid-player-routine · galway-player · matt-gray-player · chordian · hvsc · sidin