SIDin #5 — Music Engines Pattern Searching, part 2: the Lostack database (Stefano Tognon, 2004)

The format of Lada 'Ray' Lostak's database of about 450 C64 music engines: Player blocks of Name, 3–16 ID=offset:value pairs, optional Messages and the addresses where the player keeps its track and pattern positions (OrderPos, SecPos) — where the offsets come from (the C64 tool AMS), why they are relative, what the positions are good for (a live display, the true length of a tune), and a complete worked example on Future Composer 1.0. Second half of player-identification.

Key claims#

• Format — one block per engine, C-like, the { on the Player line:

• Relative offsets and the AMS heritage: ID=$0000:$10 / $0001:$11 / $0003:$12 / $0006:$13 equals the Perl pattern '\x10\x11.\x12..\x13' of part 1. The database "derived from the Advanced Music Searcher AMS 5.0.415", a C64 program that scans memory for loaded engines by applying the IDs at fixed start addresses; if it steps $1000 bytes, an entry with all offsets shifted by one finds a player at $1001 that the unshifted entry misses, and only a one-byte step makes the two equivalent — so an ID offset is relative to wherever the search is applied, and AMS most probably stepped $100, "as usually a player can be relocated in fixed memory location: you will found that a player locates the tune at $1000 but not a $1001".
• Which offsets for which program: on a real C64 the offsets must be relative to $100/$200 steps and a full scan is slow — but since C64 music now lives in PSID/RSID files "there are no needed of use this database in the real C64"; tools like PSID64 could compute the engine on the PC while converting (sid-format); SID players, C64 emulators and rip tools run on a PC and can search byte by byte. The uses he wants: an emulator option that scans emulated memory for known engines ("if a ripper know the engine he is being ripped it will be more simple his work"), the same in tools like PSIDedit, and the engine name shown in the SID player. Absolute offsets exist (ID=#$0300:$10 = the byte $10 at memory $0300) but "you must not use this technique anymore": only programs with an emulated C64 memory can test them; SIDedit, which handles the PSID file itself, cannot.
• Order/Sector, in tracker terms — Track v1: 01 02 03 01 01 00, Sector 01: ins1 dur3 C4 C5 D4 D5 — the player stores where it is in each; OrderPos=$0921,$0922,$0923, SecPos=$0924,$0924,$0926. Uses: "a sidplayer that shows in realtime what the player is doing", and "Detect the true length of the music": track ends use special values, "usually $FF and $FE", one restarting the track, the other stopping the voice; a restart shows as OrderPos jumping back, and "only when all the voices are restarting we can saw that the tune is finished" (Track1: 01 02 03 FE / Track2: 05 05 06 07 07 07 FE / Track3: 08 08 FE — complete when voice 2 wraps); with the stop value the tune is over "as soon as the SecPos value did not change anymore after the OrderPos value where changed". Players with one track for all voices (Common Track: 01 02 03 …, each sector holding all three voices) get the same address three times: Name=TAXIM? with ID=$0001:$12 / $0002:$C0 / $0004:$0D, OrderPos=#$02C1,#$02C1,#$02C1, SecPos=#$02C0,#$02C0,#$02C0 (orderlist).
• Example — Future Composer 1.0 (future-composer): ID=$0006:$AD / $0007:$74 / $0009:$C9 / $000A:$02, OrderPos=$0921,$0922,$0923, SecPos=$0924,$0924,$0926; "there are more than 1000 tunes in HVSC that use this player". In /VARIOUS/G-L/Luca/Moonlight_shadow.sid, loaded at $1800, the file starts JMP $2108 / JMP $2117 / LDA $2174 / CMP #$02 — the IDs are the opcode $AD, the operand's low byte $74, the opcode $C9 and the immediate $02: "it is used a low byte of an LDA instruction: as the tune can be relocated, using even the high byte cannot detect all the tunes", the low byte being safe because "usually the tunes are relocated in step of $100 bytes". OrderPos is $2121,X (LDY $2121,X ; read OrderPos), SecPos $2124,X; the track byte is compared with $FE and $FF (both "end mark"), $FF clearing both positions and $FE storing 2 into the flag the init tests, then a byte with bit 7 set is masked AND #$1F and one with bit 6 set AND #$3F before a pattern pointer is fetched. "The best way for finding what addresses are OrderPos and SecPos is to search for the mark indicator ($FF, $FE), as they are very commons."
• Limits and next steps: "It has a wonderful potential, but unfortunately too much engines are recognized with only 4 IDs (like in the above example), as the pattern derives from the AMS database. These can produce many false detections if you use a one by one step in searching." Planned: an extended syntax (relative addresses inside instructions, "using the actual Program Counter register"), tools that suggest a usable pattern, and "the Simon White scanner being developed is able to scan all the HVSC for 450 engines in less than 5 seconds!" The Lada player (www.unreal64.net) is "actually the unique one that take use of the database"; it is GPL, and Ray e-mails the latest version to contributors.

Practical takeaways#

• Every pattern player keeps a track index and a pattern index per voice; find them by searching for the $FF/$FE compares — the same knowledge a ripper uses to find a tune's loop point (sid-player-routine).
• Four bytes from an init routine identify a player only if the scan is aligned to pages; byte-by-byte scanning needs longer signatures — which is what sidid's list later provides.
• A per-voice track position that wraps for all three voices = the tune's true length; a SecPos that stops moving = the tune has ended.

Notable quotes#

"It has a wonderful potential, but unfortunately too much engines are recognized with only 4 IDs."

Relevance to the wiki#

The database half of player-identification; the FC 1.0 signature and track-byte handling for future-composer; OrderPos/SecPos for orderlist and sid-player-routine; the 450-engines / 5-seconds / 90 % figures for hvsc and sidid; the "you can use the players for founding the right pattern" link to hvmec.

To verify#

• (unverified) SecPos=$0924,$0924,$0926 repeats $0924 for voices 1 and 2 although the code indexes $2124,X per voice, so the second value is probably a typo for $0925 in the database or the article.
• (unverified) The track bytes masked AND #$1F (bit 7 set) and AND #$3F (bit 6 set) look like the transpose and repeat commands of Deenen's musicfile sequences, which would support the "FC is a MoN player" claim on future-composer; Tognon does not say so.

Pages touched#

player-identification · sidid · future-composer · orderlist · sid-player-routine · hvsc · hvmec · sid-format · chordian · sidin