Zumi's Scratchpad

The GBS tagging rabbit hole

April 26, 2023 — updated on Jun 29 '23 at 10:57

This is a continuation of a previous post on GBS ripping.

Ah, here we go again.

GBS is definitely one of the game music formats of all time. Other formats have some sort of tagging standard, whether it be a new format or an extension. NSF has NSFE and NSF2. SID has the STIL. What about GBS?

Let's look at the GBS specification for a moment. One thing you'll notice is that GBS leaves no space for anything else but the bare basics for song playback. There's not even any "reserved" space defined here, unlike those other formats. Basically no room for anything else, especially the possibility of tagging. In the meantime, here's the "standards" I'm currently aware of:

The NEZPlug format

I've already touched on this on the aforementioned previous post, but here's a refresher for GBS rips:

• It's basically a 7zip archive.
• The archive contains the GBS itself, and a bunch of M3U playlists.
• The M3U playlists are plain text files using NEZPlug's "Extended Playlist" format as defined in the in_nez documentation.
• The M3U is split into a "main playlist" listing all the tracks in the GBS, and several M3U files for each of the tracks.
• The M3U has comments (usually only in the main playlist) that list information about the rip, for example:

  # @TITLE       Pokemon Picross
  # @ARTIST      Jupiter
  # @COMPOSER    Toshiyuki Ueno
  # @DATE        1999
  # @RIPPER      Zumi
  # @TAGGER      ppltoast, Zumi
  

  (Worth noting that that these comments are not part of the NEZPlug spec and isn't parsed by it—instead it's what's being used de-facto for some reason. Knurek's rips follow this format)

Now, this format assumes you're playing it using NEZplug and Winamp. While Winamp users are based llama lovers, who else uses it (and especially with NEZplug) to listen to some video game music? I feel like most people switched to foobar2000 and its Game Emu Player (foo_gep) plugin—me included.

There are some oddities of playing these kinds of M3U playlists with foo_gep. First, you don't drag the main M3U file into foobar. Instead, you drag the GBS rip and it'll pick up tags from the M3U file automatically. That's file—singular—file, because dragging the other individual M3U files will not work. Second, there seems to be indexing differences between the two, which is quite annoying. That is, song indices seem to start at 1, while NEZplug's documentation specifies that GBS stuff should be indexed at 0 (and in_nez does act accordingly).

Figured it was time to investigate…

Looking at Game_Music_Emu

foo_gep's backend is a library called Game_Music_Emu (GME), I believe originating from blargg with the latest version at 0.5.2, continued by kode54 (apparently at 0.6.x) and as of this writing the latest version is Michael Pyne's version 0.6.3.

It's a solid backend for playing video game music, but for this purpose, I care about how it reads M3U files. Some inspection (and experimentation with both foo_gep and GME's provided demo player) says that unlike NEZplug, this only has 1-indexed track numbers for every format. More consistent, sure, but it does present problems for people who still use NEZplug for whatever reason.

The sensible thing to do here is file a bug and ask if this is intended behavior. Problem is, the issues tab of the maintained repo seems to be completely overrun with spam.

Then I went to foo_gep and found out that it's an archived version (since the maintainer has quit from foobar2000 extensions) and its linked source code repo is a dead link. However—looking at the link through the Wayback Machine—I found a commit detailing about "fixing M3U with 1-based indexes into formats with 0-based indexes", but I can't look at the commit itself, so I'm not sure.

And I'm not about to fork GME and try to fix it myself, so I suppose it's time to look at some other formats.

VGM

Ew, VGM. Okay, so VGM is usually associated with Sega video game music… probably because there's not much else (GYM only does YM2612 and poorly) and the hardware can handle these fine. It's like the opposite situation where there's no reliable dumping format (except maybe for games that use SMPS) so this is the next best thing.

VGM has support for several chips but is more of a register dump / log. Yes, it has support even for systems that already have ripping formats like the NES (nsf) and Game Boy (gbs). Seeing VGM "rips" for those systems feels cursed. (Apparently it's less preferred anyway since emulation bugs may creep in to the VGM log, though hardware playback may help)

But the one thing VGM has is an actual, "well-formed" tagging system known as GD3. Consistent, somewhat reliable, covers most of the relevant bases.

Again, VGM is kinda… eh (and I also don't currently have emulators to dump them in the first place). It's understandable why it's been a popular choice—you don't need to disassemble or hunt down sound engine playback code, so they're easy to make. Just hit "Dump VGM", wait, and perform post-processing like finding looping points and whatnot.

An idea I had is to provide VGM with the M3U+GBS, but it's not a good idea, I think. Why provide two copies of the same thing in one package?

There's yet another option to consider, however…

GBSX

GBSX was the proposed extended header format for GBS files. It was supported by GBSPlay up until commit eaaa3714 (2021-01-06), after which it was removed due to lack of any traction. Sad, really.

Today, GBSPlay still supports loading tags, but only for VGM. (GD3 tags are only read when loading a VGM)

To think that literally nobody is even aware of GBSX, which is such a shame as this could have been very useful. It may not be perfect, it may contain more basic info, but goddamn it would be much better than having to load up real ancient plugins!!

So, I was determined to make this work.

I checked out and built the final commit that had GBSPlay support and cooked up a script that would embed GBSX tags in the GBS file. Now, a few comments on building these tags:

1. The spec says that GBSX can be separated into its own file, and there's a proposal on how one can do that. However, I can't seem to find any indication that this is actually implemented into GBSPlay. Disappointing, but I'm fine with embedding it into the GBS anyway.
2. Seems like extra hassle to require serializing a CRC32 checksum. I get why it's in the design, but like… why?

   For reference—CRC32 checksums (standard algorithm, like ZIP) are needed for the original GBS file and the entire extended header with its CRC slot zeroed out.

3. Adding the GBSX header requires padding the GBS length to some multiple of 16. The offset to the GBSX, minus the 0x70 of the GBS's standard header, then divided by 0x10—is then written to at address 0x6e. (For whatever reason, before writing it, 0x100 seems to be added to this value)
4. (Insert something about encodings here). Speaking of encodings, I just said fuck it and used \r\n for the new line endings.

After all that:

Not much info is displayed. But it's a good sign. Although yes, it seems to ignore the track time and fades out where it would be by default. I'm not sure why, but whatever, at least it actually reads them.

A Distribution Compromise

So, knowing what I know now I decided to try these three things to my new distribution—a rip of the unreleased Katakis 3D (done by zlago):

1. Add GBSX tags to the GBS rip. Any player (if any) can read these tags if M3U support is not available.
2. Use 1-indexed track numbers in the main M3U playlist. The rationale is that foobar2000 reads this file as 1-indexed. Yes, it will assign the wrong tracks when loaded in NEZPlug.
3. Use 0-indexed track numbers in the individual M3U playlists. Rationale being that only NEZPlug can load these files, so… drag all of them into Winamp than the one main M3U I guess.

Last two may make the distribution feel cursed, but I'm trying dammit lmao.

As before, I generate these with the help of Python script to process a few JSON files to make the process more streamlined. Here's the tools used for this rip, a bit more versatile than the one I used previously: gbstools

The tools have only two real dependencies: jsonschema and py7zr, both of which are a pip install away.