Captain's Log: Stardate 78592.1

The buffer-clearing saga

Adding the new AnukariEffect plugin has ended up precipitating a lot of improvements to Anukari, because it pushed me into testing what happens when multiple instances of the plugin are running at the same time. Most of my testing is done in the standalone Anukari application. It loads extremely quickly, so it's nice for quickly iterating on a new UX change, etc. But in reality, it's likely that users will mostly use Anukari as a plugin, so obviously I need to give that configuration ample attention.

The last big issue I ran into with the plugin was that in GarageBand, loading a song that had something like 6 instances of Anukari and AnukariEffect, sometimes one of the instances would mysteriously fail. The GPU code would initialize just fine, but GPU call to process the first audio block would fail with the very helpful Metal API error, Internal Error (0000000e:Internal Error), unknown reason.

After some research, it turned out that to get a more detailed error from the Metal API, you have to explicitly enable it with MTLCommandBufferDescriptor::errorOptions, and then dig it out of the NSError.userInfo map in an obscure and esoteric manner. So I had my intern (ChatGPT) figure out how to do that and finally I got a "more detailed" error message from the Metal API: IOGPUCommandQueueErrorDomain error 14.

If you've followed my devlog for a while, it should come as no surprise that I am a bit cynical about Apple's developer documentation. So I was completely unsurprised to find that this error is not documented anywhere in Apple's official documents. Apple just doesn't do that sort of thing.

Anyway, I found various mentions of similar errors, with speculation that they were caused by invalid memory accesses, or by kernels that ran too long. I used the Metal API validation tools to check for any weird memory access and they didn't find anything weird. I figured they wouldn't, since I have some pretty abusive fuzz tests that I've run with Metal API validation enabled, and invalid memory access almost certainly would have shown up before.

So I went with the working hypothesis that the kernel was running too long and hitting some kind of GPU watchdog timer. But this was a bit confusing, since the Anukari physics simulation kernel is, for obvious reasons, designed to be extremely fast. With some careful observation and manual bisection of various code features, I realized that it was definitely not the physics kernel, but rather it was the kernel that is used to clear the GPU-internal audio sample buffer.

Some background: Anukari supports audio delay lines, and so it needs to be able to store 1 second of audio history for each Microphone that might be tapped by a delay line. To avoid allocations during real-time audio synthesis, memory is allocated up-front for the maximum number of Microphones, which is 50. But also note that there can be 50 microphones per voice instance, and there can be 16 voice instances. Long story short, the per-microphone, per-instance, per-channel buffer for 1 second of audio is about 300 MB, which is kind of huge.

It's obvious that clearing such a buffer needs to be done locally on the GPU, since transferring a bunch of zeros from the CPU to the GPU would be stupid and slow. So Anukari had a kernel that would clear the buffer at startup, or at other times when it was considered "dirty" due to various possible events (or if the user requested a physics reset).

Now imagine 6 instances of Anukari all being initialized in parallel, and each instance is trying to clear 300 MB of RAM -- that's multiple gigabytes of memory write bandwidth. And sometimes one of those kernels would get delayed or slowed enough to time out. The problem only gets worse with more instances.

Initially I considered a bunch of ideas for how to clear this memory in a more targeted way. We might clear only the memory for microphones that are actually in use. But then we have to track which microphones are live. And also, the way the memory is strided, it's not all that clear that this would help, because we'd still be touching a huge swath of memory.

I came up with a number of other schemes of increasing complexity, which was unsatisfying because complexity is basically my #1 enemy at the moment. Almost all the bugs I'm wrangling at this point have to do with things being so complex that there were corner-cases that I didn't handle.

At this point you might be asking yourself: why does all this memory need to be cleared, anyway? That's a good question, which I should have asked earlier. The simple answer is that if a new delay line is created, we want to make sure that the audio samples it reads are silent in the case that they haven't been written yet by their associated microphone. For example, at startup.

But then that raises the question: couldn't we just avoid reading those audio samples somehow? For example, by storing information about the oldest sample number for which the data in a given sample stream is valid, and consulting that low-watermark before reading the samples.

The answer is yes, we could do that instead. And in a massive face-palm moment, I realized that I had already implemented this timestamp for microphones. So in other words, the memory clearing was completely unnecessary, because the GPU code was already keeping track of the oldest valid audio sample for each stream. I think what happened is that I wrote the buffer-clearing code before the low-watermark code, and forgot to remove the buffer-clearing code. And then forgot that I wrote the low-watermark code.

Well, that's not quite the whole story. In addition to the 50 microphone streams, there are 2 streams to represent the stereo external audio input, which can also be tapped by delay lines (to inject audio into the system as an effect processor). This data did not have a low-watermark, and thus the clearing was important.

However for external audio, a low-watermark is much simpler: it's just sample number 0. This is because external audio is copied into the GPU buffer on every block, and so it never has gaps. The Microphone streams can have gaps, because a Microphone can be deleted and re-added, etc. But for external audio, the GPU code just needs to check that it's not reading anything prior to sample 0, and after that it can always assume the data is valid.

Thus ultimately the fix here was to just add 2 lines of GPU code to check the buffer access for external audio streams, and then to delete a couple hundred lines of CPU/GPU code responsible for clearing the internal buffer, marking it as dirty, etc. This resulted in a noticeable speedup for loading Anukari and completely solved the issue of unreliable initialization in the presence of multiple instances.

Pre-alpha release 0.0.13

With the last reliability bug (that I know of) solved, I was finally able to cut a new pre-alpha release this Friday. I'm super stoked about this release. It has a huge number of crash fixes, bug fixes, and usability enhancements. It also turned out to be the right time to add a few physics features that I felt were necessary before the full release. The details of what's in 0.0.13 are in the release notes and in older devlog entries, so I won't go into them here, but this release is looking pretty dang good.

The next two big things on my radar are AAX support and more factory presets. On the side I've been working to get the AAX certificates, etc., needed to release an AAX plugin, and I think that it should be pretty straightforward to get this working (famous last words). And for factory presets, I have about 50 right now but would like to release with a couple hundred. This is especially important now that I've added AnukariEffect, since only a couple of the current presets are audio effects -- most of them are instruments. So I'm kind of starting from scratch there. I think it's pretty vital to have a really great library of factory presets for both instruments and effects, and also, working on them is a great way to find issues with the plugin.

Captain's Log: Stardate 78573.2

Automatic Bypassing Workaround

While testing the new AnukariEffect plugin in various DAWs for compatibility, I found that it was doing some very strange stuff in GarageBand (and Logic Pro, which seems to share the same internals). I had noticed weird stuff in GarageBand before even with the instrument plugin, and had a TODO to do a deep dive, so I figured that now was as good a time as any to finally get the plugin working well with Apple's DAWs.

What I had seen in the past with the Anukari (instrument) plugin was that sometimes the physics simulation would inexplicably stop working. I had seen this at GarageBand startup, but also after it had been open for a while. I couldn't see any reason in Anukari's logs for the problem, and occasionally it would just start working again. But this was fairly rare and I hadn't had time to find a way to reproduce it.

But with the AnukariEffect plugin, this was happening constantly. Since it was easy to reproduce, I pretty quickly found out that GarageBand will simply stop calling into the plugin's ProcessBlock function, which is where audio processing happens, and in Anukari, is where the physics simulation occurs.

It turns out that GarageBand is extremely aggressive about this. It has some heuristics about when a plugin is no longer producing audio, and at that time it will stop calling into the plugin, to save CPU/power. For example, for an instrument, if it hasn't received MIDI input in a while it might be automatically bypassed. And for an effect, if the track is not playing or the effect is not receiving audio input, it will be automatically bypassed.

This is reasonable behavior, and other DAWs do it too, but for example in a VST3 plugin (as opposed to an AudioUnit) the plugin can specify the number of "tail samples" as being kInfiniteTail, in other words, the plugin can state "I might keep generating audio samples forever even without input." VST3 plugins can also set their sub-type to Generator, which also communicates to the DAW that they might continue to generate audio even without input. (Note that an AudioUnit can be a generator at the top level, but instruments/effects can't also be generators. Which is a pretty big oversight.)

And other DAWs that do aggressive automatic bypassing, like Cubase or Nuendo, they provide the option to disable the feature. But of course having a knob like this is anathema to Apple, especially in GarageBand, and thus it cannot be disabled.

Anyway, for an instrument or effect plugin running in an Apple DAW, aggressive automatic bypassing is just a fact of life. And if that plugin is a continuous physics simulation like Anukari, this is a huge problem, because the simulation part of the plugin will become unresponsive, and furthermore, weird discontinuous things may happen if it is bypassed and un-bypassed at inopportune moments.

So as usual for working with Apple, the solution to Apple's oversimplification of the problem is to push more complexity into the non-Apple software: Anukari now can detect that it has been automatically bypassed, and will seamlessly transfer ownership the physics simulation to a background thread. When DAW processing resumes, it seamlessly transfers ownership back. This is optional (but highly recommended), so users who really need to save power can disable it.

This really is much more complicated than I'd like, partly because Apple doesn't provide any indication that the plugin is bypassed. From what I can tell there's no notification whatsoever, except that ProcessBlock stops getting called. So detecting this condition requires a keepalive timer and a background thread that monitors it. Once it detects that ProcessBlock hasn't run for too long, it begins running the simulation directly. Then when ProcessBlock resumes being called, it detects that the keepalive is fresh again and stops.

There are some very tricky details here to do all this reliably in the real-time audio thread without priority inversion issues with a mutex. The keepalive timer is an atomic, and the monitoring thread never acquires the mutex unless the keepalive is stale. This does mean that the audio thread has to acquire the mutex for each audio block, but because we use the atomic keepalive timer to guarantee that the mutex will never be contended, this is OK, because on all the platforms where Anukari will run, an uncontended mutex acquisition is simply an atomic CAS operation. (This is a great tip I learned from Fabian Renn-Giles in his excellent ADC23 talk.)

There is one moment where the audio thread's mutex acquisition could be contended, which is when the automatic bypass is being lifted. The monitoring thread may be holding it while running the simulation itself. This is not a big deal though, because the monitoring thread releases the mutex after simulating each small sample block. The audio thread will try to acquire the mutex, fail, and return a silent buffer. But in doing so it will update the keepalive timer, and next time it runs it will acquire the mutex without contention. The reason this dropped block is not a big deal is that we're coming back from being bypassed anyway -- this just adds a few samples of latency before audio starts. No problem.

There's one last detail, which is that while all the above complexity keeps the physics simulation running, so the user can continue to interact with the plugin, there will not be any audio output while it's bypassed. This cannot be fixed, and could be a little confusing. So Anukari now displays a pulsating "BYPASSED" message on the master output level meter when it is in this state. And that message has a tooltip explaining how the DAW is doing potentially annoying things.

Less Waste Still Makes Haste

In my previous post Waste Makes Haste I wrote about how Anukari has to run a spin loop on a single GPU core to convince MacOS to actually clock up the GPU so that it performs well enough for Anukari to function.

That workaround continues to be extremely effective. However, while testing multiple Anukari instances, I realized that each instance was running a spin loop on the GPU, so e.g. 4 instances would run 4 spin loops. Running the one spin loop is pretty stupid, but gets the job done and is well worth it. But running 4 spin loops is purely wasteful, since only the 1 is required to get the GPU clocked up.

Fixing this requires coordination among all Anukari audio threads within the same process. Somehow a single audio thread needs to run the spin loop, and the others need to just do regular audio processing. But if the first thread running the loop is e.g. bypassed, another thread needs to pick up the work, and so on.

I ended up devising another overly-complicated solution here, which is to use another shared atomic keepalive timer. Each audio thread checks it periodically to see if it has expired, and if so, attempts a CAS to update it. If that CAS fails, it means some other thread got to it first. If the CAS succeeds, it means that this thread now owns the spin loop and needs to keep updating the keepalive. There are some other details, but this algorithm turned out to be mercifully easy to get right with just a couple of CAS operations and a nano timer. (And it doesn't even require that each thread sees only unique nano timestamps!)

An alternative solution would be to have an entirely separate thread run the GPU spin loop, instead of having an audio thread be responsible for tending to it. This could also be a good solution. However it would require its own tricky details so that the spin loop would pause when all audio threads were bypassed. And also it would require initializing some Metal state that each audio thread already initializes anyway. I will probably keep the current solution unless it proves unreliable, in which case I'll move to this alternative.

Mouse Cursor Hell

The last workaround I spent time on this last week is making custom mouse cursors work well in GarageBand and Logic.

Since Anukari has a somewhat sophisticated 3D editor, custom mouse cursors are very useful for helping make it clear what is happening. So for example, when the user drags the right mouse button to rotate the camera, the mouse cursor changes to a little rotation icon for the duration of the drag, and then goes back to being a pointer when the button is released.

Or, well, it goes back to being a pointer in every DAW except GarageBand and Logic, because of course Apple is doing something fucking weird with the mouse cursor in their DAWs. Humorously, as I investigated this issue, I discovered that the mouse cursor often gets stuck in GarageBand/Logic even without plugins, and that users have been complaining about this for at least 10 years. One user in a forum post basically said, "don't worry about the busted mouse cursors so much, you just get used to it." So Apple has been ignoring an obvious mouse cursor bug for a decade. Sounds about right.

Anyway, I narrowed down the problem to the fact that changing the mouse cursor using [NSCursor set] inside of a mouseDown or mouseUp event sometimes doesn't work. Err, it does work, in the sense that the call succeeds, and if you call [NSCursor currentCursor] it will return the one you just set. But visually the cursor will not change.

I tried about a billion things, and ultimately ended up with a workaround that force-sets the mouse cursor inside the next mouseMove or mouseDrag event (described in more detail here on the JUCE forums). This is not perfect, but it's pretty good, and much better than no workaround at all.

Yikes... as you can tell I'm pretty sick of dealing with compatibility with Apple's DAWs. But I'm not done yet. There are two more Apple-specific issues that I'm aware of, which hopefully I can address over the next week.

Captain's Log: Stardate 78324

I've been really digging into MacOS optimizations over the last few days. Being ALU-bound is quite a pain in the butt, because unlike being memory-bound, there are a lot fewer big changes I can make to speed things up. Mostly I've been working on instruction-level optimizations, none of which have had a big impact. I've gotten to where I don't see anything else that is really worth optimizing at this level. I've confirmed that by simply commenting out pieces of code and measuring the speedup from not running it, which gives me an upper bound on how much optimizing it could help. Reducing computation at this point is not going to be done at the instruction level.

I have a couple ideas for larger structural/algorithmic changes, but before I move on to this, I want to eliminate a few other issues that I've noticed with MacOS.

The biggest pain I've run into is the fact that MacOS gives the user very little (almost no) control over the power/performance state of the machine. I guess this is user-friendly, as Apple magically figures out how best to clock the CPU and GPU up and down to save power, but it turns out that Apple is doing this really badly for Anukari's use case.

From what I can tell, Apple's GPU clock throttling is based on something akin to a load average. This is a concept I originally ran into from Unix, where the load average is, roughly speaking, the average number of threads that are awaiting execution at any moment. If this number is higher than the number of CPU cores, it means that there's a backlog: there are regularly threads that want to run, but can't.

On the Apple GPU, it seems that the OS won't clock it up until there's a command queue with a high load average. Which makes sense for throughput-sensitive use cases, but doesn't work at all for a latency-sensitive use case. For example, for graphics, as long as the command queue isn't backing up (load average > 1), there's no need to up-clock the GPU: it is keeping up. But for Anukari, this heuristic doesn't work, because if it ever hits a load average even close to 1, that means that there's no wallclock time left over for e.g. the DAW to process the audio block, etc. It's already too late, and audio glitches are regularly occurring even at something like 0.8.

This is a serious problem. I asked on the Apple Developer Forums (optimistic, I know), and though I did get a thoughtful response, it wasn't from an Apple developer, and it didn't help anyway. I ran numerous experiments with queuing more simulation kernels in advance, but ultimately none of these ideas helped, because ultimately I can't really generate audio in advance because I need up-to-date MIDI input, etc.

Ultimately, the solution that I am going with is probably the stupidest code that I will have ever shipped to production in my career: Anukari is dedicating one GPU threadgroup warp to useless computation, simply to keep it fully saturated to communicate to the OS that the GPU needs to be up-clocked. So in this case, waste makes haste.

But while this is incredibly stupid, it is also incredibly effective. I got it working with the minimal amount of power usage by dedicating the smallest amount of compute possible to spinning (not even a whole threadgroup, just one warp). And it immediately gets the OS to clock up the GPU, which decreases the audio computation latency by about 40%.

My new golden test latency stats framework doesn't show as much of a gain as I see when using the plugin, because it was already running the GPU with smaller gaps, as it doesn't need to wait for real time to render the audio. But even in the tests, the performance improvement is dramatic. In this graph you can see the overall real time ratio for the tests. The huge drop at iteration 15 is where I implemented the GPU spin:

Here is a graph with all the tests broken out individually:

At this point, on my Macbook M1, the standalone app performance is very usable, even for quite complicated presets like the slinky-resocube.

But... of course nothing is ever simple, and somehow while these fantastic performance gains are very observable when running as a VST3 plugin in Ableton, they are not nearly as visible when running as an AU plugin in GarageBand. I don't know what is up here, but it needs to be the next performance issue that I address, because if I can get past this, I might have performance at a "good enough" level on MacOS to start spending my time elsewhere for a while.