Captain's Log: Stardate 79645.8

I finally got the 0.9.26 release out yesterday (download, release notes), and I'm really happy with everything I managed to get into it.

Anukari's RAM use had been bothering me for some time. A kind of funny thing is that I became especially aware of it after I moved the audio processing from the GPU to the CPU.

Delay Line Buffer Optimization

Anukari supports delay lines of up to 1 second in duration. Delay lines run from the virtual microphones to audio input exciters. So each mic needs to store 1 second of audio sample history. Anukari supports up to 50 mics, and to complicate things further, in polyphonic mode, there may be up to 16 instances of each mic. So storing 1 second of 32-bit precision audio data at 48 kHz for 16 instances of 50 mics requires ~154 MB of RAM.

Since the physics simulation is running in a realtime audio thread, it is highly undesirable to do dynamic memory allocation. Generally speaking, memory allocation is a blocking operation, and often will acquire a mutex, and even if it doesn't, it might take a while if memory is fragmented, etc. So for any memory blocks used by the audio thread, at startup Anukari allocates buffers that are of the maximum required size. This eliminates the need for dynamic allocation at runtime, but insofar as those buffers go unused, it wastes memory.

When Anukari's physics simulation ran on the GPU, the 154 MB delay buffer was allocated in GPU memory. This was actually kind of handy, because most audio applications don't use much GPU memory, so in some ways this memory was "free." Of course that doesn't apply to Apple's unified memory architecture. But for many users, this RAM usage was not really visible.

When I moved the simulation to the CPU, 154 MB of RAM usage that was sometimes hidden before became quite visible. Especially for users running many instances of Anukari in parallel, this can add up pretty quickly.

For the 0.9.26 release I decided it was time to do something about this. Most Anukari presets have fewer than 50 mics, don't use all 16 voices of polyphony, and don't have delay lines attached to every single mic. In fact many presets don't even have a single delay line, so this 154 MB of RAM is completely wasted.

Now, could I get away with just dynamically allocating this memory in the audio thread? Probably. The allocation would only happen when the user adds a delay line to an existing preset, increases the polyphony setting, etc. If a small audio glitch happened at that moment it probably wouldn't be super noticeable.

But I've taken a very hard line on this issue. My goal is for Anukari's audio to be absolutely rock solid with no compromises. I'm not claiming I've fully accomplished that, but it's the north star. So that means no allocation on the audio thread, no taking the easy way out.

The solution I came up with is as follows:

1. There is a background allocator thread.
2. Delay buffer pointers are atomics.
3. The audio thread can describe the allocations it needs in a struct that gets pushed into a lock-free SPSC queue that the allocator thread reads.
4. The allocator thread looks at each new allocation spec, and if required it will allocate the required memory and update the atomic delay buffer pointers.

This allows the memory allocation to happen in a non-realtime thread without ever blocking the audio thread.

There's an incredibly important wrinkle though: when a buffer is no longer needed, somehow we need to free it, or else we'll leak memory. This is a bit subtle, because somehow we need to guarantee that the audio thread will never again access a given buffer before we free it, otherwise invalid memory access will occur and we'll probably crash.

The trick I used to handle deallocation is as follows:

1. When the allocator thread reallocates or deallocates a buffer, it updates the atomic pointer, and then pushes the old pointer to a local "to-free" queue, in a struct that also contains an epoch integer.
2. The audio thread and allocator thread share an atomic epoch counter, which the audio thread increments each time it finishes processing an audio block.
3. The audio thread is allowed to safely access stale delay buffer pointers until it increments the epoch value.
4. The allocator thread periodically checks the next item in the to-free queue. If the item's epoch is less than the atomic epoch counter, it is safe to free it.
5. Thus, as long as the audio thread never caches a buffer pointer past the point when it increments the epoch counter, it will never access freed memory.

This may sound a bit complicated, but it's all less than a hundred lines of code. I am very thankful that I have a bunch of thorough fuzz tests to give me confidence that it works. :)

Note that this solution can't immediately be generalized to all memory allocations. Because it is asynchronous, and the allocator thread is not realtime, there will be a period where the audio thread has requested memory but does not yet have it. So this technique only applies to cases where the audio thread can operate without the memory for that duration.

This is fine for delay lines. It just means that after the user adds a new delay line, it may take a moment before the delay line's audio signal begins arriving at its destination. In practice, this duration is imperceptibly small, and doing things this way guarantees realtime safety so there will never be an audio glitch when adding a delay line.

Other RAM Savings

The delay line buffer allocation was the most interesting/difficult RAM optimization, but I did many other optimizations that were a lot simpler.

First, I tightened up the static size for several buffers and queues. There were several instances where I had sized a buffer way too conservatively, and was able to save a few tens of megabytes of RAM this way. Again, here I was relying on my comprehensive fuzz tests (and unit tests) to make sure that I didn't break anything by resizing these buffers.

The largest and most embarrassing savings of all, though, was by evicting assets from Anukari's 3D renderer cache. This was essentially trivial to do and obviously safe, so a bit of a facepalm that I hadn't done it before.

A bit of background: I have spent many hours obsessing over how quickly Anukari loads, both from a cold start as well as when the GUI is closed and re-opened when running as a DAW plugin.

One bottleneck in the cold start was the parsing and initialization of the 3D assets. This includes decoding the files, decompressing them when necessary, building collision data structures, stuff like that. I did some testing and found that parallelizing this work massively reduced the GUI's cold start latency.

The way I parallelized this is as follows:

1. The GUI enumerates all the assets it needs.
2. The asset cache then spawns a bunch of threads and loads the assets in parallel, writing the final results to a dictionary cache.
3. The GUI reads the assets from the cache as they become available.

The memory-wasting problem was that after the GUI finished reading all of the asset data, it never told the asset cache that it was finished. So the asset cache just stuck around for the lifetime of the 3D renderer, wasting memory for no reason.

This was an easy fix! The GUI already knew when it was done with the cache, so I simply had to add a line of code to clear it. This saved 100+ MB of memory for simple skyboxes and 3D assets. But for ultra high-resolution skyboxes and more detailed 3D models, it saves way more memory, potentially a few hundred MB. Pretty nice!

Captain's Log: Stardate 79603

I mentioned in my last post that I've begun working on a marketing plan for Anukari. I'm feeling pretty optimistic that it will be possible to get a steady trickle of sales going, and I also think it's possible that something more than a trickle could happen at some point. Fundamentally I just want enough sales to fund my continued development of Anukari, but I'd obviously be open to more sales, which could lead to further interesting possibilities.

Either way, I like to plan for success, and so one thing I've been working on is fixing a few small customer support headaches, so that they don't become big headaches in the event of lots of sales.

Long term the biggest thing will always be making sure Anukari performs well and is compatible with as many machines as possible. That is feeling pretty good right now.

Now that the product is not generating so many support requests, the next biggest support issue is with account emails failing to be delivered to users. The chief issue here is that occasionally a user won't receive the email containing their license key after they purchase Anukari.

This is immensely frustrating for the customer. The normal flow is that when they checkout via Shopify, they will receive a receipt from Shopify, and in that email, it says "you'll get a second email with the license key." Obviously I'd prefer for these to be one email, but that's not really possible at this point, and it's not that big of a deal.

But some users get the email from Shopify, and then never get the second email. I am the only customer support person, so this means that if someone buys Anukari right after I got to bed, even if they email to complain, they won't get their license key for at least 8 hours. Awful! Imagine it's Friday night, you're stoked to finally have a few hours to play with some new VSTs, you spend your hard-earned money on one, and then no license key comes until tomorrow. Ugh. I'd be super annoyed. (That said, evidently my customers are saints, because they have all been super understanding.)

Email is a tricky thing. Spam filters exist, and also things like IP reputation are really complicated. I have worked on spam/fraud prevention, so I know how difficult some of these problems are.

When email delivery fails, my first question has always been "did you check your spam folder." Curiously, this has almost never been the case.

The main reason that I've had emails fail to be delivered is that Resend, my email provider, simply chose not to send the emails as requested, due to their global email suppression list. This is a list of email addresses spanning across all Resend accounts, that have hard-bounced or been flagged by providers. If a customer's email somehow ends up on this list, Resend will simply not attempt any email delivery. Resend's help page says that a user marking an email as spam could be a reason for them to be globally suppressed. That means that if another Resend user sends a spammy-email to someone, it might block you from sending email to them later.

I've contacted Resend support about this several times. At one point I was told that I could remove addresses from this global suppression list myself. At other times, the Resend customer support folks had to remove the email address for me. The GUI for suppression seems to have changed a couple times, right now it doesn't look like Resend users can actually request removal from the suppression list. In my case, the GUI stated that the email address that failed delivery had already been removed from the suppression list; however I retried sending the email and it was suppressed again.

Over the course of several contacts with Resend support about this global suppression list, it became clear that Resend sees this as a feature, not a bug. And again, I understand that they have a difficult problem to solve, maintaining good deliverability with multiple users, some of which surely are spammers trying to abuse Resend's platform.

The thing is: to me, Resend has exactly one job, and it is to attempt deliver email when I send it. And sometimes, Resend simply chooses not to even attempt delivery. On purpose.

The email addresses that Resend has suppressed are totally normal, just everyday people, typically on GMail accounts, nothing weird. Shopify's emails invariably get delivered without issue, and then my emails from GMail to respond to their support requests also work perfectly. Perhaps there was a one-time delivery hiccup, or another Resend user sent out some spam. Either way, Resend is being super aggressive about suppression in a way that impacts all their users.

IMO the global suppression list is an incredibly bad design, especially with how heavy-handed Resend is being in populating it. I can verify firsthand that it causes plenty of grief. It's a difficult problem, but there are many approaches to managing reputation for shared email IPs. This one is no bueno.

So after several complaints to Resend without a solution, I migrated anukari.com completely off Resend, to Postmark. Postmark has a very good reputation for high deliverability, which again, is the only thing I actually want from an email provider. As far as I can tell, Postmark does not have a global email suppression list, but instead has per-account per-stream lists, which makes way more sense. Here's hoping that this leads to fewer frustrated customers!

Captain's Log: Stardate 79584.2

Some NAMM Ramifications, or NAMMifications

I've been quite busy (in a good way!) the last couple of weeks, with some of the connections that I made at NAMM bearing fruit in various ways.

One thing I've been working on is some setup for collaborating on a demo video with a popular MPE controller that I won't yet name. This is super exciting since the video will have broad reach, so I am working hard to iron out any remaining wrinkles in Anukari's MPE implementation. And I'm really stoked that a sound designer whom I met at NAMM, Jake Siders is cooking up some amazing MPE presets for the demo.

Also I have engaged the help of a marketing expert, Chris Hayzel. We met at NAMM and immediately hit it off. Anukari is a kind of weird product, and I have some extremely strong opinions on how I want the marketing to look. A huge thing for me is doing marketing in a way that I feel good about, so no used car salesman stuff, no misleading hype, no lies, basically. But also, hey, I think Anukari is pretty cool and I'd like to tell more people about it, and explain it in a way that's fun to learn about and not a bunch of dry technical talk (my forte). So it was important to me to find someone who I felt I could trust with this vision. This is in the early stages, but it's likely that this website is going to undergo a significant metamorphosis in the coming months. I'm excited.

Polishing, polishing, polishing

A lot of people at NAMM were surprised that Anukari is still in Beta, given that it seems like a completely functional product. I think it's almost ready to be called 1.0, but there remain a few small things I want to polish first.

First, there were still a couple of crashes/hangs that I was aware of from user reports.

One hang only happened in Apple's AU validation, only sometimes. This turned out to be a weird issue where if you pass too long of a work cycle time to Apple's Core Audio workgroup feature, the thread won't be created. JUCE has some comments to this effect, and has a supposed workaround, but it didn't work for me. I implemented my own workaround and it fixed the hang. This issue appeared when the AU validation set the plugin up with a huge block size (4096 samples), so the estimated time to process the block went above 50ms, which appears to be the limit where Core Audio loses its mind.

For the crash report, I only really knew that it happened on AMD hardware. So of course I prayed that I could find a way to reproduce it on my AMD test machines, and thankfully I was able to. The biggest hint was that my user said that originally Anukari worked perfectly, but later it crashed on startup. I guessed that they had changed a graphics setting, which was persisted to the preferences file before causing a crash. So then it crashed forever. I had the user delete their preferences file and things worked again! So I went through all the graphics settings and toggled everything until I found that the Reflections feature crashed. I tried to find the root cause, but the Vulkan code involved is so complex that I gave up and filed a bug with the Google Filament folks. Here's hoping they can help.

One improvement I made based on this crash report is that Anukari now waits a couple seconds before persisting preferences changes to disk. This way, if a change crashes Anukari, hopefully it won't be persisted, and Anukari will work when it's restarted.

Some of the other polish work includes audio quality improvements, such as adding a bit of smoothing for LFO retriggers to avoid pops.

MTS-ESP Support

As of the 0.9.25 testing release, Anukari is fully compatible with the MTS-ESP protocol for microtuning using an MTS-ESP control plugin such as (MTS-ESP Mini](https://oddsound.com/mtsespmini.php). This protocol lets users tune all their MTS-ESP-compatible plugins to whatever scale they want, in sync, so they all play in tune. The control plugin can import .scl files, so there are thousands of tunings to play with. People have been asking me for this support for a long time.

Given that I've been busy with NAMM follow-ups and polishing Anukari into a 1.0-ready product one might wonder how I fit in the work for MTS-ESP. I am also surprised! Basically what happened is I ended up with a few hours free where I didn't have enough time to really start a new big task, and was finished with my last big task. And right at that moment I got another email asking for MTS-ESP support. So I just said screw it, let's just see if I can bang that out this afternoon and that way I can say "yes" to the email.

The MTS-ESP site says you can integrate the library in an hour. I was skeptical, but it's mostly true. I had things working at a basic level in less than an hour. I did spend several more hours after that making it work really well, ensuring some of the optional features worked, and writing tests and GUI stuff to make it all professional.

So far users are reporting back that it works great. I tested it myself, and despite not being a big microtuning geek, it was surprisingly fun to try out different scales. I'm glad to have been able to sneak in this feature!