Skip to main content

2023-12-10 Using Udon MIDI as a data bus, retrospective

I am one of a handful of creators who have tried to communicate with Udon from an external process using MIDI.

Here’s me sharing what I learned from trying.

Why MIDI?​

For external processes to send data to Udon, there are only a few ways (non exhaustive):

  • Streaming video players and GPU readback
  • Polling a string loader
  • Input system (Joystick data, VR inputs…)
  • MIDI

MIDI, the one we’re interested in, is good for real time, and by real time I mean applications with less than a few milliseconds of delay.

For instance this can be used to sync the position of a real-life tracked object with the world.

Unfortunately, OSC is not an available option despite worlds being mentioned in the corresponding OSC docs.

VRChat Component​

Udon can receive MIDI data through the VRCMidiListener component. This component will listen to the first MIDI device it finds, unless a launch parameter is defined that specifies the name of the device to look for. All events are routed to a UdonBehaviour.

For clarity, MIDI events are not networked, so all processing is done on the local machine that sends and receives the MIDI data.

Virtual MIDI devices​

I don’t have any MIDI or musical background, I only care about MIDI as a possible way to transmit data from an external process to Udon.

In order for VRChat to receive MIDI messages, we need to create a device we can write MIDI events to. This can be done using a virtual MIDI device, and although apparently it’s natively supported in OSX, in Windows there’s no native support for virtual MIDI devices so you need to install a driver from a third party.

Scouring through the Internet I was not able to find a truly free and open source solution, so we can use proprietary or source-available free-to-download software with some usage restrictions. There are a few mentioned here: https://support.troikatronix.com/support/solutions/articles/13000081815-using-virtual-midi-ports

LoopBe1, and LoopMIDI often come up as results, but personally I’ve used sbvmidi, which worked out. Your mileage may vary.

  • These software create an input MIDI device and an output MIDI device.
  • You send events to the virtual output MIDI device.
  • The events are re-emitted out of the virtual input MIDI device.
  • That virtual input MIDI device is what VRChat listens to.

Sender library​

Personally I’ve used the DryWetMIDI .NET library so that I could use it in a non-VRChat SDK Unity project to send MIDI data.

Data types​

Midi in Udon supports 3 events:

  • NoteOn
  • NoteOff
  • ControlChange

Each of these three event types have:

  • A 4-bit number which is the channel
  • A 7-bit number which is the note
  • A 7-bit number which is either the velocity or a value

Encoding strings​

Although the data payloads that we will send from our external apps to Udon can be anything specifically crafted for a specific use in order to fit data buffer size constraints (see later in this article), let’s start with strings since it’s easy to process and debug.

Here’s the catch: I can’t find an easy way to use Encoding.UTF-8 in Udon in order to decode bytes to strings, even with tricks.

Therefore instead of encoding the string to UTF-8 bytes, I’ve decided to process each Char of the string, which is 2 bytes per char.

This way, it will be easier to reconstruct the string even if it has non-ASCII characters in it.

Transmitting 2 bytes using a single note​

All event types have a total of 18 bits, therefore it should be possible to encode a Char (16 bits) inside one event.

However, I’ve decided to do the following instead:

  • Store the 7 first low bits of each Char into the two respective 7-bit numbers
  • Use the NoteOn Event if the highest bit of the 1st Char is low
  • Use the ControlChange Event if the highest bit of the 1st Char is high
  • Use the channel 14 if the highest bit of the 2nd Char is low
  • Use the channel 15 if the highest bit of the 2nd Char is high

This limits the number of channels that our encoder will use to only 2.

The problem: There’s a bug in VRCMidiListener/VRCPortMidiInput​

Everything would work great if it weren’t for an issue in VRCMidiListener, located in the internal class VRCPortMidiInput.

VRChat will raise exceptions if there are too many MIDI events received since the last frame was processed. Too many MIDI events will trigger a buffer over-read inside VRChat, and the events won’t be able to be processed properly. This can happen as low as around 128 events since the last frame.

If VRChat needs to process more than 8 times that maximum amount, it may also able to crash to client under certain circumstances. This only affects the machine that sends the MIDI events: the other players in the instance are unaffected since MIDI events are not networked anyways.

Therefore, in order to transmit data safely to VRChat using MIDI, you need to ensure VRChat’s own stability from the caller side:

  • Stop sending more events than VRChat can process within a frame.
    • This can be done using an event or message queue.
    • Make sure you can handle queuing events for messages that are larger than what VRChat can handle in a single frame.
  • Detect when VRChat has finished processing a frame, before sending more events.
    • This can be done by monitoring VRChat’s output logs using a FileSystemWatcher.
    • In the Udon behaviour that processes the MIDI events, if it has processed at least one note this frame, then after all notes are processed that frame (use LateUpdate), print a line that the sender will be able to detect.
  • Detect when VRChat is not responding or when the user leaves the world, or handle timeouts.
    • This prevents the queue from growing too large, as the sender must detect that the frame has been processed first, which will never happen when VRChat is not responding.
  • If VRChat is lagging behind the sender, avoid queuing data to be sent.
    • You don’t want the queue to grow too large with messages that will be irrelevant by the time VRChat has processed them.

That’s it​

The lower your VRChat framerate, the slower VRChat may be able to process incoming data.

The real solution would be OSC for worlds, but realistically, are you going to wait for that if you need to get things done?