For the record, all the text below was written by hand.

Some conclusions are derived from setbacks that an AI agent encountered as part of porting worlds.

General​

• Basis may use different layers:
  • Avatars and objects may be invisible in rendertarget Cameras that specify the wrong layers.
  • This can affect rendertarget Cameras, reflection probes, mirror settings, raycasts, and other physics systems.
• Multiple players may have the same name:
  • Display names are not unique, and therefore cannot be used as an index.
  • Similarly, multiple players may have the same DID key, because joining with multiple clients is often allowed by the server.
• Player names may contain tags:
  • Player names may contain customized tags, making it more difficult to display them in rich text fields such as TextMeshPro.
• By default, Interactables are activated using the VR controller grab, not by pressing the trigger:
  • The trigger key is not used to interact with Interactables by default.
• Constraints are auto-converted to Basis constraints:
  • This can cause some systems that reference the constraints to fail.

Affects gameplay​

• Jump height is proportional to avatar height:
  • Players with small avatars can be unable to jump across platforms designed with a fixed world-space jump height.
  • Players with tall avatars can jump over obstacles designed with fixed world-space jump height.
• Collider is proportional to avatar height:
  • Players may fall through small gaps on small avatars, when they were supposed to be walkable over.
  • Players may casually walk over chasms, when they were supposed to make the player fall.

Unclear specification​

• Teleporting multiple times per frame could result in an incorrect teleport position:
  • Getting the position of the player may return the same value after teleportation.
  • A side effect of this may cause the player end position after a second teleportation to be incorrect.
• Position for VR players may be unclear:
  • Getting the position VR players may not reflect the bottommost position of their locomotion collider.
  • Instead, it might reflect the initial playspace position of when they calibrated their avatar, so it could be completely offset.
• Local player trigger collider is unclear:
  • It is unclear if the local player has a reliable trigger collider that can be used in collision events.

Visual​

• Shaders that claim to support URP might not necessarily support Adaptive Probe Volumes:
  • By default, Basis uses URP with Adaptive Probe Volumes, meaning Light Probes are not used.
  • Some shaders that claim to support URP may not necessarily support Adaptive Probe Volumes, so they will not sample light probes properly.
• Default font material is visible through everything:
  • Create a new font material and assign it to Text and TextMeshPro components to avoid this issue.
• Zero-area triangles may blow up probes:
  • If dynamic objects that use light probes are overblown, or there are square artifacts in the world, there may be zero-area triangles on world meshes that have an emissive material.
  • This can typically happen when moving away from Bakery lightmapper while migrating to URP.
• Animators and code that affects materials need to be updated for URP:
  • Properties of URP materials may need to be animated or set using different property names than in their BIRP equivalents.

Cilbox VM​

• UI Buttons should not call scripts, instead the callback should be registered:
  • Use button.onClick.AddListener(OnEraseAllClicked) to trigger events.
• Avoid the use of bool arrays unless tested:
  • Bool arrays appear to trigger several bugs in Cilbox, and should be avoided at this time of writing.
  • Examples of errors:
    • Privilege failed for System.Array.Empty generic argument 0 type
    • CilboxException: Error: Could not find reference to: [mscorlib][System.Array]
• Avoid the use of implicit cast from Vector4 to Vector3 unless tested:
  • Examples of errors:
    • AmbiguousMatchException: Ambiguous match found

Assistance from AI agents​

The below apply only when trying to orchestrate a port through the use of AI agents.

• Tell the AI that program variables (serialized fields) from the original components need to be extracted first:
  • An AI-assisted port may fail to understand that program variables the original components are serialized in a peculiar way inside the component.
  • The AI should first extract the peculiar serialized data contained within the original component, and re-apply it to the ported Cilbox components.
  • In addition, the AI may struggle to copy serialized AnimationCurve fields, so the AI should pay extra attention to AnimationCurves.
  • Should it fail the above, there is a risk that the AI may try to guess the value of the fields, which is undesirable.
    • If this happens, you should start over from a previous point rather than try to repair it.
• The AI is quite capable at autonomously debugging networking and Cilbox issues:
  • Using the Claude Sonnet model and the Claude Opus models (October 2026) revealed that they can autonomously debug issues, including network tests.
  • The models are also capable of knowing when an issue is caused by a bug within Cilbox itself (or a bug in Basis itself), and work around it.
  • Tell the AI to run tests in Play Mode:
    • The AI should try to run Cilbox scripts in Play Mode, to ensure the execution of the Cilbox behaves the way it should.
    • Cilbox sometimes contains unknown bugs that are only revealed during the execution, despite the structure of the program looking sound.
  • Tell the AI to run network tests with a real server (hosted locally) and two clients:
    • The AI should be able to autonomously test that network behaviors are working the way they should, and debug unexpected situations.
• When porting with AI agents, create an ordered list of tasks for the AI to tackle one by one:
  • Running the port as a single instruction risks the AI interleaving operations in an unspecified order, leaving most of it incomplete or unsatisfactory.
  • In addition, you may clear the context between tasks, as long as the AI keeps track of the state of the port.
  • Here's an abridged example of a task list. Some tasks may need to be clarified with further context, and some may not apply to you:

• Convert materials that use BIRP shaders to URP shaders.
  • (you may need additional instructions, such as Convert shaders to URP, migrate to a specific URP shader, or ensure custom shaders support SPS-I so that it is drawn on both eyes)
• Convert Bakery lights to Unity lights and make sure you check lights contained within prefabs.
• Convert lightmapping settings from Bakery to the Unity GPU Lightmapper.
• Add a Basis Scene component to the scene, replicating the original.
• Extract all program variables for use later.
• Convert all non-Editor scripts to Cilbox and Basis, as faithful to the original as possible, including the scene objects with the help of the previously extracted program variables.
  • Create a utility function for all conversions of GetTrackingData so that we may globally apply a correction to the head and hand rotation later that will affect all original callers.
• Identify object pools (groups of objects spawned for each player) and devise a Cilbox equivalent.
• Modify the animations of Animators and Cilbox scripts that used to target BIRP materials, so that they target the equivalent property of the URP materials.
• Convert the scene objects and scripts to use layers of the Basis Framework.
• Convert all Unity constraints to Basis constraints, including any field or Cilbox script reference.
• Improve Cilbox scripts to take full advantage of the Basis networking model.
  • Identify locations from the original scripts that used object state transfer as a disguised way to do message passing, and use message passing directly.

As a side note, in my setup, I deliberately set up a spare Windows gaming desktop machine on the network (without a monitor attached), so that Claude Code could connect to it and perform whatever tasks it might need without disrupting the operations of my main computer.

A lot of these tasks may take several hours without human intervention, so you may want to keep your main work machine operating as normal during it.

Cilbox is a scripting system that is available by default in Basis. User-generated content such as worlds, props, and avatars may contain scripts, and they will execute in separate sandboxes that have different lists of permitted types and accesses to functions.

Setup​

The class should be marked as [Cilboxable], such as:

[Cilboxable]
public class Pencil : MonoBehaviour {
    // ...

• To use a scene that uses Cilbox scripts, a single Cilbox Scene Basis component needs to exist somewhere in the scene.
  • Scene scripts written using Cilbox can be placed outside of that component.
• To use a prop that uses Cilbox scripts, a single Cilbox Prop Basis component should be put at the root of the prop.
• To use an avatar that uses Cilbox scripts, a single Cilbox Avatar Basis component should be put at the root of the avatar.

Those components do not need to be configured.

OnEnable is not called the first time​

Cilbox seems to have a quirk, where OnEnable will not be called the first time the prop is enabled.

You may have to resort to workarounds similar to the following:

private bool _isEnabled;
public void Start() 
{
    // Do one-time initialization stuff here.
    
    WhenEnable();
}
public void OnEnable() { WhenEnable(); }
private void OnDisable() { _isEnabled = false; }
private void WhenEnable()
{
    if (_isEnabled) return; // Cilbox quirk
    _isEnabled = true;
    
    // Do OnEnable stuff here.
}

Networking​

To make an object networkable, either:

• Call _networkShim = Basis.SafeUtil.MakeNetworkable(this),
• or add a BasisNetworkShim component and reference it as a field.

Check ownership using _networkShim.IsLocalOwner()

To send and receive data:

• Send packets using _networkShim.SendCustomNetworkEvent(bytes, ...)
• receive packets by registering an event listener using _networkShim.NetworkMessageReceived += OnNetworkMessageReceived

Props load at different times for different players, so you may have to detect when network is ready using _networkShim.NetworkReady += OnNetworkReady to start sending packets.

According to the creator of the Basis Framework, it is impossible for other players to load the prop faster than the person who spawns and owns the prop, therefore when OnNetworkReady is triggered on a non-owner, it should be safe to assume that the network owner is ready to receive the packets. EDIT: It is unclear if that's the case.

UI Buttons​

Follow the official Canvas UI documentation, then.

To make a UI button trigger events when clicked:

• Reference the Unity UI button as a field, and then
• register an event listener, such as _eraseAllButton.onClick.AddListener(OnEraseAllClicked);

Pickups​

Follow the official Interactable documentation on how to set up a network synced pickup.

To detect pickup and drop events:

• Reference the BasisPickupInteractable component as a field, and then
• register an event listener, such as pickup.OnInteractStartEvent.AddListener(OnPickup); and pickup.OnInteractEndEvent.AddListener(OnDrop);
• for the trigger events, use pickup.OnPickupUse.AddListener(OnPickupUse). The callback method has an enum value that tells you when the trigger press starts, continues being held, or ends.

Pickups do not affect component ownership.

• If a user grabs a synced pickup, they only own the synced pickup component.
• The other networked components that are on the same GameObject do not change ownership.

Teleportation​

Scene props can teleport the player using by calling the BasisLocalPlayer.Instance.Teleport(spawn.position, spawn.rotation) function.

Troubleshooting builds​

If a non-uploaded prop works in editor but doesn't work when the built BEE file is loaded in the real application, then consider loading the BEE file inside the Unity Editor while in Play Mode and debug it directly inside the editor.

This way, I found out that the non-uploaded prop worked in editor because it was executing using the Cilbox Scene box component instead of the Cilbox Prop box component, as the prop didn't have a Cilbox Prop box component attached.

Not supported​

Do not use Array.Empty<HVRNPSCilBeacon>()​

Privilege failed for System.Array.Empty generic argument 0 type HVR.NPS.ForCilbox.HVRNPSCilBeacon

CilboxException: Error: Could not find reference to: [mscorlib][System.Array][HVR.NPS.ForCilbox.HVRNPSCilBeacon[] Empty[HVRNPSCilBeacon]()] Type from:System.Array

Do not use lambdas​

ArgumentException: Object of type 'Cilbox.CilboxHeapInstance' cannot be converted to type 'Cilbox.CilboxProxy'.

Do not implicit cast from Vector4 to Vector3​

• Error raised on method op_Implicit of type UnityEngine.Vector4 with 0 args.
• AmbiguousMatchException: Ambiguous match found.

Bugs

Negating a value is not working properly​

The value of x in var x = -someValue; will be incorrect.

• Discovered when Random.Range(-x, x) was returning values way out of expected range.
• This can be worked around by multiplying with -1

This bug has been fixed in the Cilbox repository, but it may be reflected in the Basis Framework project and client at a later point.

Server XML configuration ignores scale​

The scale of props is ignored in the XML configuration.

When exporting a custom avatar using Unity's default Export .unitypackage function to transfer assets between projects, assets are often scattered across multiple folders as expected from a custom avatar.

However, the contents of the export often end up with extra assets that aren't particularly needed by the custom avatar.

All of these reasons can be summarized using the picture below, but the first reason listed here can be very unexpected. I suggest you read the first reason in detail, and skim over the others since many of them are predictable once you're aware of it.

A prefab instance simultaneously overrides a field and deletes the component that contains that field​

If a prefab instance overrides an asset reference on a component, and that prefab instance simultaneously removes the same component or GameObject which is present in the prefab, the override referencing the asset continues to exist.

In addition, the Unity Editor prefab override UI is incorrect and will not display overriden references inside removed components. The act of un-removing the component will expose the hidden reference that it contained.

Detecting this issue is nearly impossible without external tooling.

A reference inside the active hierarchy points to a Transform located within a prefab source that you do not use​

Components may contain stray references to sub-assets of a model or a transform within a prefab, even if you are not using that prefab source on your avatar.

This is a problem because it means every reference used by that prefab source will be required, along with any other asset referenced by them.

On my projects, I've found that the anchor override of a SkinnedMeshRenderer sometimes points to a transform located inside a prefab source.

A reference points to a sub-asset of a prefab model​

If a sub-asset of a prefab source or model is required (such as Mesh), then the entire prefab source is required.

This is expected, but it can cause a problem because of the next reason.

A prefab model (FBX) contains references to textures that you do not use​

Sometimes, textures located inside the project are referenced directly by the prefab model (FBX).

Even if you do not use the texture in any of your active materials, it might get included.

A prefab source requires assets that you have overriden​

A prefab source may be set up with asset references pointing by default to assets that you are not using because they are overriden by your prefab instance or prefab variant.

This is typically true when you are using avatars that you have purchased, but you have overriden some materials.

In turn, these assets may reference other assets, such as textures, that you may also not be using.

A prefab source requires assets in a Component or GameObject that you have removed​

Very similar to the previous reason, but instead of overriding the asset with another asset, you have removed the component or GameObject altogether.

A prefab instance or a prefab source requires assets in GameObject which is EditorOnly​

As an alternative to removing a Component or GameObject, it can be common practice to set it to EditorOnly so that you can preserve the objects in order to re-include them later.

Not only the prefab source may reference assets that you are not using; it is possible that you are yourself overriding some fields using assets on the prefab instance, which will not end up being used.

It is ambiguous whether this counts as an unused asset as different people will use EditorOnly differently in their workflow, but it is worth a mention.

A component or material requires a script, and references inside those scripts​

Some files containing code are not necessarily wanted during export, but also, these scripts can reference other objects.

• Materials require a shader. Shaders are often installed separately.
  • Shaders sometimes contain references to default textures in them. Those textures are often installed with the shader itself.
• If a GameObject references a MonoBehaviour, it may require scripts, and DLLs. Those are often installed separately.
  • If a MonoBehaviour uses an icon, then that icon can end up being included.

If you are exporting to another game, some references inside components may be undesirable​

If you are exporting from one game to another game, some references inside components can be irrelevant.

• Some components may be irrelevant in another project (e.g. Modular Avatar Menu Item) and contain references to assets such as icon Textures.
• Some proprietary assets may be incompatible in another project (e.g. Expression Menu), and those assets may themselves contain references to other assets.
• Some Unity assets may be irrelevant in another project (e.g. Animator Controller, Animation Clip).

Before and after​

How it looks like before (this is the same picture as the one at the top of this article):

And after:

While developing Skinned Mesh Constraint, I noticed something weird and I wanted to document it somewhere.

Skinned Mesh Constraint is a tool that makes Parent Constraint move the same way as how vertices move from mesh skinning.

Approach #1: Activate constraint targeting the bones​

I've always thought that activating a Parent Constraint with the Activate button made the GameObject follow the source transforms using a proportion of their weight, relative to the rest pose.

This is why, when I tried to create my Skinned Mesh Constraint component, my first approach was to collect the painted weights from the mesh, set up the corresponding Parent Constraint sources, and then use reflection to press the Activate button (on a non-VRC project).

However, when moving the bones around, the Parent Constraint would not move the same way as mesh skinning (it overshoots).

Approach #2: Activate constraint targeting new offset transforms located inside the bones​

I've then tried to create new transforms inside the bones that are offset to be located in the same position as the Position Constraint at rest.

That approach worked. So somehow I was getting confused about how Parent Constraints work internally. With these observations, I asked for help on two Discord servers.

I've then been told by someone named Sacred that my approach #2 is the correct approach for mesh skinning (Sacred credited someone else called Voxian for that approach, so I've only rediscovered it), and that there is an alternative worse way to do it by using multiple constraints (one parent constraint per bone).

This approach has the advantage of using less constraints, but needs more transform references.

Following this discussion on Discord, this triggered my research for a third approach:

Approach #3: Manually calculate the offset transforms​

When clicking the Activate button, the Parent constraint stores information about the relative offsets to the constraint sources at rest.

In Approach #1, these offsets have some value.

In Approach #2, all of these offsets are equal to zero.

In this new approach, I tried to copy the previously created offset transform position and rotations located inside the bones, into these offsets. The constraints are targeting the bones directly instead of those offset transforms.

As it turns out, writing these offsets by hand has fixed the issue. It does not require additional transforms to be referenced, and thus this is the method that I'm using in Skinned Mesh Constraint.

The offsets are calculated using (Quaternion.inverse(sourceTransform.rotation) * constraint.rotation).eulerAngles, for the rotation and sourceTransform.InverseTransformPoint(constraint.position) this answer for a translation offset.

So, what does the Activate button do on constraints?​

I don't know how the Activate button works on constraints.

I've tried to visualize what the button does, but this is strange. When clicking Activate, the offset on the leg bones are pointing to unexpected positions.

On this picture above, the tip of the red lines are pointing to the world space position of the offsets, but the object being constrained is at rest.

The red lines should not show up at all, because the rest position is the offset, when configuring the Parent Constraint in a way that follows mesh skinning.

I don't know anything more about this, but this means that pressing the Activate button on a constraint results in a different behaviour than calculating the offsets by hand.

Update 2025-05-03: I have learned new information about how the Activate button works, so this article will later be updated accordingly along with a product update.

I was building a new home world after being bored by the one that I had been using for over a year.

Initially, I just wanted to make sure that shooting walls would produce decals and the correct sound effects depending on the material being hit, after being inspired by a Half-Life 1 documentary starring Gabe Newell on the topic of fun, reinforcement, and world interactivity (2-minute watch, between 15:08-17:03). I was thinking it was odd I never spent time to implement such a basic game functionality within the Unity engine.

However, the whole idea went out of control and I ended up building an analogue of throwable and breakable objects.

Below is a random assortment of things I have learned while building this system. Some items may be specific to the

Avoiding Continuous Speculative for important collisions​

When a rigidbody is thrown and hits another collider, we need to apply an appropriate sound effect volume and damage amount, using the collision force. Collision events are raised through OnCollisionEnter, containing this information.

The issue is that depending on the collision detection type of the rigidbody, the reported amount may equal to 0 newton too often, even on objects that are thrown hard onto walls. This especially happens when the rigidbody is set to Continuous Speculative.

The fix is to set the collision detection type to Continuous Dynamic, as long as we care about these objects reporting the correct forces to produce sounds or be damaged on impact. For instance, this is not necessary on the individual debris emitted out of breakable objects.

Don't bounce away from broken objects​

If you throw a bowling ball against a fragile window, the bowling ball should go through the window because it conserves most of its energy.

Collisions with all non-moving objects will cause the thrown object to bounce away from it, even if that non-moving object breaks the moment it is hit. This also applies to some heavy breakable objects that can be moved.

The fix that I chose is to save the velocity and angular velocity of thrown objects every physics frame in FixedUpdate. When an object breaks apart, it needs to inform the other object that collided with it, so that it cancels the bounce and re-applies most of its energy based on the velocity and angular velocity of the last saved physics frame.

Getting the material hit in a multi-material mesh​

If an environment mesh/world mesh is made out of multiple materials, hitting it with bullets or collisions should produce different sounds based on what that material represents, for example if it's glass.

This information can be encoded in the Physics Material type of the collider, but there is another approach.

When a mesh has multiple materials, it is possible to use Mesh Colliders with raycasts to get the material that is hit. Whether you should use this approach is a different question.

To do this, raycasts will report a triangle index if a Mesh Collider has been hit by the raycast.

To convert the triangle index to a material, you need to get the mesh asset of that Mesh Collider, and for each sub-mesh of that mesh, use triangleCount[subMeshIndex] = mesh.GetSubMesh(subMeshIndex).indexCount / 3; to get the number of triangles for that sub-mesh (notice how we divide by three).

If a mesh has 3 sub-meshes, and their triangle count are 70, 50, 40 in that order, then a triangle index of 74 will mean you hit the sub-mesh at index 1 (which is the second one), so you can get the material slot at index 1. A triangle index of 122 would be index 2, because it is greater or equal to 70 + 50.

Again, it may be more reasonable to split into different meshes, or use different colliders for each material.

Preventing objects from breaking too early for remote players​

Using Player pools to send and receive packets (Player Objects)​

When an object is damaged, this information arrives through Manual Sync information which is located outside the VRCObjectSync hierarchy.

The way I have designed the damage networking in my world is through the use of many player pools (now known as Player Objects, in this case non-persistent ones).

• Each player gets their own set of Manual Sync objects, one for each specialized "packet" it needs to send.
• To damage an object, players would submit a packet containing all the individual pieces of damage instances this player has dealt since last serialization. That packet contains information about which one of the many damageable objects has been hit in the scene.
• Everyone receives that packet and applies the damage to the corresponding object.
• The default network owner (also known as master) would track the actual health of these objects, and makes sure everyone else agrees on the actual health of the object.

Example of one of the several packets that can be sent.

[UdonBehaviourSyncMode(BehaviourSyncMode.Manual)]
public class DamageableSyncMessage : UdonSharpBehaviour
{
    [HaiInjectable] [SerializeField] private DamageManager manager;

    [UdonSynced] public int[] bundle_entityId;
    [UdonSynced] public int[] bundle_healthReduction;

    private readonly List<DamageableSyncMessageBundle> _queued = new List<DamageableSyncMessageBundle>();
    
    // [...]
    
    public override void OnPreSerialization()
    {
        var queuedMessageCount = _queued.Count;
        bundle_entityId = new int[queuedMessageCount];
        bundle_healthReduction = new int[queuedMessageCount];
        
        for (var index = 0; index < queuedMessageCount; index++)
        {
            var bundle = _queued[index];
            bundle_entityId[index] = bundle.entityId;
            bundle_healthReduction[index] = bundle.healthReduction;
        }

        _queued.Clear();
    }
    
    public override void OnDeserialization()
    {
        // [...]
            manager._InternalReceiveDamageableSyncMessage(entityId, healthReduction);
        // [...]
    }
    
    // [...]
}
internal class DamageableSyncMessageBundle { /* ... */ }

Each player individually tracks the health of those objects locally. This has the advantage of:

• avoiding ownership transfers altogether, letting multiple players damage the same object simultaneously,
• it also avoids a round-trip between the person dealing the damage, and the default network owner confirming that the damage has been dealt,
• if the local player performs an action that breaks the object, it breaks instantly without waiting for any response from the network.

As a consequence, the objects break as soon as the message is received, and if the local player breaks the object, it also breaks locally with no delay, even if they do not own the object itself, making for a very responsive system that emphasizes visual effects over correctness and over proper damage attribution.

Damage packets arrive too soon, before the visual cause of damage happens​

This works great with damage coming from bullets.

However, damage caused by collisions with walls are always transmitted by the network owner of the rigidbody itself, but the damage packets from this collision often arrive to other players before the object is even close to hitting the walls.

This causes objects to break in midair, or worse, break before they are even thrown. This visual disconnect between the consequence and the cause of the object breaking is jarring.

The solution I chose to fix this is to create a 2nd specialized damage packet, dedicated to collisions with walls. The network owner of the rigidbody will submit those specialized damage packets, containing the position of the rigidbody when it collides with anything.

Breaking objects from collisions are intentionally delayed​

On other clients, if receiving that specialized damage packet would have resulted in the health going from a non-zero value to zero, then the breaking visual effect would be intentionally delayed until any one of the following happens:

• The rigidbody position reaches the position indicated within that specialized packet, or
• A maximum time limit of 1 second since receiving that packet is reached, or
• The object ends up damaged by any other reason than a collision, such as a bullet hitting it.

There are edge cases to this approach, for instance if the object breaks from a collision, but not as a result of itself being thrown, instead as a result of another object being thrown into it. This is still something I may have to fix. Edit 2025-02-09: This case has been handled by estimating the network delay of the damage dealer (e.g. if a volleyball is thrown onto glass, and the glass breaks, we want the network delay of the volleyball physics object owner who dealt the damage), and using that delay to break the object later.

In addition, this approach is really sensitive to deserialization order, which is a flaw. In particular, the default network owner will submit updates about the actual health of the objects so that other players may try to resync in case of a disagreement on the health of the object.

There is a special case when the default network owner declares that the health went from non-zero to zero, causing the object to break instantly if it was not already broken by a damage packet; this is a problem if the default network owner also happens to be the rigidbody owner, because the damage packets will often be received at the same time as the confirmations of changes in health.

This specific issue has not currently been solved in a sane way; in this case, I chose to move some of the deserialization handling code to the Update loop, so that damage packets are handled first, but this quick fix is not sustainable.

Disabling the broken object interrupts its networking​

When the object gets broken, the GameObject that contains the VRCObjectSync must not be disabled, otherwise the rigidbody state updates may not reach other clients and will freeze in midair. For this reason, I disable the renderer and colliders, and on the rigidbody owner, I force the position, rotation, velocity, and angular velocity of the rigidbody to be constants while it's in a broken state.

This object is also respawned invisibly with an advance delay on the rigidbody owner, before it is enabled back on all clients, so that the non-owners will not interpolate the position of the rigidbody and cause unintentional collisions with other objects.

Avoiding ownership transfers​

If a player shoots at a movable object, that object should be pushed away a little.

When two or more players shoot at the same object simultaneously, we want to avoid intensive ownership transfers.

For this reason, when a player shoots an object, that player transmits a specialized packet in the same way as described in the previous section using Player Objects. This packet describes where the bullet was shot from, what was the direction of that bullet, and where the bullet hit in the local space of that object.

When that packet is received by the owner of the rigidbody, it reapplies the force of that bullet onto the object, along with sound effects and decals. This allows the object to be pushed away a little by both players, without ownership transfers, although there is a visible delay from the shooter's perspective.

Collision sounds while picked up​

When an object is picked up, the object should still produce sounds whenever that object collides with walls.

This works correctly on the user that picked up the object, but it appears that collision events do not happen on remote users while the object is picked up.

To remediate this, only while an object is picked up, the owner will submit a limited number of networked sound effects to be played back on other clients.

Lookup tables​

In order to do this project, I ended up having a bunch of lookup tables which are baked in Edit mode:

• When a bullet hits a collider, the decals needs to stick to the visual representation of that collider, even if it's a moving object.
  • Moving objects are not necessarily object-synced. For instance, the logic behind doors use networking, but the rigidbody of the door is not itself a networked object, so even some non-networked objects need to have a way to identify them.
• If an object can be damaged, then we need to identify this object for transmission of damage packets, so that damage effects can be applied back on every client.
• Therefore, I associated a numerical identifier to every object that may need one, creating a parallel Network ID of some sorts. A single behaviour centralizes all those identifiers, so that prefabs and objects that are routinely copied don't need to carry this information.
• Those identifiers are updated using a script in Edit mode, which also happens to be the same script that handles dependency injection in Edit mode.
• When a collider is hit for any reason, we need to produce a sound effect, so each collider is associated with a collision sound effect identifier based on either its Physics Material or the materials of the renderer; or multiple ones if there are sub-meshes, so the triangle index lookup table is part of that too.

Getting the velocity of the hands​

In order to detect a flicking of the hand to emulate a similar interaction style to Half-Life Alyx gloves, we need to get the velocity of the hands. However, I do not want locomotion using the analog stick to be interpreted as a flicking of the hands.

Every frame, I create a tracking space origin TRS matrix using the position and rotation of

Then, on the current frame, I reapply the relative hand position of last frame onto the current frame's tracking space origin matrix. This gives us two world space positions, which can be used to get the hand velocity in world space.

World space allows the use of the dot product to detect the hand flicking in a specific direction (e.g. opposite of head direction, or away from a specific object).

Hello, if you're trying to run Modular Avatar on applications other than VRChat, here's what I learned when trying myself.

Modular Avatar V1.x is almost already compatible​

Modular Avatar V1.x is almost already compatible with apps other than VRChat, but some VRChat-specific functionality will be removed.

This is because most of the code specific to VRChat is already disabled in Modular Avatar if the VRChat SDK is not detected in the project.

For example, MA Merge Armature will run because it's generic, but MA Menu Item will not because it's too specific to VRChat.

All sections below describe the work needed to make it compatible.

Modular Avatar sometimes has compilation errors in some releases​

Modular Avatar is only officially supported on the VRChat platform, so in some releases, there may be compilation errors because some code paths without VRChat had been overlooked.

Fixing this often requires adding #if MA_VRCSDK3_AVATARS and #endif at the right locations to make the compiler ignore imports, field declarations, or entire code blocks that are too specific to VRChat.

If something requires context.AvatarDescriptor, you could probably assume most of the contents of the function could be removed from compilation.

See this pull request as an example.

If you use ALCOM / VCC, installing Modular Avatar will install the VRChat SDK​

Installing Modular Avatar V1.x through VCC or ALCOM will also install the VRChat SDK in the project. This is not desirable, you must not have the VRChat SDK in a non-VRChat project because the VRChat SDK contaminates the project configuration and its build process (layer names, XR project settings, etc.), which will interfere with the functionality of your targeted app.

If you use ALCOM package manager, you can:

• Close the Unity project,
• Install Modular Avatar,
• Uninstall the VRChat SDK Avatars package,
• Uninstall the VRChat SDK Base package.

If you use VCC, you can't uninstall the VRChat SDK package once it is installed. Either switch to ALCOM, even if it's just for non-VRChat projects, or install Modular Avatar manually through the source. You will also need to install NDMF.

If you use neither ALCOM nor VCC, you can install it directly through the source. You will also need to install NDMF.

I've been told that in the future, Modular Avatar V2.x will no longer cause the VRChat SDK to be installed in the project when it is installed.

Modular Avatar depends on DLLs​

Modular Avatar requires some DLLs to be present in the project, which are shipped by default in the VRChat SDK.

These DLLs are often absent from other projects like ChilloutVR or Basis.

Try to install Chillaxins through ALCOM, or install those DLLs yourself from Microsoft NuGet (System.Collections.Immutable, Lib.Harmony).

The DLLs are:

• System.Collections.Immutable DLL, because NDMF uses it.
• Lib.Harmony DLL, because NDMF 1.5 uses it.

Modular Avatar depends on the Unity Burst package​

Modular Avatar requires the Unity Burst package in the project, which is normally included automatically by the VRChat SDK.

Try to install Chillaxins through ALCOM, or go to Unity's package manager, and add Burst to your project.

Connect NDMF to the build process​

To ensure Modular Avatar runs when the avatar is being built, you need to connect NDMF to the build process.

In the ChilloutVR application, I've done it in Chillaxins (install it through ALCOM), see this code.

In the Basis framework, see this code. This is already included in the Basis source code itself.

VRChat has a feature called OSC trackers. It exists as a low-cost way to add trackers controlled by software, without needing to create custom SteamVR drivers, and also can be used on Quest standalone headsets.

For the sake of experimentation, I’ve been wanting to try using OSC trackers for locomotion “animations”, where the locomotion animation logic would come from an external program, rather than doing it in an animator controller with animation clips.

Strangely I’ve only written this down recently, but I’m convinced I had been wanting to do this for more than a year already.

Flatscreen Third person games​

You’ve played third person games, you know what I will be talking about. Grand Theft Auto, Assassin’s Creed, Uncharted, Metal Gear Solid, they all control a main character in third person. Even when it’s in first person, you have NPCs that move around. There’s an entire industry dedicated to making a game character movement look good.

The idea is to make or reuse a character controller for third person games in an external program, and then give that character OSC trackers, so that this virtual character will be controlling the avatar in VRChat instead of my physical body.

There’s a bunch of interesting talks about this subject. Here’s a few of my favorite talks and demos:

https://www.youtube.com/watch?v=KSTn3ePDt50

• Kristjan Zadziuk (Ubisoft) - GDC 2016 - Motion Matching, The Future of Games Animation

https://www.youtube.com/watch?v=7S-_vuoKgR4

• Michal Mach (Naughty Dog) - GDC 2017 - Physics Animation in Uncharted 4: A Thief's End

https://www.youtube.com/watch?v=RCu-NzH4zrs

• Joe van den Heuvel (Valve) - SIGGRAPH 2021 - Character Locomotion in Half-Life: Alyx

https://www.youtube.com/watch?v=16CHDQK4W5k

• Ubisoft La Forge - SIGGRAPH 2020 - Learned Motion Matching
  • Article

Generally, I think there’s a bunch of established tech from existing non-VR fields, which I really wonder how they would translate into VR, so I’m barely scratching the surface.

Making this project is partly a way for me to start looking for answers in which game animation techs could be reused in the context of social VR.

VR has its own constraints that put a limit on what’s acceptable, and I think a good way to find those constraints is to break them intentionally or unintentionally.

Motion Matching in Unity by JLPM22​

In GDC 2016, Kristjan Zadziuk from Ubisoft presented a technique they devised “Motion Matching” (see the first video above).

This technique caused a ripple in the games industry, as bits of animations for locomotion cycles could be replaced by sampling directly from a specially recorded and labelled continuous motion capture session, to directly drive the character locomotion.

In effect, this creates a very sophisticated character movement in the world, as the character compensates by properly shifting weight when turning or stopping.

An user by the handle of JLPM22 (Jose Luis Ponton) has published an open version of this technique, implemented in Unity:

https://github.com/JLPM22/MotionMatching

This could serve as a good starting point to experiment.

I’ve imported the project, did a few fixes (A, B), to make it run with a social VR avatar I have.

I noticed it mostly ran well, although the idle pose looked weird (the character is leaning forward).

Otherwise the character animates as expected, and it only moves on a flat surface, which should be enough for testing.

Immediately, I anticipate some points that could cause issues later, but it won’t stop me from trying:

• The character takes time to stop and start.
• The head bobs up and down during movement.
• The character takes time to turn.
• The character cannot strafe.

I do not have a motion capture studio, and my room is too small to do proper DIY motion capture for walking long distances in a straight line, so I cannot supplement the existing motion tracking data with my data, so I’ll just try to get it working within those limitations.

OSC Trackers​

Next up, add OSC trackers to that character.

This ended up being easy for me, because I’ve already written small library bits for sending OSC trackers data to VRChat for past projects like Double Hip Tracker.

The small issue to take care of is the relativity of the OSC tracker data. The character moves in world space in our external program, but I don’t move in my physical space; the OSC tracker data needs to be sent relative to the head.

There are a few different ways to solve this, one of them might have been to send the head data to VRChat, but instead, I’ve decided to recalculate the position of the OSC trackers to be relative to the X and Z position of the characters’ head, shifted to the HMD X and Z position. The Y position (height) is ignored, as the head of the character moves up and down, while my physical head remains at the same height. I don’t want the feet to sink into the ground as the character bobs up and down.

I need the position of the HMD relative to the playspace in my Unity scene. There are also several ways to get that information, but I have already made SteamVR overlays (and background apps) in Unity in past projects, which includes a way to get the HMD position. Since it’s an overlay, it can still run while VRChat runs in VR.

In order to move the character in my external program, for this iteration:

• I’m still using the thumbstick to move around.
• I’ve settled with getting the avatar parameter for Velocity through OSC, and transmit that information to the character controller.
  • In the next iteration, I’ve switched to using the trackpad, explained later.
• The character direction is controlled by the rotation of the HMD (the forward direction of the HMD).
  • There’s a few shortcomings (such as looking straight down will flip the body, this is easily solved by using a dot product with the world down direction and then fall back using the up direction of the HMD), but it’s good enough for testing.

Here’s the result:

Some of the problems I had identified earlier are glaringly visible:

The velocity on the VRChat character does not match the movement of the character in my external program, as:

• the character in the external program is better grounded in reality, so it has less reactive movement (this may depend on the requirements of the motion capture performance),
• the velocity in worlds is way too fast for a normal human character, who would be sprinting all the time (and also vary from world to world).

This causes the feet to visibly slide on the ground, as the locomotion in-game is really snappy.

Locomotion comfort?​

VRChat rightfully uses a no-acceleration character controller for comfort. All motion is linear motion that is directly mapped to the thumbstick, with no smoothing whatsoever.

Preserving the user comfort tends to be important in games that target all ages; although in recent years, some VR developers are starting to specifically target a new class of young users who are almost born with VR legs and can handle extreme forms of locomotion.

I still have experiences in VR I cannot handle, especially:

• any game that involves rolling the character sideways with acceleration without a visible cage,
• or grounded experiences where the floor cannot be distinguished from the ceiling (i.e. some social VR music venues do this, where the floor and the ceiling are both video players).

I’m still willing to experiment with this, which led me to the second iteration.

The character in my external program becomes the thumbstick​

In this iteration, I will no longer move in-game using my thumbstick. Instead, the character in my external program becomes the thumbstick.

I will command the character in my external program to move around using the trackpad of my controller. The Velocity in VRChat is no longer used.

When the character in my external program effectively moves around in the world, that movement will be sent to VRChat using OSC inputs (/input/Vertical and /input/Horizontal).

I’m reading the trackpad using the OpenVR API. I cannot use the thumbstick because it causes a conflict with VRChat thumbstick locomotion, as the OSC input does not seem to take over the controller input when it is sending small values.

Thumbstick movement is relative to the HMD direction, therefore the movement of the character in the external program needs to be converted to become relative to the HMD direction.

This is the result:

This result is much better, the feet movement seems somewhat grounded especially during hard stops; but here’s some things to notice:

• The feet still slides but not as much, which indicates a mismatch between the movement of the VRChat avatar and the character in the external program.
• Fast rotation of the HMD will not translate to a fast rotation of the upper body of the character; this may require better integration with the motion capture data in order to emphasize the head direction.
• The head cannot bob up and down, as it is driven by the HMD position.
• The static VR arms now looks really awkward in contrast to the motion of the character.
• The character movement in the external program cannot account for changes in elevation; also the motion capture data probably does not have this information.

This is how it looks from the outside.

From the inside, the character locomotion is no longer linear, but it does not have smooth acceleration either, it’s somewhere in-between.

There is also a disconnect between the trackpad input and the in-game movement, as this introduces a delay between the input and the actual VR view moving around. This is really destabilizing, but strangely, I do not feel as much VR sickness from it.

Other thoughts​

There are still a few other issues, as, if I physically move in my playspace, the legs will not move, but if I command the character to move, then it will send inputs to VRChat to move in-world when it was not necessary. This incorrect feedback is probably solvable, but I’ve already achieved the objective I had set for myself for this proof of concept.

Otherwise, it looks fun, but most likely I will not be touching this proof of concept again, because it is just too much trouble to make it cope with full body tracking.

My takeaway: If I were to build a new iteration of this project, I would try to look into whether it is possible to build a social VR app where the character in first-person is animated differently from the character in third person, so that the networked character animation during locomotion looks good to others, while the character in first-person still remains comfortable to move around**.**

• In VR, first person locomotion would remain direct and linear with the input.
• The character locomotion in first person as seen on mirrors should be reactive and consistent with the first person experience.
• However, the character locomotion in local cameras not taken from a first-person perspective could be driven by third-person locomotion animation systems, whenever the character is moving using the thumbstick, so that it looks good in recorded footage.
• The local character as seen by remote users (= networked) could be driven by third-person locomotion animation systems whenever that local character is moving using the thumbstick.
• This would not apply to the context of social VR games with a competitive aspect where the position of the character in the world is critical for gameplay.

The objective would be to make the character locomotion within the world look interesting, without causing a disconnect in first-person.

It would be nice to also experiment with recording new motion capture data specialized for social VR (maybe with snappier movement, and some emphasis on the upper torso movement), or try some more sophisticated integrations, and see if it leads to an improvement.

I’ve made a tool to improve the shading of SkinnedMeshRenderers by recalculating select blendshape normals.

I’m gonna share what I learned from doing this.

Documentation:

Recalculate Normals | Haï~
Improves the shading of SkinnedMeshRenderers by recalculating select blendshape normals (and tangents).

The tool is under the MIT License. Please use it freely under the terms of this license!

https://github.com/hai-vr/prefabulous-avatar/blob/main/Packages/dev.hai-vr.resilience.prefabulous/Scripts/VRC/Editor/PrefabulousHaiRecalculateNormalsPlugin.cs

What are normals?​

In short:

• Place a vertical pole on the floor, that is perpendicular to the floor.
• If the floor is slanted, then the vertical pole is slanted.
• Place a vertical pole on every point of a mesh by following this rule.

This vertical pole is a normal. It is a piece of information that is used to figure out the direction where that point would receive the maximum amount of light. If the sun is right above the pole, that vertex receives maximum light.

This piece of information is also used for reflections (if you look at the point from above the pole, you reflect yourself), and a ton of other stuff in shaders.

Blendshape normals in the context of VRChat​

A blendshape deforms points of a mesh by moving them around.

Not only points are moved, but the normals also should be: Since the points move, then we need new information on how to tilt that vertical pole to match the new slant of the surface below it.

However, in the context of VRChat (actually, Unity, VRChat is not at fault here), there’s an issue.

There’s been a problem in Unity where:

• If you change a blendshape of the face, it would sometimes morph the shading of the entire avatar because the mesh normals are being contaminated by the delta normals of the blendshape itself, as they were incorrectly recalculated.
• Recalculating normals on the face is often the wrong thing to do on toon-shaded avatars, because avatar artists repaint the normals of the face to be different from what normals would be usually calculated for the curved surface of the face.
• A side effect of the above quirks is that the meshes dramatically inflates in size due to Unity’s incorrect recalculation, both in VRAM, and in compressed download bundle size.

Because of this, VRChat users are often encouraged to set the mesh settings to “Legacy Blendshape Normals”, or set it to not import or recalculate blendshape normals.

A consequence of this recommendation is that blendshapes have no delta normals: Blendshapes no longer have data that describes how to tilt that pole, so the shading will look incorrect.

If you flatten parts of the body, emboss it, or make it larger, the information that describes the surface slant will continue to describe the original surface, which will make it look weird.

Unfortunately this recommendation is the best, because there are only worse alternatives.

It would make sense for blendshape normals to be recalculated accordingly, as long as that blendshape that isn’t a toon face expression or a technical blendshape (i.e. hide parts of the body).

If you want to learn more:

Unityでカスタム法線メッシュにブレンドシェイプを適用した際に法線が崩れる問題の解決法 - Qiita
はじめにUTS2.0などのNPRシェーダを使ってキャラクターを可愛く見せようとした時、顔メッシュの法線編集は必須になってくると思います。しかし、Blenderなどでカスタム法線メッシュを作成し、…

OpenKnowledge/Normals, Tangents, BlendShapeNormals のUnityインポート設定の解説
この記事では、Unity のFBXインポート設定にある「Normals」等の解説を行います。 実際の画面では、以下の場所にある設定項目を主に解説します。Unity のバージョンは 2018.4.20f1 です。 # 先に結論を VRChat向けアバターとしては　`Normals = Import, Tangents = Calculate Mikktspace, Blend Sh...

Recalculating blendshape normals through scripting​

Unity has a function to recalculate normals on a static mesh.

I’m going to use it to recalculate the delta normals of each blendshape that should be recalculated.

Unity - Scripting API: Mesh.RecalculateNormals
Thank you for helping us improve the quality of Unity Documentation. Although we cannot accept all submissions, we do read each suggested change from our users and will make updates where applicable.

During my attempts, I stumbled upon an issue, which is specified in Unity’s RecalculateNormals documentation:

Imported Meshes sometimes don't share all vertices. For example, a vertex at a UV seam is split into two vertices, so the RecalculateNormals function creates normals that are not smooth at the UV seam.

This turns out to be a practical problem: If an artist splits parts of the body into different UV seams, then even when this is one single smooth shaded vertex in the avatar artist workflow, this single artist-authored vertex will be split into multiple vertices in the mesh data representation for the game engine, because they have different UV data.

UV data is used to determine where each vertex will sample the texture from. From the perspective of a game engine renderer, this means multiple vertices will be used for shading, even though they are just one from the artists’ perspective.

This technical detail results in an issue in the output of the RecalculateNormals function: The recalculation will place multiple vertical poles at different angles for the same artist-authored vertex.

This makes the UV seam visible in the normal data. It will negatively affect shading: light, reflections, and rims will be subjected to an abrupt cut, which is arguably worse than leaving normals unchanged.

In order to remediate this, I have opted for the following strategy.

• Before recalculating normals, I store the indices of every set of points that have the same position and normal into a data structure.
  • These points are effectively UV seams.
  • I do not check for tangent equality, since the tangent is based on the UV data which may be different.
• Then I recalculate normals using RecalculateNormals.
• After recalculating normals, I reuse that data structure to re-recalculate normals of only those indices by averaging and normalizing them.
• Then I recalculate tangents using RecalculateTangents.

This isn’t perfect, but it’s a good trade-off. See the source code here, especially the ReRecalculateNormalsInUVSeams method and its callers.

Overall strategy​

The overall strategy is the following:

• Create a SkinnedMeshRenderer for the purposes of baking, and set all blendshape weights to 0.
• Use BakeMesh to build a static Mesh from that skinned mesh.
• Recalculate the normals and tangents using the remediation described above.
  • I recalculate the normals and tangents even when 0 blendshapes are active, because we want the delta normals and delta tangents to be calculated based on what Unity thinks the normals should be, rather than what the normals are.
• This mesh is the base recalculated mesh.
• Then, for each blendshape to recalculate individually (we do not recalculate all blendshapes):
  • Set the blendshape weight to the maximum (technically, do this for each weight of each frame of the original)
  • Use BakeMesh to build another static Mesh from that skinned mesh.
  • Recalculate the normals and tangents using the remediation described above.
  • Store the difference between the normals of this individual blendshape mesh with the normals of the base recalculated mesh.
  • Store the difference between the tangents of this individual blendshape mesh with the tangents of the base recalculated mesh.
  • Those are the delta normals and delta tangents we’ll have to write back into the mesh later.
• After all of this is done, rewrite the modified blendshapes back into (a copy of) the original mesh.
  • The Unity API does not provide a way to edit a single blendshape, so we have to circumvent this by clearing all blendshapes and rewriting all of them.
  • That’s why delta normals and delta tangents are stored, so that we do this rewriting operation once instead of several times.

Since the subsequently calculated deltas is only based on Unity’s judgement, it means that if Unity judges that a normal doesn’t change, the delta remains zero.

Deltas must not be calculated by comparing the artists’ judgement with Unity’s judgement, otherwise the two will disagree on almost all vertices, resulting in a change of the shading of the entire avatar. It is specifically avoided here by recalculating the normals even on the base mesh that has 0 blendshapes active.

Even vertices that don’t move will be recalculated​

All vertex normals and tangents are recalculated, even if the vertex position is not changed by a blendshape. This is because when a vertex changes position, all of its smoothed neighbours will also change normals, even when that neighbour does not change position.

If recalculation were limited to only vertices that moves, I’ve experimentally noted that this causes a weird shading “aura” effect to be visible, especially on blendshapes that flatten parts of the body. Therefore it’s better to remove this limitation so that shading remains consistent.

This should cause a small increase in VRAM usage because of those neighbouring vertices being affected, and I am counting on the stability of RecalculateNormals to produce delta normals and tangents that are equal to 0 for all unaffected vertices.

Finally​

I’ve added an UI that suggests blendshapes to add. Since face expression blendshapes should probably not be recalculated, and conflict prevention blendshapes should not either, this is a user selection process.

Anyways, this tool is under the MIT License. Please use it freely under the terms of this license! Maybe add it natively to VRCFury or something.

Documentation:

Recalculate Normals | Haï~
Improves the shading of SkinnedMeshRenderers by recalculating select blendshape normals (and tangents).

https://github.com/hai-vr/prefabulous-avatar/blob/main/Packages/dev.hai-vr.resilience.prefabulous/Scripts/VRC/Editor/PrefabulousHaiRecalculateNormalsPlugin.cs

I am one of a handful of creators who have tried to communicate with Udon from an external process using MIDI.

Here’s me sharing what I learned from trying.

Why MIDI?​

For external processes to send data to Udon, there are only a few ways (non exhaustive):

• Streaming video players and GPU readback
• Polling a string loader
• Input system (Joystick data, VR inputs…)
• MIDI

MIDI, the one we’re interested in, is good for real time, and by real time I mean applications with less than a few milliseconds of delay.

For instance this can be used to sync the position of a real-life tracked object with the world.

Unfortunately, OSC is not an available option despite worlds being mentioned in the corresponding OSC docs.

VRChat Component​

Udon can receive MIDI data through the VRCMidiListener component. This component will listen to the first MIDI device it finds, unless a launch parameter is defined that specifies the name of the device to look for. All events are routed to a UdonBehaviour.

For clarity, MIDI events are not networked, so all processing is done on the local machine that sends and receives the MIDI data.

Virtual MIDI devices​

I don’t have any MIDI or musical background, I only care about MIDI as a possible way to transmit data from an external process to Udon.

In order for VRChat to receive MIDI messages, we need to create a device we can write MIDI events to. This can be done using a virtual MIDI device, and although apparently it’s natively supported in OSX, in Windows there’s no native support for virtual MIDI devices so you need to install a driver from a third party.

Scouring through the Internet I was not able to find a truly free and open source solution, so we can use proprietary or source-available free-to-download software with some usage restrictions. There are a few mentioned here: https://support.troikatronix.com/support/solutions/articles/13000081815-using-virtual-midi-ports

LoopBe1, and LoopMIDI often come up as results, but personally I’ve used sbvmidi, which worked out. Your mileage may vary.

• These software create an input MIDI device and an output MIDI device.
• You send events to the virtual output MIDI device.
• The events are re-emitted out of the virtual input MIDI device.
• That virtual input MIDI device is what VRChat listens to.

Sender library​

Personally I’ve used the DryWetMIDI .NET library so that I could use it in a non-VRChat SDK Unity project to send MIDI data.

Data types​

Midi in Udon supports 3 events:

• NoteOn
• NoteOff
• ControlChange

Each of these three event types have:

• A 4-bit number which is the channel
• A 7-bit number which is the note
• A 7-bit number which is either the velocity or a value

Encoding strings​

Although the data payloads that we will send from our external apps to Udon can be anything specifically crafted for a specific use in order to fit data buffer size constraints (see later in this article), let’s start with strings since it’s easy to process and debug.

Here’s the catch: I can’t find an easy way to use Encoding.UTF-8 in Udon in order to decode bytes to strings, even with tricks.

Therefore instead of encoding the string to UTF-8 bytes, I’ve decided to process each Char of the string, which is 2 bytes per char.

This way, it will be easier to reconstruct the string even if it has non-ASCII characters in it.

Transmitting 2 bytes using a single note​

All event types have a total of 18 bits, therefore it should be possible to encode a Char (16 bits) inside one event.

However, I’ve decided to do the following instead:

• Store the 7 first low bits of each Char into the two respective 7-bit numbers
• Use the NoteOn Event if the highest bit of the 1st Char is low
• Use the ControlChange Event if the highest bit of the 1st Char is high
• Use the channel 14 if the highest bit of the 2nd Char is low
• Use the channel 15 if the highest bit of the 2nd Char is high

This limits the number of channels that our encoder will use to only 2.

The problem: There’s a bug in VRCMidiListener/VRCPortMidiInput​

Everything would work great if it weren’t for an issue in VRCMidiListener, located in the internal class VRCPortMidiInput.

VRChat will raise exceptions if there are too many MIDI events received since the last frame was processed. Too many MIDI events will trigger a buffer over-read inside VRChat, and the events won’t be able to be processed properly. This can happen as low as around 128 events since the last frame.

If VRChat needs to process more than 8 times that maximum amount, it may also able to crash to client under certain circumstances. This only affects the machine that sends the MIDI events: the other players in the instance are unaffected since MIDI events are not networked anyways.

Therefore, in order to transmit data safely to VRChat using MIDI, you need to ensure VRChat’s own stability from the caller side:

• Stop sending more events than VRChat can process within a frame.
  • This can be done using an event or message queue.
  • Make sure you can handle queuing events for messages that are larger than what VRChat can handle in a single frame.
• Detect when VRChat has finished processing a frame, before sending more events.
  • This can be done by monitoring VRChat’s output logs using a FileSystemWatcher.
  • In the Udon behaviour that processes the MIDI events, if it has processed at least one note this frame, then after all notes are processed that frame (use LateUpdate), print a line that the sender will be able to detect.
• Detect when VRChat is not responding or when the user leaves the world, or handle timeouts.
  • This prevents the queue from growing too large, as the sender must detect that the frame has been processed first, which will never happen when VRChat is not responding.
• If VRChat is lagging behind the sender, avoid queuing data to be sent.
  • You don’t want the queue to grow too large with messages that will be irrelevant by the time VRChat has processed them.

That’s it​

The lower your VRChat framerate, the slower VRChat may be able to process incoming data.

The real solution would be OSC for worlds, but realistically, are you going to wait for that if you need to get things done?

When using TeleportTo continuously to create an alternate locomotion system, the player may spin out of control.

Here’s a bit of knowledge that has been known for years through word of mouth, but as far as I could tell, I haven’t found articles online that fully explain it. It’s as if people are supposed to figure out how TeleportTo (U#)’s AlignRoomWithSpawnPoint (U#) (which is missing from non-U# docs) and TrackingDataType.Origin (U#) (Docs) are supposed to work together just by looking at their individual descriptions.

This is because calling the function in the intuitive way by passing Networking.LocalPlayer.GetRotation(), there is no guarantee that the rotation of the player after the teleportation will be equal to the rotation of the teleportation.

In order to teleport the player to the same rotation, you need to:

• use the variant of the TeleportTo function that accepts the parameter VRC_SceneDescriptor.SpawnOrientation.AlignRoomWithSpawnPoint which will teleport the center of the player’s room instead of teleporting the player, and
• use Networking.LocalPlayer.GetTrackingData(VRCPlayerApi.TrackingDataType.Origin) to find the current center of the player’s room in the world

You can directly use the following function in your project:

/*
  This code snippet is released into the public domain.
  If the public domain is not applicable where you are, then it's released under the terms of
  the CC0 1.0 Universal, which is essentially the same as public domain.
*/

// Teleporting to self should always result in the player not moving
StableTeleportTo(Networking.LocalPlayer.GetPosition());

// Teleport to other position
public Vector3 destinationPos = ...;
StableTeleportTo(destinationPos);

private static void StableTeleportTo(Vector3 destinationPos)
{
    var playerPos = Networking.LocalPlayer.GetPosition();
    var playerOrigin = Networking.LocalPlayer.GetTrackingData(VRCPlayerApi.TrackingDataType.Origin);
    var posToOrigin = playerOrigin.position - playerPos;
    var destinationOrigin = destinationPos + posToOrigin;

    Networking.LocalPlayer.TeleportTo(
        destinationOrigin,
        playerOrigin.rotation,
        VRC_SceneDescriptor.SpawnOrientation.AlignRoomWithSpawnPoint
    );
}

This function is designed for continuous teleportation, so it does not accept any rotation parameters.

It calculates where the center of player’s room will be in such a way that the player will be located at the destination, and teleports the center of the player’s room to that location.

ResilienceVR is a social VR app I’m developing to experiment and iterate outside of the boundaries of existing platforms.

In this article, I will explain how my first experiment works: Enabling the ability to make the avatar IK react to objects and players, despite the latency.

Networked IK in existing applications​

In social VR, users are connected together in a single virtual space, usually puppeteering a single virtual representation of themselves (an avatar).

There are currently two major ways that social VR applications transmit the virtual representation of the player to other players:

• Either transmit the hip position and bone rotations of the avatar, or
• Transmit the position and rotations of critical bones of the avatar (such as head and hands) or hardware trackers.

These approaches come with various drawbacks. In the latter approach, the end users would need to perform additional computations to deduce the other players’ poses.

Regardless, both of these approaches result in a large quantity of data that needs to be sent through the network, so additional simplifications and optimizations are needed for this to scale depending on the application’s needs.

Generally, in the worse case scenario when everyone’s avatars are in close virtual proximity in the virtual space, network usage in such applications can be estimated by the square of the number of players, as the data of any new player joining needs to be sent to all the other existing players in the instance.

Without optimizations:

• Double the players
  an instance with 10 players would be 4 times more taxing on the network than an instance with 5 players.
• Multiply the number of players by 10
  an instance with 80 players would be 100 times more taxing on the network than an instance with 8 players.

This scaling issue is a main motivator for many optimization techniques, because networking non-static data can have a serious financial cost.

The problem with latency​

Users are still separated by latency, which in some cases can result in large delays between the players in the order of 200ms to even one full second.

These delays hamper the ability for the players to puppeteer their avatar in reaction to another player’s actions. Attempting to hold hands, dance together, or manipulate an object in the virtual space that is also being manipulated by another player, visually causes a mismatch due to the latency.

Bonding interactions​

In order to provide a partial solution to this issue, ResilienceVR experiments with an approach that enables some degree of reactive interactions between players, but not all. Social VR has various contexts, and the one I’m particularly interested in is bonding between users. Provide a way for users to hold hands and dance together by putting hands on each other.

For our purposes, we’ll focus on something that is known outside of VR as “victim control”, a term that is used by stage combat in the field of acting. When acting a fight scene, the actor who is the aggressor is not actually in control of the victim. The victim preserves their full freedom of movement. This means if an actor acts grappling the victim actor’s head, the victim actor’s head can freely move around at the victim’s will, not the aggressor’s will.

In our case, the person in control of someone’s limbs is not the person who initiated the interaction.

Context is important, so this experiment does not focus on other social VR interactions where players are put in confrontation to each other, such as a hands-to-hands fighting game or gun game.

A combat-focused networking could bring additional challenges such as requiring the ability for a player to take control of another players’ limbs, as the players engaged in combat would be confrontational in this context. What would happen to your hand if someone else were to grab your elbow away? A “networked Boneworks” would be a drastically different problem to solve. This could be a subject for a future experiment in ResilienceVR.

Interactions with non-players​

Even though this system was primarily designed for bonding between players, it just so happens that the system can easily be extended to enable interactions with items and objects in the virtual space, and even items held by other players.

Heavily inspired by single player games like Boneworks and Half-Life: Alyx, the feature was further extended to make the player’s hand attracted to objects, and make that attraction visible to other players without latency.

It should be noted that this is a rudimentary form of interaction, as there is no rigidbody simulation or collisions involved unlike those singleplayer games.

How it’s done​

In short, the local player hands tells the other players if they’re attracted to something. At the moment of grabbing, it tells other players where they’ve grabbed it. It is not continuous transmission of data.

Each player then computes how strongly each player’s hand is attracted to their respective objects and runs a partial IK solver to place their hand somewhere between their networked IK hand placement and the object’s hand placement; the stronger it’s attracted, the closer it is.

In detail:

Step 1: Local user stores the player pose for networked IK​

The normal IK solver runs, and the solved pose is stored so that it can be sent as part of networked IK. Networked IK poses are always sent with the local data in its configuration as if the entire system that is described below in the following steps didn’t exist.

All the users’ avatars are posed based on networked IK.

Step 2: Local user finds the most attractive object​

In local, the player’s hand searches for objects around it that it can be attracted to (if it’s not already grabbing an object).

For our purposes, the players’ limbs are attractive objects.

Once it finds available objects, it calculates where the hand would be placed on the object if it were to be fully attracted to it given the current hand placement.

With the potential hand placements computed, we calculate a magnetization amount for each potential hand placement, which is a scalar estimate of how close the current hand placement is to the potential hand placement. Also, some objects are more attractive than others.

The object that has the greatest magnetization amount is the winner.

• If the user is attracted to something, the user sends which object it is attracted to the other users. The object is represented by a number which is an identifier for attractive objects.
• If the user decides to grab that object, the user sends the hand position and rotation relative to that object. This is not updated after the initial grab action (with one exception explained later).
• If the user is not attracted to anything, the user sends the information that it is attracted to nothing.

The advantages of doing this is that it’s cheap. It’s event driven. Attraction doesn’t always change and the data being sent is minimal. There is very little need for interpolation of networked data outside of phase changes.

Since this computation is performed locally, we don’t need to run a search for attractive objects on other players. This may cause the transition to the attractive state to be delayed.

The magnetization amount can be reused to drive user input: If the magnetization amount is greater than a certain threshold, then the user can interact with that object, useful for climbing for instance.

Note that the magnetization amount is not transmitted to other players. Magnetization will be recalculated on each player to better account for delays.

Step 3: All users run the partial IK for users that are attracted​

The users solves the IK locally for all users that apply, including themselves.

The IK solver will not run locally for a given player:

• When the given player is not visible, or
• When the given player is not attracted to anything, or
• When the player is attracted to an object that is disabled, or
• When the player is attracted to an object that is located on someone’s avatar that is hidden (i.e. the animator not being updated because it is culled)

Doing this saves iterating on users that don’t need it.

ResilienceVR uses a custom written IK solver, which provides more control on what part of the IK can run, and how it can run, so it can choose to only resolve the arm based on the currently posed avatar.

The user computes the hand placement to that specific object, and then computes the magnetization amount.

This magnetization amount is used to interpolate between the networked hand placement and the final hand placement. This leads to the hand being attracted to the object as it approaches it locally, rather than remotely.

There are two ways to iterate:

• Run all the iterations of the solver on each user separately, in the same order for all players, or
• Run one iteration solver on each user, then apply the pose on every user, then iterate again.

They both come with drawbacks. In a scenario where each user decides to grab the others’ hand, or users decide to grab each others’ elbows, this can lead to stability issues that still need to be experimented upon.

Relative movement (in step 2)​

I want to allow that player to move their grabbed hand while still benefiting from networked IK.

When a player decides to initiate that action, it sends another networked event which contains:

• the position and rotation of the hand at the moment this action was initiated, and
• the position and rotation of the object being held at the moment this action was initiated.

This is never updated afterwards, so there’s no interpolation. In the solver phase, we’ll treat the player’s hand as being a difference relative to the object. This allows it to benefit from the smoothness of any networked IK without having to deal with interpolation ourselves in this step.

When the players decides to stop this action, it sends a new grab action with freshly baked positions and rotations.

Implementation in VRChat​

The “ResilienceVR IK Demo” Udon VRChat world is a port of the above system, brought back under the allowed limitations of the VRChat platform.

The current state of Udon is limiting due to execution slowness, but good enough to showcase this feat. It should be noted that this is a naïve port with very few Udon-specific optimizations, as I want to keep the changes in the code minimal from ResilienceVR.

Despite this, I still had to disable some parts of the IK solver for simplification. If this were a native Udon project, I would have written the solver more efficiently.

Since VRChat does not provide any access to the avatar’s meshes, nor write access to its bones, the world itself contains the models of the avatars as SkinnedMeshRenderers, and moves their armature, effectively making it seem like the user is wearing an avatar.

The avatar pedestal in the world ensures that the bone rotations and lengths are correct, and that the animator is running when visible. If the animator is not running because no mesh were to be visible, Udon’s VRCPlayerApi would not return updated bone rotations.

The VRChat world works in the following way:

Step 1: Local user stores the player pose for networked IK​

All players’ bone positions and rotations (using VRCPlayerApi) are collected and copied into memory. These positions and rotations are read and written multiple times in the IK solving process.

Unlike ResilienceVR, since VRChat is the one responsible for Networked IK, we don’t have to worry about any of this, since there is nothing the world can do to influence the transmitted bones in the future.

Step 2: Local user finds the most attractive object​

This part is almost identical to ResilienceVR, but uses Udon’s manual sync to transmit the events with the help of CyanPlayerObjectPool. The event driven nature of the original makes it very easy to carry over in Udon.

Step 3: All users run the partial IK for users that are attracted​

Udon doesn’t give sufficient access to players’ data to accomplish most of the optimizations.

In this case, I’ve limited to the following:

The users solves the IK locally for all users that apply, including themselves.

The IK solver will not run locally for a given player:

• When the given player is not wearing the correct avatar
• When the given player is not attracted to anything

We can detect if the player is wearing the correct avatar by measuring the individual distances between the hips, spine, chest, and neck. This can lead to false positives if the player is wearing an original avatar of the same model.

Players are iterated individually, with the poses applied immediately. This can lead to some discrepancies as the execution order may affect the result.

The solved poses are then applied to the bones of the SkinnedMeshRenderers’ armatures located in the world.

The result​

This experiment resulted in the creation of a VRChat world where you can experience it for yourselves. I want my work to inspire existing social VR users to desire and demand more from their favorite social VR platforms.

ResilienceVR will continue to be a social VR app that I can use to iterate upon new ideas to share.

There is still a lot to research, as player-to-player interactions creates a bunch of interesting challenges in terms of user experience. I’ve received a lot of feedback, especially regarding the grabbing interaction and the strength of magnetization that I’ll be able to work upon.

https://twitter.com/vr_hai/status/1653570784999120899

Patreon​

Thank you to all of my supporters on Patreon, Pixiv Fanbox, and tippers of my free tools on Booth!

If you like my work and research, consider supporting me on Patreon!

Haï~ | creating Virtual reality gimmicks and productivity tools | Patreon
Become a patron of Haï~ today: Get access to exclusive content and experiences on the world’s largest membership platform for artists and creators.

I’ve made it possible for friends to draw using my 3D printer from within VR.

How it started​

I’ve been using my 3D printer for a couple months now, and I’ve started to wonder if I could somehow connect it to a VRChat world so that friends could control my 3D printer with a pen attached to it.

However, so far, I’ve only been printing using a SD card where I put gcode files output from slicing software, and then plug it directly to the printer.

For maximum effect, I wanted the drawing to materialize live with the lowest latency possible.

I’ve asked on Twitter with my basic pitch:

https://twitter.com/vr_hai/status/1618541782899519489

Soon I was told that it’s possible with serial port communication using the USB-B port on my Prusa MK3S+ printer. Thankfully my 3D printer is very close to my main rig, so I immediately plugged it to try.

Prior art​

After making this tweet asking for advice on how to send gcode in real time, I’ve been shown examples of prior art, which is cool! This one uses a plotter.

getting an autograph from a vtuber!?? 😱
VShojo #ENVtuber Vshojo:Twitter ⯈ https://twitter.com/VShojoOfficialInstagram ⯈ https://www.instagram.com/vshojo/Website ⯈ https://www.vshojo.com/VShojo tal...

The plan​

To make lines drawn by the pen (by myself or by other players) materialize onto the printer, the plan is to modify QvPen (which is the pen prefab) to output the pen line positions in the game logs using Udon, then write a log parser that will continuously transmit the pen positions as gcode instructions for the printer through the serial port.

Debug.Log the pen position/click/release → Log parser → Remap to gcode → Serial port

This needs to be done with the lowest possible latency for greater effect, as the printer will be live streamed back into the world.

Learning serial port communication and gcode​

After a quick google search, I’ve looked at my Device Manager to get the communication port of my 3D printer (the COM ports).

Then I searched up for references on what’s the serial port configuration for my printer (baud rate, etc.), which led to some OctoPrint logs that referenced it.

_port = new SerialPort(portName, 115200, Parity.None, 8, StopBits.One);
_port.Open();

Then I looked up for basic gcode functions: Moving the print head in the X and Y axis (which are the axes for the print bed), and the Z axis (up and down), and how to control the movement speed. The main gcode is G1.

private const int InternalFastestMillimetersPerSecond = 200;
private const int FastestMillimetersPerMinute = InternalFastestMillimetersPerSecond * 60;

// Editor's note: SafeWriteLine and other functions are described in the next section

public void AsFastAsPossible()
{
    SafeWriteLine($"G1 F{FastestMillimetersPerMinute}");
}

public void MoveTo(float xMillimeters, float yMillimeters)
{
    if (IsFloatIllegalRange(xMillimeters) || IsFloatIllegalRange(yMillimeters))
    {
        throw new ArgumentOutOfRangeException($"{xMillimeters} {yMillimeters}");
    }
    SafeWriteLine($"G1 X{xMillimeters:0.000} Y{yMillimeters:0.000}");
}

public void PressPen(float amount01)
{
    if (amount01 < 0 || amount01 > 1)
    {
        throw new ArgumentOutOfRangeException();
    }

    if (amount01 < 0.001f)
    {
        MoveZToAndIgnoreLess(Retracted);
    }
    else
    {
        MoveZToAndIgnoreLess(FullyPressed * amount01 + BarelyPressed * (1 - amount01));
    }
}

public void ReleasePen()
{
    MoveZToAndIgnoreLess(Retracted);
}

public void MoveZToAndIgnoreLess(float zMillimeters)
{
    if (IsFloatIllegalRange(zMillimeters))
    {
        throw new ArgumentOutOfRangeException();
    }

    if (zMillimeters < 9f)
    {
        return;
    }
    SafeWriteLine($"G1 Z{zMillimeters:0.000}");
}

// Editor's note: SafeWriteLine and other functions are described in the next section

Mistakes​

In the learning process I’ve done a few mistakes that caused the printer to physically malfunction. The malfunctions manifested in the following ways:

• The Z axis (print head moving down) crashed the nozzle into the bed, while the bed was moving, causing physical damage to the print bed. I tried a test print after recalibration afterwards, and it seems to work fine, but maybe there’s some additional damage that I don’t know of yet.

https://twitter.com/Ybalrid/status/1618583370199609348

• The X axis (print head moving laterally) crashed repeatedly into the left side, causing the motor to stutter.
• The X axis was no longer parallel to the print bed, which is a problem that was auto-detected by the leveling process. My printer has a process to fix itself by moving up as far as possible.

The reasons for these malfunctions are the following:

• I was accidentally sending NaN (not a number) to the printer axes due to a division issue. This can even make it move on Z axis even though movement on Z was not requested, which caused the nozzle to crash onto the bed.
• I was incorrectly communicating with the printer. After each operation sent to the printer, I need to wait for the printer to send the message ok back to my program.

These are the lessons I’ve learned from these mistakes:

• Stay ready to press the reset button or shut the printer off when trying in production for the first time.
• When trying new things, use the printer to manually move the Z axis up to the middle (On the printer, use Settings > Move axis > Z, and dial up). Since Z movement is slower than X and Y, this will provide a lead time in case the printer does something unwanted.
• Defensively verify my outputs to the printer. If an output is going to send numbers outside of an expected range, or illegal numbers such as NaN or infinity, terminate the program immediately. Note that I don’t do any sort of heating operation, and I always keep an eye on my printer, so I don’t have to worry about shutting the printer’s heaters off first.
  I don’t normally verify function outputs in general programming, only inputs, but since this deals with low level hardware with physical consequences, I now prefer taking a defensive approach.

    private bool IsFloatIllegalRange(float value)
    {
        if (IsFloatInvalid(value)) return true;
        
        return value < 0 || value > 210;
    }

    private bool IsFloatInvalid(float value)
    {
        return float.IsNaN(value) || float.IsInfinity(value);
    }

• All gcode sent to the printer is passed to a function that blocks until ok is answered.

    private void SafeWriteLine(string message)
    {
        _port.WriteLine(message);
        WaitForOkResponse();
    }

    private void WaitForOkResponse()
    {
        string readLine;
        do
        {
            readLine = _port.ReadLine();
            Console.WriteLine($"Read line: {readLine}");
        } while (!readLine.EndsWith("ok")); // FIXME: Maybe don't use EndsWith
    }

Making QvPen output to logs​

I’ve modified QvPen to output the trail renderer position every frame. QvPen normally syncs all drawn lines after the line is complete to ensure that the drawing looks the same for everyone in the instance, but I’ve opted for outputting the line as it’s being drawn to improve the effect of low latency at the expense of drawing accuracy:

Since I’m using VRCDN to stream the video of my 3D printer live back into the world, there’s less than a second of latency, which is really neat.

The changes in QvPen are the following:

// QVPen_Pen.cs
private void LateUpdate()
{
    // ...
    if (trailRenderer.gameObject.activeSelf)
    {
        var pos = inkPositionChild.position;
        Debug.Log($"[GCODETRANSFER]::POS {pos.x:0.000000} {pos.y:0.000000} {pos.z:0.000000}");
    }
}

private void StartDrawing()
{
    // ...
    Debug.Log($"[GCODETRANSFER]::START");
}

private void FinishDrawing()
{
    // ...
    Debug.Log($"[GCODETRANSFER]::FINISH");
}

I need to be present in the world instance to log this.

I’ve also modified this line to quickly disable the eraser function of the pen:

// QVPen_Pen.cs
private const float clickTimeInterval = 0.0f; // was 0.2 before

Players will be encouraged to draw on a square whiteboard, so I’ve made the pen stick to the whiteboard plane:

// QVPen_Pen.cs
public Collider[] alwaysBypassColliderMode;

private void OnTriggerEnter(Collider other)
{
    var isAlwaysColliderMode = useSurftraceMode;
    if (!isAlwaysColliderMode)
    {
        foreach (var bypass in alwaysBypassColliderMode)
        {
            if (other == bypass)
            {
                isAlwaysColliderMode = true;
                break;
            }
        }
    }
    
    // Edit the following line to replace `useSurftraceMode` with `isAlwaysColliderMode`
    if (isUser && isAlwaysColliderMode && VRC.SDKBase.Utilities.IsValid(other) && !other.isTrigger)
    // ...

The box collider of the whiteboard dragged into alwaysBypassColliderMode

Converting to gcode​

QvPen outputs the world absolute coordinates of the pen, disregarding where this will land in the picture. Converting world coordinates to printer coordinates is handled on my external program, as a personal choice. An advantage of doing it at this step is that we can correct some mistakes later in software, as long as the precision is sufficient.

Log parser is done continuously using a FileStream, StreamReader, and FileSystemWatcher. There are other ways to do it, this is the one I ended up using without any particular preference.

After filtering for the lines output by QvPen, the work is pretty much done already.

One thing to watch out: when the user starts drawing, don’t press the pen immediately. We have to to wait until the next position is received, move to it, and only then press the pen.

The whole process is blocking, there is no async magic in play: Read a valid line, and immediately try to send it to the printer, and wait for the printer to answer ok. Then read the next line, etc. Especially, there is no post processing of the received data whatsoever. Every position output by QvPen is converted into gcode. I didn’t have to do any simplification magic. It’s really dumb but it just works.

// Log parser
bool willPress = false;
void ParseLine(string line)
{
    if (line.StartsWith("POS"))
    {
        var strings = line.Split(" ");
        var x = float.Parse(strings[1]);
        var y = float.Parse(strings[2]);
        // FIXME: Whiteboard bounds
        var xRel = Math.Clamp(x + 0.5f, 0f, 1f);
        var yRel = Math.Clamp(y - 0.25f, 0f, 1f);
        prusa.DrawTo(xRel, yRel);
        if (willPress)
        {
            prusa.PressPen(1f);
            willPress = false;
        }
    }
    if (line.StartsWith("START"))
    {
        // The pen needs to move to the next drawing position before pressing
        willPress = true;
    }
    if (line.StartsWith("FINISH"))
    {
        prusa.ReleasePen();
        willPress = false;
    }
}

Attaching the pen​

The pen is held by painter’s tape. Maybe I can 3D print a more robust attachment at another time. Calibration is done by hand. Pressing and releasing the pen in software is done by moving the Z axis very slightly.

Results​

I invited a friend to draw there and it just works! I’ve made a mistake in remapping the whiteboard coordinates to printer coordinates, but it turned out fine.

The world had a live stream player connected to my VRCDN account, with low latency enabled. The latency is absurdly low, which adds to the effect.

https://twitter.com/vr_hai/status/1618915110415896576

I’ve created a Subtitle system for my “Offline” world. This was originally intended for me to watch a JP movie with English subtitles from YouTube.

Subtitle systems already exist but the few ones I know are designed for movie lists. This one is specifically for letting users pasting subtitles for arbitrary videos.

SubtitleManager.cs - Shared to twitter
Instantly share code, notes, and snippets. SubtitleManager.cs - Shared to twitter You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or window. Reload to refresh your session. Reload to refresh your session.

Another user has built upon this implementation.

https://twitter.com/loljacklul/status/1510610024925741065

Here’s a postmortem of my experience while creating this subtitle prefab.

Subtitle format​

I’ve settled with the SRT subtitle format (SubRip), as it appeared to be widely supported. It can be easily parsed, and there’s no much layout concerns about it.

Subtitles are usually sorted by the time they appear, and this becomes an assumption in my Udon implementation: Parsed order is assumed to be the appearance order.

The only weird quirk about this format is that it’s not really specified what should happen when two ranges of timestamps overlap ( https://www.youtube.com/watch?v=FqJ21-GaYew ).

The approach I’ve taken is that when two subtitles overlap, multiple subtitle text will be displayed with a line break in between, and the display order is always the timestamp order; not the “last appearance” order.

In addition, it is assumed that an overlapping subtitle A will not overlap again with a subtitle C if a subtitle B that started after A started, finishes before A finishes.

Copy pasting​

I am making a video player world, not a movie world, so there is no predefined list of movies. I cannot preload subtitles, the user is free to use any SRT subtitle of their choosing.

Some movies or documentaries have very large subtitles, crossing the 300KB range in file size. However, Unity Text fields can only support around 16K characters, which makes copy pasting fail.

To work around this, I am using a TextMeshPro Text field, which does not have this limitation, letting the user paste absurd amounts of data. This causes several problems:

Not accessible via Udon​

TextMeshPro text is not accessible via Udon. In order to copy the text, I had to wire up some Unity events to set the Text element of a Unity UI Text object. Then, I could access the UI Text in Udon.

When the text is displayed, it lags during and after pasting​

Pasting large amount of text will cause two issues:

• It causes a huge hitch right after pasting, there is no choice but to wait a few seconds.
• While subtitle text is displayed in the field, performance will drop significantly. The performance drop was significant when I tested with a 300KB subtitle file in Japanese.

The text field needs to be cleared as soon as it is submitted in order to regain the performance. However, I have personally not found an easy way to clear the field. Maybe there is a way but I don’t know. The way I did it, is to duplicate (VRC Instantiate) a hidden template copy of the Text field hierarchy, and destroy the previously used text field hierarchy.

Syncing​

If the local user wants to benefit from the subtitles pasted by another user who shares the preferred language of the instance, they need syncing.

Following advice, requesting serialization of anything above around 60KB may fail.

To work around this, I’m chopping up the string with an arbitrary number of characters (8192 characters), and sending them one by one. Since I don’t know the underlying network constraints regarding string sync, the size in characters is intentionally way lower than the maximum size in bytes.

Given the size of the problem, increasing this number or decreasing this number by a bit will not make a huge difference in transmission sync. Remember this is used for watching long movies, the subtitles are not going to change every minute, and players are expected to stay in the instance longer, so sync speeds are not a concern. For the same reason, there is no string compression because that would be overengineering in that context.

Tracking​

Before sending text to be displayed, I am tracking the video player current position.

During normal operation, the subtitle manager keeps track of the last displayed subtitle index, displays it and any other future subtitle (if there are any overlapping ones), and moves on the the next after it is complete, so that we don’t traverse the entire array every frame.

When the video player rewinds at any point, the subtitle manager reads all subtitle timestamps from the 0 index of the array up to the first timestamp that matches the current time.

Displaying​

To display the subtitles, I am exploiting two TextMeshPro Text tricks:

• Marks as a backdrop ( http://digitalnativestudios.com/textmeshpro/docs/rich-text/#mark )
  Marks allows to overlay text. Normally, this is used for outlining, but here, we want it to be used as a subtitle background. If we just outline the text, it will cause it to be darker.
  To achieve this, there are two Text objects, one for the normal text, one just for displaying the background.
  The background Text is fed with $"<mark=#000000>{buildingText}</mark>"

• Outline for black border ( http://digitalnativestudios.com/textmeshpro/docs/shaders/ )
  In order to increase the legibility of the text, I’ve added a black border to the subtitles on top of the existing backdrop. This is done by changing the outline setting of the material.

Dyslexia​

Apparently the above settings seems to work well for someone with dyslexia.

https://twitter.com/exe_elf/status/1496023742048313344

https://twitter.com/exe_elf/status/1496060055992639489

thank you! also thank you for making this yellow, that always makes it a bit easier to read subtitles with dyslexia for me
For me personally, rounder open fonts, and bright colors (like yellow!) work wonders, tysm 4 the consideration

I’ve added an option to choose a custom font claimed to be better identifiable for dyslexic users, even though the current font seems to be fine.

Vector of attacks​

The subtitle system was not designed to resist attacks, as it’s meant to be used for chilling in a trusted environment with friends.

In a public environment, a motivated attacker could use a malicious subtitle file to attempt displaying large amounts of text to cause a performance hog, or to use problematic rich text in subtitles, or to cause a large amount of text switching, or by clogging the network in one way or another causing object ownership issues.

Various sanity measures may have to be put in place to reduce the use of this system as a vector of attacks. The same is true for any system that accept user input, like Udon chat boxes.

During the development of “Animation Viewer”, I’ve encountered several issues that I want to document here.

How Animation Viewer was born​

Animation Viewer was created by accident while I was working on my project, ComboGestureExpressions 2 (also tentatively called ComboGestureDynamics).

I wanted to make it easy to drag-and-drop animations into the CGE UI, but one problem stood, is that the user cannot see the animation before it is dropped into the CGE UI.

At that point, I already had a system to generate previews on animations on an animator, but it was used inside CGE UI.

So, I was looking for ways to generate previews on animations directly in the Project View.

Failed approach using RenderStaticPreview​

I tried to create a custom Editor for the AnimationClip type.

By implementing an Editor, I was hoping to override the RenderStaticPreview method in order to bake the animation preview to be rendered in the project view just like materials. However, that did not go as planned:

RenderStaticPreview somehow updates little thumbnail that can be seen in the Inspector, but for some reason the thumbnails of the assets themselves were not getting updated.

Returning a new Texture2D would cause the animation asset not to change. However, returning the Texture2D that is returned by AssetPreview.GetMiniTypeThumbnail would result in a blank icon for some reason.

In addition, there were numerous crashes that I could not explain.

During my searches for a solution, I found and settled on the current approach using the EditorApplication.projectWindowItemOnGUI callback, which went well. In this approach, I use the callback to add items to a queue, and when items are added to a queue, a new task is scheduled for the next frame to actually render the queued items into their newly allocated texture.

Workaround: Animator moves to origin​

In order to generate the preview, I copy the entire Animator hierarchy, hide the original, render the several animations using the copy, delete the copy, and then show the original again.

The expected result is that the copy is animated in-place. However, for a reason I cannot explain, the animator actually moves to the origin of the world after the first sampling. This is despite the animator having no RootT or RootQ properties in them.

In order to fix this, I record the position and rotation of the copy, and then apply that position again manually after the sampling ends.

Workaround: Muscles are not updated​

When an animation is sampled, for some reason, the muscles of the animator will not update on subsequent AnimatorMode samplings on the same frame. This causes all animation thumbnails to have the same muscles positions, which is incorrect.

In order to fix this, I’ve found through trial and error that disabling the GameObject of the Animator and re-enabling it again on the same frame, does work around that issue.