WebXR, Vision Pro, and Multiplayer

I got hands on the Vision Pro device for a day with XCode set up 🎉

Hey all! This is Asad from Playroom and I got some exciting update to share. We applied for Apple's Developer Labs to get access to Vision Pro at Cupertino. Luckily, we got selected for the first day! Probably one of the earliest non-Apple devs to get hands on Vision Pro.

My main goal was to try a few code demos, mostly explore WebXR on it, make a multiplayer demo using Playroom Kit and try to benchmark the device for a bunch of AI use-cases that we are interested in (like inference time with Stable Diffusion running on-device). Unfortunately, I am not allowed to talk much about my experience. But it was positive overall and I had fun!

Preparing for cross-platform XR revolution

Right now we only have Oculus Quest headsets in the mainstream world, but when Vision Pro arrives, we will likely need to make cross-platform XR stack to make games in since the tech stack is completely different for both platforms. In short, the main options for making a cross-platform XR game are either by using Unity or by WebXR.

We like WebXR because there is no installation required which reduces friction, specially for a multiplayer game. This is the same reason we at Playroom are betting on web-first multiplayer games.

WebXR on Vision Pro

The simulator supports WebXR but right now it only supports the fully immersive mode and not the mixed-reality mode. WebXR is hidden behind a feature flag in Safari on Vision Pro. You need to enable it from Settings:

Developing a quick multiplayer game with WebXR and Playroom Multiplayer SDK on Vision Pro proved fairly seamless! Used three.js and Playroom's built-in Gamepad API to connect game controller to the game. Video recording wasn't allowed with Vision Pro, but here is a video of me playing the game an iPhone using a game controller WebXR Viewer.

Code Overview

This is a standard three.js game with xr enabled:

import { XRButton } from 'three/addons/webxr/XRButton.js';
// ...
renderer = new THREE.WebGLRenderer( { antialias: true, alpha: true } );
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( window.innerWidth, window.innerHeight );
renderer.setAnimationLoop( render );
renderer.xr.enabled = true;
document.body.appendChild( renderer.domElement );
document.body.appendChild( XRButton.createButton( renderer ) );

Added Playroom SDK which handles the multiplayer and gamepad API:

const { onPlayerJoin, insertCoin, isHost, myPlayer, Joystick, isStreamScreen } = Playroom;
let players = [];

// ...

await insertCoin({streamMode: true, allowGamepads: true});
onPlayerJoin((state) => {
  // Create a plane for the player
  const plane = createPlane(scene, state.getProfile().color.hex);
  // Create a joystick for the player
  const joystick = new Joystick(state, {type: "dpad"})
  players.push({state: state, plane, joystick });
});

Game can now be played using game controllers. It just needed to read the gamepad state and update the player's plane position in the main game loop:

function render() {
  renderer.render( scene, camera );
  players.forEach((player) => {
    const playerState = player.state;
    const plane = player.plane;
    const dpad = player.joystick.dpad() || {};
    if (dpad.x === "left"){
      plane.mesh.rotation.y += 0.03;
    }
    if (dpad.x === "right"){
      plane.mesh.rotation.y -= 0.03;
    }
    if (dpad.y === "up"){
      plane.mesh.rotation.z -= 0.03;
    }
    if (dpad.y === "down"){
      plane.mesh.rotation.z += 0.03;
    }

    // Move the plane forward towards the direction it is facing
    // ...
  });
}

Limitations

• Do note that WebXR on Vision Pro is not perfect. It doesn't yet support immersive-ar, which means no mixed-reality experiences yet. I hope the support for immersive-ar lands soon.