Passthrough camera access is quite the buzz in the XR (Extended Reality) community these days. While we have a clear idea where companies like Meta, Apple, and Pico stand on the issue, everyone’s curious about Google’s approach with Android XR. After a direct chat with them, I can share that they’re planning to roll out a solution akin to what we see on smartphones. Read on for the details!
## The Camera Access Conundrum
Let’s pause a moment for those who might be puzzled about what I’m referring to. As most of you know, the latest standalone VR headsets are essentially MR (Mixed Reality) devices, offering users an RGB passthrough view of their surroundings through front cameras. This capability has paved the way for fantastic mixed reality applications like Cubism, Starship Home, and Pencil, among others.
These headsets use the camera feeds to provide passthrough views, allowing developers to harness these video frames to enrich users’ experiences through AI and computer vision. As I’ve mentioned on my blog before, I’m convinced that camera access is key to unlocking true mixed reality, making applications more context-aware of the physical world. For example, I once experimented with a hacked camera access method on Quest to prototype an AI+MR app for interior design—a feat that wouldn’t have been feasible otherwise.
But with great power comes great responsibility, and the flip side to this fabulous tech is the privacy concern. If an unscrupulous developer gains camera access, they could clandestinely capture environmental images and exploit them to extract personal details, such as credit card numbers or ID information visible during app use. There’s also the risk of capturing unwitting images of faces or bodies, potentially for malicious purposes.
Balancing user privacy with mixed reality’s potential is crucial.
## XR Companies’ Stances
Initially, there were no hurdles to camera access. Some of you may recall my past experiments on the Vive Focus with NTW (around 2019), where we explored diminished reality, Aruco marker tracking, and sound reactivity.
As MR gained traction, companies prioritized privacy, restricting camera frame access to safeguard users. Giants like Meta, Pico, HTC, and Apple all followed suit, blocking developers from accessing camera frames.
This became the standard until the XR developer community recognized the critical need for camera access, pushing manufacturers to loosen these restrictions. Advocates like Cix Liv, Michael Gschwandtner, and I united to demand clear and user-consented access to camera frames to enable object recognition and advanced computer vision on XR devices. We questioned the disparity between phone and XR device policies, since smartphone camera access is often granted through a mere permission request.
This collective effort bore fruit as XR companies began reconsidering, with Meta planning a “Passthrough API” release earlier this year. But what’s Google’s game plan for Android XR?
## Android XR: Mirroring Smartphone Functionality
Most global smartphones run on Android, and currently, app developers can request camera access via user permission. If granted, developers specify the camera ID (0 usually being the rear camera) and can then manipulate the camera feeds as desired.
Google aims to maintain Android XR’s app compatibility with general Android apps, hinting at a similar framework for the new OS. Following several unofficial rumors, I reached out to Google for confirmation regarding Android XR’s camera access. Here’s what a spokesperson stated:
“Just like with any Android app, developers, upon user consent, can utilize existing camera frames for XR purposes. Our article for developers elaborates on various permissions apps can request.”
To access the main world-facing camera (like a smartphone’s rear camera), apps need to request camera permissions. Developers can get the world-facing camera stream (camera_id=0) or the selfie-camera stream (camera_id=1), whether it’s on Android phones or XR headsets. These streams are accessed through standard Android Camera APIs, such as Camera2 and CameraX.
However, when accessing the selfie-camera, apps receive a user’s avatar video stream. This stream comes from avatar provider apps on the headset, leveraging tracking data from OpenXR APIs based on physical camera input that monitors movements and expressions.
With Android Camera2 and CameraX, developers can seamlessly manage camera streams on Android XR-based headsets and glasses just as they do on phones—grabbing frames, saving media, or employing ML analysis, which is fantastic news!
While the front camera offers standard views, the rear camera presents a recreated avatar perspective, similar to Apple’s Vision Pro approach. This setup ensures consistency with smartphone behavior: apps accessing the “rear camera” see the world in front of the user, and the “selfie camera” reveals the user’s avatar face.
Google’s intent is clear: enabling Android apps to operate out of the box on Android XR by ensuring cohesive camera and permission functionalities is a smart move. I personally support maintaining permission protocol consistency between phones and headsets/glasses.
You might wonder about accessing raw camera streams. Unfortunately, here’s the current stance:
“At present, we aren’t providing a route for applications to access non-standard camera data, such as the forward-facing or reconstructed inward camera feeds.”
This response implies exclusion of non-standard stream access, which isn’t available right now but could open in the future, maybe for enterprise use.
Now, what about Unity development? Android’s Camera2 and CameraX are native classes, so what can Unity developers use? If Android XR mirrors phone protocols, the Unity class WebcamTexture could potentially grant camera stream frames. If not, some creative JNI work to create a native Unity wrapper library for CameraX features might be the workaround the community needs.
## A Quick Note on Android XR
Bear in mind, Android XR is still in preview with no officially released headsets yet. So, the outlined features could evolve before the official launch. I doubt drastic changes will occur, but it’s something to keep in mind.
## The Countdown to Expansive Camera Access
With Google and Meta paving the way, it’s likely other companies will follow, making 2025 poised for groundbreaking developments in mixed reality. I eagerly anticipate the remarkable innovations developers will bring forth!
(Header image courtesy of Samsung)
Disclaimer: This blog involves advertisements and affiliate links to sustain itself. If you click an affiliate link, I earn a small commission, and you can read my boring full disclosure here.
## Related
Share this with other innovators!
