Passthrough

Introduction

Passthrough is a feature which allows developers to show the users’ surroundings in the virtual world. It is useful in applications that requires users to be aware of their surroundings in real time, e.g. Passthrough Underlay for Mixed Reality applications.

Contents Prerequisite How to use Resources

Prerequisite

To use Passthrough, you have to install the VIVE Wave XR Plugin and VIVE Wave XR Plugin - Native packages.

How to use

The Passthrough feature can be controlled through native APIs provided in the VIVE Wave XR Plugin - Native package. To call those APIs, you have to include the Wave.Native namespace in your script.

There are three types of Passthrough available for you to use, which will be introduced in the following sections.

Overlay

Passthrough Overlay is a type of Passthrough that covers the content layer.

Here are the APIs for controlling the behavior of Passthrough Overlay:

//Passthrough Overlay APIs

bool WVR_ShowPassthroughOverlay(bool show, bool delaySubmit = false, bool showIndicator = false)
//Use this API to show/hide Passthrough Overlay.
//Parameters
//bool show: Set this to true for showing and false for hiding passthrough overlay. The system passthrough(surrounding or out-of-boundary) would be disabled if this parameter is set to true.
//bool delaySubmit: Set this to true for delay the timing for completing frame submission when the passthrough overlay is showing to improve the visual latency. False by default.
//bool showIndicator: Set this to true for showing and false for hiding the controller indicator on passthrough overlay. False by default.
//Return value
//true: Passthrough overlay is supported by the runtime on the current device.
//false: Passthrough overlay is not supported by the runtime on the current device.

WVR_Result WVR_SetPassthroughOverlayAlpha(float alpha)
//Use this API to set the transparency (alpha value) of the Passthrough Overlay.
//Parameters
//float alpha: The target alpha value. Should be a float in range [0f, 1f].
//Return value
//WVR_Result.WVR_SUCCESS: Alpha value is set successfully.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Passthrough Overlay is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Passthrough Overlay is not supported on this device.

//Passthrough Overlay Sample Code
using Wave.Native;

void ShowPassthroughOverlay()
{
    Interop.WVR_ShowPassthroughOverlay(true); //Show Passthrough Overlay
}

void SetPassthroughOverlayAlphaValue()
{
    Interop.WVR_SetPassthroughOverlayAlpha(0.5f); //Set Alpha of Passthrough Overlay to 0.5f
}

void HidePassthroughOverlay()
{
    Interop.WVR_ShowPassthroughOverlay(false); //Hide Passthrough Overlay
}

Underlay

Passthrough Underlay is a type of Passthrough that can be occluded by the content layer, i.e. objects rendered in-app by Unity can occlude the Passthrough Underlay.

Here are the APIs for controlling the behavior of Passthrough Underlay:

//Passthrough Underlay APIs

WVR_Result WVR_ShowPassthroughUnderlay(bool show)
//Use this API to show/hide Passthrough Underlay
//Parameters
//bool show: Set this to true for showing and false for hiding passthrough underlay. The system passthrough(surrounding or out-of-boundary) would be disabled if this parameter is set to true.
//Return value
//WVR_Result.WVR_SUCCESS: Parameter is valid.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Passthrough Underlay is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Passthrough Underlay is not supported on this device.

//Passthrough Underlay Sample Code
using Wave.Native;

void ShowPassthroughUnderlay()
{
    Interop.WVR_ShowPassthroughUnderlay(true); //Show Passthrough Underlay
}

void HidePassthroughUnderlay()
{
    Interop.WVR_ShowPassthroughUnderlay(false); //Hide Passthrough Underlay
}

Note

In order for Passthrough Underlay to be visible and properly occluded by in-app objects, the area which the Passthrough Underlay should be visible should be in transparent black (i.e. Color(0, 0, 0, 0)). To achieve this, you either have to use a shader similar to the one used by Multi Layer Underlays for Underlay Alpha Blending, or change the Main Camera background color to transparent black and set the Clear Flag to Solid Color.

Example of a Camera component with transparent black Background Color and Clear Flag set to Solid Color

Projected (Overlay)

Projected Passthrough is a type of Passthrough that will only be showed in a specific area, which is determined by the mesh data and pose you provide.

Here are the APIs for controlling the behavior of Projected Passthrough:

//Projected Passthrough APIs

WVR_Result WVR_ShowProjectedPassthrough(bool show)
//Use this API to show/hide Projected Passthrough
//Parameters
//bool show: Set this to true for showing and false for hiding projected passthrough. The system passthrough(surrounding or out-of-boundary) would be disabled if this parameter is set to true.
//Return value
//WVR_Result.WVR_SUCCESS: Parameter is valid.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Projected Passthrough is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Projected Passthrough is not supported on this device.

WVR_SetProjectedPassthroughPose(ref WVR_Pose_t pose)
//Use this API to set the pose of the Projected Passthrough
//Parameter
//ref WVR_Pose_t pose: A WVR_Pose_t struct which is for setting the position and rotation of the Projected Passthrough. Values should be of OpenGL coordinate system convention.
//Return value
//WVR_Result.WVR_SUCCESS: Pose value is set successfully.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Projected Passthrough is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Projected Passthrough is not supported on this device.

WVR_Result WVR_SetProjectedPassthroughMesh(float[] vertexBuffer, uint vertextCount, uint[] indices, uint indexCount)
//Use this API to set the mesh data of the Projected Passthrough
//Parameter
//float[] vertexBuffer: A float array that contains the vertices of you mesh. Vertex coordinates should be of OpenGL coordinate system convention.
//uint vertextCount: Number of vertex data. Should be the length of the vertexBuffer parameter.
//uint[] indices: A uint array for specifying the indices of the triangles of the mesh. Order should be of OpenGL convention.
//uint indexCount: Number of indices. Should be the length of the indices parameter.
//Return value
//WVR_Result.WVR_SUCCESS: Mesh values are set successfully.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Projected Passthrough is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Projected Passthrough is not supported on this device.

WVR_Result WVR_SetProjectedPassthroughAlpha(float alpha)
//Use this API to set the transparency (alpha value) of the Projected Passthrough.
//Parameters
//float alpha: The target alpha value. Should be a float in range [0f, 1f].
//Return value
//WVR_Result.WVR_SUCCESS: Alpha value is set successfully.
//WVR_Result.WVR_Error_RuntimeVersionNotSupport: Projected Passthrough is not supported by the runtime on the device.
//WVR_Result.WVR_Error_FeatureNotSupport: Projected Passthrough is not supported on this device.

//Projected Passthrough Sample Code
using Wave.Native;

void ShowProjectedPassthrough()
{
    //Set Pose of Projected Passthrough
    WVR_Pose_t pose = new WVR_Pose_t();
    pose.position.v0 = 0.0f;
    pose.position.v1 = 0.0f;
    pose.position.v2 = -2.0f;
    pose.rotation.w = 1.0f;
    pose.rotation.x = 0.0f;
    pose.rotation.y = 0.0f;
    pose.rotation.z = 0.0f;
    Interop.WVR_SetProjectedPassthroughPose(ref pose);

    //Set Mesh of Projected Passthrough
    float size = 0.25f;
    float[] vertices = { -size, -size, 0.0f,
                          size, -size, 0.0f,
                          size,  size, 0.0f,
                         -size,  size, 0.0f };
    uint[] indices = { 0, 1, 2, 0, 2, 3 };
    Interop.WVR_SetProjectedPassthroughMesh(vertices, (uint)vertices.Length, indices, (uint)indices.Length);

    Interop.WVR_ShowProjectedPassthrough(true); //Show Projected Passthrough
}

void SetProjectedPassthroughAlphaValue()
{
    Interop.WVR_SetProjectedPassthroughAlpha(0.5f); //Set Alpha of Projected Passthrough to 0.5f
}

void HideProjectedPassthrough()
{
    Interop.WVR_ShowProjectedPassthrough(false); //Hide Projected Passthrough
}

Quality

The Passthrough enables the environment camera on HMD and this costs performance. VIVE Wave™ provides quality and performance control APIs for all passthrough feature, overlay, underlay and projected underlay. These APIs can help developers to make the balance between the quality and performance.

Set Passthrough Image Quality

WVR_Result Interop.WVR_SetProjectedPassthroughImageQuality(WVR_PassthroughImageQuality quality)

• WVR_PassthroughImageQuality.DefaultMode

  After your app is launched, this is the default setting. The passthrough image will not be very sharp. And the performance is not heavy.

• WVR_PassthroughImageQuality.PerformanceMode

  Setting to this mode, the passthrough image will be softer, and the system cost will be lower.

• WVR_PassthroughImageQuality.QualityMode

  Setting to this mode, the passthrough image will be sharper, and the system cost will be higher.

Set Frame Rate

In practice, a lower traget frame rate can help the final FPS be more stable when you show passthrough underlay, and can help the actions in the VR/MR be more smooth.

This API can let you set the target frame rate.

WVR_Result Interop.WVR_SetFrameRate(int frameRate)

You should set the Frame Rate by one of available frame rate. See ‘Get Available Frame Rate’_

Get Frame Rate

This API can let you get the current target frame rate.

WVR_Result Interop.WVR_GetFrameRate(ref uint frameRate)

Get Available Frame Rates

This API can let you get the possible target frame rate supported by the system. Use the value in the return array to set the frame rate.

WVR_Result Interop.WVR_GetAvailableFrameRates(out uint[] frameRates)

Alignment

The passthorugh image is taken by single environment camera. In the passthrough, the real objects which is closed to the camera will not have a correct depth. If visual object rendered in the a correct depth, a missalignment will be happen between visual object and passthrough image.

VIVE Wave™ can align your visual controller with the real controller in the passthrough image. This can help immersive better if no multiple hand or controller is shown in your vision.

However, with this effect you will see the passthrough image distorted around your controller. The visual content around your hand will be not distorted but the background image be distored. You can choose another mode if you need the background be smooth.

In practice, if player see more visual object in your design, it is better to set to the View mode. if player focus the visual object holded on hand, it is better to set to the Scale mode. You can switch it when necessary.

Set Passthrough Image Focus

WVR_Result Interop.WVR_SetProjectedPassthroughImageFocus(WVR_PassthroughImageFocus focus)

• WVR_PassthroughImageFocus.View

  No controller alignment effect.

• WVR_PassthroughImageFocus.Scale

  This is the default mode. Your visual controller will be aligned with the real controller in passthrough image.

Resources

If you have installed the VIVE Wave XR Plugin - Essence package and imported its Samples in the Unity Package Manager, you can find the Passthrough Sample at Assets/Sample/VIVE Wave XR Plugin - Essence/<package version>/Essence/PassThrough.