Passthrough Underlay

Introduction

VIVE Wave™ provides a Passthrough Underlay feature which allows users to see the real world environment while wearing the HMD. Depending on HMD, the passthrough image may be gray scale or colored. Compared to Passthrough Overlay, Passthrough Underlay will not cover virtual objects but Passthrough Overlay will.

The Passthrough Underlay feature will blend the real world image with VR content’s image. In the VR content image, only the pixels with specific combination of alpha and color values will be processed in the blending. The pixels in the region where the passthrough should show up need to have the special color and alpha values.

VIVE Wave™ also provide two materials to help render the color and alpha value for Passthrough Underlay. Due to a limitation with materials in Unreal , two materials are required for writing the require color and alpha values separately. Therefore, we will render the same object twice, once with each material, to prepare the passthrough underlay region. See How to Use for the usage.

See also: Passthrough Quality, Performance, and Aligment

APIs

../_images/ShowPassthroughUnderlay.png

To show or hide the Passthrough Underlay. Warning: The Passthrough Underlay will not work if Passthrough Overlay is active.

  • arguments
    Show
    True means to show Passthrough Underlay, False means to hide Passthrough Underlay.
  • return
    True means that Passthrough Underlay is enabled sucessfully, while False means that is not.

See more at WVR_ShowPassthroughUnderlay.

How to use

There are two methods to choose. The method 1 is more flexible and generic but the loading will be much heavier. The method 2 will get better performance but get more limitations.

Method 1

  1. We will use a mesh to wrap the player pawn and the other visible content. This mesh will be used like a skybox to set the special color of background. In this step, we will add two mesh component into level Outliner. They are named CleanColor and CleanAlpha in this case.

  2. Set both of their meshes to SM_SkySphere which is an engine asset. Note that the size of skysphere is 8192 * 8192 * 8192, and make sure this component’s size is big enough to wrap all visible content and player area of VR Pawn.

    ../_images/UnrealPassthroughUnderlayMesh.jpg

  3. Set the material, UnderlayCleanColor, to CleanColor component. You can find the material in our plugin content.

    ../_images/UnrealPassthroughUnderlayColorClear.jpg

    If the CleanColor component is not present, your background color might blend with passthrough image. The CleanColor will make the passthrough region black.

  4. Set the material, UnderlayAlpha, to CleanAlpha component. You can find the material in our plugin content.

    ../_images/UnrealPassthroughUnderlayAlpha.jpg

  5. As we don’t want to interact with these components, we will disable the collision.

    ../_images/UnrealPassthroughUnderlayMeshNoCollision.jpg

  6. Now you can enable Passthrough Underlay to see the result.

    ../_images/PassthroughUnderLay_Show.png

In some cases, passthrough underlay is disabled in the beginning, you may set both meshes to invisible in the editor, and use your own skybox or environment. You can turn both of them visible when Passthrough Underlay is turned on.

../_images/UnrealPassthroughUnderlayMeshControl.jpg

Note:

If there are no CleanColor and CleanAlpha component, you will not have any visual hint about the Passthrough Underlay is running in the background. This will cause a lot of performance waste. In our practice, it is possible to forget turn it off when level change. You could use the BP event, End Play, to help you turn it off.

Method 2

If you want to get better performance and don’t need to keep the skysphere anymore.

Remove your Sky Sphere if you want to show Passthrough Underlay in the background of your map.

../_images/PassthroughUnderLay_SkySphere.png

Note

There are some project settings you need to follow to use Passthrough Underlay: In UE4.27, disable Mobile Multi-View. In UE5.0, set MSAA Sample Count to No MSAA if Mobile Multi-View is enabled.