Advanced Usage

Vive Hand Tracking Unreal plugin also provides several utility blueprints for some common functions.

Using Gesture As Events

HandStateChecker Component is provided to use hand gestures as triggers/events.

It’s designed as a state machine with at most 3 states:

../_images/state1.png
  • Reset state (state 0): the default state. If reset condition is met, all other states switches to reset state, useful to stop action if one or both hands are missing.
  • Prepare state (state 1): stays at this state only if prepare confition is met.
  • Trigger state (state 2): stays at this state only if trigger confition is met while in state 1. Only allow transit from state 1 by default, unless Can Skip Prepare checkbox is on.

Tip

To make the prepare and trigger state transit more robust and avoid state jumps, blueprint supports enter state until matches for several frames, and return to reset state after missing for several frames.

Condition of each hand can be multi-selected from gestures, including pre-defined gestures, pinch and custom gestures. Condition is met if hand is currently one of the selected gesture. Condition of both hands must be met to enter the state.

Note

Due to flag bits limitation, HandStateChecker blueprint supports at most 10 single hand custom gestures and 5 dual hand custom gestures. For custom gestures that exceeds the limit, see Add Custom Gestures in Skeleton Mode section below for manually checking gesture state.

HandStateChecker blueprint is used in all Sample levels. For example, in the teleport action, reset state does nothing, prepare state (right hand five gesture) triggers destination selection, trigger state (right hand fist gesture) confirms telport. Left hand condition is set to no hand or unknown gesture to avoid conflict with other samples.

To bind or unbind OnStateChanged delegate, the event function must have an integer parameter and returns void. It can be either blueprint function or C++ function decorated with UFUNCTION(BlueprintCallable). For C++ Developers, please refer to Unreal document for binding functions.

../_images/bind.png

Add Custom Gestures in Skeleton Mode

Besides pre-defined gestures, user can define custom gestures based on the VR app content. Custom gestures are defined regarding finger close/open states, with optional distance requirement between finger tips.

  • Thumb state is either close or open.
  • Other finger states are close, open or relax (or half-open).
  • Finger states are multi-selectable.
  • Restrictions can be added to make sure distance between two finger tips are close or far.
  • Custom gesture assets can be reused in all levels.

Please refer to CustomGestureMap sample level for example usages.

Add Custom Gesture Definition

Custom gesture definitions are saved as uassets in Unreal project. They can be created using Add New -> ViveHandTracking -> DualHandCustomGesture/SingleHandCustomGesture button in content browser:

../_images/custom_create1.png

There are two types of custom gestures supported in SDK:

  • Single hand custom gesture: The custom gesture is recognized for each hand separately. This is similar as pre-defined gesture.

    ../_images/custom_rock1.png

  • Dual hand custom gesture: The custom gesture requires both hands. Finger tip restrictions can be defined for both hands as well as cross-hand.

    ../_images/custom_vive1.png

The above custom gesture assets can be found in Plugins/ViveHandTracking/Content/Blueprints/CustomGesture folder and are used in CustomGestureMap level.

Using Custom Gesture In Map

To use custom gesture in the map:

  1. Make sure HandTrackingProvider actor is setup correctly in the map, and hand tracking mode is set to Skeleton.
  2. Add CustomGestureProvider actor to the map.
  3. Register single hand and dual hand custom gestures in CustomGestureProvider details.
../_images/custom_provider1.png

Note

Custom gestures must be added to CustomGestureProvider, or it cannot be detected in the map.

For the first 10 single hand custom gestures and 5 dual hand custom gestures, it’s recommended to use HandStateChecker blueprint to check gesture status and trigger UI actions.

../_images/custom_state1.png

It’s also possible to check gesture status manually using custom gesture blueprints. For single hand custom gesture, use IsLeftMatch and IsRightMatch to check if gesture is triggered for left/right hand. For dual hand custom gesture, use IsMatch to check if gesture is triggered.

Using Hand with UMG UI

WidgetInteractionActor is provided for UMG UI interaction. It contains Widget Interaction Component and Hand State Checker Component. Widget Interaction Component allows you to simulate laser pointer that can interact with widget. You can set what gesture is used to trigger mouse click in Hand State Checker Component, default is pinch.

../_images/interaction.png

To use hand with UMG UI, follow the steps below:

  1. Create your UMG UI following Unreal Document.
  2. Add Widget Interaction Actor and your widget to the level for UI interaction.

You can also reference sample WidgetMap level for example usage.

Tip

Change Prepare Condition of LeftHandStateChecker and RightHandStateChecker if you want different gestures for click action.

Switch Hand Display

In sample levels, there is a utility blueprint to switch hand display among sphere-links and hand models. The blueprint is used to show only one kind of hand display each time, while display can be changed using gestures. Make sure to check Is Model for 3d model renderer blueprints, so these will be ignored if current mode is not skeleton.

Note

Make sure there is at least one display class that supports non-skeleton mode.

../_images/switch_display1.png

HandDisplaySwitch blueprint should be used together with HandStateChecker blueprint in the same actor, in order for hand switch event to trigger. HandDisplaySwitch.OnStateChanged must bind to HandStateChecker.OnStateChanged delegate. Hand display is switched to next class whenever prepare state is entered. Trigger condition is not used and should set condition of both hands to Nothing.

Hint

In sample levels, we switch hand display for both hand together by pointing to a list of hand classes. You can switch display for each hand separately by using two HandDisplaySwitch blueprints, each pointing to a list of left/right hand classes. Make sure to add two HandStateChecker as well.