Vive Hand Tracking Unity plugin also provides several utility blueprints for some common functions.
Using Gesture As Events¶
HandStateChecker blueprint is provided to use hand gestures as triggers/events.
Make sure to select
Hand Tracking Provider and
Custom Gesture Provider to get VivehHandTracking and CustomGesture Component.
Custom Gesture Provider can be null if current map does not use custom gestures.
It’s designed as a state machine with at most 3 states:
Reset state(state 0): the default state. If reset condition is met, all other states switches to reset state, useful to stop action if one or both hands are missing.
Prepare state(state 1): stays at this state only if prepare confition is met.
Trigger state(state 2): stays at this state only if trigger confition is met while in state 1. Only allow transit from state 1 by default, unless
Can Skip Preparecheckbox is on.
To make the prepare and trigger state transit more robust and avoid state jumps, blueprint supports enter state until matches for several frames, and return to reset state after missing for several frames.
Condition of each hand can be multi-selected from gestures, including pre-defined gestures, pinch and custom gestures. Condition is met if hand is currently one of the selected gesture. Condition of both hands must be met to enter the state.
Due to flag bits limitation,
HandStateChecker blueprint supports at most 10 single hand custom gestures and 5 dual hand custom gestures.
For custom gestures that exceeds the limit, see Add Custom Gestures in Skeleton Mode section below for manually checking gesture state.
HandStateChecker blueprint is used in all Sample levels.
For example, in the teleport action, reset state does nothing, prepare state (right hand five gesture) triggers destination selection, trigger state (right hand fist gesture) confirms telport.
Left hand condition is set to no hand or unknown gesture to avoid conflict with other samples.
To bind or unbind
OnStateChanged delegate, the event function must have an integer parameter and returns void.
It can be either blueprint function or C++ function decorated with UFUNCTION(BlueprintCallable).
For C++ Developers, please refer to Unreal document for binding functions.
Add Custom Gestures in Skeleton Mode¶
Besides pre-defined gestures, user can define custom gestures based on the VR app content. Custom gestures are defined regarding finger close/open states, with optional distance requirement between finger tips.
- Thumb state is either close or open.
- Other finger states are close, open or relax (or half-open).
- Finger states are multi-selectable.
- Restrictions can be added to make sure distance between two finger tips are close or far.
- Custom gesture assets can be reused in all levels.
Please refer to
CustomGestureMap sample level for example usages.
Add Custom Gesture Definition¶
Custom gesture definitions are saved as uassets in Unreal project.
They can be created using
Add New button in content browser:
There are two types of custom gestures supported in SDK:
Single hand custom gesture: The custom gesture is recognized for each hand separately. This is similar as pre-defined gesture.
Dual hand custom gesture: The custom gesture requires both hands. Finger tip restrictions can be defined for both hands as well as cross-hand.
The above custom gesture assets can be found in Plugins/ViveHandTracking/Content/Blueprints/CustomGesture folder and are used in
Using Custom Gesture In Map¶
To use custom gesture in the map:
- Make sure
HandTrackingProvideractor is setup correctly in the map, and hand tracking mode is set to Skeleton.
CustomGestureProvideractor to the map.
- Register single hand and dual hand custom gestures in
Custom gestures must be added to
CustomGestureProvider, or it cannot be detected in the map.
For the first 10 single hand custom gestures and 5 dual hand custom gestures, it’s recommended to use
HandStateChecker blueprint to check gesture status and trigger UI actions.
It’s also possible to check gesture status manually using custom gesture blueprints.
For single hand custom gesture, use
IsRightMatch to check if gesture is triggered for left/right hand.
For dual hand custom gesture, use
IsMatch to check if gesture is triggered.
Using Hand with UMG UI¶
Widget Interaction Actor is provided for UMG UI interaction. It contains Widget Interaction Component and Hand State Checker Component. Widget Interaction Component allows you to simulate laser pointer that can interact with widget. You can set what gesture is used to trigger mouse click in Hand State Checker Component, default is pinch.
To use hand with UMG UI, follow the steps below:
Create your UMG UI following Unreal Document.
You might need to use game instance to pass value from Widget to blueprints, see this question.
WidgetMapsample level to work, you need to set game instance to the one provided by Hand Tracking SDK.
Add Widget Interaction Actor and your widget to the level for UI interaction.
You can also reference sample
WidgetMap level for example usage.
Prepare Condition of
RightHandStateChecker if you want different gestures for click action.
Switch Hand Display¶
In sample levels, there is a utility blueprint to switch hand display among sphere-links and hand models. The blueprint is used to show only one kind of hand display each time, while display can be changed using gestures. Make sure to check Is Model for 3d model renderer blueprints, so these will be ignored if current mode is not skeleton.
Make sure there is at least one display class that supports non-skeleton mode.
HandDisplaySwitch blueprint should be used together with
HandStateChecker blueprint in the same actor,
in order for hand switch event to trigger.
HandDisplaySwitch.OnStateChanged must bind to HandStateChecker.OnStateChanged delegate.
Hand display is switched to next class whenever prepare state is entered.
Trigger condition is not used and should set condition of both hands to Nothing.
In sample levels, we switch hand display for both hand together by pointing to a list of hand classes.
You can switch display for each hand separately by using two
HandDisplaySwitch blueprints, each pointing to a list of left/right hand classes.
Make sure to add two
HandStateChecker as well.