API Reference


GestureFailure StartGestureDetection(GestureOption *option)

Start detection with given option, non-blocking.

option is both in/out parameter. Mode of option may be modified if requirements is not met on the device. The resulting mode is the actual mode used in detection. If option is null, default option will be used (auto backend + best mode).

void StopGestureDetection()

Stop the detection. Blocking call until the pipeline is actually stopped.

int GetGestureResult(const GestureResult **points, int *frameIndex)

Get detection result in world coordinate. Returns at most one left and one right hand. Return value is number of detections in points, at most 2.

You should call this function periodically to get latest results.

Points:(return value) A pointer to an array of GestureResult. The pointer is valid until next call to GetGestureResult() or StopGestureDetection(). The pointer need NOT to be freed.
FrameIndex:(return value) A pointer to frame index (can be null). This can be used to check if results are updated. Set to -1 if detection is not started or stopped due to error.

Example usage:

const GestureResult* points = NULL;
int frameIndex;
int size = GetGestureResult(&points, &frameIndex);
void UseExternalTransform(bool value)

This function should be called before StartGestureDetection() to indicate if caller is providing camera transform or not. Default is false. Call it after StartGestureDetection() has no use.

If set to true, hmd positions are not queried from raw HMD API from OpenVR or WaveVR. The caller is responsible for managing hmd positions using either of the two methods below:

  1. Call SetCameraTransform function every frame to provide HMD transform (recommended)
  2. Apply camera transform to result points after GetGestureResult() function call

This is useful if camera transform is different from raw HMD data, e.g. teleporting. This is also used in Unity & Unreal plugin.

void SetCameraTransform(GestureVector3 position, GestureQuaternion rotation)

Only takes effect if UseExternalTransform(true) is called before StartGestureDetection.

Set hmd transform for use. This function should be called regularly if HMD pose is changing. The transform is used until new one is set. Default transform is idendity.

void SetARCoreSession(ArSession *session, ArFrame *frame)

This function should be called before StartGestureDetection() to set arcore session and frame instance. Call it after StartGestureDetection() has no use.

If session or frame is set to null, arcore will be disabled. Frame is used to determine if frame is updated or not, and also to acquire camera image continuously. Session and Frame will be reset to null by StopGestureDetection().


class GestureVector3

Struct representing 3D points.

float x

x component

float y

y component

float z

z component

class GestureQuaternion

Struct containing rotation information as quaternion.

float x

x component

float y

y component

float z

z component

float w

w component

class GestureOption

Struct used as input/output parameter for StartGestureDetection().

GestureBackend backend = GestureBackend::GestureBackendAuto

Backend of detection.


GestureBackend is deprecated and will be removed in future release.

GestureMode mode = GestureMode::GestureModeSkeleton

Mode of detection.


GestureMode is deprecated, skeleton mode will be the only supported mode in future release. If you want to use other modes, use GestureResult::position.

int maxFPS = -1

Limit max fps of raw detection, only used when value in range [15, 90]. Default is no limit.


This has negative impact on latency. Only use it when VR application fps slows down due to hand tracking.

class GestureResult

Struct containing detection result for one hand.

bool isLeft

Returns if this hand is left/right.

GestureVector3 position

Returns position of palm center, use this if only need hand position instead of 21 joints.

GestureVector3 joints[21]

Returns position of the hand joints. Meaning of this field is different based on actual GestureMode.

+x is right, +y is up, +z is front. Unit is meter.

GestureQuaternion rotations[21]

Returns rotation of the hand joints. Meaning of this field is different based on actual mode.

Identity rotation (assume hand is five gesture): palm face front and fingers point upward.

GestureType gesture

Returns pre-defined gesture type.

float confidence

Returns confidence of the hand, within [0, 1].

GesturePinchInfo pinch

Returns pinch information.

class GesturePinchInfo

Struct containing information of pinch (thumb & index).

float pinchLevel

Returns pinch (thumb & index) level of the hand, within [0, 1], higher means more possible to pinch. If you only need a boolean value for pinch or not, use isPinching() instead.

bool isPinching() const

Returns if currently pinching or not. If you need a range value within [0, 1], you can use pinchLevel instead.

GestureVector3 pinchStart

Returns start position of the pinch ray.

GestureVector3 pinchDirection

Returns direction of the pinch ray.


enum GestureBackend

Enum for selecting computation backend.

enumerator GestureBackendAuto = 0

Default backend, use GPU on PC and CPU on Android. Recommended.

enumerator GestureBackendCPU = 1

Use CPU, not supported on PC

enumerator GestureBackendGPU = 2

Use GPU, supported on PC/Android

enum GestureMode

Enum for detection mode. Larger mode return more info, but runs more slowly. If a mode is not supported on a device, will fallback to previous supported mode.

See also

See Hand Positions & Rotations (Modes) section for detailed modes description.

enumerator GestureMode2DPoint = 0

Fastest mode, return one 2d point for hand, supported on all devices.

enumerator GestureMode3DPoint = 1

Return one 3d point for hand, supported on all dual camera devices.

enumerator GestureModeSkeleton = 2

Return skeleton (21 points) for hand, supported on all devices.

enum GestureType

Enum for predefined gesture classification.

See also

See Pre-defined gesture classification section for detailed gesture description.

enumerator GestureTypeUnknown = 0

All other gestures not in predefined set.

enumerator GestureTypePoint = 1

Only index finger straight and upward.

enumerator GestureTypeFist = 2

All fingers folded.

enumerator GestureTypeOK = 3

Thumb and index fingers connect into a circle, other fingers straight.

enumerator GestureTypeLike = 4

Only thumb straight and upward.

enumerator GestureTypeFive = 5

All fingers straight.

enumerator GestureTypeVictory = 6

Only index and middle finger straight and upward.

enum GestureFailure

Enum for possible errors in hand detection.

enumerator GestureFailureNone = 0

No error occurs.

enumerator GestureFailureOpenCL = -1

(Only on Windows) OpenCL is not supported on the machine.

See also

See Notes for Windows for possible solutions.

enumerator GestureFailureCamera = -2

Start camera failed. This happens if Vive Hand Tracking SDK failed to get camera frames.

  • Windows: See Camera Setup for possible solutions.
  • Android phone & WaveVR: Make sure camera permission is granted before start detection.
    • Some phones does not support NDK Camera2 API (or return no cameras in API).
    • Most WaveVR devices other than Vive Focus does not support camera API yet.
    • Make sure WaveVR SDK is not included for non-WaveVR project.
enumerator GestureFailureInternal = -10

Internal errors.

enumerator GestureFailureCPUOnPC = -11

CPU backend is not supported on Windows. Please use different backend when you start detection.