Customize 3D Hand Model

The plugin provides prefabs and 3d models to display detected hands as 3d models. It also supports to use your own model for display, if your model matches the assumptions and requirements below.

Note

You can compare your custom model with the model shipped in the plugin to make sure it’s setup correctly.

Requirements of Custom Model

The custom 3d model must be imported into Unreal first to check the following requirements:

  1. Custom model must be rigged and contains Skeletal Mesh and Skeleton.
  2. Custom model must contain all the 21 keypoints defined in the SDK as joints. See Skeleton Mode for keypoints definition.
    • This includes 1 palm joint and 4 joints per finger. All these joints must be defined and rigged, since our script will modify rotation of these joints.
    • If more joints are defined, these extra joints will remain their origin rotation and never changed by script.
  3. The local axis of the finger joints must be same for all 21 keypoints within the model.
    • For all finger joints, there should be an axis that change finger open/close status.
    • For finger root joints of each finger, there should be another axis that swipe the whole finger left/right.
    • All the fingers must using same local axis, including thumb finger, even though thumb finger is facing different direction globally.

Use Custom Model

To use custom 3d model in Hand Tracking SDK, follow the steps below:

  1. Create a Blueprint Actor which is inherited from RigidHand.

  2. Change SkeletalMesh and Material in LeftHand/RightHand details panel.

    ../_images/mesh.png

  3. Check AutoDetectProperties property in ModelRenderer and set search strings.

    ../_images/auto_detect.png
    • Values of Isleft, BoneName and Init Rotation will be calculated automatically at runtime.
    • Search strings are used to identify if mesh belongs to left/right hand.
    • If hand model is not displayed correctly when Play in editor, follow next step to set values manually.

  4. If you auto detection failed:

    1. Uncheck AutoDetectProperties.

    2. Check/Set Hand Rotation in details panel of LeftHand/RightHand. Keep their palm forward and fingers upward.

      ../_images/model_rotation.png

    3. Copy hand rotation and use it to set InitialRotation in ModelRenderer.

      ../_images/model_initial_rot.png

    4. Check Is Left for left hand and uncheck for right hand.

    5. Check/Setup Nodes to contain 21 keypoints in the same order as defined in Skeleton Mode.

      ../_images/unreal_joints.png

  5. Change collision and ux effect settings if necessary.

    ../_images/handsettings.png

    Tip

    If your custom model can create and assign Physics Asset correctly, use it for physics collision detection.