Wave MR Scene And Anchor Native Development Guide

Overview

To develop MR applications using Wave SDK, please refer to following documents:

Contents 1. Permission & Manifest 2. Show passthrouh 3. Anchor 4. Scene Plane & Plane Mesh 5. Collider Mesh & Visual Mesh 6. Others

Note

• You can download the Wave SDK from here. And check Native MR sample in SDK/samples/wvr_native_anchor.
• To build the project, please reference Native SDK Getting Started.

1. Permission & Manifest

In order to use scene perception API, developer should add feature in AndroidManifest.xml. If developer want to use scene mesh API, permission is also needed.

//andoirManifest.xml
//scene perception
<uses-feature android:name="wave.feature.sceneperception" android:required="true"/>
//scene mesh
<uses-permission android:name="wave.permission.GET_SCENE_MESH"/>

//MainActivity.java
int hasPermission=checkSelfPermission(SCENE_PERMISSION);
if(hasPermission!=PackageManager.PERMISSION_GRANTED){
    Log.i(TAG, "Permission not granted try to aquire: "+SCENE_PERMISSION);
    requestPermissions(new String[]{SCENE_PERMISSION},REQUEST_CODE);
}
@Override
public void onRequestPermissionsResult(int requestCode,  String[] permissions,  int[] grantResults) {
    if (requestCode == REQUEST_CODE && grantResults != null && grantResults.length > 0) {
        if (grantResults[0] == PackageManager.PERMISSION_GRANTED) {
            mHasPermission = true;
            Log.i(TAG, "Permission granted");
        }
        else{
            Log.i(TAG, "Permission not granted");
        }
    }
}

2. Show passthrouh

Show Passthrough underlay by calling WVR_ShowPassthroughUnderlay() after WVR_Init().

static bool showUnderlay = true;
WVR_ShowPassthroughUnderlay(showUnderlay);

3. Anchor

3-1. Init

In order to use anchor API, developer should call WVR_StartScene() before calling any anchor API.

WVR_Result result = WVR_StartScene();
if (result != WVR_Success)
{
    LOGE("WVR_StartScene init fail %d", result);
    return false;
}

3-2. Get existed anchors - WVR_EnumerateSpatialAnchors()

Application should call WVR_EnumerateSpatialAnchors() twice to get the number of existed anchors and WVR_SpatialAnchor object. Application should repeat this process every time when app resume, adding anchor and deleting anchor.

//call this function when after adding anchor/deleting anchor/resume
void getExistedAnchor()
{
    // WVR API get anchor num and allocate memory to get data
    uint32_t anchorCount;
    WVR_SpatialAnchor *mAnchor;
    WVR_EnumerateSpatialAnchors(0, &anchorCount, nullptr);
// clear memory
if (mAnchor != nullptr)
    free(mAnchor);
    // assign memory space for anchor
    mAnchor = (WVR_SpatialAnchor *)malloc(sizeof(WVR_SpatialAnchor) * anchorCount);
    LOGI("get anchor anchorCount=%u", anchorCount);
    WVR_EnumerateSpatialAnchors(anchorCount, &anchorCount, mAnchor);
}

3-3. Get anchor data - WVR_GetSpatialAnchorState()

Application could send WVR_SpatialAnchor object to WVR_GetSpatialAnchorState() to update anchor status(Which include pose, name and tracking state). If the anchor’s trackingState is not equal to WVR_SpatialAnchorTrackingState_Tracking, developer should not use this anchor pose to update model pose.

WVR_SpatialAnchorState anchorState;
WVR_GetSpatialAnchorState(mAnchor[a], WVR_PoseOriginModel_OriginOnHead, &anchorState);

3-4. Add an anchor - WVR_CreateSpatialAnchor()

Application should check existed anchors after calling WVR_CreateSpatialAnchor() to add an anchor. If developer want to classify different render model with different anchor, developer can add extra information in anchor’s anchorName while creating anchor.

//in this example anchor name would be "c=scale.x=scale.y=scale.z"
std::string planeName("c");
std::string dash("=");

WVR_SpatialAnchor anchor;
WVR_SpatialAnchorCreateInfo info;

info.originModel = WVR_PoseOriginModel_OriginOnHead;
info.pose = convertToWVRPose(pose);
planeName += dash;
planeName += std::to_string(round(scale.x * 10000) / 10000);
planeName += dash;
planeName += std::to_string(round(scale.y * 10000) / 10000);
planeName += dash;
planeName += std::to_string(round(scale.z * 10000) / 10000);
planeName += dash;
std::copy(planeName.begin(), planeName.end(), info.anchorName.name);

WVR_Result res = WVR_CreateSpatialAnchor(&info, &anchor);
if (res != WVR_Success)
{
    LOGE("add anchor fail (%d)", res);
}
else
{
    LOGI("add anchor success (%d)", res);
}

//enumerate anchor again
getExistedAnchor()

3-5. Delete an anchor - WVR_DestroySpatialAnchor()

Application should check existed anchors after calling WVR_DestroySpatialAnchor() to delete an anchor.

WVR_DestroySpatialAnchor(*(mAnchor + a));

//enumerate anchor again
getExistedAnchor()

3-6. Render anchor object

In order to render a model with anchor, developer should use anchor’s pose as model pose matrix and sent it to OpenGL shader program.

//update anchor pose
WVR_GetSpatialAnchorState(mAnchor[cm->cubeVector[a]->anchorIndex], WVR_PoseOriginModel_OriginOnHead, &anchorState);
if (anchorState.trackingState == WVR_SpatialAnchorTrackingState_Tracking)
{
    allAnchorCanTrack = true;
    pose = convertToMatrix4(anchorState.pose);
    cm->cubeVector[a]->mPose = pose;
    cm->cubeVector[a]->anchorLost = false;
}
else
{
    cm->cubeVector[a]->anchorLost = true;
    LOGI("Anchor cubeVector(%d) lost tracking", a);
}

//render
Matrix4 matrix = projection * eye * view * cubeVector[a]->mPose;
glUniformMatrix4fv(mMatrixLocation, 1, false, matrix.get());
glUniform3f(mScale, cubeVector[a]->mScale.x, cubeVector[a]->mScale.y, cubeVector[a]->mScale.z);

mVAO->bindVAO();
glActiveTexture(GL_TEXTURE0);
mTexture->bindTexture();
glDrawElements(GL_TRIANGLES, singleIndexArray.size(), GL_UNSIGNED_INT, 0);
mTexture->unbindTexture();
mVAO->unbindVAO();

4. Scene Plane & Plane Mesh

4-1. Init

In order to use plane API, developer should call WVR_StartScene() and WVR_StartScenePerception() before calling any plane API.

// Loading the WVR scene
WVR_Result result = WVR_StartScene();
if (result != WVR_Success)
{
    LOGE("WVR_StartScene init fail %d", result);
    return false;
}

// start perception before get scene plane (need permission)
// can call WVR_StopScenePerception when you won't need to update data
// WVR_StopScenePerception will be automatic called when app is closed
WVR_Result startPerceptionResult_2d = WVR_StartScenePerception(WVR_ScenePerceptionTarget_2dPlane);
if (startPerceptionResult_2d == WVR_Error_PermissionNotGranted)
{
    LOGE("Permission not granted. Need permission to start scene perception");
    perceptionPermissionGrant = false;
}
else{
    perceptionPermissionGrant = true;
}

if (WVR_Success != startPerceptionResult_2d)
    LOGE("WVR_StartScenePerception init fail");

4-2. Get Plane data - WVR_GetScenePlanes()

Application should call WVR_GetScenePlanes() twice in every frame to get the number of planes and planes data.

// WVR API get scene num and allocate memory to get data
WVR_GetScenePlanes(nullptr, 0, &planeCount, WVR_PoseOriginModel_OriginOnHead, nullptr);
mPlane = (WVR_ScenePlane *)malloc(sizeof(WVR_ScenePlane) * planeCount);
WVR_GetScenePlanes(nullptr, planeCount, &planeCount, WVR_PoseOriginModel_OriginOnHead, mPlane);
//get planes data
planesVector.push_back(Plane{convertToMatrix4(mPlane[a].pose), Vector3(mPlane[a].extent.width, mPlane[a].extent.height),1});

4-3. Render Plane object

Application can use pose and extent to render rectangles.

glUniform4f(mColor, 0.0f, 0.8f, 0.0f, 1.0f);
matrix.push_back(projection * eye * view * planesVector[a].mPose);
glUniformMatrix4fv(mMatrixLocation, 1, false, matrix[a].get());
glUniform3f(mScale, planesVector[a].mScale.x, planesVector[a].mScale.y, planesVector[a].mScale.z);//scale is define by plane extent
mVAO->bindVAO();
glDrawElements(GL_TRIANGLES, singleIndexArray.size(), GL_UNSIGNED_INT, 0);
mVAO->unbindVAO();

4-4. Get plane mesh data - WVR_GetSceneMeshBuffer()

Plane mesh vertex and element index(index value: 0~3) can be got via sending meshBufferId to WVR_GetSceneMeshBuffer.

WVR_SceneMeshBuffer buffer0 = {0, 0, nullptr, 0, 0, nullptr};

// get sceneMesh size
if (WVR_GetSceneMeshBuffer(mPlane[planes[b].scenePlaneIndex].meshBufferId, WVR_PoseOriginModel_OriginOnHead, &buffer0) != WVR_Success)
{
    LOGE("Fail to get plane sceneMesh size");
    sceneMesh2D->clearAllData();
}
else
    LOGI("Get sceneMesh index size=%d", buffer0.indexCountOutput);

std::vector<WVR_Vector3f> vertexBuffer(buffer0.vertexCountOutput);
std::vector<uint32_t> indexBuffer(buffer0.indexCountOutput);
WVR_SceneMeshBuffer buffer1 = {buffer0.vertexCountOutput, buffer0.vertexCountOutput, vertexBuffer.data(), buffer0.indexCountOutput, buffer0.indexCountOutput, indexBuffer.data()};

// get sceneMesh data
if (WVR_GetSceneMeshBuffer(mPlane[planes[b].scenePlaneIndex].meshBufferId, WVR_PoseOriginModel_OriginOnHead, &buffer1) != WVR_Success)
{
    LOGE("Fail to get sceneMesh data");
    sceneMesh2D->clearAllData();
}
sceneMesh2D->mMeshIndex.push_back(indexBuffer);
sceneMesh2D->mMeshVertex.push_back(vertexBuffer);

4-5. Render plane mesh data

Developer can simply render plane meshes by vertex and index with glDrawElement().

//populate vertex buffer
glBufferData(GL_ARRAY_BUFFER, sizeof(float) * mMeshVertex[a].size()*3, mMeshVertex[a].data() , GL_STATIC_DRAW);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 3, GL_FLOAT, false, stride, 0);

// populate the index buffer
mVAO->bindElementArrayBuffer();
glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(uint32_t) * mMeshIndex[a].size(), mMeshIndex[a].data(), GL_STATIC_DRAW);

Matrix4 matrix = projection * eye * view;
glUniformMatrix4fv(mMatrixLocation, 1, false, matrix.get());
mVAO->bindVAO();
glDrawElements(GL_TRIANGLES, index[a].size(), GL_UNSIGNED_INT, 0);
mVAO->unbindVAO();

5. Collider Mesh & Visual Mesh

5-1. Init

In order to use mesh API, developer should call WVR_StartScene() and WVR_StartScenePerception() before calling any mesh API.

// Loading the WVR scene
WVR_Result result = WVR_StartScene();
if (result != WVR_Success)
{
    LOGE("WVR_StartScene init fail %d", result);
    return false;
}

// start perception before get scene mesh (need permission)
// can call WVR_StopScenePerception when you won't need to update data
// WVR_StopScenePerception will be automatic called when app is closed
WVR_Result startPerceptionResult_3d = WVR_StartScenePerception(WVR_ScenePerceptionTarget_SceneMesh);
if (startPerceptionResult_3d == WVR_Error_PermissionNotGranted)
{
    LOGE("Permission not granted. Need permission to start scene perception");
    perceptionPermissionGrant = false;
}
else{
    perceptionPermissionGrant = true;
}

if (WVR_Success != startPerceptionResult_3d)
    LOGE("WVR_StartScenePerception init fail");

5-2. Check scene perception state - WVR_GetScenePerceptionState()

Application should check if HMD device can recognize corresponding mesh. If state is not equal to WVR_ScenePerceptionState_Complete, application should not use WVR_GetSceneMeshBuffer() to get mesh data.

WVR_GetScenePerceptionState(WVR_ScenePerceptionTarget_SceneMesh,&spstate);
if(spstate==WVR_ScenePerceptionState_Completed){
    // get mesh data
    ...
}
else
{
    LOGI("Scene perception not complete");
    mSceneMeshBufferVisual_index.clear();
    mSceneMeshBufferVisual_vertex.clear();
}

5-3. Get existed collider/visual meshBufferId - WVR_GetSceneMeshes()

Application should call WVR_GetSceneMeshes() to get newest existed meshBufferId every time before calling WVR_GetSceneMeshBuffer(). If any scene mesh vertex is been changed, all meshes meshBufferId will be changed.

//ColliderMesh
uint32_t  count_mesh;
//get number of sceneMeshes
if(WVR_GetSceneMeshes(WVR_SceneMeshType_ColliderMesh, 0, &count_mesh, nullptr)!=WVR_Success)
    LOGE("Fail to get sceneMeshes count");
else
    LOGI("SceneMesh ID size=%d", count_mesh);

//Get sceneMeshes ID
std::vector<WVR_SceneMesh> meshes(count_mesh);
if(WVR_GetSceneMeshes(WVR_SceneMeshType_ColliderMesh, count_mesh, &count_mesh, meshes.data())!=WVR_Success)
    LOGE("Fail to get sceneMeshes ID");

//visualMesh
if(WVR_GetSceneMeshes(WVR_SceneMeshType_VisualMesh, 0, &count_mesh, nullptr)!=WVR_Success)
/*same as collider mesh...*/

5-4. Get collider/visual mesh data - WVR_GetSceneMeshBuffer ()

Application could use meshBufferId and WVR_GetSceneMeshBuffer() to get mesh vertex and element index. Owing to performance concern, we strongly recommend only use WVR_GetSceneMeshBuffer() when scene mesh vertex is changed.

//only update data when any vertex had changed
//If any mesh data had changed, all buffer id get from WVR_GetSceneMeshes() would be change
if(lastVisualMeshID!=meshes[0].meshBufferId){
    LOGI("visual sceneMesh data should be updated");
    WVR_SceneMeshBuffer buffer0={0, 0, nullptr, 0, 0, nullptr};
    //get sceneMesh size
    if(WVR_GetSceneMeshBuffer(meshes[a].meshBufferId, WVR_PoseOriginModel_OriginOnHead, &buffer0)!=WVR_Success){
        LOGE("Fail to get visual sceneMesh size");
        mSceneMeshBufferVisual_index.clear();
        mSceneMeshBufferVisual_vertex.clear();
        lastVisualMeshID=-1;
        return;
    }
    else
        LOGI("Get sceneMesh index size=%d", buffer0.indexCountOutput);

    std::vector<WVR_Vector3f> vertexBuffer(buffer0.vertexCountOutput);
    std::vector<uint32_t> indexBuffer(buffer0.indexCountOutput);
    WVR_SceneMeshBuffer buffer1 = {buffer0.vertexCountOutput, buffer0.vertexCountOutput,vertexBuffer.data(), buffer0.indexCountOutput, buffer0.indexCountOutput, indexBuffer.data()};

    //get sceneMesh data
    if(WVR_GetSceneMeshBuffer(meshes[a].meshBufferId, WVR_PoseOriginModel_OriginOnHead, &buffer1)!=WVR_Success){
        LOGE("Fail to get sceneMesh data");
        mSceneMeshBufferVisual_index.clear();
        mSceneMeshBufferVisual_vertex.clear();
        lastVisualMeshID=-1;
        return;
    }
    mSceneMeshBufferVisual_index.push_back(indexBuffer);
    mSceneMeshBufferVisual_vertex.push_back(vertexBuffer);
}

5-5. Render collider/visual mesh

Developer can render 3D mesh simply by vertex and index with glDrawElement(). In addition, Developer can also modify vertex and index to create better visual effect. Owing to the large vertex and element data, we strongly recommend not to push vertex to GPU every frame. Only push vertex to GPU when the vertex data had change.

//Modify color
Vector3 selectColor(int num){
    Vector3 color;
    switch (num) {
    case 1:
        color.x = 238.0 / 255;
        color.y = 132.0 / 255;
        color.z = 117.0 / 255;
        break;
    /*skip ...*/
    case 10:
        color.x = 157.0 / 255;
        color.y = 28.0 / 255;
        color.z = 200.0 / 255;
        break;
    }
    return color;
}
//modify vertex and element for better visual effect
void getOrderMesh(){
    int count;
    Vector3 color;
    mMeshOrder.clear();
    std::vector<float> buffer;
    for (int a = 0; a < mMeshIndex.size(); a++) {
        count = 0;
        buffer.clear();
        for (int b = 0; b < mMeshIndex[a].size(); b = b + 3){
            count = count % 10 + 1;
            color = selectColor(count);
            buffer.push_back(mMeshVertex[a][mMeshIndex[a][b]].v[0]);
            buffer.push_back(mMeshVertex[a][mMeshIndex[a][b]].v[1]);
            buffer.push_back(mMeshVertex[a][mMeshIndex[a][b]].v[2]);
            buffer.push_back(color.x);
            buffer.push_back(color.y);
            buffer.push_back(color.z);
        }
        mMeshOrder.push_back(buffer);
    }
}

// push data to GPU (only when the vertice were changed)
mVAO->bindVAO();
mVAO->bindArrayBuffer();
int stride = 6 * sizeof(float);
GLuint offset = 0;
float *ptr = (float *)meshData->mMeshOrder[a].data();
glBufferData(GL_ARRAY_BUFFER, sizeof(float) * meshData->mMeshOrder[a].size(), ptr, GL_STATIC_DRAW);
glEnableVertexAttribArray(0); // populate vertex
glVertexAttribPointer(0, 3, GL_FLOAT, false, stride, (const void *)offset);
offset += sizeof(float) * 3;
glEnableVertexAttribArray(1); // populate color
glVertexAttribPointer(1, 3, GL_FLOAT, false, stride, (const void *)offset);
mVAO->unbindVAO();
mVAO->unbindArrayBuffer();

// render
meshData->mVertexSize.push_back(meshData->mMeshOrder[a].size() / 6);
LOGI("render %d vertex", meshData->mVertexSize[a]);
glUniformMatrix4fv(mMatrixLocation, 1, false, matrix.get());
mVAO->bindVAO();
glDrawArrays(GL_TRIANGLES, 0, meshData->mVertexSize[a]);
mVAO->unbindVAO();

• Render Visual Mesh

• Render Collider Mesh

6. Others

While developer use last section features, developer should align WVR_PoseOriginModel in all WAVE API. If nothing wrong, develop can have the MR content with correct position as below.