Summary of the use of MRTK3 on PICO4

illustrate

        This document is a learning summary for the GitHub open source project PicoMRTK3 .

1. Use of MRTK3

        MRTK3 is currently a public preview version, and its functions are not yet stable, but there is no problem with basic use. See MRTK3 for details .

1. Add the package body to the Unity project

(1) Use of mixed reality functional tools

        Use the mixed reality function tool to import the MRTK3 package body to the Unity project, the tool download address: mixed reality function tool

1) Tools home page 

        The tool does not need to install EXE. After downloading, place it in a suitable path and create a shortcut to the desktop. After opening, click Start on the home page.

2) Select the Unity project path 

        The selected path is the Unity project path containing Assets. After selecting the path, click Discover Features.

3) Select the MRTK3 inclusion body 

        Click Select All on the right side of the MRTK3 menu to select all MRTK3 related packages.

        Note that after the OpenXR Plugin is imported, the project package reports an error, so you don't need to import the OpenXR Plugin for the time being. The OpenXR Plugin is under the Platform Support menu. 

        After selecting the MRTK3 package body, click Get Features.

4) import 

        Just click Import. Exit the tool after importing.

2. Add examples to Unity project

        MRTK3 is composed of a collection of loosely coupled individual Unity Package Manager packages, which no longer contain example scenes. Instead, maintain a UnityProjects folder at the top level of your Git repository, containing any Unity projects you want to deliver. Currently, this folder contains the MRTKDevTemplate project, which contains all the example scenes and is configured to match the recommended optimal settings.

(1) Enter the Microsoft Mixed Reality Github page

        Address: MixedRealityToolkit-Unity

(2) Select the mrtk3 branch 

        Direct connection entrance: mrtk3

(3) Clone or download the compressed package

        Use git to clone the project or download the compressed package directly. 

(4) Copy resources to Unity project

        After decompressing the downloaded compressed package, open UnityProjects-->MRTKDevTemplate-->Assets, and copy the internal Analyzers, Audio, BuildAssets, Editor, Example Assets, Prefabs, Profiles, Scenes and Scripts folders to the Assets path of the Unity project . Of course, you can also create a folder named MRTKSamples under the Assets path, and copy the above folders under the MRTKSamples folder for easy management.

2. Adapt to the PICO platform

1. PICO SDK settings

        See PICO Documentation Center for details .

(1) Import PICO Unity Integration SDK

1) SDK download

        Go to PICO SDK Download Center to download the latest version of SDK.

2) Unzip the downloaded SDK compressed package

        Create a folder named PICOUnityIntegrationSDK under the same path as the Unity project Assets, copy the downloaded compressed package to this folder and decompress it. The reason why it should be placed in the Unity project is because the SDK path will be bound when the SDK is imported from the Unity Package Manager later. In order not to lose the SDK during version management, it is safest to place the SDK in the Unity project.

        After decompression, you will get a folder containing the package.json file, which is used for subsequent import.

3) Import SDK in Unity Package Manager 

        As shown in the figure, click the "+" sign, and click Add package from disk... again.

4) Select the package.json file and import it 

5) Enter system settings and upgrade XR Interaction Toolkit 

        The Unity version used in this project is 2021.3.20f1c1, so after importing the SDK, a window will automatically pop up to prompt you to set the input system and upgrade the XR Interaction Toolkit. For lower versions of Unity (such as 2019.4.x), you need to go to the Package Manager to upgrade the XR Interaction Toolkit by yourself, and then follow the prompts in the pop-up window to set it.

        Here, click Yes, and then click I Made a Backup, Go Ahead in the subsequent pop-up window! . After that, Unity will restart, and continue with subsequent settings after the restart is complete. 

6) Click Apply 

        After clicking Apply, after Unity compiles, click Close again to close the window.

(2) Project configuration

1) Enable the PICO XR plugin

        In the Project Settings window, click XR Plug-in Management --> Android Settings icon, check PICO. 

2) Other settings of the Android platform

        Including the items in Company Name, Product Name, Version and Other Settings. 

2. MRTK3 adapts to PICO

(1) Clone or download open source projects

        Project address: PicoMRTK3 .

(2) Copy resources

        After decompressing the downloaded project, open its Assets, and copy the PicoMRTK3Support folder under the Scripts path to the Scripts path of the Unity project.

(3) Scripting Define Symbols settings

        Set in Project Settings-->Player-->Other Settings-->Script Compilation, add PICO_INSTALL and MRTK3_INSTALL. 

        After the addition is complete, click Apply. After Unity compiles, perform subsequent configurations.

(4) MRTK3 settings

        Set in Project Settings-->MRTK3.

1) Configuration file settings

        Just select the default configuration file.

2) MRTK subsystem settings

① Accessibility Subsystem settings

        Tick ​​MRTK Accessibility Subsystem and configure setting assets. By default, it is checked and configured. If it is not set, you can manually set it up, and set the asset to select the default. 

② HandsAggregatorSubsystem settings

        Check Pico MRTK Hands Aggregator Subsystem and configure the settings asset. Set assets to select the default.

③ Hands Subsystem settings

        Check Subsystem for Pico Hands API. 

(5) Scene setting

        You can copy the "HandInteractionExamples" scene in the MRTK example and configure it based on it. Note that in the copied scene, delete the prefab that reported the error. 

1) Add PXR_Manager component

        Add the PXR_Manager component to the MRTK XR Rig object, and check Hand Tracking. 

2) Configure left and right hands

① Hand model prefab creation

        Open Packages-->PICO Integration-->Assets-->Resources-->Prefabs, drag HandLeft and HandRight into the Hierarchy of the scene, and unbind the prefabs.

        Take the left hand as an example, add the PicoMRTKHandVisualizer script component to the HandLeft object, and configure related parameters. 

        Most of the above parameters are sub-nodes of the object itself, just drag the nodes of the five fingers into it one by one.

        Select the l_handMesh node under the object, and replace its material with the three materials under the path of Packages-->MRTK Input-->Visualizers-->RiggedHandVisualizer, as shown in the figure below. 

        After setting, drag the HandLeft object to the Prefabs folder created in the custom path under Assets to make it into a prefab.

        Similarly, create a HandRight prefab.

② Configuration of left and right hands

        Select MRTK LeftHand Controller under MRTK XR Rig, drag the created left-hand prefab to Model Prefab, and drag the Camera Offset object under MRTK XR Rig to Model Parent.

        Similarly, configure the right hand. 

3) Modify the script in MRTK3

① Modify ArticulatedHandController

        This script is mounted on MRTK LeftHand Controller and MRTK RightHand Controller objects under MRTK XR Rig. It is mainly to modify the UpdateInput method and add conditional compilation instructions. The modified method code is as follows.

/// <inheritdoc />
protected override void UpdateInput(XRControllerState controllerState)
{
    base.UpdateInput(controllerState);

    using (UpdateTrackingInputPerfMarker.Auto())
{
    if (controllerState == null)
    return;

    // Cast to expose hand state.
    ArticulatedHandControllerState handControllerState = controllerState as ArticulatedHandControllerState;

    // If we still don't have an aggregator, then don't update selects.
    if (XRSubsystemHelpers.HandsAggregator == null) { return; }

    bool gotPinchData = XRSubsystemHelpers.HandsAggregator.TryGetPinchProgress(
    handNode,
    out bool isPinchReady,
    out bool isPinching,
    out float pinchAmount
);

    // If we got pinch data, write it into our select interaction state.
    if (gotPinchData)
{
    // Workaround for missing select actions on devices without interaction profiles
    // for hands, such as Varjo and Quest. Should be removed once we have universal
    // hand interaction profile(s) across vendors.

    // Debounce the polyfill pinch action value.
    bool isPinched = pinchAmount >= (pinchedLastFrame ? 0.9f : 1.0f);
    #if !PICO_INSTALL
    // Inject our own polyfilled state into the Select state if no other control is bound.
    if (!selectAction.action.HasAnyControls())
{
    controllerState.selectInteractionState.active = isPinched;
    controllerState.selectInteractionState.activatedThisFrame = isPinched && !pinchedLastFrame;
    controllerState.selectInteractionState.deactivatedThisFrame = !isPinched && pinchedLastFrame;
}

    if (!selectActionValue.action.HasAnyControls())
{
    controllerState.selectInteractionState.value = pinchAmount;
}

    // Also make sure we update the UI press state.
    if (!uiPressAction.action.HasAnyControls())
{
    controllerState.uiPressInteractionState.active = isPinched;
    controllerState.uiPressInteractionState.activatedThisFrame = isPinched && !pinchedLastFrame;
    controllerState.uiPressInteractionState.deactivatedThisFrame = !isPinched && pinchedLastFrame;
}

    if (!uiPressActionValue.action.HasAnyControls())
{
    controllerState.uiPressInteractionState.value = pinchAmount;
}

    pinchedLastFrame = isPinched;
    #else
    // Debounced.

    controllerState.selectInteractionState.active = isPinched;
    controllerState.selectInteractionState.activatedThisFrame = isPinched && !pinchedLastFrame;
    controllerState.selectInteractionState.deactivatedThisFrame = !isPinched && pinchedLastFrame;
    controllerState.selectInteractionState.value = pinchAmount;


    controllerState.uiPressInteractionState.active = isPinched;
    controllerState.uiPressInteractionState.activatedThisFrame = isPinched && !pinchedLastFrame;
    controllerState.uiPressInteractionState.deactivatedThisFrame = !isPinched && pinchedLastFrame;
    controllerState.uiPressInteractionState.value = pinchAmount;

    pinchedLastFrame = isPinched;
    #endif
    }

    handControllerState.PinchSelectReady = isPinchReady;
}
}

② Modify HandConstraintPalmUp

        This script is mounted on the HandMenu object. Since the left and right hands are reversed in the PICO SDK, it is necessary to modify the value of the local variable dotProduct in the script, as shown below.

float dotProduct = Vector3.Dot(hand == XRNode.LeftHand ? -palmPose.Up : palmPose.Up, Camera.main.transform.forward);

3. About the problem of abnormal hand model upgrade of PICO4 system

        After upgrading the PICO4 system to 5.7.1, the position of the hand model is flipped and the joint points are distorted, which may be caused by PICO standardization of the hand model for gesture recognition. The solution is to upgrade the SDK to the latest version of the PICO SDK. This article uses the latest version 2.3.0.

1. Reconfigure left and right hands

        Open Packages-->PICO Integration-->Assets-->Resources-->Prefabs, drag HandLeft and HandRight into the Hierarchy of the scene, and unbind the prefabs.

        Take the left hand as an example, add the PicoMRTKHandVisualizer script component to the HandLeft object, and configure related parameters. Be careful not to forget to delete the original PXR_Hand script component and hide the child object RayPose.

         As can be seen from the figure above, PICO has standardized the name of the joint points of the hand model, so it only needs to be dragged into the corresponding position.

        Other configurations are the same as the corresponding settings in 2-2-(5)-2) above.

        Similarly, create a HandRight prefab.

2. Modify PicoMRTKHandVisualizer

        Mainly modify the code for updating the hand joint points in Update, as follows:

if (i == (int) HandJoint.JointWrist)
{
      riggedVisualJointsArray[i].localPosition =
                            handJointLocations.jointLocations[i].pose.Position.ToVector3();
      riggedVisualJointsArray[i].localRotation =
                            handJointLocations.jointLocations[i].pose.Orientation.ToQuat();
}
else
{
      //riggedVisualJointsArray[i].localRotation =
      //    handJointLocations.jointLocations[i].pose.Orientation.ToQuat();
      UnityEngine.Pose parentPose = UnityEngine.Pose.identity;

      if (i == (int)HandJoint.JointPalm ||
          i == (int)HandJoint.JointThumbMetacarpal ||
          i == (int)HandJoint.JointIndexMetacarpal ||
          i == (int)HandJoint.JointMiddleMetacarpal ||
          i == (int)HandJoint.JointRingMetacarpal ||
          i == (int)HandJoint.JointLittleMetacarpal)
      {
          parentPose = new UnityEngine.Pose(handJointLocations.jointLocations[1].pose.Position.ToVector3(), handJointLocations.jointLocations[1].pose.Orientation.ToQuat());
      }
      else
      {
          parentPose = new UnityEngine.Pose(handJointLocations.jointLocations[i - 1].pose.Position.ToVector3(), handJointLocations.jointLocations[i - 1].pose.Orientation.ToQuat());
      }

      var inverseParentRotation = Quaternion.Inverse(parentPose.rotation);
      riggedVisualJointsArray[i].localRotation = inverseParentRotation * handJointLocations.jointLocations[i].pose.Orientation.ToQuat();
}

3. Restore HandConstraintPalmUp

        This script is mounted on the HandMenu object. In the new version of PICO SDK, left and right hands are optimized normally, so it is necessary to restore the value of the local variable dotProduct in the script, as shown below.

float dotProduct = Vector3.Dot(palmPose.Up, Camera.main.transform.forward);

4. There are still problems

        After the above configuration, the hand model can be displayed normally, and the interaction is no problem.

        However, the left-hand ray is still reversed, and the left palm must face up to display the ray, which does not affect close-range interaction. If there is a small partner in this area who finds the problem, please leave a message in the comment area, thank you very much.