Bringing virtual cats to your world with Project Tango

Posted by Jason Guo, Developer Programs Engineer, Project Tango

Project Tango brings augmented reality (AR) experiences to life. From the practical to the whimsical, Project Tango apps help place virtual objects -- anything from new living room furniture to a full-sized dinosaur -- into your physical world.

Last month we showed you how to quickly and easily make a simple solar system in AR. But if you are ready for something more advanced, the tutorial below describes how to use Project Tango’s depth APIs to associate virtual objects with real world objects. It also shows you how to use a Tango Support Library function to find the planar surface in an environment.

So what’s our new tutorial project? We figured that since cats rule the Internet, we’d place a virtual cat in AR! The developer experience is designed to be simple -- when you tap on the screen, the app creates a virtual cat based on real-world geometry. You then use the depth camera to locate the surface you tapped on, and register (place) the cat in the right 3D position.

Bring on the cats!

Before you start, you’ll need to download the Project Tango Unity SDK. Then you can follow the steps below to create your own cats.

Step 1: Create a new Unity project and import the Tango SDK package into the project.

Step 2: Create a new scene. If you don’t know how to do this, look back at the solar system tutorial. Just like the solar system project, you’ll use the Tango Manager and Tango AR Camera in the scene and remove the default Main Camera gameobject. After doing this, you should see the scene hierarchy like this:

Step 3: Build and run once, making sure sure the application shows the video feed from Tango’s camera.

Step 4: Enable the Depth checkbox on the Tango Manager gameobject.

Step 5: Drag and drop the Tango Point Cloud prefab to the scene from the TangoPrefab folder.

Tango Point Cloud includes a bunch of useful functions related to point cloud, including finding the floor, transforming pointcloud to unity global space, and rendering debug points. In this case, you’ll use the FindPlane function to find a plane based on the touch event.

Step 6: Create a UI Controller gameobject in the scene. To do this, click the “Create” button under the Hierarchy tab, then click “Create Empty.” The UI Controller will be the hosting gameobject to run your UIController.cs script (which you’ll create in the next step).

Step 7: Click on “UIController gameobject” in the inspector window, then click “Add Component” to add a C# script named KittyUIController.cs. KittyUIController.cs will handle the touch event, call the FindPlane function, and place your kitty into the scene.

Step 8: Double click on the KittyUIController.cs file and replace the script with the following code

using UnityEngine;
using System.Collections;

public class KittyUIController : MonoBehaviour
{
    public GameObject m_kitten;
    private TangoPointCloud m_pointCloud;

    void Start()
    {
        m_pointCloud = FindObjectOfType();
    }

    void Update ()
    {
        if (Input.touchCount == 1)
        {
            // Trigger place kitten function when single touch ended.
            Touch t = Input.GetTouch(0);
            if (t.phase == TouchPhase.Ended)
            {
                PlaceKitten(t.position);
            }
        }
    }

    void PlaceKitten(Vector2 touchPosition)
    {
        // Find the plane.
        Camera cam = Camera.main;
        Vector3 planeCenter;
        Plane plane;
        if (!m_pointCloud.FindPlane(cam, touchPosition, out planeCenter, out plane))
        {
            Debug.Log("cannot find plane.");
            return;
        }

        // Place kitten on the surface, and make it always face the camera.
        if (Vector3.Angle(plane.normal, Vector3.up) < 30.0f)
        {
            Vector3 up = plane.normal;
            Vector3 right = Vector3.Cross(plane.normal, cam.transform.forward).normalized;
            Vector3 forward = Vector3.Cross(right, plane.normal).normalized;
            Instantiate(m_kitten, planeCenter, Quaternion.LookRotation(forward, up));
        }
        else
        {
            Debug.Log("surface is too steep for kitten to stand on.");
        }
    }
}

Notes on the code
Here are some notes on the code above:

• m_kitten is a reference to the Kitten gameobject (we’ll add the model in the following steps)
• m_pointCloud is a reference to the TangoPointCloud script on the Tango Point Cloud gameobject. We need this reference to call the FindPlane method on it
• We assign the m_pointcloud reference in the Start() function
• We check the touch count and its state in the Update() function when the single touch has ended
• We invoke the PlaceKitten(Vector2 touchPosition) function to place the cat into 3D space. It queries the main camera’s location (in this case, the AR camera), then calls the FindPlane function based on the camera’s position and touch position. FindPlane returns an estimated plane from the touch point, then places the cat on a plane if it’s not too steep. As a note, the FindPlane function is provided in the Tango Support Library. You can visit TangoSDK/TangoSupport/Scripts/TangoSupport.cs to see all of its functionalities.

Step 9: Put everything together by downloading the kitty.unitypackage, which includes a cat model with some simple animations. Double click on the package to import it into your project. In the project folder you will find a Kitty prefab, which you can drag and drop to the Kitten field on the KittyUIController.
Step 10:
Compile and run the application again. You should able to tap the screen and place kittens everywhere!We hope you enjoyed this tutorial combining the joy of cats with the magic of AR. Stay tuned to this blog for more AR updates and tutorials!

A final note on this tutorial
So you’ve just created virtual cats that live in AR. That’s great, but from a coding perspective, you’ll need to follow some additional steps to make a truly performant AR application. Check out our Unity example code on Github (especially the Augmented Reality example) to learn more about building a good AR application. Also, if you need a refresher, check out this talk from I/O around building 6DOF games with Project Tango.