Set up a face tracking project

In this learning experience, you'll learn to create your own AR face filter apps. The first app you’ll make is a simple filter that applies materials and 3D models to the user’s face.

In this tutorial, you’ll take the first step in developing an AR face filter app. You’ll set up your project to detect and track the user’s facial features and visualize the tracking functionality with a green material overlay.

As you prepare to begin your AR development work, it might be helpful to hear what other creators experience in their day-to-day work. In the video below, a few experienced AR creators discuss their personal development process and what it means to work professionally in XR.

You should have already completed Get started with AR tutorials. In those tutorials, you complete the following setup tasks:

• Install Unity version 2022.3.1f1.
• Make sure your mobile device and computer are set up for AR development.
• Create a new Unity project using the Mobile AR Template. Deploy a simple AR app to your mobile device that displays a spinning cube in your physical environment.

If you have already successfully built and run the spinning cube app on your device, you’ are ready to continue with this tutorial.

Let’s get started by setting up your project:

1. If your AR project is not already open, open it now.

2. Select File > New Scene to create a new scene from the AR template.

3. Select File > Save As, then save the scene as “Basic_Face_Filter” inside the Scenes folder.

4. Download and unzip the Basic Face Filter assets.

5. In Unity, go to the main menu and select Assets > Import Package > Custom package.

6. Navigate to the location of the unzipped folder, select the BasicFaceFilterAssets.unitypackage file, and then select Open.

7. In the Import Unity Package window, check that all the assets are selected and then select Import.

8. In Unity, go to the Project window and navigate to Assets > _BasicFaceFilter.

Take a little time to explore the assets that you have just imported before you continue.

Augmented reality experiences that overlay digital content on your face require face tracking ability. Face tracking tells your mobile device where your face is when a camera sees it. The good news is that there’s already computer code packaged up for you that makes this work! You’ve already imported some of that code, but now you need to set up your scene to make that code work and to visualize the faces that are tracked in your application.

You’ll use the AR Default Face GameObject to visualize the tracking of your face. In order to see the AR Default Face when you test, you'll create a prefab that will be activated when the camera sees your face. You'll also need to apply a material to that AR Default Face GameObject so that it shows up. A special material is provided as part of the project.

Follow these instructions to set up the AR Default Face prefab:

1. In the Hierarchy window, right-click and select XR > AR Default Face to add a new AR Default Face GameObject to the scene.

The AR Default Face is a GameObject you can customize to make your own creative face filters.

2. Next, you need to make sure that the AR Default Face will show up when a face is detected by your phone’s camera. In the Project window, navigate to Assets > _BasicFaceFilter > Materials and find the Face Debug MAT material.

3. Drag the Face Debug MAT material from the Project window onto the AR Default Face GameObject in the Hierarchy window to set the material. You can check if the material has been set by navigating to the Inspector window for the AR Default Face GameObject and scrolling down to the Material Inspector. Here, you should see the Face Debug MAT (Material).

4. In the Project window, navigate to Assets > _BasicFaceFilter > Prefabs.

5. Drag the AR Default Face GameObject from the Hierarchy window into the Prefabs folder in the Project window.

The GameObject icon and name should now be displayed in light blue in the Hierarchy window.

You have just turned the AR Default Face GameObject into a prefab. Prefabs act as templates so that you can reuse GameObjects configured in a specific way.

Next, you need to tell the XR Origin (AR Rig) which GameObject you want to display over a face when it’s detected. You’ll do this using a new component called the AR Face Manager.

Follow these instructions to set up the AR Face Manager:

1. In the Hierarchy window, delete the Cube GameObject – it is only there for testing.

2. Select the XR Origin(AR Rig) GameObject.

3. In the Inspector window, select Add Component and then search for and add the AR Face Manager component. This component will manage the addition, updating, and removal of GameObjects to represent faces.

4. In the Inspector window, go to the AR Face Manager component and locate the Face Prefab property.

5. Drag the AR Default Face prefab you created from the Project window (in Assets > _BasicFaceFilter > Prefabs) onto the Face Prefab property.

6. In the Hierarchy window, delete the AR Default Face GameObject. When a face is detected by your app, AR Foundation will generate an AR Default Face GameObject from the prefab that you have just created.

The app you built with the spinning cube used the camera on the back of your handheld device. However, this new app involves face tracking, so it makes more sense to use the camera on the front of your device.

Follow these instructions to configure the app to use the front camera:

1. In the Hierarchy window, hold the Alt key (macOS: hold the Option key) and select the foldout icon on the XR Origin (AR Rig) GameObject to expand all of its child GameObjects. Parent and child GameObjects are a way to organize the different GameObjects in a scene.

2. Select the Main Camera GameObject.

3. In the Inspector window, find the AR Camera Manager (Script) component.

4. Set the Facing Direction property to User.

It’s time to test your project! The video below demonstrates how the app should function on your device.

To test the project in its current state, follow these instructions to build to your device:

1. Save the scene.

2. Open the Build Settings window (File > Build Settings).

3. Select Add Open Scenes to add the current scene to the build. Clear any previous scenes to make sure that they are not included in your new build.

4. Make sure that your device is plugged in, and then select Build and Run to build the app on your device.

Note: For a reminder of how to build to your device, you can review the process for Android or iOS.

5. The application will open automatically on your device. When a face is detected, the application should cover the face with a simple green face filter.

Note: If you're developing for iOS and the application is still using the world-facing camera instead of the user-facing one, it may be that Face Tracking is not enabled in your project settings. From the main menu, select Edit > Project Settings. From the left panel, select ARKit, then make sure Face Tracking is enabled.

If you want to further develop your skills, explore new concepts, or improve your project, check out some of the optional activities below. Each one is tagged as either Easy, or Medium, so you can choose the level of challenge.

These activities are entirely optional, so if you’re not interested, no problem — just skip this step. We do recommend attempting at least one of them in order to get the most out of this learning experience. Good luck!

Easy: Change face debug material color

While the debug material is a fantastic shade of green, you might like to make the material a different color. One way to change the color would be to open the debug material and change the color using a color picker. Another way to change the face debug material color is to make a new material and apply that to the face.

For more information on how to work with materials, you can explore the Manage materials in a project section of the Creative Core pathway.

Medium: Create screen recordings of the application to share

Sharing what you have done with others is a great way to show off your accomplishments and learning. One of the easiest ways to share the progress that you're making on your AR journey is to record your screen as the application runs.

There are different processes for recording on iOS and on Android. Once you’ve made a recording, share it with your friends, family, and the world!

In this tutorial you created a basic face tracking application that overlays the default AR face model over your face — great job! In the next tutorial, you'll make things a little more exciting by adding custom materials to the face model.