NativePHP Face Tracking#
Let people control your app with a blink, wink, smile, or head movement. Add a camera preview, choose the gestures you want, and handle them in PHP.
Platforms: iOS · Android (beta)
What you need#
- PHP 8.4+ and
nativephp/mobile^4.5.2. - An iPhone or iPad with face-tracking support, running iOS 16 or later.
- For Android (beta), a compatible device with a front camera, running Android 7 (API 24) or later.
- Permission to use the camera. The plugin adds the required camera permission settings to your app.
Your app may require a newer OS version. Call FaceTracking::isSupported() to
check whether a device supports tracking. This checks device support, not camera
permission. Tracking is not available on desktop or in the iOS Simulator.
Installation#
composer require kirilldakhnyuk/nativephp-face-tracking
Register the plugin:
php artisan native:plugin:register kirilldakhnyuk/nativephp-face-tracking
Rebuild your native app after installing or updating the plugin.
Your first gesture#
This screen listens for blinks, smiles, and left winks. When a gesture is detected, it shows the gesture's name.
namespace App\NativeComponents; use Illuminate\View\View;use KirillDakhnyuk\FaceTracking\Events\GestureStarted;use KirillDakhnyuk\FaceTracking\Facades\FaceTracking;use KirillDakhnyuk\FaceTracking\Gesture;use Native\Mobile\Attributes\On;use Native\Mobile\Edge\NativeComponent; class FaceControls extends NativeComponent{ public string $lastGesture = 'Waiting for a face'; public function mount(): void { FaceTracking::configure([ Gesture::blink(), Gesture::smile(), Gesture::winkLeft(), ]); } #[On(GestureStarted::class)] public function gestureStarted(string $name, int $timestamp): void { $this->lastGesture = $name; } public function render(): View { return view('native.face-controls'); }}
Add the screen to routes/mobile.php:
use App\NativeComponents\FaceControls;use Illuminate\Support\Facades\Route; Route::native('/face-controls', FaceControls::class);
Create resources/views/native/face-controls.blade.php. This example uses the
layout and text components from nativephp/mobile-ui:
<native:column class="w-full h-full p-4 gap-4"> <native:face-tracking-preview mirrored class="w-full h-64" /> <native:text :text="$lastGesture" /></native:column>
The preview keeps tracking active while it is on the screen. Removing it stops
tracking unless another preview or a start() call still needs it.
Camera preview#
Use <native:face-tracking-preview /> or <x-native-face-tracking-preview />.
| Option | Default | What it does |
|---|---|---|
mirrored |
true |
Shows a mirror view of the camera |
show-mesh |
false |
Draws a face mesh on iOS; has no effect on Android |
hidden |
false |
Keeps tracking active without showing the camera |
For gestures without a visible camera feed:
<native:face-tracking-preview hidden />
Use the hidden option above instead of a Tailwind class. On Android, only the
most recently added visible preview shows the camera feed.
Choose your gestures#
Pass the gestures you want to FaceTracking::configure():
FaceTracking::configure([ Gesture::blink(), Gesture::winkLeft(), Gesture::winkRight(), Gesture::smile(),]);
| Gesture | Detects |
|---|---|
blink() |
A quick blink with both eyes |
eyesClosed() |
Both eyes held closed |
winkLeft() / winkRight() |
One eye closed while the other stays open |
smile() |
A smile |
jawOpen() |
An open mouth |
browRaise() |
Raised eyebrows |
tongueOut() |
Tongue out — iOS only |
puff() |
Puffed cheeks |
tiltLeft() / tiltRight() |
Head tilted to one side |
nod() |
A nod |
shake() |
A head shake |
Left and right always mean the person's own left and right, even with a mirrored preview.
All supported gestures are enabled until you call configure(). Use
Gesture::presets() to get all 13 presets, or configure([]) to turn off gesture
events and streaming. Android supports 12 presets; configure() lists
tongue-out and any custom rules that need it in the result's unsupported field.
Gesture settings are shared across the app. Calling configure() from another
screen replaces the current rules.
Adjust sensitivity#
Each gesture comes with defaults. You can change how easily it triggers and how long someone needs to hold it:
$smile = Gesture::smile() ->threshold(0.6) ->minDuration(200) ->cooldown(500);
threshold(): a higher value makes expressions harder to trigger; a lower value makes them easier. Expression scores range from 0 to 1.minDuration(): how long to hold an expression before it counts, in milliseconds.maxDuration(): recognizes a gesture when released within this time limit.cooldown(): how long to wait before recognizing the gesture again, in milliseconds.release(): the score at which a held gesture ends. It must be below a>threshold or above a<threshold.
For nod() and shake(), use amplitude() in degrees (greater than 0, up to 180)
and cooldown(); the other settings do not apply.
Each builder call returns a new rule, so keep the returned value as shown above.
Add a custom gesture#
Combine expression or head-angle conditions. All conditions must match:
use KirillDakhnyuk\FaceTracking\Enums\BlendShape;use KirillDakhnyuk\FaceTracking\Enums\HeadAxis; $jawAndBrow = Gesture::custom('jaw-and-brow') ->when(BlendShape::JawOpen, '>', 0.5) ->and(BlendShape::BrowInnerUp, '>', 0.4) ->minDuration(150); $lookUp = Gesture::custom('look-up') ->when(HeadAxis::Pitch, '>', 15); FaceTracking::configure([$jawAndBrow, $lookUp]);
Expression values range from 0 to 1. Head-angle conditions accept -90 to 90 degrees.
Invalid rules throw InvalidGestureException.
Read face data#
A snapshot gives you the latest face data. It returns null when no face is tracked.
use KirillDakhnyuk\FaceTracking\Enums\BlendShape; $face = FaceTracking::snapshot();$leftEyeClosed = $face?->blendShape(BlendShape::EyeBlinkLeft); // 0 to 1, or null$head = $face?->head; // yaw, pitch, roll in degrees; position in metres$gaze = $face?->gaze; // eye direction, when available
Positive yaw means turning left, positive pitch means looking up, and positive roll means tilting toward the right shoulder.
On Android, tongue-out values and gaze data are unavailable, and head position
is an estimate. Missing expression values return null.
Start and stop tracking#
The preview handles starting and stopping for you. Without a preview, call
FaceTracking::start() yourself, then FaceTracking::stop() when finished.
Match every successful start() with a stop(), including when your screen is
removed in unmount().
stop() only cancels tracking requested through start(). A preview still on
screen keeps tracking active, even if it is hidden.
| Method | What it returns |
|---|---|
isSupported() |
Whether the device supports face tracking |
status() |
Unsupported, Denied, Idle, Starting, Running, or Interrupted |
configure($gestures, stream: null) |
A result listing any unsupported gestures |
start() |
true when started or waiting for permission or the app to become active |
stop() |
Nothing; stops one matching start() request |
snapshot() |
The latest face data, or null |
Tracking pauses when the app goes into the background and resumes when it becomes
active, as long as a preview or a start() request still needs it. Losing the face
or pausing tracking ends held gestures.
If starting fails, start() returns false. Listen for TrackingFailed to handle
camera errors. A failure clears active tracking requests; call start() to retry.
On Android, you can request permission again after an ordinary denial. If access
was permanently denied, the user needs to enable it in device settings.
Events#
Use Native\Mobile\Attributes\On to listen for events, as shown in the first
example. Event classes are in KirillDakhnyuk\FaceTracking\Events. Each includes
a timestamp in Unix milliseconds.
| Event | When it happens | Other fields |
|---|---|---|
GestureStarted |
A gesture is recognized | name |
GestureEnded |
A held gesture ends | name, durationMs |
FaceDetected |
A face appears | None |
FaceLost |
The tracked face disappears | None |
FaceFrame |
A face-data update arrives, when streaming is enabled | isTracked, blendShapes, head, gaze |
TrackingFailed |
Tracking cannot start or continue | reason, message |
Quick gestures, such as blinks, emit only GestureStarted. Held gestures, such as
smiles, also emit GestureEnded. Its durationMs measures time since recognition.
Failure reasons are unsupported, camera_denied, interrupted, and failed.
Receive regular updates#
Streaming is off by default. Enable it when you need regular face-data updates:
FaceTracking::configure([Gesture::smile()], stream: 10);
This sends up to 10 FaceFrame events per second. The allowed range is 1 to 30.
For simple controls, gesture events or occasional snapshots are usually enough
and keep PHP updates light.
License#
This is paid, proprietary software. You need a valid commercial license to use it. See LICENSE and the terms supplied with your purchase.
The bundled MediaPipe model has its own Apache 2.0 license. See model attribution for details.