Briar

Gesture Warfare: Cyber Dojo

A hands-free computer game driven by real-time hand tracking in Unity.

  • Unity
  • Hand Tracking
  • Novel Input Techniques
  • Game
Built with
Unity · C# · MediaPipe · Kalman filtering

Game Mechanics

Gesture Warfare: Cyber Dojo is a third-person PvE shooter with six levels, controlled entirely through hand movements. Each hand can perform the same six gesture-based actions:

  • Aim: Extend the thumb and index finger while curling the other fingers, with the index finger pointed toward the screen.
  • Shoot: Bend the thumb while keeping the aiming pose.
  • Pulse grenade: Make a fist.
  • Recover health: Form an OK sign.
  • Reload: Give a thumbs-up.
  • Pause: Open the hand and extend all five fingers.

Raising a hand above a set height threshold makes the player jump.

Together, these controls let players play through all six levels without a conventional controller.

Demo

Work highlights

  • Integrated MediaPipe hand tracking into Unity for real-time gesture-based input.
Hand landmark mapping used for gesture input.
Hand landmark mapping for gesture input
  • Improved tracking stability and responsiveness with Kalman filtering and an event-driven interaction architecture.
  • Depth and scale estimation with a monocular camera.
Unity visualization of a far hand perspective.
Far perspective
Unity visualization of a near hand perspective.
Near perspective

Depth: Reflecting the overall far/near perspective.

Scale: Used to control the overall size of the visualization.