Where We Started:

The geometric visualiser was our first real dive into Three.js and AR. We had Platonic solids rotating in real time, hand gesture recognition, four viewports synced together, and an AR mode that projected geometry onto a Hiro marker. It worked, and it worked well. But somewhere along the way we asked ourselves, 'What if this wasn't just a maths toy? What if the same rendering pipeline served something more applied?'

The Pivot

The idea of a security camera planner came from a simple observation: placing cameras in a space is fundamentally a geometry problem. Coverage cones, field of view, blind spots — all of it is just 3D math dressed up in a practical context. We already had the rendering infrastructure. We just needed to reframe it.

The first version was a basic 2D floor plan. Click to place a camera, see a coverage cone, and identify blind spots. Simple, clean, immediately useful.

Adding the Third Dimension

The 2D view was useful but flat — literally. So we brought in a GLB house model and rebuilt the viewport system to show the same scene from four angles simultaneously: perspective, top, front, and side. The quad-viewport system from the geometric visualiser carried over almost directly. That felt like the project clicking into place.

Gesture Control

We kept gesture mode from the previous project because it made sense here too – orbiting a 3D scene with your hands rather than a mouse is just better when you’re presenting or demonstrating. MediaPipe handled the hand tracking, same as before.

What It Is Now

The current version lets you plan an entire camera layout in 2D, switch to 3D to validate placement against the actual house geometry, orbit the scene in four views simultaneously, and use gesture control throughout. It’s a tool that could genuinely be used before installing a real system.

The geometric visualizer taught us the rendering pipeline. The camera planner taught us that a good pipeline is only as useful as the problem it’s pointed at.

Hand Tracking: Teaching the App to See Your Hands

Implementing MediaPipe Hands was a turning point. Instead of sliders and buttons, users could now interact with 3D objects naturally—the way humans naturally communicate through gesture.

The Challenge: Getting hand landmarks to map smoothly onto 3D space. Our hand was visible on-screen via webcam, but translating (x, y, z) coordinates from the camera feed into meaningful cube transformations required careful calibration.

The Solution: Three core gestures emerged:

1. Pinch (Thumb + Index) — Distance between fingertips directly controls scale. Closer pinch = larger scale. It’s intuitive: pinching makes things grow, spreading fingers shrinks them.

2. Grab (Open Hand) — Palm center position maps to cube position. Moving your hand moves the cube. Pure spatial mapping.

3. Swipe (Hand History) — We tracked hand position over 10 frames. Horizontal swipes rotate around the Y-axis, vertical swipes around X. The velocity of the swipe controls rotation speed.

The so-called magic lied in the simplicity using just simple libraries, raw landmarks and basic geometry.

Gimbal Mode: When Your Device Becomes the Controller

After gesture control, we wanted users to manipulate objects by tilting their phone. This required accessing device motion sensors—a feature that works differently across browsers and operating systems.

The Discovery: Android Brave doesn’t expose device orientation the way Safari on iOS does. So we had two fallbacks:

1. Mobile: Device accelerometer via devicemotion event

2. Desktop: Arrow keys (↑↓←→) for X/Y rotation, Q/E for Z-axis

This dual approach meant gimbal mode works everywhere—you’re never locked out by hardware limitations. Demonstrating on a laptop? Use arrow keys. On a phone? Tilt it.

How It Works: Device acceleration gets fed through atan2 to calculate tilt angles. These angles become Euler rotation values for the cube. The result: your device becomes an extension of the 3D space. Tilt forward, the cube tilts. Rotate your phone, the cube spins.

Transformation Sequences: Visualizing Order Matters

The final piece was pedagogical—showing students why matrix multiplication order matters.

T Only, T+R, T+R+S buttons preset the cube to specific states, building complexity gradually:

• T Only: Pure translation. Position changes; shape stays same.

• T+R: Translation + rotation. Now the position AND orientation change.

• T+R+S: Full transformation suite. The cube translates, rotates, and scales.

• 

Then came the punchline: Order Matters.

When you toggle “Order Matters,” the same transformations apply but in different order:

• T×R×S order: Translate first (smaller effect), rotate, then scale

• S×R×T order: Scale first, rotate, then translate (translation now travels further because the cube is bigger)

This allows for students to visually see why the order in matrix multiplication actually matters and not learn to do algebraically.

Integration: Making It All Work Together

The challenge wasn’t individual features—it was unifying them:

• Hand tracking runs on Three.js

• Gimbal works on the same cube

• Transformation sequences trigger on both

• State syncs to localStorage for AR mode

• All without conflicts or performance loss

We built it modular: gesture control in ar-mode.html, Hiro AR in hiro-ar-mode.html, desktop in index.html. Each mode is self-contained but contributes to the same learning experience.

The Result

Users can now:

• Manipulate 3D objects with their hands (gestural interface)

• Control objects by tilting their phone (embodied interface)

• See transformation sequences build complexity step-by-step (pedagogical interface)

• View everything in AR space (immersive interface)

What began as abstract math—matrices, transformations, composition—became something tangible. Graspable. Literally graspable with your hands!

We began with a very simple goal: to build a system that could visually demonstrate basic geometric transformations. At this stage, the focus was only on a single cube, without any surrounding environment or context. While the functionality worked, it felt isolated and lacked depth. It did the job technically, but it didn’t feel intuitive or engaging.

2. Understanding Translation

The first transformation we explored was translation. Moving the cube along the X, Y, and Z axes helped us understand how position changes in 3D space. While this was straightforward mathematically, visualising it in real time gave a much clearer perspective than static diagrams ever could.

3. Exploring Rotation

Rotation introduced a new layer of complexity. Unlike translation, it wasn’t just about movement but orientation. Watching the cube rotate across different axes made us realise how easily one can lose track of spatial orientation without proper visualisation.

4. Building an Environment

At this point, it became clear that the cube alone wasn’t enough. We needed a better sense of space and interaction. This led us to introduce an environment around the object, making the transformations feel more grounded and visually meaningful.

5. Pushing Beyond — Introducing AR

While the system was now functional, we felt it still lacked a level of interactivity that others were exploring. This led us to experiment with camera-based interaction using marker detection. Initially, this was confusing — especially understanding how markers work and whether we needed complex AI models. Eventually, we realised that a simple marker-based system could achieve what we needed.

6. Bringing It All Together

With AR integrated, the system finally started to feel complete. The ability to detect a marker, visualize an object, and then bring that object into the transformation environment created a seamless flow. It connected real-world interaction with digital manipulation in a way that made the entire system more intuitive.