MetaMouse

MetaMouse points the Mac cursor at whatever you are aimed at. A camera on a pair of Meta's glasses sees the screen, the software works out where on that screen you are pointing, and if you hold still over something clickable the cursor appears and snaps to it. One key clicks. A second scrolls.

It started as Mouse, a plain macOS experiment with an ordinary mouse standing in for the eyes, because the interaction was the unsolved part and the tracking was not.

The bet

You do not need to know where someone is aimed to the pixel. macOS will tell you, deterministically and in well under a millisecond, which interface element sits under any point. So aim roughly, then ask the system what is actually there, and snap to it. the aim half can be bad

That is the whole idea. It converts an estimate into a correct click, and it means the aiming half can be mediocre and the thing still works.

Finding the screen

The glasses camera sees a room. Before any of that can happen it has to find the screen inside the room.

I tried markers stuck on the corners, which works and looks terrible. What shipped instead is a thin, brightly colored ring the app draws around the edge of its own display. Threshold the frame for that color, take the largest contour, pull four corners out of it, and you have a mapping from camera space to screen space.

Head, not eyes. The glasses carry a camera and not an eye tracker, and pretending otherwise would have been a much longer project than the one I wanted to do.

It fails the way a color threshold fails. Glare on the bezel washes the ring out. Something genuinely cyan in the room can beat it for largest contour. All four corners have to stay in view, so aiming hard at one corner of the screen loses the opposite corner and confidence decays.

Where it stopped

Everything is written. Twenty-eight tests pass, including a frame-by-frame check of the wire protocol between the phone and the Mac, proved against a synthetic client speaking the exact byte format.

None of it has run through the glasses. The Mac loop has never been run on a Mac, the iOS streamer has never been compiled, and whether a cyan ring reads at all through that particular camera is the one question no test can answer.

It is parked exactly where the next step needs hardware and an afternoon, and I have not given it one.