see, saw, seen · Ziyan Xie

In Virilio's War and Cinema, he describes how the logistics of modern warfare shifted from the physical territory to the visual field. The battlefield became, at some point, a problem of who could see what and how fast. The camera and the gun converged. "Perception and destruction have now become coterminous," he writes.¹

The word that stays with me from that whole tradition — Virilio, Farocki, the drone theorists, etc. — is logistics. The management of flows. Logistics, a word about how things move, and through what channels, toward what ends, and who decides.

This is how "Operational Seeing" comes into place. It's a game, or something adjacent to a playable medium, where a human player operates an interface to exert control over a crowd of artificial agents. There are no comprehensible tales of truth or affective content to keep you engaged. What I hope is to lean on the raw motivation of scoring and reward hacking so that you forget you're playing, and thus you're actually playing, and so that you've learned to see through the gaze of machines before you've had a chance to question what that gaze is for.

The interface itself would make its own violence legible before anyone has to spell it out.

#Teaching the machine's gaze

The first challenge that comes to mind: how do you even communicate the abstract math and binary codes that a machine reads naturally to a human? We can use metaphors, of course, but metaphors are symbolic representations, and the subject is already a complex symbolic system. It will soon spiral into a mess where nothing makes sense. But I don't want the experience to make total sense either. A neural net does not know anything, nor does it aspire for knowledge. It learns through back-propagated errors. It keeps grinding to minimize those errors. So why make it clear to a human what the system is about and what to do?

The knowledge should be limited, restricted, uncomfortably constraining, and only then does it feel natural to stop overthinking and focus on the task that may not make full sense. Only then would you not want to see the world outside of the aperture, so you can willfully stay within the virtual world that runs on the engine, a surrogate that temporarily stands in for your whole field of vision and knowledge. This serves as the backbone, the hidden layer, the logic of the gameplay.

#Optical flow as the medium

"I am an eye. A mechanical eye. I am the machine that reveals the world to you as only the machine can see it." Dziga Vertov wrote this in 1923, on his Kino-Eye, which liberates cinema from theater, from staging, from the lie of acted reality.² The promise inverts easily. The mechanical eye still reveals the world as only the machine can see it. But what does the machine want to see? And what does it smooth away?

The point is to create something that is confusing and engaging at the same time. We need strong visuals, and probably audio cues. First few things come to mind: we need things to move, we need colors (a lot of them), and we need splashes and senses of something being off and something that can be restored to neat order.

This simulation for machine vision found its form quickly. I decided to use optical flow as the core experience. It's a computer vision technique that encodes motion direction as hue and speed as saturation.

And it's not a metaphor for machine vision. It is machine vision (or at least a functioning miniature of it). The technique was developed to extract aggregate behavior from a scene without identifying any individual within it. If that still doesn't feel evocative enough: optical flow is how surveillance systems read crowd movement.

#The flow field and turbulence

So, we have a slowly shifting canvas of hue and brightness. A cluster of moving points, or agents, as people like to call them now, start out moving rightward, diagonally, in order. The agents themselves are nearly invisible, much like individuals in a society of control. You can see them if you force yourself to look, but the visual hierarchy pushes your eye toward the background texture, the flow field of colors.

The flow field at rest is calm, uniform, and boring. The game needs things for the player to respond to. So I inject what I call "turbulence events," forces that push agents into specific patterns of motion.

There are six types: Circular, Vortex, Scatter, Wave, Oscillation, Cluster. Inside the code they are geometric descriptions of force fields. But when your cursor overlaps an active event, the interface speaks in language you can comprehend: Scatter becomes DISPERSAL, Circular becomes ASSEMBLY, Cluster becomes BLOCKADE. No other context is given.

Just the motion and the label and the agents restoring to orderly flow under your cursor as you hold the mouse down and watch the colors settle. You would probably have an inkling of what's going on, and that inkling is the hinge. You register that the motion vector carries a social name, and that the social name makes it a judgment, and you're absorbing this while you're also busy keeping your loss score low and hunting for visual cues, so there's little cognitive capacity left for you to step back from sensor to decision.

Some more mindful players would find out that not all turbulent events need to be suppressed. Some are classified as coherent, and smoothing them out wastes your energy and nothing else. You have to wait for your suppression tool to recharge while the unstable dynamics multiply elsewhere. The system has already decided which patterns are divergent and which are coherent. You have to learn which is which, so that you get your rewards more efficiently.

Unsurprisingly, it's a tall order for most players within a five-minute window, so I don't penalize for it. It's a bonus for those who are observant. The gameplay is meant to last five minutes, enough to get a taste of what machine vision feels like, enough to sink in before you get bored and quit before the final stage.

#Designing the cursor

Let's now talk about the cursor. A crosshair or a circle would just read like a game, something lifted from shooters. My intended behavior for the cursor is when it overlaps a dense cluster of agents, the grid would spatially hash them and draw small bounding boxes around qualifying groups, with divergence score floating at the corner.

It's a subtle visual cue to guide the player's attention, and I'm borrowing a lot from the visual grammar of the stereotypical feature detection in computer vision — those regularly spaced points that a system lays over an image to carve it into analyzable cells.

Eventually, I designed dot-grid for the cursor: a square array of small sampling points that grows when you perform well and contracts when turbulence outpaces you. 3x3 at the start, expanding up to 7x7. It's driven by an accumulator that tracks your dampening output against active turbulence. If you do well, the instrument widens. But it runs the other way too.

In some way, it holds this Steyerl-ian tenet: "Resolution determines visibility." As the dot-grid grows, you resolve more. You detect more clusters, draw more boxes, assign more labels. The system widens your aperture in proportion to your obedience. There's something obscene about how satisfying this feels in the hand.

#One tool, not many

Fun story, I had actually built three tools early on: SCAN, which is sustained dampening. PULSE, an instant burst. LOCK, a freeze. The code for all three still exists. But now only SCAN is exposed to the player. This was about what kind of attention the game asks for.

Multiple tools make a strategy game. They invite optimization, min-maxing, toolkit management, and your focus drifts from the field to the interface. You start reading the HUD and lose the world it overlays. One tool keeps you in the world. SCAN is sustained attention and spatial reading. You hover, you hold, you watch the field yield and slowly recover. There is just you, applying pressure, watching color drain out of a region and knowing it will seep back.

In Workers Leaving the Factory Farocki shows the same Lumiere scene, workers streaming through a factory gate, over and over, intercut with a century of footage from industrial films and surveillance cameras.³ The camera at the factory gate in 1895 and the surveillance camera in the plaza in 2003 are doing the same work a century apart. Both are counting bodies in motion. The single tool here works this way too. You scan the same kinds of disturbances for five minutes. The patterns repeat. The labels cycle. And at some point the repetition is the content. You are noticing what your hand has been doing, maybe for the first time, and maybe too late for it to matter.

#The ending

My favorite moment is the last thing that happens after gameplay. This is my tribute to Farocki.

The screen splits. On the left, a replay of your entire session, the moments you scanned and the moments you let breathe. On the right, a narrated documentary, edited to rhyme with those same moments of abstraction and psychological flow you just went through.

The structure borrows from essay film, from the juxtaposition editing that Marker and Farocki practiced. Meaning lives in the gap between two images placed side by side and left to resonate. The whole point of the split-screen is that you watch your own performance being reframed in real time, and that reframing needs to happen in the body, in the feeling of recognition, or it falls flat.

Steyerl, in How Not to Be Seen, talks about how the most important things want to remain invisible. And there's something in that formulation that touches Operational Seeing, at a strange angle. Here everything is visible. The flow field, the labels, the bounding boxes, the score, the hint text, and the documentary at the end. The violence of the interface is on the surface.

Interface literacy creates a kind of tunnel vision. When you're good at a system you stop seeing the system and start seeing through it, and that's the whole point of training, the whole reason training works, and also the thing that makes the reveal necessary for the people who got so good at the game that they forgot to read what it was telling them in plain language all along.

#What the machine learned to see

James Bridle tells this story in New Dark Age about the US Army training a neural network to identify tanks hidden in a forest. The network performed perfectly on training data, failed on every new image. Turned out the tank photos had been shot on cloudy days, the forest photos on sunny days. The network had learned weather. Whatever artificial intelligence becomes, Bridle suggests, it will be fundamentally inscrutable, because it sees what reduces the error, and what reduces the error may have nothing to do with what we thought we were teaching it.⁴

Operational Seeing tries to put the player inside that logic. You learn to see agents as vectors in a loss function. The interface teaches you this by rewarding it. And once you've learned, the seeing feels natural and fluent, which is exactly the moment worth examining.

After the documentary, the game deposits you back at the console. You could go again. You know now what you'd be doing. And the system will still score you. And you will still want to do well, if you're being honest with yourself.

Farocki's films often end on images that don't resolve. You have to decide to leave. I want the return to the console to feel like that.

Everything you learned during the session is yours to carry. And the hand, the hand still wants the grid to grow. The hand learned that smoothness feels like competence, that stillness feels like order, that a falling score feels like something you did right.

Nothing about knowing this makes the hand want it less. The training holds. And convergence is such a warm word for what it describes.

War and Cinema: The Logistics of Perception (French orig. 1984; English translation 1989) — the subtitle is doing the same work as the essay above: perception as a supply chain, not a private experience. ↩
From Vertov's 1923 kino-eye manifesto — the theoretical basis for a cinema that sees past the limits of the human eye rather than imitating it. ↩
Farocki restages the Lumière brothers' 1895 actuality film of workers leaving their factory, cutting it against a century of surveillance and industrial footage of the same gesture: bodies exiting a gate under a fixed camera. ↩
The story gets repeated constantly in AI writing, Bridle's book included, but its original source has never been tracked down — it may be closer to a parable the field tells itself than a documented experiment. That it survives anyway, true or not, is its own kind of evidence. ↩