Who Writes The Sum?

Paul Knaggs writes:

In Gaza, Israeli intelligence officers had a word for the dead before they died. NAZA is an acronym for nezek agavi, Hebrew for collateral damage: the estimated number of civilians the Israeli military considered acceptable to kill. Twenty NAZA meant up to 20 civilian deaths approved to eliminate a single target. For one high-value target, a figure of 500 was approved. Nobody said “women and children”. They said a number, and then they pressed a button.

That word is now the title of a film. NAZA, written and directed by Yuval Abraham and Rachel Szor, opened in British cinemas on 16 October. It is built from interviews with 24 Israeli soldiers, most of them intelligence personnel, describing their roles in surveillance, target selection and strikes. Naomi Klein watched it, and then recorded a short video for her Substack with a message the technology industry would rather we did not hear.

We keep being told to fear the day artificial intelligence slips its leash. Klein’s answer is blunt: it never needed to. “It isn’t rogue,” she says. “It hasn’t gone rogue.” The speculative chatter about killer robots is, in her words, “an elaborate distraction” from crimes already committed and still being committed. What makes her argument powerful is what she refuses to blame. “I’m not saying that it’s the technology that’s to blame,” she says. The danger lies in the marriage of a powerful tool with an ideology that has already decided some lives count for less.

A PROBABILITY, NOT A PERSON 

The film’s sources describe how that marriage works in practice. Intelligence officers detail an AI-assisted system known as Lavender, which identified people for assassination using data gathered from thousands of hacked mobile phones. Klein puts it precisely: targets were generated “based on data patterns, not based on evidence”. A suspicion becomes a score; a score becomes a building; a building becomes rubble with a family inside it.

The Israeli military rejects this account, and its response deserves to be stated fairly. It insists that AI does not decide whether a person or object becomes a target, nor whether a target is struck, and that appearing in Lavender does not authorise a strike. The Guardian, which produced the film, says it stands fully behind it, and points out that the New York Times, Washington Post and Associated Press have also confirmed the use of AI systems and the loosening of civilian safeguards.

But notice what the denial concedes. Nobody disputes that the database exists, or that civilian deaths were priced in advance. The argument is over how many, and how much human judgement stood between the machine’s suggestion and the bomb. That gap is the whole story. One interviewee reaches for the oldest alibi in the book: “My role is entirely technical,” he says.

THE COMPANY IN THE ROOM

Klein demands the names of the technology firms behind this machinery. One name already sits on the public record. In January 2024 Palantir agreed a strategic partnership with the Israeli Ministry of Defence, and its executive vice president Josh Harris told Bloomberg that both sides had agreed to harness Palantir’s technology “in support of war-related missions”. Bloomberg noted that Palantir had recently introduced its Artificial Intelligence Platform, AIP, a decision system able to analyse enemy targets and propose battle plans.

To be exact about it: Palantir has denied developing, in whole or in part, the automated targeting systems used by the Israeli military in Gaza. Lavender is not proven to be its product. But a company that publicly volunteered its technology for a war now under examination for genocide is not a neutral supplier of spreadsheets. It chose a side, and said so.

The then UK Defence Secretary John Healey and Palantir CEO Alex Karp signed a £1.5 billion investment commitment in the UK. And that company is inside our health service. In November 2023 NHS England awarded Palantir the £330 million contract to run its Federated Data Platform. More than half of English NHS trusts now use it. The NHS points out that the contract bars Palantir from commercialising or marketing NHS data, even in anonymised form, and that the platform manages waiting lists and theatre schedules, not life and death. Fair enough, today. The question is not what the software does this year. It is who owns the machinery of knowing everything about every patient, and whose values will steer it when the money runs short.

THE OLD ARITHMETIC

We do not have to imagine how a British state might use an algorithm against the vulnerable. It already has. A Department for Work and Pensions fairness analysis in February 2024 found a “statistically significant outcome disparity” in the machine-learning system used to vet universal credit claims for fraud, with bias by age, disability, marital status and nationality. Claimants were not told, because the department decided they “do not need to know”. The assurance was that a human made the final decision. The humans of Gaza’s targeting rooms made final decisions too.

So the fear in this article is not science fiction. It is history waiting for better software. The idea that a disabled person is a cost to be calculated was once mainstream policy across the civilised world. In 1927 the United States Supreme Court upheld forced sterilisation in Buck v Bell. Nazi Germany’s 1933 sterilisation law put roughly 400,000 people under the knife, and its propaganda posters told citizens how many Reichsmarks a hereditarily ill person cost the national community over a lifetime. From there it was a short administrative step to Aktion T4 and the murder of disabled people in their tens of thousands, by doctors filling in forms. Britain had its own eugenic enthusiasts, many of them respectable progressives who spoke the language of efficiency.

None of them needed a computer. What a computer offers is the thing they lacked: scale, speed, and the comforting fiction that nobody in particular decided anything. Feed a model the logic of austerity, tell it to minimise lifetime cost, hand it genomic data and disability records, and it will produce a recommendation with perfect neutrality. The cruelty will already be in the question.

THE WORST USE COMES FIRST

None of this should surprise us, and nothing in it is new. Bad men will always find the cruellest use for a new tool, and they usually find it first. The first great battlefield use of industrial chemistry was chlorine gas, released over the trenches at Ypres in 1915. The first public demonstration of nuclear fission erased Hiroshima and Nagasaki. The motor car killed thousands on British roads before anyone thought to make drivers take a test, which only arrived in 1935.

In each case, the horror came first and the thinking came after. The Geneva Protocol banned gas warfare in 1925. The Non-Proliferation Treaty tried to contain the bomb. The driving test, the seatbelt and the breathalyser turned the car from a menace into something we hardly think about. Societies do learn. Technologies that began as instruments of death have become, under law and democratic control, instruments of ordinary life.

But the learning has always been paid for in lives, and bad men are only half the story. Gas did not release itself, and the men who released it did not think of themselves as monsters. They were following procedure. The soldier in NAZA who says his role was “entirely technical” is the most dangerous figure in the film, because he is the most ordinary. Evil needs a few people to give the orders and many more to feel they are only doing their jobs. Artificial intelligence is the most powerful tool yet for producing that feeling. Following orders excused nothing at Nuremberg. Following an algorithm must excuse nothing now.

THIS TIME, THE LAW MUST COME BEFORE THE BODIES 

That is why the Nuremberg tribunals refused to accept that following orders excuses anything, and why we must now establish, in law, that following an algorithm excuses nothing either. We have the advantage no earlier generation had: we know the pattern. We have seen what the first use looks like, in Gaza, and we have seen the smaller trial runs closer to home, in a welfare algorithm that singled out disabled people while telling them nothing.

The first test is close at hand. A break clause allows the government to stop Palantir’s NHS contract renewing beyond March 2027, a ministerial decision is due by December, and Prime Minister Andy Burnham has not said what he will do. He should use it and bring the health service’s data infrastructure into public ownership, answerable to Parliament. No company with an undisclosed role in an alleged genocide should hold a British public contract. No decision about a person’s benefits, treatment or entitlement should be made by an automated system without a named human being legally responsible for it. Every algorithm the state uses on its citizens should be published, audited and open to challenge in court.

Klein is right that the decisions remain human. So must the responsibility, and so must the punishment. Every new technology has been used at its worst before it was used at its best; the only choice we have is how many people pay for the lesson. The machine will never go rogue. It will simply do the sum it is given, and the only question that has ever mattered is who writes the sum.