[Ed. Note: This is part of an ongoing series at The Geek in Review Substack page. – GL]

Leo Huang did not usually come to the twelfth floor, and he never came to it at seven-forty in the morning. So when Cooper looked up from his coffee and found him in the doorway, still in yesterday’s shirt with a laptop held against his chest like a clipboard, he could tell something was going on, and from the look on Leo’s face, probably something bad.

“I think we almost sent a client something we shouldn’t have,” Leo said. “I think the intake agent cleared it, and I think it’s been clearing things it shouldn’t for a while, and I can’t find anyone whose job it is to care.”

Cooper put the coffee down. “Sit. Start at the beginning, slowly. What got cleared.”

Leo set the laptop on the desk and turned it. On the screen was the intake queue, the one the business-intake team and the junior associates worked off of every morning, the inbound non-disclosure agreements and engagement requests that the firm’s intake agent read overnight, classified by risk, and stacked into three colors. Green for standard, clear to send. Yellow for a human look. Red for a partner. The agent drafted the first-pass response too, so that by the time a person opened the item, there was already a reply sitting under it, polite and competent and ready to go out.

“This one came in Monday,” Leo said, and opened it. “Counterparty NDA, mutual, looks like every other one. The agent tagged it green. Standard, clear to send. The response was already drafted. Someone on intake would have sent it this morning, today, because that’s what you do with green, you send it.” He scrolled. “But I was staffed on the underlying matter, so I actually read the thing. And buried in the confidentiality section there’s a non-solicit. Twelve months, their employees and ours. That’s not a confidentiality term. That’s a restrictive covenant riding inside an NDA, and we have a house rule that any embedded non-solicit goes to a partner, because Maya made it a rule after that mess last year.”

“And the agent missed it.”

“The agent didn’t miss it. That’s the part that kept me up. It saw the non-solicit. It just decided it was boilerplate. Because the playbook it’s reading still says standalone non-solicits in NDAs are standard market and don’t need escalation. Which was true. Last spring. Before the rule changed.”

Cooper felt the morning go quiet around him.

“So I went looking for the playbook it’s reading,” Leo said. “It’s pointed at a document in the old KM folder. The one we superseded in March. The current house position, the one with the non-solicit rule, lives in a different place now, and nobody ever told the agent. It’s been reading a dead document with total confidence since the spring. Every green tag it’s handed out since March, it handed out from a rulebook we threw away.”

“How many is that.”

“I don’t know. That’s the other thing.” Leo looked at him. “I went to find out who owns this agent so I could tell them. There’s no one. It was a pilot. I built the clause-extraction piece, the part that pulls terms out, eighteen months ago, but I don’t own the thing it became. Somebody in KM stood up the production version, and somebody used to check it on Fridays, and I went and asked, and the person who checked it moved to the Houston office in the spring. The Friday review went with her. Except it didn’t go anywhere. It just stopped.”

Cooper was already standing, already reaching for the marker he kept by the door.

“Don’t send anything green until I get back to you,” he said. “I’m getting the room.”

• • •

He got the room by ten. Not the firefighting version from the Friday in June when a government had reached across an ocean and switched off a model while everyone watched from their kitchens. The deliberate one, Governance Committee on twelve, door closed, three hours blocked. Nora came up from the administrative floor with her iPad and the look of someone who already suspected she would not enjoy the next three hours. Maya came in between a deposition and a closing, reading glasses pushed into her hair. Jesse joined from his lab in Chicago, a window on the wall screen with whiteboards behind him. And Arthur joined the way he joined everything now, a calm rectangle in the corner, the leather spines of his home study at his back.

Cooper wrote one sentence on the board, not a phrase this time but a full sentence, and stepped away so they could all read it.

Continue Reading When a Trusted AI Agent Goes Wrong and No One is Watching (Beyond the Model)