Time: A Word, Governor?

This is normally the kind of thinking we squirrel away on the Peoplenomics.com site.  But the issue here is bigger than a paywall…

There is a fascinating AI story in The Wall Street Journal this week that deserves more than the usual round of “AI is going to kill us all” versus “AI will save everything.” Anthropic researcher Jacob Coxon quit the industry Tuesday because he believes the competitive rush toward self-improving AI could produce systems humans eventually cannot control. The concern is not imaginary. Coxon says people inside the labs are already using terms like “crunchtime” and “endgame,” while other leading researchers have publicly raised similarly severe risk estimates.

Fair enough. But here’s the part that longer-view carbons may recognize: We have been here before. Not with intelligence, certainly, but with machines whose useful output could run away from the humans operating them. Engineering eventually learned what to do about it.

When was the last time you heard about a steamboat blowing up on the Mississippi?  New Tech always has “issues.”

A Word: Governor

Before there were gasoline engines, steam engines discovered an inconvenient fact about power. Give an engine enough steam and it will turn. Give it more steam and it will turn faster. Take the load off suddenly while maintaining the same steam input and it may turn considerably faster than anyone intended. That’s bad when the machine is connected to expensive equipment, and potentially catastrophic when rotating machinery exceeds what its components were designed to withstand.

The answer became the governor. James Watt didn’t invent the underlying centrifugal-governor idea from scratch; similar speed-regulating mechanisms had already been used with mills. But in the late 18th century Watt adapted the principle to steam engines, and governors became standard equipment because their purpose was elegantly simple: maintain useful speed despite changes in load or steam pressure.

Two spinning weights moved outward as engine speed rose, and that movement began closing the steam valve. Less steam slowed the engine. When the engine slowed under load, the weights dropped inward and more steam became available. Think about what had happened: the machine’s own behavior had been turned into feedback controlling the machine. The governor didn’t make the steam engine less useful. The governor is part of what made powerful steam engines practical.

Then Gasoline Learned the Same Lesson

Internal-combustion engines inherited the problem. Especially in stationary engines, maintaining a sensible operating speed mattered more than seeing how fast the crankshaft could possibly turn. One wonderfully primitive solution was the old hit-and-miss engine.

The governor watched engine speed. Too fast? Miss. The mechanism prevented another firing cycle. Speed fell. Hit. Another power stroke. No congressional hearing, no blue-ribbon commission, and no 400-page white paper on existential combustion risk. Just a simple control loop that limited the machine before the machine limited itself by coming apart.

That brings us directly back to AI.

Rated Intelligence — Not Redline Intelligence

Much of the AI race today is still framed in terms resembling an adolescent standing beside a newly rebuilt motor asking, How fast will she go? How many parameters? How large a context? How much compute? How autonomous? Can it code, operate another computer, improve itself, or design the next model?

Wonderful engineering questions. But eventually engine companies discovered that the winner wasn’t necessarily the company whose engine could produce the highest RPM moments before distributing its connecting rods around the county. The useful questions became different: how much horsepower does it produce, at what weight and fuel consumption, how reliably, for how many hours, with what maintenance interval, and at what rated RPM?

In other words, engineering moved from fascination with maximum possible performance toward optimization of useful performance inside a controlled operating envelope. AI is approaching precisely that transition. Its important specification eventually may not be peak intelligence. It may be rated cognitive horsepower: useful work per watt, error rate under unfamiliar conditions, maximum authority envelope, recovery time after failure, cost of contained failure, and something analogous to mean time between failures — mean time between unauthorized actuations.

Those are engineering numbers. “How close are we to superintelligence?” is mostly a campfire story by comparison.

Reading Isn’t the Same as Turning the Throttle

Here’s another distinction getting blurred. An AI reading the public web is one thing. An AI changing the world it is reading about is another. And by “writing” I don’t mean merely generating words on a screen for a human to read.

The boundary that matters is unmediated actuation: changing state in somebody else’s system without a human approving that particular transaction. Deploying a software patch, moving money, opening accounts, changing access permissions, sending executable commands, modifying databases, operating physical equipment, creating cloud infrastructure, or logging into another machine and changing things are not merely forms of reading or analysis.

That’s not the AI reading the shop manual anymore. That’s the AI touching the throttle.

And this distinction isn’t theoretical. Current AI products are already being given access to outside applications and systems, while security concerns are pushing developers toward stronger monitoring, restricted access and shutdown mechanisms. Funny. Sounds suspiciously like a governor.

Put Down a Damage Bond

Here’s one possible direction that may eventually deserve serious consideration. Keep observation comparatively free, but put a price on autonomous actuation. Suppose an AI company wants to deploy an agent capable of altering resources outside its own sandbox. Fine. Post a damage bond.

Think of the old library model. You can walk in and read an irreplaceable book. Maybe you can photograph pages. But if the library is going to hand you the Gutenberg Bible and allow you to leave the building with it, somebody is going to want considerably more assurance.

Same principle. A bot publishing routine weather observations might have an almost trivial action envelope. An AI allowed to modify its own company’s test servers is another level. A system authorized to write into production systems, transfer financial assets, modify other people’s machines, or operate critical infrastructure has an entirely different potential damage envelope.

So make the bond scale with the authority granted. Don’t ask only, How intelligent is the model? Ask instead, How much of somebody else’s world can this model alter without asking first?

That’s an insurable question. And insurance companies have a lovely tendency to ask practical questions technologists sometimes overlook: What can this thing break? How often might it break it? And who’s writing the check?

The Governor Can’t Share the Same P&L

Grok, reviewing an earlier version of this argument, caught an important wrinkle. Watt’s governor worked because the governor wasn’t competing with another steam engine whose owner could increase profits tomorrow by simply disconnecting his governor.

That is much closer to Coxon’s actual concern. The danger isn’t merely that self-improving AI exists. It is that competitive pressure rewards the company willing to operate closest to redline. Company A says, “Our model needs human approval before doing that.” Company B says, “Ours doesn’t.” Guess which one demos better — until something blows up?

That means the strongest governor probably cannot live entirely inside the same profit center rewarded for opening the throttle. Technical controls should be independent of the system being controlled, and perhaps some controls need to be independent of the company benefiting financially from relaxing them.

Independent monitoring, immutable logs, outside auditing, bonding or insurance, hard permission boundaries, and shutdown mechanisms the model itself cannot modify all fit the same general idea. Which leads to a useful design rule: The controller shouldn’t be identical to the thing being controlled — and preferably shouldn’t share the same P&L.

That is less dramatic than “AI extinction.” It is also considerably more engineerable.

Self-Improvement Isn’t Actually the Question

This gets us to the scary phrase in the WSJ story: self-improving AI. Sounds ominous, but it’s still the wrong engineering question. The useful question isn’t simply, Can the machine improve itself?

The better question is: Which variables may it change, over what range, at what rate, with whose money at risk, and which control loop is forbidden from being the thing it regulates?

An AI might be allowed to modify millions of internal parameters. Fine. But not its network permissions. It might discover new software architectures. Great. But another independent system signs the production deployment. It might run millions of experiments inside a simulator. Have at it. Crossing the boundary from simulation into somebody else’s real machine? Miss. Until the independent governor says: Hit.

That’s not suppressing intelligence. It’s controlling actuation.

Finding the AI Power Band

Internal-combustion development eventually stopped being a contest to discover how fast an engine could possibly spin. Designers learned there was a power band: an operating region where the engine delivered the best combination of torque, horsepower, efficiency, temperature and durability.

AI is looking for its power band now. Peak benchmark scores aren’t it. Maximum autonomy isn’t it. Recursive self-improvement isn’t necessarily it. The winning system may instead be the one delivering the largest amount of dependable useful cognition at acceptable energy cost, error rate and external risk.

That is why some of the current AI hysteria strikes me as historically shortsighted. The danger can be real without being metaphysically unique. Steam had runaway machinery. Gas engines had overspeed. Aircraft engines acquired increasingly elaborate control systems. Jet engines eventually got FADEC. Computers discovered thermal throttling. Electrical systems got breakers. Nuclear plants got multiple independent shutdown layers.

Every powerful technology eventually discovers that control architecture is part of the technology. AI isn’t somehow exempt. The industry is simply reaching the stage where raw capability stops being the only interesting number.

Horsepower, Not Redline

So yes, take the warnings from inside the AI labs seriously. The people building these things know considerably more about their behavior than the average keyboard prophet. But don’t confuse concern with inevitability. The historical template says something else happens next.

We stop asking only, How fast can this thing go? And start asking, How much useful work can it reliably deliver without throwing a connecting rod through the internet?

That is where mature engineering begins. Steam figured it out. Gasoline figured it out. Aviation figured it out. AI will have to figure it out, too, because the objective isn’t maximum RPM.

It’s rated horsepower.

Or, in this case, rated intelligence, not redline intelligence.

A word, Governor?

~ure