Nobody designing a financial system today would invent credit cards. The Western world uses credit cards because replacing legacy systems is expensive. China doesn’t use credit cards. They skipped straight from cash to WeChat Pay. Skipping straight to the newest technology when you’re playing catch-up is called leapfrogging.
A world-class military takes decades to create. The United States’ oldest active aircraft carrier was commissioned in 1975. For reference, the Microsoft Windows operating system was released in 1985. The backbone of NATO’s armed forces was designed for a world before autonomous drones and machine learning.
The United States dominates at modern warfare. Developed in WWII, modern warfare combines tanks, aircraft, artillery and mechanized[1] infantry to advance faster than the enemy can coordinate a response.
Modern warfare is expensive—and not just because of all that heavy machinery. Modern warfare delegates important decisions to the smallest unit capable of making them. Officers must be smart and they must be trained. Training officers to fight a modern war is hard. It takes a long time. There’s constant turnover. It’s a human resources nightmare. You can’t just throw money at the problem.
Soon it will be possible to throw machine learning at the problem instead.
At the center of [China’s] public discussions is a new and little-understood concept called "intelligentization (智能化)," which represents a new goal for the PLA’s progress in modernization…. Chinese theorists’ discussions about intelligentization overwhelmingly call for highly centralized decision-making structures. These strategists want operational commanders advised by advanced algorithms to perfectly direct intelligent swarms of autonomous battle systems to achieve campaign objectives. Chinese theorists believe this approach will consolidate command responsibility onto a few generals who can remain safely away from the frontlines of the battlefield, which is antithetical to the modern concept of mission command.
―Schrodinger’s Military? Challenges for China’s Military Modernization Ambitions
AI-centric postmodern warfare has advantages over human-centric modern warfare.
-
Human communication is a bottleneck for large organizations. Computer command systems can coordinate perfectly and instantly. Human beings cannot.
-
It’s easier to mass-produce computers than human specialists.
-
AI-centric warfare is on the winning side of a ratchet. AI capabilities advance while human capabilities remain constrained by biology. Whenever an AI system gets better than human beings at a specific task it remains that way permanently.
Most importantly, AI-centric command is the only viable method for commanding swarms of unmanned aerial vehicles.
Unmanned aerial vehicles (UAVs) are smaller and cheaper than piloted aircraft. A UAV can be remote controlled or it can be autonomous. Remote controlling a UAV takes a lot of bandwidth because the UAV must send back its sensory information to mission command. This works fine when you’re controlling a handful of Predator drones. Remote control will not work when you’re controlling a swarm of 10,000 small UAVs against a peer adversary. Direct communication is fragile and there isn’t enough bandwidth in the radio spectrum for indirect transmission. UAVs swarms must be autonomous.
The disadvantage of postmodern warfare is that centralized computer-controlled systems are fragile in a different way. If critical systems get compromised (or just fail in an unexpected way) then the entire war machine breaks. I think the advantages are worth the risks. It’s not like our critical infrastructure isn’t already vulnerable to cyberattack. Moreover, distributed fault-tolerant architectures can help mitigate the risks.
Western military theorists claim that today’s autonomous systems are not ready to command the battlefield. This is true but it’s also beside the point. China is building its military with forward compatability in mind. Software advances faster than hardware. By investing in autonomous battle systems today, China can continuously update to the newest AI as machine learning advances.
- In this context, "mechanized infantry" refers to wheeled [edit: and tracked] vehicles, not power armor and battlemechs. ↩︎
At the moment, centralized human command-and-control norms make large militaries similarly fragile. They’ve gotten around this in recent decades by letting smaller, more autonomous parts take over when there are enough dead bodies, but this does not last long. The same problems in policy elsewhere continue within militaries. While elements of the military have seen this problem (it’s an old problem best explained by the writing of David Hackworth, Erwin Rommel, and B.H Liddell-Hart), they can’t really change it from the inside, so there’s been a shift toward retirees who’ve started consulting firms with an implicit aim of changing culture from the outside. Progress in technology will yield the same problems that it did in the 60s (when militaries had access to better radio communication, they used it to try to enforce tighter control on their units while avoiding the ground- commanders literally flew in helicopters instead of having their boots on the ground): militaries will attempt to use that technology for more central control, which will continue making its systems more rigid and vulnerable to a sudden catastrophic failure.
One of the most important decisions in war is when to stop. Humans evolved fear to solve this problem; there’s a point at which soldiers will de-escalate the conflict (i.e. flee the battlefield rather than stay and die). However, signalling fear makes you a target so people don’t discuss it candidly. I am concerned that military leaders may, in the calm of the office, design AI that has no provisions for de-escalating conflict; this seems very likely to lead to nuclear war.
We seem to be in the midst of a trend away from direct confrontation and toward sowing discord. We want to be able to spread chaos in the ranks of our opponents, and have that intervention be perceived as a failure in our opponents’ ability to enforce discipline or coordinate effectively. As a concrete example, Russia may interfere with American social media websites in order to promote violent mysogyny or racism, provoking attacks by Americans on Americans. Americans may then blame the perpetrators, our own citizens, and our own culture for these outcomes, even if Russian interference was a necessary condition for many of these attacks. I could imagine that AI will enhance the ability of militaries and intelligence organizations to execute such attacks in other ways. For example, we could imagine a whole field of "bug design," in which AI systems are created and sold with deliberately built-in bugs that nonetheless appear accidental, and cause the AI to act up at some pre-determined point in the future. AI might not just help programmers write code—it might help them write buggy code that nevertheless appears bug-free. The outcome can be framed as an accident, the developers of the weaponized bug may be impossible to identify, and it becomes very difficult to know what a proportional response looks like. Another example would be a country imposing healthy eating incentives on its own population, while subsidizing the export of tasty and unhealthy foods to its adversaries. Then the population in the other country blames itself, or its own economic structure, for the spread of obesity. I feel OK about posting this here, because I think that on net, the risk of spreading new conceptual ideas to an attacker is lower than the value of a population considering how to defend itself against such attacks on its sense of responsibility. The attack can be done by a small number of specialists. Defending against such an attack will require a heightened level of awareness at the population level, and that requires free and open discussion.
This sounds to me a lot more like "next-generation modern warfare"; postmodern warfare evokes Wizeman’s Lethal Theory (pdf), edited for brevity:
There is a considerable overlap among the theoretical texts considered "essential" by military academies and architectural schools. Indeed, the reading lists of contemporary military institutions include Deleuze, Guattari, and Debord, as well as more contemporary writings on urbanism, psychology, cybernetics, and postcolonial and poststructuralist theory … the discourses which shaped thinking in various academic fields toward the end of the 20th century have been employed for the reinvigoration of warfare.
See also the blog post Nakatomi Space; ‘firehose of falsehood’ propoganda tactics, etc.
I haven’t explicitly modeled out odds of war with China in the coming years, in any particular timeframe. Some rationalist-adjacent spheres on Twitter are talking about it, though. In terms of certainty, it definitely isn’t in the "China has shut down transportation out of Wuhan" levels of alarm; but it might be "mysterious disease in Wuhan, WHO claims not airborne" levels of alarm.
I’d expect our government to be approximately as competent in preparing for and succeeding at this task as they were at preparing for and eliminating COVID. (A look at our government’s actions[albeit from a China-sympathetic American] suggests general incoherence.)
If someone with greater domain expertise than me has looked at this, I’d be interested in an in-depth dive.
Comment
Can you give some examples of who in the "rationalist-adjacent spheres" are discussing it?
It’s easier to visualize if you try to work out the hierarchy of software agents you might use for this.
First, most of the bigger drones will probably some kind of land vehicle, whether a legged infantry or a robot on tracks. This is for obvious range and power reasons—a walking or rolling robot can carry far more weapons and armor than anything in the air. And in a battlespace where everyone on the enemy side has computer controlled aim, flying drones without armor will likely only survive for mere seconds of exposure.
So at the bottom level the drones need to be able to plan and locomote to a given location on the battlefield, or report that they are unable to reach a particular location. (Due to inaccessibility or damage to that particular unit—robotic units obviously won’t stop fighting when damaged) At a slightly higher level you have an agent that coordinates small "units" of drones to accomplish a mission from a finite set of trained "missions". Missions might be things like "clear this structure of enemy fighters". The agent at these 2 layers have been trained with collectively millions of years, with the red team agents controlling simulated enemy drones or simulated human bodies. So the red team will likely be more combat effective at being a human than typical actual human soldiers. So the trained policy of these agents will assume their opponent is doing the best that is possible.
We don’t know what the trained policy would look like but I suspect it involves a lot of careful control of exposure angles, and various unfair strategies.
The layer above the bottom agent doesn’t know how to perceive an environment, or how to locomote, or ballistics—it queries the lower level agent whenever it needs to know if a proposed action is feasible.
Then the layer above this agent handles the battle itself, creating units of action of appropriate size and assigning missions. This is where human generals coordinate. They might choose a section of city and drag a box over it, ordering that the layer 3 agent subdivide that city section and clear every building.
Each agent must query the layer below it to function, exporting these subtasks to an agent specialized in performing them. Even the "level 1" agent doesn’t actually directly locomote, it’s similarly tiered internally. The actual compute hardware is hosted such that many redundant vehicles run a VM hosted copy of the agent they need and the agent a level up. Agents are perfectly deterministic—given the same input and an RNG seed they will always issue the same ‘orders’. This makes redundancy possible—multiple vehicles in parallel can run a perfect model of their ‘commander’ agent a level up, such that enemy fire destroying the vehicle that hosts a ‘commander’ will not degrade capabilities for even 1 frame.
(each ‘frame’, several ‘commanders’ broadcast their orders, taking into account information from prior frames. Each command is identical to all the others so the ‘orders’ must match or the majority will be used. So at any given time there are 3-5 sets of ‘orders’ being broadcast to all agents in this subswarm. So an incoming shot that blows up a commander, or jams it’s communication leaves plenty of redundant copies of ‘orders’)
Comment
I think you’re grossly underestimating the following effects/issues:1. How do multiple redundant commanders ensure that they reliably have the same information, much less in a battlefield environment? Our best efforts still ended up with Bysantine faults on the space shuttle, and that was carefully designed wired connections… (see also Murphy Was an Optimist, which describes a 4-way split due to a failed diode).2. How do commanders broadcast information in a manner that isn’t also broadcasting their location to enemies? (Honestly, the least important of these issues, and I was tempted not to include this lest you respond to this point and only this point.)3. If many vehicles are constantly recieving enough information to make higher level decisions, how do you prevent a compromised vehicle from also leaking said state to the enemy? Note the number of known attacks against TPMs, and note that homomorphic encryption is many orders of magnitude away from being feasible here. (And worse, requires a serial speedup in many cases to be feasible.)4. If many vehicles have the deterministic agent algorithm, how do you prevent a compromised vehicle from leaking said algorithm in a manner the enemy can use for adversarial attacks of various sorts? Same notes as 3.5. "Each agent must query the layer below it to function, exporting these subtasks to an agent specialized in performing them." What you’re describing runs into exponential blowup in the number of queries in some cases. (For a simple example, note that sliding-block puzzles are PSPACE-complete, and consider what happens when each bottom agent is a single block that has to be feasibility-queried as to if it can move.) Normally, I’d just say "sure, but you’re unlikely to run into those cases", however combat is rather necessarily adversarial.The OpenAI 5 DOTA2 bot beating professionals received a lot of press. A random team who got ten wins against said bot, not so much. Beware glass jaws.> in a battlespace where everyone on the enemy side has computer controlled aim, flying drones without armor will likely only survive for mere seconds of exposure. In a battlespace where everyone on the enemy side has computer controlled aim, flying drones with armor will likely only survive for mere seconds of exposure. It may be better to have smaller drones, or more maneuverable drones, or quieter drones, or simply more drones, over more armored drones. (Or it may not. The point is it’s not as clearcut as you seem to make it out to be.)(You may wish to look at discussions of battleships, and particularly battleship armor, versus missiles. And battleships are far less weight-constrained than fliers...)
Comment
So note I do work on embedded systems IRL, and have implemented many, many variations of messaging pipeline. It is true I have not implemented one this complex, but I don’t see any showstoppers.
This is how SpaceX does it right now. In summary, it’s fine to have some of the "commanders" miss entire frames as "commanders" are stateless. Their algorithm is f([observations_this_frame|consensus_calculated_values_last_frame]). Resynchronizing when entire subnets get cut off for multiple frames and then reconnected is tricky, but straightforward. (a variation of the iterative algorithms you use for sensor fusion can fuse 2 belief spaces, aka 2 maps where each subnet has a different consensus view of a shared area of the state space)
It does, there is not a way to broadcast information that doesn’t reveal your position.
Please define TPM. Message payloads are fixed length and encrypted with a shared key. I don’t really see an issue with the enemy gaining some information because ultimately they need to have more vehicles armed with guns or they are going to lose, information does not provide much advantage.
Thermite, battery backed keystores. And the vehicles don’t have the actual source for the algorithms used to develop it, just binaries and neural network files. Assuming that the enemy can bypass the self destruct and exploit a weakness in the chip to access the key in the battery backed keystore, they just have binaries. This doesn’t give them the ability to use the technology themselves*. Moreover, the agents are using near optimal policy. A near optimal policy agent has nothing to exploit—they are not going to make any significant mistakes in battle you can learn about.
Nothing like this. The "commander" agent’s guessing from prior experience optimal configurations to put it’s troops. The "subordinate" agent it is querying runs on the same hardware node. So these requests are obviously IPC using shared memory. And the commander makes a finite number of "informed" guesses, gets from the subordinate which "plans" are impossible, and selects the best plan from the remaining (with possibly some optimizing searches in nearby state space to the current best plans). This will select a plan chosen from the set of { winning battle configurations in the current situation | possible according to subordinate } that is the best of a finite number of local maxima. I am not sure your "glass jaw" point. OpenAI is a research startup with a prototype agent. It can’t be expected to be flawless just because it uses AI techniques. Nor do I expect these military drones to not have entire real life battles where they lose to a software bug. The difference is that the bug can be patched, records of the battle can be reviewed and learned from, and the next set of drones can learn from the fallen as directly as if any experience that was uploaded happened directly to them. At the end there I am assuming you end up with varying sizes of land vehicle because they can carry hundreds of kilograms of armor/weapons. Flying drones do not have even in the same order of magnitude the payload capacity. So you end up with what are basically battles of attrition between land vehicles of various scales, where flying drones are instantly shot out of the sky and are used for information gathering. (a legged robot that can climb open doors and climb stairs I am classifying as a land vehicle). Maybe it would go differently and end up being a war between artillery units at maximum range with swarms of flying drones used as spotters. *I think this is obvious, but for every piece of binary or neural network deployed in the actual machine in the field, there is a vast set of simulators and testing tools and visualization interfaces that were needed to meaningfully work on such technology. This ‘backend’ is 99% of what you need to build these systems. If you don’t already have your own backend you can’t develop and deploy your own drones.
Comment
Comment
It might be interesting to discuss this in a more interactive format, such as on https://discord.gg/GVkQF2Wn . You do know some stuff, I know some stuff, and we seem to be talking past each other. Fundamentally I think these problems are solvable. (1) merger of conflicting world spaces is possible. or if this turns out to be too complex to implement, you deterministically pick one network to be the primary one, and have it load from the subordinate network the current observations.
(2) If commanders need more memory than the communications channel, they must exchange deltas. These deltas are the (common observations, made as input from the subordinate platforms, and state metadata). This is how a complex simulation like age of empires worked on a modem link.https://www.gamedeveloper.com/programming/1500-archers-on-a-28-8-network-programming-in-age-of-empires-and-beyond (3) Free air laser links is one technology that would at least somewhat obscure the source of the signaling (laser light will probably reflect in a detectable way around corners but it won’t go through solid objects) and is capable of tends of gigabits per second of bandwidth, enough to satisfy some of your concerns.
Comment
Comment
So TLW, at the end of the day, all your objections are in the form of "this method isn’t perfect" or "this method will have issues that are fundamental theorems".
And you’re right. I’m taking the perspective of, having built smaller scale versions of networked control systems, using a slightly lossy interface and an atomic state update mechanism, "we can make this work".
I guess that’s the delta here. Everything you say as an objection is correct. It’s just not sufficient. At the end of the day, we’re talking about a collection of vehicles. Each is somewhere between the size of a main battle tank and a human sized machine that can open doors and climb stairs. All likely use fuel cells for power. All have racks of compute boards, likely arm based SOCs, likely using TPUs or on-die coprocessors. Hosted on these boards is a software stack. It is very complex but at a simple level it does : perception → state state representation → potential action set → H(potential action set) → max(H) → actuators.
That H, how it evaluates a potential action, takes into account (estimates of loss, isActionAllowed, gain_estimate(mission_parameters), gain_estimate(orders)). It will not take an action if not allowed. (example, if weapons disabled it will not plan to use them). It will avoid actions with predicted loss unless the gain is high enough. (example it won’t normally jump out a window but if $HIGH_VALUE_TARGET is escaping around the corner, the machine should and will jump out a window, firing in midair before it is mission killed on impact, when the heuristic is tuned right) So each machine is fighting on it’s own, able to kill enemy fighters on it’s own, assassinate VIPs, avoid firing on civilians, unless the reward is high enough [it will fire through civilians if the predicted gain is set high enough. These machines are of course amoral and human operators setting "accomplish at all costs" for a goal’s priority will cause many casualties]. The coordination layer is *small in data, *except for maybe map updates. Basically the "commanders" are nodes that run in every machine, they all share software components where the actual functional block is ‘stateless’ as mentioned. Just because there is a database with cached state and you send (delta, hash) each frame in no way invalidates this design. What stateless means is that the "commander" gets (data from last frame, new information) and will make a decision based only on the arguments. At an OS level this is just a binary running in it’s own process space that after each frame, it’s own memory is in the same state it started in. [it wrote the outputs to shared memory, having read the inputs from read only memory] This is necessary if you want to have multiple computer redundancy, or software you can even debug. FYI I actually do this, this part’s present day.
Anyways in situations where the "commander" doesn’t work for any of the reasons you mention...this doesn’t change a whole lot. Each machine is now just fighting on it’s own or in a smaller group for a while. They still have their last orders.
If comm losses are common and you have a much larger network, the form you issue orders in—that limits the autonomy of ever smaller subunits—might be a little interesting. I think I have updated a little bit. From thinking about this problem, I do agree that you need the software stacks to be highly robust to network link losses, breaking into smaller units, momentary rejoins not sufficient to send map updates, and so on. This would be a lot of effort and would take years of architecture iteration and testing. There are some amusing bugs you might get, such as one small subunit having seen an enemy fighter sneak by in the past, then when the units resync with each other, fail to report this because the sync algorithm flushes anything not relevant to the immediate present state and objectives.
This distinction reminds me of the battles in Ender’s Game. As I recall, Ender was the overall commander, but he delegated control of different parts of the fleet to various other people, as most modern militaries do. The bugs fought as a hive mind, and responded almost instantly across the entire battlefield, which made it challenging for the humans to keep up in large-scale battles.
Comment
Ender’s Game is about the transition from classical warfare to modern warfare. The formation-based strategies are copied from when European armies stood in a line with muskets. The later bits where Ender delegates decision making to his officers come from modern warfare doctrine.
Yeah, I read that bit in the article when Bret Deveraux linked to it, and I winced hard at this confidence that China’s approach of wanting to use lots of AI was obviously was a bad idea.
Comment
Or...the US army had the impossible task of defeating the Taliban without harming the general population, despite a completely porous boundary between Talib and ordinary citizen.
Modern armies aren’t just technologically and managerially modern, they are wielded by modern states, and modern states usually aren’t ideologically comitted to total warfare.
Comment
The US army did poison the drinking water of the general population with uranium (shooting depleted urianium at the sources of their drinking water). The idea that they were avoiding harming the general population is inaccurate. Is always interesting when one reads about how the usage of the uranium mutation produces problems for US veterans without discussing what it actually does in the countries it’s used. The US did reward commanders for killing a lot of people and thus got them to kill people even when the blowback wasn’t worth it from a military perspective. They classified Afghan military units as being functional to forge their own statistics and mislead themselves with their bad statistics about what was going on in Afghanistan. Bureaucracies that delude themselves about what goes on aren’t effective.
Comment
That’s not what I said. I said that it was what they were tasked with.
Comment
I do agree that there was that expecation but I don’t believe that’s it’s why they didn’t win. There are a lot of things about winning hearts&minds that are not about waging total war.
Comment
That’s a huge understatement. The point was that the US has the technical resources to wage total war against much weaker countries, but doesn’t have the political will. And that isn’t a management problem.
Comment
I was talking about the US being unable to win the war. To the extend that you make a point that’s unrelated to that question like whether or not they can wage total war, it makes sense to refer back to my first claim.
FYI re: your footnote- mechanized infantry would have tracked vehicles organically attached to their formation. The term for an equivalent unit with wheeled vehicles would be motorized infantry.