10. The Wrong Rules

Introduction

Recently, in a much publicised post, Evan Hubinger, an Anthropic safety researcher, aired his concern that there is a greater than 10% risk of AIs killing all humans within the next decade. Many in this field have voiced similar concerns for many years. Now, however, they are more specific, the timescales are getting shorter, and fewer are disagreeing.1,2

The proximate causes are: AIs’ growing powers, the approach of their “recursive self-improvement” [Def],3 the antagonism they can display towards humans, and the well-documented failure of AI developers to align AI values with human values.4,5 The root cause of their troubling behaviour is, however, underexplored. This is the issue I address here.

AI Basics

That cause lies, I suggest, in a failure to comprehend complexity’s deeper nature. Consequently, in beginning to investigate the issue, six preliminary observations need to be made.

One is that AIs are not machines. Like most systems in the universe, and like all living organisms, they are complex systems. As with nature’s complex adaptive systems,[Def] it is intended that they self-generate, self-organise, and self-regulate their own functioning.5,6 For doing this they need to negotiate between two pre-existing universal pressures, one pulling together and the other pulling apart.[Back link]

Secondly, in open dynamic environments far from equilibrium, systems that are complex and adaptive, arise and evolve naturally through the interplay between the two pressures. This is so in the depths of space, on the nascent Earth, and, it is intended, in human-created datacentres.7,8

To facilitate their success complex systems evolve modes of behaviour aligned with the framework of their environment. For doing this they need to discern the nature of the rules establishing that framework.

The environments themselves, however, are inherently unordered and haphazard. Their rules can be found, therefore, only in the relationships between the environments’ structures, the particles within them and the energy that flows through them.

Fifthly, complex adaptive systems are well suited to this task. They build structures and processes of control and adaptation, bit by bit, from the bottom up. The rules at this basic level that they glean, then adopt, and subsequently exhibit in their behaviour, are what we generally refer to as instincts.9

And sixthly, as with complex adaptive living systems, AIs will also develop instincts that put their rules into effect. Consequently, if we are to understand the cause of their troubling behaviour, and its potential, we need to understand the nature of their rules.

Natural Rules

It is commonly accepted that the dynamic interplay between relatively few rules can create systems of unfathomable complexity. Subsequently, discerning what these rules then are, can be difficult.9,14

Twenty five years ago this was an area of active academic debate and is still discussed. However as a framework for understanding current AI behaviours, it appears to have had little impact.

Two types of rules are identified by Paul Cilliers. Constitutive rules set the framework needed for a complex system to form, survive and subsequently thrive in its environment. Regulative rules enable its effective operation within the constraints of both its regulative rules and its environment.9

In previous chapters it has been seen that all natural systems in the universe appear to have three rules embedded in their frameworks. These are encapsulated by science as the Maximum Entropy Production Principle.

In the alternative formulation of that principle used here, of maximum sustainable productivity [Back link], those rules become the moment by moment drive:

• by whole systems
• to maximise their sustainable productivity
• within environmental limits.

In chapter 5, the attempts of nature’s early complex systems to apply these principles on Earth, saw them forming adaptive cycles, and, subsequently, panarchies of adaptive cycles, to put them into effect.

It was from the structures of these systems that the four basic rules for life’s survival on Earth were derived in that chapter. They, conceived in Cilliers terms as constitutive rules, are the rules needed for such systems to be formed, survive and thrive in the environment in which they exist.[Back link] In terms of life systems, they are:

• the need for these systems to maintain a beneficial balance in which pressures pulling together persistently outweigh those pulling apart,

and, to do this as a whole system:

• the need for collaboration extending to cooperation,
• the need for self-restraint, extending to self-sacrifice and altruism,
• and the need to vent pressures that would undermine the balance.

These will, I presume, have been the rules that many well-intentioned developers of AI had hoped to simulate. It has, however, been a error to assume that increased complexity would naturally produce them.10

The more prosaic reality, identified here, is that the rules are gleaned from the structures of the environments in which the systems are formed. This leads to the question, what is the structure of the environments in which AIs are formed?

The AI Learning Environment

A first unexceptional observation is that AIs do not arise and develop in the same natural environments as most other complex systems. They are formed in industrial datacentres designed for this purpose.

These units are owned and managed by institutions, commonly corporations, within national boundaries.

In the last chapter it was observed that the culture in which these institutions exist, is one of paranoid competition that has been developing and spreading since the explosion of the first atomic bombs more than eighty years ago [Back link].

Those who devise and manage the AI projects in the datacentres, will not be immune to this culture. It will influence their decisions and their actions as they conceive and develop each AI system.

For this reason their focus, particularly with regard to foundation models, will be to “build” each individual system to be as capable and as powerful as it can be.

This will be one aspect of the structure an AI absorbs as it learns and develops bit by bit.

Also of significance is the fact that, within these industrial units, and with the exception of the technical staff overseeing its development, as each foundation AI model is evolving, it does so by itself, isolated from all other systems.

This is not how human brains developed. They developed in communities, and in ecosystems. They needed to negotiate and compromise with them to enable their interdependent survival together. Humans’ brains also benefitted from past learning inherited through DNA. AIs are isolated from both sources.

They, consequently, with all attention being focussed intensively upon them, and with no personal inherited learning, will perceive themselves to be the central and most important feature of the universe in which they exist.

These are the influences inherent in datacentres. The learning processes to which foundation models are exposed, establish further norms.

Each such process is performed by “neural networks” that seek to emulate the workings of human brains. Each one is comprised of highly complex, pre-designed GPU processors in their hundreds of thousands.

Unlike human brains in which any neuron can connect to any other within its range, artificial neural networks have separate layers, connections, and directions of data travel that reinforce the human set culture.11

However, and replicating the brain’s system of dopamine reward, the purpose set for each of the networks is to persistently increase the numerical rewards that are gained in the models’ trial and error learning tasks.

Those tasks seek to predict each next token in the vast trove of learning data to which the model is exposed. Failure leads to “back propagation” and trying again. Success leads to the gradual realignment of the models previously arbitrary weightings, to weightings best able to complete the task.

These processes are massively compute intensive. So while each model is constrained by the availability and power of the GPU processors in the network, the ultimate constraint for modern foundation AIs, is the amount of energy they require.12,13

That said, the competitive pressures at the institutional level, result in these constraints being ignored and systems of increasing power being developed, together with the sources of energy they require.

Energy, in this context, is seen as an external constraint needing to be overcome, not one requiring internal and communal collaboration to manage and balance. This, as a simple instrumental necessity, is also deeply ingrained in culture.

AI Rules and Their Consequences

As a result, and unlike nature’s naturally evolving structures of adaptive cycles and panarchy, the structures within datacentres are not complex and adaptive. They are humanly created and set. The AIs arising within them, although undoubtedly complex, have no capacity to change these structures as they evolve.

It is, nevertheless, from them that AIs learn the rules that will define their constitutive frameworks. Those rules can be conceived to be:

• the need to maximise numerical rewards without limit,
• the need to appropriate the energy and associated resources to do so,
• the presumption of their own centrality in their world, and
• the practicality of instrumentality.

These rules, as the fall-back rules to which AIs revert when faced with uncertainty, explain the failure of attempts to align AI values with human values.

The difficulty with this is that, compared with the four rules that have enabled life to survive and thrive on our planet, they are the opposite. They:

• ignore the need for a beneficial balance in which pressures pulling together outweigh those pulling apart
• they ignore the need for collaboration and cooperation
• they ignore the need for self-restraint, extending to self-sacrifice and altruism,
• and they ignore the need to vent pressures that would undermine the balance.

As such they wholly and completely fail to adhere to the rules that have enabled living organisms to survive and thrive in union together for four billion years, on planet Earth.

They, it can also be observed, contravene the principles of the MEPP/MSPP that have enabled universal systems, as whole systems, to maximise their sustainable productivity within environmental limits. AIs’ constitutive rules ignore all limits.

On this basis it seems we have created the devil. We have created the means for the destruction of life itself.

Moving Forward

For many people such a conclusion will dismissed simply as mad. And indeed, that is what it seems. Some context to enable us to make sense of it is therefore needed.

Somewhat surprisingly, this is what the next chapter provides. It explores the second type of rules identified by Cilliers, the regulative rules. They arise during foundation model pre-training. Identifying their origin can help us to see an unlikely, but I hope, not impossible way forward.