We still can't talk about robot writers

#2/n of the Language Models and Literature Series: The 'autocomplete' fallacy and its consequences

Welcome back to the Language Models and Literature Series. You can read the introduction here. My next two posts in this series will consider from a technical perspective how an AI agent can write a novel, and then will look at how it can write a good novel. Before we get there, I wanted to do some spring cleaning of the concepts we use when talking about language models. 

---

One of Silicon Valley’s favourite hobbies is the misuse of metaphors. If something is bad - no matter what its origin or characteristics - it is called a ‘bug’. If it is good it is called ‘a feature’. It is very common to hear some clever clogs say, in the language of Bayesian reasoning, that they are ‘updating their priors’, which approximately 100% of the time simply means ‘changing my mind’. This is usually harmless, because, to their credit, tech-aligned people understand these figures of speech serve specific, limited purposes. Harmless, that is, until the metaphors escape from Silicon Valley and run amok in normie-land. When this happens, non-technical people are prone to mistaking these half-serious metaphors for rigorous explanations of technical concepts.

One such that has absconded from its captivity and is now at large in the population is ‘autocomplete’, used to describe the inference and training process of language models (LMs). This seems to have been originally employed in a narrowly technical sense, to foreground the idea that the atomic unit of the LM training process is the minimisation of prediction error of the next token in a sequence. 

Yet the metaphor has now become a catch-all slogan about the general behaviour of language models. Last week in the New Yorker, Ted Chiang dedicated prestigious column inches to various speculations about the current and future capabilities of language models. As I was reading I got the sense that the writer was a person for whom being a writer, and cultivating writing as a skill, formed an important part of his self-worth. I could tell he was well-read and had good taste. I also could tell he had - at best - limited experience working hands-on with transformer architectures. From this mixture of intellectual ambition, human protectiveness, and technical limitation, I knew - just knew! - that the ‘autocomplete’ metaphor would pop up. And there, towards the end, it did:

More than our ability to solve algebraic equations, our ability to cope with unfamiliar situations is a fundamental part of why we consider humans intelligent. Computers will not be able to replace humans until they acquire that type of competence, and that is still a long way off; for the time being, we’re just looking for jobs that can be done with turbocharged auto-complete.

Look, it’s no fun being the language police. People should be creative and transgressive and push language to its limits. However, if, in doing so, they are using language as part of a rhetorical strategy which is combatively making untrue claims, then they expose themselves to counter-insurgency measures. I do not do this willingly, but I was drafted into this service, and will carry out my duty.

If I try to run with Chiang’s point, he seems to be calling the capabilities of LMs ‘turbocharged autocomplete’ to argue that these models have a narrow ‘understanding’ of the world, because they can only ‘think’ one token at a time. He also is implying that because they are trained only text data from the internet then they do not exhibit generalisable capabilities that can be applied to novel and surprising situations. 

The ‘one-token-at-a-time’ thesis is often used as a gotcha by those semi-literate about LMs, but it doesn’t quite hold. You can easily prompt your model to do things like end a sentence with a particular phrase, or tell a story that ends in a certain way - and there are more complicated token lookahead behaviours they are getting better at all the time. This is because LMs are able to model relationships between different words in a sequence both backwards and forwards. During training, a sentence (for the sake of argument) is passed to the model with one token (word) masked, and the model is asked to predict what the value of the masked tokens might be. But in this process, it is not simply looking at all the tokens before the masked token, it also includes representations of the relationships between the masked tokens and all the tokens which come after it in the sentence.

The second implication of the autocomplete metaphor, that language models cannot represent general concepts and thus are unable to deal with unknown situations, is more contentious, but baldly stating that LMs are incapable of this is simply not being sensitive enough to our current understanding of the technology. Research has suggested that in a number of domains LMs are capable of using and representing concepts not in their training sets, such as human emotion and spatio-temporal ideas. These results are not unequivocal, and of course involve significant ambiguity regarding what does and does not define a ‘concept’, but what they do mean is that we cannot rule out the ability of LMs to generalise and infer from any number of non-linguistic concepts or capabilities. We know, for instance, that you can train LMs with no knowledge of the rules of othello on simple lists of moves, and they will construct a representation of the game pretty much exactly resembling the actual rules. 

You could even go all in, and simply say that there is no great chasm between autocomplete and human understanding. I won’t go into this here, but Geoff Hinton has claimed that “by training something to be really good at predicting the next word, you’re actually forcing it to understand. Yes, it’s ‘autocomplete’—but you didn’t think through what it means to have a really good autocomplete.”

Technically speaking, the autocomplete metaphor misrepresents the mechanisms that LMs use and makes them seem far more limited than we know them actually to be. And given that autocomplete is familiar to most people, and we are comfortable with it, it implies a level of certainty regarding these models that we fundamentally do not have. While we don’t know what all the capabilities and behaviours of LMs really are, we certainly know that they are not limited to autocomplete. 

(There is a tangential point here: the autocomplete system in your email and maybe on your phone keyboard will likely be running inference across a transformer model just like the LMs we’re talking about. So calling an LM ‘autocomplete’ is just a tautology.)

You could say that this whole discussion is just playing out a pattern seen time and time again in the adoption of technology. Sceptics make their arguments using popular and accessible concepts, and then in return those with technical understanding accuse sceptics of not knowing how the thing works. The difference here I think is that when it comes to LMs you often find that the capability sceptics speak with greater certainty than the AI builders. Once you spend any time working with language models, no matter what size your p(doom), it becomes very hard to make grand statements about the ultimate limits of the technology, both as it exists now and as it might develop. 

It’s in this kind of uncertain situation that wishful thinking creeps in. Because the current state of affairs with regards to AI capabilities is not fully known, there isn’t an agreed set of facts upon which you can pass judgement, saying ‘X capability of AI is good’ and ‘Y capability is bad’. And in the absence of such agreed facts, people must expend a lot of energy trying to describe the world in the hope that those descriptions might one day become the accepted facts. But what seems to happen is that people still want to pass judgement while doing this descriptive work, because people always want to scratch the itch of promulgating their values. This means that descriptions get skewed because they don’t just represent what you think is the case, but what you want to be the case. The autocomplete metaphor is simply one particularly widespread example of how people can use flawed descriptions to smuggle in ideas of how they the world to be: a world in which language models are fully legible, and on the things we most care about - namely creativity and linguistic beauty - ultimately less than we are.

It is difficult entirely to expunge the optative quality of language: that a description you use is not just the one that is accurate, but is also the one you want to be accurate. Yet we must be able to dwell in uncertainty just a little bit longer — simplification is not always a virtue. In this regard, maybe we can learn something from the way LMs use language. When they predict the next token in a sequence, they generate a series of probabilities of what it could be. Instead of the description of the world they offer being an inevitable fact, we can see that if we tweaked a few dials, or took the second-most probable instead of the first, we’d get an entirely different description.