I’ve written previously about the need to capture and codify emotional context and meaning as we look to build computationally creative AI systems. Today I’d like to address an often overlooked aspect of computational creativity; the subtle but essential difference between examples and ideas.
It’s well known that current LLMs (and other pretrained transformer algorithms) are wonderfully proficient predictors of ‘the next token’ in a series. When given enough training data, they can even discover non-trivial, conceptually abstracted logic from pure language. In fact, pretrained transformers have been around since 2017, and it was a surprise to everyone (even the inventors!) when they suddenly began demonstrating their current depth of knowledge and expertise.
Ghost in the Machine
As far back as 1967, author Italo Calvino was thinking seriously about the role of artificial intelligence in the creative process. Incredibly, he all but predicted the design of today’s LLMs in his essay Cybernetics and Ghosts:
Literature as I knew it was a constant series of attempts to make one word stay put after another by following certain definite rules; or, more often, rules that were neither definite nor definable, but that might be extracted from a series of examples, or rules made up for the occasion — that is to say, derived from the rules followed by other writers.
Italo Calvino: Cybernetics and Ghosts
It’s certainly true – that feeling we get from interacting with AI that provides meaningful and interpretive responses suggests our brains are fairly well modeled by neural net-type systems. But as we’ll see, the syntactic grammar found in text examples is relatively predictable. Things get a bit more complicated when we hope to see novel ideas appear.
Searching for Genius
In many ways, human makers of literature and music follow a mental flow similar to deep neural nets when immersed in their craft. They navigate the creative process through tokenized symbols of meaning, consider influence from the memory of past examples, and constantly make predictions of what should come next.
But we also know people value the distinction between art generated from a statistical mashup from known examples and the inspiring ideas we attribute to artistic genius.
We face a difficult contradiction between today’s pretrained transformer algorithms and the often unconscious artistic processes that result in the longevity and repeatability of highly-valued ideas. Ultimately music’s long-term value is measured through listeners’ interpretation and interaction, and even transcendence of the original maker’s intentions and we want our algorithms to behave accordingly.
Inspiration vs. Perspiration
More than 55 years ago Calvino described a noticeable shift in perspective between ‘continuous’ and ‘discrete’ approaches to the analysis of literary narrative. Unlike some at the time, Calvino felt that the combination of a ‘digital’ and ‘analog’ perspective offered the most convincing theoretical model of our own creative process.
One of the most arduous intellectual efforts of the Middle Ages has only now become entirely real: I refer to the Catalan monk Raymond Lully and his ars combinatoria. The process going on today is the triumph of discontinuity, divisibility, and combination over all that is flux, or a series of minute nuances following one upon the other.
Italo Calvino: Cybernetics and Ghosts
What interested Calvino was the development of a functional perspective in preparation for the coming utopia of intelligent machines. What sort of results might the unification of ‘continuous’ and ‘discrete’ produce? How will we know when the ars combinatoria Renaissance has arrived? Are there suitable tests to help us judge human- vs. machine-generated art?
His conclusion is striking:
The true literature machine will be one that itself feels the need to produce disorder, as a reaction against its preceding production of order: a machine that will produce avant-garde work to free its circuits when they are choked by too long a production of classicism.
Italo Calvino: Cybernetics and Ghosts
Toward a Creative AI
I undoubtedly see a future where machines play an active role in the creative process, challenging conventional notions of authorship and artistic inspiration. And like Calvino, I view the ultimate creative machine as one that is capable of producing avant-garde work because it feels the need to break free from its own trained prediction model of past examples.
As designers of these systems, I encourage us to seek out and capture the romanticized notions of inspiration over the manipulation of emotion through the well-established and familiar means offered by past examples. This requires additions and adjustments to the capabilities of current LLMs and in my view, will open the door to a new world of creativity and artistic exploration.