Economics of AI Slop and productivity
I want to give a general overview of how to get productivity despite of AI slop.
Please watch the following video.
In that video, I show a linear production. Linear production means more input yields more output in a straight line. Its a simplified model.
In business as usual, more input yields more verified output. When AI output feels euphoric compared with business as usual, I get more output for less input. Then extra gains stall into slop that is still better than doing the work myself. After that they stall into pure slop that is not useful. The Fabulous AI Output Frontier is where I hit the target output with less input than business as usual.
The economic knowledge and understanding I have acquired have proved very useful when working with an LLM. Still, I have realized that to make full use of the LLM capabilities. I need an extra skill beyond just domain knowledge.
For instance, when examining the effect of a policy change, my economic knowledge enables me to spot the relevant variables. However, I have to know how to prompt and organize the LLM to obtain meaningful insights or to simulate various scenarios.
There is a great deal of potential in prompt engineering, context engineering, and harness engineering. Prompt engineering means writing the instructions for the model. Context engineering means choosing what text the model sees. Harness engineering means building the setup around the model. I still need a completely different but fundamental set of skills if I am to make use of LLMs.
The missing skill¶
The key ability is the skill of doing research. That skill includes dealing with unknown unknowns. Unknown unknowns are vague problems with vague solutions. I use my knowledge of the subject area. I break a large problem into smaller modular sections. Those smaller modular problems are independent, or at least less closely connected. A module is a separate section of the problem. This practice keeps problem complexity under control. Complexity means the growing load of parts and links.
This skill relates to systems thinking. Systems thinking means seeing a whole as linked parts. In my opinion, the skill goes a bit further than that.
How complexity grows¶
I am using LLMs, so complexity inevitably tends to increase over time. It rises because the number of dependencies between the modular problem sets grows. It rises because of a greater degree of obscurity. Obscurity means the links are harder to see. It rises because the blast radius gets wider. Blast radius means tinkering with one module affects several others.
The key idea is to let complexity increase at a decreasing rate. At the same time, make progress at least at a linear rate. Linear means the gains keep coming in a straight line. If we cannot tame this complexity, the LLM will produce slop.
No loops¶
The main problem that lets complexity grow without limit is circularity. Circularity means the dependencies among modular problem sets run in a loop. To limit the complexity, those dependencies must be acyclic. Acyclic means the links never run in a circle.
I apply acyclic dependencies to modular problem sets. Just as in the case of modular problem sets, if the solution of B depends on that of A and if the solution of C depends on that of B, then the solution of C should not depend on the solution of A. In such situations where circular dependencies occur, it becomes considerably more difficult to untangle the relationships and to identify the independent components.
Join the pieces¶
Once I have those modules, I can use a relatively simple LLM. That model can have a very limited context window. A context window is the text the model can hold at once.
I still need careful synthesis and coordination. Synthesis means integrating the solutions of the subproblems into a coherent whole. In this regard, knowledge of the subject is very useful.
The hard case¶
Yet a large part of the research that we economists carry out involves what is notoriously known as endogenous or circular relationships. Endogenous means the result is set inside the system, not from outside. It is almost always unknown how to divide such problems into independent modular sets.
But some problems can be broken into modules with acyclic dependencies. For those problems, you can be productive by controlling the chaos of the complexity. The need is not speed. It is control of the chaos.
