For some time now, figures in the AI industry have been saying things along the lines of: “We want to build AI that augments, but doesn’t replace, humans.” I think the idea is great. But so far, we haven’t seen much evidence that expressing pro-augmentation views has led to firm commitments about how to actually make this happen. Perhaps these statements are shaping internal priorities (more compute for interface experiments, more product work on copilots, more attention to human-AI workflows, etc.), but we can't be sure.
Allocating more resources toward interface-focused research will likely produce tools that are better at augmentation. However, I do not think that making AI systems better at augmenting workers will do much, by itself, to prevent substitution or replacement. In fact, augmenting systems may accelerate replacement in domains that are currently data scarce.
Anytime a worker uses an AI system to perform some task requiring what we might now think of as "human stuff" (judgment, taste, domain knowledge, private context, etc.) they produce rich workflow traces and outcome data. If those traces are captured by the AI firm or employer, they become training and evaluation data. The next model becomes better at doing the task with less human input. The worker’s marginal contribution and bargaining power fall, even though the original system was “augmenting” at deployment time.
If we really believe in data scaling, we should expect scaling to apply to any capability domain that can be captured in data records. Many areas where models are currently bad are areas where data is harder to get. But if we had the data, and no countervailing force, why wouldn't models be able to learn the necessary patterns? (Of course, I do not mean to argue that every social or relational dimension of work will disappear or that demand for human labor will approach zero. People may continue to value human presence, accountability, care, etc., but even these aspects of labor will not be immune to data scaling.)
One tempting response is to say that we should simply impose “augment-only” rules at the modelling level: build systems that assist workers but are somehow prevented from replacing them. I am skeptical that this is technically coherent. Once a model has learned a capability, it is very hard to guarantee that the capability will only be used in complementarity with human labor rather than in substitution for it. The same ability that makes a system useful as a copilot often makes it useful as a replacement, especially once the surrounding workflow is redesigned around the model. So I do not think “augment, don’t replace” can be secured primarily through model behaviour. It has to be secured through constraints on data capture, deployment, ownership, and use.
So I'd contend: an AI system will not be stably augmentative just because it is deployed as a copilot. It can be stably augmentative only if the institutions around it preserve meaningful control over use-time information and rights over downstream training/evaluation data. Many efforts to build augmenting systems -- with the best of intentions -- may directly support replacement unless they somehow restrict the flow of data. This friction could come in the form of increased individual data rights and/or an approach emphasizing data intermediaries and collective bargaining.