Current AI systems are very primitive.
For them to be truly useful long term, they will need enhancements. I want to talk about one which is hardly available in the current widely deployed systems.
I will be discussing specifically in the context of coding agents. The points are likely very applicable in most other areas too.
That feature is often named as continual learning in the literature. If we are to compare a human with an AI system today, this is likely the biggest functional difference.
The Naming Problem
Before explaining what continual learning is about, I want to first make an observation related to the nomenclature here. It is to do with the meaning of the word “learning”. I claim that learning is likely to mean two very different things for a common man and an ML person.
In the real world, learning is not simply the ability to give meaningful answers, but more importantly, it is the ability to keep bettering those answers. When we look at most machine “learning” systems today, this is exactly what they cannot do.
As a matter of fact, the term “machine learning” is misleading on two counts rather than one. In common usage, to do something “mechanically” means to do the same thing over and over without any improvement. This is in fact the opposite of what is commonly understood as “learning”. So, to use two almost opposite words together is honestly quite misleading.
I submit that a better name is required. Hopefully, the following paragraphs will at least make a case for why the current name - machine learning - is not fit for purpose anymore.
The so-called “machine learning” systems of today, including large language models (LLMs), are static in terms of their knowledge. They produce practically the same answer, without any improvement, over and over again. This is NOT “learning”. Yet the field proudly calls it machine “learning”. To see why this is a problem, all we need is to think about what name we should use for a system that could truly learn?
If we survey the literature, there already seems to be a consensus in the research community on what such systems should be called - “continual learning”. Irrespective of what name ultimately becomes popular, the key thing to note here is that we are having to invent new words to represent what should have been simply called “learning”!
In other words, what we today call a “continual learning machine” is what “machine learning” should have been about from the very beginning. But as anyone who remembers pre-ChatGPT will tell, getting a machine to produce a meaningful answer was already a lofty goal in the early days that only a few people would have complained that the bar was set too low back then.
Why Coding Agents Need It
Now let’s carry on with the nomenclature that we have - continual learning. Why is it important for coding agents?
Simply because, without it, we are likely to end up with reams and reams of brittle markdown trying to steer the AI agents to produce the code we want. The biggest productivity boost from coding agents has been the fact that they seem to just “get it”. This is what has allowed the programmers to prompt largely at the system level and trust the agent to do the right thing when it comes to the details. Without this, we are essentially back to square one.
Put another way, programmers enjoy the most benefits when they are willing to leave the details to the discretion of the coding agent, rather than hand-hold it. But this works only if the agent will do the “right thing” when it comes to the details.
So far, the number of people who find this to be the case is quite high. But will it remain so? What happens when the dependencies evolve - a better library, a better paradigm?
We would expect the coding agents to keep doing the right thing. The question is, could they possibly?
There is a future world where coding agents release new versions, with updates to not just the tooling, but also their knowledge base! This may be possible if a large number of people adopt the same “standard” stack. This is not unheard of - I believe the Microsoft stack is one such example.
The Hard Part
Even in this limited case, research suggests that the vendor (e.g. Microsoft) is likely to run into some fundamental challenges. The key challenge is about how the new knowledge is going to be incorporated. Retraining a model from scratch with the new knowledge is likely to be impractical for a couple of reasons.
One, the training data is still going to have an overwhelming amount of old patterns. These old patterns are not easily “edited”. Secondly, the sheer cost of doing a full training is likely to be massive.
So, what is feasible is a more targeted approach that builds on top of an existing model, to bring it up to date. This is where the research community has identified the first big hurdle - catastrophic forgetting. This is simply the phenomenon of a model forgetting existing knowledge when attempting to learn new things.
Solving this challenge is a key objective of continual learning research.
Let’s hope practical needs will force more and more effort in the area and that we get some exciting news in the near future.
Before closing, let’s briefly imagine the world once we have such continual learning systems. I think the easiest way to think of such systems is as a co-worker. An entity with its own life cycle and development progression. Given that such co-workers are machines, they could be deployed paired to individual humans or even to multiple humans (learning from all of them).