I went on The Interlude Show to talk with Jake and Vatsal about what I’ve been working on in my research. We covered the cerebellum as a forward model over cortical states and what that suggests for machine introspection, Zipfian grokking, why browser agents should read the web with vision rather than the DOM, whether Jevons’ paradox saves software engineering, and what humans were evolutionarily designed for. What follows is a transcript of our conversation.
Cold open There are certain parts of the brain, the most notable of which is the cerebellum, which looks a little bit like a generalized world model that can talk to the cortex in ways that we don’t necessarily understand super well. And for this particular interplay of these systems, I’m interested in the idea that this allows us to experience the shadow of our own neural computation. There’s a theory by a guy named Masao Ito, a Japanese neuroscientist who died a couple of years ago, who thought of the cerebellum as a generalized thought-forward prediction model. You can look at the sum of the parts as a system that has good interpretability because of the interplay between the different brain regions that goes on. And this is something that I find interesting.
Jake: Welcome to Interlude, where we talk about the pre-AI and post-AI world. I’m Jake, one of the co-hosts of this podcast.
Vatsal: I’m Vatsal, one of the co-hosts of the Interlude podcast. Today we are interviewing Jasper. Jasper is a member at South Park Commons, researching new types of AI architectures inspired by how the brain works, touching on how humans learn sample-efficiently, are capable of introspection, and learn robust motor skills. Before that, he was at Yutori, working on web agents that effectively emulate human users. Jasper and I met at a South Park Commons park hang, and all three of us are part of the same community, so we’re very excited to have you here.
Jasper: Thanks for having me, guys.
Vatsal: So we’ll dive right into the questions. Tell us, what are you exploring at South Park Commons?
Jasper: When I first got bitten by the research bug about nine months ago, I started looking into model representations, and how models form representations that generalize from data that inherently doesn’t generalize in and of itself. So I started looking into specific questions around model introspection, how generalizable representations form, and ideas around interpretability, and started going from there into learning a lot more about how the brain works, based on my own curiosity.
There are a lot of interesting parts of the brain that feel like they don’t really have a place in the standard architectures that we currently train models with, and so that was really interesting to me. I got nerd-sniped by that a little bit. Since then, I’ve been working a bit more on some of the things that behaviorally we do that the models don’t, one of the biggest of which is motor control — that’s a specific thing that I’ve spent a lot of time focusing on recently. So that’s the kind of thing I’m looking into.
Vatsal: Just to clarify: so this is mostly focused on ML interpretability. How do you tie in your interest in how the human brain works with interpretability? How do both of them tie together?
Jasper: Right now — I think I started a little bit with interpretability, but one interesting property that the brain has that I think the models don’t is that we seem to be really good at a sort of loose automatic interpretability, in ways that the models find it difficult to do. We can’t dissect our own thinking perfectly, but we can dissect our own thinking reasonably well. And for the models, you have to do post-hoc, intensive, expensive interpretability techniques that break down the model’s process in ways that would just be completely infeasible for the brain to do. So I was thinking, there’s got to be a better way to think about how we can do interpretability, based on how biological learners do something that looks a little bit like interpretability. That’s the intersection there.
Vatsal: Can you give us an example?
Jasper: So for example, there are certain parts of the brain, the most notable of which is the cerebellum, which looks a little bit like a generalized world model that can talk to the cortex in ways that we don’t necessarily understand super well. And for this particular interplay of these systems, I’m interested in the idea that this allows us to experience the shadow of our own neural computation. There’s a theory by a guy named Masao Ito, a Japanese neuroscientist who died a couple of years ago, who thought of the cerebellum as a generalized thought-forward prediction model. And this, for example, is a form of — you can look at the sum of the parts as a system that has good interpretability because of the interplay between the different brain regions that goes on. And this is something that I find interesting.
Vatsal: So we’re able to dissect our own thinking whenever we make any sorts of actions. But with models, you have to do a lot of post-hoc processing to be able to do that. And that’s the sort of difference that you noticed?
Jasper: Exactly, yeah. And that’s because the models don’t really have a true analog for action in the sense that we take. We can imagine the consequences of our actions in the real world in a grounded way, specifically because the cerebellum provides a sort of online correction to our actions. This is how we don’t need to make constant adjustments to our actions — we can just smoothly execute movements based on the world modeling that the cerebellum is constantly providing.
And so it’s a little bit of a lesser known theory, but I think a very interesting theory, that the cerebellum also does this in a generalized sense for people’s neural representations.
Vatsal: I think something I read in one of your blogs was about the forward models, and that reminds me — you’re talking about how we try to understand the actions that we take and learn from them in our brains, but that particular article was trying to peek into the model layer, right? So could you maybe talk about that paper a little bit, and what you were looking into?
Jasper: So this paper, the forward self-modeling paper, was explicitly inspired by the idea that the cerebellum is a forward predictor over cortical states — so, the activation analog.
Vatsal: Could you explain that?
Jasper: Yeah. The idea in the paper is that you take a standard transformer and you learn to predict the later layer activations based on the earlier layer activations that the transformer has. And in the process of doing this, the meta-model that you’re training, that lives alongside the main model, is deliberately smaller than the amount of parameters that it is modeling in the main model. And because of this, it learns a compressed basis of the computation that the main model is undergoing. So it doesn’t get everything perfect that the main model does, but it gets the coarse details right.
And that means that when you get to the end, you have the predicted activations versus the actual activations that the main model computed. The difference between those is an interpretable residual that reflects the complexity that the main model computed that the forward model — which is a compressed version of the main model — didn’t expect to see show up. So it’s a kind of grounded novelty signal, or computational difficulty signal.
Vatsal: So I’m just going to repeat what you said, and I might dilute what you said a little bit, just maybe simplify it, just to make sure we’re on the same page. Essentially what you’re doing is you have a big model and you have a smaller model. If I understand correctly, while the inference on the big model is happening, you use the small model beforehand to predict what the big model is going to do — like you mentioned, which layers in the network will be activated. So you’re going to predict how the big model is going to think, essentially. And that is similar to how the human brain works. Is that a sort of correct interpretation of it? Check me if I’m wrong.
Jasper: Definitely, yeah. So the forward model and the main model are predicting the same objects. They’re both based on the activations that the main model is working with, and so when you get to the downstream layer, the main model has predicted a set of activations and the forward model has also predicted a set of activations. So you can directly compare those two quantities.
Vatsal: Okay, let’s do an analogy with the brain first. So imagine I’m playing chess, all right? So if I’m playing chess and I make a move — I think about making a move and I think, okay, this is the optimal move. In my brain, how would my forward model work in that example?
Jasper: Yeah. So in the real brain it’s a little bit more complicated, because it doesn’t work just based on your actual activations. It takes two inputs: both your actual activations, and what’s called the efference copy. That’s a separate neural signal that your brain fires when it’s about to take an action with your muscles. And so the real cerebellum takes both inputs and then produces the expected output based on, what do I think will occur if I do this?
And so in the chess example, you might be looking at the board state and observe your situation, and then what you would do is, your brain would fire off an efference copy saying, here’s this action that I am theoretically about to take, what do I expect the outcome from this to be? And this is a simplified version, because the original usage of this was actually online motor control in real time, as opposed to discrete movements — but the analogy is a good one and it applies either way.
Vatsal: And this is what inspired your feed-forward models as well. And if, for example, now you’ve got a model that was trained to play chess, now your — I forget the term — your smaller model in this case will essentially be able to, like a world model would, simulate what’s going to happen if you move the pawn to e4. It’s going to simulate that, and then your big model is going to simulate that but with a lot more parameters, and then you can sort of compare the output in both cases. Is that correct?
Jasper: There’s a couple of subtleties here. So in the case of the big model, I think the interesting properties theoretically might happen not because the big model is doing better than the small model, but because it is able to offload that world modeling computation to the small model, such that in principle you might get a more modular design architecture, something like that.
There’s also the subtlety that in the real human brain — and actually all mammalian brains — the cerebellum is actually a very ancient part of our brain. There’s a complex wiring where the cerebellum injects its predictions into the middle of your cortex layers. And so your cortex, because it is a looped system that has both backwards and forwards components, is able to probably experience the outputs of the cerebellum as essentially an auxiliary sensory signal.
And for this reason, my working theory is that the cortex has the ability to get a grounded experience of what it expects to happen in various domains, such that — the canonical example is, you can’t tickle yourself, because your cerebellum is predicting what’s going to happen and it actually cancels out the signal of the predicted sensory input from the actual sensory input that you get. So that’s the interplay that I think theoretically might happen, and on the sensory side and the motor control side there is already good evidence that this is the case.
Vatsal: I’m curious — taking a step back, how did you get interested in analyzing the brain and how it works, and in AI? Where did that interest even come from in the beginning?
Jasper: Yeah, it came purely from an AI perspective. I have a CS degree. I’m by no means a neuroscience person, but I got interested in looking at the brain as inspiration for where we might look to solve the deficits of current AI models.
I also have a lot of interest in evolutionary biology and paleontology. So it’s especially interesting for me to look at that not as a present problem of how do we get the brain as it exists currently, but also how did the human brain exist in prior phases of human evolution? How does the human brain compare to other animals? The mammal brain is very, very complex in all cases, and the differences between our brains and cetacean brains — whales — or other primates are in many cases quite subtle, and the differences show up not where you would most expect them.
So thinking about that from a functional biology standpoint is really interesting to me, as well as thinking about it purely from a how-can-we-build-better-AI-architectures standpoint.
Vatsal: That makes sense. Going back, before you started at South Park Commons, you presented at the demo fair about Zipfian grokking. I found the talk pretty interesting — I watched it on X. Could you maybe explain that a little bit, and why it’s important?
Jasper: So for reference, for those who may not know the grokking task, the basic idea is you just take modular arithmetic — so A plus B mod C — and you compute a list of rules for a particular modulus, and you train a model to predict the output of that given two inputs with a particular modulus set.
Vatsal: So just to clarify, if you’re doing A plus B mod 10, let’s say, your output will only ever be between those 10 numbers.
Jasper: Exactly.
Vatsal: So your output set is constrained. And then I imagine your input set is also — oh, it’s not constrained. Okay.
Jasper: Yeah. And the basic grokking task, which was originally described in an OpenAI paper from around 2021, is that if you train a model on just a particular list of data points, it won’t generalize for a really long time. It’ll just memorize the particular examples that you have. And for a long time it’ll look like nothing is happening — it will get 0% accuracy on examples it hasn’t seen. And then at some point, really late into training, if you just train it for long enough, it’ll all of a sudden generalize to held-out examples that it hasn’t seen before.
This is often given as an example of a problem where we really, really understand what is going on at a representational level, in ways that we don’t really for much more complex models like LLMs. And so I got into this problem as a way of thinking about, all right, this is the one example we know where we really understand what is going on under the hood, so how can we maybe modify this setup such that it has properties that resemble those of a frontier LLM a little bit better?
Vatsal: So just to clarify, you’re training a model to predict what A plus B mod 10 will be, given A plus B, but the model doesn’t actually have access to any sort of arithmetic or programs, so it’s not able to do that — you’re just training an MLP, a multi-layer perceptron, to perhaps do this thing? So that’s what the problem of grokking is?
Jasper: That’s exactly right. And it’s an interesting problem because, like you said, it doesn’t have the ability to use functions, cosines, or anything like that. But what happens under the hood during grokking is that it actually re-derives this analytically known solution that involves trigonometry. And it uses that in weight space to generalize to examples that it’s never seen before, just because this is analytically known to be a way of describing the rules underlying modular arithmetic.
Vatsal: But the set that you train it on — if you just test it again on the train set itself, it’s always high on accuracy on the train set, but on the test set, as you mentioned, it’s zero at first and then suddenly fully accurate.
Jasper: Exactly. And the reasoning for that is it has to specifically develop that trigonometric representation under the hood that describes modular arithmetic in the maximally compact way. So it has to start with a long description length solution — a bunch of memorized examples has a long description length. But what I find interesting about this is that the trigonometric solution has a really short description length; it’s a minimum description length of a handful of bits. And in the process of that, you need to use weight decay in order to find this solution.
Vatsal: So giving it a prior. Weight decay?
Jasper: Yeah, weight decay is just a prior that you can give to your model that says, the larger the weights, this is something that I will penalize you for. So you can use the weights to learn a description of function space, but the larger it is, the more you get penalized for it. As a result, it eventually finds the minimum description length trigonometric solution, and that is grokking.
Vatsal: Okay. And then where does the Zipfian piece come from?
Jasper: Yeah, so the Zipfian piece comes specifically from — I had the insight that a lot of real-world data sets are biased according to Zipf’s law, which says that there’s a really heavy-tailed distribution of data. The most common data is very common, and then there’s a long tail of less common data. As opposed to in a traditional grokking setup, where all data is equal.
And in some sense, all of the data in modular arithmetic optimally points to the correct solution, because you’re giving the model a bias saying, hey, each of these modular arithmetic examples that I’ve given you, each of them points equally to the trigonometric solution — which is true, because the trigonometric solution is equally valid for all of them.
So that inspired the idea of, what happens if you were to put more of these modular arithmetic examples in the train set than others? What if you were to weight each example according to a Zipfian distribution? And what happens is that it learns the correct solution under pressure from weight decay, but then it accumulates a bias such that the trigonometric solution under Zipfian pressure actually goes away, and it forgets the correct solution it’s learned.
Vatsal: So grokking doesn’t work under a Zipfian distribution? I didn’t really follow on why.
Jasper: Yeah. So Zipfian grokking has the unique properties that it does — specifically, that it discovers the trigonometric solution and forgets it. Accuracy goes right back down. Then it rediscovers the trigonometric solution, then forgets it again. It’s an infinite oscillator. And the reason this occurs is because weight decay, which is the force that says I want you to find the smallest possible solution, is effective in proportion to the size of the weights in the model. So once it has found the trigonometric solution, it has reached the minimum description length solution to the grokking task.
The problem is that weight decay is now, because of the Zipfian distribution, not the only force acting on the model. There’s also a force that says, these particular modular arithmetic examples that I weighted highly, I want you to really make sure that you get those right. And because the model is computing trigonometry in weight space, it’s not exactly 100% mathematically perfect. There are small errors that are, for all intents and purposes, meaningless, but they get magnified when you start really upweighting particular modular arithmetic examples.
Vatsal: How do you exactly upweight these examples while modeling?
Jasper: You just have an auxiliary loss term that says, for this train example you really have to get this one right, and these other ones are not that important. And the result is that it oscillates infinitely, because once it forgets the trigonometric solution after these memorization corrections, the weights are large again. So now weight decay can have an effect on the weights of the model again, re-derives the trigonometric solution, and the cycle continues.
Vatsal: So I want to pivot to your experience at Yutori. I actually used to work at a browser agents company, so I’m a little bit familiar with the space and was excited to ask you some questions. We took a more deterministic approach where we relied more on XPath interactions with the browser, and then used LLMs for recovery. But I’m curious: what was Yutori’s approach? I know you used vision over the DOM. And what kinds of use cases were Yutori’s agents best suited for?
Jasper: This is something that’s near and dear to my heart. Even before working at Yutori, I loved working on browser agents for my own hobbyist purposes. I as well started working on this in the GPT-4 era, when it was like, oh my God, we have language models that can look at the DOM and take concrete actions. But at Yutori specifically, we were really interested in getting agents that are reliable, agents that emulate human users as effectively as possible, and agents that are cheap.
And we found that, for a couple of reasons, vision just tends to work best, especially on cost. But if you’re willing to eat the modeling costs of developing multimodal models, vision tends to work best for long-horizon rollouts too. It’s more token-efficient and honestly is probably more reliable at scale, largely because humans use the web in a multimodal fashion.
So for example, one thing I did was take a bunch of web pages and run them through an embedding model as their DOM, and run them through an embedding model when rendered visually. And one really interesting thing that fell out was that the variance in the embeddings was way, way higher over the DOM embeddings than over the vision embeddings. Which makes sense when you think about it. There are a lot of different ways you can implement a web page on the back end, but it all boils down to how it renders to the user. And if you think about it, that’s also why humans would use the web visually rather than using the DOM: it’s designed for quick consumption.
We want it to be in a domain that is well-constructed and narrow, as opposed to ill-defined and precise but broader in distribution. So for cost reasons, and because of that lower variance, we found that vision agents tend to generalize better to unseen websites.
Vatsal: Something interesting about the DOM too is that some websites would have JavaScript that checks for interactions with the JavaScript, and so they would catch us as bots. And so we actually used the OS cursor to interact with the web page as a way to seem more human.
You said you wanted these agents to browse the internet as if they were more human-like. I’m curious what that means — what kinds of techniques did you deploy there?
Jasper: Yeah, it definitely can mean that: if you consume the web page via the DOM, you’re not necessarily getting the version that humans will see. So even something as simple as clicking on elements that only would be clicked on by a human, not tripping honeypots like that — that’s an example of it.
Another thing this can mean is largely on the infrastructure side: are you sending traffic to the website from data center IPs or residential IPs? And there’s a lot of device tracking that goes into web pages these days. There’s an interesting website you can go to where it will show you all of the device-based profiling that websites can do based on the device you’re using. It’s surprisingly a lot. And this means that if you’re trying to use, for example, a hosted cloud browser service, it’s pretty much an impossible game of cat and mouse to emulate that perfectly.
So one of the things I did at Yutori was working on stuff just like this: how do we make sure that our cloud browser agents are sending traffic from machines that look like real humans would be using them, coming from real-human-looking IP addresses, stuff like that. Which is a fun cat-and-mouse game that can never really be solved perfectly, but I find it interesting to play that game for a while.
Vatsal: No, for sure. I definitely resonate. I’ve had to play the same game too, and a lot of fun stuff got uncovered.
I guess this is my last question about this topic. A lot of people have the perception, when you’re building browser agents, that the internet has a lot of weird edge cases that pop up when you interact with it. So things like random pop-ups come up, web pages will vastly change their UI and stuff. How did you deal with recovering from that? Because that hits into the reliability aspect of browser agents.
Jasper: That comes down to a combination of factors, I guess, starting from a point of just being comfortable using long horizons. You can buy recoverability just by operating over long horizons, but that also requires doing actions in a token-efficient manner, which goes back to the vision question: how do you run, I don’t know, a 50-click interaction with a web page in a way that both retains what you did previously, such that you have reliability, but doesn’t blow up your token budget? And as far as I’m aware, one of the best ways you can do that is using vision, which is a couple hundred tokens per interaction, rather than using the DOM and needing to throw out a bunch of content at each interaction step, such that you might be a little bit more likely to run into traps, or run in circles of forgetting you took prior actions, getting ungrounded, that kind of thing. So it all comes back down to that, in my experience.
Vatsal: Did you rely on foundation models for your vision models, or did you end up post-training your own?
Jasper: Yutori does post-training on top of Qwen models. I wasn’t on the modeling team, so I was doing more of the infrastructure and harness surrounding the model. But there are a lot of really good foundations that are being laid by open model companies these days, obviously. And there are still a lot of things you can do on top of that if you put a lot of effort into thinking through how you want to construct your harness. Having the right sorts of problems — knowing what sorts of problems people are deploying these models for — can often be a really big boost in terms of being able to construct synthetic data effectively to target those use cases.
And that’s, in some sense, what Yutori’s secret sauce is, as far as my perspective on it goes: having an insight into what people use the models for, and being able to wrap those insights into the training process such that the models can be deployed at a Pareto-beating accuracy and cost.
Vatsal: This is kind of a destructive thought, but if someone were training a browser agent to just work on the visual — a model that would just look at the visual and then make decisions based on that rather than the DOM — I’d want to create adversarial examples where something looks like a button but isn’t really a button. And I would want to create a synthetic dataset of all those adversarial examples. I just wonder how good the model would perform on those.
Jasper: I think you could definitely construct them. I feel like there’s a bad design competition I’ve seen on Reddit where it’s all kinds of wacky web interaction stuff that is kind of cursed. I think the main issue that comes to mind that you would run into is that you would probably horribly confuse your human users as well. And so you could design an uninteractable website, but I imagine it would be equally uninteractable to humans and to vision-based web agents.
Vatsal: All right, so switching topics a bit — a little more high level. What are your thoughts on the future of software engineering? The reason behind this question is that I read your article, “Why Jevons Paradox Won’t Save Your Software Engineering Job.” It’s from last year, and in it you made the following predictions: that in 6 to 12 months, machines will be able to, one, hold more information in working memory than a human; two, use a computer at least as effectively as a human to augment its context when necessary; and three, architect novel systems better than a human. I can grant you the first, but not the second and third with as much confidence. Has your view shifted? What are your predictions now?
Jasper: I think I would stand behind all three claims personally. Specifically, on holding more information in memory — I don’t know, you can squint and say that the file system is a version of that. And you said you agree with that one anyway.
For where I was thinking on using a computer better than a human, and on the question of architecting novel systems better: what I mean by using a computer better is that I used to use the terminal on my computer a lot more. For instance, a lot of the time I don’t bother going into settings anymore. I just open Claude Code on my computer and say, execute some terminal commands to change my computer. That’s not an example of UI-based computer use, but I think it’s equally valid in some sense. I don’t bother using the macOS native search functionality anymore, because it’s easier to just tell — one, that feature sucks, and two, Claude Code is just good at grepping through your file system anyway. So that’s half of the equation.
On the second question, I think also for repetitive, visual-based tasks — take an example of something that’s really schleppy that involves clicking through stuff, like HR training at a company or something like that. I would definitely like to think that Claude Code is better at using a computer to complete HR trainings than me. Certainly I don’t want my comparative specialization in this life to be taking HR trainings on the computer. So that’s the kind of thing I mean: the intersection of those two computer-use abilities. And I would argue that at this point, the models are probably better at that than me.
And then as far as architecting novel systems goes, I really meant that in kind of a SWE-native sense. Given a high-level description of, let’s say, some web app you want to build that’s slightly novel in some ways — I would think it would be a bad use of my time to manually implement a web app myself, even a slightly novel one. I don’t think there’s a universe in which I would not delegate that to Claude. Admittedly, there might be some funky stuff that Claude would need to handle. But honestly, I don’t know the details of web architectures that well anymore. I feel like a lot of this stuff would be better done by an auto research loop than by me at this point. So that’s what I meant from a SWE perspective for the third question.
Vatsal: So based on your predictions, what do you think will happen to software engineers in 10 years?
Jasper: I think the part of that last thing that seems not automated right now is generating the description of some novel system — purely on the description side. To me, that’s related to the question of why LLMs can’t write well, which is a fundamental thing that I would argue has to do with their cognitive mechanisms, and is probably related to why they can’t think of ideas de novo quite as well.
For now, it seems like the job of technical people and software engineers has just bumped up a level of abstraction, to managing that level, which humans can do pretty well. Admittedly, there are a lot of software engineers from 10 years ago who couldn’t really do that, and who were mostly focused on the lower-level implementation stuff. So that’s the kind of software engineering that I have argued will be obsolete, and which I imagine is in the process of going out.
But at least until we have models that are more creative in some sense — whatever that means — I do think there’s still a place, especially at startups, I’d imagine, for people who can think creatively and, to the extent that is necessary, supervise the models doing the actual building. But I think there’s probably no way of getting around either human interaction or creativity in the workplace for the next n number of years, to my estimation.
Vatsal: So you don’t believe in full-scale automation?
Jasper: I believe in automation of what we defined software engineering as five to ten years ago. I think we’re already most of the way there. These days, if I have to edit a string in a code base, I’m not going to command-F and search for that string. I’ll just tell Claude to do it. And I don’t think I’ve touched code in the past six months, to be honest.
Vatsal: Maybe we can also — in this article that you’ve written, you talked about Jevons paradox. And you’d given this example, an agriculture example. You said that in the 1900s there were farmers, and a big part of the economy was farming. And then there was a lot of technological progress that improved the food output per farmer. But the demand is inelastic, so people still need that same amount of food. Now farmers, since they’re very technically able, are able to do other jobs, and farming is mechanized, and only two percent of employment is farming. That’s the sort of progression you saw when you had this technological revolution in agriculture, and I guess you can compare that to a technological revolution within tech, in a way. So that’s where my question was inspired from — automation or augmentation. Do you think that, I don’t know, 50 years from now only two percent of people will be doing software engineering, and the rest will be able to do higher-abstraction, different things that I’m not even able to predict right now?
Jasper: That’s what I would imagine, yeah. For clarity, I do think that we’re a long, long ways away from being able to observe that. Just because, for example, when the first mechanized farming implements came about, my impression is that the demand for food would still have been so great relative to what could be supplied that there was a long period — I would imagine, and I would probably want to look this up first — of increased output without necessarily a corresponding decrease in employment.
But obviously we eventually got to the point of a corresponding decrease in employment. And I think that we’re in a similar place right now with software, where there’s starting to be a dramatic increase in output, but the demand for software is so far pretty elastic, so I don’t think we’ve really explored the curve to the point where it gets inelastic and software is no longer worth investing in producing more of. So in terms of what people spend their time doing, we’ve probably already started to explore the place where software demand was inelastic. But as far as actual production of software, I imagine that we have quite a ways to go before we start exploring the space where this is no longer an important thing for anyone, human or machine, to be thinking about that hard.
Vatsal: So why do you think the demand for software is elastic? Is it price sensitivity — if there were a price increase, would you imagine people not desiring software as much, or if there were a price decrease, people would really try to use a lot of software? Why do you think that is?
Jasper: My mental model of software is as something along the lines of an exoskeleton of intelligence, where it’s this ossified artifact that intelligent entities can create that presents the ability to rerun specific artifacts of that intelligence really cheaply.
And to the extent that the demand for cheaply executable intelligence is very elastic, I think that’s probably reflected in demand for software — that it’s just a fundamental thing that has a lot of demand for it. And if you apply the rough analogy of, I don’t know, DNA in nature — I’m sure it’s not a perfect analogy — but there’s to some extent elastic demand for DNA insofar as there is free energy to be soaked up from the sun, right?
Vatsal: Oh, I see. Okay.
Jasper: Rough analogy, but I think that’s my mental model for why the demand for software is a broader section of the demand for intelligence, and therefore pretty elastic for a while, as far as we can tell.
Vatsal: I guess on that note, beyond software, what are your views for humanity in general? Where do you think the future of humanity will be going?
Jasper: For the time being, I think we’re actually in a little bit of a desirable position, such that, as far as I view it, we’ve invented machines that do the schleppy parts of cognition without the creative parts of cognition. And so for the time being, it feels like we have this awesome window where it’s okay to be a creativity machine, and that’s where a lot of the value hopefully accrues to. That feels like where we’re at from a technological development standpoint, at least for the next maybe five to ten years — we’ll see.
I do think — and this is something that I’m working on — I want to understand why humans have this thing that we call creativity and the machines don’t, and I’m looking into the neuroscience and neuro-AI explanations for how this could occur. So it’s hard to say the timelines on which we’ll be able to get meaningful machine creativity in the sense that we would define it from a human. And when that happens, which I’m sure we will get there at some point, it’s hard to say what exactly we humans will pivot to. But I feel like one thing that is pretty comfortably established throughout time is that we humans are pretty narcissistic and we really like talking. And so I’m sure we’ll use the creative systems to augment our own creativity and become truer versions of our creative selves, or something along those lines, that will still be — we’ll still view it as meaningful, I’m sure, even if in some abstract objective sense it wouldn’t have been considered meaningful 50 years ago, maybe.
Vatsal: Do you feel like AI will — it sounds like you think AI will always be used to augment our creativity. Do you feel like it can almost replace it? For example, you might have AI filmmaking tools, or video generation. Do you think it can get to a point where it even takes over, or even becomes better than humans at that?
Jasper: I’m sure that it will, but I feel like we’ll just stop caring, probably. For example, in the Industrial Revolution, we used to have to rely on horses to get around, and then cars came around and they were better than horses. And we used to have to rely upon human manual labor to do physical manufacturing tasks, and that is to some extent going the way of the dodo bird as well. And was physical manufacturing ever a thing that we should have hung our hat on as a species? Probably not, I would say. I’m sure that there were people — but also, were there people who cared about that? But also, we tend to do a good job of caring about vanity metrics. Why do we go to the gym? Certainly not because it’s economically productive to be a physically fit person, if you live in San Francisco at least. But we still do it because, I don’t know, we’re evolutionarily designed to need to go to the gym and it’s fun to feel good, I guess. And I imagine there will be similar things going on: why do we learn math and go to the gym? Probably the same reason we will continue to be creative in the future, even if it’s not strictly necessary and we can get further with our creativity with machine aid than otherwise.
Vatsal: On that note, I’m quoting one of your articles. You said, quote: “The fact of the matter is humans were not evolutionarily designed to do symbolic reasoning, and we happen to have invented a machine that is more or less designed to do just that.” And now you’ve just described to me that you think in the future creativity will be something like gymming — something that’s not necessary but we’ll still continue doing. So what are humans evolutionarily designed for? And how do you envision that sort of future coming along? Because if I think of that sort of future where so many things are just optional — I think Yuval Noah Harari had this vision of the future where everyone’s just lazy and not doing anything. So when you say that, that’s what I imagine. I just wanted to get your thoughts on that.
Jasper: I think you can take a couple of views of what humans are designed for evolutionarily. The trivial view is that we’re designed to reproduce and survive into the next generation, because that is the physical mechanism that brought us here, obviously. But I think that’s also a little bit reductive, in the sense that comparing LLMs to next-token prediction machines is reductive. That is obviously trivially true, but it’s not necessarily a good descriptor of what brought us here, or what is unique, or anything like that.
I think maybe a more human-centric vision is to look at what happened on the evolutionary timescale since our most common ancestor with the genus Pan, which includes chimpanzees and bonobos. One thing that has been hypothesized as a significant factor influencing human evolution during this time period was a shift in the primary sources of predation and natural environmental dangers, going from the natural environment to other humans. So for example, we have found fossils of early hominins like Paranthropus and Australopithecus that were frequently preyed upon by things like eagles on the African savanna, leopards, hyenas, that kind of thing. And as we go more recent in the fossil record, that starts dying out as humans become apex predators, in a sense.
We start moving into this environment where there is presumably really strong selection acting on humans’ ability to survive in groups of other humans, because we became apex predators — but we also became apex predators not in an individualist sense but in a collective sense. Could I re-implement all of the software in a car? Absolutely not. But humans collectively have built this. And in the same sense, humans collectively were very good at hunting large game and surviving in the environment, but only by working together and communicating and augmenting our natural capabilities with tools.
And so I think in some sense, from a human-centric point of view, one of the best answers I can give for what we’re evolutionarily designed for is surviving and thriving in this environment where the most important thing you can do for survival is hanging out with other humans, forming alliances, cooperating with others to survive in the natural environment. And I think this speaks a little bit to why we are so good at generalizing to unseen scenarios and adapting really quickly. We’re really good at picking up on social cues.
And this is to some extent related to some of the neuroscience stuff, because one really interesting thing that I find is that the parts of the brain that are specifically enlarged in modern human evolution — so Homo sapiens compared to earlier hominins like Neanderthals, Denisovans, and Homo erectus — are specifically the parts of our brain that are most involved in social cognition and theory of mind. The default mode network, the precuneus and the cerebellum are three things that are highly implicated in those capabilities and that are specifically enlarged in humans relative to the most common other hominins with which we share an ancestor. So that’s my answer for what we’re designed for, to some extent: surviving and thriving and generalizing in groups of other humans.
Vatsal: So is the next step to design models that are able to interact with each other and have a very high desire and need for social interaction?
Jasper: Maybe. I feel like it would probably fall out considering that there’s convergence between what we found to work well in environments of navigating interaction with other intelligent entities and the sorts of things that might be necessary for cognitive flexibility in these environments. So it certainly seems plausible, yeah.
Vatsal: So we’ve talked about the future of AI and humanity, and we also went a long way back into the past to take inspiration from. We’ve talked about really long-horizon topics, and we originally started by talking about intelligence and ML interpretability. I want to come back to the medium to short term now, and come back to intelligence.
There’s a whole field of post-training, which is about augmenting current large language models to be really good at certain domain-specific tasks. What do you think about the future of post-training? And when I say future, I mean the next 10 to 20 years. Do you think post-training is still going to be a thing? Or do you believe that models themselves will become so good that you won’t need to have them specialize in one thing or the other?
Jasper: So for the purpose of grounding a little bit, let’s define post-training as the things I think of most commonly: stuff like RLVR, like instruction tuning; even something like a data curriculum would fall under what I would think of as canonical post-training tasks. A lot of these feel to me like the sorts of things that humans tend to do in — not the default mode network, but in our task-oriented parts of our brain. The prefrontal cortex is really good at planning abstractly and synthesizing information in order to do so. A lot of this feels a little bit to me like the kind of thing where there are analogs to what humans do that look like post-training.
So let’s say you’re studying for a test. There are analogs to post-training insofar as you might be able to take a test, see how you did, and then generalize back from your score into: okay, what do I need to study further? So, something like data curation. You might be able to do some kind of form of credit assignment based on what was I thinking on this particular question that led to me getting this wrong.
But I do find that the way post-training is done, to my knowledge, at the frontier tends to be a little bit different from how humans would go about this. Typically, if I’m working on a long-horizon rollout type problem and I got the answer wrong, I would wind it back in my head to think: okay, what did I do from steps one through four? I was pretty confident in these steps, let’s say, and I wasn’t confident in steps five and seven. I would be able to retroactively introspect on my own thought process that led to the incorrect answer, and be able to do credit assignment in a way that is localized to, let’s say, steps five and seven, which I think may be more likely to have gone wrong.
Right now we can’t really do this in a true introspective sense in the models, because chain of thought is somewhat close to a kind of record of the thought process, but there have been a couple of interesting papers recently that have shown that this is not necessarily faithful to what the models are actually computing in latents in order to produce each token. And I would argue that in order to get a first-class sense of what the models are thinking, there needs to be a signal with privileged introspective capabilities, such that for any given token, at the token level, we can say what went into producing this token, and how can we do credit assignment in a way that is specifically localized to the qualitative nature of the computation that went into it.
And that’s something that I’m really interested in this forward self-models program as doing, because — remember I mentioned the residual, the difference between the predicted and the actual activations — this is, in some sense, an introspective signal. A model that can sense its own residual, sense what a compressed version of itself might have predicted: in some sense you can use that to say, what did I compute here that was novel? What did I compute here that I’m relatively confident in? And how does this relate to the likely results of the positive or negative signal that I got?
Vatsal: I guess you could also get your reward earlier, because you have your small one that’s already sort of predicted which layers are going to be activated and has an output. So you get your reward, in a way, before the inference for the big model is completed?
Jasper: Yeah. And there’s all sorts of really interesting places where the human cerebellum is deeply connected to the actual dopaminergic signaling side of our brain. So it’s plausible that when you get this novelty signal in your brain, there’s immediate tagging of that with positive or negative dopaminergic signaling as: this is a good sort of novelty, or this is a bad sort of novelty. And I find that to be suggestively interesting about the methods of our cognition.
Vatsal: Well, thank you so much. It was really great having you here, Jasper.
Jasper: Thanks so much for having me, guys.
Vatsal: Yeah, thank you so much. I guess that concludes the episode. And until next time.
Jasper: Cool. Thank you.