> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
I have an agentic workflow and Luna just always gets stuck, SOL and grok 4.6 don’t. I like Luna in theory I just find not much practical work for it yet in coding type work.
Now I think Luna is plenty good for many applications inside a very good harness/scaffold. And I think there are a lot of those usecases. So I think these small models are really good for application developers.
But for entrepreneurial knowledge work all of my work still benefits a lot from more intelligence.
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
> crazy complicated algorithms they cannot implement themselves
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
It probably depends on the background and the company. For example if one works at a startup that happens to use technology, it’s unlikely to happen because SWEs just translate business rules to code. But if one works at the place where the technology itself is the focus, then yes most people will fall in that bucket.
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
Sol is 100 EUR/m and I prefer it over Fable, and you get practically unlimited usage unless you go really overboard with it. So I don't get the cost argument. Would be better to own hardware, but with the shortages that's not very attractive atm.
IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
As a developer, none related to the model. It’s faster, lighter, makes less assumptions.
For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.
But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.
For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.
For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).
For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.
Just like for warships, the complexity and cost of building cutting edge hardware has grown exponentially up to a point where a significant chunk of the world's computing is dependent on 2 companies: ASML, TSMC. We shouldn't extrapolate linearly from examples from the 80s.
No, but I wouldn’t expect it to stagnate like with Intel in the 2010s either. Maybe the biggest caveat is that most people will be fine with using cloud providers, so the market for non-server hardware won’t be subject to as much competition.
I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I would be shocked if the current high prices were to persist for a decade. Eventually the percentage of people who can afford that hardware will grow to be a decent chunk of the computer-owning population. And they'll be following READMEs written by the early adopters, for installing open-source harnesses to work with open-weight models.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
Newer cards aimed at consumer market are not capable of being used for local models the way 3090s are. That's on purpose: this capability is now used to price-differentiate between "normies playing games" and "companies in data center business".
For now. That won't last forever. Yes, it'll take quite some time to work through the current production backlog, which is why I'm predicting five years, not one or two. But the trajectory has always been "new video card comes out, game devs push the limits of what it can do, gamers buy new card so the hot new game can run faster, rinse and repeat". And that includes wanting more VRAM so the game can load more of the scene at once, load higher-res textures, etc.
Which means it's inevitable that eventually, even the consumer game market will be buying GPUs with 32 or 64 GB of RAM. And there are decent models that will run at that size. Even the "normies playing games" market, as you call it, will end up with the capacity to run local models. It'll take a few more years than it would have if the data-center companies weren't trying to buy up all the GPUs, but it's not like gamers are going to stop wanting to play games. So in the long run, Anthropic et al are still going to have to figure out how to deal with competition from local models that run on your gaming video card. Which won't ever be at parity with the models that take terabytes of VRAM to run, but are very rapidly approaching "good enough for what most people want to do".
I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price.
There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.
So will the stock of 3090s, as well as their ability to run contemporary local models. And there will be no supply of newer equivalents of those GPUs, because NVIDIA has since wisened up, and is using the very capabilities you need for local LLMs as market segment differentiator.
I keep coming back to this: why do I need to run a local model on my own GPU? Open models can run in dedicated clouds and while, yeah, they may be more expensive per token than my own GPU, when accounting for depreciation, energy usage, and opportunity cost (money not spent on my GPU will instead sit in my portfolio appreciating with its particular blend of returns), I'm pretty sure I break even or even net lose money with a GPU.
Don't get me wrong, there are advantages to a fully local model in that, I can have agents looping 24/7 even when my internet is not working. But this is niche enough that if I had to price the advantages they don't seem worth it.
If I'm willing to pay the Openrouter tax, I can fire up Openrouter today and just get access to whatever model I want, and still pay a fraction for tokens as what I'm paying with the big guys.
Unless you value privacy, pay for openrouter. You still get the benefits of cheap tokens and programmatic usage.
3090 pricing is something of a wild card. Since the only big-mem consume cards are the xx90s, and a 5090 is pushing $5000, resale value has gone way up. The bottom hit ~$700 last year. It's still a very good GPU, if power hungry.
I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.
It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.
Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.
Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers.
It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.
Spending that kind of moment on a product that gives you personal happiness for years up to decades and then will still have residual worth, which people save up for ages for, is an entirely different proposition than buying a product that may make you faster professionally, but which in the short time can also be achieved by a few dollars worth of subscriptions to a hosted model for even greater effect.
Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.
Most people in the US don't drive a new car, and used cars can be had for far less than $40k. An $80k purchase would be just shy of the median annual household income -- anyone who thinks nothing of that has financial resources far above typical. You are in a bubble.
An $80k purchase is far more affordable when you're looking at an 84 month loan. You trade in your current $20k truck with $30k in debt on it for your $80,000 car, get a couple grand in incentives and a $10k down payment, and boom you're only looking at a bit under $1,200/mo in payments. The median household is bringing home ~$84k before taxes, hypothetical person lives in a no income tax state, they take home ~$5k/mo. Easy peasy, its not like you were planning on taking any vacations anyway since you're always working.
What matters is you've got the Duramax HD King Ranch TRD Big-Boy machine. Doesn't matter the cost. You can tow anything, drive anywhere, do anything, and do it all in comfort. Other than parking in a normal parking spot comfortably. Or even park it in your own garage at home.
I've seen this exact scenario many times personally.
There are many payday loan operators and those willing to sell predatory loans to those workers you mention buying boats or RVs. I've yet to see a payday loan open up in SF to help tech workers buy hardware.
They cost more to run than hosted anyway. But that isn't the point of having them. They are a playground, a backup when the internet is down, or claude is down. They can render Blender scenes pretty well. They play any game I want.
You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out.
I keep seeing this comment. This is _hacker news_ where, back in the day, people just hacked on things, because it was a hobby. They weren't "moneymaxxing" or desperately trying to be as insanely efficient as possible. They hacked on stuff with a can of surge at 3am because it was fun.
Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.
Back in the day the business backing this platform wasn't incubating companies like Flock (YC S17). I think the increased focus on "moneymaxxing" in the community reflects a similar change by its owners.
Not really. 2 years ago that was a pretty normal amount of GPU hardware for a hacker or gamer. It's all relative. They are not accessible to most people yet, but for someone that cares and is a technologist? Likely accessible.
I have trouble getting simple extraction to work sometimes. I have a block of text describing people and their roles at a company and their ages, and i asked for structured results of an array of these things with the text span that it appears in and all i can say is: nope.
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.
By annoying snippets of incomplete conversations, do you mean Musk's clamoring for legalisation of child porn, or Altman's obsession with doomsday in talking about AI, or Zuckerberg's "Aut Zuck Aut Nihil"?
The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
- ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then
- claiming that AI would make all those people who put in the work redundant
It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.
I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.
This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks.
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
DeepBlue beat Kasparov with essentially raw compute thrown at alpha-beta search. That does support the premise of the bitter lesson in general. But that does not mean the bitter lesson is correctly being applied here. The point is that even if throwing raw compute at a task is better than careful human-crafted algorithms, it's still task-dependent. The current trend with the people blowing hundreds of billions of dollars is developing an all-purpose model that is everything to everyone, but you don't need hundreds of billions of dollars to create a task-specific model that outperforms their model at a given task.
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
> Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right?
I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.
Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.
This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.
If you are just taking a chess model vs. an LLM, you are right.
But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.
Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.
AFAIK Stockfish still remains a traditional (some-variant-of-) minimax-search engine, but now uses a microscopic neural network to evaluate the positions at the maximum search depth rather than heuristics. I expect any pure ML engine can learn to exploit any fixed heuristics, but now Stockfish is trying to balance both strategies, which still heavily ends up on the side of raw search depth.
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
You are technically correct. The best kind of correct.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur
I think the gp found a viable counter to the ggp's version of the bitter lesson, which seems so extreme as to certainly exceptions. IE, they seem to say nothing but the latest frontier model is ultimately viable as an AI business.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.
So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer?
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true.
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.
Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.
> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.
More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!
But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983
> This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks?
Oh no!
The whole field was full of people whose entire career was built around the idea of developing smart priors.
To quote Wikipedia:
> For computer vision in particular, much progress came from manual feature engineering, such as SIFT features, SURF features, HoG features, bags of visual words, etc. It was a minority position in computer vision that features can be learned directly from data
This undersells the change though! David Lowe's reputation as the best image researcher in the world was based on his SIFT patent[1]
This approach worked until 30 September 2012.
That was a bitter day for many, many computer science researchers.
Is being wrong/ignorant about whether/how something can be automated the same as having a preference for doing it manually? Maybe so if it's your patent, your thesis I guess..
But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.
There's two different goals to AI research - one was to get results - a chess engine thst wins, etc. But the other goal (which seems to have been abandoned in the deep learning era) was to use AI to help understand how human minds work. A chess engine modeled after human grandmasters is much more interesting in that regard than either a min-max algorithm like beat Kasparov or modern deep learning engines.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.
Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen.
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.
The Bitter Lesson says that the only things that scale are search and learning.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
> A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.
Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
Now I think Luna is plenty good for many applications inside a very good harness/scaffold. And I think there are a lot of those usecases. So I think these small models are really good for application developers.
But for entrepreneurial knowledge work all of my work still benefits a lot from more intelligence.
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
replit is already leading the way with free luna usage
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.
But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.
For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.
For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).
For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
Which means it's inevitable that eventually, even the consumer game market will be buying GPUs with 32 or 64 GB of RAM. And there are decent models that will run at that size. Even the "normies playing games" market, as you call it, will end up with the capacity to run local models. It'll take a few more years than it would have if the data-center companies weren't trying to buy up all the GPUs, but it's not like gamers are going to stop wanting to play games. So in the long run, Anthropic et al are still going to have to figure out how to deal with competition from local models that run on your gaming video card. Which won't ever be at parity with the models that take terabytes of VRAM to run, but are very rapidly approaching "good enough for what most people want to do".
Don't get me wrong, there are advantages to a fully local model in that, I can have agents looping 24/7 even when my internet is not working. But this is niche enough that if I had to price the advantages they don't seem worth it.
If I'm willing to pay the Openrouter tax, I can fire up Openrouter today and just get access to whatever model I want, and still pay a fraction for tokens as what I'm paying with the big guys.
3090 pricing is something of a wild card. Since the only big-mem consume cards are the xx90s, and a 5090 is pushing $5000, resale value has gone way up. The bottom hit ~$700 last year. It's still a very good GPU, if power hungry.
It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.
Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.
It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.
Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.
What matters is you've got the Duramax HD King Ranch TRD Big-Boy machine. Doesn't matter the cost. You can tow anything, drive anywhere, do anything, and do it all in comfort. Other than parking in a normal parking spot comfortably. Or even park it in your own garage at home.
I've seen this exact scenario many times personally.
You could sell those and have enough money to pay for hosted inference for years.
You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out.
Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
- ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then
- claiming that AI would make all those people who put in the work redundant
It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.
https://stockfishchess.org/blog/2023/stockfish-16/
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.
Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.
This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.
If you are just taking a chess model vs. an LLM, you are right.
But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.
Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.
https://en.wikipedia.org/wiki/Stein%27s_example
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
Any similar model aimed at coding?
A >10B model for mass spawning/swarming and reporting back to a larger model
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
> maybe if we use a blend of raw data and hand-crafted heuristics
I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
Just kidding of course
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000
This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.
More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!
But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983
Oh no!
The whole field was full of people whose entire career was built around the idea of developing smart priors.
To quote Wikipedia:
> For computer vision in particular, much progress came from manual feature engineering, such as SIFT features, SURF features, HoG features, bags of visual words, etc. It was a minority position in computer vision that features can be learned directly from data
This undersells the change though! David Lowe's reputation as the best image researcher in the world was based on his SIFT patent[1]
This approach worked until 30 September 2012.
That was a bitter day for many, many computer science researchers.
[1] https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...
But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
In the case of stockfish, the harness is a tree search around the neural network evaluations.
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.