Hello, and happy Sunday! As you rest and reflect on the week past and the week to come, we're thinking about AI benchmarks. We may think of benchmarks as a simple yardstick, but for today's models they are so much more—as our ownAlex Duffywrites, they're a critical means of giving some direction to the wild AI ride we find ourselves on. Meanwhile,Katie Parrottwrote a fun first-hand account of how vibe coding tools led her to want to learn to code on her own. AndRhea Purohitpenned a fascinating account of machine creativity that is guaranteed to stoke wonder, perhaps along with your own creative juices. —Michael ReillyWas this newsletter forwarded to you?Sign upto get it in your inbox.Benchmarks lead the wayFor most of us, driving a car means harnessing a controlled explosion. You sit behind a masterfully engineered hunk of metal that turns burning gasoline into progress at 70 miles per hour. With hands lightly on a steering wheel and a foot on a pedal, you control incredible power.Benchmarks steer AI the same way. AI is powerful—explosive even—but without a clear sense of where you want to go, it’s easy to confuse activity with achievement.Last week, Hugging Face shut down their famous Open LLM Leaderboard. For two years, they evaluated over 13,000 models, helping sort good from great. But as AI evolved, these benchmarks stopped measuring real-world impact. Models have been rapidly gaining new abilities—like reasoning and agents—that the leaderboard didn’t capture. Some teams were even training models for the express purpose of acing these benchmarks—essentially “training on the test,” which was no longer representative of real-world performance.This week, Metr re-captured the AI community’s attention with a new and well-chosen benchmark. Their blog post showed off a clear demonstration of AI’s impact in striking terms.Click hereto read the full postWant the full text of all articles in RSS?Become a subscriber, orlearn more.
