A test for AGI is closer to being solved — but it may be flawed

Date:

Share post:


A well-known test for artificial general intelligence (AGI) is closer to being solved. But the tests’s creators say this points to flaws in the test’s design, rather than a bonafide research breakthrough.

In 2019, Francois Chollet, a leading figure in the AI world, introduced the ARC-AGI benchmark, short for “Abstract and Reasoning Corpus for Artificial General Intelligence.” Designed to evaluate whether an AI system can efficiently acquire new skills outside the data it was trained on, ARC-AGI, Francois claims, remains the only AI test to measure progress towards general intelligence (although others have been proposed.)

Until this year, the best-performing AI could only solve just under a third of the tasks in ARC-AGI. Chollet blamed the industry’s focus on large language models (LLMs), which he believes aren’t capable of actual “reasoning.”

“LLMs struggle with generalization, due to being entirely reliant on memorization,” he said in a series of posts on X in February. “They break down on anything that wasn’t in the their training data.”

To Chollet’s point, LLMs are statistical machines. Trained on a lot of examples, they learn patterns in those examples to make predictions, like that “to whom” in an email typically precedes “it may concern.”

Chollet asserts that while LLMs might be capable of memorizing “reasoning patterns,” it’s unlikely that they can generate “new reasoning” based on novel situations. “If you need to be trained on many examples of a pattern, even if it’s implicit, in order to learn a reusable representation for it, you’re memorizing,” Chollet argued in another post.

To incentivize research beyond LLMs, in June, Chollet and Zapier co-founder Mike Knoop launched a $1 million competition to build open source AI capable of beating ARC-AGI. Out of 17,789 submissions, the best scored 55.5% — ~20% higher than 2023’s top scorer, albeit short of the 85%, “human-level” threshold required to win.

This doesn’t mean we’re ~20% closer to AGI, though, Knoop says.

In a blog post, Knoop said that many of the submissions to ARC-AGI have been able to “brute force” their way to a solution, suggesting that a “large fraction” of ARC-AGI tasks “[don’t] carry much useful signal towards general intelligence.”

ARC-AGI consists of puzzle-like problems where an AI has to, given a grid of different-colored squares, generate the correct “answer” grid. The problems were designed to force an AI to adapt to new problems it hasn’t seen before. But it’s not clear they’re achieving this.

Tasks in the ARC-AGI benchmark. Models must solve ‘problems’ in the top row; the bottom row shows solutions. Image Credits:ARC-AGI

“[ARC-AGI] has been unchanged since 2019 and is not perfect,” Knoop acknowledged in his post.

Francois and Knoop have also faced criticism for overselling ARC-AGI as benchmark toward AGI — at a time when the very definition of AGI is being hotly contested. One OpenAI staff member recently claimed that AGI has “already” been achieved if one defines AGI as AI “better than most humans at most tasks.”

Knoop and Chollet say that they plan to release a second-gen ARC-AGI benchmark to address these issues, alongside a 2025 competition. “We will continue to direct the efforts of the research community towards what we see as the most important unsolved problems in AI, and accelerate the timeline to AGI,” Chollet wrote in an X post.

Fixes likely won’t come easy. If the first ARC-AGI test’s shortcomings are any indication, defining intelligence for AI will be as intractable — and inflammatory — as it has been for human beings.





Source link

Lisa Holden
Lisa Holden
Lisa Holden is a news writer for LinkDaddy News. She writes health, sport, tech, and more. Some of her favorite topics include the latest trends in fitness and wellness, the best ways to use technology to improve your life, and the latest developments in medical research.

Recent posts

Related articles

Republican Congressman Jim Jordan asks Big Tech if Biden tried to censor AI

On Thursday, House Judiciary Chair Jim Jordan (R-OH) sent letters to 16 American technology firms, including Google...

Bench is charging people for services they already paid for, some customers say

After Employer.com acquired bankrupt accounting startup Bench in a fire-sale late last year, CEO Jesse Tinsley pledged...

AI coding assistant Cursor reportedly tells a ‘vibe coder’ to write his own damn code

As businesses race to replace humans with AI “agents,” coding assistant Cursor may have given us a...

Profitable Klarna files for a potentially blockbuster IPO

Swedish fintech Klarna took the next step in its highly anticipated U.S. IPO on Friday when it...

Google is replacing Google Assistant with Gemini

Google will replace Google Assistant on Android phones with Gemini later this year, the company announced on...

‘Open’ model licenses often carry concerning restrictions

This week, Google released a family of open AI models, Gemma 3, that quickly garnered praise for...

Testing the Uber-Waymo robotaxi, Rivian goes hands-free, and Travis Kalanick has AV FOMO 

Welcome back to TechCrunch Mobility — your central hub for news and insights on the future of...

Tern AI’s low-cost GPS alternative actually works

We’ve all experienced that moment of frustration when the GPS glitches and you miss an exit on...