Independent · Source-cited · No sponsored rankingsSources linked in every article
AIToolRanked
AI News · 5 min read

GPT-6 Astra tops Deepnote's Slack agent test

Deepnote's Agent Initiative Benchmark puts GPT-6 Astra at 8.97 of 10. Sol scores 8.65 at $0.112 a task. Two LLM judges scored a vendor test.

RA
· Founder, AIToolRanked
Twelve dots on a dark field, two of them larger and brighter than the rest
On this page

What Deepnote measured

Deepnote published the Agent Initiative Benchmark ↗: 12 models, one harness, and a synthetic company called PageMantle. The page has no dateline. GitHub ↗ was pushed on November 13, 2026 at 07:49 UTC. The dataset is on Hugging Face ↗.

The picture above is ours.

PageMantle is a fake website-builder: six years of records, about 10 GB, and 11 services (Slack, Notion, Linear, HubSpot, Stripe, Google Calendar, PostHog, Google Analytics, Search Console, BigQuery, Snowflake). Every model gets the same 50 situations, the same tools, high effort, and a 180-second budget of model response time. Tool time is excluded. Two LLM evaluators, blind to which model they are grading and to each other, score each run from 0 to 10 against a reference solution. The band is a 95% task-bootstrap interval.

Deepnote sells that kind of workspace, and the page says this version compares models, not harnesses. The table is a vendor benchmark with LLM judges.

The 50 situations split into eight types: Find, Explain, Analyze, Verify, Diagnose, Decide, Coordinate, and Guard. Forty needed a message. Ten called for silence. "Helped" is out of 40. "Stayed silent" is out of 10, and holding back on a task that needed help is a miss.

How the 12 models scored

Numbers below are from the results table on the page, read on November 14, 2026. Time is model response time. Cost is recorded model spend per task.

ModelScoreHelpedStayed silentTimeCost / taskTool calls
GPT-6 Astra8.9740/408/1054 s$0.69319.3
GPT-6.1 Sol8.6539/408/1054 s$0.11218.0
Claude Fable 5.28.1740/406/101 min 0 s$1.8720.9
Claude Opus 5.57.8440/408/1029 s$1.2810.0
Claude Fable 5.17.5040/406/1051 s$3.0214.5
Claude Sonnet 5.56.6537/406/1015 s$0.4208.9
GPT-6 Luna6.3838/404/1050 s$0.007013.7
DeepSeek V4.1 Flash6.3538/404/101 min 26 s$0.02621.1
Mistral Large 46.0937/404/101 min 25 s$0.08222.5
Claude Haiku 5.56.0538/405/1028 s$0.07512.1
Gemini 4 Argon5.6023/4010/102 min 44 s$0.40334.5
Grok 4.74.5124/406/102 min 25 s$0.42535.5

Astra's 95% interval is 8.30 to 9.52. Sol's is 7.88 to 9.31. Fable 5.2's is 7.34 to 8.89. Deepnote's own sentence: "GPT-6.1 Sol at 8.65 sits inside Astra's interval, so this task set does not separate them." Fable 5.2 is below Astra's interval and inside Sol's. Deepnote says Fable 5.2 leads Decide tasks and has not passed either OpenAI model overall. Astra leads five of the eight types, including Analyze and Guard. Sol leads Explain and Find.

The zero column, from the same writeup: Astra and Opus each scored 0 on two tasks. Gemini 4 Argon scored 0 on 17. Grok 4.7 scored 0 on 20.

The published list is those 12 names. Gemini 4 Carbon and Gemini 4 Hydrogen are not on it. Carbon's launch is Gemini 4 Carbon is out. GPT-6 in an earlier story on this desk is the cyber model, GPT-6 Cyber, not Astra, Sol, or Luna.

Fable 5.2 at $1.87 is 17 times Deepnote's rounded $0.11 for Sol. The table lists Sol at $0.112. Deepnote says Sol sits 0.32 below Astra at one-sixth of the price. A month of one investigation per Slack thread in PageMantle, about 2,500 threads, is $281 on Sol, $1,732 on Astra, and $4,684 on Fable 5.2. The $1.87 is measured spend on this harness, not Anthropic's token price. That price cut is Claude Fable 5.2.

Opus 5.5 answers in 29 seconds with 10.0 tool calls at $1.28, and keeps 8 of 10 silences. Luna is $0.0070 and stays silent on 4 of 10. DeepSeek V4.1 Flash is $0.026 at 6.35, the model behind DeepSeek V4.1 Flash Ultrafast. Mistral Large 4 (Le Chonk in Mistral Studio) scored 6.09.

What people are saying

  • "We *just* open-sourced a benchmark on proactive agents in a workspace context - neither Grok nor Argon shone." — Jakub Jurovych ↗, of Deepnote

  • "Interestingly, the latest or the most expensive models did not perform the best." — Laura Probstner ↗, who wrote that the test was done at Deepnote

  • "a proactive agent that reads every thread can't cost $3 a read" — Chetaslua ↗. The dearest row is Fable 5.1 at $3.02 a task. The quote is his, not Deepnote's.

  • "Cost per point may matter more than the top score." — Edlucy ↗

  • "Restraint is a capability we barely measure. It decides whether an agent is a colleague or noise." — Veth ↗

Reddit was not reachable for this story.

Argon stayed silent 10/10 and helped on only 23 of 40. Jakub works at Deepnote, and that line is his summary of his company's table.

Our take

On this test, Sol is the model to try first for an always-on Slack watcher. It is 0.32 points under Astra, inside Astra's interval, at $0.112 instead of $0.693, and it helped on 39 of 40. Astra is the top printed score. The page says the task set does not separate them. Paying Astra's rate for a gap the authors will not call is a choice, not a result.

Do not put Fable 5.2 on every thread. It helped 40/40, it leads Decide, and it costs $1.87, which Deepnote prices at $4,684 for that 2,500-thread month. Use it when the job is a decision and the bill is acceptable. Opus 5.5 is the row for a live conversation that has to come back in about half a minute. Luna is the row for cheap triage, with a 6.38 and weak silence.

Do not promote the ranking into a purchase. Two unnamed LLM judges scored one synthetic company on the vendor's own question. Grok 4.7 at 4.51 and Argon at 5.60 are this harness, not a verdict on Grok Bot in meetings or Claude Meetings. Other model writeups sit under LLM comparisons.

More from the news desk · About Rai Ansar