devadmin/OpenCodeGoEvals
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: weekly evaluation refresh 2026-10-26

Browse Source
This commit is contained in:
opencode-go-evals
2026-10-26 05:24:20 +00:00
parent 99489ca17b
commit f3259fdc0a
6 changed files with 1508 additions and 1184 deletions
Download Patch File Download Diff File Expand all files Collapse all files
MODELS.md
+59 -19
View File
@@ -5,7 +5,7 @@ Short, practical notes on every model in the OpenCode Go catalogue: what each on
what it costs, and what to watch out for. what it costs, and what to watch out for.
Evaluated scores come from Artificial Analysis (single consistent harness) plus vendor model cards. Evaluated scores come from Artificial Analysis (single consistent harness) plus vendor model cards.
Catalogue metadata comes from models.dev's OpenCode Go provider. Data snapshot: **2026-10-21**. Catalogue metadata comes from models.dev's OpenCode Go provider. Data snapshot: **2026-10-26**.
The interactive version with the full task matrix lives at `site/index.html`. The interactive version with the full task matrix lives at `site/index.html`.
> No model in the Go catalogue generates images. Every Go model is text-output; several accept image, > No model in the Go catalogue generates images. Every Go model is text-output; several accept image,
@@ -28,6 +28,16 @@ The interactive version with the full task matrix lives at `site/index.html`.
### undefined ### undefined
**moonshotai** · released 2026-10-23 · $2 in / $10 out per 1M tokens · 1.05M context · text, image, video in → text out
Multimodal Kimi model with 1M context and always-on reasoning for coding, 3D creation, financial research, and agentic work
_No hand-written write-up yet._
*Evaluated:* Intelligence Index 55.1 · FrontierCode 47.1 · TB4.0 67.2 · AutomationBench 73.3 · HLE 52.5 · LCR 86.3 · MMMU-Pro 83.9
### undefined
**google** · released 2026-10-13 · $4 in / $20 out per 1M tokens · 1.05M context · text, image, video, audio, pdf in → text out **google** · released 2026-10-13 · $4 in / $20 out per 1M tokens · 1.05M context · text, image, video, audio, pdf in → text out
Targeted for: **Frontier coding, Deep research, Legal & financial work, Security, Multimodal** Targeted for: **Frontier coding, Deep research, Legal & financial work, Security, Multimodal**
@@ -46,6 +56,16 @@ Hardest frontier work: large codebase refactors and optimisation (Google used it
### undefined ### undefined
**google** · released 2026-10-13 · $2 in / $10 out per 1M tokens · 1.05M context · text, image, video, audio, pdf in → text out
Google's flagship reasoning model for complex software engineering, deep math and science, and long-horizon autonomous agents
_No hand-written write-up yet._
*Evaluated:* Intelligence Index 52.6 · FrontierCode 43.5 · TB4.0 57.1 · AutomationBench 77.5 · HLE 57.1 · LCR 79.7
### undefined
**meta** · released 2026-09-02 · $0.1 in / $0.2 out per 1M tokens · 1.05M context · text, image, video, pdf, audio in → text out **meta** · released 2026-09-02 · $0.1 in / $0.2 out per 1M tokens · 1.05M context · text, image, video, pdf, audio in → text out
Targeted for: **Long-horizon agents, Multi-agent workflows, Coding, Multimodal, Cheap** Targeted for: **Long-horizon agents, Multi-agent workflows, Coding, Multimodal, Cheap**
@@ -183,6 +203,16 @@ Anything that has to answer fast and often: interactive completions, classificat
### undefined ### undefined
**minimax** · released 2026-10-19 · $0.15 in / $0.6 out per 1M tokens · 1M context · text, image, video in → text out
Frontier multimodal coding model with 1M context window and tunable thinking depth
_No hand-written write-up yet._
*Evaluated:* Intelligence Index 42.7 · FrontierCode 38.6 · TB4.0 25.8 · AutomationBench 72.5 · HLE 48 · LCR 79.7 · MMMU-Pro 79.1
### undefined
**minimax** · released 2026-10-19 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1M context · text, image, video in → text out **minimax** · released 2026-10-19 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1M context · text, image, video in → text out
Targeted for: **Coding, Multimodal, Agentic, Free tier** Targeted for: **Coding, Multimodal, Agentic, Free tier**
@@ -200,7 +230,7 @@ Day-to-day coding-agent work on real repositories: reading long codebases, editi
### undefined ### undefined
**zhipuai** · released 2026-08-26 · $0.15 in / $0.5 out per 1M tokens · 1M context · text, image, video, pdf in → text out **zhipuai** · released 2026-08-26 · $0.15 in / $0.5 out per 1M tokens · 1M context · text, image, video in → text out
Targeted for: **Efficient coding, Long-horizon agents, Multimodal, Cheap** Targeted for: **Efficient coding, Long-horizon agents, Multimodal, Cheap**
@@ -360,7 +390,7 @@ Reflection's first open-weight model: a 501B-parameter sparse Mixture-of-Experts
Cost-efficient coding and agent loops. The MoE design activates only 23B parameters per token, so it is a cheap way to run a lot of terminal work, tool calls and multi-step reasoning. Text-only, so it is a pure code/reasoning worker rather than a perception model. Cost-efficient coding and agent loops. The MoE design activates only 23B parameters per token, so it is a cheap way to run a lot of terminal work, tool calls and multi-step reasoning. Text-only, so it is a pure code/reasoning worker rather than a perception model.
*Evaluated:* Intelligence Index 30.3 · TB4.0 3 · AutomationBench 48.5 · HLE 34.7 · LCR 78 *Evaluated:* Intelligence Index 30.3 · FrontierCode 24.3 · TB4.0 3 · AutomationBench 48.5 · HLE 34.7 · LCR 78
**Caveats** **Caveats**
@@ -505,20 +535,20 @@ Long-context reasoning with tool calling at a modest price — useful for repo-s
### undefined ### undefined
**tencent** · released 2026-08-28 · $0.834 in / $2.501 out per 1M tokens · 1.02M context · text in → text out **undisclosed** · released 2026-10-12 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1.05M context · text in → text out
Targeted for: **Agentic workflows, Productivity, Complex task execution, Open weights** Targeted for: **Free tier, Coding, Agentic workflows, Structured output**
A next-generation productivity model with significantly enhanced agent and complex task execution capabilities. A text-only reasoning model for coding and agentic workflows, with tool calling and structured JSON outputs.
Productivity and workflow automation: multi-step office/enterprise tasks, tool orchestration and long-running assistants. Tencent positions it around agentic capability and complex task execution rather than raw chat quality, with a 1M window for large corpora. An anonymous free tier aimed at agents: tool calling and structured JSON output make it useful as a sub-agent or a formatter that has to hand machine-readable results to the next step.
*Evaluated:* no public third-party evaluation published yet. *Evaluated:* no public third-party evaluation published yet.
**Caveats** **Caveats**
- Preview release with no third-party evaluations published yet, so performance is unverified outside the vendor's own numbers. - No vendor, no model card, no public evaluation — verify behaviour before depending on it.
- Text input only. - Not listed in the live Go catalogue endpoint.
### undefined ### undefined
@@ -538,6 +568,23 @@ Same model as the paid Step 5 Preview, without the token bill — useful for lon
### undefined ### undefined
**tencent** · released 2026-08-28 · $0.834 in / $2.501 out per 1M tokens · 1.02M context · text in → text out
Targeted for: **Agentic workflows, Productivity, Complex task execution, Open weights**
A next-generation productivity model with significantly enhanced agent and complex task execution capabilities.
Productivity and workflow automation: multi-step office/enterprise tasks, tool orchestration and long-running assistants. Tencent positions it around agentic capability and complex task execution rather than raw chat quality, with a 1M window for large corpora.
*Evaluated:* no public third-party evaluation published yet.
**Caveats**
- Preview release with no third-party evaluations published yet, so performance is unverified outside the vendor's own numbers.
- Text input only.
### undefined
**meituan** · released 2026-09-25 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1M context · text, image in → text out **meituan** · released 2026-09-25 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1M context · text, image in → text out
Targeted for: **Coding, Agentic, Image understanding, Free tier** Targeted for: **Coding, Agentic, Image understanding, Free tier**
@@ -573,21 +620,14 @@ An unreleased preview model with full multimodal input and tool use, priced like
### undefined ### undefined
**undisclosed** · released 2026-10-12 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 1.05M context · text in → text out **undisclosed** · released 2026-10-24 · $0 in / $0 out per 1M tokens · **free tier (limited time)** · 0.26M context · text in → text out
Targeted for: **Free tier, Coding, Agentic workflows, Structured output** Stealth model, free for a limited time
A text-only reasoning model for coding and agentic workflows, with tool calling and structured JSON outputs. _No hand-written write-up yet._
An anonymous free tier aimed at agents: tool calling and structured JSON output make it useful as a sub-agent or a formatter that has to hand machine-readable results to the next step.
*Evaluated:* no public third-party evaluation published yet. *Evaluated:* no public third-party evaluation published yet.
**Caveats**
- No vendor, no model card, no public evaluation — verify behaviour before depending on it.
- Not listed in the live Go catalogue endpoint.
## Free models outside the Go plan ## Free models outside the Go plan
### MiMo-V2.6-Flash Free (Xiaomi) ### MiMo-V2.6-Flash Free (Xiaomi)
site/assets/catalog.js
+447 -375
View File
@@ -2,227 +2,35 @@
// Source: models.dev/api.json (provider "opencode-go") + Artificial Analysis. // Source: models.dev/api.json (provider "opencode-go") + Artificial Analysis.
export const CATALOG = [ export const CATALOG = [
{ {
"id": "hy4-preview", "id": "mimo-v2.5",
"n": "Hy4 preview", "n": "MiMo V2.5",
"lab": "tencent", "lab": "xiaomi",
"released": "2026-08-28", "released": "2026-04-22",
"price": { "price": {
"in": 0.834, "in": 0.14,
"out": 2.501 "out": 0.28
}, },
"free": false, "free": false,
"ctx": 1024000, "ctx": 1000000,
"modalities": "text in → text out", "modalities": "text, image, audio, video in → text out",
"description": "A next-generation productivity model with significantly enhanced Agent and complex task execution capabilities.", "description": "MiMo omni model for text, image, video, audio, and agents",
"canonical": "tencent/hy4-preview", "canonical": "xiaomi/mimo-v2.5",
"openWeights": true "openWeights": true
}, },
{ {
"id": "claude-haiku-5-5", "id": "gpt-6-luna",
"n": "Claude Haiku 5.5", "n": "GPT-6 Luna",
"lab": "anthropic", "lab": "openai",
"released": "2026-10-07", "released": "2026-09-22",
"price": { "price": {
"in": 0.1, "in": 0.1,
"out": 0.5 "out": 0.5
}, },
"free": false, "free": false,
"ctx": 1000000,
"modalities": "text, image, pdf in → text out",
"description": "Fast Claude model for responsive assistance, classification, and lightweight agents",
"canonical": "anthropic/claude-haiku-5-5",
"openWeights": false
},
{
"id": "step-5-preview-free",
"n": "Step 5 Preview Free",
"lab": "stepfun",
"released": "2026-09-16",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "StepFun's next-generation flagship base model for coding and professional knowledge work, with native text, image, and video input and a 1M-token context window",
"canonical": "stepfun/step-5-preview",
"openWeights": true
},
{
"id": "deepseek-v4.1-flash",
"n": "DeepSeek V4.1 Flash",
"lab": "deepseek",
"released": "2026-09-10",
"price": {
"in": 0.15,
"out": 0.6
},
"free": false,
"ctx": 1000000,
"modalities": "text, image in → text out",
"description": "DeepSeek V4.1 Flash model for reasoning and agentic coding",
"canonical": "deepseek/deepseek-v4.1-flash",
"openWeights": true
},
{
"id": "longcat-2.5-preview-free",
"n": "LongCat 2.5 Preview Free",
"lab": "meituan",
"released": "2026-09-25",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image in → text out",
"description": "Meituan's multimodal reasoning model for coding and agent tasks, with image understanding and a 1M-token context window",
"canonical": "meituan/longcat-2.5-preview",
"openWeights": false
},
{
"id": "minimax-m2.7",
"n": "MiniMax-M2.7",
"lab": "minimax",
"released": "2026-03-18",
"price": {
"in": 0.3,
"out": 1.2
},
"free": false,
"ctx": 204800,
"modalities": "text in → text out",
"description": "MiniMax model for chat, coding, office work, and agentic tasks",
"canonical": "minimax/MiniMax-M2.7",
"openWeights": true
},
{
"id": "mimo-v2.6-pro",
"n": "MiMo-V2.6-Pro",
"lab": "xiaomi",
"released": "2026-09-22",
"price": {
"in": 0.435,
"out": 0.87
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, audio, video in → text out",
"description": "Stronger MiMo Pro tier for multimodal reasoning and coding-agent execution",
"canonical": "xiaomi/mimo-v2.6-pro",
"openWeights": true
},
{
"id": "kimi-k3",
"n": "Kimi K3",
"lab": "moonshotai",
"released": "2026-07-16",
"price": {
"in": 3,
"out": 15
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video in → text out",
"description": "Multimodal Kimi model with 1M context and toggleable max-effort thinking for long-horizon agent work",
"canonical": "moonshotai/kimi-k3",
"openWeights": true
},
{
"id": "hy3",
"n": "Hy3",
"lab": "tencent",
"released": "2026-07-06",
"price": {
"in": 0.14,
"out": 0.58
},
"free": false,
"ctx": 256000,
"modalities": "text in → text out",
"description": "Tencent Hy reasoning model for coding, instruction following, and agent tasks",
"canonical": "tencent/hy3",
"openWeights": true
},
{
"id": "mimo-v2.5-pro",
"n": "MiMo V2.5 Pro",
"lab": "xiaomi",
"released": "2026-04-22",
"price": {
"in": 0.435,
"out": 0.87
},
"free": false,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "MiMo pro model for strong multimodal reasoning and agent execution",
"canonical": "xiaomi/mimo-v2.5-pro",
"openWeights": true
},
{
"id": "beam-free",
"n": "Beam Free",
"lab": "reflection",
"released": "2026-10-05",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "Sparse mixture-of-experts reasoning model with 23B active parameters for coding and agentic workflows",
"canonical": "reflection/beam-1.0-501b-a23b",
"openWeights": true
},
{
"id": "qwen3.8-flash",
"n": "Qwen3.8 Flash",
"lab": "alibaba",
"released": "2026-08-26",
"price": {
"in": 0.15,
"out": 0.47
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Qwen vision-language model for visual reasoning, documents, and agent tasks",
"canonical": "alibaba/qwen3.8-flash",
"openWeights": false
},
{
"id": "qwen3.8-max",
"n": "Qwen3.8 Max",
"lab": "alibaba",
"released": "2026-08-03",
"price": {
"in": 2,
"out": 6
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "2.4-trillion-parameter multimodal flagship for coding, professional work, and long-horizon agentic workflows",
"canonical": "alibaba/qwen3.8-max",
"openWeights": false
},
{
"id": "gpt-5.6-luna",
"n": "GPT-5.6 Luna",
"lab": "openai",
"released": "2026-07-09",
"price": {
"in": 0.2,
"out": 1.2
},
"free": false,
"ctx": 1050000, "ctx": 1050000,
"modalities": "text, image, pdf in → text out", "modalities": "text, image, pdf in → text out",
"description": "Cost-efficient GPT-5.6 model for fast, high-volume workloads", "description": "OpenAI's most efficient model for focused, high-volume tasks",
"canonical": "openai/gpt-5.6-luna", "canonical": "openai/gpt-6-luna",
"openWeights": false "openWeights": false
}, },
{ {
@@ -242,35 +50,51 @@ export const CATALOG = [
"openWeights": true "openWeights": true
}, },
{ {
"id": "glm-5.2", "id": "hy3",
"n": "GLM-5.2", "n": "Hy3",
"lab": "zhipuai", "lab": "tencent",
"released": "2026-06-13", "released": "2026-07-06",
"price": { "price": {
"in": 1.4, "in": 0.14,
"out": 4.4 "out": 0.58
}, },
"free": false, "free": false,
"ctx": 1000000, "ctx": 256000,
"modalities": "text in → text out", "modalities": "text in → text out",
"description": "Open flagship GLM for long-horizon coding agents and million-token context work", "description": "Tencent Hy reasoning model for coding, instruction following, and agent tasks",
"canonical": "zhipuai/glm-5.2", "canonical": "tencent/hy3",
"openWeights": true "openWeights": true
}, },
{ {
"id": "deepseek-v4-pro", "id": "cat-jam-free",
"n": "DeepSeek V4 Pro (New)", "n": "Cat Jam Free",
"lab": "deepseek", "lab": "",
"released": "2026-04-24", "released": "2026-10-12",
"price": { "price": {
"in": 0.66, "in": 0,
"out": 1.98 "out": 0
},
"free": true,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "A text-only reasoning model for coding and agentic workflows, with tool calling and structured JSON outputs.",
"canonical": null,
"openWeights": false
},
{
"id": "mimo-v2.6-pro",
"n": "MiMo-V2.6-Pro",
"lab": "xiaomi",
"released": "2026-09-22",
"price": {
"in": 0.435,
"out": 0.87
}, },
"free": false, "free": false,
"ctx": 1000000, "ctx": 1048576,
"modalities": "text in → text out", "modalities": "text, image, audio, video in → text out",
"description": "Flagship DeepSeek model for coding, reasoning, and agentic work", "description": "Stronger MiMo Pro tier for multimodal reasoning and coding-agent execution",
"canonical": "deepseek/deepseek-v4-pro", "canonical": "xiaomi/mimo-v2.6-pro",
"openWeights": true "openWeights": true
}, },
{ {
@@ -289,54 +113,6 @@ export const CATALOG = [
"canonical": "meta/muse-spark-1.3", "canonical": "meta/muse-spark-1.3",
"openWeights": false "openWeights": false
}, },
{
"id": "minimax-m3.1-flash-free",
"n": "MiniMax M3.1 Flash Free",
"lab": "minimax",
"released": "2026-10-19",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Frontier multimodal coding model with 1M context window and tunable thinking depth",
"canonical": "minimax/MiniMax-M3.1-Flash",
"openWeights": false
},
{
"id": "grok-4.7",
"n": "Grok 4.7",
"lab": "xai",
"released": "2026-09-21",
"price": {
"in": 2,
"out": 6
},
"free": false,
"ctx": 500000,
"modalities": "text, image, pdf in → text out",
"description": "xAI's frontier model for long-running agents, coding, knowledge work, and visual projects",
"canonical": "xai/grok-4.7",
"openWeights": false
},
{
"id": "qwen3.7-plus",
"n": "Qwen3.7 Plus",
"lab": "alibaba",
"released": "2026-06-02",
"price": {
"in": 0.4,
"out": 1.6
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Multimodal reasoning model for visual analysis, planning, and tool use",
"canonical": "alibaba/qwen3.7-plus",
"openWeights": false
},
{ {
"id": "gemini-4-argon-contributor", "id": "gemini-4-argon-contributor",
"n": "Gemini 4 Argon Contributor", "n": "Gemini 4 Argon Contributor",
@@ -353,6 +129,150 @@ export const CATALOG = [
"canonical": "google/gemini-4-argon", "canonical": "google/gemini-4-argon",
"openWeights": false "openWeights": false
}, },
{
"id": "step-5-preview-free",
"n": "Step 5 Preview Free",
"lab": "stepfun",
"released": "2026-09-16",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "StepFun's next-generation flagship base model for coding and professional knowledge work, with native text, image, and video input and a 1M-token context window",
"canonical": "stepfun/step-5-preview",
"openWeights": true
},
{
"id": "gpt-5.6-luna",
"n": "GPT-5.6 Luna",
"lab": "openai",
"released": "2026-07-09",
"price": {
"in": 0.2,
"out": 1.2
},
"free": false,
"ctx": 1050000,
"modalities": "text, image, pdf in → text out",
"description": "Cost-efficient GPT-5.6 model for fast, high-volume workloads",
"canonical": "openai/gpt-5.6-luna",
"openWeights": false
},
{
"id": "kimi-k3.1-pro",
"n": "Kimi K3.1 Pro",
"lab": "moonshotai",
"released": "2026-10-23",
"price": {
"in": 2,
"out": 10
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video in → text out",
"description": "Multimodal Kimi model with 1M context and always-on reasoning for coding, 3D creation, financial research, and agentic work",
"canonical": "moonshotai/kimi-k3.1-pro",
"openWeights": false
},
{
"id": "qwen3.8-flash",
"n": "Qwen3.8 Flash",
"lab": "alibaba",
"released": "2026-08-26",
"price": {
"in": 0.15,
"out": 0.47
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Qwen vision-language model for visual reasoning, documents, and agent tasks",
"canonical": "alibaba/qwen3.8-flash",
"openWeights": false
},
{
"id": "step-5-preview",
"n": "Step 5 Preview",
"lab": "stepfun",
"released": "2026-09-16",
"price": {
"in": 1,
"out": 2.7
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "StepFun's next-generation flagship base model for coding and professional knowledge work, with native text, image, and video input and a 1M-token context window",
"canonical": "stepfun/step-5-preview",
"openWeights": true
},
{
"id": "qwen3.8-max",
"n": "Qwen3.8 Max",
"lab": "alibaba",
"released": "2026-08-03",
"price": {
"in": 2,
"out": 6
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "2.4-trillion-parameter multimodal flagship for coding, professional work, and long-horizon agentic workflows",
"canonical": "alibaba/qwen3.8-max",
"openWeights": false
},
{
"id": "hy4-preview",
"n": "Hy4 preview",
"lab": "tencent",
"released": "2026-08-28",
"price": {
"in": 0.834,
"out": 2.501
},
"free": false,
"ctx": 1024000,
"modalities": "text in → text out",
"description": "A next-generation productivity model with significantly enhanced Agent and complex task execution capabilities.",
"canonical": "tencent/hy4-preview",
"openWeights": true
},
{
"id": "gemini-4-argon",
"n": "Gemini 4 Argon",
"lab": "google",
"released": "2026-10-13",
"price": {
"in": 2,
"out": 10
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video, audio, pdf in → text out",
"description": "Google's flagship reasoning model for complex software engineering, deep math and science, and long-horizon autonomous agents",
"canonical": "google/gemini-4-argon",
"openWeights": false
},
{
"id": "glm-5.2",
"n": "GLM-5.2",
"lab": "zhipuai",
"released": "2026-06-13",
"price": {
"in": 1.4,
"out": 4.4
},
"free": false,
"ctx": 1000000,
"modalities": "text in → text out",
"description": "Open flagship GLM for long-horizon coding agents and million-token context work",
"canonical": "zhipuai/glm-5.2",
"openWeights": true
},
{ {
"id": "glm-5.3", "id": "glm-5.3",
"n": "GLM-5.3", "n": "GLM-5.3",
@@ -380,107 +300,43 @@ export const CATALOG = [
}, },
"free": false, "free": false,
"ctx": 1000000, "ctx": 1000000,
"modalities": "text, image, video, pdf in → text out", "modalities": "text, image, video in → text out",
"description": "Native multimodal GLM model for efficient coding and long-horizon agent tasks", "description": "Native multimodal GLM model for efficient coding and long-horizon agent tasks",
"canonical": "zhipuai/glm-5.3-flash", "canonical": "zhipuai/glm-5.3-flash",
"openWeights": true "openWeights": true
}, },
{ {
"id": "space-bunny", "id": "qwen3.7-plus",
"n": "Space Bunny", "n": "Qwen3.7 Plus",
"lab": "", "lab": "alibaba",
"released": "2026-09-23", "released": "2026-06-02",
"price": {
"in": 0.4,
"out": 1.6
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Multimodal reasoning model for visual analysis, planning, and tool use",
"canonical": "alibaba/qwen3.7-plus",
"openWeights": false
},
{
"id": "deepseek-v4.1-flash",
"n": "DeepSeek V4.1 Flash",
"lab": "deepseek",
"released": "2026-09-10",
"price": { "price": {
"in": 0.15, "in": 0.15,
"out": 0.6 "out": 0.6
}, },
"free": false, "free": false,
"ctx": 1048576,
"modalities": "text, image, video in → text out",
"description": "Anonymous preview reasoning model for coding, agentic tasks, tool use, and multimodal input",
"canonical": null,
"openWeights": false
},
{
"id": "kimi-k2.7-code",
"n": "Kimi K2.7 Code",
"lab": "moonshotai",
"released": "2026-06-12",
"price": {
"in": 0.95,
"out": 4
},
"free": false,
"ctx": 262144,
"modalities": "text, image, video in → text out",
"description": "Coding-focused Kimi model, stronger on long-horizon repo work with less overthinking",
"canonical": "moonshotai/kimi-k2.7-code",
"openWeights": true
},
{
"id": "cat-jam-free",
"n": "Cat Jam Free",
"lab": "",
"released": "2026-10-12",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "A text-only reasoning model for coding and agentic workflows, with tool calling and structured JSON outputs.",
"canonical": null,
"openWeights": false
},
{
"id": "gpt-6-luna",
"n": "GPT-6 Luna",
"lab": "openai",
"released": "2026-09-22",
"price": {
"in": 0.1,
"out": 0.5
},
"free": false,
"ctx": 1050000,
"modalities": "text, image, pdf in → text out",
"description": "OpenAI's most efficient model for focused, high-volume tasks",
"canonical": "openai/gpt-6-luna",
"openWeights": false
},
{
"id": "mimo-v2.5",
"n": "MiMo V2.5",
"lab": "xiaomi",
"released": "2026-04-22",
"price": {
"in": 0.14,
"out": 0.28
},
"free": false,
"ctx": 1000000, "ctx": 1000000,
"modalities": "text, image, audio, video in → text out", "modalities": "text, image in → text out",
"description": "MiMo omni model for text, image, video, audio, and agents", "description": "DeepSeek V4.1 Flash model for reasoning and agentic coding",
"canonical": "xiaomi/mimo-v2.5", "canonical": "deepseek/deepseek-v4.1-flash",
"openWeights": true "openWeights": true
}, },
{
"id": "muse-spark-1.2-contributor",
"n": "Muse Spark 1.2 Contributor",
"lab": "meta",
"released": "2026-08-05",
"price": {
"in": 0.1,
"out": 0.2
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video, pdf, audio in → text out",
"description": "Muse Spark 1.2 is a coding-focused update to Muse Spark 1.1 with improvements in code generation, complex debugging, codebase understanding, and end-to-end developer workflows.",
"canonical": "meta/muse-spark-1.2",
"openWeights": false
},
{ {
"id": "mimo-v2.6-flash", "id": "mimo-v2.6-flash",
"n": "MiMo-V2.6-Flash", "n": "MiMo-V2.6-Flash",
@@ -497,6 +353,22 @@ export const CATALOG = [
"canonical": "xiaomi/mimo-v2.6-flash", "canonical": "xiaomi/mimo-v2.6-flash",
"openWeights": true "openWeights": true
}, },
{
"id": "kimi-k2.7-code",
"n": "Kimi K2.7 Code",
"lab": "moonshotai",
"released": "2026-06-12",
"price": {
"in": 0.95,
"out": 4
},
"free": false,
"ctx": 262144,
"modalities": "text, image, video in → text out",
"description": "Coding-focused Kimi model, stronger on long-horizon repo work with less overthinking",
"canonical": "moonshotai/kimi-k2.7-code",
"openWeights": true
},
{ {
"id": "longcat-2.0", "id": "longcat-2.0",
"n": "LongCat-2.0", "n": "LongCat-2.0",
@@ -514,32 +386,232 @@ export const CATALOG = [
"openWeights": true "openWeights": true
}, },
{ {
"id": "step-5-preview", "id": "minimax-m3.1-flash",
"n": "Step 5 Preview", "n": "MiniMax M3.1 Flash",
"lab": "stepfun", "lab": "minimax",
"released": "2026-09-16", "released": "2026-10-19",
"price": { "price": {
"in": 1, "in": 0.15,
"out": 2.7 "out": 0.6
}, },
"free": false, "free": false,
"ctx": 1000000, "ctx": 1000000,
"modalities": "text, image, video in → text out", "modalities": "text, image, video in → text out",
"description": "StepFun's next-generation flagship base model for coding and professional knowledge work, with native text, image, and video input and a 1M-token context window", "description": "Frontier multimodal coding model with 1M context window and tunable thinking depth",
"canonical": "stepfun/step-5-preview", "canonical": "minimax/MiniMax-M3.1-Flash",
"openWeights": false
},
{
"id": "claude-haiku-5-5",
"n": "Claude Haiku 5.5",
"lab": "anthropic",
"released": "2026-10-07",
"price": {
"in": 0.1,
"out": 0.5
},
"free": false,
"ctx": 1000000,
"modalities": "text, image, pdf in → text out",
"description": "Fast Claude model for responsive assistance, classification, and lightweight agents",
"canonical": "anthropic/claude-haiku-5-5",
"openWeights": false
},
{
"id": "longcat-2.5-preview-free",
"n": "LongCat 2.5 Preview Free",
"lab": "meituan",
"released": "2026-09-25",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image in → text out",
"description": "Meituan's multimodal reasoning model for coding and agent tasks, with image understanding and a 1M-token context window",
"canonical": "meituan/longcat-2.5-preview",
"openWeights": false
},
{
"id": "grok-4.7",
"n": "Grok 4.7",
"lab": "xai",
"released": "2026-09-21",
"price": {
"in": 2,
"out": 6
},
"free": false,
"ctx": 500000,
"modalities": "text, image, pdf in → text out",
"description": "xAI's frontier model for long-running agents, coding, knowledge work, and visual projects",
"canonical": "xai/grok-4.7",
"openWeights": false
},
{
"id": "mimo-v2.5-pro",
"n": "MiMo V2.5 Pro",
"lab": "xiaomi",
"released": "2026-04-22",
"price": {
"in": 0.435,
"out": 0.87
},
"free": false,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "MiMo pro model for strong multimodal reasoning and agent execution",
"canonical": "xiaomi/mimo-v2.5-pro",
"openWeights": true "openWeights": true
},
{
"id": "space-bunny",
"n": "Space Bunny",
"lab": "",
"released": "2026-09-23",
"price": {
"in": 0.15,
"out": 0.6
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video in → text out",
"description": "Anonymous preview reasoning model for coding, agentic tasks, tool use, and multimodal input",
"canonical": null,
"openWeights": false
},
{
"id": "kimi-k3",
"n": "Kimi K3",
"lab": "moonshotai",
"released": "2026-07-16",
"price": {
"in": 3,
"out": 15
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video in → text out",
"description": "Multimodal Kimi model with 1M context and toggleable max-effort thinking for long-horizon agent work",
"canonical": "moonshotai/kimi-k3",
"openWeights": true
},
{
"id": "minimax-m3.1-flash-free",
"n": "MiniMax M3.1 Flash Free",
"lab": "minimax",
"released": "2026-10-19",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1000000,
"modalities": "text, image, video in → text out",
"description": "Frontier multimodal coding model with 1M context window and tunable thinking depth",
"canonical": "minimax/MiniMax-M3.1-Flash",
"openWeights": false
},
{
"id": "minimax-m2.7",
"n": "MiniMax-M2.7",
"lab": "minimax",
"released": "2026-03-18",
"price": {
"in": 0.3,
"out": 1.2
},
"free": false,
"ctx": 204800,
"modalities": "text in → text out",
"description": "MiniMax model for chat, coding, office work, and agentic tasks",
"canonical": "minimax/MiniMax-M2.7",
"openWeights": true
},
{
"id": "iceman-contributor-free",
"n": "Iceman Contributor Free",
"lab": "",
"released": "2026-10-24",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 262144,
"modalities": "text in → text out",
"description": "Stealth model, free for a limited time",
"canonical": null,
"openWeights": false
},
{
"id": "beam-free",
"n": "Beam Free",
"lab": "reflection",
"released": "2026-10-05",
"price": {
"in": 0,
"out": 0
},
"free": true,
"ctx": 1048576,
"modalities": "text in → text out",
"description": "Sparse mixture-of-experts reasoning model with 23B active parameters for coding and agentic workflows",
"canonical": "reflection/beam-1.0-501b-a23b",
"openWeights": true
},
{
"id": "deepseek-v4-pro",
"n": "DeepSeek V4 Pro (New)",
"lab": "deepseek",
"released": "2026-04-24",
"price": {
"in": 0.66,
"out": 1.98
},
"free": false,
"ctx": 1000000,
"modalities": "text in → text out",
"description": "Flagship DeepSeek model for coding, reasoning, and agentic work",
"canonical": "deepseek/deepseek-v4-pro",
"openWeights": true
},
{
"id": "muse-spark-1.2-contributor",
"n": "Muse Spark 1.2 Contributor",
"lab": "meta",
"released": "2026-08-05",
"price": {
"in": 0.1,
"out": 0.2
},
"free": false,
"ctx": 1048576,
"modalities": "text, image, video, pdf, audio in → text out",
"description": "Muse Spark 1.2 is a coding-focused update to Muse Spark 1.1 with improvements in code generation, complex debugging, codebase understanding, and end-to-end developer workflows.",
"canonical": "meta/muse-spark-1.2",
"openWeights": false
} }
]; ];
export const SNAPSHOT = { export const SNAPSHOT = {
"generated": "2026-10-21T17:14:47.585Z", "generated": "2026-10-26T05:24:19.608Z",
"models": 33, "models": 37,
"newModels": [], "newModels": [
"kimi-k3.1-pro",
"gemini-4-argon",
"minimax-m3.1-flash",
"iceman-contributor-free"
],
"removedModels": [], "removedModels": [],
"deltas": [], "deltas": [],
"notInLiveCatalog": [ "notInLiveCatalog": [
"cat-jam-free",
"gemini-4-argon-contributor",
"step-5-preview-free", "step-5-preview-free",
"longcat-2.5-preview-free",
"space-bunny", "space-bunny",
"cat-jam-free" "minimax-m3.1-flash-free"
] ]
}; };
site/assets/evals.js
+782 -670
View File
File diff suppressed because it is too large Load Diff
site/assets/history.js
+36
View File
@@ -1,6 +1,42 @@
// AUTO-GENERATED by scripts/sync-catalog.mjs — do not edit by hand. // AUTO-GENERATED by scripts/sync-catalog.mjs — do not edit by hand.
// Rolling week-over-week snapshot of the Intelligence Index. // Rolling week-over-week snapshot of the Intelligence Index.
export const HISTORY = [ export const HISTORY = [
{
"date": "2026-10-26",
"scores": {
"mimo-v2.5": 21.8,
"gpt-6-luna": 38.1,
"minimax-m3": 29.2,
"hy3": 25.3,
"mimo-v2.6-pro": 46.3,
"muse-spark-1.3-contributor": 48.1,
"gemini-4-argon-contributor": 52.6,
"gpt-5.6-luna": 37.3,
"kimi-k3.1-pro": 55.1,
"qwen3.8-flash": 39.8,
"step-5-preview": 43.7,
"qwen3.8-max": 45.4,
"gemini-4-argon": 52.6,
"glm-5.2": 33.7,
"glm-5.3": 44.8,
"glm-5.3-flash": 41.8,
"qwen3.7-plus": 25.2,
"deepseek-v4.1-flash": 39.5,
"mimo-v2.6-flash": 37.9,
"kimi-k2.7-code": 25.8,
"longcat-2.0": 19.1,
"minimax-m3.1-flash": 42.7,
"claude-haiku-5-5": 43.4,
"grok-4.7": 46.4,
"mimo-v2.5-pro": 26,
"kimi-k3": 43.6,
"minimax-m3.1-flash-free": 42.7,
"minimax-m2.7": 22.8,
"beam-free": 30.3,
"deepseek-v4-pro": 36,
"muse-spark-1.2-contributor": 39.6
}
},
{ {
"date": "2026-10-21", "date": "2026-10-21",
"scores": { "scores": {
site/assets/history.json
+36
View File
@@ -1,4 +1,40 @@
[ [
{
"date": "2026-10-26",
"scores": {
"mimo-v2.5": 21.8,
"gpt-6-luna": 38.1,
"minimax-m3": 29.2,
"hy3": 25.3,
"mimo-v2.6-pro": 46.3,
"muse-spark-1.3-contributor": 48.1,
"gemini-4-argon-contributor": 52.6,
"gpt-5.6-luna": 37.3,
"kimi-k3.1-pro": 55.1,
"qwen3.8-flash": 39.8,
"step-5-preview": 43.7,
"qwen3.8-max": 45.4,
"gemini-4-argon": 52.6,
"glm-5.2": 33.7,
"glm-5.3": 44.8,
"glm-5.3-flash": 41.8,
"qwen3.7-plus": 25.2,
"deepseek-v4.1-flash": 39.5,
"mimo-v2.6-flash": 37.9,
"kimi-k2.7-code": 25.8,
"longcat-2.0": 19.1,
"minimax-m3.1-flash": 42.7,
"claude-haiku-5-5": 43.4,
"grok-4.7": 46.4,
"mimo-v2.5-pro": 26,
"kimi-k3": 43.6,
"minimax-m3.1-flash-free": 42.7,
"minimax-m2.7": 22.8,
"beam-free": 30.3,
"deepseek-v4-pro": 36,
"muse-spark-1.2-contributor": 39.6
}
},
{ {
"date": "2026-10-21", "date": "2026-10-21",
"scores": { "scores": {
site/assets/snapshot.json
+148 -120
View File
@@ -1,129 +1,113 @@
{ {
"generated": "2026-10-21T17:14:47.585Z", "generated": "2026-10-26T05:24:19.608Z",
"models": [ "models": [
"hy4-preview", "mimo-v2.5",
"claude-haiku-5-5", "gpt-6-luna",
"step-5-preview-free",
"deepseek-v4.1-flash",
"longcat-2.5-preview-free",
"minimax-m2.7",
"mimo-v2.6-pro",
"kimi-k3",
"hy3",
"mimo-v2.5-pro",
"beam-free",
"qwen3.8-flash",
"qwen3.8-max",
"gpt-5.6-luna",
"minimax-m3", "minimax-m3",
"glm-5.2", "hy3",
"deepseek-v4-pro", "cat-jam-free",
"mimo-v2.6-pro",
"muse-spark-1.3-contributor", "muse-spark-1.3-contributor",
"minimax-m3.1-flash-free",
"grok-4.7",
"qwen3.7-plus",
"gemini-4-argon-contributor", "gemini-4-argon-contributor",
"step-5-preview-free",
"gpt-5.6-luna",
"kimi-k3.1-pro",
"qwen3.8-flash",
"step-5-preview",
"qwen3.8-max",
"hy4-preview",
"gemini-4-argon",
"glm-5.2",
"glm-5.3", "glm-5.3",
"glm-5.3-flash", "glm-5.3-flash",
"space-bunny", "qwen3.7-plus",
"kimi-k2.7-code", "deepseek-v4.1-flash",
"cat-jam-free",
"gpt-6-luna",
"mimo-v2.5",
"muse-spark-1.2-contributor",
"mimo-v2.6-flash", "mimo-v2.6-flash",
"kimi-k2.7-code",
"longcat-2.0", "longcat-2.0",
"step-5-preview" "minimax-m3.1-flash",
"claude-haiku-5-5",
"longcat-2.5-preview-free",
"grok-4.7",
"mimo-v2.5-pro",
"space-bunny",
"kimi-k3",
"minimax-m3.1-flash-free",
"minimax-m2.7",
"iceman-contributor-free",
"beam-free",
"deepseek-v4-pro",
"muse-spark-1.2-contributor"
], ],
"scores": [ "scores": [
{ {
"id": "hy4-preview", "id": "mimo-v2.5",
"ii": null "ii": 21.8
}, },
{ {
"id": "claude-haiku-5-5", "id": "gpt-6-luna",
"ii": 43.4 "ii": 38.1
},
{
"id": "step-5-preview-free",
"ii": null
},
{
"id": "deepseek-v4.1-flash",
"ii": 39.5
},
{
"id": "longcat-2.5-preview-free",
"ii": null
},
{
"id": "minimax-m2.7",
"ii": 22.8
},
{
"id": "mimo-v2.6-pro",
"ii": 46.3
},
{
"id": "kimi-k3",
"ii": 43.6
},
{
"id": "hy3",
"ii": 25.3
},
{
"id": "mimo-v2.5-pro",
"ii": 26
},
{
"id": "beam-free",
"ii": 30.3
},
{
"id": "qwen3.8-flash",
"ii": 39.8
},
{
"id": "qwen3.8-max",
"ii": 45.4
},
{
"id": "gpt-5.6-luna",
"ii": 37.3
}, },
{ {
"id": "minimax-m3", "id": "minimax-m3",
"ii": 29.2 "ii": 29.2
}, },
{ {
"id": "glm-5.2", "id": "hy3",
"ii": 33.7 "ii": 25.3
}, },
{ {
"id": "deepseek-v4-pro", "id": "cat-jam-free",
"ii": 36 "ii": null
},
{
"id": "mimo-v2.6-pro",
"ii": 46.3
}, },
{ {
"id": "muse-spark-1.3-contributor", "id": "muse-spark-1.3-contributor",
"ii": 48.1 "ii": 48.1
}, },
{
"id": "minimax-m3.1-flash-free",
"ii": 42.7
},
{
"id": "grok-4.7",
"ii": 46.4
},
{
"id": "qwen3.7-plus",
"ii": 25.2
},
{ {
"id": "gemini-4-argon-contributor", "id": "gemini-4-argon-contributor",
"ii": 52.6 "ii": 52.6
}, },
{
"id": "step-5-preview-free",
"ii": null
},
{
"id": "gpt-5.6-luna",
"ii": 37.3
},
{
"id": "kimi-k3.1-pro",
"ii": 55.1
},
{
"id": "qwen3.8-flash",
"ii": 39.8
},
{
"id": "step-5-preview",
"ii": 43.7
},
{
"id": "qwen3.8-max",
"ii": 45.4
},
{
"id": "hy4-preview",
"ii": null
},
{
"id": "gemini-4-argon",
"ii": 52.6
},
{
"id": "glm-5.2",
"ii": 33.7
},
{ {
"id": "glm-5.3", "id": "glm-5.3",
"ii": 44.8 "ii": 44.8
@@ -133,49 +117,93 @@
"ii": 41.8 "ii": 41.8
}, },
{ {
"id": "space-bunny", "id": "qwen3.7-plus",
"ii": null "ii": 25.2
}, },
{ {
"id": "kimi-k2.7-code", "id": "deepseek-v4.1-flash",
"ii": 25.8 "ii": 39.5
},
{
"id": "cat-jam-free",
"ii": null
},
{
"id": "gpt-6-luna",
"ii": 38.1
},
{
"id": "mimo-v2.5",
"ii": 21.8
},
{
"id": "muse-spark-1.2-contributor",
"ii": 39.6
}, },
{ {
"id": "mimo-v2.6-flash", "id": "mimo-v2.6-flash",
"ii": 37.9 "ii": 37.9
}, },
{
"id": "kimi-k2.7-code",
"ii": 25.8
},
{ {
"id": "longcat-2.0", "id": "longcat-2.0",
"ii": 19.1 "ii": 19.1
}, },
{ {
"id": "step-5-preview", "id": "minimax-m3.1-flash",
"ii": 43.7 "ii": 42.7
},
{
"id": "claude-haiku-5-5",
"ii": 43.4
},
{
"id": "longcat-2.5-preview-free",
"ii": null
},
{
"id": "grok-4.7",
"ii": 46.4
},
{
"id": "mimo-v2.5-pro",
"ii": 26
},
{
"id": "space-bunny",
"ii": null
},
{
"id": "kimi-k3",
"ii": 43.6
},
{
"id": "minimax-m3.1-flash-free",
"ii": 42.7
},
{
"id": "minimax-m2.7",
"ii": 22.8
},
{
"id": "iceman-contributor-free",
"ii": null
},
{
"id": "beam-free",
"ii": 30.3
},
{
"id": "deepseek-v4-pro",
"ii": 36
},
{
"id": "muse-spark-1.2-contributor",
"ii": 39.6
} }
], ],
"newModels": [], "newModels": [
"kimi-k3.1-pro",
"gemini-4-argon",
"minimax-m3.1-flash",
"iceman-contributor-free"
],
"removedModels": [], "removedModels": [],
"deltas": [], "deltas": [],
"liveCatalogSize": 37, "liveCatalogSize": 38,
"notInLiveCatalog": [ "notInLiveCatalog": [
"cat-jam-free",
"gemini-4-argon-contributor",
"step-5-preview-free", "step-5-preview-free",
"longcat-2.5-preview-free",
"space-bunny", "space-bunny",
"cat-jam-free" "minimax-m3.1-flash-free"
] ]
} }