devadmin
4e5a13f7b6
Deploy to Cloudflare Pages / deploy (push) Successful in 30s
- scripts/sync-catalog.mjs: rebuilds the Go catalogue from models.dev and benchmark scores from Artificial Analysis, and records what changed in snapshot.json - scripts/build-models-md.mjs: regenerates MODELS.md from the data layers - scripts/notify-new-models.mjs: files an issue when a new model needs a write-up - weekly-eval.yml: cron every Monday 05:23 UTC -> refresh -> commit -> Pages deploy - site now renders every model in the catalogue (33) with a data-snapshot stamp; auto-added models are flagged 'no write-up' until described - replaced scripts/refresh-evals.mjs with the wider sync script
145 lines
6.2 KiB
Markdown
145 lines
6.2 KiB
Markdown
# OpenCodeGoEvals
|
||
|
||
Benchmarks and plain-language notes for every model in the **OpenCode Go** catalog:
|
||
a task-by-task performance matrix, cost/value analysis, and a description of what each
|
||
model is actually targeted for (agentic work, coding, data analysis, image understanding,
|
||
deep research, high-volume chores).
|
||
|
||
Static site, no build step — it is plain HTML/CSS/ES modules, deployed to **Cloudflare Pages**.
|
||
|
||
## What's here
|
||
|
||
| Path | What it is |
|
||
|---|---|
|
||
| `site/index.html` | The site: performance matrix, price/value charts, model guide |
|
||
| `site/assets/data.js` | Model catalogue — pricing, context, modality, target descriptions, caveats |
|
||
| `site/assets/evals.js` | Benchmark scores (0–100) per model per task family |
|
||
| `site/assets/app.js` | Rendering logic for the matrix, charts and guide |
|
||
| `site/assets/style.css` | Styling |
|
||
| `MODELS.md` | The same model descriptions in Markdown |
|
||
| `scripts/sync-catalog.mjs` | Weekly refresh: models.dev catalogue + Artificial Analysis → `catalog.js`, `evals.js`, `snapshot.json` |
|
||
| `scripts/build-models-md.mjs` | Regenerates `MODELS.md` from the three data layers |
|
||
| `scripts/notify-new-models.mjs` | Files a Gitea issue when a new model needs a write-up |
|
||
| `scripts/deploy.sh` | Manual Cloudflare Pages deploy |
|
||
| `.gitea/workflows/weekly-eval.yml` | Weekly scheduled evaluation refresh |
|
||
| `wrangler.toml` | Cloudflare Pages project configuration |
|
||
| `.gitea/workflows/pages.yml` | CI: builds and deploys to Cloudflare Pages on every push to `main` |
|
||
|
||
## Data sources
|
||
|
||
- [OpenCode Go docs](https://opencode.ai/docs/go) — pricing, monthly allowances, privacy/retention, client support
|
||
- [OpenCode Data](https://opencode.ai/data) — catalog metadata, context windows, usage and retention stats
|
||
- [Artificial Analysis](https://artificialanalysis.ai) — benchmark scores and the Intelligence Index
|
||
- Vendor announcements — Google (Gemini 4 Argon), Reflection AI (Beam), Meituan (LongCat),
|
||
MiniMax (M3.1 Flash), xAI (Grok 4.7), StepFun (Step 5), Qwen, Zhipu, Moonshot, DeepSeek, Xiaomi, Tencent
|
||
|
||
Data snapshot date: **2026-10-21**. Re-run the collection script to refresh.
|
||
|
||
## Publishing to Cloudflare Pages
|
||
|
||
### 1. Create the Pages project
|
||
|
||
```bash
|
||
npm install
|
||
npx wrangler login # or export CLOUDFLARE_API_TOKEN
|
||
npx wrangler pages project create opencodogoevals
|
||
```
|
||
|
||
### 2. Give CI the credentials
|
||
|
||
Add these as repository secrets (Gitea: *Settings → Actions → Secrets*):
|
||
|
||
| Secret | Value |
|
||
|---|---|
|
||
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token with **Cloudflare Pages: Edit** (and Account Settings: Read) |
|
||
| `CLOUDFLARE_ACCOUNT_ID` | The Cloudflare account that owns the Pages project |
|
||
|
||
The workflow in `.gitea/workflows/pages.yml` then:
|
||
|
||
1. checks out the repo and installs Wrangler,
|
||
2. stages `MODELS.md` next to the site,
|
||
3. creates the Pages project if it does not exist yet (idempotent),
|
||
4. deploys `site/` on every push to `main`, on a manual *Run workflow*, and hourly.
|
||
|
||
A runner must be registered on the Gitea instance for the workflow to execute.
|
||
|
||
### 3. Deploy manually (optional)
|
||
|
||
```bash
|
||
./scripts/deploy.sh # uses env vars or your wrangler login session
|
||
npm run deploy # same thing, wrangler only
|
||
npm run dev # local preview on http://localhost:8788
|
||
```
|
||
|
||
### Custom domain
|
||
|
||
Set it in Cloudflare Pages → *Custom domains*, then add a CNAME record pointing at
|
||
`<project>.pages.dev`.
|
||
|
||
## Local development
|
||
|
||
The site is static — open `site/index.html` directly, or use `npm run dev` for a
|
||
Cloudflare-compatible local server.
|
||
|
||
## The weekly cycle
|
||
|
||
Two workflows run in Gitea Actions:
|
||
|
||
| Workflow | Trigger | What it does |
|
||
|---|---|---|
|
||
| `weekly-eval.yml` | Mondays 05:23 UTC (cron), or manual | Re-scrapes the catalogue and benchmark scores, rebuilds `MODELS.md`, opens an issue for any new model needing a write-up, and commits the result |
|
||
| `pages.yml` | every push to `main` | Deploys `site/` to Cloudflare Pages |
|
||
|
||
The chain is: **schedule → data refresh → commit → Pages deploy**, so the live site
|
||
updates itself without anyone touching it. The weekly run is idempotent — if nothing
|
||
changed, nothing is committed and nothing is redeployed.
|
||
|
||
A run leaves behind a trail you can audit:
|
||
|
||
- `site/assets/snapshot.json` — when the data was taken, which models are new or gone,
|
||
and every benchmark score that moved since the previous run
|
||
- the commit `chore: weekly evaluation refresh <date>` in the history
|
||
- the Actions run log for the week
|
||
- a Gitea issue when a new model appears in Go without a write-up
|
||
|
||
### What is automated vs. hand-written
|
||
|
||
| File | Owner | Contents |
|
||
|---|---|---|
|
||
| `site/assets/catalog.js` | auto-generated | every model in Go: pricing, context, modalities, release date |
|
||
| `site/assets/evals.js` | auto-generated | benchmark scores per model per task family |
|
||
| `site/assets/data.js` | **hand-written** | what each model is targeted for, caveats, tags |
|
||
| `site/assets/snapshot.json` | auto-generated | timestamp, added/removed models, score deltas |
|
||
|
||
When you want to change a description, edit `site/assets/data.js` and run
|
||
`node scripts/build-models-md.mjs`; the weekly refresh will never overwrite it.
|
||
|
||
### Running it by hand
|
||
|
||
```bash
|
||
node scripts/sync-catalog.mjs # refresh catalogue + scores
|
||
node scripts/build-models-md.mjs # rebuild MODELS.md
|
||
node scripts/notify-new-models.mjs # file the "needs write-up" issue
|
||
```
|
||
|
||
### If the scheduler is not firing
|
||
|
||
Gitea only runs scheduled workflows when a runner is registered and awake. If Mondays
|
||
slip past, trigger it manually from the Actions tab (*Run workflow*), or run the three
|
||
commands above on any machine with network access and push the result.
|
||
|
||
## Caveats
|
||
|
||
- Task-matrix cells use the best published evaluation per task family; when a model has
|
||
not been run on a benchmark the column falls back to another one, so the underlying
|
||
benchmark can differ between rows in a column. Hover a cell to see which one it is.
|
||
- Free-tier models (MiniMax M3.1 Flash Free, Beam Free, LongCat 2.5 Preview Free) are
|
||
limited-time offers.
|
||
- Contributor-tier models (Gemini 4 Argon, Muse Spark 1.3) may use your prompts and
|
||
completions for training — check the Go docs' privacy table.
|
||
- No Go model generates images; they are all text-output.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|