Automate the evaluation on a weekly schedule
Deploy to Cloudflare Pages / deploy (push) Successful in 30s
Deploy to Cloudflare Pages / deploy (push) Successful in 30s
- scripts/sync-catalog.mjs: rebuilds the Go catalogue from models.dev and benchmark scores from Artificial Analysis, and records what changed in snapshot.json - scripts/build-models-md.mjs: regenerates MODELS.md from the data layers - scripts/notify-new-models.mjs: files an issue when a new model needs a write-up - weekly-eval.yml: cron every Monday 05:23 UTC -> refresh -> commit -> Pages deploy - site now renders every model in the catalogue (33) with a data-snapshot stamp; auto-added models are flagged 'no write-up' until described - replaced scripts/refresh-evals.mjs with the wider sync script
This commit is contained in:
@@ -17,6 +17,11 @@ Static site, no build step — it is plain HTML/CSS/ES modules, deployed to **Cl
|
||||
| `site/assets/app.js` | Rendering logic for the matrix, charts and guide |
|
||||
| `site/assets/style.css` | Styling |
|
||||
| `MODELS.md` | The same model descriptions in Markdown |
|
||||
| `scripts/sync-catalog.mjs` | Weekly refresh: models.dev catalogue + Artificial Analysis → `catalog.js`, `evals.js`, `snapshot.json` |
|
||||
| `scripts/build-models-md.mjs` | Regenerates `MODELS.md` from the three data layers |
|
||||
| `scripts/notify-new-models.mjs` | Files a Gitea issue when a new model needs a write-up |
|
||||
| `scripts/deploy.sh` | Manual Cloudflare Pages deploy |
|
||||
| `.gitea/workflows/weekly-eval.yml` | Weekly scheduled evaluation refresh |
|
||||
| `wrangler.toml` | Cloudflare Pages project configuration |
|
||||
| `.gitea/workflows/pages.yml` | CI: builds and deploys to Cloudflare Pages on every push to `main` |
|
||||
|
||||
@@ -76,15 +81,52 @@ Set it in Cloudflare Pages → *Custom domains*, then add a CNAME record pointin
|
||||
The site is static — open `site/index.html` directly, or use `npm run dev` for a
|
||||
Cloudflare-compatible local server.
|
||||
|
||||
## Updating the data
|
||||
## The weekly cycle
|
||||
|
||||
1. `node scripts/refresh-evals.mjs` re-scrapes every model's benchmark scores from
|
||||
Artificial Analysis and rewrites `site/assets/evals.js`. Model names, OpenCode Go
|
||||
prices and free-tier flags come from `site/assets/data.js` (hand-maintained from the
|
||||
Go docs), so the chart always shows Go pricing.
|
||||
2. Descriptions, modalities, release dates and caveats live in `site/assets/data.js`;
|
||||
update them there and regenerate `MODELS.md`.
|
||||
3. Commit both files — CI publishes the site on push.
|
||||
Two workflows run in Gitea Actions:
|
||||
|
||||
| Workflow | Trigger | What it does |
|
||||
|---|---|---|
|
||||
| `weekly-eval.yml` | Mondays 05:23 UTC (cron), or manual | Re-scrapes the catalogue and benchmark scores, rebuilds `MODELS.md`, opens an issue for any new model needing a write-up, and commits the result |
|
||||
| `pages.yml` | every push to `main` | Deploys `site/` to Cloudflare Pages |
|
||||
|
||||
The chain is: **schedule → data refresh → commit → Pages deploy**, so the live site
|
||||
updates itself without anyone touching it. The weekly run is idempotent — if nothing
|
||||
changed, nothing is committed and nothing is redeployed.
|
||||
|
||||
A run leaves behind a trail you can audit:
|
||||
|
||||
- `site/assets/snapshot.json` — when the data was taken, which models are new or gone,
|
||||
and every benchmark score that moved since the previous run
|
||||
- the commit `chore: weekly evaluation refresh <date>` in the history
|
||||
- the Actions run log for the week
|
||||
- a Gitea issue when a new model appears in Go without a write-up
|
||||
|
||||
### What is automated vs. hand-written
|
||||
|
||||
| File | Owner | Contents |
|
||||
|---|---|---|
|
||||
| `site/assets/catalog.js` | auto-generated | every model in Go: pricing, context, modalities, release date |
|
||||
| `site/assets/evals.js` | auto-generated | benchmark scores per model per task family |
|
||||
| `site/assets/data.js` | **hand-written** | what each model is targeted for, caveats, tags |
|
||||
| `site/assets/snapshot.json` | auto-generated | timestamp, added/removed models, score deltas |
|
||||
|
||||
When you want to change a description, edit `site/assets/data.js` and run
|
||||
`node scripts/build-models-md.mjs`; the weekly refresh will never overwrite it.
|
||||
|
||||
### Running it by hand
|
||||
|
||||
```bash
|
||||
node scripts/sync-catalog.mjs # refresh catalogue + scores
|
||||
node scripts/build-models-md.mjs # rebuild MODELS.md
|
||||
node scripts/notify-new-models.mjs # file the "needs write-up" issue
|
||||
```
|
||||
|
||||
### If the scheduler is not firing
|
||||
|
||||
Gitea only runs scheduled workflows when a runner is registered and awake. If Mondays
|
||||
slip past, trigger it manually from the Actions tab (*Run workflow*), or run the three
|
||||
commands above on any machine with network access and push the result.
|
||||
|
||||
## Caveats
|
||||
|
||||
|
||||
Reference in New Issue
Block a user