Automate the evaluation on a weekly schedule
Deploy to Cloudflare Pages / deploy (push) Successful in 30s
Deploy to Cloudflare Pages / deploy (push) Successful in 30s
- scripts/sync-catalog.mjs: rebuilds the Go catalogue from models.dev and benchmark scores from Artificial Analysis, and records what changed in snapshot.json - scripts/build-models-md.mjs: regenerates MODELS.md from the data layers - scripts/notify-new-models.mjs: files an issue when a new model needs a write-up - weekly-eval.yml: cron every Monday 05:23 UTC -> refresh -> commit -> Pages deploy - site now renders every model in the catalogue (33) with a data-snapshot stamp; auto-added models are flagged 'no write-up' until described - replaced scripts/refresh-evals.mjs with the wider sync script
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
name: Weekly evaluation refresh
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Mondays, 05:23 UTC — off the hour so it does not pile onto every other cron
|
||||
- cron: "23 5 * * 1"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
refresh:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Refresh catalogue and benchmark scores
|
||||
run: node scripts/sync-catalog.mjs
|
||||
|
||||
- name: Rebuild MODELS.md
|
||||
run: node scripts/build-models-md.mjs
|
||||
|
||||
- name: Report new models
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN || gitea.token }}
|
||||
GITHUB_API_URL: ${{ github.api_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
run: node scripts/notify-new-models.mjs
|
||||
|
||||
- name: Commit refreshed data
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN || gitea.token }}
|
||||
run: |
|
||||
git config user.name "opencode-go-evals"
|
||||
git config user.email "[email protected]"
|
||||
if git diff --quiet -- site/assets MODELS.md; then
|
||||
echo "no data changes — nothing to commit"
|
||||
exit 0
|
||||
fi
|
||||
git add site/assets MODELS.md
|
||||
git commit -m "chore: weekly evaluation refresh $(date -u +%Y-%m-%d)"
|
||||
# the checkout action's credentials are usually enough; fall back to the token
|
||||
if ! git push origin HEAD:main; then
|
||||
git remote set-url origin \
|
||||
"https://x-access-token:${GITEA_TOKEN}@git.wallacenet.ca/devadmin/OpenCodeGoEvals.git"
|
||||
git push origin HEAD:main
|
||||
fi
|
||||
echo "pushed — the Cloudflare Pages workflow will publish the update"
|
||||
Reference in New Issue
Block a user