this post sux
sux bench
1 folder · 32 files

README.md

bench: Jev vs Laya vs EmbeddingGemma for in-browser feed filtering #

100 current Bluesky Discover posts, labeled by Claude on five sux-style rules, asked with the exact question wording sux sends Jev (data/tasks.json is generated from shared.js). Results: results.md.

node fetch_posts.mjs 300          # data/pool.json (skips !no-unauthenticated authors)
# label data/pool_part{0,1,2}.json -> data/labels_part{0,1,2}.json (done by Claude subagents)
node make_set.mjs                 # data/set.json: 100 posts, rare positives kept
node make_tasks.mjs               # data/tasks.json from sux's buildQuestions
source ../typesafe_api_key.secret.sh && node run_jev.mjs
uv run python run_laya.py english mps     # also: multilingual, cpu
node run_gemma.mjs q8                     # also: fp32, q4 (Transformers.js, onnx-community build)
uv run python analyze.py                  # -> results.md

Caveats: one labeler family (Claude), 100 posts, 12–31 positives per rule, so differences under ~0.05 are noise. Gemma latency is Node/onnxruntime CPU, not browser WASM/WebGPU. Laya ships no ONNX or npm package; its timings are PyTorch.