dev.permadeath.com #

tally is the deduplicated commit-activity service behind dev.permadeath.com. Repository: tangled.org/permadeath.com/dev.permadeath.com.

Tally answers four questions in one place:

1. How much have I committed on Tangled, per identity?
2. How much have my didbot agents committed there?
3. How much have I committed on GitHub, including private and work repositories?
4. What fraction of all Tangled commits is mine?

The site is static. A scheduled Rust worker crawls, writes JSON and HTML, and publishes them to S3 behind CloudFront. There is no public write path and no server process in the request path.

Preview #

The interactive, self-contained snapshot is docs/mockup.html; open it in a browser to use the period selector, table view and theme toggle. The screenshots contain real activity data and are intentionally committed. Rebuild it from a built site with python3 scripts/build_mockup.py dist/data/site.json. The heatmap's day panel fetches data/days/YYYY-MM.json, so it needs the built site behind a local HTTP server (python3 -m http.server -d dist); the mockup can't load it.

Each build also renders the link preview: og.png (by default four of the page's charts as wordless red shapes: the 90-day calendar, the 24 h wheel, written-to-landed and project timelines; TALLY_OG_LAYOUT=year brings back the year's matrix) and its alt text, with the 30-day count of the owner's and their agents' commits together. The meta tags' title and description are fixed text. The image URL carries a fingerprint of the PNG, so a crawler that caches by URL refetches it only when the card changed.

Prior Tangled census #

The earlier network research lives at blogposts/tangled-stats. As of 4 September 2026 its closure crawl reached 182 knots, 14,528 repositories and 17,939 accounts. It counted commits from merged-PR patches: 278,018 all-time, of which permadeath.com was 3,014 (1.1%), and 16,966 / 2,426 (14.3%) for August 2026. Its own notes say commits pushed straight to a branch are not counted and that pull-record shapes changed over the crawl.

Tally takes the branch-walk route instead: every branch is read, direct pushes are included, and commits collapse by SHA and Change-Id. Its denominator is therefore larger and its share is not directly comparable to the patch-derived census. The dashboard labels both the definition and the current index coverage.

Commit definition #

One definition applies to Tangled, GitHub and the redacted work inbox:

A logical change is a distinct Change-Id: when the commit carries one (atgc writes these trailers), otherwise a distinct Git object ID (SHA). A commit counts once its repository's default branch reaches it (merged work; TALLY_BRANCH_SCOPE=all counts every branch instead), and counts once regardless of how many branches, resubmits, rebases, tags, or forges report it. Exact resubmits have the same SHA and are already excluded. A rebase keeps the Change-Id, so the rewritten object collapses into the same logical change. The same SHA on Tangled and GitHub is one change. Merge commits count once. In the share, a rebase without a Change-Id counts once when its author, author time and subject match; elsewhere it remains a separate object. Tags are not counted separately. Attribution uses the Git author, and dates are author dates in UTC. The dashboard also shows the raw unique Git-object count for context.

Tangled's index is built from ref updates, so the same commit can be announced on every branch and again on every resubmit of a pull request. A smoke crawl of 17 repositories read 130,690 branch-reported commits and found 6,680 distinct commits — about 19.6x inflation. Tally walks every branch, stops when a branch joins history it has already indexed, and stores one row per SHA. The GitHub and work crawlers walk branches the same way and deduplicate by SHA; co-authored commits count for their primary author.

• A commit reachable from five branches is one row.
• A commit seen on Tangled and mirrored on GitHub is one row.
• Author dates are used, not push dates; the committer is often Tangled <noreply@tangled.sh> with the time of the push.
• The global denominator is the distinct SHA set across every public Tangled repository the worker has reached. The dashboard shows repository coverage next to the share and labels the figure preliminary until coverage is high.
• The previous self-reported figure (32.6% on 2026-09-04) is shown as context, not silently reproduced.

The share is period-scoped (30d, 90d, 1y, or all time) and is all of your Tangled activity across all repositories — your handles plus your bots — over that period. It is not a per-repository or per-handle ratio. GitHub is tracked separately and is not part of the Tangled denominator. The dashboard's data covers at least the last 365 days.

Identities #

Personal Tangled identities are configured with a stable, validated colour:

identity	colour
permadeath.com	series-red (hard red)
lance.blue	series-1 (blue)
permadeath.games	series-3 (aqua)
atgc.codes	series-4 (yellow)
personal GitHub	series-6 (green)
did.bot agents	series-5 (magenta)
work GitHub	series-7 (violet)

The date chart and the time wheel stack by identity in these colours, or, by default, by project: the year's six busiest, owner and agents together, each keeping its colour between builds while it stays in the six, plus grey "other". A "project · identity" toggle switches both (?by=identity); the period pickers filter days and never recolour. The redacted work account carries no repository names. Project and recent-commit rows carry a small T (Tangled) or GH (GitHub) source badge, and Tangled project links use the fully qualified handle/repository form.

Local development #

cargo run -- crawl --db state.sqlite3 --out dist --owned-only --skip-github
cargo run -- serve --dir dist --bind 127.0.0.1:8787

--owned-only scans only the configured personal and bot repositories, which is the fast feedback loop. Without it the worker enumerates the public Tangled index and resumes the global backfill across runs.

A run is bounded by TALLY_CRAWL_BUDGET_SECS (default 3000). Repositories and branches are committed to SQLite as they are read, so the next run resumes exactly where this one stopped.

Two crawl schedules #

The production deployment uses two tasks and two databases:

• Scheduled global (monthly) — crawl --global, with TALLY_STATE_KEY=state/global.sqlite3. It walks every public Tangled repository, ignores the time budget, skips GitHub and the inbox, writes TALLY_GLOBAL_SUMMARY_OUT, uploads periodic SQLite checkpoints, and does not publish the site. It can run as N shards (below).
• Hourly owned — crawl --owned-only, with the normal state/tally.sqlite3. It scans the personal and bot repositories, ingests recent GitHub reports, reads TALLY_GLOBAL_SUMMARY, publishes the site and uploads the hourly snapshot.

The global summary is:

{"version":1,"generated_at":"2026-10-21T03:07:00Z",
 "repos_reached":14528,"repos_total":14528,"partial":false,
 "days":{"2026-10-20":1234}}

The global pass skips fork records, excludes commits before TALLY_GLOBAL_IMPORTED_BEFORE and before each repo's creation date minus TALLY_GLOBAL_REPO_AGE_GRACE_DAYS, and flags a repository whose default branch exceeds TALLY_GLOBAL_REVIEW_REPO_COMMITS for review. A per-repo page cap is the backstop for a log whose total is unavailable.

Which repositories count is decided by a versioned rule set (src/rules.rs), recomputed from scratch on every run, so removing a rule or an exclude brings a repository back. Each repository gets an evidence vector, signals in tiers:

tier	for	against
manual		excludes file, fork record, description says mirror/fork/backup
strong	commits by Tangled identities (DID authors); history born on Tangled	50+ authors with under 0.5% Tangled-identity commits; stopped at the page cap
medium	at most 10 authors; owner keeps other repos; an original description	10+ authors before the repo existed
weak	complete walk; plausible size	history from before the repo existed

A manual signal excludes. A strong negative excludes, or holds the repository for review when a strong positive contradicts it; a page-cap stop holds. Otherwise a repository counts with one strong positive, two medium ones, or one medium and no medium negative. 10+ authors before the repo existed also drops its grace period. At the merge, a repository over 5% of any window (and at least 1,000 changes in it) is held unless its DID is in global-reviewed.txt (TALLY_GLOBAL_REVIEWED). Rewritten commits without a Change-Id (same author email, author time and subject) count once, and nothing dated more than a day after the run counts. The summary records rules (version, parameters), rules_hash (SHA-256 of them, the excludes and the reviewed list), the verdict tallies, held_for_review with what each held repository would add, held_days, the evidence of the largest excluded or held repositories, and top_repos.

crawl --global --filter-only applies the rule set and the oversized-repo log to an existing database, rewrites the summary, and exits without walking commits. It uses indexed commit counts, so a giant repository that was never scanned is not visible to the audit until a normal run reaches it.

Shards #

TALLY_SHARD_INDEX of TALLY_SHARD_COUNT (or crawl --global --shard i/N) crawls only the repositories whose DID hashes to that shard, under its own state key beside the single-task one: state/global.sqlite3 becomes state/global-shard-2-of-8.sqlite3. A shard with no state yet starts from the single-task database, trimmed to its repositories, so moving to shards is not a re-crawl. Instead of the summary, each shard publishes its counted changes (state/global-shard-2-of-8.counts.sqlite3): one row per logical change with its earliest day in each count. global-merge (with the same TALLY_SHARD_COUNT, or --counts files) merges them and writes TALLY_GLOBAL_SUMMARY_OUT. A change reachable from repositories on two shards counts once, so the merge equals a single-task summary; adding per-shard daily counts would not. The merged summary is as old as its oldest shard and partial if any shard was.

A repository whose history no walk has finished (a branch left unindexed, or commits none of which is marked as on the default branch) is due whatever the refresh window says, so a seeded shard walks every default branch once. A walk saves its place (the head it reads and how far into its log) after every page, and a branch is indexed only when a walk reaches its end, the import cutoff or known history; a walk cut short by a timeout, a failed page or the page cap resumes from that place next run. Until then the repository is not scanned for coverage: one stopped at the cap is a failure, the rest count in repos_unwalked and coverage.repos_walk_incomplete. Before writing a summary, the single task and global-merge refuse one that looks wrong: coverage under TALLY_GLOBAL_MIN_COVERAGE_PCT, too many repositories still to walk, or default-branch counts under TALLY_GLOBAL_MIN_DEFAULT_RATIO of the all-branch ones. The run fails and the published summary stays.

Discovery and coverage #

The global run's universe is Tangled search, including records from before repositories had a DID (their knot serves them as <owner DID>/<name>; the mirror doesn't), plus the repo records of accounts the relay lists and search doesn't. A sharded run finds it once: global-discover writes the list to TALLY_GLOBAL_DISCOVERY before the shards start, and each shard keeps its own repositories from it. Knots on localhost, private addresses and reserved names are not asked at all. A failure is tried once more after the circuit cooldown.

Coverage is what the run read over what it could read. repos_total is the reachable repositories, repos_fresh those read without error this run (at or after coverage.scanned_since, the run's start less the refresh window), and repos_reached those ever read. A repository is unreachable, and outside repos_total, when its knot is private, or when its host or the repository itself has failed three times in a row over at least two days; anything else that fails stays in as a failure. missing_changes_30d estimates the logical changes of the last 30 days the unread repositories hold: each one's rate over the 30 days before its last good read, over the days since. coverage has the detail per shard and summed: discovered, excluded, reachable, scanned, failed, not attempted and unreachable counts, failure and unreachable reasons, how repositories were found, and the same estimate for unreachable ones.

A checkpoint writes the same file with partial:true. The hourly run sums days for each selected period and hides the share when the summary is missing, stale, or partial below TALLY_GLOBAL_MIN_COVERAGE_PCT. A partial summary at or above that coverage is shown with its coverage beside the number. The age cutoff is TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS (default 9, range 1–365); use about 35 for a monthly crawl.

Configuration #

variable	default	meaning
TALLY_PERSONAL_DIDS	the four identities	did|handle|series-N comma separated
TALLY_PERSONAL_EMAILS	empty	address=did aliases for older commits
TALLY_DIDBOT_PDS	https://pds.did.bot	comma-separated didbot PDS bases
TALLY_TANGLED_API	https://api.tangled.org	appview XRPC
TALLY_TANGLED_MIRROR	https://mirror.tangled.network	commit/branch API
TALLY_GLOBAL_DISCOVERY	unset	where global-discover writes the repository list and a sharded run's shards read it (s3://… or a path); unset or older than 36 hours, a run finds the universe itself
TALLY_TANGLED_RELAY	https://relay1.us-east.bsky.network	relay whose sh.tangled.repo writers the global run lists when search lacks them; empty is off
TALLY_CONCURRENCY	8	repositories scanned in parallel
TALLY_PER_HOST_RPS	10	per-host request pacing
TALLY_CRAWL_BUDGET_SECS	3000	hourly budget; --global ignores it
TALLY_GLOBAL_REFRESH_SECS	86400	how often a global repo is revisited; use 2592000 for monthly
TALLY_OWNED_REFRESH_SECS	3600	how often an owned repo is revisited
TALLY_GLOBAL_CONCURRENCY	4	global-run repository parallelism
TALLY_GLOBAL_PER_HOST_RPS	2	global-run per-host request pacing
TALLY_GLOBAL_HTTP_ATTEMPTS	3	attempts per global request
TALLY_GLOBAL_SKIP_FORKS	true	exclude fork records from the global denominator
TALLY_GLOBAL_SKIP_DESCRIBED_COPIES	true	also exclude repos whose own description says they are a mirror, fork or backup ([READ-ONLY] Mirror of …, fork of …)
TALLY_BRANCH_SCOPE	default	which branches count: default, what each repository's default branch reaches (merged work), or all, every branch. Both are stored and summarized, so switching needs no recrawl
TALLY_GLOBAL_IMPORTED_BEFORE	2025-01-01	network-wide floor for imported history
TALLY_GLOBAL_REPO_AGE_GRACE_DAYS	30	also drop commits older than each repo's Tangled creation date minus this grace
TALLY_GLOBAL_MAX_PAGES	40	per-repo page cap; stops runaway histories
TALLY_GLOBAL_REVIEW_REPO_COMMITS	20000	flag repos above this for manual review; not auto-excluded
TALLY_GLOBAL_EXCLUDES	global-excludes.txt	one repository DID per line
TALLY_GLOBAL_REVIEWED	global-reviewed.txt	held repositories the owner has reviewed and released, one DID per line
TALLY_GLOBAL_PROGRESS_SECS	600	progress log interval
TALLY_GLOBAL_PROGRESS_REPOS	250	progress log interval in repos
TALLY_GLOBAL_CHECKPOINT_SECS	1800	seconds of crawling between checkpoint uploads
TALLY_GLOBAL_CHECKPOINT_REPOS	0	also checkpoint every this many repos; 0 is off
TALLY_SHARD_INDEX	0	this task's shard of the global crawl
TALLY_SHARD_COUNT	1	number of global shards; 1 is the whole crawl in one task
TALLY_REPO_TIMEOUT_SECS	900	give up on one repository after this long; 0 is no limit
TALLY_GLOBAL_MIN_COVERAGE_PCT	90	minimum coverage for a partial share, and for publishing a global summary at all
TALLY_GLOBAL_MIN_DEFAULT_RATIO	0.4	refuse to publish when a default-branch count falls below this share of the all-branch one; 0 is off
TALLY_CIRCUIT_FAILURES	5	consecutive failures before a host circuit opens
TALLY_CIRCUIT_COOLDOWN_SECS	60	host circuit cooldown
TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS	9	1–365; hide the share after this many days (2 nightly, 9 weekly, 16 fortnightly, ~35 monthly)
TALLY_GLOBAL_SUMMARY_OUT	unset	global output: s3://… or local path
TALLY_GLOBAL_SUMMARY	unset	hourly input: s3://…, https://… or local path
TALLY_MAX_REPOS_PER_RUN	unset	optional cap for a run
TALLY_TIMEZONE	unset	IANA zone for the time-of-day wheel, such as America/New_York; unset, each viewer sees their own zone. Converted in the browser per date, so daylight saving is right. The link preview's wheel reads hours in it too, in UTC when unset
TALLY_ACTIVITY_URL	https://vibescrobble.com	https vibescrobble service the live "agentic activity" card polls in the browser for the first personal DID; empty turns the card off
TALLY_ACTIVITY_TEXT	true	show each scrobble's text, model and repo in that card; false keeps only emoji, agent and age
TALLY_OG_LAYOUT	first in og::LAYOUTS	the link preview's design by name: dashboard or year
TALLY_WORK_INBOX	unset	s3://…, https://… or local path for one inbox payload
TALLY_INBOXES	unset	comma-separated inbox payloads for multiple accounts; overrides TALLY_WORK_INBOX
GITHUB_TOKEN	unset	personal GitHub token
TALLY_GITHUB_LOGIN	token's login	personal login
TALLY_GITHUB_OWNERS	permadeath-games	extra owners to enumerate
TALLY_GITHUB_REDACT_ALL	false	redact every personal-account repo
TALLY_GITHUB_REDACT_REPOS	empty	personal repos to redact by full name
GITHUB_WORK_TOKEN	unset	optional direct work token; prefer the inbox
TALLY_WORK_REDACT_ALL	true	redact every work repository
TALLY_WORK_REDACT_REPOS	empty	comma-separated work repos to redact instead
TALLY_PUBLISH	0	upload the site for hourly runs, or state + summary for global runs
TALLY_SITE_BUCKET	unset	public site bucket
TALLY_STATE_BUCKET	unset	index snapshot and inbox bucket
TALLY_STATE_KEY	state/tally.sqlite3	index snapshot key
TALLY_S3_PREFIX	empty	optional key prefix for the site

GitHub #

Personal account and permadeath-games #

Create a fine-grained PAT with:

• Metadata: Read-only
• Contents: Read-only on the repositories you want counted, including the private repositories in the permadeath-games organization.
• Organization approval for permadeath-games, if the org requires it.

The worker lists /user/repos and also enumerates every owner named in TALLY_GITHUB_OWNERS, so permadeath-games is included even when it is not returned by the authenticated account's repository list. Commits are filtered to author=<login>.

Store the token in SSM SecureString and give its parameter name to the infra stack as github_token_parameter_name.

Either account: upload a report instead of a token #

The service never needs the account tokens. Run the scraper once per account on the machine that holds that token:

# personal account; public repositories are named, private ones are hashed
GITHUB_TOKEN=... python3 scripts/github_activity.py \
  --label personal --login <personal-login> \
  --s3 s3://tally-state-ACCOUNT/inbox/personal-activity.json

# work account; every name is hashed, including public repositories
GITHUB_WORK_TOKEN=... \
  python3 scripts/github_activity.py --label work \
  --s3 s3://tally-state-ACCOUNT/inbox/work-activity.json

Private and internal repositories are hashed by default, including when --no-redact-repos is passed. Public repository names are used unless --redact-all is passed. A --label work report hashes every repository, public or private, because a public repository name can still disclose what the work account was doing. --reveal-private-repos names private repositories on a personal report and should only be used for repositories you are willing to publish.

The repo-hash salt is created on first use at ~/.config/tally/repo-salt with mode 0600, or at $TALLY_SALT_FILE / --salt-file if set. No salt argument is needed. Keep that file stable and private: changing it changes every repository hash. --salt and WORK_REPO_SALT still override it for one-off runs.

On the first run each collects 365 days across every branch; later runs re-read two days of overlap and the service deduplicates by SHA/Change-Id. The state file is per label, so the two runs do not overwrite each other. It also records each repository's pushed_at, so unchanged repositories are skipped without listing their branches; --full ignores that cache.

Each commit is marked with whether the repository's default branch has it, and the newest report's mark replaces an older one. Reports from before the mark existed count every commit; run each label once with --full so history uploaded earlier is marked too.

--upload-url sends the payload as a plain HTTPS PUT, with no AWS CLI involved. The bearer token is read from TALLY_UPLOAD_TOKEN, then TALLY_UPLOAD_TOKEN_FILE, then ~/.config/tally/upload-token. That is the hook for a stable ingest endpoint; a presigned S3 URL also works but expires and has to be regenerated.

Point the service at both payloads:

TALLY_INBOXES=s3://tally-state-ACCOUNT/inbox/personal-activity.json,s3://tally-state-ACCOUNT/inbox/work-activity.json

The payload contains only:

{"version":1,"label":"personal","redacted":false,
 "generated_at":"…","commits":[
  {"sha":"<40 or 64 hex>","authored_at":"2026-10-19T15:00:00Z",
   "repo":"owner/public-repo","redacted":false,"change_id":"I…"},
  {"sha":"<40 or 64 hex>","authored_at":"2026-10-19T16:00:00Z",
   "repo":"h_ab12…","redacted":true,"change_id":""}
]}

Each commit carries its own redacted flag, so one payload can mix named public repositories with hashed private ones. The top-level redacted is true only when every repository is hashed; older payloads may omit the per-commit flag and fall back to it. For --label work, repo is normally a salted h_ab12… hash. There is no field for a login, commit message, URL, author name or email. change_id is only the Gerrit Change-Id: trailer value, so rebases collapse to one logical change just like atgc commits. The service reads only the fields above and ignores extras.

The work machine's credential needs one permission:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject"],
    "Resource": ["arn:aws:s3:::tally-state-ACCOUNT/inbox/work-activity.json"]
  }]
}

The infrastructure repo owns the state bucket and the write-only policy. Point the scraper at the s3://…/inbox/work-activity.json URI it provides.

A direct GITHUB_WORK_TOKEN is also supported, but it requires the service to hold the token and see repository names. Prefer the inbox.

Deployment #

This repository is the application only. AWS infrastructure lives in the private platform Terraform repo (envs/devinfra/site*.tf and modules/site). See docs/GLOBAL-RERUN.md for the rebuild and global re-run contract.

The app side of the contract is:

• build the image from Dockerfile and push it to the platform ECR repo
• run the nightly task as crawl --global, with its own TALLY_STATE_KEY=state/global.sqlite3, TALLY_GLOBAL_SUMMARY_OUT, no time budget, and no GitHub/inbox configuration
• run the hourly task as crawl --owned-only, with TALLY_STATE_KEY pointing at the hourly snapshot and TALLY_GLOBAL_SUMMARY pointing at the nightly summary
• give the task roles the S3 state/site permissions, SSM token access and networking described in the infra repo

The worker is a one-shot Fargate task. CloudFront serves the static site; there is no load balancer or always-on service.

Tests #

cargo test
cargo check --features aws
cargo fmt --check
cargo clippy --all-targets --features aws -- -D warnings

The unit tests cover the de-duplication invariant (same SHA on many branches and in two repositories), identity upgrades without duplication, the Git SHA validator, repository-hash sanitising, the palette's stable bot colours, the S3 prefix/object-key rules and the global-summary period maths.

Limits #

• Private Tangled repositories are not part of the public index and are not counted.
• The global share is only meaningful once the repository coverage shown next to it is high. Until then the site labels it preliminary.
• A work repository is only as hidden as its hash and the local salt. The hash prevents the public site from naming it; it is not encryption.
• GitHub commits are filtered to the authenticated login. Co-authored commits are counted under their primary author.