A local Git merge queue with content-bound gate verdicts
28 kB · 445 lines
Markdown
at main

The nine rules #

Nine rules decide every name, argument and output of this specification. Each rule names the failure it prevents. A choice elsewhere that seems arbitrary follows from one of them. Two terms come first, since every rule uses them.

A check is a named command (build, test, the pre-commit hook) that the rules define and a run executes on a tree. Its result is passed, failed, stuck (it reached its time bound) or error (it could not run). Each branch lists the checks a land into it must pass.

A tree is checked for a branch when every check that branch lists passed on that exact tree: the Git tree of NAME replayed onto the branch's current tip.

The tree of NAME's own tip does not count, and neither does a replay onto an older tip of the branch. A land of fix-ttl into staging builds the commit staging would move to, runs staging's checks on its tree, and fast-forwards staging to that same commit. The branch therefore holds, byte for byte, a tree whose checks passed. A pull request whose checks ran on its own head can still break the branch it merges into, because the merged tree was never tested. Landing only checked trees closes that gap.

A result is stored under the tree, the check and the build configuration. The build configuration covers the check's own definition in the rules, with the commit of each other repository the check declares it reads, and nothing of the machine: a result is a fact about a tree under a check's definition. The platform that ran the check is a field of the result, never part of its key. One pass on any platform the rules support passes a check, unless the check's section lists the platforms its verdict must hold on, and then a pass on each of them is required (Configuration). A tree already checked costs a lookup unless --retry asks for a rerun, which is how a check gets a verdict under a changed tool, and mq pull brings in the results other machines recorded (Check results). A run the scheduler started reuses only results that runs the scheduler started recorded, since a run a caller started keeps that caller's confinement (runs), and mq status says checks passed by mq land only of a tree whose passes are all such a run's. The engine refines "passed" in one way, and every output that relies on it says so: a failure the target's tip already has is not charged to the change being landed. A flake never stops a check gating. It is recorded and shown, and a check leaves a branch's list only by a land into mq/config.

1. One name per piece of work: the branch #

Every branch can be landed into. A branch that takes its commits from another along an edge of the DAG is landed into from that branch alone. No branch is a different kind from another. A queue is the line of lands waiting on a branch; it exists while something waits. A branch's name is the only name a user types for a piece of work.

staging, main and release/1.2 are branches like any other. How they feed each other is a directed acyclic graph (DAG) declared in the rules (Configuration). An edge staging -> main says that main takes staging, and when: as staging moves, on a machine whose scheduler runs the queue, or on demand. It also says that main holds nothing staging lacks, so a land along an edge is always a fast-forward and git log main..staging is exactly what main has not taken yet. Each branch names the checks a land into it must pass. A branch of the DAG is any branch the rules name, by its own name or by a pattern such as release/*, whether or not an edge reaches it.

A branch made by mq new points at a branch of the DAG through its upstream, and is not one itself: fix-ttl lands into staging because mq new recorded staging as its base, and mq push never publishes it. A branch made with git switch -c is no different, and lands the same way. The entry branch is where new work lands by default. It is the one branch the rules name by its own name that takes no other, or the one they mark with entry when there are several. A pattern such as release/* is never the entry branch. The smallest DAG is one branch, main, with no edge, and every example in this specification that names staging reads main there. The DAG says where work flows and what checks it. It does not stop anyone from moving a branch (rule 4). "Work branch" names a role: it is the branch being landed. The one distinction the commands make is between a branch the rules name, which mq never deletes and rewrites only in the unpublished part a pull replays (Several machines), and every other branch.

There is no entry id, no series noun, no clone path and no commit-set name on the command line. A branch's queue is the line of lands waiting on that name. The issue becomes metadata on the branch. It is no longer the identity.

Failure prevented: a command that accepts the issue while its neighbour wants the entry id, so the user has to translate between five names.

2. State lives where Git keeps it #

Every fact mq keeps has one place where it is the truth (The state). Git's own state stays in Git, and mq keeps no copy of it: the rules are a branch, every move mq makes is in the moved branch's reflog, a stopped conflict is a git rebase in progress, a worktree is Git's, a check result is a line of the note on its tree, evidence is a note on its commit, and a commit mq keeps for a person to take back is in the reflog of refs/mq/keep. mq's own facts are Git's too where their nature is: a queued land is a ref to its record, a candidate under check a ref to the replayed commit, a move about to be finished an intent ref written in the move's own transaction, a freeze a ref whose reflog is its history (The refs). git worktree list -v, git reflog, git log --notes=mq and git status therefore show what they show of the queue on a machine where nobody has read mq's help.

The journal holds the history of events: what a run did and when, each check's duration, each wait and its reason, the delivery's position. One SQLite database, the index, holds no fact of its own: it is a view over Git and the journal, refreshed by the operation that writes Git and by a reader that finds it older than the refs it mirrors, so deleting it changes no reading (What the index holds). The facts another machine needs travel as the branches of the DAG, the rules and two notes refs, the results about trees as refs/notes/mq-checks and the origins and landing records of commits as refs/notes/mq; a third, refs/notes/mq-evidence, carries the evidence about commits to a reader of a public copy, and nothing else of mq's travels. Where the code still keeps a fact elsewhere, design.md's Where the code stands names it.

The base and the description are not in .git/config. An agent sandbox refuses every write to that file, since a value there such as core.hooksPath runs code outside the sandbox, and mq is run by such agents: mq new, mq field, mq land, mq dequeue and mq drop write no key of it. For a branch mq keeps no record of, such as one made with git branch --track, mq reads the base and the description Git holds for it, branch.NAME.merge when it names a local branch and branch.NAME.description, and writes them only into its record, the first time it records something of that branch. The cost is that git status and git branch -vv do not show the base of a branch mq made.

Six consequences follow.

  • mq writes its refs under refs/mq/ and refs/notes/, and none under refs/heads/. Its check worktrees stand on a detached HEAD. git branch then lists the DAG's branches, the rules (mq/config, Configuration) and the work, and nothing else. A queue that keeps them under refs/heads/ buries the work: in one measured checkout, 43 of 59 local branches were the queue's own.
  • A check result is a line of the note on the tree it is about, in refs/notes/mq-checks (Check results). git notes --ref=mq-checks show COMMIT^{tree} prints which checks passed on the tree a landing left, under which rules, on which platform and on which machine, without mq installed, and mq checks COMMIT prints them by check and platform.
  • Every move mq makes is in the branch's reflog. mq writes the entry Git would write, with a message that names the land or the pull that moved the branch (mq land fix-ttl: 1 commit). git reflog show staging is therefore the list of landings into staging on this machine, and staging@{1} is its tip before the last one. mq events adds what moved no ref (The commands).
  • A landed branch is merged in Git's eyes. A replay gives the landed commits new ids, and a branch left on the old ones would read as unmerged for ever. A branch mq made a worktree for is deleted when it lands (After a land). Any other branch is moved to the commit it landed as and, if it had no base, given its target as one in its record, so it is level with its target and git branch -d NAME succeeds.
  • A branch that a checkout has out moves with its checkout, or not at all. Moving a branch under a checkout leaves that checkout's index on the old commit, and its next commit reverts whatever the move brought. mq therefore moves a checked-out branch as git merge --ff-only moves it: the files come along, and uncommitted changes the move does not touch stay where they are. Only a change the move would overwrite stops it. When the checkout holds the target of a land, or a branch a pull moves, the call ends stuck, naming the checkout and the paths, since a scheduled land has no caller to read a refusal. When it holds the landed work branch, the landing stands and the branch stays on its old commits (After a land). A pull that takes a checked-out branch back to the agreed copy's value moves the checkout as git reset --keep does, under the same condition.
  • A long-lived branch keeps its remote upstream. staging tracks origin/staging, as Git sets it up, so git status counts what is not pushed. An upstream that is a remote-tracking branch of another name (origin/staging, for a branch made from it) names the local branch (staging) wherever this specification says "NAME's upstream".

mq new sets no upstream and no push remote in Git's configuration, so git push in a work branch asks where to push, as Git asks for any branch with no upstream, unless push.autoSetupRemote is set; git push -u origin NAME publishes it, ready for a pull request, and never touches staging. Whatever upstream that sets, the base stays in mq's record. mq new records the base twice, as the record's upstream and as a base field in the description, which land reads when the upstream no longer names a local branch.

Failure prevented: a second copy of Git's state under .git/mq/ that drifts from the first. The index derives every row from Git or the journal and is checked against them before a read, and a commit one of mq's records must keep is kept in refs/mq/keep's reflog in the same transaction that takes it from its last branch.

3. A Git word only when Git's behaviour carries over exactly #

A command that changes work (a ref, a worktree, a remote) takes a Git name only when it passes three tests. It takes the same arguments in the same order, has the same effect on refs and worktree, and gives every flag it shares with Git the same meaning. Otherwise it takes a word Git does not use. A flag Git uses with one meaning across its commands is held to the same tests.

A familiar word that does something else (a false friend) costs more than an unfamiliar word. An unfamiliar word sends the reader to --help. A false friend gets typed with confidence. The table gives seven that a command line named after Git would hold.

Git-named command What a Git user reads What mq would do Consequence
mq branch NAME -m MSG -m renames a branch set the description a rename typed as a description, or the reverse
mq merge staging bring staging into my branch queue staging into main the commonest Git command moves main, and --force skips the checks
mq log fix-ttl the commits on fix-ttl print the queue's events the wrong history
mq land --detach --detach detaches HEAD (git switch, git worktree add) queue without waiting a flag that mq new's own worktrees give a Git meaning to
mq drop --force --force discards what stands in the way throw uncommitted changes away an override that does not say what it overrides, typed from habit
mq land --until main --until is a date (git log --until) wait for a branch a flag whose value a Git user expects to be a time
mq new --trailer 'Bug: X' --trailer writes into the commit message write into the branch's description, which is never pushed metadata the author believes is in the history

The rule keeps Git's nouns (branch, upstream, worktree, rebase, hook), since the data model is shared, and gives mq its own commands, since the behaviour is not. Taking no Git word at all would lose rule 2, since Git's own tools would no longer show mq's state. Graphite (gt create, gt submit), Jujutsu and Phabricator's arc land made the same choice. -C PATH, --onto and --dry-run pass the tests where this design uses them, and so does --no-verify, which skips the checks as git commit --no-verify skips the hooks. --force, --detach, --until, --trailer and -m appear nowhere in it.

Two kinds of command are outside the tests, and the naming table says which each one is. A command that only reads, or that sets mq itself up, may share a word with Git when it does for the queue what Git's does for the repository: mq status and mq init. A command that changes work may keep a Git word that fails the tests only when its Git-habit misuse is refused or moves nothing. mq push and mq pull pass that test. The direction, the refusal when the remote has moved and the remedy are Git's. mq push origin main pushes one branch as git push origin main does, and mq push origin fix-ttl, typed from habit on a work branch, is refused with a sentence that says a work branch is pushed with Git. merge and --force fail it: their habitual use would move a branch.

One accident the rule cannot name away is prevented by an argument. A Git user standing in a work branch types mq land staging from the habit of git rebase staging, and would land staging into main. A branch the rules name is therefore landed only with --onto, and the refusal prints no next:: printing mq land staging --onto main would hand an agent the very command the refusal exists to stop.

4. A land is checked, or says it is not, and mq controls nothing else #

land moves a branch only by a fast-forward to a tree that passed that branch's checks, unless the caller types --no-verify, which skips them and says so: the landing is journaled as unverified, with who made it and when, and mq status and mq stats show it. mq pull is the one other command that moves a branch another way. It takes the agreed copy's value, and sends this machine's own landings back through the checks (Several machines). Moving staging into main is mq land staging --onto main, the same command as any other land, so no separate command promotes a branch.

Every other way a branch moves is ordinary Git and none of mq's business. A pull request merged on a forge is one, and so are a git push and a commit on a checked-out branch. mq takes each branch as it finds it, and never refuses, complains about or undoes such a move. A branch the rules name that is deleted with Git stays deleted: a run never recreates it, a land queued into it ends refused naming the missing target, and mq status names it. mq status states one fact about each branch of the DAG, whoever moved it: whether its tip is checked. A red that a move brings is a failure of the tip, so the next land is not blamed for it. Push rights and review belong to the team's workflow and its server's settings (Teammates, forges and peers).

GitHub's rulesets carry a bypass list, and Chromium documents a direct commit that goes around its commit queue. Git has its own: git commit --no-verify and git push --no-verify skip the hooks (githooks(5)). A branch's checks are mq's hooks, so mq's bypass is the same flag, mq land staging --onto main --no-verify (The commands). It has exactly Git's safety and nothing more: it moves main as git merge --ff-only staging moves it in main's checkout, so only a fast-forward moves it, the checkout comes along, and only a change the move would overwrite stops it. The raw Git way is awkward, which is why mq spells it: git push . staging:main is refused while main is checked out, and the right command has to be typed in that checkout. A move made with Git stays a bypass mq never refuses, and the flag adds only the record of who skipped which checks.

A commit placed on main while main takes staging is normal operation, and mq absorbs it. main then holds something staging lacks, and no fast-forward of main to staging exists. The next run replays main's own commits, those since the two branches forked, onto staging, which only grows, checks the replayed tree like any land into main, and moves staging and then main to it, so the edge is a fast-forward again. A replay that conflicts names the repair into staging, mq new repair/main staging. The serve leaves out each commit of main that staging holds as a copy: a copy with its patch, as git rebase leaves out a commit whose change the branch already holds, and a copy resolved by hand against staging's own changes, which is the resolution. A copy that leaves out a path staging already holds as main's commit makes it, as git cherry-pick does, is still that commit's copy, and so is a copy resolved at staging's tip, after later changes next to its own. Only the rest of main's commits are replayed; when none is left, the replay is staging's tip itself. main's move is the one that is no fast-forward: each of its commits made with Git is replaced by its replay or by staging's copy, so its change is never discarded, on this machine or by a pull, and main's reflog keeps the original. When a land into staging moved it past the tip the serve replayed onto, the land leaves staging there, lacking main's replays, and the run rebases it onto main before any land of the reading starts, as git rebase main would: its own commits since that tip are replayed onto main's tip, or onto the commit the serve landed as when they conflict there, and staging's reflog keeps the commits it held before. No later serve then replays a change main already holds. mq writes no merge commit: a land is a replay. A commit placed on main this way is a normal event: lands into staging keep landing while the edge waits for its serve, checked against staging's tip as it stands.

Only a true conflict needs a decision: two different edits of one region that neither side has resolved. The serve is a land no terminal typed, so that conflict meets the resolver as any such land's does when the rules send it there ([resolve] default = "auto", Conflicts): the replay stops at each commit of main that conflicts with staging, the resolver stages its answer, and the resolved series is checked under main's checks and lands, the outcome naming the resolver. A conflict the resolver refuses, one the rules keep from it, or a red replay, moves nothing and ends the edge's land needs_work, which holds the edge alone, until staging, main, the rules or a repository main's checks declare they read move, and never the lands into staging. The outcome counts the commits main holds and staging lacks, names the oldest, and names what to repair into staging. A conflict names main, whose commit is the newcomer, and is told as any land's conflict is (Conflicts), with its paths and main's commit the replay tried. The conflict is a function of the two commits, an observation of the index made again by a dry merge when missing (The state), so it outlives the journal. A red is decided by checking main's own commits first, as they stand on main, and their replay onto staging last: when one of main's commits fails on its own, the culprit is that commit made with Git; when main's tip passes, it is the replay. Either is repaired as any change, landed through mq into staging, which main then takes, never by a hand edit the outcome prescribes: the next: is mq new repair/main staging. A copy of main's commit resolved against staging's change and landed so is one the next serve takes as held. How mq push publishes a main so moved, which is no fast-forward of its agreed copy, and how another machine's mq pull takes it, is an open point.

Failure prevented. Without this rule, a user would have to learn a second kind of branch before landing anything, and the tool would object to what Git allows.

5. Every call ends in one outcome #

A waiting command writes its progress to standard error and its outcome to standard output. The outcome is one block. Its first line starts with the outcome word, and the indented lines under it are time:, log:, worktree: and next:, each present only when it has something to say. --json prints the same block as one object. Exit 0 means that the call did what it was asked, and the outcome word says which: landed, passed for a dry run, queued for a call asked not to wait. Every other outcome has one exit code, and each exit code has one party whose move it is.

There are nine outcome words. Five are the engine's own states, landed, passed, needs_work, stuck and removed. queued ends a call that was asked not to wait, and interrupted one that Ctrl-C stopped. refused ends a call that cannot be made in this state, and unavailable one the machine could not serve. The text, the JSON and the event stream use this one vocabulary for outcomes. mq status adds two states for a branch still in the queue, queued and checking, and the event stream adds the words of the commands that do not wait (Outcomes and exit codes).

next: is one command this caller can run as printed. It is absent when the next move is the operator's, after unavailable and stuck, and when the only command to print would be the accident a refusal prevents. The one exception is an unavailable whose cause mq doctor reports with its remedy, such as rules that do not load, where it is mq doctor. After needs_work it is the command that leads to the fix. That is mq land NAME, to run once the change is fixed, or mq new NAME when the branch has no worktree to fix it in.

On a terminal the progress is a live panel. Anywhere else it is one line for each change of state, and at least one line every thirty seconds while a check runs and nothing changes. A caller whose harness ends a silent call can then tell a slow check from a dead one. A wait for another process's run says where it waits once, and again when its place changes: the line repeated would say nothing new, and a run that is gone ends the wait at its bound.

Failure prevented: a script or an agent that parses a progress panel, or that cannot tell "your check failed" from "you called it wrong" from "the machine failed".

6. A worktree path is handed out once #

A path handed out for fix-ttl never becomes another branch's worktree, or a later branch of the same name, while anything that holds its serial stands or a line of refs/mq/keep's reflog keeps its work. The directory behind it is parked and reused once nothing stands in it; a shell or editor left in it after a land finds the directory gone, and never commits into someone else's work (After a land).

Failure prevented: the worked session W=$(mq new fix-ttl) leaving $W, or a shell standing in it, in a worktree handed out for other work.

7. A machine never waits for another #

Every machine runs its own queue, with its own checks, and land finishes on a laptop with no network. A remote holds history that others agreed to, whether a team shares it or one person publishes it (Several machines and Teammates, forges and peers). The user moves work between machines with two explicit steps, mq pull and mq push, as with Git. No machine is the primary and no server runs mq: a remote is a plain Git repository, and no command needs another machine to be reachable except the pull or push the user typed. A team may let one machine alone write a shared branch. Losing that machine stops the shared copy moving, never anyone's local work.

Failure prevented: a land that blocks because one host is unreachable, and a primary whose loss stops every other machine.

8. Nothing leaves the machine except by a network command #

init, pull and push are the only commands with which mq contacts another machine, and only when the user types them. land never does, and mq never talks to a forge's interface. No background process fetches or pushes. A command the rules name (a check, the resolver, the hook) is the team's own, and does what its author wrote.

Failure prevented. No command fails, and no branch moves, because of a network nobody asked mq to use.

9. No mq process stays resident #

mq has no daemon. A run is a few bounded rounds: each reads the queue, decides, starts the tasks that take time (a check, a replay, a cone ask, a resolver or hook call) under leases, writes, and returns (Who runs the checks, design). A task writes its own answer and the next round reads it, so no process waits on another and a run lasts seconds. A run is started by the land or the pull that needs it, by the operating system's scheduler on a machine registered with it, or by mq/wake, and never by a process that stays behind. The scheduler's tick and the reference-transaction hook start mq briefly and often; each such process finds nothing to do and exits within a second. The scheduler is registered with a stable path to mq, so the next round uses an upgraded mq with nothing to restart, and a task finishes under the binary that started it. A run whose mq writes a registration other than the one the machine holds registers again before it lands, and journals it. Gradle's daemon, which is resident, serves only its own version, and an idle one stops only after three hours.

Failure prevented. No resident process runs last week's binary, no install step comes before the first land, and no verdict depends on which caller happened to start the checker.