Performance Testing #

Blit-Tech has CPU micro-benchmarks for hot methods. This guide explains when to use them, how to add a new benchmark, and how CI uses the results.

Table of Contents #

• Overview
• CPU Benchmarks
• Adding a New CPU Benchmark
• Commands
• CI Benchmark Workflow
• Recommended Workflow for New Performance Work

---

Overview #

Blit-Tech uses Vitest bench for CPU micro-benchmarks. These measure isolated methods, hot loops, cache lookups, math helpers, and allocation patterns.

For visual correctness (not performance), use the visual regression tests: pnpm test:visual. They run Playwright with Chromium + WebGPU and produce PNG snapshots. See docs/testing.md for details.

When to Use CPU Benchmarks #

Use CPU benchmarks when you want to measure a single method, hot loop, cache lookup, math helper, allocation pattern, or batching helper in isolation.

Examples:

• Vector2i.add() vs Vector2i.addInPlace()
• Color32.toFloat32Array() vs Color32.writeToFloat32Array()
• BitmapFont.measureText() cold vs warm cache
• Rect2i.containsXY() vs Rect2i.contains()

---

CPU Benchmarks #

CPU benchmarks are implemented with Vitest bench and colocated next to the source as *.bench.ts files.

Current benchmark files:

• src/utils/Vector2i.bench.ts
• src/utils/Color32.bench.ts
• src/utils/Rect2i.bench.ts
• src/assets/BitmapFont.bench.ts
• src/assets/PaletteEffect.bench.ts

Metrics #

CPU benchmarks report ops/sec. Higher numbers are better.

If one benchmark shows:

• 20,000,000 ops/sec

and another shows:

• 10,000,000 ops/sec

the first one is roughly twice as fast.

Why This Is the Default Choice #

CPU benchmarks are:

• faster to run locally
• easier to write
• easier to reason about
• already integrated into CI regression checks when the PR is labeled perf

If you add a new hot method and want immediate automated regression protection, this is the first tool to use.

---

Adding a New CPU Benchmark #

If you add a new method and it can run without a browser, start here.

File Location #

Create or extend a *.bench.ts file near the code being measured.

Examples:

src/utils/MyType.bench.ts
src/render/SpritePipeline.bench.ts

Basic Structure #

import { bench, describe } from 'vitest';

import { MyType } from './MyType';

describe('MyType hot paths', () => {
  const instance = new MyType();

  bench('newMethod()', () => {
    instance.newMethod();
  });

  bench('oldMethod()', () => {
    instance.oldMethod();
  });
});

What to Compare #

Good benchmark comparisons usually measure one meaningful tradeoff:

• new method vs previous method
• allocating vs in-place
• small input vs large input
• cold cache vs warm cache
• vector argument vs raw x, y argument

Benchmark Design Rules #

• keep the benchmark focused on one hot behavior
• use realistic inputs, not absurd synthetic values unless stress testing is intentional
• compare alternatives in the same file so the relative result is obvious
• prefer stable setup outside the bench(...) callback
• if mutation is involved, reset state inside the benchmark or in setup so each iteration is valid

Run CPU Benchmarks #

pnpm bench
pnpm bench:json

pnpm bench gives the terminal report.

pnpm bench:json writes benchmark-results.json, which is what CI uses.

---

Commands #

pnpm bench       # Run all CPU benchmarks
pnpm bench:json  # Run benchmarks and write benchmark-results.json

Which Command Should I Use? #

• New hot method: pnpm bench
• Need machine-readable result for comparison: pnpm bench:json

---

CI Benchmark Workflow #

CPU benchmark regression detection is wired into GitHub Actions.

What Happens on main #

On pushes to main, CI:

1. runs pnpm bench:json
2. produces benchmark-results.json
3. uploads that file as the latest benchmark baseline artifact

That artifact becomes the reference point for future pull requests.

What Happens on a Pull Request #

On PRs targeting main with the perf label, CI:

1. runs the normal quality job first
2. runs the benchmark job
3. runs pnpm bench:json
4. downloads the latest successful main benchmark baseline artifact
5. compares PR results against the main baseline
6. posts or updates a PR comment with a benchmark comparison table
7. fails the job if any benchmark is more than 25% slower

PRs without the perf label skip the benchmark job to reduce CI cost.

What the PR Comment Contains #

The comparison comment includes:

• benchmark name
• baseline ops/sec
• current ops/sec
• delta percent
• pass/fail status

What Counts as Failure #

The benchmark CI job fails if:

• any benchmark regresses by more than 25%
• a previously existing benchmark disappears from the PR run

---

Recommended Workflow for New Performance Work #

If you add a new sprite operation, follow this order:

1. Write the code clearly first.
2. Add a CPU benchmark if the method can run in Node.
3. Run pnpm bench locally to compare the new method against the old behavior or an alternative implementation.
4. Run pnpm bench:json if you want to inspect the machine-readable output used by CI.
5. Open a PR and add the perf label for CPU benchmark comparison.

Best Default #

If you are unsure where to start:

• use CPU benchmarking with Vitest bench

It is simple, fast, and already supported by the label-gated benchmark CI.