solstone
personal memory agent
Star
Fork
Watch
solstone docs testing.md
5.8 kB · 123 lines
Markdown
at main

Testing #

Test Structure #

  • Framework: pytest; coverage reporting comes from make test-cov, make verify, or make coverage, not make test or make ci
  • Unit Tests: live under tests/ (and each app's tests/ dir)
    • Fast, with mocked process/thread/clock/network/repository boundaries
    • No external API calls, real browser, heavyweight build, or shared fixture writes
    • Read tests/fixtures/journal/ mock data; use journal_copy or tmp_path for any scan, index rebuild, or mutation
    • Test individual functions and modules
  • Integration Tests: marked @pytest.mark.integration
    • Opt-in via make test-integration; excluded from make test and make ci
    • Real local processes/builds and persisted-index contracts
    • Still use disposable tmp_path state and never the owner's journal
  • Release Tests: marked @pytest.mark.release
    • Run serially via make test-release; excluded from development make test and make ci
    • Cover release transactions, release entrypoints, packaging/install probes, and release-host tool contracts
    • make release-checks combines them with the real advisory-policy liveness and minisign gates; candidate construction runs that target automatically
  • Naming: Files test_*.py, functions test_*
  • Fixtures: Shared fixtures in tests/conftest.py

Fixture Journal #

# The autouse set_test_journal_path fixture in tests/conftest.py does this
# for unit tests. Set it explicitly only when a test needs a different journal.
os.environ["SOLSTONE_JOURNAL"] = "tests/fixtures/journal"
# Now all journal operations work with test data

The tests/fixtures/journal/ directory contains immutable mock input with sample facets, agents, transcripts, and indexed data. Tests may read it directly. Any test that writes, scans, or rebuilds journal/index state must use the journal_copy fixture or a smaller journal under tmp_path.

Running Tests #

  • make test runs all unit tests — tests/ + every solstone/apps/*/tests/, in one parallel run
  • make test-cov — the same suite with coverage reporting
  • make test-integration — opt-in real local build/process and persisted-index contracts
  • make test-release — serial release transactions and release-host probes
  • make release-checks — complete candidate-host validation, including the release tests, advisory-policy liveness, and real minisign signing
  • make test-app APP=<name> and make test-only TEST=path are the focused development loop
  • make coverage to generate a coverage report
  • make ci once on the settled final tree before merge or release (install checks plus the full unit suite)
  • Always run journal restart-convey after editing solstone/convey/ or solstone/apps/ to reload code

OpenAPI Verification Lanes #

There are two API referees with different jobs:

  • tests/test_openapi_schemathesis.py fuzzes a small allowlist from the committed native-client contract at docs/openapi/convey-clients.json against the Flask WSGI app. It runs with the normal unit suite because it uses an isolated tmp journal and does not use HTTP transport sockets.
  • make verify-api checks SPA/API response baselines against a running sandbox. That lane verifies rendered baseline behavior, not the native-client OpenAPI contract.

Run the Schemathesis lane directly with:

make test-only TEST=tests/test_openapi_schemathesis.py

The operator live lane is:

make verify-schemathesis

verify-schemathesis starts a disposable sandbox, sets SOLSTONE_SCHEMATHESIS_LIVE=1, and resolves the base URL through solstone.think.convey_client.resolve_base_url(). Set SOLSTONE_SCHEMATHESIS_BASE_URL to override the target. The live lane may grow to include mutating routes because it targets disposable instances; do not run it as part of ordinary unit CI.

The WSGI lane uses Schemathesis' default checks, but pins input generation to positive cases only. That means checks such as status-code conformance, content-type conformance, response headers, response schema, server-error rejection, and ignored-auth behavior still run, while negative_data_rejection has no generated negative inputs in this lane. The positive-only setting keeps the floor assertions meaningful: every generated floor case must return 2xx.

Known findings are reported by this lane and triaged separately; the lane must not auto-fix them or widen the contract to pass. A contract-vs-implementation mismatch is triaged by asking which side is wrong. Current recorded findings:

  • GET /app/network/api/status materializes link CA key material on a read path: ca_dir().exists() first creates journal/link/ca, then _ca_fingerprint() calls load_or_generate_ca() and writes private.pem plus cert.pem.
  • GET /app/network/api/status reaches _detect_lan_ip(), which opens a UDP socket to 8.8.8.8:80 to read the kernel-selected source address. The WSGI test harness reaches that syscall.

Worktree Development #

Run the full stack (supervisor + callosum + sense + cortex + convey) against test fixture data:

make dev                    # Start stack (Ctrl+C to stop)

In a second terminal, hit endpoints:

export SOLSTONE_JOURNAL=tests/fixtures/journal
export PATH=$(pwd)/.venv/bin:$PATH
curl -s http://localhost:$(cat tests/fixtures/journal/health/convey.port)/

Notes:

  • Agents won't execute without API keys — this is expected in worktrees
  • Output artifacts go in scratch/ (git-ignored)
  • Service logs: tests/fixtures/journal/health/<service>.log
  • make dev writes runtime artifacts (stats cache, health logs, task logs) into the fixtures journal — these are covered by tests/fixtures/journal/.gitignore and should never be committed