diff --git a/.gitignore b/.gitignore index bbd64a0..5896d43 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ coverage/ .claude/settings.local.json data/ .vscode +site-config.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 442bc93..274f392 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ What changed in each release of SCN Chat, for the people who use it, run it, or ## [Unreleased] +### Added + +- For admins: the whole admin configuration, its settings, roles, plugins, and admin models, can be exported to a JSON file and applied from one with `site-config export` and `site-config apply`, run on the server, so a deployment tool like Ansible can set up a server and keep it as described. `docs/deployment.md` has the commands. + ### Changed - A streaming reply's words fade in as they arrive, rather than appearing at once. Nothing moves under reduced motion. @@ -11,10 +15,12 @@ What changed in each release of SCN Chat, for the people who use it, run it, or - Loading a provider's models while adding an API key lists them to tick, rather than adding every one. - A key's page has a Delete card, so a key can be deleted where its models are edited. - Rebuilding this device's copy of your chats asks to confirm first. +- Plugin instances are named after the plugin they make, such as `web-search` or `openai-compatible-litellm`, rather than a random ID. Instances added before this are renamed at startup, keeping their options and secrets. ### Fixed - A reply's step dots no longer show over the composer or the header as the chat scrolls under them. +- Saving a plugin in the admin area while another plugin change is being applied no longer undoes that change. ## [0.2.2] - 2026-10-16 diff --git a/CLAUDE.md b/CLAUDE.md index 4d5a5f8..0445312 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ An AI assistant chat app where each conversation is an atproto permissioned spac - `apps/server` is the Hono server. `src/server.ts` wires every service together. - `apps/web` is the TanStack Start app, which the server runs inside Hono. Its code is grouped by feature in `src/features/`, with thin routes in `src/routes/` and cross-feature code in `src/shared/`. `src/parts/` holds the views that draw message parts, and `src/store/` is the browser's local SQLite copy. - `plugins/` holds the shipped plugins. -- `docs/architecture.md` is the human-facing architecture overview, and `docs/plugins.md` the plugin author guide. Update them when a change affects what they describe. +- `docs/architecture.md` is the human-facing architecture overview, and `docs/plugins.md` the plugin author guide. `README.md` and everything in `docs/` are for people, so a person has the final word on them. When a change affects what they describe, draft the update and point it out, so the person can rewrite it in their own words. - `CHANGELOG.md` records each change a user, admin, or forker would notice: add its line under Unreleased in the same commit. `docs/releasing.md` has the versioning rules and the steps for cutting a release. - `GLOSSARY.md` holds the project's terms, and `docs/adr/` the decisions behind its less obvious choices. Read the relevant ADR before changing what it decided. - `pnpm dev` runs the server, which serves the web app through Vite, as one process, and `pnpm dev --scan` also loads React Scan in the browser. `pnpm preview` builds the web app and runs the production build locally. `pnpm dev:remote` and `pnpm preview:remote` do the same behind a localhost.run tunnel over ssh, with the tunnel's HTTPS URL as `PUBLIC_URL`, for testing from a phone. `pnpm start` runs the production build for a deployment. `pnpm check` runs lint, type checking, and every test. `pnpm codegen` regenerates the lexicon code after a lexicon change, and `pnpm publish-lexicons ` publishes the lexicons from the lexicon authority account. `pnpm keys` prints fresh secrets. @@ -26,7 +26,7 @@ These are settled. Raise a concern before working against any of them. - Every record belongs to the user, including the assistant's replies. The running app never writes records under its own DID. The one exception is publishing the lexicons, which the maintainers do from a lexicon authority account, outside the app. - One space per conversation, of type `network.sharedcomputer.chat.conversation`. Sharing a chat means changing that space's read policy. Fallback users cannot share or view shared chats. -- There is no public HTTP API, apart from XRPC procedures defined in our own lexicons, currently `requestSync`, and routes opened to admin-issued API keys, currently cron. The user's PDS is the API: a client can hold a whole conversation by writing records directly. The conventions for that are in `lexicons/README.md`, and the server must honor them exactly. +- There is no public HTTP API, apart from XRPC procedures defined in our own lexicons, currently `requestSync`, and routes opened to admin-issued API keys, currently cron and site config. The user's PDS is the API: a client can hold a whole conversation by writing records directly. The conventions for that are in `lexicons/README.md`, and the server must honor them exactly. - Users whose PDS does not support spaces fall back to local storage in SQLite or Postgres. It sits behind the same PDS-shaped record interface as the spaces backend, so it can be deleted once PDSs support spaces. Local rows store the exact lexicon record JSON under the same space and record keys, so migrating a user to their PDS is a replay. - Each conversation has an entry record in the user's settings space. Together the entries are the user's chat index, which is how the server and browser find conversations and notice changes with one call per user. - The browser keeps its own local copy of the user's chats in SQLite compiled to WebAssembly, for the chat list, the open conversation, and search. That copy lives on the user's device, not on the server. diff --git a/apps/server/package.json b/apps/server/package.json index 441a44d..d4be6dc 100644 --- a/apps/server/package.json +++ b/apps/server/package.json @@ -13,7 +13,8 @@ "start:remote": "node src/tunnel-cli.ts --start", "migrate": "node src/migrate-cli.ts", "typecheck": "tsc --noEmit -p .", - "keys": "node src/keys-cli.ts" + "keys": "node src/keys-cli.ts", + "site-config": "node src/site-config-cli.ts" }, "dependencies": { "@atproto/common-web": "0.0.0-spaces-alpha-20261001173819", diff --git a/apps/server/src/admin/config-routes.ts b/apps/server/src/admin/config-routes.ts new file mode 100644 index 0000000..7a4c512 --- /dev/null +++ b/apps/server/src/admin/config-routes.ts @@ -0,0 +1,22 @@ +import { Hono } from 'hono' +import { z } from 'zod' +import { requireApiKey } from '../auth/api-keys.ts' +import { ADMIN_ROLE } from '../auth/roles.ts' +import { jsonBody } from '../body.ts' +import type { Db } from '../db/index.ts' +import type { AppEnv } from '../env.ts' +import type { Logger } from '../logger.ts' +import type { SiteConfig } from './site-config.ts' + +/** The whole admin configuration, for `site-config` and other callers with an admin key. */ +export function siteConfigRoutes(deps: { db: Db; logger: Logger; siteConfig: SiteConfig }) { + return new Hono() + .use(requireApiKey(deps, ADMIN_ROLE)) + .get('/', async (c) => c.json(await deps.siteConfig.export())) + .put('/', async (c) => { + const raw = await jsonBody(c, z.unknown()) + const key = c.get('apiKey') + if (!key) throw new Error('requireApiKey let a request through without a key') + return c.json(await deps.siteConfig.apply(raw, `api-key:${key.id}`)) + }) +} diff --git a/apps/server/src/admin/routes.ts b/apps/server/src/admin/routes.ts index 7bf7e09..246927d 100644 --- a/apps/server/src/admin/routes.ts +++ b/apps/server/src/admin/routes.ts @@ -4,17 +4,19 @@ import { sql } from 'kysely' import { z } from 'zod' import { type Access, CannotSuspendAdmin } from '../auth/access.ts' import { getAccount } from '../auth/accounts.ts' -import { issueApiKey, listApiKeys, revokeApiKey } from '../auth/api-keys.ts' +import { apiKeysHolding, issueApiKey, listApiKeys, revokeApiKey } from '../auth/api-keys.ts' import type { IdentityResolver } from '../auth/identity.ts' import { addInvite, listInvites, removeInvite } from '../auth/invites.ts' import { addMember, createRole, deleteRole, + hostnames, listRoles, RoleExists, removeMember, roleExists, + roleName, updateRole, } from '../auth/role-store.ts' import { ADMIN_ROLE, IMPLICIT_ROLE } from '../auth/roles.ts' @@ -34,16 +36,9 @@ export type AdminRoutesDeps = PluginAdminRoutesDeps & { } const PAGE_SIZE = 50 -const ROLE_NAME = /^[a-z][a-z0-9-]*$/ -const HOSTNAME = - /^(?=.{1,253}$)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+$/ - -const hostnames = z.array( - z.string().trim().toLowerCase().regex(HOSTNAME, 'must be a hostname, such as example.com'), -) const newRole = z.object({ - name: z.string().regex(ROLE_NAME, `must match ${ROLE_NAME}`), + name: roleName, description: z.string().default(''), }) @@ -69,9 +64,7 @@ async function roleUses(deps: AdminRoutesDeps, name: string): Promise if ((JSON.parse(model.roles_json) as string[]).includes(name)) uses.push(`model ${model.provider}/${model.model_id}`) } - for (const key of await listApiKeys(deps.db)) { - if (key.roles.includes(name)) uses.push(`API key ${key.label}`) - } + for (const label of await apiKeysHolding(deps.db, name)) uses.push(`API key ${label}`) return uses } diff --git a/apps/server/src/admin/site-config.ts b/apps/server/src/admin/site-config.ts new file mode 100644 index 0000000..9c67382 --- /dev/null +++ b/apps/server/src/admin/site-config.ts @@ -0,0 +1,499 @@ +import { isDeepStrictEqual } from 'node:util' +import { isValidDid } from '@atproto/syntax' +import { z } from 'zod' +import { apiKeysHolding } from '../auth/api-keys.ts' +import { + addMember, + hostnames, + insertRole, + listRoles, + type RoleSummary, + removeMember, + removeRole, + roleName, + updateRole, +} from '../auth/role-store.ts' +import { ADMIN_ROLE, IMPLICIT_ROLE } from '../auth/roles.ts' +import { describeIssues, InvalidBody, type Issue, validate } from '../body.ts' +import type { Db } from '../db/index.ts' +import type { Logger } from '../logger.ts' +import { type InstalledPlugins, optionsJsonSchema, secretKeys } from '../plugins/installed.ts' +import { type PluginInstance, readInstances, replaceInstances } from '../plugins/instances.ts' +import { + instantiate, + PluginInstanceError, + type PluginRuntime, + type RuntimeBuilder, + type RuntimeHolder, +} from '../plugins/runtime.ts' +import { + type AdminModel, + adminModelSchema, + listAdminModels, + replaceAdminModels, + servesAdminModels, +} from '../providers/admin-models.ts' +import { jsonObject } from '../schemas.ts' +import type { SecretBox } from '../secrets.ts' +import { type SettingKey, type Settings, settingSchemas } from '../settings/schemas.ts' +import { SettingsStore } from '../settings/store.ts' + +const INSTANCE_ID = /^[A-Za-z0-9][A-Za-z0-9_-]*$/ + +const roleEntry = z.object({ + name: roleName, + description: z.string().optional(), + pdsHosts: hostnames.optional(), + handleDomains: hostnames.optional(), + members: z + .array(z.string().refine((value): boolean => isValidDid(value), 'must be a DID')) + .optional(), +}) + +type RoleEntry = z.infer + +const pluginEntry = z.object({ + id: z.string().regex(INSTANCE_ID, `must match ${INSTANCE_ID}`), + package: z.string().min(1), + enabled: z.boolean().default(true), + options: jsonObject.default({}), + secrets: z.record(z.string(), z.unknown()).default({}), + secretsSet: z.array(z.string()).optional(), +}) + +/** A server's whole admin configuration, as `site-config` exports and applies it. */ +export const siteConfigSchema = z.object({ + version: z.literal(1), + // A setting group left out keeps its stored value. + settings: z.object(settingSchemas).partial(), + roles: z.array(roleEntry), + plugins: z.array(pluginEntry), + models: z.array(adminModelSchema), +}) + +export type ConfigDocument = z.infer + +export type ApplyResult = { changed: boolean; changes: string[]; warnings: string[] } + +export type SiteConfigDeps = { + db: Db + box: SecretBox + settings: SettingsStore + installed: InstalledPlugins + holder: RuntimeHolder + build: RuntimeBuilder + logger: Logger +} + +const SETTING_KEYS = Object.keys(settingSchemas) as SettingKey[] + +const isRecord = (value: unknown): value is Record => + typeof value === 'object' && value !== null && !Array.isArray(value) + +/** Fields of the input that parsing dropped, however deep, such as a misspelled nested setting. */ +function droppedFields(input: unknown, output: unknown, path: Issue['path'] = []): string[] { + if (Array.isArray(input) && Array.isArray(output)) + return input.flatMap((item, i) => droppedFields(item, output[i], [...path, i])) + if (!isRecord(input) || !isRecord(output)) return [] + return Object.entries(input).flatMap(([key, value]) => + key in output ? droppedFields(value, output[key], [...path, key]) : [[...path, key].join('.')], + ) +} + +/** Does a plugin keep this role's members in sync? */ +const pluginManaged = (role: RoleSummary) => + role.members.some((member) => member.addedBy.startsWith('plugin:')) + +/** Added, updated, removed, and reordered entries between two keyed lists. */ +function listChanges(what: string, before: T[], after: T[], key: (item: T) => string): string[] { + const old = new Map(before.map((item) => [key(item), item])) + const changes: string[] = [] + for (const item of after) { + const previous = old.get(key(item)) + if (!previous) changes.push(`${what} ${key(item)} added`) + else if (!isDeepStrictEqual(previous, item)) changes.push(`${what} ${key(item)} updated`) + } + const kept = new Set(after.map(key)) + for (const item of before) if (!kept.has(key(item))) changes.push(`${what} ${key(item)} removed`) + const order = (list: T[]) => list.map(key).filter((id) => kept.has(id) && old.has(id)) + if (!isDeepStrictEqual(order(before), order(after))) changes.push(`${what}s reordered`) + return changes +} + +/** Exporting the admin configuration, and making the server match a document of it. */ +export class SiteConfig { + private readonly deps: SiteConfigDeps + + constructor(deps: SiteConfigDeps) { + this.deps = deps + } + + async export(): Promise> { + const { db, box, settings } = this.deps + const [roles, instances, models] = await Promise.all([ + listRoles(db), + readInstances(db, box), + listAdminModels(db), + ]) + return { + version: 1, + settings: Object.fromEntries( + SETTING_KEYS.map((key) => [key, settings.get(key)]), + ) as ConfigDocument['settings'], + roles: roles.map((role) => ({ + name: role.name, + description: role.description, + pdsHosts: role.pdsHosts, + handleDomains: role.handleDomains, + ...(pluginManaged(role) ? {} : { members: role.members.map((member) => member.did) }), + })), + plugins: instances.map((instance) => ({ + id: instance.id, + package: instance.package, + enabled: instance.enabled, + options: instance.options, + secretsSet: Object.keys(instance.secrets), + })), + models, + } + } + + /** + * Make the server match the document. A document that would stop the server starting or break + * its references is refused, and anything a plugin can recover from is stored with a warning. + */ + apply(raw: unknown, by: string): Promise { + const doc = validate(siteConfigSchema, raw) + const warnings = droppedFields(raw, doc).map( + (path) => `${path} isn't a known field, so it was left out`, + ) + return this.deps.holder.exclusive(async () => { + const result = await this.applyNow(doc, by) + return { ...result, warnings: [...warnings, ...result.warnings] } + }) + } + + private async applyNow(doc: ConfigDocument, by: string): Promise { + const { db, box, holder } = this.deps + const [roles, stored, models] = await Promise.all([ + listRoles(db), + readInstances(db, box), + listAdminModels(db), + ]) + const removedRoles = roles.filter( + (role) => role.name !== ADMIN_ROLE && !doc.roles.some((entry) => entry.name === role.name), + ) + const proposed = this.proposeInstances(doc, stored) + await this.check(doc, removedRoles, stored, proposed) + const missing = SETTING_KEYS.filter((key) => doc.settings[key] === undefined) + const warnings = [ + ...missing.map((key) => `settings.${key} isn't in the file, so it was left as it is`), + ...this.pluginWarnings(doc, stored, proposed), + ] + const comparable = (instances: PluginInstance[]) => + instances.map((instance) => ({ + ...instance, + unreadableSecrets: instance.unreadableSecrets ?? null, + })) + const settingChanges = SETTING_KEYS.filter( + (key) => + doc.settings[key] !== undefined && + !isDeepStrictEqual(doc.settings[key], this.deps.settings.get(key)), + ) + const roleChanges = this.roleChanges(doc, roles, removedRoles) + const pluginChanges = listChanges( + 'plugin', + comparable(stored), + comparable(proposed), + (i) => i.id, + ) + const modelChanges = listChanges('model', models, doc.models, (m) => `${m.provider}/${m.id}`) + const changes = [ + ...settingChanges.map((key) => `setting ${key} updated`), + ...roleChanges, + ...pluginChanges, + ...modelChanges, + ] + const current = holder.current() + if (!changes.length) + return { changed: false, changes, warnings: [...warnings, ...health(doc, current)] } + + const runtime = pluginChanges.length ? await this.deps.build(proposed) : current + try { + await db.transaction().execute(async (tx) => { + for (const key of settingChanges) + await SettingsStore.write(tx, key, doc.settings[key] as Settings[typeof key], by) + if (roleChanges.length) await writeRoles(tx, doc, roles, removedRoles, by) + if (pluginChanges.length) await replaceInstances(tx, box, proposed, by) + if (modelChanges.length) await replaceAdminModels(tx, doc.models as AdminModel[], by) + }) + } catch (err) { + if (runtime !== current) await runtime.host.close() + throw err + } + if (runtime !== current) holder.swap(runtime) + for (const key of settingChanges) + this.deps.settings.adopt(key, doc.settings[key] as Settings[typeof key]) + this.deps.logger.info({ by, changes }, 'site config applied') + return { changed: true, changes, warnings: [...warnings, ...health(doc, runtime)] } + } + + /** Refuse a document whose references don't hold together, or that removes a role an API key holds. */ + private async check( + doc: ConfigDocument, + removedRoles: RoleSummary[], + stored: PluginInstance[], + proposed: PluginInstance[], + ) { + const issues: Issue[] = [] + const listed = new Set() + for (const [i, role] of doc.roles.entries()) { + if (role.name === IMPLICIT_ROLE) + issues.push({ path: ['roles', i, 'name'], message: `"${IMPLICIT_ROLE}" is built in` }) + else if (listed.has(role.name)) + issues.push({ path: ['roles', i, 'name'], message: `"${role.name}" is listed twice` }) + listed.add(role.name) + } + const roleNames = new Set([ADMIN_ROLE, IMPLICIT_ROLE, ...listed]) + const access = doc.settings.access ?? this.deps.settings.get('access') + for (const [i, role] of access.inviteRoles.entries()) { + if (!roleNames.has(role)) + issues.push({ + path: ['settings', 'access', 'inviteRoles', i], + message: `No such role: ${role}`, + }) + } + for (const role of removedRoles) { + for (const label of await apiKeysHolding(this.deps.db, role.name)) + issues.push({ + path: ['roles'], + message: `The "${role.name}" role is held by the API key ${label}, so it can't be removed`, + }) + } + + const ids = new Set() + for (const [i, entry] of doc.plugins.entries()) { + if (ids.has(entry.id)) + issues.push({ path: ['plugins', i, 'id'], message: `"${entry.id}" is listed twice` }) + ids.add(entry.id) + const installed = this.deps.installed.get(entry.package) + if ( + installed && + !installed.multiple && + doc.plugins.findIndex((p) => p.package === entry.package) < i + ) + issues.push({ + path: ['plugins', i, 'package'], + message: `${entry.package} can only be added once`, + }) + } + issues.push(...this.pluginIdIssues(stored, proposed)) + + const modelKeys = new Set() + for (const [i, model] of doc.models.entries()) { + const key = `${model.provider}/${model.id}` + if (modelKeys.has(key)) + issues.push({ path: ['models', i], message: `${key} is listed twice` }) + modelKeys.add(key) + for (const role of model.roles) { + if (!roleNames.has(role)) + issues.push({ path: ['models', i, 'roles'], message: `No such role: ${role}` }) + } + } + if (doc.models.filter((model) => model.default).length > 1) + issues.push({ path: ['models'], message: 'Only one model can be the default' }) + + if (issues.length) throw new InvalidBody(describeIssues(issues), issues) + } + + /** + * Refuse a change that renames a plugin, which strands what users stored under its old ID, or + * that gives two enabled instances the same plugin. + */ + private pluginIdIssues(stored: PluginInstance[], proposed: PluginInstance[]): Issue[] { + const issues: Issue[] = [] + const byId = new Map(stored.map((instance) => [instance.id, instance])) + const seen = new Set() + for (const [i, instance] of proposed.entries()) { + const pluginId = this.pluginIdOf(instance) + if (!pluginId) continue + const before = byId.get(instance.id) + const previous = before?.package === instance.package ? this.pluginIdOf(before) : undefined + if (previous && previous !== pluginId) + issues.push({ + path: ['plugins', i, 'options'], + message: `This would rename the plugin from ${previous} to ${pluginId}. Add a new instance instead.`, + }) + if (!instance.enabled) continue + if (seen.has(pluginId)) + issues.push({ path: ['plugins', i], message: `Two plugins would use the ID ${pluginId}` }) + seen.add(pluginId) + } + return issues + } + + /** The ID the instance's plugin has, when its package is installed and its options load. */ + private pluginIdOf(instance: PluginInstance): string | undefined { + try { + return instantiate(this.deps.installed, instance).id + } catch (err) { + if (err instanceof PluginInstanceError) return undefined + throw err + } + } + + /** What the document asks of the plugins that they can live with, but a person should know. */ + private pluginWarnings( + doc: ConfigDocument, + stored: PluginInstance[], + proposed: PluginInstance[], + ): string[] { + const byId = new Map(stored.map((instance) => [instance.id, instance])) + return doc.plugins.flatMap((entry, i) => { + const warnings: string[] = [] + const before = byId.get(entry.id) + if (before && before.package !== entry.package && Object.keys(before.secrets).length) + warnings.push(`plugin ${entry.id} changed package, so its stored secrets were dropped`) + const kept = Object.keys(proposed[i]?.secrets ?? {}).sort() + if (entry.secretsSet && !isDeepStrictEqual([...entry.secretsSet].sort(), kept)) + warnings.push( + `plugin ${entry.id}'s secretsSet is only for reading, so it was ignored. To clear a secret, set it to null under secrets`, + ) + const installed = this.deps.installed.get(entry.package) + if (!installed) { + if (Object.keys(entry.options).length) + warnings.push( + `plugin ${entry.id}'s package ${entry.package} isn't installed, so its secrets can't be told apart from its options. Give any secrets under secrets, or they are stored unencrypted`, + ) + return warnings + } + const fields = this.optionFields(entry.package) + const secrets = secretKeys(installed.optionsSchema) + for (const key of Object.keys(entry.options).filter((key) => !fields.has(key))) + warnings.push( + `plugin ${entry.id} has no option ${key}; it's stored, and the plugin ignores it`, + ) + for (const key of Object.keys(entry.secrets).filter((key) => !secrets.includes(key))) + warnings.push( + `plugin ${entry.id} has no secret ${key}; it's stored, and the plugin ignores it`, + ) + return warnings + }) + } + + private readonly fields = new Map>() + + /** The option fields a package's schema has. */ + private optionFields(pkg: string) { + let fields = this.fields.get(pkg) + if (!fields) { + const plugin = this.deps.installed.get(pkg) + fields = new Set(Object.keys((plugin && optionsJsonSchema(plugin).properties) ?? {})) + this.fields.set(pkg, fields) + } + return fields + } + + /** + * The document's instances, keeping stored secrets it leaves out and clearing the ones it nulls. + * A secret field under options moves to secrets, so it is stored encrypted. + */ + private proposeInstances(doc: ConfigDocument, stored: PluginInstance[]): PluginInstance[] { + const byId = new Map(stored.map((instance) => [instance.id, instance])) + return doc.plugins.map((entry) => { + const before = byId.get(entry.id) + const same = before?.package === entry.package ? before : undefined + const secretFields = secretKeys(this.deps.installed.get(entry.package)?.optionsSchema) + const options = Object.fromEntries( + Object.entries(entry.options).filter(([key]) => !secretFields.includes(key)), + ) + const given = { + ...Object.fromEntries( + Object.entries(entry.options).filter(([key]) => secretFields.includes(key)), + ), + ...entry.secrets, + } + const secrets = { ...same?.secrets } + for (const [key, value] of Object.entries(given)) { + if (value === null) delete secrets[key] + else secrets[key] = value + } + const cleared = Object.values(given).some((value) => value === null) + return { + id: entry.id, + package: entry.package, + enabled: entry.enabled, + options, + secrets, + ...(same?.unreadableSecrets && !cleared + ? { unreadableSecrets: same.unreadableSecrets } + : {}), + } + }) + } + + private roleChanges(doc: ConfigDocument, roles: RoleSummary[], removed: RoleSummary[]) { + const byName = new Map(roles.map((role) => [role.name, role])) + const changes: string[] = [] + for (const entry of doc.roles) { + const before = byName.get(entry.name) + if (!before) { + changes.push(`role ${entry.name} added`) + continue + } + if (definitionChanged(entry, before)) changes.push(`role ${entry.name} updated`) + if (entry.members && !sameMembers(entry.members, before)) + changes.push(`role ${entry.name} members updated`) + } + for (const role of removed) changes.push(`role ${role.name} removed`) + return changes + } +} + +/** Plugins that aren't loaded and models users can't see, as the server is after an apply. */ +function health(doc: ConfigDocument, runtime: PluginRuntime): string[] { + const warnings: string[] = [] + for (const entry of doc.plugins) { + const status = runtime.statuses.get(entry.id) + if (status?.state === 'failed') + warnings.push(`plugin ${entry.id} isn't loaded: ${status.error}`) + } + for (const model of doc.models) { + if (!servesAdminModels(runtime.host.providers, model.provider)) + warnings.push( + `model ${model.provider}/${model.id} is hidden from users, since no loaded plugin serves its provider with an admin key`, + ) + } + return warnings +} + +const sameMembers = (members: string[], role: RoleSummary) => + isDeepStrictEqual(new Set(members), new Set(role.members.map((member) => member.did))) + +/** Does the entry change any part of the role's definition it gives? */ +const definitionChanged = (entry: RoleEntry, role: RoleSummary) => + (entry.description !== undefined && entry.description !== role.description) || + (entry.pdsHosts !== undefined && !isDeepStrictEqual(entry.pdsHosts, role.pdsHosts)) || + (entry.handleDomains !== undefined && !isDeepStrictEqual(entry.handleDomains, role.handleDomains)) + +/** Bring the stored roles and members in line with the document. */ +async function writeRoles( + db: Db, + doc: ConfigDocument, + roles: RoleSummary[], + removed: RoleSummary[], + by: string, +) { + for (const role of removed) await removeRole(db, role.name) + const byName = new Map(roles.map((role) => [role.name, role])) + for (const entry of doc.roles) { + const before = byName.get(entry.name) + if (!before) await insertRole(db, entry) + else if (definitionChanged(entry, before)) await updateRole(db, entry.name, entry) + if (!entry.members || (before && sameMembers(entry.members, before))) continue + const listed = new Set(entry.members) + const current = new Set(before?.members.map((member) => member.did)) + for (const did of current) if (!listed.has(did)) await removeMember(db, entry.name, did) + for (const did of listed) if (!current.has(did)) await addMember(db, entry.name, did, by) + } +} diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts index 77a5673..ba5868c 100644 --- a/apps/server/src/app.ts +++ b/apps/server/src/app.ts @@ -2,7 +2,9 @@ import { randomBytes } from 'node:crypto' import { serveStatic } from '@hono/node-server/serve-static' import { Hono } from 'hono' import { sql } from 'kysely' +import { siteConfigRoutes } from './admin/config-routes.ts' import { type AdminRoutesDeps, adminRoutes } from './admin/routes.ts' +import type { SiteConfig } from './admin/site-config.ts' import { requireApiKey } from './auth/api-keys.ts' import { SessionExpired } from './auth/pds.ts' import { ADMIN_ROLE } from './auth/roles.ts' @@ -76,6 +78,7 @@ export type AppDeps = { sync?: SyncRoutesDeps admin?: AdminRoutesDeps cron?: Cron + siteConfig?: SiteConfig } export function createApp(deps: AppDeps) { @@ -125,6 +128,8 @@ export function createApp(deps: AppDeps) { throw err } }) + if (deps.siteConfig) + api.route('/admin/config', siteConfigRoutes({ db, logger, siteConfig: deps.siteConfig })) if (deps.auth) api.route('/', authApiRoutes(deps.auth)) if (deps.plugins) api.route('/plugins', pluginRoutes(deps.plugins)) diff --git a/apps/server/src/auth/api-keys.ts b/apps/server/src/auth/api-keys.ts index 3645074..c072edc 100644 --- a/apps/server/src/auth/api-keys.ts +++ b/apps/server/src/auth/api-keys.ts @@ -50,6 +50,11 @@ export async function listApiKeys(db: Db): Promise { })) } +/** Return the list of API keys (label only) that are assigned to a given role. */ +export async function apiKeysHolding(db: Db, role: string): Promise { + return (await listApiKeys(db)).filter((key) => key.roles.includes(role)).map((key) => key.label) +} + /** Returns false when there is no such key. */ export async function revokeApiKey(db: Db, id: string): Promise { const result = await db.deleteFrom('api_key').where('id', '=', id).executeTakeFirst() @@ -87,6 +92,7 @@ export function requireApiKey( .where('id', '=', row.id) .execute() deps.logger.info({ path: c.req.path, key: row.label }, 'API key accepted') + c.set('apiKey', { id: row.id, label: row.label }) await next() } } diff --git a/apps/server/src/auth/role-store.ts b/apps/server/src/auth/role-store.ts index 2f47a77..55ba02d 100644 --- a/apps/server/src/auth/role-store.ts +++ b/apps/server/src/auth/role-store.ts @@ -1,7 +1,19 @@ import { isValidDid } from '@atproto/syntax' +import { z } from 'zod' import type { Db } from '../db/index.ts' import { ADMIN_ROLE } from './roles.ts' +const ROLE_NAME = /^[a-z][a-z0-9-]*$/ +const HOSTNAME = + /^(?=.{1,253}$)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+$/ + +export const roleName = z.string().regex(ROLE_NAME, `must match ${ROLE_NAME}`) + +/** PDS hosts or handle domains a role matches. */ +export const hostnames = z.array( + z.string().trim().toLowerCase().regex(HOSTNAME, 'must be a hostname, such as example.com'), +) + export type RoleMember = { did: string; addedAt: string; addedBy: string } export type RoleSummary = { @@ -45,17 +57,26 @@ export async function roleExists(db: Db, name: string): Promise { export async function createRole(db: Db, role: { name: string; description: string }) { if (await roleExists(db, role.name)) throw new RoleExists(role.name) + await insertRole(db, role) +} + +/** Store a role, leaving one that already exists alone. */ +export async function insertRole( + db: Db, + role: { name: string; description?: string; pdsHosts?: string[]; handleDomains?: string[] }, +) { const now = new Date().toISOString() await db .insertInto('role') .values({ name: role.name, - description: role.description, - pds_hosts_json: '[]', - handle_domains_json: '[]', + description: role.description ?? '', + pds_hosts_json: JSON.stringify(role.pdsHosts ?? []), + handle_domains_json: JSON.stringify(role.handleDomains ?? []), created_at: now, updated_at: now, }) + .onConflict((oc) => oc.column('name').doNothing()) .execute() } @@ -79,10 +100,13 @@ export async function updateRole( } export async function deleteRole(db: Db, name: string): Promise { - await db.transaction().execute(async (tx) => { - await tx.deleteFrom('role_member').where('role', '=', name).execute() - await tx.deleteFrom('role').where('name', '=', name).execute() - }) + await db.transaction().execute((tx) => removeRole(tx, name)) +} + +/** `deleteRole` within a transaction the caller holds. */ +export async function removeRole(db: Db, name: string): Promise { + await db.deleteFrom('role_member').where('role', '=', name).execute() + await db.deleteFrom('role').where('name', '=', name).execute() } export async function addMember(db: Db, role: string, did: string, addedBy: string) { diff --git a/apps/server/src/auth/routes.ts b/apps/server/src/auth/routes.ts index bd5e387..ee79e24 100644 --- a/apps/server/src/auth/routes.ts +++ b/apps/server/src/auth/routes.ts @@ -72,7 +72,7 @@ const isLocalPath = (path: string) => /^\/(?![/\\])/.test(path) const isSharePath = (path: string) => /^\/shared\/[^/]+\/[^/]+$/.test(path) /** API paths opened to API keys, which never use the session cookie. */ -const API_KEY_PATHS = [/^\/api\/cron$/] +const API_KEY_PATHS = [/^\/api\/cron$/, /^\/api\/admin\/config$/] /** API paths a viewer may use. */ const VIEWER_PATHS = [/^\/api\/me$/, /^\/api\/logout$/, /^\/api\/shared\//, /^\/api\/health$/] diff --git a/apps/server/src/env.ts b/apps/server/src/env.ts index cd4e498..d2b15dd 100644 --- a/apps/server/src/env.ts +++ b/apps/server/src/env.ts @@ -9,5 +9,8 @@ export type AuthUser = { admin: boolean } +/** The API key a request was made with, on routes opened to keys. */ +export type ApiKeyCaller = { id: string; label: string } + /** Hono context variables shared by every route. */ -export type AppEnv = { Variables: { user: AuthUser | null } } +export type AppEnv = { Variables: { user: AuthUser | null; apiKey?: ApiKeyCaller } } diff --git a/apps/server/src/plugins/admin.ts b/apps/server/src/plugins/admin.ts index 1aa091a..064de9e 100644 --- a/apps/server/src/plugins/admin.ts +++ b/apps/server/src/plugins/admin.ts @@ -1,4 +1,3 @@ -import { randomUUID } from 'node:crypto' import type { ModelInfo } from '@scn-chat/plugin-api' import { InvalidBody } from '../body.ts' import type { Db } from '../db/index.ts' @@ -64,6 +63,44 @@ function splitOptions( return { options, secrets } } +const RANDOM_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i + +/** Rename instances still under the random IDs they were once added with to their plugin's ID. */ +export async function nameInstancesAfterPlugins( + db: Db, + box: SecretBox, + installed: InstalledPlugins, + logger: Logger, +): Promise { + const instances = await readInstances(db, box) + const taken = new Set(instances.map((instance) => instance.id)) + for (const instance of instances) { + if (!RANDOM_ID.test(instance.id)) continue + let id: string + try { + id = instantiate(installed, instance).id + } catch (err) { + if (!(err instanceof PluginInstanceError)) throw err + logger.info( + { instance: instance.id, err }, + "kept a plugin instance's random ID, since it doesn't load", + ) + continue + } + if (taken.has(id)) { + logger.info( + { instance: instance.id, pluginId: id }, + "kept a plugin instance's random ID, since its plugin ID is taken", + ) + continue + } + await db.updateTable('plugin_instance').set({ id }).where('id', '=', instance.id).execute() + taken.delete(instance.id) + taken.add(id) + logger.info({ from: instance.id, to: id }, 'plugin instance renamed after its plugin') + } +} + /** Adding, changing, and removing plugin instances, each applied without a restart. */ export class PluginAdmin { private readonly deps: PluginAdminDeps @@ -124,64 +161,70 @@ export class PluginAdmin { }) } + /** Add an instance, named after the plugin it makes. */ async add(pkg: string, input: Record, adminDid: string): Promise { const installed = this.deps.installed.get(pkg) if (!installed) throw new InvalidBody(`The plugin package "${pkg}" isn't installed`) - const id = randomUUID() - const instances = await readInstances(this.deps.db, this.deps.box) - if (!installed.multiple && instances.some((instance) => instance.package === pkg)) - throw new InvalidBody(`${pkg} is already added, and can only be added once`) - instances.push({ id, package: pkg, enabled: true, ...splitOptions(installed, input) }) - await this.apply(instances, id, adminDid) + const draft = { id: 'new', package: pkg, enabled: true, ...splitOptions(installed, input) } + let id: string + try { + id = instantiate(this.deps.installed, draft).id + } catch (err) { + if (err instanceof PluginInstanceError) throw new InvalidBody(err.message, err.issues) + throw err + } + await this.change((instances) => { + if (!installed.multiple && instances.some((instance) => instance.package === pkg)) + throw new InvalidBody(`${pkg} is already added, and can only be added once`) + if (instances.some((instance) => instance.id === id)) + throw new InvalidBody(`An instance of the plugin ${id} is already added`) + return { proposed: [...instances, { ...draft, id }], changedId: id } + }, adminDid) return id } async update(id: string, change: InstanceChange, adminDid: string): Promise { - const instances = await readInstances(this.deps.db, this.deps.box) - const index = instances.findIndex((instance) => instance.id === id) - const instance = instances[index] - if (!instance) throw new InstanceNotFound(id) - const installed = this.deps.installed.get(instance.package) - const split = splitOptions( - installed, - change.options ?? { ...instance.options }, - instance.secrets, - change.clearSecrets, - ) - instances[index] = { - ...instance, - ...split, - enabled: change.enabled ?? instance.enabled, - // Clearing any secret discards the ones that can't be decrypted. - unreadableSecrets: change.clearSecrets?.length ? undefined : instance.unreadableSecrets, - } - await this.apply(instances, id, adminDid) + await this.change((instances) => { + const index = instances.findIndex((instance) => instance.id === id) + const instance = instances[index] + if (!instance) throw new InstanceNotFound(id) + const installed = this.deps.installed.get(instance.package) + const split = splitOptions( + installed, + change.options ?? { ...instance.options }, + instance.secrets, + change.clearSecrets, + ) + const proposed = [...instances] + proposed[index] = { + ...instance, + ...split, + enabled: change.enabled ?? instance.enabled, + // Clearing any secret discards the ones that can't be decrypted. + unreadableSecrets: change.clearSecrets?.length ? undefined : instance.unreadableSecrets, + } + return { proposed, changedId: id } + }, adminDid) } async remove(id: string, adminDid: string): Promise { - const instances = await readInstances(this.deps.db, this.deps.box) - if (!instances.some((instance) => instance.id === id)) throw new InstanceNotFound(id) - await this.apply( - instances.filter((instance) => instance.id !== id), - undefined, - adminDid, - ) + await this.change((instances) => { + if (!instances.some((instance) => instance.id === id)) throw new InstanceNotFound(id) + return { proposed: instances.filter((instance) => instance.id !== id) } + }, adminDid) } async reorder(ids: string[], adminDid: string): Promise { - const instances = await readInstances(this.deps.db, this.deps.box) - const byId = new Map(instances.map((instance) => [instance.id, instance])) - if ( - ids.length !== instances.length || - new Set(ids).size !== ids.length || - ids.some((id) => !byId.has(id)) - ) - throw new InvalidBody('The new order must list every plugin instance once') - await this.apply( - ids.map((id) => byId.get(id) as PluginInstance), - undefined, - adminDid, - ) + await this.change((instances) => { + const byId = new Map(instances.map((instance) => [instance.id, instance])) + if ( + ids.length !== instances.length || + new Set(ids).size !== ids.length || + ids.some((id) => !byId.has(id)) + ) + throw new InvalidBody('The new order must list every plugin instance once') + return { proposed: ids.map((id) => byId.get(id) as PluginInstance) } + }, adminDid) } /** @@ -225,10 +268,17 @@ export class PluginAdmin { } } - /** Build a runtime from the proposed instances, and switch to it unless it breaks something. */ - private apply(proposed: PluginInstance[], changedId: string | undefined, adminDid: string) { + /** + * Read the instances under the lock, so changes never overwrite each other, then build a runtime + * from the proposed ones and switch to it unless it breaks something. + */ + private change( + propose: (instances: PluginInstance[]) => { proposed: PluginInstance[]; changedId?: string }, + adminDid: string, + ) { const { holder } = this.deps return holder.exclusive(async () => { + const { proposed, changedId } = propose(await readInstances(this.deps.db, this.deps.box)) const current = holder.current() const candidate = await this.deps.build(proposed) try { diff --git a/apps/server/src/plugins/instances.ts b/apps/server/src/plugins/instances.ts index c4bf799..5f7a5f8 100644 --- a/apps/server/src/plugins/instances.ts +++ b/apps/server/src/plugins/instances.ts @@ -39,31 +39,39 @@ export async function writeInstances( box: SecretBox, instances: PluginInstance[], adminDid: string, +): Promise { + await db.transaction().execute((tx) => replaceInstances(tx, box, instances, adminDid)) +} + +/** `writeInstances` within a transaction the caller holds. */ +export async function replaceInstances( + db: Db, + box: SecretBox, + instances: PluginInstance[], + adminDid: string, ): Promise { const now = new Date().toISOString() - await db.transaction().execute(async (tx) => { - const ids = instances.map((instance) => instance.id) - let removal = tx.deleteFrom('plugin_instance') - if (ids.length) removal = removal.where('id', 'not in', ids) - await removal.execute() - for (const [position, instance] of instances.entries()) { - const entered = Object.keys(instance.secrets).length > 0 - const row = { - package: instance.package, - position, - enabled: instance.enabled ? 1 : 0, - options_json: JSON.stringify(instance.options), - secrets_encrypted: entered - ? box.encrypt(JSON.stringify(instance.secrets)) - : (instance.unreadableSecrets ?? null), - updated_at: now, - updated_by: adminDid, - } - await tx - .insertInto('plugin_instance') - .values({ id: instance.id, created_at: now, ...row }) - .onConflict((oc) => oc.column('id').doUpdateSet(row)) - .execute() + const ids = instances.map((instance) => instance.id) + let removal = db.deleteFrom('plugin_instance') + if (ids.length) removal = removal.where('id', 'not in', ids) + await removal.execute() + for (const [position, instance] of instances.entries()) { + const entered = Object.keys(instance.secrets).length > 0 + const row = { + package: instance.package, + position, + enabled: instance.enabled ? 1 : 0, + options_json: JSON.stringify(instance.options), + secrets_encrypted: entered + ? box.encrypt(JSON.stringify(instance.secrets)) + : (instance.unreadableSecrets ?? null), + updated_at: now, + updated_by: adminDid, } - }) + await db + .insertInto('plugin_instance') + .values({ id: instance.id, created_at: now, ...row }) + .onConflict((oc) => oc.column('id').doUpdateSet(row)) + .execute() + } } diff --git a/apps/server/src/providers/admin-models.ts b/apps/server/src/providers/admin-models.ts index 82e7210..59d44c6 100644 --- a/apps/server/src/providers/admin-models.ts +++ b/apps/server/src/providers/admin-models.ts @@ -104,6 +104,32 @@ export async function deleteAdminModel(db: Db, provider: string, id: string): Pr return result.numDeletedRows > 0n } +/** Replace every model with this list, in this order, within a transaction the caller holds. */ +export async function replaceAdminModels(db: Db, models: AdminModel[], adminDid: string) { + const now = new Date().toISOString() + const keep = new Set(models.map((model) => `${model.provider}/${model.id}`)) + for (const row of await db.selectFrom('admin_model').select(['provider', 'model_id']).execute()) { + if (keep.has(`${row.provider}/${row.model_id}`)) continue + await deleteAdminModel(db, row.provider, row.model_id) + } + for (const [position, model] of models.entries()) { + const row = { + name: model.name, + position, + capabilities_json: JSON.stringify(model.capabilities), + roles_json: JSON.stringify(model.roles), + is_default: model.default ? 1 : 0, + updated_at: now, + updated_by: adminDid, + } + await db + .insertInto('admin_model') + .values({ provider: model.provider, model_id: model.id, created_at: now, ...row }) + .onConflict((oc) => oc.columns(['provider', 'model_id']).doUpdateSet(row)) + .execute() + } +} + /** Put the models in this order. */ export async function reorderAdminModels(db: Db, refs: { provider: string; id: string }[]) { await db.transaction().execute(async (tx) => { diff --git a/apps/server/src/server.ts b/apps/server/src/server.ts index daffd67..a7553dd 100644 --- a/apps/server/src/server.ts +++ b/apps/server/src/server.ts @@ -1,3 +1,4 @@ +import { SiteConfig } from './admin/site-config.ts' import { createApp, type Web } from './app.ts' import { Access } from './auth/access.ts' import { type Account, getAccount } from './auth/accounts.ts' @@ -15,7 +16,7 @@ import { Cron } from './cron.ts' import type { Db } from './db/index.ts' import { migrateToLatest } from './db/migrate.ts' import type { Logger } from './logger.ts' -import { PluginAdmin } from './plugins/admin.ts' +import { nameInstancesAfterPlugins, PluginAdmin } from './plugins/admin.ts' import { fireChatHooks } from './plugins/chat-hooks.ts' import type { PluginServices } from './plugins/host.ts' import type { InstalledPlugins } from './plugins/installed.ts' @@ -131,6 +132,7 @@ export async function createServer(deps: ServerDeps) { }), logger, }) + await nameInstancesAfterPlugins(db, box, deps.installed, logger) const initial = await build(await readInstances(db, box)) for (const [instance, status] of initial.statuses) { if (status.state === 'failed') @@ -267,6 +269,15 @@ export async function createServer(deps: ServerDeps) { }, admin: { db, roles, access, cron, settings, identity, plugins, runtime, logger }, cron, + siteConfig: new SiteConfig({ + db, + box, + settings, + installed: deps.installed, + holder, + build, + logger, + }), }) const stops: (() => void)[] = [] diff --git a/apps/server/src/settings/store.ts b/apps/server/src/settings/store.ts index e60c2ea..54ae466 100644 --- a/apps/server/src/settings/store.ts +++ b/apps/server/src/settings/store.ts @@ -50,20 +50,30 @@ export class SettingsStore { /** Validate, store, and announce a setting. Throws InvalidBody when it fails its schema. */ async set(key: K, input: unknown, adminDid: string): Promise { const value = validate(settingSchemas[key], input) as Settings[K] + await SettingsStore.write(this.db, key, value, adminDid) + this.adopt(key, value) + this.logger.info({ key, admin: adminDid }, 'admin setting changed') + return value + } + + /** Store a validated setting without announcing it. Call `adopt` once the write commits. */ + static async write(db: Db, key: K, value: Settings[K], adminDid: string) { const row = { value_json: JSON.stringify(value), updated_at: new Date().toISOString(), updated_by: adminDid, } - await this.db + await db .insertInto('app_setting') .values({ key, ...row }) .onConflict((oc) => oc.column('key').doUpdateSet(row)) .execute() + } + + /** Hold and announce a setting that is already stored. */ + adopt(key: K, value: Settings[K]): void { this.values[key] = value - this.logger.info({ key, admin: adminDid }, 'admin setting changed') for (const listener of this.listeners.get(key) ?? []) (listener as Listener)(value) - return value } /** Call the listener after each change to the setting. */ diff --git a/apps/server/src/site-config-cli.ts b/apps/server/src/site-config-cli.ts new file mode 100644 index 0000000..4d284eb --- /dev/null +++ b/apps/server/src/site-config-cli.ts @@ -0,0 +1,117 @@ +import { existsSync, readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { text } from 'node:stream/consumers' +import { fileURLToPath } from 'node:url' +import { parseArgs } from 'node:util' +import { issueApiKey, revokeApiKey } from './auth/api-keys.ts' +import { ADMIN_ROLE } from './auth/roles.ts' +import { DEFAULT_DATABASE_URL } from './config.ts' +import { createDb } from './db/index.ts' + +const CLI_LABEL = 'site-config CLI' +// Age of a CLI key before a later run clears it out. +const STALE_MS = 60 * 60_000 +const USAGE = + 'Usage: site-config export [--url ]\n site-config apply [file] [--url ]' + +// Resolve the file from where the command ran, and the server's own paths from its package. +const callerDir = process.env.INIT_CWD ?? process.cwd() +process.chdir(fileURLToPath(new URL('..', import.meta.url))) +if (existsSync('.env')) process.loadEnvFile('.env') +else if (existsSync('../../.env')) process.loadEnvFile('../../.env') + +const { values, positionals } = parseArgs({ + allowPositionals: true, + options: { url: { type: 'string' } }, +}) +const [command, file, ...extra] = positionals +if ( + !(command === 'export' || command === 'apply') || + extra.length || + (command === 'export' && file) +) { + console.error(USAGE) + process.exit(2) +} + +/** Stop with a problem in how the CLI was run. */ +function misuse(message: string): never { + console.error(message) + process.exit(2) +} + +if (command === 'apply' && !file && process.stdin.isTTY) + misuse(`Give the file to apply, or pipe it in.\n${USAGE}`) +const server = values.url ?? `http://127.0.0.1:${process.env.PORT || 3000}` +if (!URL.canParse(server)) misuse(`--url ${server} isn't a URL, such as http://127.0.0.1:3000.`) +const databaseUrl = process.env.DATABASE_URL || DEFAULT_DATABASE_URL +const sqlitePath = databaseUrl.startsWith('sqlite:') ? databaseUrl.slice('sqlite:'.length) : null +if (sqlitePath && !existsSync(sqlitePath)) + misuse( + `There's no database at ${resolve(sqlitePath)}. Set DATABASE_URL to the server's database.`, + ) + +const body = + command === 'apply' + ? file + ? readFileSync(resolve(callerDir, file), 'utf8') + : await text(process.stdin) + : undefined + +/** Call the running server with the key, print what it says, and return the exit code. */ +async function run(key: string): Promise { + let response: Response + try { + response = await fetch(new URL('/api/admin/config', server), { + method: command === 'export' ? 'GET' : 'PUT', + headers: { authorization: `Bearer ${key}`, 'content-type': 'application/json' }, + body, + }) + } catch (err) { + if (!(err instanceof TypeError)) throw err + console.error( + `Couldn't reach the server at ${server}. Is it running? Use --url if it listens elsewhere.`, + ) + return 1 + } + type Answer = { message?: string; changed?: boolean; changes?: string[]; warnings?: string[] } + let result: Answer + try { + result = (await response.json()) as Answer + } catch (err) { + if (!(err instanceof SyntaxError)) throw err + console.error( + `${server} answered with something other than SCN Chat's API (${response.status}).`, + ) + return 1 + } + if (response.status === 401) { + console.error( + `${server} didn't accept the CLI's key, so it isn't using this database. Point --url at the server that uses the database in DATABASE_URL.`, + ) + return 1 + } + if (!response.ok) { + console.error(result.message ?? `The server refused the request (${response.status}).`) + return 1 + } + if (command === 'export') console.log(JSON.stringify(result, null, 2)) + else console.log(result.changed ? (result.changes ?? []).join('\n') : 'No changes.') + for (const warning of result.warnings ?? []) console.error(`Warning: ${warning}`) + return 0 +} + +const db = createDb(databaseUrl) +// Clear out keys left by runs that were killed. +await db + .deleteFrom('api_key') + .where('created_by', '=', CLI_LABEL) + .where('created_at', '<', new Date(Date.now() - STALE_MS).toISOString()) + .execute() +const { id, key } = await issueApiKey(db, { label: CLI_LABEL, roles: [ADMIN_ROLE] }, CLI_LABEL) +try { + process.exitCode = await run(key) +} finally { + await revokeApiKey(db, id) + await db.destroy() +} diff --git a/apps/server/tests/admin/plugin-routes.test.ts b/apps/server/tests/admin/plugin-routes.test.ts index 449b2ad..d3614b1 100644 --- a/apps/server/tests/admin/plugin-routes.test.ts +++ b/apps/server/tests/admin/plugin-routes.test.ts @@ -1,7 +1,12 @@ import { definePlugin, type ModelProvider } from '@scn-chat/plugin-api' import { describe, expect, it } from 'vitest' import { z } from 'zod' +import { migrateToLatest } from '../../src/db/migrate.ts' +import { writeInstances } from '../../src/plugins/instances.ts' +import { SecretBox } from '../../src/secrets.ts' import { adminHarness } from '../helpers/admin.ts' +import { TEST_ADMIN, TEST_SECRET_KEY } from '../helpers/config.ts' +import { createSqliteDb } from '../helpers/db.ts' import { installedFrom } from '../helpers/plugins.ts' const caps = { vision: true, reasoning: false, tools: true } @@ -107,6 +112,53 @@ describe('plugin instances', () => { expect(h.holder.current().host.providers.get('fake')?.hasAdminKey).toBe(true) }) + it('names each instance after the plugin it makes', async () => { + const h = await setup() + expect((await h.add({ id: 'one' })).body).toEqual({ id: 'prov-one' }) + expect((await h.add({ id: 'two' })).body).toEqual({ id: 'prov-two' }) + }) + + it('renames instances added under random IDs after their plugin at startup, keeping their secrets', async () => { + const random = '0b5e2a1c-7d44-4c1e-9a3f-2f6d8e9b1c00' + const broken = '9f1d3c2b-1a2b-4c3d-8e4f-5a6b7c8d9e0f' + const db = createSqliteDb() + await migrateToLatest(db) + await writeInstances( + db, + new SecretBox(Buffer.from(TEST_SECRET_KEY, 'base64')), + [ + { + id: random, + package: 'provider-plugin', + enabled: true, + options: { id: 'fake' }, + secrets: { apiKey: 'sk-admin' }, + }, + { + id: 'litellm', + package: 'provider-plugin', + enabled: true, + options: { id: 'other' }, + secrets: {}, + }, + { + id: broken, + package: 'provider-plugin', + enabled: true, + options: { id: 'Not Valid' }, + secrets: {}, + }, + ], + TEST_ADMIN, + ) + const h = await adminHarness({ installed: fixtures().installed, db }) + const { instances } = (await h.call('GET', '/admin/plugins')).body as { + instances: { id: string; secretsSet: string[] }[] + } + expect(instances.map((instance) => instance.id)).toEqual(['prov-fake', 'litellm', broken]) + expect(instances[0]?.secretsSet).toEqual(['apiKey']) + }) + it('refuses options that fail the schema, pointing at the field', async () => { const h = await setup() const res = await h.add({ id: 'Not Valid' }) diff --git a/apps/server/tests/admin/site-config.test.ts b/apps/server/tests/admin/site-config.test.ts new file mode 100644 index 0000000..87e579f --- /dev/null +++ b/apps/server/tests/admin/site-config.test.ts @@ -0,0 +1,689 @@ +import { execFile } from 'node:child_process' +import { existsSync, mkdtempSync, writeFileSync } from 'node:fs' +import type { AddressInfo } from 'node:net' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { serve } from '@hono/node-server' +import { definePlugin, type ModelProvider } from '@scn-chat/plugin-api' +import { afterEach, describe, expect, it } from 'vitest' +import { z } from 'zod' +import { recordLogin } from '../../src/auth/accounts.ts' +import { issueApiKey, listApiKeys } from '../../src/auth/api-keys.ts' +import { syncMembers } from '../../src/auth/role-store.ts' +import { createDb, type Db } from '../../src/db/index.ts' +import { migrateToLatest } from '../../src/db/migrate.ts' +import { writeInstances } from '../../src/plugins/instances.ts' +import { SecretBox } from '../../src/secrets.ts' +import { adminHarness } from '../helpers/admin.ts' +import { TEST_ADMIN, TEST_SECRET_KEY } from '../helpers/config.ts' +import { createSqliteDb } from '../helpers/db.ts' +import { installedFrom } from '../helpers/plugins.ts' + +const caps = { vision: true, reasoning: false, tools: true } + +/** Holds the setup of a plugin with `hold`, so a test can act while an apply is building. */ +let gate = Promise.withResolvers() + +/** A provider plugin whose provider ID and admin key come from its options, and a plain one. */ +const installed = installedFrom({ + 'provider-plugin': { + multiple: true, + optionsSchema: z + .object({ + id: z + .string() + .regex(/^[a-z]+$/) + .default('fake'), + apiKey: z.string().min(1).optional().meta({ secret: true }), + crash: z.boolean().default(false), + hold: z.boolean().default(false), + }) + .strict(), + factory: (options: { id: string; apiKey?: string; crash: boolean; hold: boolean }) => + definePlugin({ + id: `prov-${options.id}`, + name: `Provider ${options.id}`, + apiVersion: 1, + async setup(ctx) { + if (options.crash) throw new Error('setup exploded') + if (options.hold) await gate.promise + ctx.providers.register({ + id: options.id, + name: options.id, + hasAdminKey: Boolean(options.apiKey), + userKeys: true, + replay: 'drop', + createModel: () => { + throw new Error('not in these tests') + }, + } as ModelProvider) + }, + }), + }, + 'plain-plugin': { + factory: () => definePlugin({ id: 'plain', name: 'Plain', apiVersion: 1, setup: () => {} }), + }, +}) + +type Doc = { + version: number + settings: Record> + roles: { name: string; members?: string[]; [key: string]: unknown }[] + plugins: { id: string; package: string; secretsSet?: string[]; [key: string]: unknown }[] + models: { provider: string; id: string; [key: string]: unknown }[] +} + +const model = (overrides: Record = {}) => ({ + provider: 'fake', + id: 'big', + name: 'Big', + capabilities: caps, + roles: ['user'], + default: false, + ...overrides, +}) + +const providerPlugin = (overrides: Record = {}) => ({ + id: 'fake-provider', + package: 'provider-plugin', + enabled: true, + options: {}, + secrets: { apiKey: 'sk-admin' }, + ...overrides, +}) + +const providerInstance = ( + id: string, + options: Record, + secrets: Record, +) => ({ id, package: 'provider-plugin', enabled: true, options, secrets }) + +async function setup(options: { db?: Db } = {}) { + const h = await adminHarness({ installed, ...options }) + const { key } = await issueApiKey(h.db, { label: 'ansible', roles: ['admin'] }, TEST_ADMIN) + const config = async (method: 'GET' | 'PUT', body?: unknown) => { + const res = await h.app.request('/api/admin/config', { + method, + headers: { authorization: `Bearer ${key}`, 'content-type': 'application/json' }, + body: body === undefined ? undefined : JSON.stringify(body), + }) + return { status: res.status, body: (await res.json()) as Record } + } + const exported = async () => (await config('GET')).body as unknown as Doc + /** Apply the server's own export with the changes made to it. */ + const apply = async (change: (doc: Doc) => Doc) => config('PUT', change(await exported())) + return { ...h, config, exported, apply } +} + +/** A server with a provider plugin, its model, and a staff role. */ +async function configured() { + const h = await setup() + const res = await h.apply((doc) => ({ + ...doc, + roles: [...doc.roles, { name: 'staff', description: 'Staff' }], + plugins: [providerPlugin()], + models: [model({ roles: ['staff'] })], + })) + expect(res.status).toBe(200) + return h +} + +describe('site config export', () => { + it('lists every section, naming the secrets that are set without their values', async () => { + const h = await configured() + const doc = await h.exported() + expect(Object.keys(doc).sort()).toEqual(['models', 'plugins', 'roles', 'settings', 'version']) + expect(Object.keys(doc.settings).sort()).toEqual([ + 'access', + 'general', + 'sessions', + 'sync', + 'turns', + ]) + expect(doc.plugins[0]?.secretsSet).toEqual(['apiKey']) + expect(JSON.stringify(doc)).not.toContain('sk-admin') + }) + + it('leaves out the members of a role a plugin keeps in sync', async () => { + const h = await configured() + await recordLogin(h.db, { + did: 'did:plc:member', + handle: 'member.test', + pdsUrl: 'https://pds.test', + spacesAllowed: false, + }) + await syncMembers(h.db, 'staff', ['did:plc:member'], 'plugin:registry') + const staff = (await h.exported()).roles.find((role) => role.name === 'staff') + expect(staff).not.toHaveProperty('members') + }) +}) + +describe('site config apply', () => { + it('changes nothing when given its own export', async () => { + const h = await configured() + expect((await h.config('PUT', await h.exported())).body).toEqual({ + changed: false, + changes: [], + warnings: [], + }) + }) + + it('adds plugins, models, roles, and settings that work without a restart', async () => { + const h = await setup() + const res = await h.apply((doc) => ({ + ...doc, + settings: { ...doc.settings, general: { ...doc.settings.general, appName: 'Ansible Chat' } }, + roles: [...doc.roles, { name: 'staff' }], + plugins: [providerPlugin()], + models: [model({ roles: ['staff'] })], + })) + expect(res.body).toEqual({ + changed: true, + changes: [ + 'setting general updated', + 'role staff added', + 'plugin fake-provider added', + 'model fake/big added', + ], + warnings: [], + }) + expect(h.holder.current().host.providers.get('fake')?.hasAdminKey).toBe(true) + expect(h.settings.get('general').appName).toBe('Ansible Chat') + }) + + it('removes what the document leaves out, and keeps the order it lists', async () => { + const h = await configured() + await h.apply((doc) => ({ + ...doc, + plugins: [providerPlugin({ id: 'other', options: { id: 'other' } }), ...doc.plugins], + models: [model({ provider: 'other', id: 'small' }), ...doc.models], + })) + const res = await h.apply((doc) => ({ + ...doc, + roles: doc.roles.filter((role) => role.name !== 'staff'), + plugins: [...doc.plugins].reverse(), + models: [model({ provider: 'other', id: 'small' })], + })) + expect(res.status).toBe(200) + const doc = await h.exported() + expect(doc.roles.map((role) => role.name)).toEqual(['admin']) + expect(doc.plugins.map((plugin) => plugin.id)).toEqual(['fake-provider', 'other']) + expect(doc.models.map((m) => `${m.provider}/${m.id}`)).toEqual(['other/small']) + }) + + it('keeps a stored secret the document leaves out, and clears one set to null', async () => { + const h = await configured() + const kept = await h.apply((doc) => ({ + ...doc, + plugins: doc.plugins.map((plugin) => ({ ...plugin, options: { crash: false } })), + })) + expect(kept.body).toMatchObject({ changed: true }) + expect(h.holder.current().host.providers.get('fake')?.hasAdminKey).toBe(true) + const cleared = await h.apply((doc) => ({ + ...doc, + models: [], + plugins: doc.plugins.map((plugin) => ({ ...plugin, secrets: { apiKey: null } })), + })) + expect(cleared.status).toBe(200) + expect((await h.exported()).plugins[0]?.secretsSet).toEqual([]) + }) + + it('sets the members of a role that lists them, and keeps those of a role that does not', async () => { + const h = await setup() + await h.apply((doc) => ({ + ...doc, + roles: [ + ...doc.roles, + { name: 'listed', members: ['did:plc:a', 'did:plc:b'] }, + { name: 'synced', members: ['did:plc:c'] }, + ], + })) + await h.apply((doc) => ({ + ...doc, + roles: doc.roles.map((role) => { + if (role.name === 'listed') return { ...role, members: ['did:plc:b', 'did:plc:d'] } + const { members: _members, ...rest } = role + return role.name === 'synced' ? rest : role + }), + })) + const members = (name: string) => + h.db + .selectFrom('role_member') + .select('did') + .where('role', '=', name) + .orderBy('did') + .execute() + .then((rows) => rows.map((row) => row.did)) + expect(await members('listed')).toEqual(['did:plc:b', 'did:plc:d']) + expect(await members('synced')).toEqual(['did:plc:c']) + }) + + it("keeps a role's description and matching rules when its entry leaves them out", async () => { + const h = await setup() + const admin = (await h.exported()).roles.find((role) => role.name === 'admin') + const res = await h.apply((doc) => ({ + ...doc, + roles: doc.roles.map((role) => + role.name === 'admin' ? { name: 'admin', members: ['did:plc:a'] } : role, + ), + })) + expect(res.body.changes).toEqual(['role admin members updated']) + expect((await h.exported()).roles.find((role) => role.name === 'admin')).toEqual({ + ...admin, + members: ['did:plc:a'], + }) + }) + + /** A server whose `broken` plugin failed to load at startup. */ + async function withBrokenPlugin() { + const db = createSqliteDb() + await migrateToLatest(db) + await writeInstances( + db, + new SecretBox(Buffer.from(TEST_SECRET_KEY, 'base64')), + [ + providerInstance('fake-provider', { id: 'fake' }, { apiKey: 'sk-admin' }), + providerInstance('broken', { id: 'other', crash: true }, {}), + ], + TEST_ADMIN, + ) + return setup({ db }) + } + + it('applies models while an unchanged plugin is failing to load', async () => { + const h = await withBrokenPlugin() + const res = await h.apply((doc) => ({ ...doc, models: [model()] })) + expect(res.body).toEqual({ + changed: true, + changes: ['model fake/big added'], + warnings: [expect.stringContaining('broken')], + }) + }) + + it('changes other plugins while an unchanged plugin is failing to load', async () => { + const h = await withBrokenPlugin() + const res = await h.apply((doc) => ({ + ...doc, + plugins: [...doc.plugins, providerPlugin({ id: 'third', options: { id: 'third' } })], + })) + expect(res.body).toEqual({ + changed: true, + changes: ['plugin third added'], + warnings: [expect.stringContaining('broken')], + }) + }) + + it('keeps ADMIN_DIDS admins when the document lists no admins', async () => { + const h = await setup() + await h.apply((doc) => ({ + ...doc, + roles: doc.roles.map((role) => (role.name === 'admin' ? { ...role, members: [] } : role)), + })) + expect((await h.call('GET', '/admin/roles')).status).toBe(200) + }) + + type Path = (string | number)[] + + it.each<[string, (doc: Doc) => unknown, Path]>([ + ['a missing section', ({ models: _models, ...doc }) => doc, ['models']], + ['an unknown version', (doc) => ({ ...doc, version: 2 }), ['version']], + [ + 'a setting the server would refuse to start with', + (doc) => ({ + ...doc, + settings: { ...doc.settings, turns: { ...doc.settings.turns, ratePerMinute: 'ten' } }, + }), + ['settings', 'turns', 'ratePerMinute'], + ], + [ + 'a duplicate plugin ID', + (doc) => ({ + ...doc, + plugins: [...doc.plugins, providerPlugin({ options: { id: 'other' } })], + }), + ['plugins', 1, 'id'], + ], + [ + 'a second instance of a single-instance package', + (doc) => ({ + ...doc, + plugins: [ + ...doc.plugins, + { id: 'one', package: 'plain-plugin' }, + { id: 'two', package: 'plain-plugin' }, + ], + }), + ['plugins', 2, 'package'], + ], + [ + 'an unknown role', + (doc) => ({ ...doc, models: [model({ roles: ['ghost'] })] }), + ['models', 0, 'roles'], + ], + [ + 'a member that is not a DID', + (doc) => ({ ...doc, roles: [...doc.roles, { name: 'x', members: ['bob'] }] }), + ['roles', 2, 'members', 0], + ], + [ + 'options that would rename a plugin', + (doc) => ({ ...doc, plugins: [providerPlugin({ options: { id: 'renamed' } })] }), + ['plugins', 0, 'options'], + ], + [ + 'two instances of the same plugin', + (doc) => ({ + ...doc, + plugins: [...doc.plugins, providerPlugin({ id: 'again', options: { id: 'fake' } })], + }), + ['plugins', 1], + ], + ])('refuses %s and changes nothing', async (_name, change, path) => { + const h = await configured() + const before = await h.exported() + const res = await h.config('PUT', change(structuredClone(before))) + expect(res.status).toBe(400) + expect(res.body.issues).toContainEqual(expect.objectContaining({ path })) + expect(await h.exported()).toEqual(before) + }) + + it.each<[string, (doc: Doc) => unknown, string]>([ + ['an unknown key', (doc) => ({ ...doc, extra: true }), 'extra'], + [ + 'an unknown key in a nested setting', + (doc) => ({ + ...doc, + settings: { + ...doc.settings, + sync: { ...doc.settings.sync, safetyNet: { enabeld: false } }, + }, + }), + 'settings.sync.safetyNet.enabeld', + ], + [ + 'an option the plugin does not have', + (doc) => ({ ...doc, plugins: [providerPlugin({ options: { colour: 'red' } })] }), + 'colour', + ], + [ + 'a plugin that fails to load', + (doc) => ({ + ...doc, + plugins: [ + ...doc.plugins, + providerPlugin({ id: 'broken', options: { id: 'other', crash: true } }), + ], + }), + 'broken', + ], + [ + 'a model whose provider has no admin key', + (doc) => ({ ...doc, plugins: [providerPlugin({ secrets: { apiKey: null } })] }), + 'fake/big', + ], + ])('applies %s with a warning', async (_name, change, mentions) => { + const h = await configured() + const res = await h.config('PUT', change(await h.exported())) + expect(res.status).toBe(200) + expect(res.body.warnings).toContainEqual(expect.stringContaining(mentions)) + }) + + it('keeps a setting group the file leaves out, and warns about it', async () => { + const h = await setup() + await h.apply((doc) => ({ + ...doc, + settings: { ...doc.settings, general: { ...doc.settings.general, appName: 'Kept' } }, + })) + const res = await h.apply(({ settings: { general: _general, ...settings }, ...doc }) => ({ + ...doc, + settings, + })) + expect(res.body).toMatchObject({ + changed: false, + warnings: [expect.stringContaining('general')], + }) + expect(h.settings.get('general').appName).toBe('Kept') + }) + + it('stores a plugin whose package is not installed, and exports it as stored', async () => { + const h = await configured() + const orphan = { id: 'later', package: 'not-yet-installed', enabled: true, options: { a: 1 } } + const res = await h.apply((doc) => ({ ...doc, plugins: [...doc.plugins, orphan] })) + expect(res.body.warnings).toContainEqual(expect.stringContaining('later')) + const doc = await h.exported() + expect(doc.plugins[1]).toEqual({ ...orphan, secretsSet: [] }) + expect((await h.config('PUT', doc)).body).toMatchObject({ changed: false }) + }) + + it('stores an option the plugin does not have, even when nothing else changes', async () => { + const h = await configured() + const res = await h.apply((doc) => ({ + ...doc, + plugins: doc.plugins.map((plugin) => ({ ...plugin, options: { colour: 'red' } })), + })) + expect(res.body).toMatchObject({ changed: true }) + const doc = await h.exported() + expect(doc.plugins[0]?.options).toEqual({ colour: 'red' }) + expect((await h.config('PUT', doc)).body).toMatchObject({ changed: false }) + }) + + it.each<[string, (doc: Doc) => Doc, string]>([ + [ + 'its package changes and its secrets are dropped', + (doc) => ({ + ...doc, + models: [], + plugins: [{ id: 'fake-provider', package: 'plain-plugin' }], + }), + 'fake-provider', + ], + ])('warns when %s', async (_name, change, mentions) => { + const h = await configured() + const res = await h.apply(change) + expect(res.status).toBe(200) + expect(res.body.warnings).toContainEqual(expect.stringContaining(mentions)) + }) + + it('keeps an apply when an admin saves a plugin while it runs', async () => { + const h = await configured() + const doc = await h.exported() + await h.signIn(TEST_ADMIN) + gate = Promise.withResolvers() + const applying = h.config('PUT', { + ...doc, + plugins: [ + ...doc.plugins, + providerPlugin({ id: 'third', options: { id: 'third', hold: true } }), + ], + }) + await new Promise((resolve) => setTimeout(resolve, 50)) + const saving = h.call('PUT', '/admin/plugins/fake-provider', { enabled: true, options: {} }) + await new Promise((resolve) => setTimeout(resolve, 50)) + gate.resolve() + expect((await applying).status).toBe(200) + expect((await saving).status).toBe(200) + expect((await h.exported()).plugins.map((plugin) => plugin.id)).toEqual([ + 'fake-provider', + 'third', + ]) + }) + + it('warns that it ignored a secretsSet that differs from the stored secrets', async () => { + const h = await configured() + const res = await h.apply((doc) => ({ + ...doc, + plugins: doc.plugins.map((plugin) => ({ ...plugin, secretsSet: [] })), + })) + expect(res.body.warnings).toContainEqual(expect.stringContaining('fake-provider')) + expect((await h.exported()).plugins[0]?.secretsSet).toEqual(['apiKey']) + }) + + it('warns once more when a plugin that is not installed is given options', async () => { + const warningsFor = async (options: Record) => { + const h = await configured() + const later = { id: 'later', package: 'not-yet-installed', options } + const res = await h.apply((doc) => ({ ...doc, plugins: [...doc.plugins, later] })) + return res.body.warnings as string[] + } + const without = await warningsFor({}) + expect(await warningsFor({ apiKey: 'sk-plain' })).toHaveLength(without.length + 1) + }) + + it('stores a secret given under options as a secret', async () => { + const h = await configured() + await h.apply((doc) => ({ + ...doc, + plugins: [providerPlugin({ options: { apiKey: 'sk-new' }, secrets: {} })], + })) + const [plugin] = (await h.exported()).plugins + expect(plugin?.options).toEqual({}) + expect(plugin?.secretsSet).toEqual(['apiKey']) + }) + + it('refuses to remove a role an API key holds', async () => { + const h = await configured() + await issueApiKey(h.db, { label: 'staff key', roles: ['staff'] }, TEST_ADMIN) + const res = await h.apply((doc) => ({ + ...doc, + roles: doc.roles.filter((role) => role.name !== 'staff'), + models: [], + })) + expect(res.status).toBe(400) + expect(res.body.issues).toContainEqual(expect.objectContaining({ path: ['roles'] })) + expect((await h.exported()).roles.map((role) => role.name)).toContain('staff') + }) +}) + +describe('site config routes', () => { + it('refuse a signed-in admin without a key', async () => { + const h = await setup() + expect((await h.call('GET', '/admin/config')).status).toBe(401) + }) +}) + +const CLI = fileURLToPath(new URL('../../src/site-config-cli.ts', import.meta.url)) +const servers: { close: () => void }[] = [] +afterEach(() => { + for (const server of servers.splice(0)) server.close() +}) + +/** A server listening on a free port, over a database file the CLI opens too. */ +async function listening() { + const dir = mkdtempSync(join(tmpdir(), 'site-config-')) + const databaseUrl = `sqlite:${join(dir, 'db.sqlite')}` + const h = await setup({ db: createDb(databaseUrl) }) + const server = serve({ fetch: h.app.fetch, port: 0, hostname: '127.0.0.1' }) + servers.push(server) + await new Promise((resolve) => server.once('listening', resolve)) + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}` + /** Run the CLI against the server, from `cwd`, with `input` on stdin. */ + const cli = ( + args: string[], + options: { input?: string; cwd?: string; url?: string; databaseUrl?: string } = {}, + ) => + new Promise<{ code: number; stdout: string; stderr: string }>((resolve) => { + const { INIT_CWD: _caller, ...env } = process.env + const child = execFile( + process.execPath, + [CLI, ...args, '--url', options.url ?? url], + { + cwd: options.cwd ?? dir, + env: { ...env, DATABASE_URL: options.databaseUrl ?? databaseUrl }, + }, + (err, stdout, stderr) => resolve({ code: err ? Number(err.code) : 0, stdout, stderr }), + ) + child.stdin?.end(options.input ?? '') + }) + const cliKeys = async () => + (await listApiKeys(h.db)).filter((key) => key.label === 'site-config CLI') + return { ...h, dir, cli, cliKeys } +} + +describe('site-config CLI', () => { + it('exports through a single-use key it revokes afterwards', async () => { + const h = await listening() + const { code, stdout } = await h.cli(['export']) + expect(code).toBe(0) + expect(JSON.parse(stdout)).toEqual(await h.exported()) + expect(await h.cliKeys()).toEqual([]) + }) + + it('applies a document from stdin', async () => { + const h = await listening() + const doc = await h.exported() + const { code } = await h.cli(['apply'], { + input: JSON.stringify({ ...doc, roles: [...doc.roles, { name: 'staff' }] }), + }) + expect(code).toBe(0) + expect((await h.exported()).roles.map((role) => role.name)).toContain('staff') + }) + + it('applies a document from a file named from where it ran', async () => { + const h = await listening() + const doc = await h.exported() + writeFileSync( + join(h.dir, 'site.json'), + JSON.stringify({ ...doc, roles: [...doc.roles, { name: 'staff' }] }), + ) + const { code } = await h.cli(['apply', 'site.json'], { cwd: h.dir }) + expect(code).toBe(0) + expect((await h.exported()).roles.map((role) => role.name)).toContain('staff') + }) + + it('refuses a --url that is not a URL, before touching the database', async () => { + const h = await listening() + expect((await h.cli(['export'], { url: 'not a url' })).code).toBe(2) + expect(await h.cliKeys()).toEqual([]) + }) + + it('refuses a database file that does not exist, without creating one', async () => { + const h = await listening() + const missing = join(h.dir, 'missing', 'db.sqlite') + expect((await h.cli(['export'], { databaseUrl: `sqlite:${missing}` })).code).toBe(2) + expect(existsSync(missing)).toBe(false) + }) + + it('prints warnings and still exits 0', async () => { + const h = await listening() + const { code, stderr } = await h.cli(['apply'], { + input: JSON.stringify({ ...(await h.exported()), extra: true }), + }) + expect(code).toBe(0) + expect(stderr).toContain('extra') + }) + + it('exits 1 with the issues when the apply is refused', async () => { + const h = await listening() + const doc = await h.exported() + const { code, stderr } = await h.cli(['apply'], { + input: JSON.stringify({ ...doc, models: [model({ roles: ['ghost'] })] }), + }) + expect(code).toBe(1) + expect(stderr).toContain('ghost') + }) + + /** An admin key issued two hours ago by `by`, labelled like the CLI's own. */ + async function oldKey(h: Awaited>, by: string) { + const { id } = await issueApiKey(h.db, { label: 'site-config CLI', roles: ['admin'] }, by) + await h.db + .updateTable('api_key') + .set({ created_at: new Date(Date.now() - 2 * 60 * 60_000).toISOString() }) + .where('id', '=', id) + .execute() + } + + it('revokes CLI keys a killed run left behind more than an hour ago', async () => { + const h = await listening() + await oldKey(h, 'site-config CLI') + await h.cli(['export']) + expect(await h.cliKeys()).toEqual([]) + }) + + it('keeps an old key an admin issued, whatever its label', async () => { + const h = await listening() + await oldKey(h, TEST_ADMIN) + await h.cli(['export']) + expect(await h.cliKeys()).toHaveLength(1) + }) +}) diff --git a/apps/server/tests/helpers/admin.ts b/apps/server/tests/helpers/admin.ts index df2d1cc..0299fb1 100644 --- a/apps/server/tests/helpers/admin.ts +++ b/apps/server/tests/helpers/admin.ts @@ -1,5 +1,6 @@ import pino from 'pino' import { vi } from 'vitest' +import type { Db } from '../../src/db/index.ts' import { migrateToLatest } from '../../src/db/migrate.ts' import type { InstalledPlugins } from '../../src/plugins/installed.ts' import { createServer } from '../../src/server.ts' @@ -10,9 +11,12 @@ import { installedFrom } from './plugins.ts' type Json = Record -/** A whole server with TEST_ADMIN as its admin, and helpers to call it as any DID. */ -export async function adminHarness(options: { installed?: InstalledPlugins } = {}) { - const db = createSqliteDb() +/** + * A whole server with TEST_ADMIN as its admin, and helpers to call it as any DID. Pass `db` to + * share a database file with another process. + */ +export async function adminHarness(options: { installed?: InstalledPlugins; db?: Db } = {}) { + const db = options.db ?? createSqliteDb() await migrateToLatest(db) let signingIn = TEST_ADMIN const identity = fakeIdentity() diff --git a/docs/deployment.md b/docs/deployment.md index eeca5b0..845e069 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -25,3 +25,42 @@ ``` */5 * * * * curl -fsS -X POST -H "Authorization: Bearer " https://chat.example.com/api/cron ``` + +## Config import/export + +All admin settings can be exported and imported via the CLI. + +1. Export the configuration: + + ``` + docker compose exec -T scn-chat node apps/server/src/site-config-cli.ts export > site-config.json + ``` + + The JSON file contains all admin settings, roles, plugins, and admin models. Secrets are not exported. + +2. Import the config: + + ``` + docker compose exec -T scn-chat node apps/server/src/site-config-cli.ts apply < site-config.json + ``` + + Changes take effect without a restart. If the import includes setting changes that will prevent the server from starting, the script will bail. Broken plugin settings that don't affect startup will still be imported with a warning. + + +Applying the config is a full reset so any admin config that wasn't in the initial export will be cleared. + +- Secrets missing from the file are ignored / left unchanged on the server. To clear a secret, set it to `null`, such as `"secrets": { "tavilyApiKey": null }`. +- A role in the JSON that leaves `members` unset will keep its members. +- Plugin packages need to be added to `package.json` to be available in the site. + +For Ansible, template the file from your variables and Vault and apply it: + +```yaml +- name: Apply the SCN Chat configuration + ansible.builtin.shell: > + docker compose exec -T scn-chat node apps/server/src/site-config-cli.ts apply < site-config.json + args: + chdir: /srv/scn-chat + register: site_config + changed_when: site_config.stdout != 'No changes.' +``` diff --git a/package.json b/package.json index 4b4390e..449c27e 100644 --- a/package.json +++ b/package.json @@ -23,7 +23,8 @@ "publish-lexicons": "pnpm --filter @scn-chat/lexicons publish-lexicons", "migrate": "pnpm --filter @scn-chat/server migrate", "keys": "pnpm --filter @scn-chat/server keys", - "release": "pnpm --filter @scn-chat/release release" + "release": "pnpm --filter @scn-chat/release release", + "site-config": "pnpm --filter @scn-chat/server site-config" }, "devDependencies": { "@biomejs/biome": "2.5.14", diff --git a/specs/README.md b/specs/README.md index 78a4e93..e1d4410 100644 --- a/specs/README.md +++ b/specs/README.md @@ -22,10 +22,11 @@ The overview at the end of this file lists every spec and whether it is active, ### Active +- [[site-config]]: the whole admin configuration exported and applied as one JSON document, through a CLI on the server, for deployment tools like Ansible. + - [[extending]]: the web app made easy to change in a fork, with tables of part and tool views, tool status and steps, message actions as a list with Copy, a Markdown seam with code block copy, smaller components, and a guide in `docs/extending.md`. Phase 3 of the web rebuild, partly built. ### Planned - - [[encrypted-conversations]] ### Completed diff --git a/specs/admin-plugins.md b/specs/admin-plugins.md index c1725e5..d3b842c 100644 --- a/specs/admin-plugins.md +++ b/specs/admin-plugins.md @@ -26,7 +26,7 @@ A plugin package is installed when it is listed in the `dependencies` of the roo ``` plugin_instance ( - id text primary key, -- random, from the server + id text primary key, -- the plugin's ID, or one a site config chose package text, position integer, enabled integer, @@ -38,6 +38,7 @@ plugin_instance ( - **Secrets.** Top-level option fields marked `.meta({ secret: true })` are stored in `secrets_encrypted`, with the same `SecretBox` as user keys. The API never returns them. It returns the names of the secret fields that have a value. On save, a blank or missing secret keeps the stored value, and a field listed in `clearSecrets` is removed. Every shipped plugin marks its `apiKey` option as secret. Secrets that no current or retired key can decrypt read as unset, so the server still starts and the plugin loads or fails as it would without them. The API reports `secretsUnreadable` for that instance, and its secret fields say their saved value can't be read. The ciphertext is written back unchanged until the admin enters a secret or ticks Clear on any secret field, which discards it. - **Order.** `position` is the load order, which decides hook order as the plugins spec describes. +- **Instance IDs.** An instance added in the admin area takes the ID of the plugin it makes, such as `web-search` or `openai-compatible-litellm`, and adding one whose plugin ID an instance already has is refused. Instances from before this, under random UUIDs, are renamed to their plugin's ID at startup, keeping their options and secrets. One whose options no longer load, or whose plugin ID another instance has, keeps its UUID, and the log says why. Nothing else stores instance IDs, so renaming is safe. - **Disabled instances** keep their options and aren't loaded. - **Plugin IDs.** The factory decides the plugin's ID, as it does today, and some plugins derive it from their options, like `openai-compatible-`. User settings, tool switches, provider IDs in model references, and user credentials all key on those IDs, so an edit that would change an instance's plugin ID is refused with "This change would rename the plugin from X to Y. Add a new instance instead." - **Deleting** an instance leaves its users' settings and tool switches in place, so adding the same plugin back restores them. diff --git a/specs/api-keys.md b/specs/api-keys.md index 5225381..7261dd0 100644 --- a/specs/api-keys.md +++ b/specs/api-keys.md @@ -54,7 +54,7 @@ Keys never appear in logs. A rejected request logs the route and the reason, and - No keys acting as a user. A key has no DID and can't read or write chats. - No keys on existing routes. Only routes that name roles with `requireApiKey` accept them. - No expiry. A key lasts until it is revoked. -- No keys in `.env` or files. Keys are issued and revoked in the admin area. +- No keys in `.env` or files. Keys are issued and revoked in the admin area, apart from the single-use keys the `site-config` CLI issues itself and revokes, from [[site-config]]. ## Edge Cases and Decisions diff --git a/specs/site-config.md b/specs/site-config.md new file mode 100644 index 0000000..ad5b737 --- /dev/null +++ b/specs/site-config.md @@ -0,0 +1,199 @@ +# Site Config + +## Summary + +A server's whole admin configuration can be exported as one JSON document and applied from one, so a tool like Ansible can set up a server and keep it as described. The document holds the admin settings, roles, plugin instances, and admin models. `site-config export` prints it, and `site-config apply` makes the server match it, removing anything it doesn't list. Both are commands run on the server, which call the running server through `GET` and `PUT /api/admin/config`, so a change takes effect at once, as it does from the admin area. + +## Motivation + +Every runtime setting lives in the app database and is edited in the admin area, which needs a person signed in with a browser. A deployment managed by Ansible, or any configuration management tool, needs to set up a fresh server without that person, and to bring an existing one back in line on every run. Those tools work by running commands over SSH and are built around stating what a server should look like, so they need a single document to apply that reports whether it changed anything. + +Writing to the database from the command line isn't enough, because the running server holds the settings and the loaded plugins in memory and only reloads them when a change goes through it. Applying through the server also reuses the checks the admin area already makes, such as a plugin's options loading, and a model keeping its provider. + +Installing plugin packages stays outside this, as everywhere else. A plugin is a dependency in the fork's `package.json`, and the deployment tool adds it, rebuilds, and restarts before applying a config that uses it. + +## Design + +### The document + +```json +{ + "version": 1, + "settings": { + "general": { "appName": "SCN Chat", "description": "…" }, + "access": { "registration": "invite", "inviteRoles": ["scn-member"] }, + "sessions": { … }, + "turns": { … }, + "sync": { … } + }, + "roles": [ + { + "name": "scn-member", + "description": "Active members of the Shared Computer Network.", + "pdsHosts": [], + "handleDomains": [] + }, + { + "name": "staff", + "description": "Can use the larger models.", + "pdsHosts": [], + "handleDomains": ["example.com"], + "members": ["did:plc:abc123"] + } + ], + "plugins": [ + { + "id": "web-search", + "package": "@scn-chat/plugin-web-search", + "enabled": true, + "options": { "engine": "tavily" }, + "secrets": { "tavilyApiKey": "tvly-…" } + } + ], + "models": [ + { + "provider": "anthropic", + "id": "claude-opus-5-5", + "name": "Claude Opus 5.5", + "capabilities": { "vision": true, "reasoning": true, "tools": true }, + "roles": ["user"], + "default": true + } + ] +} +``` + +- **`version`** is 1. Any other version is refused, so a later format can't be misread. +- **`settings`** has the admin setting groups from [[admin-settings]], each in the shape its schema reads. A group left out keeps its stored value, with a warning, so a file written before an upgrade added a group still applies. A field left out of a group takes its default. +- **`roles`** are the roles besides the built-in `admin` and `user`, which always exist. A role's `members` is optional: + - **Listed:** the role has exactly those members, including removing ones a plugin or an admin added. + - **Left out:** the role's members are left alone. This is for roles a plugin keeps in sync, such as `scn-member` from [[scn-member-registry]], whose members would otherwise be removed on every apply until the next cron run. + - `description`, `pdsHosts`, and `handleDomains` are optional too. One left out keeps its stored value, or starts empty for a new role. + - `admin` may appear with only `members`, to set its hand-added admins. `ADMIN_DIDS` are admins whatever the document says. +- **`plugins`** are the plugin instances, in load order. An instance's `id` is chosen by the document. Instances added in the admin area are named after their plugin, as [[admin-plugins]] describes, so an export already has readable IDs such as `web-search`. Changing an instance's ID makes it a new instance, which doesn't keep the old one's secrets. `options` holds the plain options, and `secrets` the secret ones by field name. A secret left out of `secrets` keeps its stored value, and one set to `null` is cleared. A secret field given under `options` is stored as a secret. A package that isn't installed is stored as given, and loads once it is installed. +- **`models`** are the admin models, in the order users see them, in the shape `adminModelSchema` reads. + +Every section must be present. An empty list or object is accepted and means none, but a missing section is refused, so a cut-off file can't wipe a server. + +### Export + +`GET /api/admin/config` answers with the document for the server as it is. Secrets are never included. Each plugin instead lists the secret fields that are set, as `"secretsSet": ["tavilyApiKey"]`, which apply accepts and ignores. Exporting and applying the result changes nothing, since every secret left out keeps its value. + +- A plugin's options are exported as stored, including any its package no longer has, so applying the export changes nothing. Apply warns about those, so they can be removed from the file. +- A role with any member a plugin added, recorded as `plugin:`, is exported without `members`, so applying the export leaves that role to its plugin. + +### Apply + +`PUT /api/admin/config` makes the server match the document, as a full reset. It refuses only what would stop the server starting or leave references it can't resolve. Anything a plugin can recover from, it stores and warns about, so a plugin changing its options in an upgrade doesn't fail a deploy. Someone fixes that plugin in the admin area afterwards. + +1. **Read.** The body is checked against the document's schema, each setting against its own schema, and each model against `adminModelSchema`. A setting value its schema refuses is refused, since the server won't start with it stored. +2. **Check against itself.** + - Plugin IDs are unique, and an installed package that allows one instance appears once. + - No change to an instance's options renames its plugin, which would strand the user settings, tool switches, credentials, and models stored under the old plugin ID, and no two enabled instances make the same plugin. These match the admin area's checks. + - Every role a model, `access.inviteRoles`, or a member list names is in the document or built in. + - Every member is a DID. + - No role being removed is held by an API key, since keys aren't part of the document. +3. **Compare.** When the server already matches, the apply ends here and answers `changed: false`, without rebuilding anything. +4. **Build.** When the plugins change, a plugin runtime is built from the document's instances. Plugins are only rebuilt when they change, so an apply that changes other things doesn't restart them. +5. **Write.** Settings, roles and members, instances, and models are written in one transaction, removing every role, instance, and model the document doesn't list. Then the new runtime replaces the old, and the settings store takes the new values, so they apply without a restart. + +The whole apply holds the plugin holder's lock, so an apply and a change from the admin area never interleave. If anything is refused, nothing changes and the answer is a 400 with the issues, in the same shape as the admin area's errors, each pointing into the document, such as `plugins.1.id`. + +The answer is `{ changed, changes, warnings }`. `changes` lists what was added, updated, and removed, such as `"plugin web-search updated"`, so the deployment tool can show it, and `changed` is false when the server already matched. `warnings` lists what was applied but needs a person's attention: + +- A field the document's schema doesn't know, however deep, such as a misspelled nested setting. It is left out. +- A setting group the document leaves out. It keeps its stored value. +- An option or secret an installed plugin doesn't have. It is stored, and the plugin ignores it. +- An instance whose package changed, which drops the secrets it had, since another package's secrets mean nothing to it. +- A `secretsSet` that differs from the secrets that will be stored, since apply ignores it, and clearing a secret takes `null` under `secrets`. +- Options for a package that isn't installed, since its secret fields can't be told apart and anything under `options` is stored unencrypted. +- A plugin that isn't loaded, with why, such as its package not being installed or its options failing its schema. +- A model hidden from users because no loaded plugin serves its provider with an admin key. + +The plugin and model warnings describe the server after the apply, so an apply that changes nothing still reports them. + +Changes are recorded with the key that made them, as `api-key:`, in the same columns that record an admin's DID. + +### Access + +Only API keys with the `admin` role reach `GET` and `PUT /api/admin/config`, through `requireApiKey` from [[api-keys]]. Like cron, the routes are registered before the session-guarded admin routes, and the cross-origin check skips them. A signed-in admin can't use them, and the rest of the admin API stays closed to keys, so a key can never issue other keys. + +### The CLI + +`apps/server/src/site-config-cli.ts`, with a root script `pnpm site-config`: + +- `site-config export` prints the document to stdout. +- `site-config apply [file]` applies the document in the file, or from stdin when no file is given. It prints the changes, and the warnings to stderr, and exits 0 whether or not anything changed, or 1 with the issues when the apply is refused. +- It exits 2 with the usage when run wrong, before touching the database: an unknown command, `apply` with no file at a terminal, a `--url` that isn't a URL, or a SQLite `DATABASE_URL` naming a file that doesn't exist, which would otherwise be created empty. +- When the server refuses the CLI's key, the server at `--url` isn't using the CLI's database, and the CLI says so without printing `DATABASE_URL`, which can hold a password. + +The CLI runs on the server, with the server's `.env` and database. Each run issues itself a single-use API key holding `admin`, labelled `site-config CLI`, calls the running server at `http://127.0.0.1:$PORT`, or at the address given with `--url`, and revokes the key when it's done. `--url` is for a server listening somewhere else, such as a local server that moved to the next free port because its own was taken. A run that is killed can leave its key behind, so each run first revokes any keys the CLI issued, recorded as created by `site-config CLI`, more than an hour old. A key an admin issues is never touched, whatever its label. The server must be running. The CLI doesn't change the database itself, apart from its key. + +In the Docker deployment, a deployment tool runs it in the container, and pipes the document in so it needs no file inside the container: + +``` +docker compose exec -T scn-chat node apps/server/src/site-config-cli.ts export > site-config.json +docker compose exec -T scn-chat node apps/server/src/site-config-cli.ts apply < site-config.json +``` + +Calling `node` directly keeps pnpm's script banner out of the exported JSON. + +### Documentation + +- `CLAUDE.md`: the routes opened to API keys are cron and site config. +- `docs/deployment.md`: a section on exporting and applying the config, written by a maintainer. +- [[api-keys]]: the CLI issues and revokes its own single-use keys, an exception to keys being issued only in the admin area. + +## Scope Boundaries + +- No installing plugin packages. The deployment tool manages `package.json`, the build, and the restart. +- No people. Accounts, invites, suspensions, and API keys aren't in the document, apart from role members. +- No `.env` settings. The document holds only what the admin area edits. +- No partial apply. A document is the whole configuration, and a section can't be left out to keep it. +- No user settings, such as a user's own API keys or plugin settings. +- No secrets in the export. + +## Edge Cases and Decisions + +- **Full reset.** Anything the document doesn't list is removed, so the deployment's files are the record of how the server is set up. A change made in the admin area is undone by the next apply. +- **Optional members.** Leaving out `members` keeps them, so plugin-synced roles don't empty on every apply and leave members without their models until cron runs again. +- **Single-use keys rather than a key in `.env`.** Running the CLI needs shell access on the server, which already gives full control, so it needs no secret of its own, and no permanent key exists for anyone to leak. +- **Through the server rather than the database.** The server keeps settings and plugins in memory and reloads them only on its own changes, and it already holds the checks a change must pass. +- **Plugins are stored, not refused.** A plugin that fails to load, an option it doesn't have, or a package that isn't installed yet is stored with a warning, as the server treats them at startup. A deploy that upgrades a plugin whose options changed still succeeds, and the plugin shows as failed in the admin area until someone fixes it. +- **Refusals are kept for what can't be fixed in the admin area:** a setting value that stops the server starting, and references between roles, models, and members that don't resolve. +- **Defaults fill in what an entry leaves out**, such as a setting's unlisted fields and a plugin being enabled, so a hand-written document can stay short. A whole section can't be left out. +- **A refusal's issues point into the document**, such as `plugins.1.id`, so the CLI's message says where the problem is. +- **Unknown keys are warnings,** so a typo is visible in the deploy's output without failing it, and a field an upgrade removed doesn't either. +- **Removing a plugin instance** removes its stored options and secrets, as removing it in the admin area does. + +## Acceptance Criteria + +- [ ] Export answers with every section, and no secret values, only the names of the secrets that are set. +- [ ] Applying an exported document changes nothing and answers `changed: false`. +- [ ] Apply adds, updates, and removes plugin instances, models, and roles to match the document, in the document's order, and answers with the changes. +- [ ] Settings, plugins, and models changed by an apply take effect without a restart. +- [ ] A secret left out of `secrets` keeps its stored value, and one set to `null` is cleared. +- [ ] A role with `members` ends with exactly those members, and a role without `members` keeps its members. +- [ ] `ADMIN_DIDS` stay admins whatever the document says. +- [ ] A missing section, an unknown version, a setting value its schema refuses, a duplicate plugin ID, a second instance of a single-instance package, options that rename a plugin, two instances of the same plugin, an unknown role, a member that isn't a DID, or removing a role an API key holds refuses the apply, and nothing changes. +- [ ] An unknown field, an option a plugin doesn't have, a plugin that fails to load, a model whose provider isn't served, and a package change that drops secrets are applied, and each is reported in `warnings`. +- [ ] An option a plugin doesn't have is stored even when nothing else changes. +- [ ] A setting group the document leaves out keeps its stored value, with a warning. +- [ ] A plugin change made in the admin area while an apply runs reads the instances after the apply, so it doesn't undo it. +- [ ] A plugin whose package isn't installed is stored as given, and its export applies without changes. +- [ ] A secret field given under `options` is stored as a secret. +- [ ] The config routes accept an `admin` API key, refuse a web session, and need no `Origin` header. +- [ ] The CLI exports to stdout and applies from a file or stdin, through a single-use key it revokes afterwards, and revokes keys it issued more than an hour old, never an admin's. +- [ ] The CLI prints warnings and exits 0, and exits 1 with the issues when an apply is refused. +- [ ] `--url` sends the CLI's requests to the given address. +- [ ] The CLI exits 2 for a `--url` that isn't a URL, and for a SQLite database file that doesn't exist, without creating it. +- [ ] A `secretsSet` that differs from the stored secrets, and options for a package that isn't installed, are reported in `warnings`. + +## Files + +- `apps/server/src/admin/config-routes.ts`, `apps/server/src/admin/site-config.ts` +- `apps/server/src/site-config-cli.ts` +- `apps/server/src/app.ts`, `apps/server/src/auth/routes.ts` +- `apps/server/src/plugins/admin.ts`, `apps/server/src/settings/store.ts`, `apps/server/src/auth/role-store.ts`, `apps/server/src/providers/admin-models.ts` +- `package.json`, `apps/server/package.json` +- `CLAUDE.md`, `docs/deployment.md`, `specs/api-keys.md`