diff --git a/README.md b/README.md index e0bd2d2..59377fe 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,16 @@ bun install bun run dev # builds both, serves the planner on :5173 ``` -Then load `dist/` unpacked at `chrome://extensions`, sign in to Self-Service in -another tab, and click capture. +Then turn on **Developer mode** at `chrome://extensions`, load `dist/` +unpacked, sign in to Self-Service in another tab, and open the planner. The +extension has no icon, so it is the grey puzzle piece in the toolbar. + +The first student to open a term crawls it, which takes a minute or so and +fills the shared cache for everybody after them. Nothing is cached until +somebody does; there is no catalog in this repo to start from. + +Optionally, `bun run companion` keeps a copy of your capture on your own +machine for the scripts under `scripts/` to read. Nothing starts it for you. The extension is built against one origin, because the manifest has to name it literally — `APP_ORIGIN=https://plan.example.edu bun run build` for a hosted @@ -32,14 +40,16 @@ shows the entire diff first and writes nothing until you confirm it. See src/content.ts runs on selfservice.cedarville.edu; fetches, nothing else src/background.ts the bridge; only whitelisted origins may call it src/types.ts raw Colleague shapes, shared by both halves + src/crawl.ts the paging loop, over whoever has a session src/requirements.ts Ellucian's 40-field Group as a tagged union src/merge.ts which course satisfies a requirement in both majors src/schedule.ts meeting times, seat counts, and date-aware conflicts src/prereqs.ts what a course needs, and what needs it src/planner.ts which term each requirement lands in src/catalog.ts the one shape that is public rather than personal - src/server/colleague.ts guest client: the catalog needs no session - src/server/crawler.ts one term per crawl, ~60 pages + src/server/colleague.ts guest client, for wherever a guest endpoint is left + src/server/crawler.ts the crawl loop bound to a guest session and the store + src/server/ingest.ts what it takes to trust somebody else's crawl src/server/store.ts SQLite cache of the section catalog src/client/ the planner: no framework, one CSS file, mount/destroy views src/client/planning.ts one projection, shared by every tab that reads one @@ -55,7 +65,7 @@ major recorded against the first one's program is not. The split is by change rate. Auth bridging is stable and security-sensitive; the planner changes every time we learn something new about Colleague. The two -halves share only `types.ts`. +halves share `types.ts` and `crawl.ts`. Two kinds of data, and only one of them is yours. Section times, seats and instructors are identical for every student, so they are cached in SQLite on @@ -63,6 +73,11 @@ the server and one student's crawl spares everyone else's. An evaluation is a student record and never leaves the machine it was fetched on. There is no account system because there is nothing here to attach to a person. +The catalog half of that used to be the server's own work. Cedarville put the +public course search behind SSO, so the crawl now runs in whichever student's +browser gets there first and is offered back to the cache. See [crawled by +whoever is signed in](#crawled-by-whoever-is-signed-in). + ### deploying without deploying anyone's transcript That promise used to be kept by accident: the planner only ran on localhost, @@ -78,23 +93,27 @@ So the halves are split by what they may hold, not by where they run. `APP_ORIGIN=https://plan.example.edu bun run build` writes the manifest with that origin allowed to talk to the extension, alongside localhost so a development build keeps working. The extension posts each capture to a -companion on `127.0.0.1:7749` — started by the MCP server under `--personal`, -and by nothing else. It accepts `POST /capture` and only from -`chrome-extension://`; a page cannot set its own `Origin`, so no -other tab can reach it. There is no way to read a capture back out over the -port: it takes, it never gives. +companion on `127.0.0.1:7749`, which runs only when you start it with +`bun run companion` and by nothing else. It accepts `POST /capture` and only +from `chrome-extension://`; a page cannot set its own `Origin`, +so no other tab can reach it. There is no way to read a capture back out over +the port: it takes, it never gives. Nothing is lost when no companion runs. The post fails, the planner carries on -in the browser, and the MCP server says which file it was looking for. +in the browser, and the scripts under `scripts/` say which file they were +looking for. The same channel carries what the student decided. "Copy my plan" sends the pins, tracks and credit load through the extension to `POST /picks`, so the -planning tools answer about the degree you chose rather than the cheapest one -that fits — and say which of the two they did. +scripts answer about the degree you chose rather than the cheapest one that +fits, and say which of the two they did. CEDARVILLE_CAPTURE where a capture is kept (default: XDG data dir) CEDARVILLE_PORT the companion's port CEDARVILLE_COMPANION 0 to decline the listener entirely + CATALOG_DB the catalog cache (default: .data/catalog.sqlite) + CRAWL "off" to skip the server's own boot crawl + NODE_ENV "production" to close /dev/capture ### what it refuses to guess @@ -202,58 +221,84 @@ weekday and an hour and never coexist. Every meeting carries its own date range and every comparison uses it; a day-and-time check alone invents clashes and makes half the catalog look unschedulable. -### the catalog needs no login +### the catalog used to need no login -Self-Service gates `/Student/Student/Courses/*` but serves `/Student/Courses/*` -to anyone: it is what the signed-out search page uses. One search in -`SectionListing` view returns sections directly, so a whole term is about sixty -pages rather than one request per course. +It did, once. Self-Service gated `/Student/Student/Courses/*` and served +`/Student/Courses/*` to anyone, because that is what the signed-out search +page used. The server crawled the catalog itself, anonymously, and one pass +served every student. -The server therefore crawls the catalog itself, anonymously, on boot and then -whenever a term's cache passes six hours old, and caches it in SQLite. It -checks every thirty minutes rather than every six hours: ticking at exactly -the staleness threshold means the catalog is always a few seconds too young -when the timer fires, so every other cycle gets skipped and the real cadence -quietly doubles. Fall 2026 is 1784 sections across 943 -courses and takes about thirty seconds. No student session is spent on data -that is identical for all of them, and the registrar sees one crawl instead of -one per user. +Cedarville moved the whole catalog behind SSO. The guest search page now +bounces to the login form, so that crawl is gone and no amount of retrying +brings it back. `GuestColleague` says so in as many words rather than blaming +a missing antiforgery token the page was never going to render. -The extension is then only needed for the one thing that is genuinely -personal: your own program evaluation. +What is left with a session is a student's browser. So the crawl moved there. -### mcp +### crawled by whoever is signed in -An MCP server exposes the catalog, and optionally your own requirements, as -tools. Add it to Claude Code with: +The loop did not have to be rewritten, because it never knew who it was +talking to. Every crawl asks one question of one object: -```sh -claude mcp add cedarville -- bun /abs/path/to/the-cedarville-app/src/mcp/server.ts -``` + interface Searcher { + search(criteria: SearchCriteria): Promise + } + +`GuestColleague` satisfies it, for as long as there is a guest endpoint +anywhere. So does `SelfService`, which is the authenticated twin: the same +body and the same paging, one `/Student` more in the path. The planner wraps +that in a `Searcher` whose `search` hops through the extension, and +`src/crawl.ts` runs over either one without knowing the difference. + +One page per message, not one term, so the paging, the delay between requests +and the stop button all stay on the page where the student can see them. A +term is about sixty pages in `SectionListing` view, plus half as many again in +`CatalogListing` for the requisite text. + +### ingesting somebody else's crawl + +Crawling per student would undo the thing the server-side crawl was for: sixty +pages per student per term, against the registrar, every time. So a crawl is +offered back. + + POST /catalog/:term/ingest + +Whoever opens a term first pays for it, and everybody after them reads the +cache. The property the old design got by construction is bought back with +three guards, because a crawl the server performed was true and a crawl posted +to the server is a claim that every other student will read. + + shape every section checked field by field, and one bad section + refuses the batch. A skipped section is indistinguishable + from a cancelled one. + + completeness the client says whether it reached the last page, and only a + complete crawl may replace a term. `store.replace` deletes + what the crawl did not see, so a student closing the tab + halfway would cancel half a term. -Seven tools are always available, over public catalog data: -`list_terms`, `search_courses`, `course_details`, `list_sections`, -`check_conflicts`, `live_seats`, `refresh_catalog`. + no shrinking a crawl may lose up to a fifth of a term and no more. + Without this one, three well-formed sections and + `complete: true` empty the catalog for everybody. -Those last two are the ones that touch the network. `live_seats` asks the -registrar for current seat counts on a handful of named courses and prints -them beside what the last crawl saw, which is what you want during -registration; `refresh_catalog` re-crawls a whole term and takes about a -minute. +Sections must also carry the term they were posted under, or a Fall crawl +could be written into the Spring cache, where the shrink guard counts rows +rather than reading them. -Five more read a captured evaluation, and are **only registered when the -server is started with `--personal`** (or `CEDARVILLE_MCP_PERSONAL=1`): -`my_requirements`, `my_eligibility`, `compare_programs`, `plan_terms`, -`critical_path`. +There is deliberately no identity here. There is no account system and nothing +to attach one to, so every guard is about the claim rather than the claimant. A +student with a real session can still post a plausible lie about the timetable; +what they cannot do is quietly delete it. That is the honest limit of a shared +cache with no accounts, and it is worth stating rather than implying otherwise. -The opt-in is structural, not a runtime check. Without the flag those tools do -not appear in `tools/list` and calling one returns "tool not found" — there is -nothing to refuse, because nothing is registered. Nothing writes to Colleague; -no tool can register for a class or drop one, and the only writes are into the -local catalog cache. +`POST /catalog/:term/refresh` still exists and still crawls server-side, for +wherever a guest endpoint is open. It now refuses a term the catalog has never +heard of: terms are a closed set the registrar publishes, and accepting any +string meant one caller could start unlimited outbound crawls by inventing +spellings. -The server reads `.data/catalog.sqlite` directly, so the planner server does -not need to be running, but a crawl must have happened at least once. +The extension is needed for two things now: your own program evaluation, which +was always personal, and the catalog, which did not used to be. ### one course list, any major @@ -363,10 +408,12 @@ for courses" button in the degree audit calls. POST /rules/resolve [{requirement, subrequirement, group}, …] ``` -No session is needed — the triple names a place in the catalog, not a student -— so the server resolves it anonymously and caches the answer in SQLite, -shared by everyone. `DABIOL25` is five biology labs; the history elective is -forty-seven courses. +No session used to be needed here: the triple names a place in the catalog, +not a student, so the server resolved it anonymously and cached the answer in +SQLite, shared by everyone. The caching and the sharing still hold. The +anonymous part does not, now that the search is behind SSO, so the server +answers from cache and a miss has to wait for somebody signed in to fill it. +`DABIOL25` is five biology labs; the history elective is forty-seven courses. One kind is deliberately not expanded. A filter naming no subject and no department ("32 hours of upper-division work") matches most of the catalog and @@ -385,7 +432,8 @@ a heavier load, and a spring-only course cannot move to autumn. bun scripts/plan-doc.ts # writes .data/plan.md ``` -The same engine backs the `plan_terms` and `critical_path` MCP tools. +It reads whichever capture the companion wrote, so a capture that landed in +the XDG data directory is found as readily as one in `.data/`. Two things a prerequisite cannot say, and the planner reads both out of the catalog rather than from a table anyone maintains. **Class standing** gates 58 diff --git a/bun.lock b/bun.lock index 1a73aee..57541c2 100644 --- a/bun.lock +++ b/bun.lock @@ -5,7 +5,6 @@ "": { "name": "the-cedarville-app", "dependencies": { - "@modelcontextprotocol/server": "^2.0.0", "zod": "^4.4.3", }, "devDependencies": { @@ -36,10 +35,6 @@ "@biomejs/cli-win32-x64": ["@biomejs/cli-win32-x64@2.5.8", "", { "os": "win32", "cpu": "x64" }, "sha512-I2czzXTY61f3nFJxXoMDq80t7MivxDEnCjE+8sDKoFfcKMaoQdkqhIFQ3KyY0XLzeSpUBYeNAXgD+iOV/BU0VA=="], - "@modelcontextprotocol/core": ["@modelcontextprotocol/core@2.0.0", "", { "dependencies": { "zod": "^4.2.0" } }, "sha512-pJCEwGG7Lfr/+PQp9ZTwKXNeO5wzbfKL7H3MYpCorM4oFBoQrdjnBgEoqG+RjhsvS1FKrDbKux+M1HhlnGWqcA=="], - - "@modelcontextprotocol/server": ["@modelcontextprotocol/server@2.0.0", "", { "dependencies": { "@modelcontextprotocol/core": "2.0.0", "zod": "^4.2.0" } }, "sha512-YhHWdHfpFMQfd0prsEnxKeS3Qz3ytIGmsS0sth4KDjnacIT7hxk6hXHkJ9KysxlkvTM+WZAtQbbcUhdoP4Hvtw=="], - "@types/bun": ["@types/bun@1.3.14", "", { "dependencies": { "bun-types": "1.3.14" } }, "sha512-h1hFqFVcvAvD9j9K7ZW7vd82aSA+rTdznZa+5bwvCwqSB1jmmfLcbIWhOLx1/+boy/xmjgCs/OMUL8hRJSmnPw=="], "@types/chrome": ["@types/chrome@0.0.268", "", { "dependencies": { "@types/filesystem": "*", "@types/har-format": "*" } }, "sha512-7N1QH9buudSJ7sI8Pe4mBHJr5oZ48s0hcanI9w3wgijAlv1OZNUZve9JR4x42dn5lJ5Sm87V1JNfnoh10EnQlA=="], diff --git a/package.json b/package.json index b1b4998..8dd6708 100644 --- a/package.json +++ b/package.json @@ -14,8 +14,7 @@ "format": "biome format --write", "check": "biome check --write", "ci": "biome ci . && tsc --noEmit && bun test", - "mcp": "bun src/mcp/server.ts", - "mcp:personal": "bun src/mcp/server.ts --personal" + "companion": "bun scripts/companion.ts" }, "devDependencies": { "@biomejs/biome": "^2.5.7", @@ -25,7 +24,6 @@ "typescript": "^5.5.0" }, "dependencies": { - "@modelcontextprotocol/server": "^2.0.0", "zod": "^4.4.3" } } diff --git a/scripts/companion.ts b/scripts/companion.ts new file mode 100644 index 0000000..076a149 --- /dev/null +++ b/scripts/companion.ts @@ -0,0 +1,28 @@ +#!/usr/bin/env bun + +/* + * Keeps a capture on your own machine. + * + * The extension posts every capture to a loopback port, and this is what + * listens on it. Nothing else does: the port only opens when you run this, + * so a student who never wants a transcript on disk simply never starts it. + * + * bun run companion + * + * Nothing is lost when it is not running. The extension's post fails, the + * planner carries on in the browser, and the scripts under scripts/ say which + * file they were looking for. + */ + +import { capturePath, picksPath, serveCompanion } from "../src/server/companion"; + +const companion = serveCompanion((path) => console.log(`wrote ${path}`)); + +if (!companion) { + console.error(`something already holds the port; only one companion can listen`); + process.exit(1); +} + +console.log(`companion on 127.0.0.1:${companion.port}, for the extension only`); +console.log(` capture ${capturePath()}`); +console.log(` picks ${picksPath()}`); diff --git a/scripts/plan-doc.ts b/scripts/plan-doc.ts index 0104e69..d05f8e1 100644 --- a/scripts/plan-doc.ts +++ b/scripts/plan-doc.ts @@ -28,6 +28,7 @@ import { import { sequencesFrom } from "../src/sequences"; import type { Book } from "../src/server/book"; import { resolveGroup } from "../src/server/colleague"; +import { capturePath, readCapture } from "../src/server/companion"; import { CatalogStore } from "../src/server/store"; const store = new CatalogStore(); @@ -93,7 +94,11 @@ const books = ( ).sort((a, b) => b.year.localeCompare(a.year)); const book = books[0]; -const snapshot = await Bun.file(".data/evaluations.json").json(); +// Wherever the companion put it, not just the repo's own .data: a capture +// that landed in the XDG directory is still the student's capture. +const snapshot = await readCapture<{ evaluations: Record }>(); +if (!snapshot) + throw new Error(`no capture found at ${capturePath()}; press capture in the planner`); // Whatever was captured, in capture order: no program code is named here. const trees = Object.values(snapshot.evaluations).map((raw) => normalize(raw as never)); const cy = trees[0]; diff --git a/serve.ts b/serve.ts index c38f20e..2bcfd25 100644 --- a/serve.ts +++ b/serve.ts @@ -22,6 +22,7 @@ import { refreshAllCourses, refreshTerm, } from "./src/server/crawler"; +import { ingest } from "./src/server/ingest"; import { CatalogStore, type RuleKey, ruleKey } from "./src/server/store"; import { loopback } from "./src/where"; @@ -44,6 +45,16 @@ const store = new CatalogStore(); /** Terms already being fetched, so a reload cannot start a second crawl. */ const running = new Map>(); +/** + * A term code the catalog has actually heard of. + * + * Guards the crawl trigger. Terms are a closed set the registrar publishes, + * so an arbitrary string is never a term, and accepting one meant any caller + * could start an unlimited number of outbound crawls by inventing spellings. + */ +const known = (term: string): boolean => + term === ALL_COURSES || store.stats().some((row) => row.term === term); + function refresh(term: string): Promise { const existing = running.get(term); if (existing) return existing; @@ -98,9 +109,53 @@ async function api(request: Request, pathname: string): Promise }); } + /** + * A term, crawled by a student and offered to everybody. + * + * The server cannot read the catalog any more: Cedarville put it behind + * SSO, so the only session that can page it belongs to a browser. This is + * where that crawl lands. Every guard on it lives in `ingest`, which is + * worth reading before changing anything here. + */ + const ingestTerm = /^\/catalog\/([^/]+)\/ingest$/.exec(pathname)?.[1]; + if (ingestTerm && request.method === "POST") { + const term = decodeURIComponent(ingestTerm); + let raw: unknown; + try { + raw = await request.json(); + } catch { + return json({ error: "not json" }, 400); + } + // The term is named twice, in the path and in the body, so say which one + // disagreed rather than silently trusting either. + if ((raw as { term?: string })?.term !== term) { + return json({ error: `body is for ${(raw as { term?: string })?.term}, not ${term}` }, 400); + } + + const verdict = ingest(store, raw); + if (!verdict.ok) { + console.warn(`${term}: refused a crawl — ${verdict.why}`); + return json({ error: verdict.why }, 422); + } + console.log( + `${term}: ingested ${verdict.sections} sections and ${verdict.courses} courses` + + (verdict.replaced ? " (replacing what was held)" : " (first crawl of this term)"), + ); + return json(verdict); + } + + /** + * Re-crawl from the server, which only works while a guest endpoint does. + * Kept because it is how the catalog gets filled anywhere Self-Service is + * still open, and because it fails loudly rather than pretending. + */ const refreshTermCode = /^\/catalog\/([^/]+)\/refresh$/.exec(pathname)?.[1]; if (refreshTermCode && request.method === "POST") { const term = decodeURIComponent(refreshTermCode); + // Only a term the catalog already knows, or one a crawl has listed. An + // arbitrary string used to start its own crawl, and the dedupe map keys + // on the string, so N spellings of one term meant N crawls. + if (!known(term)) return json({ error: `unknown term ${term}` }, 404); void refresh(term); return json({ refreshing: term }, 202); } diff --git a/src/client.ts b/src/client.ts index 854708a..bb67ee9 100644 --- a/src/client.ts +++ b/src/client.ts @@ -6,6 +6,7 @@ * here; the browser already has one. */ +import type { SearchCriteria, Searcher, SearchPage } from "./crawl"; import type { CatalogVocabulary, DegreePlanDto, @@ -13,8 +14,6 @@ import type { DegreePlanView, EvaluationResponse, ProgramSummary, - SearchCriteria, - SearchResponse, SectionsResponse, } from "./types"; @@ -49,7 +48,7 @@ export class UnauthorizedError extends Error { } } -export class SelfService { +export class SelfService implements Searcher { #token: string | null = null; /** @@ -161,11 +160,22 @@ export class SelfService { }); } - searchCourses(criteria: Partial): Promise { + /** + * Course search, on the student's own session. + * + * The authenticated twin of the guest endpoint the server used to crawl: + * same body, same paging, one `/Student` more in the path. Cedarville put + * the guest half behind SSO, so this is now the only way the catalog can be + * read at all, which is why it satisfies the same `Searcher` shape the + * crawl loop takes, and why that loop runs here unchanged. + * + * No default view. The crawl asks for the one it wants, and choosing on its + * behalf is how a section crawl quietly comes back full of courses. + */ + search(criteria: SearchCriteria): Promise { return this.post("/Student/Student/Courses/PostSearchCriteria", { pageNumber: 1, quantityPerPage: 100, - searchResultsView: "CatalogListing", ...criteria, }); } diff --git a/src/client/app.ts b/src/client/app.ts index a91f9b8..ceb61e2 100644 --- a/src/client/app.ts +++ b/src/client/app.ts @@ -7,13 +7,15 @@ * apart the moment a fourth thing needs to know about them. Here, setting * state is the only way to change what is on screen. * - * Everything below runs on the student's own machine. The catalog comes from - * this app's server, which fetched it anonymously; the evaluation comes from - * the extension and goes nowhere else. + * Everything below runs on the student's own machine. The catalog is crawled + * through the extension, on this student's own session, and offered to the + * app's server so the next student does not have to crawl it again; the + * evaluation comes from the extension and goes nowhere else. */ import { type TermCatalog, termNow } from "../catalog"; import type { Capture } from "../content"; +import { crawlTerm } from "../crawl"; import { enumeratedCourseIds, normalize, openGroups, type ProgramTree } from "../requirements"; import { capture, @@ -22,8 +24,9 @@ import { fetchAllCourses, fetchCatalog, installed, + offerCatalog, programs, - refreshCatalog, + searcher, terms, } from "./bridge"; import { $ } from "./dom"; @@ -229,6 +232,39 @@ async function awaitCrawl(term: string) { } } +/** + * Crawls a term here, on this student's session, and offers it to the server. + * + * The server used to do this anonymously. Cedarville closed the guest + * endpoint, so the only session that can read a timetable is the one in this + * browser, and the crawl runs through the extension instead. + * + * The offer afterwards is what keeps the old promise: whoever opens a term + * first pays about sixty pages for it, and everybody after them reads the + * cache. A refusal costs nothing here — the sections are already in hand, and + * the server declining to share them is its business, not this student's. + */ +async function crawlHere(term: string): Promise { + const crawled = await crawlTerm(searcher, term, { + onProgress: ({ page, pages, sections, phase }) => + store.set({ + progress: + `${phase === "courses" ? "course details" : "sections"} ` + + `page ${page}/${pages}, ${sections} so far`, + }), + }); + + store.set({ progress: `offering ${crawled.sections.length} sections to the shared cache…` }); + const shared = await offerCatalog(crawled); + say( + shared + ? `crawled ${crawled.sections.length} sections for ${term} and shared them` + : `crawled ${crawled.sections.length} sections for ${term}; the server kept its own copy`, + "ok", + ); + return crawled; +} + $("#load-sections").addEventListener("click", async () => { const term = $("#term").value; if (!term) return say("pick a term first", "err"); @@ -238,25 +274,33 @@ $("#load-sections").addEventListener("click", async () => { const status = await catalogStatus(); const cached = status.terms.find((t) => t.term === term); + if (status.refreshing.includes(term)) await awaitCrawl(term); + + let sections: TermCatalog; if (!cached || cached.sections === 0) { - say(`asking the server to fetch ${term}…`); - await refreshCatalog(term); - await awaitCrawl(term); - } else if (status.refreshing.includes(term)) { - await awaitCrawl(term); + // Nobody has crawled this term yet, so this student does. Needs the + // extension: it is the only thing holding a session. + if (!installed()) { + throw new Error( + `no catalog for ${term} yet, and the bridge extension is not installed. ` + + "Install it and sign in to Self-Service, then try again.", + ); + } + say(`no catalog for ${term} yet; crawling it here…`); + sections = await crawlHere(term); + } else { + const courseIds = candidateCourseIds(); + sections = await fetchCatalog(term, courseIds.length ? courseIds : undefined); + say( + `${sections.sections.length} sections for ${term}, ` + + `fetched ${new Date(sections.fetchedAt).toLocaleTimeString()}`, + "ok", + ); } - const courseIds = candidateCourseIds(); - const sections = await fetchCatalog(term, courseIds.length ? courseIds : undefined); localStorage.setItem(SECTIONS, JSON.stringify(sections)); void dumpForDev("catalog", sections); - store.set({ sections, view: "schedule" }); - say( - `${sections.sections.length} sections for ${term}, ` + - `fetched ${new Date(sections.fetchedAt).toLocaleTimeString()}`, - "ok", - ); } catch (err) { say(message(err), "err"); } finally { diff --git a/src/client/bridge.ts b/src/client/bridge.ts index c8b548e..a812dec 100644 --- a/src/client/bridge.ts +++ b/src/client/bridge.ts @@ -8,10 +8,12 @@ import type { TermCatalog } from "../catalog"; import type { Applied, Capture, ColleaguePlan, Reply, ReplyMap, Request } from "../content"; +import type { SearchCriteria, SearchPage } from "../crawl"; import type { Change } from "../sync"; import type { ProgramSummary } from "../types"; +import { EXTENSION_ID } from "../where"; -export const EXTENSION_ID = "dijggphdklmdeegidljleaogedbahpjo"; +export { EXTENSION_ID }; /** `chrome.runtime` appears on the page only when a connectable extension is installed. */ interface Runtime { @@ -70,6 +72,19 @@ export const colleaguePlan = (): Promise => send({ type: "colleag export const applyPlan = (changes: Change[]): Promise => send({ type: "applyPlan", changes }); +/** + * A `Searcher` that reads the catalog through the student's own session. + * + * This is the whole of what the pivot to user crawls needed on this side. + * The crawl loop in `src/crawl.ts` asks for one method, and this is that + * method with an extension hop in the middle, so every crawl the server used + * to run anonymously now runs here instead and the loop does not know the + * difference. + */ +export const searcher = { + search: (criteria: SearchCriteria): Promise => send({ type: "search", criteria }), +}; + /** * Hands a capture to the local dev server, which writes it to .data/ so the * agent working on this code can read a real response instead of guessing at @@ -122,9 +137,32 @@ export interface CatalogStatus { export const catalogStatus = async (): Promise => (await fetch("/catalog")).json() as Promise; -/** Asks the server to re-crawl. Returns immediately; the crawl runs on. */ -export const refreshCatalog = (term: string) => - fetch(`/catalog/${encodeURIComponent(term)}/refresh`, { method: "POST" }); +/** + * Offers a crawl this browser performed to the shared cache. + * + * The one write in this file that other students read, so it is the one the + * server is entitled to refuse: a crawl that stopped early, or that would + * shrink a term, is turned down and the reason logged there. Returns whether + * it was accepted, and never throws, because the sections are already in hand + * either way and a student who crawled a term should not be shown an error + * about somebody else's cache. + */ +export async function offerCatalog(catalog: TermCatalog & { complete: boolean }): Promise { + try { + const res = await fetch(`/catalog/${encodeURIComponent(catalog.term)}/ingest`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(catalog), + }); + if (!res.ok) { + const { error } = (await res.json()) as { error?: string }; + console.warn(`the server declined the crawl: ${error ?? res.status}`); + } + return res.ok; + } catch { + return false; + } +} /** Current availability for the courses on screen. Never cached. */ export async function liveSeats( diff --git a/src/content.ts b/src/content.ts index 748885a..f5bbf5d 100644 --- a/src/content.ts +++ b/src/content.ts @@ -8,6 +8,7 @@ */ import { SelfService, UnauthorizedError } from "./client"; +import type { SearchCriteria, SearchPage } from "./crawl"; import { normalize, programFor, unservedCredentials } from "./requirements"; import type { Change, PlannedCourse } from "./sync"; import type { @@ -209,6 +210,19 @@ export type Request = * app decides what the changes are; this only carries them out. */ | { type: "applyPlan"; changes: Change[] } + /** + * One page of the course catalog, read on the student's session. + * + * Cedarville put the public catalog behind SSO, so a signed-in browser is + * the only thing left that can read a timetable at all. This is a read of + * data that is identical for every student: it carries no transcript, and + * what comes back is the same page the "Search for courses" screen shows. + * + * One page per message rather than a whole term, so the paging, the delay + * between requests and the abort all stay on the page side where the + * student can see a progress bar and stop it. + */ + | { type: "search"; criteria: SearchCriteria } /** * What the student chose, on its way to their own machine. Never reaches * this file: the background answers it without troubling Self-Service, @@ -234,6 +248,7 @@ export interface ReplyMap { capture: Capture; colleaguePlan: ColleaguePlan; applyPlan: Applied; + search: SearchPage; } chrome.runtime.onMessage.addListener((msg: Request, _sender, reply) => { @@ -262,6 +277,8 @@ chrome.runtime.onMessage.addListener((msg: Request, _sender, reply) => { } case "applyPlan": return { ok: true, data: await applyPlan(msg.changes) }; + case "search": + return { ok: true, data: await api.search(msg.criteria) }; default: /* * A request this build has never heard of, which in practice means diff --git a/src/crawl.test.ts b/src/crawl.test.ts new file mode 100644 index 0000000..8c13172 --- /dev/null +++ b/src/crawl.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, test } from "bun:test"; +import { + ALL_COURSES, + crawlAllCourses, + crawlGroup, + crawlSeats, + crawlSections, + crawlTerm, + type SearchCriteria, + type Searcher, + type SearchPage, +} from "./crawl"; + +/** + * A searcher that serves canned pages and records what it was asked. + * + * Every crawl here is a paging loop over somebody else's `TotalPages`, so the + * things worth asserting are what it asked for and when it stopped. A fake + * that remembers its calls is the only way to see either. + */ +function fake(pages: SearchPage[]): Searcher & { asked: SearchCriteria[] } { + const asked: SearchCriteria[] = []; + return { + asked, + search(criteria) { + asked.push(criteria); + const page = pages[(criteria.pageNumber ?? 1) - 1]; + if (!page) throw new Error(`asked for page ${criteria.pageNumber}, which was not set up`); + return Promise.resolve(page); + }, + }; +} + +const sectionPage = (ids: string[], pages = 1, term = "2026FA"): SearchPage => ({ + Sections: ids.map((Id) => ({ Id, CourseId: `c${Id}`, TermId: term })), + TotalItems: ids.length, + TotalPages: pages, + CurrentPageIndex: 0, +}); + +const coursePage = (codes: string[], pages = 1): SearchPage => ({ + CourseFullModels: codes.map((code) => { + const [subject, number] = code.split("-"); + return { Id: code, SubjectCode: subject, Number: number }; + }), + TotalItems: codes.length, + TotalPages: pages, + CurrentPageIndex: 0, +}); + +const noDelay = { delayMs: 0 }; + +describe("paging", () => { + test("follows TotalPages rather than guessing", async () => { + const client = fake([sectionPage(["a"], 3), sectionPage(["b"], 3), sectionPage(["c"], 3)]); + + const { sections, complete } = await crawlSections(client, "2026FA", noDelay); + + expect(sections.map((s) => s.Id)).toEqual(["a", "b", "c"]); + expect(complete).toBe(true); + expect(client.asked.map((c) => c.pageNumber)).toEqual([1, 2, 3]); + }); + + test("asks for the view it wants, never the endpoint's default", async () => { + const client = fake([sectionPage(["a"])]); + await crawlSections(client, "2026FA", noDelay); + expect(client.asked[0]?.searchResultsView).toBe("SectionListing"); + + const courses = fake([coursePage(["ACCT-2110"])]); + await crawlAllCourses(courses, noDelay); + expect(courses.asked[0]?.searchResultsView).toBe("CatalogListing"); + }); + + test("a section seen twice across pages lands once", async () => { + // Sections shift between pages while a crawl is running, so the same one + // legitimately arrives on page 1 and page 2. + const client = fake([sectionPage(["a", "b"], 2), sectionPage(["b", "c"], 2)]); + + const { sections } = await crawlSections(client, "2026FA", noDelay); + expect(sections.map((s) => s.Id)).toEqual(["a", "b", "c"]); + }); + + test("a one-page crawl makes one request", async () => { + const client = fake([sectionPage(["a"])]); + await crawlSections(client, "2026FA", noDelay); + expect(client.asked).toHaveLength(1); + }); + + test("reports progress per page, with the phase it is in", async () => { + const seen: string[] = []; + const client = fake([sectionPage(["a"], 2), sectionPage(["b"], 2)]); + + await crawlSections(client, "2026FA", { + ...noDelay, + onProgress: ({ page, pages, phase }) => seen.push(`${phase} ${page}/${pages}`), + }); + + expect(seen).toEqual(["sections 1/2", "sections 2/2"]); + }); +}); + +describe("an interrupted crawl says so", () => { + // The distinction the shared cache is built on: a term that lost half its + // sections and a crawl that stopped halfway look identical in the data, so + // the crawl has to report which it was. + test("aborting mid-crawl comes back incomplete", async () => { + const abort = new AbortController(); + const client: Searcher = { + search(criteria) { + if ((criteria.pageNumber ?? 1) === 2) abort.abort(); + return Promise.resolve(sectionPage([`s${criteria.pageNumber}`], 5)); + }, + }; + + const { sections, complete } = await crawlSections(client, "2026FA", { + ...noDelay, + signal: abort.signal, + }); + + expect(complete).toBe(false); + // Page 2 was fetched, then the abort stopped page 3. + expect(sections).toHaveLength(2); + }); + + test("a crawl that reaches the end is complete", async () => { + const client = fake([sectionPage(["a"], 2), sectionPage(["b"], 2)]); + expect((await crawlSections(client, "2026FA", noDelay)).complete).toBe(true); + }); + + test("a term is complete only when both halves are", async () => { + const abort = new AbortController(); + const client: Searcher = { + search(criteria) { + if (criteria.searchResultsView === "CatalogListing") { + abort.abort(); + return Promise.resolve(coursePage(["ACCT-2110"], 4)); + } + return Promise.resolve(sectionPage(["a"], 1)); + }, + }; + + const term = await crawlTerm(client, "2026FA", { ...noDelay, signal: abort.signal }); + expect(term.sections).toHaveLength(1); + expect(term.complete).toBe(false); + }); +}); + +describe("a term, both ways round", () => { + test("carries sections, courses and the term it was asked for", async () => { + const client: Searcher = { + search: (criteria) => + Promise.resolve( + criteria.searchResultsView === "CatalogListing" + ? coursePage(["ACCT-2110", "ACCT-2120"]) + : sectionPage(["s1", "s2"]), + ), + }; + + const term = await crawlTerm(client, "2026FA", noDelay); + + expect(term.term).toBe("2026FA"); + expect(term.sections).toHaveLength(2); + expect(term.courses).toHaveLength(2); + expect(Date.parse(term.fetchedAt)).toBeGreaterThan(0); + expect(term.complete).toBe(true); + }); + + test("scopes both halves to the one term", async () => { + const client = fake([sectionPage(["s1"]), coursePage(["ACCT-2110"])]); + const asked: (string[] | undefined)[] = []; + await crawlTerm( + { + search: (criteria) => { + asked.push(criteria.terms); + return client.search(criteria); + }, + }, + "2026FA", + noDelay, + ); + expect(asked).toEqual([["2026FA"], ["2026FA"]]); + }); +}); + +describe("every course that exists", () => { + test("sends no term filter, because the question is not about a term", async () => { + const client = fake([coursePage(["ACCT-2110"])]); + await crawlAllCourses(client, noDelay); + expect(client.asked[0]?.terms).toBeUndefined(); + }); + + test("prefers the record being taught when one code has two", async () => { + // Real case: a course being retired beside its replacement, both live + // during the transition, both ACCT-2110. + const page: SearchPage = { + CourseFullModels: [ + { Id: "old", SubjectCode: "ACCT", Number: "2110", MatchingSectionIds: [] }, + { Id: "new", SubjectCode: "ACCT", Number: "2110", MatchingSectionIds: ["s1"] }, + ], + TotalItems: 2, + TotalPages: 1, + CurrentPageIndex: 0, + }; + + const { courses } = await crawlAllCourses({ search: () => Promise.resolve(page) }, noDelay); + expect(courses).toHaveLength(1); + expect(courses[0]?.Id).toBe("new"); + }); + + test("names its progress after the sentinel rather than a term", async () => { + const seen: string[] = []; + await crawlAllCourses( + { search: () => Promise.resolve(coursePage(["ACCT-2110"])) }, + { ...noDelay, onProgress: ({ term }) => seen.push(term) }, + ); + expect(seen).toEqual([ALL_COURSES]); + }); +}); + +describe("seats", () => { + test("asks for nothing when given no courses", async () => { + const client = fake([]); + expect(await crawlSeats(client, "2026FA", [])).toEqual({}); + expect(client.asked).toHaveLength(0); + }); + + test("reads availability per section id", async () => { + const page: SearchPage = { + Sections: [ + { + Id: "s1", + Available: 3, + Capacity: 30, + Enrolled: 27, + Waitlisted: 1, + AvailabilityStatus: "Open", + }, + ], + TotalItems: 1, + TotalPages: 1, + CurrentPageIndex: 0, + }; + + const seats = await crawlSeats({ search: () => Promise.resolve(page) }, "2026FA", ["c1"]); + expect(seats.s1).toEqual({ + available: 3, + capacity: 30, + enrolled: 27, + waitlisted: 1, + status: "Open", + }); + }); +}); + +describe("expanding a rule", () => { + test("carries the group's coordinates and sorts the answer", async () => { + const client = fake([ + { + Courses: [ + { SubjectCode: "BIO", Number: "2010" }, + { SubjectCode: "BIO", Number: "1010" }, + ], + TotalItems: 2, + TotalPages: 1, + CurrentPageIndex: 0, + }, + ]); + + const courses = await crawlGroup(client, { + requirement: "R1", + subrequirement: "S1", + group: "G1", + }); + + expect(courses).toEqual(["BIO-1010", "BIO-2010"]); + expect(client.asked[0]).toMatchObject({ + requirement: "R1", + subrequirement: "S1", + group: "G1", + searchResultsView: "CatalogListing", + }); + }); + + test("a course missing either half of its code is skipped", async () => { + const courses = await crawlGroup( + { + search: () => + Promise.resolve({ + Courses: [{ SubjectCode: "BIO" }, { SubjectCode: "BIO", Number: "1010" }], + TotalItems: 2, + TotalPages: 1, + CurrentPageIndex: 0, + }), + }, + { requirement: "R1", subrequirement: "S1", group: "G1" }, + ); + expect(courses).toEqual(["BIO-1010"]); + }); +}); diff --git a/src/crawl.ts b/src/crawl.ts new file mode 100644 index 0000000..1773bb0 --- /dev/null +++ b/src/crawl.ts @@ -0,0 +1,364 @@ +/** + * Paging the course search, independently of who is signed in. + * + * Colleague's catalog used to be readable by anyone, so the crawl lived on the + * server and one pass served every student. Cedarville moved the whole thing + * behind SSO, and the only thing left with a session is a student's own + * browser. So the loop had to leave the server. + * + * It did not have to be rewritten. Every crawl here asks one question of one + * object — `search(criteria)` — so the loops are parameterised over that and + * nothing else. The server passes a guest client when there is still a guest + * endpoint to use; the planner passes one that round-trips through the + * extension. Same paging, same dedupe, same politeness delay, one + * implementation, because two would drift and only one of them would be + * tested. + * + * Deliberately free of imports from `server/`: this module is bundled into + * the browser, and `server/store` opens SQLite. + */ + +import type { CatalogCourseRecord, ListingSection, TermCatalog } from "./catalog"; + +/** + * What the crawl sends. A partial of Colleague's own criteria object: the + * endpoint fills in every field it is not given, and naming all of them here + * would mean restating defaults we do not care about. + */ +export interface SearchCriteria { + terms?: string[]; + subjects?: string[]; + courseIds?: string[]; + /** Degree-audit coordinates: asks Colleague to evaluate that group's rule. */ + requirement?: string; + subrequirement?: string; + group?: string; + keyword?: string; + pageNumber?: number; + quantityPerPage?: number; + searchResultsView?: "SectionListing" | "CatalogListing"; +} + +/** + * What it sends back. Which of the three collections is populated depends on + * `searchResultsView`, so all three are optional and the caller reads the one + * it asked for. + */ +export interface SearchPage { + Sections?: unknown[]; + Courses?: unknown[]; + CourseFullModels?: unknown[]; + TotalItems: number; + TotalPages: number; + CurrentPageIndex: number; +} + +/** + * The whole contract a crawl needs. Both halves of the app have something + * that satisfies it, and neither had to be built for this. + */ +export interface Searcher { + search(criteria: SearchCriteria): Promise; +} + +export interface CrawlProgress { + term: string; + page: number; + pages: number; + sections: number; + phase?: "sections" | "courses"; +} + +export interface CrawlOptions { + /** Gap between pages. This is a registrar, not a load test. */ + delayMs?: number; + onProgress?: (progress: CrawlProgress) => void; + signal?: AbortSignal; +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +/** + * Runs the pager and hands each page to a collector. + * + * Every crawl below is this loop plus a different thing done with each page, + * and the parts worth getting right are the shared ones: `TotalPages` comes + * from the response rather than being guessed, an abort is checked before + * each request rather than after, and the delay is skipped on the last page + * so a one-page crawl does not sit waiting for nothing. + * + * Returns how many pages were actually read, which is how a caller tells a + * complete crawl from one that was cut short. + */ +async function page( + searcher: Searcher, + criteria: SearchCriteria, + options: CrawlOptions, + take: (result: SearchPage) => void, + progress: (page: number, pages: number) => void, +): Promise<{ pages: number; read: number; total: number }> { + const { delayMs = 300, signal } = options; + let current = 1; + let pages = 1; + let read = 0; + let total = 0; + + while (current <= pages) { + if (signal?.aborted) break; + + const result = await searcher.search({ ...criteria, pageNumber: current }); + pages = Math.max(result.TotalPages ?? 1, 1); + total = result.TotalItems ?? 0; + take(result); + read++; + + progress(current, pages); + current++; + if (current <= pages && delayMs > 0) await sleep(delayMs); + } + return { pages, read, total }; +} + +/** + * A term's sections, which is the timetable: when each one meets, where, with + * whom, and how many seats are left. + * + * One search in `SectionListing` view returns sections directly, so a term is + * about sixty pages rather than one request per course. + */ +export async function crawlSections( + searcher: Searcher, + term: string, + options: CrawlOptions = {}, +): Promise<{ sections: ListingSection[]; complete: boolean }> { + const sections: ListingSection[] = []; + const seen = new Set(); + + const { pages, read } = await page( + searcher, + { terms: [term], searchResultsView: "SectionListing" }, + options, + (result) => { + for (const raw of result.Sections ?? []) { + const section = raw as ListingSection; + // A section shifting between pages mid-crawl must not double up. + if (!section?.Id || seen.has(section.Id)) continue; + seen.add(section.Id); + sections.push(section); + } + }, + (current, totalPages) => + options.onProgress?.({ + term, + page: current, + pages: totalPages, + sections: sections.length, + phase: "sections", + }), + ); + + return { sections, complete: read >= pages }; +} + +/** + * The same term again in catalog view, which is the only place requisites + * come back as readable text rather than an opaque rule id. Half the pages of + * the section crawl, and it is what makes "what blocks what" answerable. + */ +export async function crawlCourses( + searcher: Searcher, + term: string, + options: CrawlOptions = {}, +): Promise<{ courses: CatalogCourseRecord[]; complete: boolean }> { + const byId = new Map(); + + const { pages, read } = await page( + searcher, + { terms: [term], searchResultsView: "CatalogListing" }, + options, + (result) => { + for (const raw of result.CourseFullModels ?? []) { + const course = raw as CatalogCourseRecord; + if (course?.Id) byId.set(course.Id, course); + } + }, + (current, totalPages) => + options.onProgress?.({ + term, + page: current, + pages: totalPages, + sections: byId.size, + phase: "courses", + }), + ); + + return { courses: [...byId.values()], complete: read >= pages }; +} + +/** + * A term, both ways round. + * + * `complete` is true only when both halves paged all the way to the end. An + * interrupted crawl is still worth keeping in the browser, but it must never + * be allowed to replace a whole term in the shared cache: a partial crawl + * looks exactly like a term where half the sections were cancelled. + */ +export async function crawlTerm( + searcher: Searcher, + term: string, + options: CrawlOptions = {}, +): Promise { + const sections = await crawlSections(searcher, term, options); + const courses = await crawlCourses(searcher, term, options); + + return { + term, + fetchedAt: new Date().toISOString(), + sections: sections.sections, + courses: courses.courses, + complete: sections.complete && courses.complete, + }; +} + +/** The sentinel term under which "every course that exists" is stored. */ +export const ALL_COURSES = "ALL"; + +/** + * Every course in the catalog, whether or not it runs this year. + * + * The per-term crawl answers "what is offered"; this answers "what exists". + * They have to be separate, because a prerequisite is often a course nobody + * is teaching this year — EGEE-2010 roots a four-course engineering chain and + * appears in neither cached term. Built from term-scoped data alone, the + * graph loses 36% of its prerequisite nodes and silently understates depth on + * a third of its courses. + */ +export async function crawlAllCourses( + searcher: Searcher, + options: CrawlOptions = {}, +): Promise<{ courses: CatalogCourseRecord[]; complete: boolean }> { + const byId = new Map(); + + const { pages, read } = await page( + searcher, + // No term filter: the whole catalog rather than one term's offerings. + { searchResultsView: "CatalogListing" }, + options, + (result) => { + for (const raw of result.CourseFullModels ?? []) { + const course = raw as CatalogCourseRecord; + if (course?.Id) byId.set(course.Id, course); + } + }, + (current, totalPages) => + options.onProgress?.({ + term: ALL_COURSES, + page: current, + pages: totalPages, + sections: byId.size, + phase: "courses", + }), + ); + + return { courses: dedupeByCode([...byId.values()]), complete: read >= pages }; +} + +/** + * One record per course code, chosen deliberately. + * + * About 1% of codes carry two records: a course being retired beside its + * replacement, both live during the transition ("Prin Accounting I" and + * "Financial Accounting" are both ACCT-2110 this term). Colleague tells them + * apart by id and picks per the student's catalog year; requisite text only + * ever names a code, so a graph keyed by code has to choose. Choosing by + * whichever the crawl happened to see last is not a choice at all — prefer the + * one being taught, then the one that states requisites. + */ +export function dedupeByCode(records: CatalogCourseRecord[]): CatalogCourseRecord[] { + const best = new Map(); + const score = (c: CatalogCourseRecord) => + ((c as { MatchingSectionIds?: string[] }).MatchingSectionIds?.length ?? 0) * 10 + + (c.CourseRequisites?.length ?? 0); + + for (const course of records) { + const code = `${course.SubjectCode}-${course.Number}`; + const held = best.get(code); + if (!held || score(course) > score(held)) best.set(code, course); + } + return [...best.values()]; +} + +export interface Seats { + available: number; + capacity: number; + enrolled: number; + waitlisted: number; + status: string; +} + +/** + * Current availability for a handful of courses. + * + * Seat counts are the one field that moves by the minute during registration, + * so serving them from a six-hour-old crawl makes the number decorative at + * exactly the moment it matters. This asks Colleague directly, scoped to the + * courses a student is actually looking at: one request, under a second. + */ +export async function crawlSeats( + searcher: Searcher, + term: string, + courseIds: string[], + options: CrawlOptions = {}, +): Promise> { + if (courseIds.length === 0) return {}; + + const seats: Record = {}; + await page( + searcher, + { terms: [term], courseIds, searchResultsView: "SectionListing" }, + { ...options, delayMs: options.delayMs ?? 0 }, + (result) => { + for (const raw of result.Sections ?? []) { + const s = raw as ListingSection; + if (!s?.Id) continue; + seats[s.Id] = { + available: s.Available, + capacity: s.Capacity, + enrolled: s.Enrolled, + waitlisted: s.Waitlisted, + status: s.AvailabilityStatus, + }; + } + }, + () => {}, + ); + return seats; +} + +/** + * Expands a requirement group whose eligible courses Colleague will not + * enumerate in an evaluation, by asking the search to evaluate the rule. + */ +export async function crawlGroup( + searcher: Searcher, + ids: { requirement: string; subrequirement: string; group: string }, + options: CrawlOptions = {}, +): Promise { + const names = new Set(); + + await page( + searcher, + { ...ids, searchResultsView: "CatalogListing" }, + { ...options, delayMs: options.delayMs ?? 0 }, + (result) => { + for (const raw of result.Courses ?? []) { + const course = raw as { SubjectCode?: string; Number?: string }; + if (course.SubjectCode && course.Number) { + names.add(`${course.SubjectCode}-${course.Number}`); + } + } + }, + () => {}, + ); + return [...names].sort(); +} diff --git a/src/mcp/server.test.ts b/src/mcp/server.test.ts deleted file mode 100644 index 4baa161..0000000 --- a/src/mcp/server.test.ts +++ /dev/null @@ -1,357 +0,0 @@ -/* - * Drives the MCP server the way a client does: over stdio, in JSON-RPC. - * - * The assertion that matters most is the opt-in one. "Personal tools are not - * registered" has to mean the client cannot see or call them, not that a - * handler politely declines, so these check tools/list and tools/call rather - * than any internal flag. - */ - -import { afterAll, beforeAll, describe, expect, test } from "bun:test"; -import { rm } from "node:fs/promises"; -import type { ListingSection } from "../catalog"; -import { CatalogStore } from "../server/store"; - -const SERVER = new URL("./server.ts", import.meta.url).pathname; -const ROOT = new URL("../..", import.meta.url).pathname; - -/** - * The test preload pins CATALOG_DB for this process, but a spawned child does - * not inherit that, so without an explicit env these tests would read the real - * catalog. A seeded temp file is passed instead: hermetic, and unlike - * ":memory:" it is visible to another process. - */ -const DB = `${ROOT}.data/test-mcp-${process.pid}.sqlite`; - -beforeAll(() => { - const store = new CatalogStore(DB); - store.replace({ - term: "2026FA", - fetchedAt: "2026-08-12T00:00:00.000Z", - sections: [ - { - Id: "s1", - CourseId: "c1", - CourseName: "CS-2210", - Number: "01", - Title: "Data Structures", - Synonym: "5108", - TermId: "2026FA", - MinimumCredits: 3, - MaximumCredits: null, - Capacity: 20, - Enrolled: 19, - Available: 1, - Waitlisted: 0, - AvailabilityStatus: "Open", - IsNonStandardDates: false, - StartDate: "2026-08-19T00:00:00-04:00", - EndDate: "2026-12-11T00:00:00-05:00", - FacultyDisplay: ["Mr. Nicholas J. Parry"], - Meetings: [ - { - Days: [1, 3, 5], - StartTime: "2026-08-11T16:00:00+00:00", - EndTime: "2026-08-11T16:50:00+00:00", - StartDate: "2026-08-19T00:00:00-04:00", - EndDate: "2026-12-11T00:00:00-05:00", - Room: "ENS*241", - Frequency: "W", - IsOnline: false, - InstructionalMethodCode: "LEC", - }, - ], - FormattedMeetingTimes: [], - } as unknown as ListingSection, - ], - courses: [ - { - Id: "c1", - SubjectCode: "CS", - Number: "2210", - Title: "Data Structures", - MinimumCredits: 3, - CourseRequisites: [ - { - DisplayText: "Take CS-1220", - DisplayTextExtension: "- Must be completed prior to taking this course.", - IsRequired: true, - }, - ], - }, - ], - }); - store.close(); -}); - -afterAll(async () => { - for (const suffix of ["", "-wal", "-shm"]) await rm(`${DB}${suffix}`, { force: true }); -}); - -interface Rpc { - id?: number; - result?: { tools?: { name: string }[]; content?: { text: string }[]; isError?: boolean }; - error?: { code: number; message: string }; -} - -/** Runs one session: initialize, then the given calls, then read replies. */ -async function session(args: string[], calls: object[]): Promise> { - const lines = [ - { - jsonrpc: "2.0", - id: 1, - method: "initialize", - params: { - protocolVersion: "2025-06-18", - capabilities: {}, - clientInfo: { name: "test", version: "1" }, - }, - }, - { jsonrpc: "2.0", method: "notifications/initialized" }, - ...calls, - ]; - - const child = Bun.spawn(["bun", SERVER, ...args], { - stdin: new TextEncoder().encode(`${lines.map((l) => JSON.stringify(l)).join("\n")}\n`), - stdout: "pipe", - stderr: "pipe", - cwd: ROOT, - // Explicit, because the preload's pin does not cross a spawn. The - // companion is off for the same reason the database is pinned: a test may - // not hold a real port, and it may not read or write a real transcript. - env: { - ...process.env, - CATALOG_DB: DB, - CEDARVILLE_COMPANION: "0", - CEDARVILLE_CAPTURE: "/nonexistent/cedarville-tests/evaluations.json", - }, - }); - - const out = await new Response(child.stdout).text(); - child.kill(); - - const byId = new Map(); - for (const line of out.split("\n")) { - if (!line.trim().startsWith("{")) continue; - const message = JSON.parse(line) as Rpc; - if (message.id !== undefined) byId.set(message.id, message); - } - return byId; -} - -const listTools = (args: string[] = []) => - session(args, [{ jsonrpc: "2.0", id: 2, method: "tools/list" }]).then( - (m) => m.get(2)?.result?.tools?.map((t) => t.name) ?? [], - ); - -const CATALOG = [ - "list_terms", - "search_courses", - "course_details", - "list_sections", - "check_conflicts", - "live_seats", - "refresh_catalog", -]; -const PERSONAL = ["my_requirements", "my_eligibility", "compare_programs"]; - -describe("the opt-in boundary", () => { - test("catalog tools are always available", async () => { - const tools = await listTools(); - for (const name of CATALOG) expect(tools).toContain(name); - }, 20_000); - - test("personal tools are absent by default, not merely refusing", async () => { - const tools = await listTools(); - for (const name of PERSONAL) expect(tools).not.toContain(name); - }, 20_000); - - test("--personal adds them", async () => { - const tools = await listTools(["--personal"]); - for (const name of [...CATALOG, ...PERSONAL]) expect(tools).toContain(name); - }, 20_000); - - // Not registered has to mean not callable, or the opt-in is decorative. - test("calling a personal tool without opting in fails as unknown", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 3, - method: "tools/call", - params: { name: "my_eligibility", arguments: { term: "2026FA" } }, - }, - ], - ); - const reply = replies.get(3); - expect(reply?.error?.message ?? "").toContain("my_eligibility"); - expect(reply?.result?.content).toBeUndefined(); - }, 20_000); -}); - -describe("planning tools", () => { - // These need a captured evaluation, which the seeded temp DB does not have, - // so assert the failure is legible rather than a crash. - test("plan_terms explains itself when there is no capture", async () => { - const replies = await session( - ["--personal"], - [ - { - jsonrpc: "2.0", - id: 11, - method: "tools/call", - params: { name: "plan_terms", arguments: { program: "NOPE.XX" } }, - }, - ], - ); - const body = replies.get(11)?.result?.content?.[0]?.text ?? ""; - expect(replies.get(11)?.result?.isError).toBe(true); - expect(body.toLowerCase()).toMatch(/no captured evaluation|evaluations\.json/); - }, 20_000); - - // Regression: both planning entry points named "2026FA" literally, so a - // newer catalog would land and be ignored without anything looking wrong. - test("planning reads the newest cached term, not a named one", async () => { - const source = await Bun.file(new URL("./server.ts", import.meta.url).pathname).text(); - const planning = source.slice(source.indexOf("function planningContext")); - // Whitespace-normalised, because the formatter is free to break the chain - // across lines and this should assert on code, not on layout. - const flat = planning.replace(/\s+/g, ""); - expect(flat).not.toMatch(/store\.read\("\d{4}(FA|SP|SU)"\)/); - expect(flat).toContain("store.stats()"); - }); - - test("critical_path is registered only with --personal", async () => { - expect(await listTools()).not.toContain("critical_path"); - expect(await listTools(["--personal"])).toContain("plan_terms"); - expect(await listTools(["--personal"])).toContain("critical_path"); - }, 20_000); -}); - -describe("catalog tools", () => { - test("an unknown term is an error, not an empty success", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 4, - method: "tools/call", - params: { name: "search_courses", arguments: { term: "1999XX", query: "anything" } }, - }, - ], - ); - const reply = replies.get(4)?.result; - expect(reply?.isError).toBe(true); - expect(reply?.content?.[0]?.text).toContain("1999XX"); - }, 20_000); - - test("reads the seeded catalog", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 7, - method: "tools/call", - params: { name: "course_details", arguments: { term: "2026FA", code: "CS-2210" } }, - }, - ], - ); - const body = replies.get(7)?.result?.content?.[0]?.text ?? ""; - expect(body).toContain("Data Structures"); - expect(body).toContain("Take CS-1220"); - }, 20_000); - - test("list_sections converts the UTC meeting time to campus time", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 8, - method: "tools/call", - params: { name: "list_sections", arguments: { term: "2026FA", code: "CS-2210" } }, - }, - ], - ); - // 16:00Z is noon in Ohio in August. - expect(replies.get(8)?.result?.content?.[0]?.text ?? "").toContain("MonWedFri 12:00pm-12:50pm"); - }, 20_000); - - // A prerequisite that is not taught this term is exactly the bottleneck a - // student needs to see, so it must not read as "no such course". - test("a course known only as a prerequisite reports what it gates", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 10, - method: "tools/call", - params: { name: "course_details", arguments: { term: "2026FA", code: "CS-1220" } }, - }, - ], - ); - const reply = replies.get(10)?.result; - expect(reply?.isError).toBeUndefined(); - expect(reply?.content?.[0]?.text ?? "").toContain("not offered in 2026FA"); - expect(reply?.content?.[0]?.text ?? "").toContain("CS-2210"); - }, 20_000); - - test("an unknown course says so rather than returning nothing", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 5, - method: "tools/call", - params: { name: "course_details", arguments: { term: "2026FA", code: "ZZZZ-9999" } }, - }, - ], - ); - expect(replies.get(5)?.result?.isError).toBe(true); - }, 20_000); - - test("check_conflicts rejects ids it does not recognise", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 6, - method: "tools/call", - params: { - name: "check_conflicts", - arguments: { term: "2026FA", section_ids: ["nope", "also-nope"] }, - }, - }, - ], - ); - const reply = replies.get(6)?.result; - expect(reply?.isError).toBe(true); - expect(reply?.content?.[0]?.text).toContain("nope"); - }, 20_000); - - // Both live tools reach the registrar, so the assertions here are about the - // paths that must not: a course we hold no sections for is answerable from - // the cache alone, and answering it over the network would be a bug. - test("live_seats needs no network to reject a course it has never seen", async () => { - const replies = await session( - [], - [ - { - jsonrpc: "2.0", - id: 7, - method: "tools/call", - params: { name: "live_seats", arguments: { term: "2026FA", codes: ["ZZZZ-9999"] } }, - }, - ], - ); - const reply = replies.get(7)?.result; - expect(reply?.isError).toBe(true); - expect(reply?.content?.[0]?.text).toContain("ZZZZ-9999"); - }, 20_000); -}); diff --git a/src/mcp/server.ts b/src/mcp/server.ts deleted file mode 100644 index 1693b81..0000000 --- a/src/mcp/server.ts +++ /dev/null @@ -1,838 +0,0 @@ -#!/usr/bin/env bun -/* - * An MCP server over the course catalog and, if you ask for it, your own - * degree requirements. - * - * The opt-in is structural rather than a runtime check. Catalog tools are - * always registered because that data is public and identical for every - * student. The tools that read a transcript are only registered when the - * server is started with --personal, so a client that has not opted in never - * sees them in tools/list: there is nothing to call, not merely something - * that refuses. - * - * Nothing here writes to Colleague. There is no tool that registers for a - * class or drops one; the only writes are into the local catalog cache, which - * is a copy of public data we were going to fetch anyway. - */ - -import { McpServer } from "@modelcontextprotocol/server"; -import { serveStdio } from "@modelcontextprotocol/server/stdio"; -import * as z from "zod/v4"; -import { nextPlannableTerm, runsIn, seasonsOffered, yearsOffered } from "../catalog"; -import { aliasesOf, buildEquivalences } from "../equivalence"; -import { merge } from "../merge"; -import { criticalPath, projectPlan, STANDING_CREDITS, type TermSlot, termsFrom } from "../planner"; -import { buildGraph, depth, downstream, eligibility, nodeOf } from "../prereqs"; -import { - completedCourses, - coursesNeededAcross, - gaps, - inProgressCourses, - normalize, - openGroups, - type ProgramTree, - splitByCredential, - stillRequiring, - substitutionsIn, - unreadModifications, -} from "../requirements"; -import { conflicts, DAY_NAMES, formatTime, offeringsFromListing } from "../schedule"; -import { capturePath, readCapture, readPicks, serveCompanion } from "../server/companion"; -import { liveSeats, refreshTerm } from "../server/crawler"; -import { CatalogStore } from "../server/store"; - -const store = new CatalogStore(); -const text = (body: string) => ({ content: [{ type: "text" as const, text: body }] }); -const fail = (body: string) => ({ - content: [{ type: "text" as const, text: body }], - isError: true, -}); - -function graphFor(term: string) { - const catalog = store.read(term); - return { catalog, graph: buildGraph((catalog.courses ?? []).map(nodeOf)) }; -} - -/** Crawls in flight, so two calls a minute apart do not crawl twice. */ -const running = new Map>(); - -function refresh(term: string): Promise { - const held = running.get(term); - if (held) return held; - const job = refreshTerm(term, store).finally(() => running.delete(term)); - running.set(term, job); - return job; -} - -// ---- catalog tools, always available ----------------------------------- - -function registerCatalog(server: McpServer) { - server.registerTool( - "list_terms", - { - description: - "Terms whose catalog is cached locally, with how many sections and courses each holds.", - inputSchema: z.object({}), - }, - async () => { - const rows = store.stats(); - if (rows.length === 0) - return fail("No terms cached yet. Run the planner server once to crawl."); - return text( - rows - .map( - (r) => - `${r.term} ${r.sections} sections, ${r.courses} courses, fetched ${r.fetchedAt}`, - ) - .join("\n"), - ); - }, - ); - - server.registerTool( - "search_courses", - { - description: - "Search the course catalog for a term. Returns course codes, titles, credits and how many sections each has.", - inputSchema: z.object({ - term: z - .string() - .describe('Term code, e.g. "2026FA". Use list_terms to see what is cached.'), - query: z - .string() - .optional() - .describe('Match against code and title, e.g. "algorithms" or "CS-3".'), - subject: z.string().optional().describe('Subject code, e.g. "CS" or "EGCP".'), - limit: z.number().int().min(1).max(200).default(40), - }), - }, - async ({ term, query, subject, limit }) => { - const { catalog } = graphFor(term); - if (catalog.sections.length === 0) return fail(`No catalog cached for ${term}.`); - - const sectionsByCourse = new Map(); - for (const s of catalog.sections) { - sectionsByCourse.set(s.CourseId, (sectionsByCourse.get(s.CourseId) ?? 0) + 1); - } - - const needle = query?.toLowerCase(); - const rows = (catalog.courses ?? []) - .map((c) => ({ c, code: `${c.SubjectCode}-${c.Number}` })) - .filter(({ c, code }) => { - if (subject && c.SubjectCode.toUpperCase() !== subject.toUpperCase()) return false; - if (!needle) return true; - return `${code} ${c.Title}`.toLowerCase().includes(needle); - }) - .slice(0, limit); - - if (rows.length === 0) return text("No matching courses."); - return text( - rows - .map( - ({ c, code }) => - `${code.padEnd(11)} ${String(c.MinimumCredits ?? "?").padStart(4)}cr ` + - `${String(sectionsByCourse.get(c.Id) ?? 0).padStart(2)} sections ${c.Title}`, - ) - .join("\n"), - ); - }, - ); - - server.registerTool( - "course_details", - { - description: - "Everything known about one course: credits, description, prerequisites in the registrar's own words, and how many later courses depend on it.", - inputSchema: z.object({ - term: z.string(), - code: z.string().describe('Course code, e.g. "CS-3410".'), - }), - }, - async ({ term, code }) => { - const wanted = code.toUpperCase(); - const { catalog, graph } = graphFor(term); - const record = (catalog.courses ?? []).find( - (c) => `${c.SubjectCode}-${c.Number}`.toUpperCase() === wanted, - ); - - // A course can be absent from a term's catalog and still matter: it is - // named as a prerequisite by courses that *are* offered. Reporting only - // "not found" would hide exactly the bottlenecks worth knowing about. - if (!record) { - const gated = downstream(graph, wanted); - if (gated.size === 0) return fail(`${wanted} is not in the ${term} catalog.`); - const direct = [...(graph.unlocks.get(wanted) ?? [])].sort(); - return text( - [ - `${wanted} is not offered in ${term}, but ${gated.size} course${gated.size === 1 ? "" : "s"} depend on it.`, - "", - `directly gates: ${direct.join(" ")}`, - `transitively: ${[...gated].sort().join(" ")}`, - "", - "Check another term for when it is taught.", - ].join("\n"), - ); - } - - const node = graph.courses.get(wanted); - const unlocks = downstream(graph, wanted); - const lines = [ - `${wanted} ${record.Title} (${record.MinimumCredits ?? "?"} cr)`, - record.Description?.trim() ? `\n${record.Description.trim()}\n` : "", - "prerequisites:", - ...(node?.requisites.length - ? node.requisites.map( - (r) => - ` ${r.required ? "required" : "recommended"} [${r.timing}] ${r.text}` + - (r.understood ? "" : " (prose; not machine-checkable)"), - ) - : [" none"]), - "", - `depth in its own chain: ${depth(graph, wanted)}`, - `unlocks ${unlocks.size} later course${unlocks.size === 1 ? "" : "s"}` + - (unlocks.size ? `: ${[...unlocks].sort().join(" ")}` : ""), - ]; - return text(lines.filter((l) => l !== "").join("\n")); - }, - ); - - server.registerTool( - "list_sections", - { - description: - "Meeting times, rooms, instructors and seat counts for a course's sections. Seats are from the last crawl, not live.", - inputSchema: z.object({ term: z.string(), code: z.string() }), - }, - async ({ term, code }) => { - const wanted = code.toUpperCase(); - const catalog = store.read(term); - const offerings = offeringsFromListing( - catalog.sections.filter((s) => s.CourseName?.toUpperCase() === wanted), - ); - if (offerings.length === 0) return fail(`No sections of ${wanted} in ${term}.`); - - return text( - offerings - .map((o) => { - const when = o.meetings.length - ? o.meetings - .map( - (m) => - `${m.days.map((d) => DAY_NAMES[d]).join("")} ` + - `${formatTime(m.start)}-${formatTime(m.end)}` + - (m.room ? ` ${m.room}` : ""), - ) - .join("; ") - : "no set meeting time"; - return ( - `${o.courseName}-${o.number} id=${o.id} syn=${o.synonym} ${when} ` + - `${o.seats.available}/${o.seats.capacity} open` + - (o.instructors.length ? ` ${o.instructors.join(", ")}` : "") + - (o.nonStandardDates ? " [partial term]" : "") - ); - }) - .join("\n"), - ); - }, - ); - - server.registerTool( - "check_conflicts", - { - description: - "Do these sections collide? Compares weekday, time and date range, so two half-semester sections in different halves do not count as a clash.", - inputSchema: z.object({ - term: z.string(), - section_ids: z.array(z.string()).min(2).describe("Section ids from list_sections."), - }), - }, - async ({ term, section_ids }) => { - const catalog = store.read(term); - const wanted = new Set(section_ids); - const offerings = offeringsFromListing(catalog.sections.filter((s) => wanted.has(s.Id))); - - const missing = section_ids.filter((id) => !offerings.some((o) => o.id === id)); - if (missing.length) return fail(`Unknown section ids: ${missing.join(", ")}`); - - const found = conflicts(offerings); - const credits = offerings.reduce((n, o) => n + o.credits.min, 0); - if (found.length === 0) { - return text(`No conflicts. ${offerings.length} sections, ${credits} credits.`); - } - return text( - [ - `${found.length} conflict${found.length === 1 ? "" : "s"} across ${offerings.length} sections (${credits} credits):`, - ...found.map((c) => { - const [x, y] = c.meetings; - const day = x.days.find((d) => y.days.includes(d)) ?? 0; - return ` ${c.a.courseName}-${c.a.number} vs ${c.b.courseName}-${c.b.number}: ${DAY_NAMES[day]} ${formatTime(Math.max(x.start, y.start))}-${formatTime(Math.min(x.end, y.end))}`; - }), - ].join("\n"), - ); - }, - ); - - server.registerTool( - "live_seats", - { - description: - "Seat counts fetched from Colleague right now, for a few named courses, next to what the last crawl recorded. Every other tool here serves cached data; seats are the one field that moves by the minute during registration, so ask this before telling anyone a seat exists.", - inputSchema: z.object({ - term: z.string(), - codes: z - .array(z.string()) - .min(1) - .max(20) - .describe('Course codes, e.g. ["CS-2210", "HON-1010"]. One request covers them all.'), - }), - }, - async ({ term, codes }) => { - const wanted = new Set(codes.map((c) => c.toUpperCase())); - const catalog = store.read(term); - const mine = catalog.sections.filter((s) => wanted.has(String(s.CourseName).toUpperCase())); - if (mine.length === 0) return fail(`No sections of ${[...wanted].join(", ")} in ${term}.`); - - const seats = await liveSeats(term, [...new Set(mine.map((s) => s.CourseId))]); - if (Object.keys(seats).length === 0) { - return fail(`Colleague returned nothing for ${term}; the cached counts still stand.`); - } - - const rows = mine - .map((s) => ({ s, live: seats[s.Id] })) - .sort((a, b) => - `${a.s.CourseName}-${a.s.Number}`.localeCompare(`${b.s.CourseName}-${b.s.Number}`), - ) - .map(({ s, live }) => { - const name = `${s.CourseName}-${s.Number}`; - // A section can vanish between crawls — cancelled, or merged into - // another. Saying so is more useful than printing the stale count. - if (!live) - return `${name.padEnd(13)} no longer listed (cached: ${s.Available}/${s.Capacity})`; - const moved = live.available !== s.Available ? ` was ${s.Available}` : ""; - const wait = live.waitlisted ? ` ${live.waitlisted} waitlisted` : ""; - return `${name.padEnd(13)} ${live.available}/${live.capacity} open ${live.status}${moved}${wait}`; - }); - - return text([...rows, "", `cached crawl: ${catalog.fetchedAt}`].join("\n")); - }, - ); - - server.registerTool( - "refresh_catalog", - { - description: - "Re-crawl a whole term into the local cache: sections, meeting times and course records. Takes about a minute and is paced out of courtesy to the registrar, so for a seat count on a handful of courses use live_seats instead.", - inputSchema: z.object({ - term: z.string().describe('Term code, e.g. "2026FA".'), - }), - }, - async ({ term }) => { - const before = store.stats().find((s) => s.term === term); - const sections = await refresh(term); - // refreshTerm keeps the old catalog when a crawl comes back empty, which - // is the right call but would otherwise read here as a silent success. - if (sections === 0) { - return fail(`Crawl of ${term} came back empty. The previous catalog is untouched.`); - } - const after = store.stats().find((s) => s.term === term); - const delta = before ? sections - before.sections : 0; - return text( - `${term}: ${sections} sections, ${after?.courses ?? 0} courses, fetched ${after?.fetchedAt}.` + - (before ? ` (${delta >= 0 ? "+" : ""}${delta} since ${before.fetchedAt})` : ""), - ); - }, - ); -} - -// ---- personal tools, only when opted in -------------------------------- - -async function loadTrees(): Promise> { - const snapshot = await readCapture<{ evaluations: Record }>(); - if (!snapshot) { - throw new Error( - `no capture yet at ${capturePath()} — open the planner and press capture ` + - `while this server is running, and the extension will hand one over`, - ); - } - return Object.fromEntries( - Object.entries(snapshot.evaluations).map(([code, raw]) => [code, normalize(raw)]), - ); -} - -/** Shared setup for the planning tools: graph, credits, seasons. */ -function planningContext() { - // Newest terms first; naming them literally is how a planner goes stale. - const cached = store - .stats() - .map((s) => s.term) - .sort() - .reverse(); - const regular = cached.find((t) => !t.includes("SU")); - const summerTerm = cached.find((t) => t.includes("SU")); - - const fall = regular ? store.read(regular) : { sections: [], courses: [] }; - const summer = summerTerm ? store.read(summerTerm) : { sections: [], courses: [] }; - // Every course we hold, not the two terms we happen to have crawled: a - // requisite names courses nobody is teaching, and a graph built from one - // term's listing loses about a third of its depth. Here that was 892 of - // 2047 courses. - const held = store.readCourses("ALL"); - const records = held.length ? held : [...(fall.courses ?? []), ...(summer.courses ?? [])]; - - const credits = new Map( - records.map((c) => [`${c.SubjectCode}-${c.Number}`, c.MinimumCredits ?? 0]), - ); - const titles = new Map(records.map((c) => [`${c.SubjectCode}-${c.Number}`, c.Title])); - const graph = buildGraph(records.map(nodeOf)); - - // Colleague publishes course equivalence on section records, not on the - // catalog view, so a transcript from an earlier catalog year still matches. - const codeForId = new Map(records.map((c) => [String(c.Id), `${c.SubjectCode}-${c.Number}`])); - const equivalences = buildEquivalences( - [...fall.sections, ...summer.sections] - .map((s) => ({ - code: s.CourseName, - // Declared on the nested course record, not on the section itself. - equatedIds: ((s as { Course?: { EquatedCourseIds?: string[] } }).Course?.EquatedCourseIds ?? - []) as string[], - })) - .filter((x) => x.code && x.equatedIds.length), - (id) => codeForId.get(id), - ); - - // The registrar states when a course runs and in which years; inferring it - // from one term's listing was wrong for 367 courses. - const seasons = new Map(records.map((c) => [`${c.SubjectCode}-${c.Number}`, seasonsOffered(c)])); - const cycles = new Map(records.map((c) => [`${c.SubjectCode}-${c.Number}`, yearsOffered(c)])); - const offeredIn = (code: string, slot: TermSlot) => { - const stated = seasons.get(code); - if (stated?.length && !stated.includes(slot.season)) return false; - return runsIn(cycles.get(code) ?? "all", slot.year, slot.season); - }; - - return { - graph, - titles, - offeredIn, - aliases: (c: string) => aliasesOf(equivalences, c), - credits: (c: string) => credits.get(c) ?? 3, - }; -} - -function registerPersonal(server: McpServer) { - server.registerTool( - "my_requirements", - { - description: - "Your own degree requirements that are still open, from a captured evaluation. Reads a local file; nothing is fetched.", - inputSchema: z.object({ - program: z - .string() - .optional() - .describe("Program code. Defaults to every captured evaluation."), - only_gaps: z - .boolean() - .default(false) - .describe("Only requirements with nothing on your degree plan covering them."), - }), - }, - async ({ program, only_gaps }) => { - const trees = await loadTrees(); - const chosen = (program ? [trees[program]] : Object.values(trees)).filter( - (t): t is ProgramTree => Boolean(t), - ); - if (chosen.length === 0) return fail(`No captured evaluation for ${program}.`); - - const out: string[] = []; - for (const tree of chosen as ProgramTree[]) { - const rows = only_gaps ? gaps(tree) : openGroups(tree); - out.push( - `${tree.code} ${tree.title} ${tree.credits.completed}/${tree.credits.minimum} credits, ${rows.length} open`, - ); - for (const { requirement, group } of rows) { - const need = [ - group.min.credits && `${group.min.credits}cr`, - group.min.courses && `${group.min.courses} courses`, - ] - .filter(Boolean) - .join(" "); - const c = group.constraint; - const pool = - c.kind === "take-all" || c.kind === "choose-from" - ? c.courses.map((x) => x.CourseName).join(" ") - : c.kind === "rule-based" - ? "(decided by a Colleague rule; course list not published)" - : ""; - out.push(` [${group.status.completion}] ${need} ${group.text || requirement.text}`); - if (pool) out.push(` ${pool}`); - } - } - return text(out.join("\n")); - }, - ); - - server.registerTool( - "my_eligibility", - { - description: - "Which courses you can take now, and what is blocking the rest, using your completed courses against the prerequisite graph.", - inputSchema: z.object({ - term: z.string(), - subject: z.string().optional().describe('Narrow to one subject, e.g. "CS".'), - state: z.enum(["open", "blocked", "unknown", "all"]).default("all"), - }), - }, - async ({ term, subject, state }) => { - const trees = Object.values(await loadTrees()); - const done = completedCourses(trees); - const now = inProgressCourses(trees); - - // Listed from the term, judged on everything we hold. A requisite names - // courses nobody is teaching this term, and asking a one-term graph - // whether they exist gets the answer "no" and calls the course open. - const offered = graphFor(term).graph; - const { graph, aliases } = planningContext(); - const rows: string[] = []; - for (const [code, listed] of [...offered.courses].sort()) { - if (subject && !code.toUpperCase().startsWith(`${subject.toUpperCase()}-`)) continue; - const node = graph.courses.get(code) ?? listed; - const verdict = eligibility(node, done, now, { - exists: (c) => graph.courses.has(c), - aliases, - }); - if (state !== "all" && verdict.state !== state) continue; - rows.push( - `${code.padEnd(11)} ${verdict.state.padEnd(8)} ` + - (verdict.state === "blocked" - ? `needs ${verdict.blockedBy.join(", ")}` - : verdict.state === "unknown" - ? verdict.why.join(" ").slice(0, 80) - : `unlocks ${downstream(graph, code).size}`), - ); - } - if (rows.length === 0) return text("Nothing matches."); - return text( - [`${done.size} completed, ${now.size} in progress`, ...rows.slice(0, 200)].join("\n"), - ); - }, - ); - - server.registerTool( - "plan_terms", - { - description: - "Project which term each remaining requirement lands in, respecting prerequisites, seasons and a credit cap. Answers 'when do I graduate' rather than 'how many credits'.", - inputSchema: z.object({ - program: z - .string() - .optional() - .describe("Program code. Defaults to the first captured evaluation."), - credits_per_term: z - .number() - .int() - .min(6) - .max(21) - .optional() - .describe("Defaults to the load set in the planner, or 15."), - summers: z - .number() - .int() - .min(0) - .max(4) - .optional() - .describe("How many summers to plan, earliest first. Zero plans none."), - keep_semesters_full: z - .boolean() - .optional() - .describe( - "Hold work back from a summer rather than leave the semester behind it part time.", - ), - start: z - .string() - .optional() - .describe( - 'First term to plan, e.g. "SP27". Defaults to the term after the one under way.', - ), - }), - }, - async ({ program, credits_per_term, summers, keep_semesters_full, start }) => { - const trees = await loadTrees(); - // Every captured program by default, solved as one cover. Planning the - // first alone buys a shared requirement twice and drops the rest of a - // second major, which is the whole question a dual major is asking. - const chosen = (program ? [trees[program]] : Object.values(trees)).filter( - (t): t is ProgramTree => Boolean(t), - ); - if (chosen.length === 0) - return fail( - `No captured evaluation${program ? ` for ${program}` : ""}. Have: ${Object.keys(trees).join(", ") || "none"}`, - ); - - const { graph, credits, offeredIn, titles, aliases } = planningContext(); - const have = new Set([...completedCourses(chosen), ...inProgressCourses(chosen)]); - const pursuing = new Set(chosen.flatMap((t) => [...t.majors, ...t.minors])); - - // The degree the student chose, when they have handed their choices - // over, and the cheapest one when they have not. Both are real answers; - // reporting which is which is the part that matters. - const picks = await readPicks(); - const load = picks?.load; - const perTerm = credits_per_term ?? load?.perTerm ?? 15; - const useSummers = summers ?? load?.summers ?? 4; - const keepFull = keep_semesters_full ?? load?.fullSemesters ?? true; - - const solved = coursesNeededAcross(chosen, { - credits, - have, - pursuing, - pinned: new Set(picks?.pinned ?? []), - tracks: new Map(Object.entries(picks?.tracks ?? {}).map(([k, v]) => [k, [v]])), - }); - const from = start - ? { - season: start.startsWith("SP") ? ("spring" as const) : ("fall" as const), - year: 2000 + Number(start.slice(2)), - } - : nextPlannableTerm(new Date()); - - const plan = projectPlan({ - need: solved.courses, - completed: have, - // Standing is measured on the transcript, not on what the pools we can - // price happen to add up to. - earnedCredits: Math.max(...chosen.map((t) => t.credits.completed + t.credits.inProgress)), - graph, - credits, - offeredIn, - aliases, - keepSemestersFull: keepFull, - slots: termsFrom(from, 12, { - capacity: perTerm, - summerCapacity: load?.summer ?? 7, - summers: useSummers, - minimum: 12, - }), - }); - - // Two majors on one bachelor's share a single credit total, so the - // requirement is the largest of them and never their sum. - const largest = (pick: (t: ProgramTree) => number) => Math.max(...chosen.map(pick)); - const toGo = - largest((t) => t.credits.minimum) - - largest((t) => t.credits.completed) - - largest((t) => t.credits.inProgress); - const lines = [ - `${chosen.map((t) => t.code).join(" + ")}: ${toGo} credits remain; ` + - `this plan schedules ${plan.totalCredits} across named requirements.`, - `Finishes ${plan.finishes ?? "never within the horizon"} at ${perTerm}/term` + - `${useSummers ? ` with ${useSummers} summer${useSummers === 1 ? "" : "s"}` : " without summers"}.`, - "", - ]; - for (const term of plan.terms) { - lines.push(`${term.slot.name} (${term.credits} cr)`); - for (const c of term.courses) { - lines.push( - ` ${c.code.padEnd(11)} ${String(c.credits).padStart(2)} ${titles.get(c.code) ?? ""}${c.caution ? " [verify: unparsed condition]" : ""}`, - ); - } - } - if (plan.unscheduled.length) { - lines.push("", "not placed:"); - for (const u of plan.unscheduled) lines.push(` ${u.code.padEnd(11)} ${u.why}`); - } - const { unenumerable } = solved; - if (unenumerable.length) { - lines.push("", "not plannable — Colleague does not publish the eligible courses:"); - for (const u of unenumerable) { - lines.push(` ${u.credits ? `${u.credits}cr` : " "} ${u.text.slice(0, 70)}`); - } - } - lines.push( - "", - picks - ? `Planned with your own choices: ${picks.pinned?.length ?? 0} courses pinned, ` + - `${Object.keys(picks.tracks ?? {}).length} tracks settled.` - : "This is the cheapest degree rather than the one you have chosen — press " + - '"copy my plan" in the planner to hand your pins and tracks over.', - ); - return text(lines.join("\n")); - }, - ); - - server.registerTool( - "critical_path", - { - description: - "The longest chain of prerequisites still ahead. This is the floor on how many terms remain, and no credit load shortens it.", - inputSchema: z.object({ - program: z.string().optional().describe("Defaults to the first captured evaluation."), - }), - }, - async ({ program }) => { - const trees = await loadTrees(); - const chosen = (program ? [trees[program]] : Object.values(trees)).filter( - (t): t is ProgramTree => Boolean(t), - ); - if (chosen.length === 0) - return fail( - `No captured evaluation${program ? ` for ${program}` : ""}. Have: ${Object.keys(trees).join(", ") || "none"}`, - ); - - const { graph, credits, titles } = planningContext(); - const have = new Set([...completedCourses(chosen), ...inProgressCourses(chosen)]); - const path = criticalPath( - graph, - coursesNeededAcross(chosen, { credits, have }).courses, - have, - ); - if (path.length === 0) return text("Nothing left with prerequisites."); - - return text( - [ - `${path.length} terms minimum, whatever the credit load:`, - "", - ...path.map((c, i) => ` ${i + 1}. ${c.padEnd(11)} ${titles.get(c) ?? ""}`), - ].join("\n"), - ); - }, - ); - - server.registerTool( - "my_record", - { - description: - "What the registrar holds: the majors and minors your enrolment covers, credits and class standing, and any substitution an advisor granted by hand. Reads a local file; nothing is fetched.", - inputSchema: z.object({}), - }, - async () => { - const trees = Object.values(await loadTrees()); - const out: string[] = []; - - for (const tree of trees) { - const held = tree.credits.completed + tree.credits.inProgress; - const standing = - held >= STANDING_CREDITS.senior - ? "senior" - : held >= STANDING_CREDITS.junior - ? "junior" - : held >= STANDING_CREDITS.sophomore - ? "sophomore" - : "freshman"; - const next = ["sophomore", "junior", "senior"].find( - (name) => held < STANDING_CREDITS[name as keyof typeof STANDING_CREDITS], - ); - - out.push( - `${tree.code} ${tree.title} catalog ${tree.catalog}`, - ` covers: ${[...tree.majors, ...tree.minors].join(", ") || tree.title}`, - ` ${tree.credits.completed} completed, ${tree.credits.inProgress} in progress of ${tree.credits.minimum}`, - ` ${standing}` + - (next - ? `, ${STANDING_CREDITS[next as keyof typeof STANDING_CREDITS] - held} credits to ${next}` - : ""), - ); - - // Colleague applies these and then goes on listing the replaced - // course, so the only place they exist is this prose. - const applied = new Set( - tree.requirements.flatMap((r) => - r.subrequirements.flatMap((s) => - s.groups.flatMap((g) => g.applied.map((a) => a.CourseName)), - ), - ), - ); - for (const swap of substitutionsIn(tree)) { - const asking = stillRequiring(trees, swap.waives, applied).filter( - (r) => r.requirement !== swap.requirement, - ); - out.push( - ` substitution: ${swap.taken} replaces ${swap.waives} for ${swap.requirement}` + - (asking.length - ? ` — but ${asking.map((r) => r.requirement).join(" and ")} still lists it` - : ""), - ); - } - for (const unread of unreadModifications(tree)) { - out.push(` modification (unparsed): ${unread.text}`); - } - out.push(""); - } - return text(out.join("\n").trimEnd()); - }, - ); - - server.registerTool( - "compare_programs", - { - description: - "Where two captured programs overlap: which requirement in one draws on the same courses as a requirement in the other.", - inputSchema: z.object({ - a: z.string().describe('A program code or a credential, e.g. "Computer Science".'), - b: z.string(), - }), - }, - async ({ a, b }) => { - const trees = await loadTrees(); - // A second major recorded against the first one's program is one - // evaluation covering two credentials, so the parts of a tree are - // comparable to each other and to any other captured program. - const parts: Record = { ...trees }; - for (const tree of Object.values(trees)) { - for (const part of splitByCredential(tree)) parts[part.code] = part; - } - - const left = parts[a]; - const right = parts[b]; - if (!left || !right) { - return fail(`Nothing captured for that. Have: ${Object.keys(parts).join(", ")}`); - } - - const result = merge(left, right); - const out = [ - `${a} + ${b}: ${result.shared.length} shared requirement pairs, ${result.certainSharedCourses.length} distinct courses`, - ]; - for (const pool of result.shared) { - out.push( - `[${pool.significance}] ${(pool.a.group.text || pool.a.requirement).slice(0, 60)}`, - ` <-> ${(pool.b.group.text || pool.b.requirement).slice(0, 60)}`, - ` ${pool.courses.map((c) => c.CourseName).join(" ")}`, - ); - } - if (result.unresolved.length) { - out.push("", "cannot be compared automatically:"); - for (const u of result.unresolved) { - out.push(` [${u.reason}] ${u.at.program} ${u.at.group.text || u.at.group.code}`); - } - } - return text(out.join("\n")); - }, - ); -} - -// ---- start ------------------------------------------------------------- - -const personal = process.argv.includes("--personal") || process.env.CEDARVILLE_MCP_PERSONAL === "1"; - -// The intake belongs to the personal half and opens with it: a server nobody -// opted into has no business holding a port open for a transcript. Logged to -// stderr, because stdout is the protocol. -// `CEDARVILLE_COMPANION=0` turns it off: a listener is a reasonable thing to -// decline, and a test that spawns this server must not hold a real port or -// write to a real home directory. -if (personal && process.env.CEDARVILLE_COMPANION !== "0") { - const companion = serveCompanion((path) => - console.error(`companion: capture written to ${path}`), - ); - console.error( - companion - ? `companion: listening on 127.0.0.1:${companion.port} for the extension` - : "companion: another one already holds the port; reading its file", - ); -} - -serveStdio(() => { - const server = new McpServer({ name: "cedarville", version: "0.1.0" }); - registerCatalog(server); - if (personal) registerPersonal(server); - return server; -}); diff --git a/src/requirements.test.ts b/src/requirements.test.ts index f815458..6d46ff7 100644 --- a/src/requirements.test.ts +++ b/src/requirements.test.ts @@ -529,7 +529,10 @@ describe("what a student still owes", () => { const credits = (c: string) => (baseCode(c) === "HON-4950" ? 2 : 1); const pick = (pinned: string[]) => - [...coursesNeeded(normalize(raw), { credits, have: new Set(), pinned: new Set(pinned) }).courses].sort(); + [ + ...coursesNeeded(normalize(raw), { credits, have: new Set(), pinned: new Set(pinned) }) + .courses, + ].sort(); // Cost alone would take the colloquium. expect(pick([])).toEqual(["HON-4910", "HON-4920"]); diff --git a/src/server/colleague.ts b/src/server/colleague.ts index a29c092..879efca 100644 --- a/src/server/colleague.ts +++ b/src/server/colleague.ts @@ -11,6 +11,8 @@ * file is allowed to exist on a server. */ +import { crawlGroup, type SearchCriteria, type Searcher, type SearchPage } from "../crawl"; + const ORIGIN = "https://selfservice.cedarville.edu"; const TOKEN_PAGE = "/Student/Courses/Search"; const TOKEN_RE = /name="__RequestVerificationToken"[^>]*value="([^"]+)"/; @@ -18,28 +20,9 @@ const TOKEN_RE = /name="__RequestVerificationToken"[^>]*value="([^"]+)"/; /** Ellucian ships this exact malformed content type. Mirroring it. */ const JSON_CT = "application/json, charset=UTF-8"; -export interface SearchCriteria { - terms?: string[]; - subjects?: string[]; - courseIds?: string[]; - /** Degree-audit coordinates: asks Colleague to evaluate that group's rule. */ - requirement?: string; - subrequirement?: string; - group?: string; - keyword?: string; - pageNumber?: number; - quantityPerPage?: number; - searchResultsView?: "SectionListing" | "CatalogListing"; -} - -export interface SearchPage { - Sections?: unknown[]; - Courses?: unknown[]; - CourseFullModels?: unknown[]; - TotalItems: number; - TotalPages: number; - CurrentPageIndex: number; -} +// Defined with the crawl loop rather than here, because the planner's own +// client has to satisfy the same shape and cannot import from `server/`. +export type { SearchCriteria, SearchPage } from "../crawl"; export interface Vocabulary { Subjects: { Code: string; Description: string; ShowInCourseSearch: boolean }[]; @@ -51,7 +34,7 @@ export interface Vocabulary { * no cookie jar, so the pairing is kept by hand: a token is only valid * alongside the .ColleagueSelfServiceAntiforgery cookie minted with it. */ -export class GuestColleague { +export class GuestColleague implements Searcher { #cookies = ""; #token = ""; @@ -153,29 +136,7 @@ const USER_AGENT = "the-cedarville-app (student course planner; github.com/tacit * No session is needed. The triple names a place in the catalog, not a * student, so one lookup serves everybody. */ -export async function resolveGroup( +export const resolveGroup = ( ids: { requirement: string; subrequirement: string; group: string }, - client = new GuestColleague(), -): Promise { - const names = new Set(); - let page = 1; - let pages = 1; - - while (page <= pages) { - const result = await client.search({ - requirement: ids.requirement, - subrequirement: ids.subrequirement, - group: ids.group, - pageNumber: page, - searchResultsView: "CatalogListing", - }); - pages = Math.max(result.TotalPages ?? 1, 1); - - for (const raw of result.Courses ?? []) { - const course = raw as { SubjectCode?: string; Number?: string }; - if (course.SubjectCode && course.Number) names.add(`${course.SubjectCode}-${course.Number}`); - } - page++; - } - return [...names].sort(); -} + client: Searcher = new GuestColleague(), +): Promise => crawlGroup(client, ids); diff --git a/src/server/crawler.ts b/src/server/crawler.ts index 02b5476..fd9e6e7 100644 --- a/src/server/crawler.ts +++ b/src/server/crawler.ts @@ -1,111 +1,55 @@ /** - * Fetching a whole term, once, for everybody. + * The server's side of the crawl: the shared loop, bound to a guest session + * and to the SQLite cache. * - * A single search in SectionListing view returns sections directly, so Fall - * 2026 is about sixty pages rather than one request per course. Doing that - * here instead of in each student's browser is both faster for them and far - * less traffic for the registrar: one crawl serves every user of the app. + * The loop itself lives in `src/crawl.ts`, because the planner runs the same + * one in the browser now that Cedarville has put the catalog behind SSO. + * What is left here is the part that is genuinely server-only: supplying a + * guest client, and writing what comes back into the store. + * + * Every function here still works, and every one of them fails the moment the + * guest endpoint is gone. That is not a bug to paper over: a server that can + * no longer read the catalog should say so, and the ingest route is how the + * catalog gets filled instead. */ -import type { CatalogCourseRecord, ListingSection, TermCatalog } from "../catalog"; +import type { CatalogCourseRecord, TermCatalog } from "../catalog"; +import { + ALL_COURSES, + type CrawlOptions, + type CrawlProgress, + crawlAllCourses as crawlAll, + crawlSeats, + crawlTerm as crawlTermWith, + dedupeByCode, + type Searcher, + type Seats, +} from "../crawl"; import { GuestColleague } from "./colleague"; import type { CatalogStore } from "./store"; -export interface CrawlProgress { - term: string; - page: number; - pages: number; - sections: number; - phase?: "sections" | "courses"; -} - -export interface CrawlOptions { - /** Gap between pages. This is a registrar, not a load test. */ - delayMs?: number; - onProgress?: (progress: CrawlProgress) => void; - signal?: AbortSignal; -} +export type { CrawlOptions, CrawlProgress, Searcher, Seats }; +export { ALL_COURSES, dedupeByCode }; -const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); - -export async function crawlTerm( +export const crawlTerm = ( term: string, options: CrawlOptions = {}, - client = new GuestColleague(), -): Promise { - const { delayMs = 300, onProgress, signal } = options; - const sections: ListingSection[] = []; - const seen = new Set(); - - // Colleague caps the page size at its own value regardless of what we ask, - // so paging is driven by the TotalPages it reports back. - let page = 1; - let pages = 1; - - while (page <= pages) { - if (signal?.aborted) break; + client: Searcher = new GuestColleague(), +): Promise => crawlTermWith(client, term, options); - const result = await client.search({ terms: [term], pageNumber: page }); - pages = Math.max(result.TotalPages ?? 1, 1); - - for (const raw of result.Sections ?? []) { - const section = raw as ListingSection; - // A section shifting between pages mid-crawl must not double up. - if (!section?.Id || seen.has(section.Id)) continue; - seen.add(section.Id); - sections.push(section); - } - - onProgress?.({ term, page, pages, sections: sections.length }); - page++; - if (page <= pages && delayMs > 0) await sleep(delayMs); - } - - return { - term, - fetchedAt: new Date().toISOString(), - sections, - courses: await crawlCourses(term, options, client), - }; -} - -/** - * The same term again in catalog view, which is the only place requisites - * come back as readable text rather than an opaque rule id. Half the pages of - * the section crawl, and it is what makes "what blocks what" answerable. - */ -export async function crawlCourses( - term: string, +export const crawlAllCourses = async ( options: CrawlOptions = {}, - client = new GuestColleague(), -): Promise { - const { delayMs = 300, onProgress, signal } = options; - const byId = new Map(); - - let page = 1; - let pages = 1; - - while (page <= pages) { - if (signal?.aborted) break; - - const result = await client.search({ - terms: [term], - pageNumber: page, - searchResultsView: "CatalogListing", - }); - pages = Math.max(result.TotalPages ?? 1, 1); + client: Searcher = new GuestColleague(), +): Promise => (await crawlAll(client, options)).courses; - for (const raw of result.CourseFullModels ?? []) { - const course = raw as CatalogCourseRecord; - if (course?.Id) byId.set(course.Id, course); - } +export const liveSeats = ( + term: string, + courseIds: string[], + client: Searcher = new GuestColleague(), +): Promise> => crawlSeats(client, term, courseIds); - onProgress?.({ term, page, pages, sections: byId.size, phase: "courses" }); - page++; - if (page <= pages && delayMs > 0) await sleep(delayMs); - } - return [...byId.values()]; -} +/** Every term Colleague currently lists as searchable. */ +export const availableTerms = (client = new GuestColleague()) => client.terms(); /** Crawls and stores in one step. Returns how many sections landed. */ export async function refreshTerm( @@ -115,127 +59,13 @@ export async function refreshTerm( ): Promise { const catalog = await crawlTerm(term, options); // An empty crawl means something went wrong upstream; keeping the previous - // catalog beats replacing a working timetable with nothing. - if (catalog.sections.length === 0) return 0; + // catalog beats replacing a working timetable with nothing. An incomplete + // one is worse than empty, because it looks like a term that lost half its + // sections, and `replace` would delete the rest. + if (catalog.sections.length === 0 || !catalog.complete) return 0; return store.replace(catalog); } -/** Every term Colleague currently lists as searchable. */ -export const availableTerms = (client = new GuestColleague()) => client.terms(); - -/** - * Current availability for a handful of courses. - * - * Seat counts are the one field that moves by the minute during registration, - * so serving them from a six-hour-old crawl makes the number decorative at - * exactly the moment it matters. This asks Colleague directly, scoped to the - * courses a student is actually looking at: one request, under a second. - */ -export async function liveSeats( - term: string, - courseIds: string[], - client = new GuestColleague(), -): Promise> { - if (courseIds.length === 0) return {}; - - const seats: Record = {}; - let page = 1; - let pages = 1; - - while (page <= pages) { - const result = await client.search({ terms: [term], courseIds, pageNumber: page }); - pages = Math.max(result.TotalPages ?? 1, 1); - - for (const raw of result.Sections ?? []) { - const s = raw as ListingSection; - if (!s?.Id) continue; - seats[s.Id] = { - available: s.Available, - capacity: s.Capacity, - enrolled: s.Enrolled, - waitlisted: s.Waitlisted, - status: s.AvailabilityStatus, - }; - } - page++; - } - return seats; -} - -export interface Seats { - available: number; - capacity: number; - enrolled: number; - waitlisted: number; - status: string; -} - -/** - * Every course in the catalog, whether or not it runs this year. - * - * The per-term crawl answers "what is offered"; this answers "what exists". - * They have to be separate, because a prerequisite is often a course nobody - * is teaching this year — EGEE-2010 roots a four-course engineering chain and - * appears in neither cached term. Built from term-scoped data alone, the - * graph loses 36% of its prerequisite nodes and silently understates depth on - * a third of its courses. - */ -export const ALL_COURSES = "ALL"; - -export async function crawlAllCourses( - options: CrawlOptions = {}, - client = new GuestColleague(), -): Promise { - const { delayMs = 300, onProgress, signal } = options; - const byId = new Map(); - - let page = 1; - let pages = 1; - - while (page <= pages) { - if (signal?.aborted) break; - - // No term filter: the whole catalog rather than one term's offerings. - const result = await client.search({ pageNumber: page, searchResultsView: "CatalogListing" }); - pages = Math.max(result.TotalPages ?? 1, 1); - - for (const raw of result.CourseFullModels ?? []) { - const course = raw as CatalogCourseRecord; - if (course?.Id) byId.set(course.Id, course); - } - - onProgress?.({ term: ALL_COURSES, page, pages, sections: byId.size, phase: "courses" }); - page++; - if (page <= pages && delayMs > 0) await sleep(delayMs); - } - return dedupeByCode([...byId.values()]); -} - -/** - * One record per course code, chosen deliberately. - * - * About 1% of codes carry two records: a course being retired beside its - * replacement, both live during the transition ("Prin Accounting I" and - * "Financial Accounting" are both ACCT-2110 this term). Colleague tells them - * apart by id and picks per the student's catalog year; requisite text only - * ever names a code, so a graph keyed by code has to choose. Choosing by - * whichever the crawl happened to see last is not a choice at all — prefer the - * one being taught, then the one that states requisites. - */ -export function dedupeByCode(records: CatalogCourseRecord[]): CatalogCourseRecord[] { - const best = new Map(); - const score = (c: CatalogCourseRecord) => - ((c as { MatchingSectionIds?: string[] }).MatchingSectionIds?.length ?? 0) * 10 + - (c.CourseRequisites?.length ?? 0); - - for (const course of records) { - const code = `${course.SubjectCode}-${course.Number}`; - const held = best.get(code); - if (!held || score(course) > score(held)) best.set(code, course); - } - return [...best.values()]; -} - /** Crawls the full catalog and stores it under the ALL sentinel. */ export async function refreshAllCourses( store: CatalogStore, diff --git a/src/server/ingest.test.ts b/src/server/ingest.test.ts new file mode 100644 index 0000000..46c6fc6 --- /dev/null +++ b/src/server/ingest.test.ts @@ -0,0 +1,210 @@ +import { describe, expect, test } from "bun:test"; +import { ingest } from "./ingest"; +import { CatalogStore } from "./store"; + +/** ":memory:" keeps each test's database to itself. */ +const store = () => new CatalogStore(":memory:"); + +const section = (id: string, over: Record = {}) => ({ + Id: id, + CourseId: `c-${id}`, + CourseName: "ACCT-2110", + TermId: "2026FA", + MinimumCredits: 3, + Capacity: 30, + Enrolled: 10, + Available: 20, + Meetings: [], + ...over, +}); + +const body = (over: Record = {}) => ({ + term: "2026FA", + complete: true, + sections: [section("s1"), section("s2")], + courses: [{ Id: "c-s1", SubjectCode: "ACCT", Number: "2110" }], + ...over, +}); + +/** Fills a term so the shrink guard has something to protect. */ +function held(db: CatalogStore, count: number) { + db.replace({ + term: "2026FA", + fetchedAt: "2026-08-12T00:00:00.000Z", + sections: Array.from({ length: count }, (_, i) => section(`held-${i}`)) as never, + }); +} + +describe("a crawl the server can use", () => { + test("accepts a complete crawl of a term nobody has filled", () => { + const db = store(); + const verdict = ingest(db, body()); + + expect(verdict).toMatchObject({ ok: true, sections: 2, courses: 1, replaced: false }); + expect(db.read("2026FA").sections).toHaveLength(2); + db.close(); + }); + + test("says when it replaced data rather than filling a gap", () => { + const db = store(); + held(db, 2); + expect(ingest(db, body())).toMatchObject({ ok: true, replaced: true }); + db.close(); + }); + + test("carries the whole section through, not just the validated fields", () => { + // The schema names nine fields; a section holds dozens, and the schedule + // is built from the meeting times that are not named here. + const db = store(); + ingest( + db, + body({ + sections: [section("s1", { FormattedMeetingTimes: [{ Days: "MWF" }], Synonym: "12345" })], + }), + ); + + const [stored] = db.read("2026FA").sections; + expect(stored).toMatchObject({ Synonym: "12345", FormattedMeetingTimes: [{ Days: "MWF" }] }); + db.close(); + }); +}); + +describe("shape", () => { + test("refuses a batch for one bad section rather than skipping it", () => { + // A skipped section is indistinguishable from a cancelled one, and + // `replace` deletes what the crawl did not report. + const db = store(); + const verdict = ingest(db, body({ sections: [section("s1"), { Id: "s2" }] })); + + expect(verdict.ok).toBe(false); + expect(db.read("2026FA").sections).toHaveLength(0); + db.close(); + }); + + test("names the field that was wrong", () => { + const db = store(); + const verdict = ingest(db, body({ sections: [section("s1", { Capacity: "lots" })] })); + + expect(verdict).toMatchObject({ ok: false }); + if (!verdict.ok) expect(verdict.why).toContain("Capacity"); + db.close(); + }); + + test("a section with no id is refused, because the id is the key", () => { + const db = store(); + expect(ingest(db, body({ sections: [section("")] })).ok).toBe(false); + db.close(); + }); + + test("unknown fields are allowed through, so an upstream addition is not an outage", () => { + const db = store(); + const verdict = ingest( + db, + body({ sections: [section("s1", { SomethingEllucianAddedLastTuesday: true })] }), + ); + expect(verdict.ok).toBe(true); + db.close(); + }); + + test("refuses something that is not a crawl at all", () => { + const db = store(); + for (const junk of [null, 42, "2026FA", [], {}]) { + expect(ingest(db, junk).ok).toBe(false); + } + db.close(); + }); +}); + +describe("completeness", () => { + test("refuses a crawl that stopped early", () => { + const db = store(); + const verdict = ingest(db, body({ complete: false })); + + expect(verdict).toMatchObject({ ok: false }); + if (!verdict.ok) expect(verdict.why).toContain("partial"); + db.close(); + }); + + test("an incomplete crawl leaves a filled term exactly as it was", () => { + // The failure this guard exists for: a student closes the tab halfway, + // and the term loses the sections the crawl never reached. + const db = store(); + held(db, 40); + ingest(db, body({ complete: false })); + expect(db.read("2026FA").sections).toHaveLength(40); + db.close(); + }); + + test("refuses an empty crawl, which is a failure rather than an empty term", () => { + const db = store(); + expect(ingest(db, body({ sections: [] })).ok).toBe(false); + db.close(); + }); +}); + +describe("no shrinking", () => { + test("refuses a crawl that would gut a term", () => { + // Without this guard, three well-formed sections and complete: true + // empty the catalog for everybody. + const db = store(); + held(db, 100); + + const verdict = ingest(db, body({ sections: [section("s1"), section("s2")] })); + + expect(verdict).toMatchObject({ ok: false }); + if (!verdict.ok) expect(verdict.why).toContain("shrink"); + expect(db.read("2026FA").sections).toHaveLength(100); + db.close(); + }); + + test("allows the ordinary churn of late registration", () => { + const db = store(); + held(db, 100); + + // 90 of 100: sections do get cancelled, and refusing that would refuse + // every honest refresh. + const sections = Array.from({ length: 90 }, (_, i) => section(`s${i}`)); + expect(ingest(db, body({ sections })).ok).toBe(true); + expect(db.read("2026FA").sections).toHaveLength(90); + db.close(); + }); + + test("a term growing is never suspicious", () => { + const db = store(); + held(db, 10); + const sections = Array.from({ length: 200 }, (_, i) => section(`s${i}`)); + expect(ingest(db, body({ sections })).ok).toBe(true); + db.close(); + }); +}); + +describe("a crawl must be about the term it claims", () => { + test("refuses sections belonging to another term", () => { + // Mixing terms would write Fall sections into the Spring cache, and the + // shrink guard counts rows rather than reading them. + const db = store(); + const verdict = ingest( + db, + body({ sections: [section("s1"), section("s2", { TermId: "2027SP" })] }), + ); + + expect(verdict).toMatchObject({ ok: false }); + if (!verdict.ok) expect(verdict.why).toContain("2027SP"); + db.close(); + }); + + test("refuses the sentinel the whole catalog lives under", () => { + const db = store(); + const verdict = ingest(db, body({ term: "ALL", sections: [section("s1", { TermId: "ALL" })] })); + + expect(verdict).toMatchObject({ ok: false }); + if (!verdict.ok) expect(verdict.why).toContain("not a term"); + db.close(); + }); + + test("refuses a term code long enough to be something else", () => { + const db = store(); + expect(ingest(db, body({ term: "x".repeat(64) })).ok).toBe(false); + db.close(); + }); +}); diff --git a/src/server/ingest.ts b/src/server/ingest.ts new file mode 100644 index 0000000..352741b --- /dev/null +++ b/src/server/ingest.ts @@ -0,0 +1,157 @@ +/** + * Taking a crawl from a student, for everybody. + * + * The catalog used to be readable without a session, so the server crawled it + * and trusted itself. Cedarville put it behind SSO, and the only thing left + * with a session is a student's browser. So the crawl moved there, and the + * shared cache now has to accept data from the people using it. + * + * That is a different problem from fetching it. A crawl the server performed + * was true by construction; a crawl posted to the server is a claim. The + * cache is read by every other student, so a bad claim is not one person's + * problem. + * + * Three guards, in the order a bad payload meets them: + * + * shape every section is checked field by field, and one bad + * section refuses the batch rather than being skipped. A + * skipped section is indistinguishable from a cancelled one. + * + * completeness the client says whether it paged to the end, and only a + * complete crawl may replace a term. `store.replace` deletes + * what the crawl did not see, so an interrupted crawl would + * cancel half a term's sections. + * + * no shrinking a crawl that would remove a large share of a term it did + * not fill is refused. This is the one that makes the route + * safe to leave open: without it, three well-formed sections + * and `complete: true` empty the catalog for everyone. + * + * What is deliberately not here is identity. There is no account system and + * nothing to attach one to, so the guards are all about the claim rather than + * the claimant. A determined student with a real session can still post a + * plausible lie about the catalog; what they cannot do is quietly delete it. + */ + +import { z } from "zod"; +import type { TermCatalog } from "../catalog"; +import type { CatalogStore } from "./store"; + +/** + * The fields the planner actually reads, required, and everything else + * allowed through untouched. + * + * Strict on what is used and loose on the rest, deliberately: Ellucian adds + * fields between releases, and a schema that rejects unknown keys would turn + * every upstream addition into an outage. A missing `Id` is a different + * matter, because it is the primary key. + */ +const Section = z + .object({ + Id: z.string().min(1), + CourseId: z.string().min(1), + CourseName: z.string(), + TermId: z.string().min(1), + MinimumCredits: z.number(), + Capacity: z.number(), + Enrolled: z.number(), + Available: z.number(), + Meetings: z.array(z.unknown()), + }) + .passthrough(); + +const Course = z + .object({ + Id: z.string().min(1), + SubjectCode: z.string().min(1), + Number: z.string().min(1), + }) + .passthrough(); + +export const Ingest = z.object({ + term: z.string().min(1).max(32), + /** Whether the crawl paged all the way to the end. Only a complete one may replace. */ + complete: z.boolean(), + sections: z.array(Section).max(20_000), + courses: z.array(Course).max(20_000).optional(), +}); + +export type IngestBody = z.infer; + +export type Verdict = + | { ok: true; sections: number; courses: number; replaced: boolean } + | { ok: false; why: string }; + +/** + * How much of a term a crawl is allowed to remove. + * + * Sections genuinely disappear between crawls, so demanding a crawl never + * shrink a term would refuse every honest refresh late in registration. A + * fifth is far more than the churn we have measured and far less than the + * loss a truncated or malicious crawl would cause. + */ +const SHRINK_LIMIT = 0.2; + +/** + * The term under which the whole catalog lives, which never arrives by + * ingest: it is 900-odd courses with no sections, and the shrink guard + * counts sections. Refused by name rather than by accident. + */ +const SENTINELS = new Set(["ALL"]); + +export function ingest(store: CatalogStore, raw: unknown): Verdict { + const parsed = Ingest.safeParse(raw); + if (!parsed.success) { + const first = parsed.error.issues[0]; + return { ok: false, why: `malformed crawl: ${first?.path.join(".")} ${first?.message}` }; + } + const body = parsed.data; + + if (SENTINELS.has(body.term)) { + return { ok: false, why: `${body.term} is not a term and cannot be ingested` }; + } + if (!body.complete) { + return { ok: false, why: "crawl did not reach the last page; a partial term is not a term" }; + } + if (body.sections.length === 0) { + return { ok: false, why: "a term with no sections is a failed crawl, not an empty term" }; + } + + // Every section must claim the term it was posted under. A crawl that + // mixes terms would write Fall sections into the Spring cache, and the + // shrink guard counts rows rather than reading them. + const astray = body.sections.find((s) => s.TermId !== body.term); + if (astray) { + return { ok: false, why: `section ${astray.Id} belongs to ${astray.TermId}, not ${body.term}` }; + } + + const held = store.read(body.term).sections.length; + const floor = Math.floor(held * (1 - SHRINK_LIMIT)); + if (body.sections.length < floor) { + return { + ok: false, + why: + `refusing to shrink ${body.term} from ${held} sections to ${body.sections.length}; ` + + `a crawl may lose up to ${Math.round(SHRINK_LIMIT * 100)}% of a term, not more`, + }; + } + + const catalog: TermCatalog = { + term: body.term, + fetchedAt: new Date().toISOString(), + // Validated above for the fields the planner reads, and carried through + // whole: Colleague's own records hold far more than this file names, and + // narrowing them to the validated subset would throw away the meeting + // times the schedule is built from. + sections: body.sections as unknown as TermCatalog["sections"], + courses: body.courses as unknown as TermCatalog["courses"], + }; + const sections = store.replace(catalog); + + return { + ok: true, + sections, + courses: body.courses?.length ?? 0, + replaced: held > 0, + }; +} diff --git a/src/types.ts b/src/types.ts index 4000513..5c2059c 100644 --- a/src/types.ts +++ b/src/types.ts @@ -204,37 +204,11 @@ export interface DegreePlanResponse { StudentPrograms: { Code: string; Title: string; Catalog: string; StudentId: string }[]; } -export interface SearchCriteria { - keyword: string; - terms: string[]; - subjects: string[]; - courseIds: string[]; - sectionIds: string[]; - days: string[]; - faculty: string[]; - locations: string[]; - academicLevels: string[]; - courseLevels: string[]; - /** Minutes since midnight. */ - startTime: number; - endTime: number; - openSections: boolean; - openAndWaitlistedSections: boolean; - pageNumber: number; - quantityPerPage: number; - searchResultsView: string; -} - -export interface SearchResponse { - Courses: (CourseRef & { MatchingSectionIds: string[]; MinimumCredits: number })[]; - TotalItems: number; - TotalPages: number; - CurrentPageIndex: number; - Subjects: Facet[]; - TermFilters: Facet[]; - Faculty: Facet[]; - DaysOfWeek: Facet[]; -} +// The search criteria and its response live in `src/crawl.ts`, beside the loop +// that pages them. Two copies of the shape lived here once, and the authenticated +// one went stale: it declared `Courses` and never the `Sections` that +// `SectionListing` view actually returns, which is the half the timetable +// is built from. export interface Meeting { /** Sometimes names ("Monday"), sometimes 0-6 integers, depending on endpoint. */