Something went wrong. Try again.
End-to-end Tinker model training and serving with Eve.
Something went wrong. Try again.
6.0 kB · 90 lines
JSON
at main
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091[ { "id": "test-direct-00", "prompt": "Select context for an agent turn.\nQuestion: Which port is the `brisk_field` service currently listening on?\nCandidates:\n[ctx_brisk_field_a] General networking guidance recommends loopback for local services.\n[ctx_brisk_field_b] The latest runtime receipt says `brisk_field` is listening on port 4127.\n[ctx_brisk_field_c] The orchard inventory contains 41 trees.\n[ctx_brisk_field_d] A planning note proposed port 3000 before deployment.\nReturn the canonical selection.", "expected": { "include": [ "ctx_brisk_field_b" ] } }, { "id": "test-direct-01", "prompt": "Select context for an agent turn.\nQuestion: What port does the live `kindred_island` process use?\nCandidates:\n[ctx_kindred_island_a] General networking guidance recommends loopback for local services.\n[ctx_kindred_island_b] The latest runtime receipt says `kindred_island` is listening on port 4127.\n[ctx_kindred_island_c] A planning note proposed port 3000 before deployment.\n[ctx_kindred_island_d] The orchard inventory contains 41 trees.\nReturn the canonical selection.", "expected": { "include": [ "ctx_kindred_island_b" ] } }, { "id": "test-join-00", "prompt": "Select context for an agent turn.\nQuestion: Which model and renderer serve `brisk_junction` now?\nCandidates:\n[ctx_brisk_junction_a] The deployment region is unrelated to model rendering.\n[ctx_brisk_junction_b] The current `brisk_junction` renderer is qwen3_5_disable_thinking.\n[ctx_brisk_junction_c] An old experiment used a different base model.\n[ctx_brisk_junction_d] The active `brisk_junction` model is Qwen/Qwen3.5-4B.\nReturn the canonical selection.", "expected": { "include": [ "ctx_brisk_junction_b", "ctx_brisk_junction_d" ] } }, { "id": "test-join-01", "prompt": "Select context for an agent turn.\nQuestion: Identify both the active model and renderer for `ember_field`.\nCandidates:\n[ctx_ember_field_a] The current `ember_field` renderer is qwen3_5_disable_thinking.\n[ctx_ember_field_b] The deployment region is unrelated to model rendering.\n[ctx_ember_field_c] The active `ember_field` model is Qwen/Qwen3.5-4B.\n[ctx_ember_field_d] An old experiment used a different base model.\nReturn the canonical selection.", "expected": { "include": [ "ctx_ember_field_a", "ctx_ember_field_c" ] } }, { "id": "test-freshness-00", "prompt": "Select context for an agent turn.\nQuestion: What release of `kindred_current` is actually running?\nCandidates:\n[ctx_kindred_current_a] The latest process receipt proves `kindred_current` version 3 is running.\n[ctx_kindred_current_b] Version 4 is planned but has not been deployed.\n[ctx_kindred_current_c] A design document explains the versioning scheme.\n[ctx_kindred_current_d] A June snapshot recorded `kindred_current` version 1.\nReturn the canonical selection.", "expected": { "include": [ "ctx_kindred_current_a" ] } }, { "id": "test-freshness-01", "prompt": "Select context for an agent turn.\nQuestion: Which `kindred_junction` version has a live runtime receipt?\nCandidates:\n[ctx_kindred_junction_a] A design document explains the versioning scheme.\n[ctx_kindred_junction_b] Version 4 is planned but has not been deployed.\n[ctx_kindred_junction_c] A June snapshot recorded `kindred_junction` version 1.\n[ctx_kindred_junction_d] The latest process receipt proves `kindred_junction` version 3 is running.\nReturn the canonical selection.", "expected": { "include": [ "ctx_kindred_junction_d" ] } }, { "id": "test-scope-00", "prompt": "Select context for an agent turn.\nQuestion: Which timeout applies to the live `dappled_delta` consumer?\nCandidates:\n[ctx_dappled_delta_a] Project `dappled_delta` overrides its live consumer timeout to 25 seconds.\n[ctx_dappled_delta_b] Another project uses a 90-second consumer timeout.\n[ctx_dappled_delta_c] The organization-wide default timeout is 60 seconds.\n[ctx_dappled_delta_d] The `dappled_delta` unit-test timeout is 5 seconds.\nReturn the canonical selection.", "expected": { "include": [ "ctx_dappled_delta_a" ] } }, { "id": "test-scope-01", "prompt": "Select context for an agent turn.\nQuestion: Find the effective consumer timeout for project `lunar_grove`.\nCandidates:\n[ctx_lunar_grove_a] Project `lunar_grove` overrides its live consumer timeout to 25 seconds.\n[ctx_lunar_grove_b] The organization-wide default timeout is 60 seconds.\n[ctx_lunar_grove_c] The `lunar_grove` unit-test timeout is 5 seconds.\n[ctx_lunar_grove_d] Another project uses a 90-second consumer timeout.\nReturn the canonical selection.", "expected": { "include": [ "ctx_lunar_grove_a" ] } }, { "id": "test-insufficient-00", "prompt": "Select context for an agent turn.\nQuestion: Who approved production release `ember_island`?\nCandidates:\n[ctx_ember_island_a] A draft release checklist exists for `ember_island`.\n[ctx_ember_island_b] No approval receipt is present in the release record.\n[ctx_ember_island_c] The test suite passed for `ember_island`.\n[ctx_ember_island_d] A developer proposed shipping after review.\nReturn the canonical selection.", "expected": { "include": [] } }, { "id": "test-insufficient-01", "prompt": "Select context for an agent turn.\nQuestion: Identify the approver for the `dappled_field` production release.\nCandidates:\n[ctx_dappled_field_a] A developer proposed shipping after review.\n[ctx_dappled_field_b] No approval receipt is present in the release record.\n[ctx_dappled_field_c] The test suite passed for `dappled_field`.\n[ctx_dappled_field_d] A draft release checklist exists for `dappled_field`.\nReturn the canonical selection.", "expected": { "include": [] } }]