From a1bfafa5564e089e5a734ffb023aa91aa63df406 Mon Sep 17 00:00:00 2001 From: Patrick Wozniak Date: Wed, 2 Sep 2026 00:22:26 +0200 Subject: [PATCH] test(e2e): default the GOAT vision phase to gpt-5.6-luna google/gemini-3.7-flash currently fails on the Provider API with a zero-data-retention routing 404 for every request, which made the GOAT live suite red regardless of provider changes. GPT-5.6 Luna is available on every plan, accepts image input, and completes the vision request. --- README.md | 2 +- tests/test-live-e2e.mjs | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1d2633b..ef5db91 100644 --- a/README.md +++ b/README.md @@ -209,7 +209,7 @@ COMMANDCODE_E2E_GOAT_API_KEY_FILE=/path/to/goat-key \ Each profile runs with an isolated Pi agent directory and asserts transport selection, reasoning across turns, quota plan identity, abort handling, tool calls, and the packed npm artifact. Go must select `generate` and reject unsupported images; GOAT must select `provider` and complete a live vision request. The profile-specific `*_API_KEY` environment variables are also supported for CI secrets, but key files are preferred for local use. -The Go profile defaults to DeepSeek V4 Flash; GOAT defaults to Grok 4.6 because its Provider API stream exposes reasoning consistently across consecutive turns. Override them with `COMMANDCODE_E2E_GO_MODEL`, `COMMANDCODE_E2E_GOAT_MODEL`, or `COMMANDCODE_E2E_PROVIDER_MODEL`. A successful live Anthropic `/provider/v1/messages` test requires a paid account whose plan includes the selected Claude model. +The Go profile defaults to DeepSeek V4 Flash; GOAT defaults to Grok 4.6 because its Provider API stream exposes reasoning consistently across consecutive turns. Override them with `COMMANDCODE_E2E_GO_MODEL`, `COMMANDCODE_E2E_GOAT_MODEL`, or `COMMANDCODE_E2E_PROVIDER_MODEL`. The GOAT vision phase defaults to GPT-5.6 Luna and can be overridden with `COMMANDCODE_E2E_GOAT_VISION_MODEL`. A successful live Anthropic `/provider/v1/messages` test requires a paid account whose plan includes the selected Claude model. See [CONTRIBUTING.md](CONTRIBUTING.md) for local setup and tests. See [RELEASE.md](RELEASE.md) for the release process. diff --git a/tests/test-live-e2e.mjs b/tests/test-live-e2e.mjs index f17ffa5..7a91d88 100644 --- a/tests/test-live-e2e.mjs +++ b/tests/test-live-e2e.mjs @@ -40,7 +40,9 @@ const expectedPlan = : testProfile === "provider" ? "provider" : undefined -const goatVisionModel = process.env.COMMANDCODE_E2E_GOAT_VISION_MODEL ?? "google/gemini-3.7-flash" +// GPT-5.6 Luna is available on every plan and has a ZDR-capable upstream; +// Gemini 3.7 Flash currently fails with a zero-data-retention routing 404. +const goatVisionModel = process.env.COMMANDCODE_E2E_GOAT_VISION_MODEL ?? "gpt-5.6-luna" const marker = "commandcode-live-e2e-ok" function findPiBinary() {