Executive assessment

Historical preliminary report — superseded for current status. This document preserves the repository/offline assessment produced before live access was available. The deployed assessment at https://opti-mcp.vercel.app/ and the source in lib/assessment.ts contain the current live-validation findings and SOW closeout status. Live access, the development key, the target MCP deployment, and representative production replays are now available; the remaining Phase 1 closeout is the complete agreed replay/adjudication, final consolidated editable/PDF report, and review session.

Purpose: Give the executive team a plain-language view of what has been built, what has been verified, what prevents a dependable demonstration, and what must happen next to complete the Phase 1 scope.

Assessment status: Preliminary and decision-ready for repository findings. The live accuracy baseline remains pending a safe credential, the running prototype, and the deployed agent configuration.

Source basis: Repository commit 3da43bf, the supplied redlined SOW and planning workbook, the 50-case historical reconstruction, source-matched offline fixtures, the bundled API materials, and the current public OptiCat service description. No customer credential was used.

Executive conclusion

OptiCat has a real and useful proof of concept. It is an actual Python MCP server with 16 read-only tools connected directly to the OptiCat automotive catalog. It already covers the intended business journeys in some form: finding parts for a vehicle, decoding a VIN, checking where a part fits, finding cross-references, reading supersession relationships, and retrieving product details.

The recommendation is to improve the existing service, not replace it. The technology foundation is reasonable. The central problem is functional consistency: the service can retrieve correct catalog information and still lose important details before the answer reaches the customer. Result lists may be partial without saying so, fitment conditions can be dropped, a part number can be ambiguous before a cross-reference lookup, and an unavailable or unauthorized API response can look like “no results.” Instructions telling the AI not to guess help, but they cannot substitute for complete, verifiable data.

The Phase 1 review is substantially complete at the repository and offline-test levels. It is not yet complete under the SOW because the current prototype has not been replayed against the approved test set with a safe development key, confirmed entitlements, and stored end-to-end transcripts. The credential pasted into this task should be treated as exposed, revoked or rotated, and replaced through an approved secret channel before testing.

The business decision

The executive team does not need to decide whether to start over. It needs to decide how much of the existing proof of concept to make dependable first.

Recommended direction

1. Make vehicle-to-part lookup accurate and complete.
2. Require exact evidence before saying that a part fits a vehicle.
3. Keep cross-reference, fitment, and supersession relationships distinct.
4. Make incomplete results, missing entitlement, and API unavailability explicit.
5. Add VIN-to-parts and part imagery once access and response behavior are confirmed.

This produces a credible demonstration without introducing a new platform merely for the sake of change.

What has been accomplished

The assessment work completed to date includes:

• A complete repository and architecture map.
• A verbatim inventory of all 16 MCP tools, their parameters, outputs, limits, and upstream operations.
• A review of the agent instructions, tool descriptions, deployment paths, security controls, and runtime behavior.
• An endpoint and data-path assessment covering vehicle search, part search, details, applications, VIN, and assets.
• Six focused diagnoses covering answer verification, truncation, workflow design, fitment data, instructions, and operational controls.
• A 50-case historical QA reconstruction with failure categories kept separate from current accuracy.
• An offline harness that executed the real tool functions against deterministic fixtures.
• A target functional architecture, prioritized opportunity register, implementation workstreams, success metrics, and an interactive customer-facing assessment.
• An interactive use-case proof covering all 18 supplied workbook queries, with side-by-side current/recommended workflows and an optional technical change map.
• A standalone natural-language planner that produces a transparent API plan in dry-run mode and supports safe read-only probing after a rotated key is supplied through the environment.
• A current public API documentation check confirming nine documented service operations, including VIN.
• A safe access and live-validation plan for completing the SOW baseline.

Functional readiness

What the repository proves

Sound and worth retaining

• Python is appropriate for this API-backed service.
• The official MCP Python SDK and FastMCP pattern are appropriate for the proof of concept.
• One shared OptiCat API boundary is a good foundation.
• The service is read-only and already covers the core catalog domains.
• The canonical FastMCP and Lambda registries both contain the same 16 tool names.
• The source is small enough to repair without a platform rewrite.

Needs repair

• Most tools return formatted prose instead of a complete result record the host can verify.
• Several lists display ten results without a machine-readable indication that more may exist.
• Fitment qualifiers and application notes are not consistently attached to each part result.
• Product lifecycle and image fields are available upstream but omitted from current outputs.
• Caught API errors are often returned as ordinary text rather than a distinct error state.
• The public JSON endpoint returns authentication failure inside an HTTP 200 response. The shared client checks the HTTP status but does not centrally classify the response body, so invalid access can be mistaken for an empty catalog result.
• One static deployment schema exposes seven tools while the canonical registries expose sixteen.
• The MCP dependency is not safely pinned: the source uses the v1 FastMCP API while current dependency declarations can resolve MCP v2.
• Secret-handling scripts and deployment documentation require remediation before live testing.

Should be consolidated or replaced

• The separate seven-tool static schema should not remain an independently maintained source of truth.
• The hand-written Lambda protocol adapter should be retired or clearly treated as a compatibility adapter if the FastMCP Streamable HTTP runtime is selected.
• Prompt-only accuracy controls should be replaced by result-level verification that travels with the service.

How the API and access model actually work

The service posts JSON requests to the OptiCat AutoCare catalog endpoint and authenticates with an X-Api-Key header. The current public service description exposes nine operations:

1. getSearchResults
2. getAutoCareVersion
3. getAutoCarePartDetails
4. getAutoCareSearchResults
5. getAutoCareBrandsLastUpdatedDates
6. getAutoCareVehicleAssets
7. getAutoCareVehicleResults
8. getAutoCareVehiclesByVIN
9. getAutoCarePartApplications

The repository calls six of those nine operations. It does not currently use the version/health operation, the brand-update operation, or the combined AutoCare/TecDoc search.

The OptiCat manual recommends starting with the hosted tester at https://webservice.opticatonline.com/autocare/v1/tester.html: enter the key, submit a wrapped JSON operation, and compare the raw response with what the application returns. It documents three authentication options—X-Api-Key header, api_key query parameter, and Basic Auth username. The repository’s X-Api-Key header is appropriate and avoids placing the key in the URL.

The manual also confirms the upstream data required by the recommended design:

• getAutoCarePartApplications returns ACES application rows with base vehicle, quantity, notes, position/other attribute IDs, and readable names;
• getAutoCareSearchResults supports vehicle and part filters, ACES attributes, facets, totals, page limits, optional PIES parts, and deep paging with nextCursorMark/cursorMark when enabled;
• getAutoCarePartDetails returns the PIES product record, lifecycle/extended information, relationships, and part image URLs;
• getAutoCareVehicleAssets is the separate base-vehicle ACES asset path; and
• the API key limits the catalog data returned by brand, so entitlement must be recorded for every test run.

The manual’s main operation overview predates the VIN addition, while the current public WSDL includes getAutoCareVehiclesByVIN. The public WSDL is therefore the current operation source for VIN; the manual remains the detailed source for the other request and response contracts.

The public endpoint is reachable without credentials, but a request without a valid key returns a body-level 401 status while the HTTP request itself succeeds. This means access validation must happen before catalog testing and must inspect the response content, not only the HTTP status.

Credential decision

The credential shared in this task should be considered compromised because it was transmitted in conversation text. It was not used during this review. The safe path is:

1. Revoke or rotate the exposed value.
2. Supply a new development key through an approved secret manager or secure handoff channel.
3. Confirm in writing which brands, datasets, VIN features, paging features, and rate limits apply to the key.
4. Record only a non-secret key identifier in test results.
5. Validate the key with getAutoCareVersion before running catalog cases.
6. Keep all request/response evidence redacted of credentials.

What the planning workbook contributes

The supplied workbook defines 18 populated example queries across the five intended lookup families:

• vehicle to part;
• VIN decode and VIN-to-parts;
• part number to vehicle fitment;
• part number to cross-reference;
• part number to supersession.

It is a useful product-planning artifact, not yet an acceptance suite. It lists likely API calls but does not provide a complete expected result, required qualifiers, entitlement, grading rule, or current live outcome for every row. It should be combined with the 50 historical QA cases rather than used as a replacement for them.

The workbook also contains two important planning examples not aligned with the older repository CSV: a Ford OE-to-Gates interchange query and a different VIN example. Those should be preserved as candidate test cases and adjudicated with OptiCat before the live baseline is frozen.

What the historical QA tells us—and what it does not

The historical set contains 50 cases. Thirteen carry an earlier pass label; the remaining records include false negatives, unsupported positive answers, contradictions, partial answers, missed attributes, qualifier problems, and infrastructure failures.

This does not mean the current system has a 26% accuracy rate. The cases were not rerun uniformly against the current deployed configuration, the API, and the same key scope. They are best used as regression and diagnosis cases for the live baseline.

SOW completion status

Why the SOW is not yet fully complete

The SOW makes the live baseline a required activity. The final Access Date cannot be treated as complete until the following package is available:

• a partner identity with appropriate permissions;
• the repository and deployed revision;
• the running prototype and its connection method;
• a rotated development key;
• written brand, dataset, VIN, paging, and rate-limit entitlement;
• the actual deployed model, prompt, settings, and tool bindings;
• original handover and prior QA materials;
• a named OptiCat API/catalog owner available for adjudication.

Recommended target behavior

The target is a simple, explainable path:

1. Understand the customer’s task.
2. Resolve the exact vehicle or part identity.
3. Ask one question only when the missing answer would change the fitment or relationship.
4. Retrieve all required catalog pages or disclose that the result is incomplete.
5. Preserve fitment conditions, lifecycle, notes, and image records beside the result they govern.
6. Verify every part-number and fitment statement against the current catalog response.
7. Answer, qualify, or abstain; never fill a catalog gap from model memory.

Recommended Phase 2 workstreams

1. Reliable catalog responses

Return totals, returned counts, paging, completeness, source, and distinct error states. Reuse one API client with a clear request budget and health check.

2. Exact vehicle and fitment resolution

Carry engine, side, position, submodel, drive type, quantity, and notes through the full lookup. Require exact application evidence for every “fits” statement.

3. Part identity and relationships

Resolve ambiguous part numbers before cross-reference or supersession. Keep interchange, fitment, and replacement relationships separate and explain lifecycle status.

4. Verified answer behavior

Use one clarification and abstention policy. Validate the final answer against the tool results so unsupported part numbers cannot be added after retrieval.

5. Release reliability

Use one tool manifest, safe secret handling, consistent deployment, traceable logs, and a release gate based on the approved test set.

Demonstration acceptance criteria

A demonstration build should not be accepted until:

• no part number absent from the current catalog evidence appears in an answer;
• every unconditional fitment claim has exact application evidence;
• incomplete result sets always disclose that they are incomplete;
• unavailable, unauthorized, invalid, and complete-zero outcomes are distinct;
• engine, side, position, and other supplied fitment conditions remain attached to the relevant result;
• cross-reference is never mislabeled as supersession or fitment;
• all approved tools are present on the chosen deployed runtime;
• the canonical test set has stored API, tool, and full-agent evidence;
• all earlier accepted cases remain passing regression guards.

Success measures

Implementation sequence

1. Access and baseline: rotate the key, freeze the deployed configuration, confirm entitlement, and replay the approved cases.
2. Response foundation: complete result records, paging, errors, health, and request budgets.
3. Identity and relationships: part resolution, interchange, lifecycle, and supersession.
4. Fitment and product detail: qualifiers, exact fitment, attributes, images, and VIN-to-parts.
5. Host integration: clarification, abstention, final-answer checking, and traceability.
6. Stabilization: full replay, performance validation, security cleanup, and demonstration rehearsal.

A previous working estimate placed this at roughly 39–58 engineer-days for the recommended demonstrable prototype. That estimate is not a fixed-price offer and must be recalibrated after safe live access.

Vercel and Supabase decision

Vercel and Supabase can help with a polished demonstration, but they are not required to repair the MCP service.

Good optional uses

• Vercel: a stakeholder-facing demo interface, evaluation dashboard, or shareable result explorer.
• Supabase: storing test cases, run metadata, redacted transcripts, judgments, and KPI history.

What they should not replace

• the Python MCP service;
• the OptiCat API adapter;
• server-side identity and fitment verification;
• the authoritative catalog;
• secure key storage appropriate to the chosen runtime.

The recommended decision is to keep the core service platform-neutral, finish the functional PRD and live baseline, then choose Vercel/Supabase only if the demonstration experience or evaluation workflow benefits from them.

Executive meeting narrative

The 60-second explanation

“The team built a real MCP service with broad OptiCat catalog coverage. We do not need to start over. The main issue is that some important catalog details are lost between retrieval and the customer-facing answer, so the system can be incomplete or overconfident even when the API has the right data. We have mapped those gaps and produced a clear repair plan. Before we can claim a current accuracy rate or finalize a build proposal, we need a rotated key, confirmed entitlements, the running prototype, and the deployed agent configuration. Once access is in place, we can run the approved test set and convert this preliminary assessment into the final SOW report.”

Decisions requested from the executive team

1. Approve “repair, do not rewrite” as the architecture direction.
2. Confirm the four core demonstration journeys and whether VIN and imagery are required.
3. Assign an owner to rotate credentials and provide written entitlement.
4. Provide the running prototype and deployed host export.
5. Confirm the canonical QA set and the OptiCat adjudicator.
6. Authorize commercial scoping only after the live baseline, or explicitly accept the uncertainty of pricing before it.

Immediate next steps