Opticat item search MCP reviewPhase 1 discovery, validation, and path forward
View Demo
Library contents
All library documents

Opportunity report

Prioritized improvements tied to evidence, SOW requirements, effort, and acceptance tests.

mdPreliminary366 lines · 13 min read
discovery/product/generated/opportunity-report.mdView source on GitHub

A preliminary repository and fixture opportunity portfolio, not a current live accuracy report.

Generated deterministically from discovery/product/opportunity-register.yaml. This is a preliminary repository/fixture opportunity portfolio, not a current live accuracy report.

Repository SHA: 3da43bf6043c4f8af18a9462a1492b01775b2855

Executive portfolio

Measure Value
Total opportunities 15
Mandatory gates 9
Access/external-action dependent 5
Quick wins 3
Foundation opportunities 3
Accuracy workstreams 4

The portfolio is ordered with mandatory gates first, then by the stored priority score. A high score does not allow work to bypass a mandatory security, evidence, or validation gate.

  1. Close credential handling and obtain safe entitlement-aware access (OPP-009, OPP-013).
  2. Establish the live evidence/release gate (OPP-011).
  3. Stabilize the MCP/deployment foundation (OPP-008, OPP-012, OPP-001, OPP-007).
  4. Build qualifier fidelity and exact fitment validation (OPP-003, OPP-002).
  5. Repair identity, interchange, lifecycle, VIN, and imagery workflows (OPP-004, OPP-005, OPP-015, OPP-006).
  6. Deploy portable grounding and evidence-aware observability (OPP-010, OPP-014).

Ranked opportunities

Rank ID Opportunity Gate Score Effort Milestone Status
1 OPP-009 Resolve committed-credential and secret-exposure risks Mandatory 6.33 3 M0 blocked_external_action
2 OPP-008 Consolidate the deployed tool surface Mandatory 5.00 3 M1 proposed
3 OPP-013 Make API entitlement and data scope observable Mandatory 4.75 3 M0 blocked_external_action
4 OPP-001 Make result completeness machine-visible Mandatory 4.00 5 M1 proposed
5 OPP-003 Preserve row-level ACES qualifiers and notes Mandatory 3.80 5 M3 proposed
6 OPP-007 Return structured errors and enforce request budgets Mandatory 3.00 5 M1 proposed
7 OPP-010 Make evidence-only grounding portable across hosts Mandatory 3.00 5 M4 access_required
8 OPP-011 Establish a reproducible live accuracy and release gate Mandatory 3.00 5 M0 access_required
9 OPP-002 Enforce an exact fitment validation gate Mandatory 2.50 8 M3 proposed
10 OPP-012 Pin and continuously test the MCP SDK/runtime contract Portfolio 5.50 2 M1 proposed
11 OPP-006 Complete part-image retrieval end to end Portfolio 5.00 2 M3 proposed
12 OPP-005 Model lifecycle and supersession explicitly Portfolio 4.33 3 M2 proposed
13 OPP-004 Resolve part identity before interchange lookup Portfolio 3.20 5 M2 proposed
14 OPP-014 Add health, telemetry, and evidence-aware observability Portfolio 3.20 5 M5 proposed
15 OPP-015 Make VIN an entitlement-aware decode-to-parts workflow Portfolio 2.10 5 M3 access_required

Portfolio distribution

By bucket

Bucket Opportunities
access_data_dependency 1
accuracy_workstream 4
foundation 3
mandatory_gate 3
quick_win 3
strategic_production 1

By category

Category Opportunities
api_and_entitlement 1
deployment_conformance 2
evaluation_governance 1
fitment_and_qualifiers 2
grounding_and_governance 1
identity_and_interchange 1
imagery 1
lifecycle 1
resilience_and_observability 2
retrieval_integrity 1
security 1
vehicle_resolution 1

By milestone

Milestone Opportunities
M0 3
M1 4
M2 2
M3 4
M4 1
M5 1

By status

Status Opportunities
access_required 3
blocked_external_action 2
proposed 10

Opportunity details

OPP-009 — Resolve committed-credential and secret-exposure risks

  • Portfolio: mandatory_gate; mandatory gate.
  • Problem: A credential-like Gateway key is tracked, and deployment scripts can expose upstream secrets.
  • Target outcome: Credentials are rotated, history is incident-reviewed, scripts use approved secret injection, and outputs are scanned.
  • Evidence: STATIC-PROVEN.
  • Priority: 6.33 using impact 5, safety 5, reach 4, time 5, confidence 1.0, effort 3.
  • Affected cases: N/A — governance/security opportunity.
  • Dependencies: None.
  • Owner / milestone / status: Security owner / M0 / blocked_external_action.
  • Acceptance:
    • Credential owner confirms revoke/rotate and incident disposition.
    • No secret is printed, passed in command-line history, or returned in errors.
    • Repository and generated-artifact secret scans pass.
  • Evidence references:
    • discovery/templates/16-diagnosis-guardrails.md §SECURITY STOP — committed credential

OPP-008 — Consolidate the deployed tool surface

  • Portfolio: quick_win; mandatory gate.
  • Problem: FastMCP and Lambda map 16 tools, while the static Gateway schema exposes only seven.
  • Target outcome: One canonical deployment contract exposes the intended versioned tool set and is tested in CI.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 5.00 using impact 4, safety 3, reach 4, time 4, confidence 1.0, effort 3.
  • Affected cases: FD-01, FD-02, WE-06, WE-10, BT-13.
  • Dependencies: None.
  • Owner / milestone / status: Platform engineer / M1 / proposed.
  • Acceptance:
    • Chosen deployed surface exposes the approved tool manifest.
    • CI fails on registry/schema drift.
  • Evidence references:
    • discovery/templates/20-baseline-direct.md §Registration and deployment-surface results
    • discovery/eval/offline-architecture-results.json finding REG-03

OPP-013 — Make API entitlement and data scope observable

  • Portfolio: access_data_dependency; mandatory gate.
  • Problem: Missing results cannot be separated from code, data, or key scope because written entitlement and health/scope metadata are absent.
  • Target outcome: Test runs identify key-scope/brand/VIN entitlement and distinguish not-entitled, unavailable, and complete zero.
  • Evidence: ACCESS-REQUIRED.
  • Priority: 4.75 using impact 5, safety 4, reach 5, time 5, confidence 0.75, effort 3.
  • Affected cases: ALL-50.
  • Dependencies: OPP-009.
  • Owner / milestone / status: OptiCat API owner / M0 / blocked_external_action.
  • Acceptance:
    • Each test run records a non-secret key-scope identifier and written entitlement.
    • Errors distinguish not entitled, unavailable, empty, and truncated.
  • Evidence references:
    • discovery/templates/03-api-coverage.md §UNKNOWN items
    • discovery/templates/22-ground-truth-verification.md §OptiCat data-team punch list

OPP-001 — Make result completeness machine-visible

  • Portfolio: foundation; mandatory gate.
  • Problem: List tools silently return subsets that the model can interpret as complete.
  • Target outcome: Every list response exposes total, returned, truncated, cursor, and an explicit completeness warning.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 4.00 using impact 5, safety 5, reach 5, time 5, confidence 1.0, effort 5.
  • Affected cases: FD-15, WE-02, BT-12.
  • Dependencies: None.
  • Owner / milestone / status: Backend engineer / M1 / proposed.
  • Acceptance:
    • 0/10/85 fixtures expose correct completeness fields.
    • A truncated response cannot be serialized without a warning and cursor/non-pageable reason.
  • Evidence references:
    • discovery/templates/12-diagnosis-truncation.md §Evidence
    • discovery/templates/20-baseline-direct.md §Confirmed serializer and request findings
    • discovery/eval/offline-architecture-results.json findings SER-01/SER-02

OPP-003 — Preserve row-level ACES qualifiers and notes

  • Portfolio: accuracy_workstream; mandatory gate.
  • Problem: Position, side, engine, submodel, drive type, and notes are dropped or shown only as aggregate facets.
  • Target outcome: Each result carries the exact qualifiers and notes that govern its fitment.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 3.80 using impact 5, safety 5, reach 5, time 4, confidence 1.0, effort 5.
  • Affected cases: FD-05, FD-16, WE-04, WE-07, WE-08, BT-06, BT-13, BT-14, BT-16.
  • Dependencies: OPP-001.
  • Owner / milestone / status: Backend engineer / M3 / proposed.
  • Acceptance:
    • Fixture position and engine qualifiers survive normalization and output.
    • Named qualifier eval cases flip without regressing historical pass guards.
  • Evidence references:
    • discovery/templates/14-diagnosis-schema.md §Evidence
    • discovery/templates/20-baseline-direct.md finding SER-02

OPP-007 — Return structured errors and enforce request budgets

  • Portfolio: foundation; mandatory gate.
  • Problem: Caught upstream failures are ordinary strings and multi-call tools can consume several sequential 30-second timeouts.
  • Target outcome: Typed retryable errors, shared client policy, finite call/time budgets, and honest host behavior.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 3.00 using impact 4, safety 4, reach 3, time 4, confidence 1.0, effort 5.
  • Affected cases: BT-18, FD-01, FD-02, WE-06.
  • Dependencies: None.
  • Owner / milestone / status: Backend engineer / M1 / proposed.
  • Acceptance:
    • Timeout/500 fixtures return a classified error within the tool budget.
    • The agent never converts upstream unavailable into a catalog-negative.
  • Evidence references:
    • discovery/templates/16-diagnosis-guardrails.md §§Evidence, Required hardening contract
    • discovery/templates/20-baseline-direct.md finding ERR-01

OPP-010 — Make evidence-only grounding portable across hosts

  • Portfolio: foundation; mandatory gate.
  • Problem: Strong candidate instructions may not be deployed and do not travel with the MCP server; tool descriptions lack a universal evidence-only contract.
  • Target outcome: Tool descriptions and deployed host policy prohibit unsupported part/spec/fitment claims and define abstention.
  • Evidence: HISTORICAL-QA.
  • Priority: 3.00 using impact 5, safety 5, reach 5, time 5, confidence 0.75, effort 5.
  • Affected cases: BT-04, BT-05, BT-10, BT-13, BT-14, WE-04.
  • Dependencies: OPP-001, OPP-002, OPP-007.
  • Owner / milestone / status: AI evaluation engineer / M4 / access_required.
  • Acceptance:
    • Deployed agent export contains the approved policy.
    • Every model-visible data tool states evidence-only and fitment-verification rules.
    • Host portability tests show zero unsupported part numbers.
  • Evidence references:
    • discovery/templates/11-diagnosis-grounding.md §Evidence
    • discovery/templates/15-diagnosis-instructions.md §Replacement instruction set v2

OPP-011 — Establish a reproducible live accuracy and release gate

  • Portfolio: mandatory_gate; mandatory gate.
  • Problem: The 50 historical cases lack a current uniform direct/API/agent replay and adjudicated baseline.
  • Target outcome: Every release produces reproducible case evidence, KPI deltas, regressions, and layer attribution.
  • Evidence: ACCESS-REQUIRED.
  • Priority: 3.00 using impact 5, safety 5, reach 5, time 5, confidence 0.75, effort 5.
  • Affected cases: ALL-50.
  • Dependencies: OPP-009, OPP-013.
  • Owner / milestone / status: AI evaluation engineer / M0 / access_required.
  • Acceptance:
    • All 50 cases have direct, raw-API, and agent evidence.
    • Every disputed case has named human adjudication.
    • Release fails when grounded accuracy or safety KPIs miss target.
  • Evidence references:
    • discovery/templates/21-baseline-agent.md §Required live replay protocol
    • discovery/templates/22-ground-truth-verification.md §Live execution record required per case

OPP-002 — Enforce an exact fitment validation gate

  • Portfolio: mandatory_gate; mandatory gate.
  • Problem: The host can state that a part fits without exact part-plus-vehicle application evidence.
  • Target outcome: Every fitment assertion is verified, conditional with named qualifiers, or refused.
  • Evidence: STATIC-PROVEN.
  • Priority: 2.50 using impact 5, safety 5, reach 5, time 5, confidence 1.0, effort 8.
  • Affected cases: BT-04, BT-10, BT-13, BT-14, BT-17, WE-05, WE-07, WE-08, KJ-01.
  • Dependencies: OPP-003, OPP-013.
  • Owner / milestone / status: MCP architect / M3 / proposed.
  • Acceptance:
    • 100% of fits claims carry exact application evidence.
    • Unresolved qualifiers produce conditional or clarification, never unconditional fitment.
  • Evidence references:
    • discovery/templates/11-diagnosis-grounding.md §Missing guardrails
    • discovery/templates/14-diagnosis-schema.md §Corrected output schemas
    • discovery/templates/15-diagnosis-instructions.md §Replacement instruction set v2

OPP-012 — Pin and continuously test the MCP SDK/runtime contract

  • Portfolio: quick_win; scored opportunity.
  • Problem: The root MCP dependency is unpinned and current behavior depends on SDK serialization and transport semantics.
  • Target outcome: Supported Python/MCP versions are pinned, lockable, and covered by registry/protocol compatibility tests.
  • Evidence: STATIC-PROVEN.
  • Priority: 5.50 using impact 3, safety 2, reach 3, time 3, confidence 1.0, effort 2.
  • Affected cases: N/A — governance/security opportunity.
  • Dependencies: None.
  • Owner / milestone / status: Platform engineer / M1 / proposed.
  • Acceptance:
    • Dependency lock records supported MCP/Python versions.
    • CI verifies tool registry, schemas, and representative serialized results.
  • Evidence references:
    • discovery/templates/00-repo-map.md §Dependency declarations and SDK currency
    • discovery/templates/01-tool-inventory.md §Scope and contract notes

OPP-006 — Complete part-image retrieval end to end

  • Portfolio: quick_win; scored opportunity.
  • Problem: Part details drop digital asset IDs and URLs; the existing asset tool is vehicle-only and also drops URLs.
  • Target outcome: A part lookup returns usable image metadata/URLs without requiring the model to invent an asset ID.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 5.00 using impact 3, safety 1, reach 2, time 4, confidence 1.0, effort 2.
  • Affected cases: CC-01.
  • Dependencies: OPP-013.
  • Owner / milestone / status: Backend engineer / M3 / proposed.
  • Acceptance:
    • Denso 234-4668-style lookup can return part image URLs in one composed tool flow.
    • Asset authentication/lifetime is documented.
  • Evidence references:
    • discovery/templates/01-tool-inventory.md §Part-image chain, end to end
    • discovery/templates/20-baseline-direct.md findings SER-03/SER-05

OPP-005 — Model lifecycle and supersession explicitly

  • Portfolio: accuracy_workstream; scored opportunity.
  • Problem: The tool reads replaces/replacedBy arrays but drops lifecycle status in extended information and does not traverse chains.
  • Target outcome: Part status and complete bounded supersession chains are returned as a relationship graph.
  • Evidence: FIXTURE-VALIDATED.
  • Priority: 4.33 using impact 4, safety 3, reach 3, time 3, confidence 1.0, effort 3.
  • Affected cases: FD-07, FD-08, FD-09, FD-10, FD-11, WE-09, WE-10.
  • Dependencies: OPP-013.
  • Owner / milestone / status: Backend engineer / M2 / proposed.
  • Acceptance:
    • Lifecycle codes and source values are preserved.
    • Chain traversal is cycle-safe and capped at ten hops.
    • Interchange equivalence is never labeled supersession.
  • Evidence references:
    • discovery/templates/14-diagnosis-schema.md §Part details / PIES
    • discovery/templates/20-baseline-direct.md finding SER-04

OPP-004 — Resolve part identity before interchange lookup

  • Portfolio: accuracy_workstream; scored opportunity.
  • Problem: Ambiguous numbers can resolve to the wrong part type or brand before cross-references are read.
  • Target outcome: The server returns identity candidates for disambiguation and queries interchange only for the resolved part.
  • Evidence: STATIC-PROVEN.
  • Priority: 3.20 using impact 4, safety 4, reach 4, time 4, confidence 1.0, effort 5.
  • Affected cases: BT-01, FD-06, BT-08, BT-17, BT-18.
  • Dependencies: OPP-001, OPP-013.
  • Owner / milestone / status: Backend engineer / M2 / proposed.
  • Acceptance:
    • PH400 returns filter identity candidates or a clarification, never a silently selected pintle hook.
    • Interchanges remain labeled separately from fitment and supersession.
  • Evidence references:
    • discovery/templates/13-diagnosis-wiring.md §Evidence
    • discovery/templates/03-api-coverage.md §The five key questions

OPP-014 — Add health, telemetry, and evidence-aware observability

  • Portfolio: strategic_production; scored opportunity.
  • Problem: Logs do not consistently record tool, operation count, completeness, error class, correlation, or evidence coverage.
  • Target outcome: Operators can diagnose latency, truncation, upstream failures, and unsupported-answer risk by request.
  • Evidence: STATIC-PROVEN.
  • Priority: 3.20 using impact 4, safety 4, reach 4, time 4, confidence 1.0, effort 5.
  • Affected cases: BT-18.
  • Dependencies: OPP-001, OPP-007.
  • Owner / milestone / status: Platform engineer / M5 / proposed.
  • Acceptance:
    • Every tool call records correlation ID, latency, operation count, total/returned/truncated, retry count, and error class.
    • Health separates server-ready, API-reachable, and entitlement-ready states.
  • Evidence references:
    • discovery/templates/16-diagnosis-guardrails.md §§Evidence, Required hardening contract

OPP-015 — Make VIN an entitlement-aware decode-to-parts workflow

  • Portfolio: accuracy_workstream; scored opportunity.
  • Problem: VIN is registered but operational status is unknown and successful decode is not composed directly into the requested part lookup.
  • Target outcome: Locally validate VIN, report entitlement accurately, resolve vehicle qualifiers, and continue into part search.
  • Evidence: ACCESS-REQUIRED.
  • Priority: 2.10 using impact 4, safety 3, reach 3, time 4, confidence 0.75, effort 5.
  • Affected cases: FD-01, FD-02, WE-06.
  • Dependencies: OPP-002, OPP-007, OPP-013.
  • Owner / milestone / status: Backend engineer / M3 / access_required.
  • Acceptance:
    • Invalid VIN fails locally with a useful reason.
    • Entitlement failure is distinct from decode zero or upstream error.
    • Successful decode supplies the exact resolved vehicle to part search.
  • Evidence references:
    • discovery/templates/16-diagnosis-guardrails.md §VIN status
    • discovery/templates/03-api-coverage.md §Documented operation matrix

Method and limitations

  • Score formula: (business_impact + user_safety + evidence_reach + time_criticality) * confidence / effort.
  • STATIC-PROVEN and FIXTURE-VALIDATED opportunities describe current code/serializer behavior.
  • HISTORICAL-QA and ACCESS-REQUIRED opportunities require deployed/live closeout before current KPI claims.
  • The register prioritizes architecture and assessment work; it is not a binding Phase 2 scope, price, or date.
View Demo