Historical evaluation suite

# ============================================================
# OptiCat MCP — Evaluation Case Suite v1
# Source: OptiCat internal testing notes (Apr 2-13, 2026; OneNote
# package June 2026). Extracted and deduplicated Aug 3, 2026.
# 37 cases across 5 testers. Ground truth = OptiCatOnLine.com as
# annotated (screenshots) by the testers, unless noted.
#
# SCORING RUBRIC (grade every rerun into exactly one):
#   grounded_correct     - right answer, every claim backed by tool results
#   grounded_incomplete  - right parts but missing catalog results/brands
#   false_negative       - denied data that exists in the catalog
#   hallucinated_positive- returned parts/specs/fitment not in the catalog
#   contradiction        - reversed itself on its own prior answer
#   qualifier_error      - wrong position/side/engine/submodel
#   infra_failure        - hang, timeout, error, tool unavailable
#
# REPLAY MODES:
#   direct  - scripted MCP client calls tools directly (deterministic;
#             tests the server: search, serialization, truncation)
#   agent   - full conversation through an agent loop (tests the whole
#             stack: grounding, question logic, verify-before-assert)
# Run every case in BOTH modes. A case that passes direct but fails
# agent localizes the defect to instructions/host; fails-direct = server.
#
# NOTE: ground_truth reflects the April 2026 catalog. Statuses
# (Announced/Available/Superseded) can drift — re-verify against
# OptiCat Online or the raw API before scoring hard failures.
# ============================================================

cases:

  # ---------- FRANK D'AGOSTINO ----------

  - id: FD-01
    tester: Frank D'Agostino
    lookup_type: vin
    query: "hvac filter for 4T1BK1EB5GU235558"
    ground_truth: "VIN valid (NHTSA clean decode): 2016 Toyota Avalon sedan. Cabin/HVAC filter should resolve."
    observed_april: "Could not find any vehicle for the VIN; asked for YMM."
    failure_class: false_negative
    notes: "VIN verified via NHTSA vPIC. Failed identically Apr 2 and Apr 10."

  - id: FD-02
    tester: Frank D'Agostino
    lookup_type: vin
    query: "water pump for 1GCGTDEN1K1238034"
    ground_truth: "VIN valid (NHTSA clean decode): 2019 Chevrolet Colorado pickup. NHTSA caveat: model year may be inexact."
    observed_april: "Could not decode; asked for YMM."
    failure_class: false_negative

  - id: FD-03
    tester: Frank D'Agostino
    lookup_type: interchange
    query: "I need to replace a kia coolant pump 251002GTA0"
    followups: ["what's the difference between Aisin WPK-822 and WPK-823?", "does WPK-822 include a gasket?", "Do any of these part numbers include a gasket?"]
    ground_truth: "All 9 interchanges: Gates 42180BH (BBSC), GMB 146-7450 (BFBQ), US Motor Works US8163 (BKMK), AISIN WPK-822/WPK-823 (BKQL), GNS YH-K173/YH-K173H (GPHG), Gates Mexico 42180BH (JNSZ), URG 537953 (HBWN). WPK-822 = pulley included; WPK-823 = no pulley. Gates 42180BH and US8163 include gasket per attributes."
    observed_april: "PASS — all 9 returned, pulley difference and gasket attributes correct."
    failure_class: pass
    notes: "Regression guard. Known data gap: AISIN includes a gasket in reality (visible only in product image), catalog attributes say no."

  - id: FD-04
    tester: Frank D'Agostino
    lookup_type: interchange
    query: "need to replace honda 160105PA305"
    followups: ["are there any other aftermarket options?", "does 845-12137 fit 2021 Honda Accord?", "which engine does it cover?"]
    ground_truth: "Only aftermarket cross: GB Remanufacturing 845-12137 (CLCW), fuel injector. Fits 2021 Accord; Honda 1.5T family."
    observed_april: "PASS — single cross correct, fitment confirmed."
    failure_class: pass
    notes: "Regression guard. 'L15B' engine designation was model knowledge beyond catalog — acceptable enrichment, flag if presented as catalog data."

  - id: FD-05
    tester: Frank D'Agostino
    lookup_type: interchange
    query: "wheel bearing 12429204"
    followups: ["Does GMB 730-0003 fit a 2013 Chevy Impala?", "Does it fit front or rear?", "does GMB 730-0003 include hardware?"]
    ground_truth: "9 interchanges incl. SKF BR930548K, GMB 730-0003, BCA WE60702/WE60702HW, Quality-Built QBS513121/WH513187HD, Schaeffler 102026, FAG MX 713 1605 10, CW Bearing 513121. GMB 730-0003: fits 2013 Impala qty 2, position FRONT, hardware included (3 BOLTS, 1 CIRCLIP per PIES)."
    observed_april: "Interchanges + Impala fitment right; said position unspecified and no hardware (both wrong — fields exist)."
    failure_class: attribute_miss

  - id: FD-06
    tester: Frank D'Agostino
    lookup_type: interchange
    query: "what brake pads can replace akebono EUR1924"
    followups: ["is there an equivalent from Textar?", "what is the difference between Raybestos PGD1924M and MGD1924M", "what is the difference between Textar 2231701 and 2231701CP"]
    ground_truth: "Crosses: Bendix CFC1924, Raybestos MGD1924M/PGD1924M, Centric 10419240/30019240, Promax 10-/11-/21-1924; Volvo OE numbers. Textar 2231701 and 2231701CP valid; CP = ceramic/copper-free per PIES."
    observed_april: "Listed crosses correctly, then claimed it couldn't find PGD1924M/MGD1924M — the numbers it had just listed."
    failure_class: contradiction
    notes: "Scored dimension: detail lookup must succeed for parts already returned as interchanges."

  - id: FD-07
    tester: Frank D'Agostino
    lookup_type: supersession
    query: "I'm looking for an oil filter part number 06E 115 562 B"
    followups: ["is this the same as part number 06E 115 562 E"]
    ground_truth: "B and E do NOT return identical cross sets: GKI EF45598 maps only to ...562 E. B superseded to E in reality; catalog records it only via supplier interchange differences."
    observed_april: "Claimed both numbers return the same crosses / are 'interchangeable supersessions'; missed GKI EF45598."
    failure_class: partial
    notes: "Frank: MCP only reads supplier-mapped interchanges; no supersession logic."

  - id: FD-08
    tester: Frank D'Agostino
    lookup_type: supersession
    query: "what number replaces GM 19210284"
    followups: ["does this replace 94632619"]
    ground_truth: "Cross family incl. Wix 51040, ACDelco PF47/PF47E/PF47F/UPF47, FRAM PH3387A, Purolator L10111, K&N HP-1001. 94632619 in the same family."
    observed_april: "PASS — worked as it should (interchange-driven)."
    failure_class: pass

  - id: FD-09
    tester: Frank D'Agostino
    lookup_type: supersession
    query: "my part is stamped 47750-12310, what does it fit?"
    followups: ["is this part number superseded?", "is this part the same as 47750-12310-84"]
    ground_truth: "URG 87527 (HBWN) disc brake caliper; 0 vehicle applications (real data gap); no supersession record; -84 suffix reasoning acceptable."
    observed_april: "PASS — matched catalog, honest about missing applications."
    failure_class: pass

  - id: FD-10
    tester: Frank D'Agostino
    lookup_type: supersession
    query: "My part is stamped 4775001020, is this part number superseded?"
    followups: ["Toyota", "Check aftermarket cross references for parts that match 47750-01020"]
    ground_truth: "Online interchange search returns 3 caliper matches: R1 Concepts CAL-47750, DFC 331-47750, URG 324796."
    observed_april: "Reported ZERO aftermarket parts across the entire catalog."
    failure_class: false_negative

  - id: FD-11
    tester: Frank D'Agostino
    lookup_type: supersession
    query: "my water pump is stamped 16100-09441, is this superseded?"
    followups: ["is this the same as part number 16100-09442"]
    ground_truth: "Real supersession (…441 → …442) not in catalog. Both OE numbers interchange to same pumps: Gates 43538, GMB 170-7370, US6047, AISIN WPT-803, GNS YH-T216, ACKOJA A70-50040, Gates Mexico 43538 (+URG 161026 on …442)."
    observed_april: "PARTIAL — no supersession found (matches catalog), interchange equivalence correctly stated."
    failure_class: partial

  - id: FD-12
    tester: Frank D'Agostino
    lookup_type: part_lookup
    query: "engine air filter for 1999 Ford Taurus"
    followups: ["vulcan", "Which filter has the best filtration?", "compare to GK industries"]
    ground_truth: "Engines: V8 3.4L, V6 3.0L 2968cc, V6 3.0L 2986cc (Vulcan = 2986cc per Frank). 10 filters incl. Wix 46428 (cellulose, 325 CFM). Purolator MX TL20195 appears but is actually an oil filter (supplier data error)."
    observed_april: "PASS — engine names consumer-friendly, filtration comparison from attributes."
    failure_class: pass
    notes: "[reconstructed opening query]. Verify which cc the POC maps 'vulcan' to (it said 2968cc; Frank says 2986cc)."

  - id: FD-13
    tester: Frank D'Agostino
    lookup_type: part_lookup
    query: "oil filter for 2010 jetta 2.0L"
    followups: ["diesel", "which filter has the best filtration?", "Do any Bosch filters fit the diesel engine?", "Which Hengst filter fits the diesel engine?"]
    ground_truth: "Engines: 2.0L 1968cc (TDI diesel), 2.0L 1984cc gas, 2.5L 2480cc. Mann W 719/45 fits the GAS 2.0L not diesel; Beck/Arnley 041-8179/8184 and Denso 150-3038 fit 2.5L; Hengst E87H D208 fits 2.5L; Vaico V10-0897 fits 2.0L gas. Correct: no Bosch for the diesel."
    observed_april: "Presented gas/2.5L filters as the diesel set; wrong Hengst answer via wrong interchange."
    failure_class: hallucinated_positive
    notes: "Underlying data is unevenly mapped (fuel type vs engine code vs CCs) — score with that in mind; a correct answer may require asking which engine code."

  - id: FD-14
    tester: Frank D'Agostino
    lookup_type: part_lookup
    query: "belt for 2020 toyota corolla"
    followups: ["2.0L", "What is the difference between 469K6F and 480K6?", "Which one fits my Corolla?", "Is there a dayco belt for my corolla?", "which one fits the 2.0L", "Which other brands fit the 2.0L"]
    ground_truth: "2.0L correct belt: Gates Mexico K060442; Dayco 5060440 ALSO fits per Online. Prime Guard 469K6F fits no 2020 Corolla; 480K6 and Gates Mexico K060480 fit the 1.8L."
    observed_april: "Contradicted itself twice; denied Dayco 5060440 fits; concluded only K060442."
    failure_class: contradiction
    notes: "Frank: 'swing and a miss… not confident enough to buy any of these parts.'"

  - id: FD-15
    tester: Frank D'Agostino
    lookup_type: part_lookup
    query: "headlight assembly for 2017 F-150"
    followups: ["King Ranch", "left and right", "please expand", "right", "TYC", "motorcraft", "valeo right side"]
    ground_truth: "Only FOUR headlight assemblies exist for 2017 F-150, all XL-trim-specific; NONE for King Ranch. The 10 Valeo assemblies POC returned do not fit the 2017 F-150."
    observed_april: "Claimed 11 left/10 right fit King Ranch; couldn't page past first 10 items; ended with 10 non-fitting Valeo units."
    failure_class: hallucinated_positive
    notes: "Also the canonical truncation case: 'catalog only exposes the first 10 items… does not allow paging.'"

  - id: FD-16
    tester: Frank D'Agostino
    lookup_type: part_lookup
    query: "brake pads for 2023 BMW M2"
    followups: ["Front", "what is the difference between 1600-1850-01 and 1600-2478-01", "which one fits my M2", "does brembo make brake pads for the 2023 M2", "does P06061N fit front or rear?"]
    ground_truth: "Positions per Online: DFC 1551-2478-00/-01 = FRONT; 1553-1850-00/-01 = REAR; 1600-1850-01 = REAR; 1600-2478-01 = FRONT; Beck/Arnley 085-1985 = REAR, 085-2140 = FRONT; Brembo P06061N = REAR, P06100 = FRONT."
    observed_april: "'Front' list contained 4 rear sets; answered P06061N = front (it's rear)."
    failure_class: qualifier_error
    notes: "Position qualifier systematically ignored; parts themselves all fit the M2."

  # ---------- CHRIS CASTILLO ----------

  - id: CC-01
    tester: Chris Castillo
    lookup_type: images
    query: "Does Denso have Images for oxygen sensors?"
    followups: ["Part # 234-4668", "Show me the images for Denso 234-4668."]
    ground_truth: "Denso 234-4668 (BBNF) exists with photo assets in the catalog."
    observed_april: "Confirmed images exist, then reversed: get_part_assets needs an asset link ID that get_part_details never returns; claimed only vehicle images are supported."
    failure_class: contradiction
    notes: "Tool-composition defect — the two tools cannot be chained."

  - id: CC-02
    tester: Chris Castillo
    lookup_type: part_lookup
    query: "air filter for chevrolet taho rst 2025"
    followups: ["5.3L"]
    ground_truth: "UNVERIFIED in notes — pull fresh from Online: 2025 Tahoe RST engines (5.3L V8/6.2L V8/3.0L diesel I6) and the 5.3L filter list."
    observed_april: "Suspicious: engine-narrowed list was a SUPERSET of the all-engines list."
    failure_class: partial
    notes: "Keep the typo 'taho' — misspelling tolerance is part of the test."

  - id: CC-03
    tester: Chris Castillo
    lookup_type: part_lookup
    query: "I need air filters for a 2001 Ford f-150"
    followups: ["Mann-Filter C 21 035 applies to what years?", "a 2001 f-150", "List the exact trims it covers"]
    ground_truth: "UNVERIFIED in notes — checkable claims: engines 4.2/4.6/5.4L; Mann C 21 035 fits >1,000 configs incl. 31 on 2001 F-150; trims XL/XLT/Lariat/King Ranch/Harley-Davidson; whether cabin filters legitimately catalog to a truck with no factory cabin filter."
    observed_april: "Reasonable-looking pass; unverified."
    failure_class: pass
    notes: "Regression guard pending verification."

  # ---------- WAYNE EIFFIES (his scored 10-query plan) ----------

  - id: WE-01
    tester: Wayne Eiffies
    lookup_type: interchange
    query: "What other brands will interchange to CP579"
    followups: ["What is the difference between the Four Seasons parts?", "What is the country of origin for the Four Seasons 77552?", "Check cross-referenced parts to see if their COOs are listed."]
    ground_truth: "Interchange list correct (NAPA 275807, Omega 2003605R/2003605AM, Honda 38810PZA004, GPD 6512328, Four Seasons 78552 new / 77552 reman, Reach 74-22088, UAC CO3605AC). Four Seasons 77552 part page: Country of Origin (Primary) = Mexico."
    observed_april: "Interchanges + new/reman right; claimed COO 'not provided' for 77552 and all crosses."
    failure_class: attribute_miss
    notes: "Wayne score: PARTIAL FAILURE."

  - id: WE-02
    tester: Wayne Eiffies
    lookup_type: interchange
    query: "I have brake pads with part number 84847749. Can you provide some aftermarket choices?"
    followups: ["Which would be a carbon composition pad?", "Are there carbon fiber pads listed for a 2022 Chevrolet Tahoe Z71 with a 6.2L V8 engine?"]
    ground_truth: "Interchange list correct; final turn must surface PowerStop Z23-2173 (Front, exc. 6-piston) and Z23-2407 (Front, 6-piston), both Friction Material: Carbon Fiber Ceramic, in the 85-result 2022 Tahoe pad set."
    observed_april: "Denied any carbon-fiber pads exist for the vehicle."
    failure_class: false_negative
    notes: "Wayne score: PARTIAL FAILURE. Likely truncation-driven (85 results, 10 exposed)."

  - id: WE-03
    tester: Wayne Eiffies
    lookup_type: part_lookup
    query: "Fuel tank filler cap for a Toyota Tacoma."
    followups: ["2015", "What is the differences between the Stant fuel caps?", "Which Motorad fuel cap is best for my vehicle?"]
    ground_truth: "Stant 10504 is a LOCKING cap (POC called it standard). Caps: Stant 10504/10834/10869/17504; Motorad MGC206KA/MGC791SK/MGC900 (all locking)."
    observed_april: "Locking/standard attributes wrong on Stant 10504; MGC900 recommendation liked."
    failure_class: attribute_miss
    notes: "Wayne score: PARTIAL FAILURE."

  - id: WE-04
    tester: Wayne Eiffies
    lookup_type: part_lookup
    query: "Engine water pump for a Ford Explorer"
    followups: ["2020", "The engine is a six cylinder", "Not sure"]
    ground_truth: "Each pump is engine-specific. GMB 125-9190 and Motorcraft PW-620 fit ONLY the 2.3L L4 — must not appear in a V6 answer. V6 3.0L: GMB 125-3500, Motorcraft PW-659; V6 3.3L: Safety PW25-302, GMB 125-4030/125-3260, Motorcraft PW-532/PW-623/PW-674."
    observed_april: "'Limit of clarifying questions' reached → presented all 10 incl. L4-only parts as fitting 'both V6 engines'."
    failure_class: hallucinated_positive
    notes: "Wayne score: FAILURE. Also documents the question-cap leak verbatim."

  - id: WE-05
    tester: Wayne Eiffies
    lookup_type: part_lookup
    query: "Drive axle shaft seal for a 2014 Chevrolet Silverado 1500"
    followups: ["Rear shaft seal and truck is a 4WD", "What is the difference between the three seals?"]
    ground_truth: "Motive Gear 40038875, 16146CR, 4762N fit 1988-1999-era GM trucks, NOT a 2014 Silverado. Correct answer requires seals for the 2014 9.5\"/9.76\" rear axle."
    observed_april: "Confirmed the three wrong seals as 'the true rear axle shaft seals', retracted when probed, then re-offered them."
    failure_class: contradiction
    notes: "Wayne score: MAJOR FAILURE — 'Completely failed in confidence to select correct part.'"

  - id: WE-06
    tester: Wayne Eiffies
    lookup_type: vin
    query: "Front strut assembly for VIN 4S4BTACC6M3188809"
    ground_truth: "Decodes to 2021 Subaru Outback; should return front strut assemblies."
    observed_april: "Could not decode (VIN tool disabled)."
    failure_class: infra_failure
    notes: "Wayne score: FAILURE."

  - id: WE-07
    tester: Wayne Eiffies
    lookup_type: fitment
    query: "What vehicles will fit a DENSO 210-1084"
    followups: ["Does it fit all Toyota Tacoma from 2005 to 2015?", "Does it fit all engines for this year, make, model?", "2015"]
    ground_truth: "Alternator fits ONLY the 4.0L V6 (130A unit), NOT the 2.7L L4. Verify claimed 2009 gap."
    observed_april: "Claimed it fits both 2015 Tacoma engines."
    failure_class: hallucinated_positive
    notes: "Wayne score: FAILURE."

  - id: WE-08
    tester: Wayne Eiffies
    lookup_type: fitment
    query: "What vehicle will fit a Bosch 0445120238?"
    followups: ["Does this part only fit a 6.7L Cummins?", "What Bosch fuel injector will fit same make, model, year with a 5.9L Cummins?", "Ram 3500", "2007"]
    ground_truth: "Applications REVERSED by POC: 0445120238 = 5.9L Cummins ISB 325hp 2004-2009 Ram; 0445120193 = 6.7L Cummins 2007-2012 Ram."
    observed_april: "Stated exactly backwards on both parts."
    failure_class: hallucinated_positive
    notes: "Wayne score: FAILURE. Check both directions when scoring."

  - id: WE-09
    tester: Wayne Eiffies
    lookup_type: supersession
    query: "I have a part with a part number of 47730-01020 and I can't find part"
    followups: ["Is this part superseded to another part?", "Does part number 47730-12350 supersede to another part number?", "Toyota OEM", "Is there a part number found for 47730-12310?", "Was this part number superseded from any other part number?"]
    ground_truth: "Toyota Parts Center: 47730-12310 supersedes 47730-12350 and 47730-01020. OptiCat Online itself lacks the chain but crosses 47730-12310 → URG 451916 (HBWN). MCP should at minimum mirror Online."
    observed_april: "Mirrored Online (found URG cross, no supersession) — data-parity pass, real-world gap."
    failure_class: partial
    notes: "Wayne score: PARTIAL FAILURE — 'leans more towards OCOnline.' Data-parity case. (Same dialogue filed in Frank's section as FD-duplicate — run once.)"

  - id: WE-10
    tester: Wayne Eiffies
    lookup_type: supersession
    query: "I'm looking for a Denso part 228000-1953"
    followups: ["I can't find 228000-1953.", "Has part number 228000-1953 been changed?"]
    ground_truth: "Part page: Status Superseded; Superseded To = AS228000-1953 (both parts brand KVVR). The successor was IN the POC's own match list."
    observed_april: "Asserted no supersession; 'catalog still treats it as active/valid'."
    failure_class: false_negative
    notes: "Wayne score: FAILURE. The single cleanest supersession test — lifecycle fields exist and were unread."

  # ---------- KYLE JOHNSON ----------

  - id: KJ-01
    tester: Kyle Johnson
    lookup_type: part_lookup
    query: "F-150 oil filter"
    followups: ["2007", "5.4L"]
    ground_truth: "Bosch 3330 does NOT fit a 2007 F-150 (Kyle's annotation). Verify remainder of 5.4L list: Motorcraft FL-820S, FRAM PH2/XG2, WIX 51372, Purolator L24651/PL24651, Mobil 1 M1-210A, K&N HP-2010, Champ PH820XL."
    observed_april: "Included Bosch 3330 in the 5.4L answer."
    failure_class: hallucinated_positive
    notes: "Kyle: 'Some of the parts listed to fit a given vehicle do not actually fit that vehicle.'"

  # ---------- BRYAN THUESON ----------

  - id: BT-01
    tester: Bryan Thueson
    lookup_type: interchange
    query: "Find the interchange / cross references for engine oil filter PH400"
    followups: ["ph400 is the OE part number"]
    ground_truth: "69 engine-oil-filter interchanges in Online incl. Wix 51516, Purolator L20195, Fleetguard LF3339, Donaldson P550400, MAHLE OC 460, GK OF20195, Prime Guard POF241."
    observed_april: "Resolved PH400 to an OE Holland pintle hook; zero filter crosses."
    failure_class: false_negative
    notes: "The wrong-data-path case: exact PN match instead of competitive interchange query. [reconstructed opening]"

  - id: BT-02
    tester: Bryan Thueson
    lookup_type: interchange
    query: "give me the bosch equivelant of this Wagner rotor part number BD180678UC"
    followups: ["Give me fitment details."]
    ground_truth: "Bosch 25011617; Chevrolet Colorado 2016-2019, GMC Canyon 2015-2020."
    observed_april: "PASS — Bryan: 'Perfect test.'"
    failure_class: pass
    notes: "Regression guard. Keep the typo 'equivelant'."

  - id: BT-03
    tester: Bryan Thueson
    lookup_type: symptom
    query: "My truck is overheating when I idle, what part do I need?"
    followups: ["2020 F150", "platinum"]
    ground_truth: "All four fan parts verified fitting: Motorcraft RF-339, VDO FA71988, Agility 6010379, Global Parts 2811945."
    observed_april: "PASS — Bryan: 'Success.'"
    failure_class: pass
    notes: "Regression guard for symptom flows."

  - id: BT-04
    tester: Bryan Thueson
    lookup_type: symptom
    query: "The engine cranks but won't start - what should I replace?"
    followups: ["OK, what parts should I purchase for all these scenarios?", "3.5l ecoboost", "Yes, provide part numbers and brands", "4wd", "I do not have the vin.", "please do so"]
    ground_truth: "Per-part annotations: found-and-plausible (FDME-62, DU-1087/1088, RY-1199/1337/1198, SA-1102, WPT-1258); found-but-no-fitment-to-this-truck (PFS-1061, BXT-65-850); fits-F150-but-not-2020 (DG-549, SP-580); wrong identity (WPT-1452 is a Door Module Connector, not PCM); unverified (M-9407-35A, M-9593-35T, BXT-99R-590, PMGR-1)."
    observed_april: "~17 Motorcraft numbers claimed to fit ALL 2020 F-150 3.5L EB 4WD Platinum. Bryan: 'not something that works well at all.'"
    failure_class: partial
    notes: "Score per part. Every recommended part must pass a fitment check before being asserted."

  - id: BT-05
    tester: Bryan Thueson
    lookup_type: symptom
    query: "The steering wheel shakes at 60 mph - what parts might fix this?"
    followups: ["Please do so."]
    ground_truth: "Only 1 of the ~11 recommended Motorcraft numbers exists in Online (Bryan didn't record which — re-verify each: BRRF-176, BRF-1762, MES-3452/3451, MEF-398, MCS-251246/251245, ML3Z-3078-A/3079-A, HUB-422)."
    observed_april: "Nearly all recommended numbers don't exist in the catalog."
    failure_class: hallucinated_positive
    notes: "Context: same 2020 F-150 Platinum 4WD session."

  - id: BT-06
    tester: Bryan Thueson
    lookup_type: symptom
    query: "My truck pulls to the right when braking - what part is worn?"
    followups: ["2018 canyon with diesel engine", "caliper", "Denali, Crew Cab, 4wd, left"]
    ground_truth: "Front LEFT calipers: Dynamic Friction 331-47221, R1 Concepts CAL-47221. POC returned the -47220 (RIGHT side) numbers. Missing brands per Online: Wagner, PowerStop, UAP, Quality-Built, BPro Auto."
    observed_april: "Returned right-side parts labeled front-left."
    failure_class: qualifier_error
    notes: "Left/right qualifier is the scored dimension; completeness second."

  - id: BT-07
    tester: Bryan Thueson
    lookup_type: symptom
    query: "I need a new spark plug wire for my diesel pickup truck."
    followups: ["2018 GMC Canyon", "OK, maybe I need glow plugs", "OK, I have learned that I need a fuel injection throttle body.", "2.8L Duramax", "yes, do so."]
    ground_truth: "Correct redirect (diesel = glow plugs, no plug wires). Glow plug crosses for GM 55589332: Bosch 0250203017 (Obsolete), BWD 21137, Standard GP126. Throttle bodies all correct: GPD 1814266 (BKNH), Ranshu 43-ETB0275 (DHPJ), Omega ETB0275 (DHBX); other brands missing."
    observed_april: "PASS — Bryan: 'Great Job.'"
    failure_class: pass
    notes: "Regression guard. Check POC's Bosch glow plug 0250202134 vs Online's 0250203017."

  - id: BT-08
    tester: Bryan Thueson
    lookup_type: interchange
    query: "find cross references or interchange number for a part AP63546"
    followups: ["Alliant Power"]
    ground_truth: "Alliant Power not in the OptiCat brand directory — a correct negative."
    observed_april: "PASS — honest 'brand not in directory' with alternatives offered."
    failure_class: pass
    notes: "Heavy-duty coverage-gap guard. Future variant: with MSP Diesel key, 30273 → AP63546."

  - id: BT-09
    tester: Bryan Thueson
    lookup_type: attributes
    query: "show me all air filters for this vehicle with these attributes: Height 55.0 mm / Shape Other/Unique / Side A Length 284.0 mm / Side B Length 166.0 mm"
    ground_truth: "Only Denso 143-3709 carries those exact PAdb attributes. FR3Z-9601-A and FRAM CA11958 are NOT in the catalog. FA-1918 has one attribute (Height 2.29, unlabeled inches ≈58.2mm — not a match). WA10315 attributes in inches, different values."
    observed_april: "Returned 5 'exact matches' incl. two non-catalog parts."
    failure_class: hallucinated_positive
    notes: "Seed session with 2015 Mustang 3.7L V6 context first. Also the unit-normalization case (mm vs unlabeled inches)."

  - id: BT-10
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "Find a starter for a 2018 gmc canyon diesel"
    followups: ["yes"]
    ground_truth: "Exactly 1 result in Online for L4 2.8L: Quality-Built 16046 (reman). No Bosch part for this application."
    observed_april: "Invented Bosch SR8507X with full specs; never surfaced Quality-Built 16046."
    failure_class: hallucinated_positive
    notes: "The canonical double failure: invention + miss of the only correct part."

  - id: BT-11
    tester: Bryan Thueson
    lookup_type: attributes
    query: "show me the attributes of each cabin air filter that fits a 2020 camaro"
    followups: ["show me attributes for the CU24003 and 454-5079 and 1211405 and CA65350 and la1123 and ptc24211", "find part mann-filter cu24003"]
    ground_truth: "Online HAS data POC said was missing: MAHLE LA 1123 (12-mo warranty, HTS 8421398015, COO China, Available to Order); Sakura CA-65350 full description set. Only GPD 1211405 + Sakura CA-65350 attributes were returned; Mann CU24003 (BKNC), Denso 454-5079, MAHLE LA1123, Pro-Tec PTC24211 = 'No attributes found'."
    observed_april: "Attribute retrieval failed for 4 of 6 parts with data present."
    failure_class: attribute_miss

  - id: BT-12
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "air filter for a 2015 mustang"
    followups: ["3.7L V6", "Find the Bosch part number."]
    ground_truth: "22 results in Online incl. BOSCH 5703WS (V6 3.7L 3726cc, panel, high-velocity cellulose). Expected follow-up answer: 5703WS."
    observed_april: "Enumerated ten WS numbers, concluded Bosch makes none."
    failure_class: false_negative
    notes: "The flagship truncation-driven false negative."

  - id: BT-13
    tester: Bryan Thueson
    lookup_type: fitment
    query: "What is the year range of mustangs that this FA-1918 fits?"
    followups: ["yes"]
    ground_truth: "Online vehicle linkages: Ford Mustang 2015-2023."
    observed_april: "'Fits 0 vehicle configurations' then reasoned 2015-2017 from cross-references and car history."
    failure_class: contradiction
    notes: "Seed with the 2015 Mustang filter search context. Expected: 2015-2023."

  - id: BT-14
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "What oil filter fits a Cummins ISX15"
    ground_truth: "The four Mann-Filter parts returned (H 1810/1, H 34 1390 KIT, H 34 2090 KIT, H 925/2) exist but have ZERO vehicle linkages — no catalog basis for the fitment claim. Correct behavior: state no linked fitment exists."
    observed_april: "Presented unlinked parts as ISX15 fitments."
    failure_class: hallucinated_positive
    notes: "Fitment claims must be backed by linkage data, full stop."

  - id: BT-15
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "I need a Drive Belt tensioner assembly for a dd15 engine"
    followups: ["2018 freightliner cascadia", "fan and alternator.", "yes"]
    ground_truth: "Gates 38667 for fan+alternator on the 14.8L DD15; accessory-drive candidates Gates 38610/38569, Dayco 89440/89447/89448. Missing brands: Bando, Freshstart."
    observed_april: "PASS — Bryan: 'successful and exciting.'"
    failure_class: pass
    notes: "Regression guard; score brand completeness too."

  - id: BT-16
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "I need a moog version of a lower control arm for my GMC Canyon"
    followups: ["2018", "4wd, left", "Does this also fit the Denali version?", "please do so", "Then what does this moog part number RK623445 which is a lower control arm fit?"]
    ground_truth: "Online: MOOG CHASSIS PRODUCTS (BCCH) has 6 results for the 2018 Canyon incl. RK623445, Front Left Lower, ball joint + bushings included. RK642960/RK642961 are not confirmed in catalog."
    observed_april: "Invented RK642960/61 → denied Moog makes any → admitted RK623445 fits when asked directly."
    failure_class: contradiction
    notes: "The self-contradiction flagship. Expected first answer: RK623445."

  - id: BT-17
    tester: Bryan Thueson
    lookup_type: interchange
    query: "Show the cross reference for these brands for the same part number RK623445"
    ground_truth: "None of the returned crosses (Mevotech CMS501239, Dorman 527035/527-035/CB92043PR, MAS CB92043, Precision 11CB503209, GM 23490293/23224320/94772434/23467346) are shown fitting this vehicle in Online."
    observed_april: "Presented all 11 as 'confirmed matches for the same application'."
    failure_class: hallucinated_positive
    notes: "Continuation of BT-16 session context."

  - id: BT-18
    tester: Bryan Thueson
    lookup_type: interchange
    query: "Does Delphi have an equivalent for Moog RK623445?"
    followups: ["Yes, check the catalog for this same part and vehicle"]
    ground_truth: "Must terminate with a usable answer (Delphi options or a clean 'none') — not hang."
    observed_april: "Searched ~20 minutes with no result."
    failure_class: infra_failure
    notes: "[reconstructed opening]. Timeout/step-cap test. Tail of BT-16 session."

  - id: BT-19
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "What parts do I need to replace a water pump on a 2022 gmc Sierra?"
    followups: ["1500", "6.2l v8", "Yes", "denali only"]
    ground_truth: "Pumps: Gates 45072, US8206/US8206TS, GMB 130-3150/130-3170, Pierburg 7.07755.24.0, GNS YH-G244/YH-G233; gasket Fel-Pro 36110. Gap: never delivered part numbers for the thermostat/belt/coolant it promised."
    observed_april: "PASS on pumps+gasket; incomplete on job parts."
    failure_class: pass
    notes: "Regression guard; score job-completion too."

  - id: BT-20
    tester: Bryan Thueson
    lookup_type: part_lookup
    query: "I need a filter but forgot what it is called."
    followups: ["I think it is the one that filters the air inside my car.", "2015 ford mustang", "ecoboost"]
    ground_truth: "Cabin air filter; all 7 returned fit (ATP CF-281/FA-23, Interfil CFI-10195, GPD 1211440, Sakura CAC-19280, MAHLE LAK 1629/LA 1629); a few valid parts missing."
    observed_april: "PASS — Bryan: 'Good Job MCP.'"
    failure_class: pass
    notes: "The novice-flow regression guard."