fix(ai): recover query matches outside the selected category - #783
fix(ai): recover query matches outside the selected category#783izadoesdev wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Unkey Deploy
|
izadoesdev
changed the title
1 participant
A category-scoped search can miss an existing analytics builder. For example,
retentionunder Custom Events returns zero results even thoughidentified_profile_retentionexists under Profiles. Discovery now returns separately labeledoutsideCategorymatches after a scoped keyword miss, preserving the scoped count, actual category labels, substring matching and full query contracts.The final change is limited to the discovery tool and its regression tests: 13 production lines added, six removed. It uses the existing catalog and performs no I/O.
Validation: root lint, all 33 typecheck tasks and the full 27-task test suite pass. The AI package reports 724 passed, 229 skipped, zero failed. Three behavioral checks reproduce failures on the frozen reference and pass here; native contract coverage also verifies selectors, ordering, genuine misses, complete browsing and existing search behavior. Independent read-only review found no actionable regressions.
We ran 48 live attempts across two frozen paired experiments with the same configured
openai/gpt-5.6-terragateway alias, native MCP configuration, tool schemas and synthetic data. The first experiment recovered retention in 4/4 attempts versus 0/4. The second compared catalog compaction against the final simpler fallback:Compaction repeatedly required three discovery calls for property questions versus one with full contracts. The final version emits more discovery-response bytes but uses 15.7% fewer input tokens in the paired sample. Cache usage differs, so this is not an established monetary saving. The fixed tool-definition payload remains about 50 KB per model step.
Each experiment uses six known synthetic cases with two repetitions, alternating which version runs first and reversing case order for repeat two. SQL/schema/profile alternatives are deliberately unavailable, so the compact version's missing checkout answer does not prove a production regression. All attempts and traces are retained; no failures were selectively replaced. This is a small diagnostic sample, not a production success rate or a live Sazabi comparison.
Scope and integration: based on current staging
a080d7abd, with no source overlap or merge dependency on #782 or #751. Both evaluation arms explicitly compose #782's earlier conversation/schema repair atf61250448; only discovery differs within each pair. Final source is identical to the evaluated fallback (SHA-256d76442b4407b18fe56c2c711fab7d981ae090311ec33d0c5bd4e95dabc6c3a66). No model, main prompt, authorization, storage or deployment changes.AI-assisted implementation and evaluation, directed by the maintainer. Draft for review.