Validate claims before using them.
Separate extraction from validation. Check the citation location, quoted text and support for the claim. Keep rejected and uncertain evidence visible.
Independent product design
VendorLens
Evidence a reviewer can trace. Decisions a human can own.
- Ownership
- Problem framing, product scope and system design
- User
- Third-party risk analyst
- Scope
- New-vendor onboarding
Vendor documents can disagree, omit material facts or contain claims the source does not support. A fluent summary does not resolve those problems.
Designed a workflow that turns documents into a citation-backed evidence pack, policy results and a recommendation. The analyst makes the final decision.
Follow one case
A questionnaire says a control is enabled. An assurance report lists an exception. What should the system do?
Illustrative design walkthrough, not a live assessment.
Step 1 of 6
Preserve what was submitted.
Keep the questionnaire and assurance report as immutable originals, each with its own identity and version.
Design choice: Every downstream claim must lead back to the exact source used.
Output: two source documents, preserved separately.
What if a tool fails?
A tool failure is a workflow state, not evidence that the vendor failed a policy.
Record the failure and checkpoint. Retry within allowed limits; if unresolved, block completion. A failed policy evaluation cannot become a ready-for-review package.
System architecture
The full review flow: traceable evidence, controlled investigation and a human decision.
Product decisions
Preserve disagreements.
A source-supported claim can still conflict with another source. Record both; do not let reconciliation silently choose a convenient answer.
Keep policy outside the model.
Apply versioned rules to an exact evidence-ledger revision. New evidence triggers reevaluation; missing evidence must not become an automatic pass.
Separate reasoning from permission.
The agent chooses an investigation action. The harness validates and records it. Independent controls enforce permissions, approvals, budgets and retry limits before execution.
Make completion a checked state.
Gate the review package on successful policy evaluation, current evidence references and required findings. An agent saying “done” is not enough.
Evaluation design
Primary risk: an unsupported “proceed” recommendation.
Offline golden cases cover contradictions, missing evidence, scanned-table exceptions, lost qualifiers, wrong policy versions and prohibited actions.
Measure citation correctness, missed gaps, false-pass recommendations, recovery behavior and human corrections. Track analyst review time and clarification rounds alongside quality.