From Bright Pattern Documentation
Jump to: navigation, search

AI Evaluators

Evaluators ensure that AI Flows provide factual answers rooted in existing knowledge by tracking, reporting, and mitigating hallucinations and otherwise inadequate or non-factual answers.

An AI Evaluator rejecting a response and providing feedback to the LLM


There are two types of Evaluators:

Approvers

Approvers, or AI Evaluators in Approve mode, analyze tool calls at specified execution steps, including the query, the content retrieved, and the results generated. Approvers evaluate response quality in real time before it is sent to the customer, and can force the following actions:

  • Regenerate the response
  • Ask clarifying questions
  • Call a different tool
  • Retry the tool call with corrected parameters
  • Block the response or action, and deliver a fallback message
  • Escalate to a human agent

Approved actions are executed and approved responses are delivered to the customer as is. Rejected responses require an explanation and corrective recommendation from the Evaluator. The AI Agent takes this feedback into account and generates revised responses, until a response is approved or the maximum number of retry attempts is exceeded.

Reporters

Reporters, or AI Evaluators in Report mode, process historical interactions to report detected hallucinations or inadequate answers. Reporters can provide the following information:

  • Conversation steps where a hallucination was detected
  • Hallucination types
  • Explanations of why something was considered a hallucination
  • Requested metric values
  • Decisions made (e.g. score is insufficient)
  • Actions taken (e.g. accepted, rejected, or escalated)

Reach out to your service provider to request custom AI Evaluators.

< Previous | Next >