Best Practices for Auto QA Configuration

​

General Configuration Principles

• Maintain diversity and realism in metric configurations.
• Ensure configurations reflect plausible agent responses.
• Review and update configurations regularly to reflect current use cases.

​

Sentence Length and Complexity

• Limit configurations to concise utterances — typically 1-3 sentences.
• Avoid configuring very long sentences or extensive text passages.

​

Handling Specific Identifiers

​

Company and Brand Names

• Include when necessary, but be aware they can impact semantic adherence.
• Use a moderate threshold (for example, 50-60% match) to accommodate variations.

​

Agent Names

• Minimize the use of specific agent names in configurations.
• Use [Agent Name] as a placeholder where the agent references themselves or other agents.

​

Product Names and Model IDs

• Avoid including specific model identifiers unless necessary.
• If included, expect precise usage in agent responses.
• Use a moderate threshold (for example, 50-60% match) for these configurations.

​

Threshold Recommendations

​

Testing and Validation

• Validate configurations against real agent responses.
• Monitor and analyze adherence rates to identify areas for reconfiguration.