Factuality and QA
Answer relevance
Implemented
factuality.answer_relevanceDefinition
How directly an answer addresses the question: mean cosine similarity between the question's embedding and embeddings of questions regenerated from the answer (RAGAS answer relevancy).
Formula
mean_k cos(e(question), e(regenerated question_k))
Range: [-1, 1]
Inputs and outputs
- question_embeddings: (n, dim)
- generated_question_embeddings: per question, embeddings of regenerated questions
Returns: MetricResult (float, or per-example array with average=None)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.answer_relevance(question_embeddings, regenerated_embeddings)References
- Es S, James J, Espinosa-Anke L, Schockaert S. RAGAS: automated evaluation of retrieval augmented generation. EACL (demos). 2024:150-158.