Long context and summarization
Summary compression ratio
Implemented
long-context.compression_ratioDefinition
How much shorter summaries are than their sources: source words / summary words (Newsroom compression), the mean over documents, with its median.
Formula
mean_d |source_d| / |summary_d|
Range: [1, ∞) typically
Inputs and outputs
- sources: see the signature of es.compression_ratio
- summaries: see the signature of es.compression_ratio
Returns: MetricResult (value plus counts, intervals and breakdowns in params)
Assumptions
No assumptions beyond valid, aligned inputs of the documented types.
Limitations
No metric-specific limitations are documented yet. Interpret the value alongside the task, data, and other metrics.
Python API
import evalsuite as es
es.compression_ratio(["a long source text here"], ["short"])References
- Grusky M, Naaman M, Artzi Y. Newsroom: a dataset of 1.3 million summaries with diverse extractive strategies. NAACL. 2018:708-719.