A Distributional and Orthographic Aggregation Model for English Derivational Morphology. - Daniel Deutsch; John Hewitt; Dan Roth 001 (ACL (1), 2018)

A General-Purpose Algorithm for Constrained Sequential Inference. - Daniel Deutsch; Shyam Upadhyay; Dan Roth 001 (CoNLL, 2019)

Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization. - Daniel Deutsch; Dan Roth 001 (EMNLP/IJCNLP (1), 2019)

Is Killed More Significant than Fled? A Contextual Model for Salient Event Detection. - Disha Jindal; Daniel Deutsch; Dan Roth 001 (COLING, 2020)

SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics. - Daniel Deutsch; Dan Roth 001 (CoRR, 2020)

Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries. - Daniel Deutsch; Dan Roth 001 (CoRR, 2020)

Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries. - Daniel Deutsch; Dan Roth 001 (CoNLL, 2021)

Question-Based Salient Span Selection for More Controllable Text Summarization. - Daniel Deutsch; Dan Roth 001 (CoRR, 2021)

Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. - Daniel Deutsch; Tania Bedrax-Weiss; Dan Roth 001 (Trans. Assoc. Comput. Linguistics, 2021)

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. - Daniel Deutsch; Rotem Dror; Dan Roth 001 (Trans. Assoc. Comput. Linguistics, 2021)

Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics. - Daniel Deutsch; Dan Roth 001 (ACL (Findings), 2022)

On the Limitations of Reference-Free Evaluations of Generated Text. - Daniel Deutsch; Rotem Dror; Dan Roth 001 (EMNLP, 2022)

Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics. - Daniel Deutsch; Rotem Dror; Dan Roth 001 (NAACL-HLT, 2022)

Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code. - Daniel Deutsch; Dan Roth 001 (CoRR, 2022)

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. - Sebastian Gehrmann; Abhik Bhattacharjee; Abinaya Mahendiran; Alex Wang; Alexandros Papangelis; Aman Madaan; Angelina McMillan-Major; Anna Shvets; Ashish Upadhyay; Bingsheng Yao; Bryan Wilie; Chandra Bhagavatula; et al. (CoRR, 2022)

Needle in a Haystack: An Analysis of Finding Qualified Workers on MTurk for Summarization. - Lining Zhang; João Sedoc; Simon Mille; Yufang Hou 001; Sebastian Gehrmann; Daniel Deutsch; Elizabeth Clark; Yixin Liu 003; Miruna Clinciu; Saad Mahamood; Khyathi Raghavi Chandu (CoRR, 2022)

A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization. - Lining Zhang; Simon Mille; Yufang Hou 001; Daniel Deutsch; Elizabeth Clark; Yixin Liu 003; Saad Mahamood; Sebastian Gehrmann; Miruna Clinciu; Khyathi Raghavi Chandu; João Sedoc (ACL (1), 2023)

Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selection. - Daniel Deutsch; Dan Roth 001 (EACL, 2023)

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration. - Daniel Deutsch; George F. Foster; Markus Freitag (EMNLP, 2023)

The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics. - Christoph Leiter; Juri Opitz; Daniel Deutsch; Yang Gao 021; Rotem Dror; Steffen Eger (Eval4NLP, 2023)

Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level. - Daniel Deutsch; Juraj Juraska; Mara Finkelstein; Markus Freitag (WMT, 2023)

The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. - Patrick Fernandes; Daniel Deutsch; Mara Finkelstein; Parker Riley; André F. T. Martins; Graham Neubig; Ankush Garg; Jonathan H. Clark; Markus Freitag; Orhan Firat (WMT, 2023)

Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent. - Markus Freitag; Nitika Mathur; Chi-kiu Lo; Eleftherios Avramidis; Ricardo Rei; Brian Thompson 001; Tom Kocmi; Frédéric Blain; Daniel Deutsch; Craig Stewart; Chrysoula Zerva; Sheila Castilho; et al. (WMT, 2023)

MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task. - Juraj Juraska; Mara Finkelstein; Daniel Deutsch; Aditya Siddhant; Mehdi Mirzazadeh; Markus Freitag (WMT, 2023)

Quality Estimation Using Minimum Bayes Risk. - Subhajit Naskar; Daniel Deutsch; Markus Freitag (WMT, 2023)

There's No Data like Better Data: Using QE Metrics for MT Data Filtering. - Jan-Thorsten Peter; David Vilar; Daniel Deutsch; Mara Finkelstein; Juraj Juraska; Markus Freitag (WMT, 2023)

Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation. - Daniel Deutsch; George F. Foster; Markus Freitag (CoRR, 2023)

Pinpoint, Not Criticize: Refining Large Language Models via Fine-Grained Actionable Feedback. - Wenda Xu; Daniel Deutsch; Mara Finkelstein; Juraj Juraska; Biao Zhang; Zhongtao Liu; William Yang Wang; Lei Li 005; Markus Freitag (CoRR, 2023)

On the Role of Summary Content Units in Text Summarization Evaluation. - Marcel Nawrath; Agnieszka Nowak; Tristan Ratz; Danilo C. Walenta; Juri Opitz; Leonardo F. R. Ribeiro; João Sedoc; Daniel Deutsch; Simon Mille; Yixin Liu 003; Sebastian Gehrmann; Lining Zhang; et al. (NAACL (Short Papers), 2024)

Finding Replicable Human Evaluations via Stable Ranking Probability. - Parker Riley; Daniel Deutsch; George F. Foster; Viresh Ratnakar; Ali Dabirmoghaddam; Markus Freitag (NAACL-HLT, 2024)

LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback. - Wenda Xu; Daniel Deutsch; Mara Finkelstein; Juraj Juraska; Biao Zhang; Zhongtao Liu; William Yang Wang; Lei Li 005; Markus Freitag (NAACL-HLT (Findings), 2024)

Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy. - Brian Thompson 001; Nitika Mathur; Daniel Deutsch; Huda Khayrallah (WMT, 2024)

Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task. - Markus Freitag; Nitika Mathur; Daniel Deutsch; Chi-kiu Lo; Eleftherios Avramidis; Ricardo Rei; Brian Thompson 001; Frédéric Blain; Tom Kocmi; Jiayi Wang; David Ifeoluwa Adelani; Marianna Buchicchio; et al. (WMT, 2024)

MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. - Juraj Juraska; Daniel Deutsch; Mara Finkelstein; Markus Freitag (WMT, 2024)

Mitigating Metric Bias in Minimum Bayes Risk Decoding. - Geza Kovacs; Daniel Deutsch; Markus Freitag (WMT, 2024)

Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data. - Zhongtao Liu; Parker Riley; Daniel Deutsch; Alison Lui; Mengmeng Niu; Apurva Shah; Markus Freitag (WMT, 2024)

Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data. - Zhongtao Liu; Parker Riley; Daniel Deutsch; Alison Lui; Mengmeng Niu; Apu Shah; Markus Freitag (CoRR, 2024)

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set. - Mara Finkelstein; Daniel Deutsch; Parker Riley; Juraj Juraska; Geza Kovacs; Markus Freitag (CoRR, 2024)

WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. - Daniel Deutsch; Eleftheria Briakou; Isaac Rayburn Caswell; Mara Finkelstein; Rebecca Galor; Juraj Juraska; Geza Kovacs; Alison Lui; Ricardo Rei; Jason Riesa; Shruti Rijhwani; Parker Riley; et al. (ACL (Findings), 2025)

Enhancing Human Evaluation in Machine Translation with Comparative Judgement. - Yixiao Song; Parker Riley; Daniel Deutsch; Markus Freitag (ACL (1), 2025)

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation. - Muhammed Yusuf Kocyigit; Eleftheria Briakou; Daniel Deutsch; Jiaming Luo; Colin Cherry; Markus Freitag (CoRR, 2025)

Enhancing Human Evaluation in Machine Translation with Comparative Judgment. - Yixiao Song; Parker Riley; Daniel Deutsch; Markus Freitag (CoRR, 2025)

SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? - Senyu Li; Jiayi Wang; Felermino D. M. A. Ali; Colin Cherry; Daniel Deutsch; Eleftheria Briakou; Rui Sousa-Silva; Henrique Lopes Cardoso; Pontus Stenetorp; David Ifeoluwa Adelani (CoRR, 2025)

Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation. - Colten DiIanni; Daniel Deutsch (CoRR, 2025)

Generating Difficult-to-Translate Texts. - Vilém Zouhar; Wenda Xu; Parker Riley; Juraj Juraska; Mara Finkelstein; Markus Freitag; Daniel Deutsch (CoRR, 2025)

Deconstructing Self-Bias in LLM-generated Translation Benchmarks. - Wenda Xu; Sweta Agrawal; Vilém Zouhar; Markus Freitag; Daniel Deutsch (CoRR, 2025)

Searching for Difficult-to-Translate Test Examples at Scale. - Wenda Xu; Vilém Zouhar; Parker Riley; Mara Finkelstein; Markus Freitag; Daniel Deutsch (CoRR, 2025)

MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation. - Parker Riley; Daniel Deutsch; Mara Finkelstein; Colten DiIanni; Juraj Juraska; Markus Freitag (CoRR, 2025)

MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task. - Juraj Juraska; Tobias Domhan; Mara Finkelstein; Tetsuji Nakagawa; Geza Kovacs; Daniel Deutsch; Pidong Wang; Markus Freitag (CoRR, 2025)