(view Google Scholar for a complete, up-to-date list.)
World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds
Shannon Zejiang Shen*, Yapei Chang*, Qiyao Liang*, Benjamin Newman*, Jenny Liang*, Rulin Shao*, Ziqiao Ma, Junjie Oscar Yin, Yung-Sung Chuang, Weijia Shi, Yejin Choi, Luke Zettlemoyer, Pang Wei Koh, Mohit Iyyer, Tongshuang Wu, David Sontag
Preprint 2026
[pdf] [site] [tldr]
We design a framework for studying collaborative multi-agent software engineering, focusing on communication, ownership and resource allocation.
Adapting Language Models for Microbiome Prediction and Scientific Reasoning
Benjamin Newman*, Ahnjae Shin*, Yifan Gao, Pan Lu, Seungju Han, Chiho Im, Sean Spencer, Kerwyn Casey Huang, Justin L. Sonnenburg, Yejin Choi
Preprint 2026
[pdf] [tldr]
We propose datasets to adapt a pretrained language model to support both microbiome classification and scientific reasoning abilities, and introduce new microbiome-focused benchmarks.
Realize What Matters: Principled Context Representation for Large-Scale Reasoning
Michael Theologitis, Dean Light, Shuyue Stella Li, Benjamin Newman, Yulia Tsvetkov, Dan Suciu
Preprint 2026
[pdf] [code] [tldr]
We design a harness for document corpus reasoning tasks, in which the model iteratively extracts task-relevant context, structures the outputs and synthesizes them into a final answer. We outperform claude code with a 35B open-weight model.
Deep Reasoning in General Purpose Agents via Structured Meta-Cognition
Dean Light, Michael Theologitis, Kshitish Ghate, Shuyue Stella Li, Benjamin Newman, Chirag Shah, Aylin Caliskan, Pang Wei Koh, Dan Suciu, Yulia Tsvetkov
Neurips 2026
[pdf] [code] [tldr]
We create a formal language for capturing human-like meta-reasoning and build an agent system that implements this language. This agent enables just-in-time agentic metareasoning scaffolds and leads to improved performance across four different reasoning-intensive tasks.
FDARxBench: Benchmarking regulatory and clinical reasoning on FDA generic drug assessment
Betty Xiong, Jillian Fisher, Benjamin Newman, Meng Hu, Shivangi Gupta, Yejin Choi, Lanyan Fang, Russ B Altman
EMNLP 2026
[pdf] [code] [tldr]
We introduce a new complex document question-answering benchmark for FDA drug review task with input from drug review profesionals.
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi
EMNLP 2026
[pdf] [tldr]
We introduce two practical techniques to generate better synthetic data for knowledge learning: generating synthetic question-answer pairs and focal document rewriting.
Spectrum tuning: Post-training for distributional coverage and in-context steerability
Taylor Sorensen, Benjamin Newman, Jared Moore, Chan Young Park, Jillian Fisher, Niloofar Mireshghallah, Liwei Jiang, Yejin Choi
ICLR 2026
[pdf] [code] [models] [tldr]
We contribute a novel dataset and post-training method to improve in-context steerability and distributional alignment and coverage, and characterize weaknesses to current post-training techniques along these desiderata.
The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality
Benjamin Newman, Abhilasha Ravichander, Jaehun Jung, Rui Xin, Hamish Ivison, Yegor Kuznetsov, Pang Wei Koh, Yejin Choi
Preprint 2025
[pdf] [code] [tldr]
We show evidence that training on knowledge models believe to be factual can help reduce hallucinations in long-form generation tasks more effectively than training on factual knowledge.
Verifying the verifiers: Unveiling pitfalls and potentials in fact verifiers
Wooseok Seo, Seungju Han, Jaehun Jung, Benjamin Newman, Seungwon Lim, Seungbeen Lee, Ximing Lu, Yejin Choi, Youngjae Yu
COLM 2025
[pdf] [code] [tldr]
We share three findings intended to guide future development of more robust fact verifiers: address annotation errors, frontier LLMs with few-shot examples are strong baselines, and small models still struggle with complex reasoning.
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, Kyle Lo
EMNLP 2023
[pdf] [tldr]
We propose a question-answering framework for decontextualizing snippets from scientific documents and investigate the performance of state-of-the-art LLMs under this framework.
Conducted during a PYIship at Semantic Scholar
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, Kyle Lo
EMNLP 2023
[pdf] [tldr]
We propose a question-answering framework for decontextualizing snippets from scientific documents and investigate the performance of state-of-the-art LLMs under this framework.
Conducted during a PYIship at Semantic Scholar
Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Liye Fu, Benjamin Newman, Maurice Jakesch, Sarah Kreps
CHI 2023
[pdf] [tldr]
LLMs can best help as email writing assistants for legislative staff members when they provide message-level suggestions compared to sentence-level ones.
P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
Benjamin Newman, Prafulla Kumar Choubey, Nazneen Rajani
ICLR 2022 (Poster)
[pdf][tldr]
Querying Pretrained NLP models with semantically equivalent prompts (e.g. "[MASK] is Canada's capital" and "Canada, which has capital city [MASK]") do not necessarily lead to the same predictions. Can low-parameter fine-tuning methods increase model consistency? Yes! And they don't need as many annotations as other methods.
Conducted during an internship at Salesforce Research
Refining Targeted Syntactic Evaluation
Benjamin Newman, Kai Siang-Ang, Julia Gong, John Hewitt
NAACL 2021
[pdf] [code] [cite] [blog]
[tldr]
How should we evaluate the syntactic understanding of NLP systems? We argue for evaluating models' likely behavior and systematicty, and build off of minimal pair evaluation to address these goals. We find models are better at conjugating verbs they deem likely.
The EOS Decision and Length Extrapolation
Benjamin Newman, John Hewitt, Percy Liang and Chris Manning
Blackbox NLP@EMNLP 2020 (Outstanding Paper)
[pdf] [code] [cite]
[tldr]
Why do sequence models struggle to extrapolate? For many reasons, but the decision to train models with End of Sequence tokens at the end of training examples is one of them. We investigate and visualize the effect that this decision has on neural models' extrapolative abilities.
