Stephen Roller

About

I am currently a Research Scientist at Google DeepMind. My research and work primarily focuses on pretraining the largest-scale language models.

Previously, I worked as a Member of the Technical Staff at Character.AI, where I lead the pre-training and training infrastructure efforts, and as a Research Engineer at Meta's Fundamental AI Research (FAIR), where I was a leading member of the Large Language Model and dialogue teams. Much of my work from FAIR is open source.

Before FAIR, I completed my PhD in Computer Science at the University of Texas at Austin under the supervision of Katrin Erk.

Contact

I'm most easily reached via my personal email, me@stephenroller.com.

Publications

My scholarly publications appear below in reverse chronological order. Stars by names indicate joint first authorship. Due to the sensitivity of my work in pretraining, my work has been unpublished since I left FAIR.

FAIR (2017 — 2023)

• Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, Susan Zhang. A Theory on Adam Instability in Large-Scale Machine Learning. arXiv 2023.
• Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, Luke Zettlemoyer. Scaling Laws for Generative Mixed-Modal Language Models. arXiv 2023.
• FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, Markus Zijlstra. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 2022.
• Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, Jason Weston. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv 2022.
• Susan Zhang*, Stephen Roller*, Naman Goyal*, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. arXiv 2022.
• Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, Jason Weston. Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion. arXiv 2022.
• Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, Jason Weston. Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents. NLP for Conversational AI Workshop. (Best paper award).
• Moya Chen, Paul A. Crook, Stephen Roller. Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue. arXiv 2021.
• Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston. Hash Layers For Large Sparse Models. NeurIPS 2021. (Spotlight presentation).
• Da Ju, Stephen Roller, Sainbayar Sukhbaatar, Jason Weston. Staircase Attention for Recurrent Processing of Sequences. arXiv 2021.
• Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, Angela Fan. Not All Memories are Created Equal: Learning to Expire. ICML 2021.
• Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, Claire Cardie. Adding Chit-Chats to Enhance Task-Oriented Dialogues. NAACL 2021.
• Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston. Recipes for building an open-domain chatbot. EACL 2021.
• Stephen Roller*, Y-Lan Boureau*, Jason Weston*, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, Pratik Ringshia, Kurt Shuster, Eric Michael Smith, Arthur Szlam, Jack Urbanek, Mary Williamson. Open-domain conversational agents: current progress, open problems, and future directions. arXiv 2020.
• Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, Jason Weston. Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training. ACL 2020.
• Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, Jason Weston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. ACL 2020.
• Margaret Li, Jason Weston, Stephen Roller. ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons. NeurIPS workshop on Conversational AI 2019.
• Sean Welleck*, Ilia Kulikov*, Stephen Roller, Emily Dinan, Kyunghyun Cho, Jason Weston. Neural Text Generation with Unlikelihood Training. ICLR 2020.
• Abigail See, Stephen Roller, Douwe Kiela, Jason Weston. What makes a good conversation? How controllable attributes affect human judgments. NAACL 2019.
• Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, Maximilian Nickel. Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings. ACL 2019.
• Emily Dinan*, Stephen Roller*, Kurt Shuster*, Angela Fan, Michael Auli, Jason Weston. Wizard of Wikipedia: Knowledge-Powered Conversational agents. ICLR 2019.
• Stephen Roller, Douwe Kiela, Maximilian Nickel. Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora. ACL 2018.

PhD (2010 — 2017)

• Su Wang, Stephen Roller, Katrin Erk. Distributional modeling on a diet: One-shot word learning from text only. IJCNLP 2017.
• Stephen Roller. Identifying Lexical Relationships and Entailments with Distributional Semantics. PhD dissertation 2017. (Corresponding proposal)
• Stephen Roller, Katrin Erk. Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment. EMNLP 2016.
• Iz Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk, Raymond Mooney. Representing Meaning with a Combination of Logical and Distributional Models. Special Issue of Computational Linguistics on Formal Distributional Semantics, 2016.
• Stephen Roller, Katrin Erk. PIC a Different Word: A Simple Model for Lexical Substitution in Context. NAACL 2016.
• Ye Zhang, Stephen Roller, Byron Wallace. MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification. NAACL 2016.
• Stephen Roller, Katrin Erk, Gemma Boleda. Inclusive yet Selective: Supervised distributional hypernymy detection. COLING 2014.
• Iz Beltagy, Stephen Roller, Gemma Boleda, Katrin Erk, Raymond Mooney. UTexas: Natural Language Semantics using Distributional Semantics and Probabilistic Logic. SemEval 2014.
• Stephen Roller, Sabine Schulte im Walde. Feature Norms of German Noun Compounds. Multiword Expression Workshop of EACL 2014.
• Stephen Roller, Sabine Schulte im Walde. A Multimodal LDA Model integrating Textual, Cognitive and Visual Modalities. EMNLP 2013.
• Stephen Roller, Sabine Schulte im Walde and Silke Scheible. The (Un)expected Effects of Applying Standard Cleansing Models to Human Ratings on Compositionality. Multiword Expression Workshop of NAACL 2013.
• Sabine Schulte im Walde, Stefan Müller and Stephen Roller. Exploring Vector Space Models to Predict the Compositionality of German Noun-Noun Compounds. StarSem 2013.
• Stephen Roller, Mike Speriousu, Sarat Rallapalli, Ben Wing and Jason Baldridge. Supervised Text-based Geolocation Using Language Models on an Adaptive Grid. EMNLP 2012.

Undergrad (2010)

• Mike Dominguez, R Michael Young and Stephen Roller. Design and Evaluation of Afterthought, A System that Automatically Creates Highlight Cinematics for 3D Games. AIIDE 2011.
• Mike Dominguez, R Michael Young and Stephen Roller. Automatic Identification and Generation of Highlight Cinematics for 3D games. FDG 2011.