Publications

My Google Scholar profile
My ACL Anthology profile
My Semantic Scholar profile

2022

Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma, Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, Daan Van Esch, Vera Axelrod, Simran Khanuja, Jonathan H Clark, Orhan Firat, Sebastian Ruder, Jason Riesa, Melvin Johnson (2022). XTREME-S: Evaluating Cross-lingual Speech Representations. In Proceedings of Interspeech 2022.

Xinyi Wang, Sebastian Ruder, Graham Neubig (2022). Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation. In Proceedings of ACL 2022.

Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, Sebastian Ruder (2022). One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia. In Proceedings of ACL 2022.

Michael Tänzer, Sebastian Ruder, Marek Rei (2022). Memorisation versus Generalisation in Pre-trained Language Models. In Proceedings of ACL 2022.

Sebastian Ruder*, Ivan Vulić*, Anders Søgaard* (2022). Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold. In Findings of ACL 2022.

Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, Clara E. Rivera (2022). Writing system and speaker metadata for 2,800+ language varieties. In Proceedings of LREC 2022.

Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Said Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Anuoluwapo Aremu, Saheed Abdul, Pavel Brazdil (2022). Naijasenti: A nigerian twitter sentiment corpus for multilingual sentiment analysis. In Proceedings of LREC 2022.

Yi Tay*, Vinh Q. Tran*, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler (2021). Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. In Proceedings of ICLR 2022.

Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, Donald Metzler (2022). ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning. In Proceedings of ICLR 2022.

2021

Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, Phil Blunsom (2021). Mind the gap: Assessing temporal generalization in neural language models. In Proceedings of NeurIPS 2021.

Stefan Daniel Dumitrescu, Petru Rebeja, Beata Lorincz, Mihaela Gaman, Andrei Avram, Mihai Ilie, Andrei Pruteanu, Adriana Stan, Lorena Rosia, Cristina Iacobescu, Luciana Morogan, George Dima, Gabriel Marchidan, Traian Rebedea, Madalina Chitez, Dani Yogatama, Sebastian Ruder, Radu Tudor Ionescu, Razvan Pascanu, Viorica Patraucean (2021). LiRo: Benchmark and leaderboard for Romanian language tasks. In Proceedings of NeurIPS 2021 Datasets and Benchmarks Track.

Sebastian Ruder, Avirup Sil (2021). Multi-domain multilingual question answering. In Proceedings of EMNLP 2021: Tutorial Abstracts.

Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, Melvin Johnson (2021). XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. In Proceedings of EMNLP 2021.

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder (2021). UNKs Everywhere: Adapting Multilingual Language Models to New Scripts. In Proceedings of EMNLP 2021.

Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Leylia Khodra, Ayu Purwarianti, Pascale Fung (2021). IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation. In Proceedings of EMNLP 2021.

Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, Anna Korhonen (2021). MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer. In Findings of EMNLP 2021.

Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig (2021). Efficient Test Time Adapter Ensembling for Low-resource Language Varieties. In Findings of EMNLP 2021.

Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder (2021). Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Proceedings of NeurIPS 2021.

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson (2021). Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Proceedings of ACL 2021.

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. In Proceedings of ACL 2021.

Xinyi Wang, Sebastian Ruder, Graham Neubig (2021). Multi-view Subword Regularization. In Proceedings of NAACL 2021.

Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, Sebastian Ruder (2021). Rethinking embedding coupling in pre-trained language models. In Proceedings of ICLR 2021.

Yi Tay*, Mostafa Dehghani*, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler (2021). Long Range Arena: A Benchmark for Efficient Transformers. In Proceedings of ICLR 2021.

Nicolas Garneau, Mareike Hartmann, Anders Sandholm, Sebastian Ruder, Ivan Vulić, Anders Søgaard (2021). Analogy Training Multilingual Encoders. In Proceedings of AAAI 2021.

Angeliki Lazaridou*, Adhiguna Kuncoro*, Elena Gribovskaya*, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama, Kris Cao, Tomas Kocisky, Susannah Young, Phil Blunsom (2021). Pitfalls of Static Language Modelling. arXiv preprint arXiv:2102.01951.

*: Equal contribution.

2020

Paula Czarnowska, Sebastian Ruder, Ryan Cotterell, Ann Copestake (2020). Morphologically Aware Word-Level Translation. In Proceedings of COLING 2020.

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych (2020). AdapterHub: A Framework for Adapting Transformers. In Proceedings of EMNLP 2020: System Demonstrations. Code

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder (2020). MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of EMNLP 2020. Code

Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic (2020). AxCell: Automatic Extraction of Results from Machine Learning Papers. In Proceedings of EMNLP 2020. Code

Ivan Vulić*, Sebastian Ruder*, Anders Søgaard* (2020). Are All Good Word Vector Spaces Isomorphic? In Proceedings of EMNLP 2020. Code

Junjie Hu*, Sebastian Ruder*, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson (2020). XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. In Proceedings of ICML 2020. Code, website

Aida Nematzadeh*, Sebastian Ruder*, Dani Yogatama∗ (2020). On Memory in Human and Artificial Language Processing Systems. In Proceedings of the Bridging AI and Cognitive Science Workshop at ICLR 2020.

Mikel Artetxe*, Sebastian Ruder*, Dani Yogatama, Gorka Labaka, Eneko Agirre (2020). A Call for More Rigor in Unsupervised Cross-lingual Learning. In Proceedings of ACL 2020.

Mikel Artetxe, Sebastian Ruder, Dani Yogatama  (2020). On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020. XQuAD data

2019

Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, Dani Yogatama (2019). Episodic Memory in Lifelong Language Learning. In Proceedings of NeurIPS 2019.

Julian Eisenschlos*, Sebastian Ruder*, Piotr Czapla*, Marcin Kadras*, Sylvain Gugger and Jeremy Howard (2019). MultiFiT: Efficient Multi-lingual Language Model Fine-tuning. In Proceedings of EMNLP 2019.

Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell and Ann Copestake (2019). Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction. In Proceedings of EMNLP 2019.

Jonas Pfeiffer*, Aishwarya Kamath*, Iryna Gurevych, Sebastian Ruder (2019). What do Deep Networks Like to Read? arXiv preprint arXiv:1909.04547.

Sebastian Ruder, Anders Søgaard, Ivan Vulić (2019). Unsupervised Cross-Lingual Representation Learning. In Proceedings of ACL 2019: Tutorial Abstracts, pages 31–38. Slides.

Sebastian Ruder, Ivan Vulić, Anders Søgaard (2019). A Survey Of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research.

Anders Søgaard, Ivan Vulić, Sebastian Ruder, Manaal Faruqui (2019). Cross-lingual Word Embeddings. Morgan & Claypool Publishers.

Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, Thomas Wolf (2019). Transfer Learning in Natural Language Processing. In Proceedings of NAACL 2019: Tutorials, pages 15-18. Slides, Colab Notebook, Code

Matthew Peters*, Sebastian Ruder*, Noah A. Smith (2019). To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. In Proceedings of RepL4NLP at ACL 2019, Florence, Italy.

Goran Glavaš, Robert Litschko, Sebastian Ruder, Ivan Vulić (2019). How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In Proceedings of ACL 2019, Florence, Italy.

Sebastian Ruder (2019). Neural Transfer Learning for Natural Language Processing. PhD thesis. National University of Ireland, Galway.

Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, Anders Søgaard (2019). Latent Multi-task Architecture Learning. In Proceedings of AAAI 2019, Honolulu, Hawaii. Code

Victor Sanh, Thomas Wolf, Sebastian Ruder (2019). A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks. In Proceedings of AAAI 2019, Honolulu, Hawaii. Code

2018

Chris Hokamp, Sebastian Ruder, John Glover (2018). Off-the-Shelf Unsupervised NMT. arXiv preprint arXiv:1811.02278.

Sebastian Ruder*, Ryan Cotterell*, Yova Kementchedjhieva, Anders Søgaard (2018). A Discriminative Latent-Variable Model for Bilingual Lexicon Induction. In Proceedings of EMNLP 2018, Brussels, Belgium. Code.

Yova Kementchedjhieva, Sebastian Ruder, Ryan Cotterell, Anders Søgaard (2018). Generalizing Procrustes Analysis for Better Bilingual Dictionary Induction. In Proceedings of CoNLL 2018, Brussels, Belgium.

Sebastian Ruder, Barbara Plank (2018). Strong Baselines for Neural Semi-supervised Learning under Domain Shift. In Proceedings of ACL 2018, Melbourne, Australia. Slides, code.

Jeremy Howard*, Sebastian Ruder* (2018). Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018, Melbourne, Australia. Poster, code and pretrained models.

Anders Søgaard, Sebastian Ruder, Ivan Vulić (2018). On the Limitations of Unsupervised Bilingual Dictionary Induction. In Proceedings of ACL 2018, Melbourne, Australia. Slides.

Sebastian Ruder, John Glover, Afshin Mehrabani, Parsa Ghaffari (2018). 360° Stance Detection. In Proceedings of NAACL-HLT 2018: System Demonstrations, New Orleans, US.

Isabelle Augenstein*, Sebastian Ruder*, Anders Søgaard (2018). Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate Label Spaces. In Proceedings of NAACL-HLT 2018, New Orleans, US. Code, slides

2017

Sebastian Ruder, Barbara Plank (2017). Learning to select data for transfer learning with Bayesian Optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 372–382, Copenhagen, Denmark. Code, poster

Sebastian Ruder (2017). An Overview of Multi-Task Learning in Deep Neural Networks. arXiv preprint arXiv:1706.05098.

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2017). Data Selection Strategies for Multi-Domain Sentiment Analysis. arXiv preprint arXiv:1702.02426.

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2017). Knowledge Adaptation: Teaching to Adapt. arXiv preprint arXiv:1702.02052.

2016

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). Towards a continuous modeling of natural language domains. In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods, pages 53-57, Austin, Texas, US. Poster

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 999–1005, Austin, Texas, US. Poster

Ian D. Wood, Sebastian Ruder (2016). Emoji as emotion tags for tweets. In Emotion and Sentiment Analysis Workshop, LREC, Portorož, Slovenia.

Sebastian Ruder, Peiman Barnaghi, John G. Breslin (2016). Analysis and Applications of a Novel Corpus of Influencers on Twitter. In Twitter for Research Conference, Galway, Ireland.

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). INSIGHT-1 at SemEval-2016 Task 4: Convolutional Neural Networks for Sentiment Classification and Quantification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, US.

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). INSIGHT-1 at SemEval-2016 Task 5: Convolutional Neural Networks for Multilingual Aspect-based Sentiment Analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, US.

Sebastian Ruder (2016). An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747.

Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). Character-level and Multi-channel Convolutional Neural Networks for
Large-scale Authorship Attribution. arXiv preprint arXiv:1609.06686.