Invited speakers

Titolo: TBA

Abstract: TBA

Short bio: Karën Fort is a Professor at Université de Lorraine and does her research at the LORIA laboratory in Nancy, in the Inria team Semagramme. Her primary research interest is ethics in natural language processing (NLP), of which she is a pioneer: she organized the first colloquium on the subject in 2014, in France, followed by a national workshop (ETeRNAL 2015 and 2020) and a special issue of the TAL journal in 2016. She initiated the ethics and NLP French blog as well as the first survey on ethics in NLP (Fort & Couillault, 2016). She was co-chair of the first two ethics committees in the field (EMNLP 2020 and NAACL 2021) and is co-chair of the ethics committee of the association for computational linguistics (ACL). Beside her work on stereotypical biases (Névéol et al., 2022), she is interested in deontological ethics using NLP4NLP techniques (Abdalla et al, 2023).

---

Titolo: TBA

Abstract: Foundation models (FMs) process information as a sequence of internal representations; however, the length of this sequence is fixed and entirely determined by tokenization. This essentially decouples representation granularity from information content, which exacerbates the deployment costs of FMs and narrows their “horizons” over long sequences. What if, instead, we could free FMs from tokenizers by modelling bytes directly, while making them faster than current tokenizer-bound FMs? To achieve this goal, I will show how to: 1) learn tokenization end-to-end, by dynamically pooling representations in internal layers and progressively learning abstractions from raw data; 2) compress the KV cache (memory) of Transformers adaptively during generation without loss of performance; 3) predict multiple bytes per time step in an efficient yet expressive way; 4) retrofit existing tokenizer-bound FMs into byte-level FMs through cross-tokenizer distillation. By blending these ingredients, we may soon witness the emergence of new, more efficient architectures for foundation models.

Short bio: Edoardo M. Ponti is an assistant professor in Natural Language Processing at the University of Edinburgh and a visiting professor at NVIDIA. Previously, he was a visiting postdoctoral scholar at Stanford University and a postdoctoral fellow at Mila Montreal. In 2021, he obtained a PhD in computational linguistics from the University of Cambridge, St John’s College. His main research foci are efficient architectures for foundation models (tokenizer-free and adaptive memory), modular deep learning, and grounded typology. His research earned him a Google Research Faculty Award and 3 best paper awards. He is a board member and co-founder of SIGTYP and a scholar of the European Lab for Learning and Intelligent Systems (ELLIS). He is also a (terrible) violinist and an aspiring practitioner of heroic viticulture.