Research

Kolibri

Aleph Alpha · Technical report · 2026

I contributed to Kolibri pre-training through expert load balancing, training optimization and hyperparameter tuning. Kolibri is an English-German MoE language model with 78.1B total parameters and 3.46B active parameters per token.

Technical report (opens in a new tab) · Model (opens in a new tab)

Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts

Pit Neitemeier, Jiaze Li (equal contribution), Alessio Serra, Philipp Scholl, Sohir Maskey · 2026

I co-developed EQB and LEI for global and local expert load balancing in MoE models.

Paper (opens in a new tab)

SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models

Pit Neitemeier, Alessio Serra, Jiaze Li, Sascha Wirges, Lukas Balles, Jan Hendrik Metzen · 2026

We measure and steer learned segment boundaries in byte-level language models, aligning computation with positions that are harder to predict.

Paper (opens in a new tab)