en_100m_s3

An English-only seed control. It repeats en_100m with a different random seed. It is one of the 40 checkpoints released with the paper "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations" (EMNLP 2026).

Paper (EMNLP 2026) | arXiv | Project page | Code

Model

All 40 models in the study share one architecture, tokenizer, training recipe and data source (BabyBabelLM). Each is a decoder-only model with about 310M parameters (12 layers, hidden size 768, 12 attention heads, 512-token context), trained from scratch. The tokenizer and vocabulary come from meta-llama/Llama-3.2-1B; no pretrained Llama weights are loaded.

This checkpoint:

• Language: English
• English data: the full English corpus of BabyLM-community/babylm-eng
• Training tokens: about 98.3M (3,000 steps of 32,768 tokens)
• Seed: 303

In the paper the English-only seed runs give the reference for how much two models differ from random seed alone.

The full list of runs, with data sources, token budgets, seeds and optimization settings, is in the training manifest of the code repository.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("iamshnoo/en_100m_s3")
model = AutoModelForCausalLM.from_pretrained("iamshnoo/en_100m_s3")

These are small models trained on about 50M to 100M tokens for a study of internal representations. They are meant for representation analysis and for reproducing the paper, and they are not meant for text generation in applications.

Citation

@inproceedings{mukherjee-etal-2026-double,
    title = "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations",
    author = "Mukherjee, Anjishnu  and
      Zhu, Ziwei  and
      Anastasopoulos, Antonios",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Junyi Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.2062/",
    doi = "10.18653/2026.emnlp-main.2062",
    pages = "46953--46977",
    ISBN = "979-8-89176-459-0"
}