---
language:
- en
license: mit
library_name: pytorch
tags:
- gpt2
- text-generation
- pytorch
- causal-lm
- research
- ablation
datasets:
- HuggingFaceFW/fineweb
pipeline_tag: text-generation
---

# GPT-2 (124M) - Research Ablation Baseline

## Model Summary
This is a **124M parameter Causal Language Model** (GPT-2 Small architecture) trained entirely from scratch using PyTorch. 

It was created as a baseline for a research ablation study to investigate training dynamics, achieving a validation loss of **4.485**.

## Model Details
- **Architecture:** Custom GPT-2 Small (Decoder-only Transformer)
- **Parameters:** 124M
- **Context Window:** 256 tokens
- **Dimensions:** Embedding: 768, Heads: 12, Layers: 12
- **Training Steps:** ~28,500
- **Validation Loss:** 4.485

## How to Use
**⚠️ Important:** Because this model was trained using a custom PyTorch class (not the standard Hugging Face `GPT2LMHeadModel`), you **must** define the model architecture in your code before loading the weights.