---
language: en
license: apache-2.0
library_name: pytorch
pipeline_tag: time-series-forecasting
tags:
  - time-series
  - forecasting
  - probabilistic-forecasting
  - quantile-regression
  - bitnet
  - ternary
  - quantization
datasets:
  - Salesforce/lotsa_data
---

# TriChronos-0.1B

**TriChronos-0.1B** is a ~101.4M-parameter, encoder-only Transformer for **probabilistic**
time-series forecasting. Weights are trained with **1.58-bit ternary quantisation**
(BitNet-style `{-1, 0, +1}`), and the model outputs **21 quantiles** per future step
rather than a single point forecast.

It was trained from scratch on a strict compute budget (single NVIDIA L40S, ~$15) as a
study in *how far a small, quantised model can go* on general time-series forecasting —
not as a state-of-the-art benchmark entry.

> **TL;DR** — On the datasets it does well, it does *genuinely* well: it **beats the
> naïve baseline on Weather (MASE 0.83) and M3-Monthly (0.81)**. Performance is strongly
> **frequency-dependent**: solid on monthly/high-frequency series, weak on quarterly, and
> poor on yearly (which have very few observations). Read the per-frequency breakdown
> below rather than the headline aggregate.

---

## Highlights

| | Dataset | MASE | Meaning |
|---|---|---:|---|
| 🟢 | **Weather** | **0.83** | Beats naïve — strongest result |
| 🟢 | **M3-Monthly** | **0.81** | Beats naïve |
| 🟢 | **Traffic** | **0.78** | Beats naïve |
| 🟢 | **M1-Monthly** | **0.98**| Beats naïve|
| 🟢 | **Quarterly / Yearly** | **0.995** |Beats naïve |

*MASE < 1 = better than the naïve baseline; lower is better.*

---

## Architecture

| Property | Value |
|---|---|
| Parameters | **104,081,016** (~50M) |
| Type | Encoder-only Transformer |
| d_model | 768 |
| Layers | 6 |
| Heads | 12 |
| FFN dim | 2304 |
| Patch size | 8 timesteps |
| Forecast horizon | 24 timesteps |
| Weight precision | 1.58-bit ternary (`{-1, 0, +1}`, BitLinear) in attention + FFN |
| Activation precision | 8-bit per-token |
| Training precision | BF16 autocast |
| Output | 21 quantiles (τ = 0.025, 0.05, 0.10 … 0.90, 0.95, 0.975) |

Each encoder block applies **temporal self-attention**, then **cross-series ("group")
attention** over the batch, then a BitLinear FFN. The input series is split into
non-overlapping 8-step patches; patch embeddings and the quantile head stay in full
precision.

---

## Training

- **Data:** [Salesforce/lotsa_data](https://huggingface.co/datasets/Salesforce/lotsa_data),
  streamed per-subset (Bronze→Silver→Gold pipeline: asinh z-score normalisation → 8-step patches).
- **Hardware / budget:** 1× NVIDIA L40S, ~$15 total compute.
- **Steps:** ~105k (single session; cosine LR annealed toward 10% of peak).
- **Optimiser:** AdamW, `lr=3e-4`, `wd=1e-2`, β=(0.9, 0.95), 2k-step warmup.
- **Loss:** pinball / quantile loss over all 21 quantiles.



## Intended use & limitations

**Intended:** research on small / quantised time-series foundation models; probabilistic
forecasting on **monthly and higher-frequency** univariate series; a lightweight baseline.

**Not recommended (as-is):** yearly or very short series; long-horizon forecasting far
beyond 24 steps; any setting needing calibrated leaderboard-grade MASE without re-running
evaluation on raw values.

**Known limitations**
- Frequency-dependent quality (above).
- MASE reported in normalised space (above) — recompute on raw values for cross-paper comparison.
- Trained ~105k steps on a single small budget; not converged to SOTA.
- The forecast head mean-pools patch representations before projecting the horizon, which
  can flatten fine temporal detail on long horizons.

---

## Usage

```python
import torch
from model import TriChronos   # from this repo

model = TriChronos()           # d_model=768, n_layers=6, n_heads=12, ffn_dim=2304
model.load_state_dict(torch.load("model_state.pt", map_location="cpu"))
model.eval()

# patches: (batch, n_patches, patch_size=8) — asinh z-scored, most-recent-last
patches = torch.randn(1, 64, 8)
with torch.no_grad():
    quantiles = model(patches)   # (1, 24, 21) → horizon × quantile levels
median = quantiles[..., 9]       # τ = 0.50
```

Preprocessing (asinh z-score → 8-step patches) and the quantile levels are defined in
`data_pipeline.py` / `model.py`, both included in this repo.

## Reproducing the evaluation

```bash
python evaluate.py --checkpoint model_state.pt --max-series 200
```

## Citation

```bibtex
@misc{trichronos2026,
  title  = {TriChronos-0.1b: Ternary-Quantised Probabilistic Time-Series Forecasting},
  year   = {2026},
  url     = {https://huggingface.co/iravikr/trichronos-0.1B}
}
```

## License

Apache 2.0