eole-nlp/metricx-23-large-v2p0-eole

MetricX-23 Large FP32 (EOLE)

This is google/metricx-23-large-v2p0 converted to EOLE format.

Model weights are stored as fp32. This repository contains a format conversion for use with EOLE's native transformer_encoder_decoder_scorer implementation.

This is not an upstream Google MetricX, Transformers, or T5X checkpoint layout. Use it with EOLE.

Model Details

Original model

google/metricx-23-large-v2p0

EOLE architecture

transformer_encoder_decoder_scorer

Weight dtype

fp32

Scoring type

metricx

EOLE scorer

EOLE-METRICX

Supported input modes

reference

Requires reference

Yes

Max input length

1024

Input template

candidate: {tgt} reference: {ref}

Score interpretation

MetricX scores are lower-is-better error scores, normally in the 0..25 range.

Usage

Requires an EOLE version with native MetricX scorer support.

Validation Metric

valid_metrics: ["EOLE-METRICX"]
metricx_model: eole-nlp/metricx-23-large-v2p0-eole
metricx_compute_dtype: fp32
metricx_batch_size: 8

Direct Scoring

eole predict \
  --model_path eole-nlp/metricx-23-large-v2p0-eole \
  --src /path/to/src.txt \
  --tgt /path/to/mt.txt \
  --ref /path/to/ref.txt \
  --output /path/to/scores.txt \
  --with_score \
  --compute_dtype fp32

By default, direct scoring writes one segment score per input line. To emit a single aggregate system score, add --score_level system:

eole predict \
  --model_path eole-nlp/metricx-23-large-v2p0-eole \
  --src /path/to/src.txt \
  --tgt /path/to/mt.txt \
  --ref /path/to/ref.txt \
  --output /path/to/system-score.txt \
  --with_score \
  --score_level system \
  --compute_dtype fp32

score_level: system writes one numeric line containing the arithmetic mean of the segment scores.

Conversion

Converted with:

eole convert MetricX \
  --model google/metricx-23-large-v2p0 \
  --dtype fp32 \
  --output metricx-23-large-v2p0-eole

Precision Note

MetricX uses mT5-style encoder-decoder scorer models, which can be numerically sensitive under reduced precision. This repository publishes fp32 converted weights as the reference EOLE artifact. Users may convert fp16 or bf16 variants separately if they accept possible score drift.

Original Model and Attribution

This model is a conversion of google/metricx-23-large-v2p0. The model weights were converted to EOLE format.

The upstream MetricX-23 models were proposed for the WMT 2023 Metrics Shared Task. They are learned regression-based metrics trained with direct assessment and MQM ratings, including synthetic training data for robustness. MetricX scores are lower-is-better error scores.

This converted repository follows the upstream model license: Apache-2.0. Please refer to the original model card and license for training data, intended use, limitations, and citation details.

MetricX GitHub repository: https://github.com/google-research/metricx

If you use MetricX-23 in your research, please cite:

@inproceedings{juraska-etal-2023-metricx,
    title = {{MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task}},
    author = "Juraska, Juraj  and
      Finkelstein, Mara  and
      Deutsch, Daniel  and
      Siddhant, Aditya  and
      Mirzazadeh, Mehdi  and
      Freitag, Markus",
    editor = "Koehn, Philipp  and
      Haddow, Barry  and
      Kocmi, Tom  and
      Monz, Christof",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.wmt-1.63",
    doi = "10.18653/v1/2023.wmt-1.63",
    pages = "756--767",
}