README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - chest X-ray report generation
  - radiology report generation
  - image captioning
  - chest X-ray
  - X-ray
  - radiology
  - cxrmate
  - cxrmate-2
  - cxrmate-ed
  - cxrmate-rrg24
  - report
  - radiology report
  - multimodal
  - patient data
  - mimic-cxr

CXRMate-2 — Chest X-ray Radiology Report Generation

From: CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation

The pre-print for CXRMate-2 is available at:

@misc{nicolson2026clinicallyacceptablechestxray,
      title={CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation}, 
      author={Aaron Nicolson and Elizabeth J. Cooper and Hwan-Jin Yoon and Claire McCafferty and Ramya Krishnan and Michelle Craigie and Nivene Saad and Jason Dowling and Ian A. Scott and Bevan Koopman},
      year={2026},
      eprint={2604.18967},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.18967}, 
}

Load CXRMate-2:

alias = 'aehrc/cxrmate-2'

model = transformers.AutoModelForCausalLM.from_pretrained(alias, trust_remote_code=True).to(device='cuda')
model.eval()

generation_config = transformers.GenerationConfig.from_pretrained(alias, trust_remote_code=True)

processor = transformers.AutoProcessor.from_pretrained(alias, trust_remote_code=True)

Generate from URL:

url = 'https://prod-images-static.radiopaedia.org/images/220869/76052f7902246ff862f52f5d3cd9cd_big_gallery.jpg'

display(Image(url=url))

processed = processor(images=url)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids) 
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')

Generate from study DICOMs:

dcm_path = [
    './physionet.org/files/mimic-cxr/2.0.0/files/p12/p12000264/s55271473/522f9570-7cb12ecb-6327c8b8-b248b4be-58bb3dfd.dcm',
    './/datasets/work/hb-mlaifsp-mm/work/repositories/25_cxrmate2/work/data/physionet.org/files/mimic-cxr/2.0.0/files/p12/p12000264/s55271473/d0b61aff-f64c4ecf-85fae310-43668cf1-0c1d4c2d.dcm',
]

processed = processor(images=dcm_path)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids) 
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')

Generate from ReXgradient HuggingFace dataset:

Scripts to create the ReXgradient, CheXpert Plus, and MIMC-CXR Hugging Face datasets will be released soon

# Run prepare_chexpert_plus.py or prepare_mimic_cxr_jpg.py or prepare_rexgradient.py to create dataset:
test_set = datasets.load_from_disk('/scratch3/nic261/database/cxrmate2/rexgradient_160k_dataset')['test']

# Wrap dataset to get priors:
test_set = processor.wrap_dataset(test_set)

random_idx = random.randint(0, len(test_set) - 1)
example = test_set[random_idx]

processed = processor(
    images=example['images'],  # This includes both the current and prior images.
    image_datetime=example['image_datetime'],  # This includes the datetimes for both the current and prior images.
    views=example['views'],  # This includes both the current and prior views.
    indication=example.get('indication', None),
    history=example.get('history', None),
    comparison=example.get('comparison', None),
    technique=example.get('technique', None),
    study_datetime=example.get('study_datetime', None),
    prior_findings=example.get('prior_findings', None),
    prior_impression=example.get('prior_impression', None),
    prior_study_datetime=example.get('prior_study_datetime', None),
)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids)
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')

Environment requirements:

Environment requirements can be found here: https://github.com/aehrc/cxrmate-ed/blob/main/requirements.txt.

Training:

The training pipeline for CXRMate-2, is available at: https://github.com/aehrc/cxrmate-2.