A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
Accepted to ECCV 2026 (Poster)
This is not a drop-in Qwen3-VL checkpoint. Its vision tower takes 6 channels: RGB
concatenated with an optical-flow visualisation (config.json → vision_config.in_channels: 6).
A standard AutoProcessor / Qwen3VLForConditionalGeneration pipeline supplies 3 channels and
will not produce correct results. The weights load, but the input is wrong.
Traditional Shot Boundary Detection (SBD) looks for isolated cut points, which breaks down on
gradual transitions and frequently yields corrupted shots. TransVLM targets Shot Transition
Detection (STD) instead: it predicts the continuous temporal segments over which a transition
happens.
The model is Qwen3-VL-4B-Instruct with one architectural change. Its vision patch embedding is
widened from 3 to 6 input channels (zero-padded at initialisation), so an optical-flow
visualisation enters the model alongside RGB at the input stage. Motion is what separates a
dissolve from a camera pan, and a model that only sees appearance has to infer it. Because the
flow is fused before patchification, the language backbone carries no additional visual tokens,
so temporal awareness comes for free at the token budget.
Training used a scalable FFmpeg-based data engine covering 59 transition effects, which addresses
the severe class imbalance in public shot-boundary data.
6 (RGB + optical-flow visualisation, concatenated on the channel axis)
Sampling frame rate
25 fps
Optical flow
NeuFlow v2, computed at inference time. Its weights (~30 MB) come from the Hub on first use, so expect a second download
Input
One video; flow is computed for you
Output
Transition segments as start/end times in seconds
Saved with
transformers 4.57.3
Intended Use and Limitations
Intended for detecting shot transitions in videos, both hard cuts and gradual effects, and
as a baseline on the STD benchmark.
Limitations
Not a frame-level cut-point classifier. The output is a time span per transition, not a single
boundary frame.
Not a general video-chat model. It was fine-tuned on one task with one prompt; the prompt ships
with the inference code and changing it changes the task.
Long videos should be split before inference. Cost is linear in duration, and the flow
visualisation is normalised over its whole input, so a long video and its parts are not the same
signal.
Usage
No standalone transformers snippet is given here on purpose: a 3-channel one would run and
return nonsense. Use the inference package, which needs Python 3.12, a CUDA GPU, and ffmpeg
on PATH:
git clone https://github.com/heygen-com/TransVLM
cd TransVLM/inference
uv venv && source .venv/bin/activate
uv sync --group cu130 --group dev # cu128 if your driver is older than 570# Do not skip this. PyTorch 2.9.1 has a Conv3d bug below cuDNN 9.15, and the 6-channel# patch embed IS a Conv3d, so it is on the hot path of every forward pass. `uv sync`# resolves cuDNN back down, so re-run this after every sync. Activate the venv first:# unlike `uv sync`, `uv pip` installs into whatever environment is active.
uv pip install nvidia-cudnn-cu13==9.16.0.29 # nvidia-cudnn-cu12 for cu128 / cu126
python -c "import torch; print(torch.backends.cudnn.version())"# must print 91600
hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1
python infer_video.py \
--video /path/to/video.mp4 \
--ckpt-dir ./pretrained/TransVLM-v1 \
--output-jsonl out.jsonl
The cuDNN package name tracks the CUDA flavour, and installing the wrong one is a silent
no-op: it lands an unused package while torch keeps loading the other family. That is why
the check prints a version number instead of trusting the install.
Each output line carries segments as {start_time, end_time} pairs in seconds on the original
video's timeline, plus timings and the full configuration that produced them.
Three backends are supported: HuggingFace (default), vLLM and SGLang. Environment setup, every
option and the output schema are documented in
inference/README.md.
Evaluation
Qualitative results and the comparison against baselines are on the
project page.
The scorer that produced the paper's metric is released at
evaluation/. Note that the
published figures cannot be reproduced bit-for-bit from the released artifacts alone. The
benchmark ground truth and the pre-computed optical flow used for that run are not part of this
release. The reasons are spelled out in
evaluation/README.md.
Release Progress
Model weights
Inference code
Evaluation code
Data engine code
STD benchmark data
Re-annotated dataset labels
Leaderboard
🚧 The remaining items are being prepared for release.
Citation
@inproceedings{chen2026transvlm,
title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
booktitle={European Conference on Computer Vision},
year={2026},
organization={Springer}
}
TransVLM was developed as academic research and is published at ECCV 2026. It is not used in
any HeyGen product; HeyGen's production system for this task is a different model, trained
separately from this one.
Training data
Training combined HeyGen's own generated and annotated footage, which accounts for the
majority of the sampling distribution, with four public shot-boundary datasets:
AutoShot,
ClipShots,
MovieShots2 (SCTSNet, whose cut annotations
build on the Condensed Movies
Dataset), and SportsShot.
Together these four are roughly 42% of the training sampling distribution, of which SportsShot is
about 7.5%.
SportsShot is licensed CC BY-NC 4.0 and is credited here accordingly. The other three
distribute annotations rather than footage, so consult each dataset's own terms before relying on
them. No dataset is redistributed by this repository, and no dataset footage is contained in these
weights.
