Hugging Face's logo

prithivMLmods
/
rf-detr-mobile-gui-detection

Object Detection
Transformers
Safetensors
English
rf_detr
text-generation-inference
mobile-gui-detection
Model card Files
xet
 Community
rf-detr-mobile-gui-detection
File size: 6,730 Bytes
6566193
 
f66760e
 
 
 
 
 
 
 
 
6ee048f
 
908c106
 
217481e
6746dfc
217481e
6746dfc
217481e
 
6746dfc
 
 
 
217481e
 
 
 
 
 
6746dfc
217481e
 
 
ab19029
a585f7e
 
 
 
f74f926
 
 
 
908c106
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6c55a43
 
34aa61a
 
6c55a43
dc42e63
 
 
 
 
9e8a97c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
---
license: apache-2.0
language:
- en
base_model:
- Roboflow/rf-detr-medium
pipeline_tag: object-detection
library_name: transformers
tags:
- text-generation-inference
- mobile-gui-detection
datasets:
- mrtoy/mobile-ui-design
---

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #e2e8f0; border-radius: 8px; box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;">
  <div style="background: #1e40af; padding: 20px; color: white;">
    <h1 style="margin: 0; font-size: 22px; font-weight: 700; color: white; border: none;">rf-detr-mobile-gui-detection</h1>
    <p style="margin: 8px 0 0 0; font-size: 13px; color: #bfdbfe; font-weight: 500;">Mobile GUI grounding model built on top of roboflow/rf-detr-medium</p>
  </div>
  <div style="display: flex; gap: 8px; flex-wrap: wrap; padding: 12px 20px; background: #f8fafc; border-bottom: 1px solid #e2e8f0;">
    <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Object Detection</span>
    <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">DETR</span>
    <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Mobile GUI</span>
    <span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Grounding</span>
  </div>
  <div style="padding: 20px; display: flex; flex-direction: column; gap: 16px;">
    <p style="margin: 0; font-size: 14px; color: #334155; line-height: 1.6;"><b>rf-detr-mobile-gui-detection</b> is a mobile gui grounding model built on top of <b>roboflow/rf-detr-medium</b> using the <b>rfdetrforobjectdetection</b> architecture. rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone, an rf-detr windowing pattern for efficient attention, a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.</p>

  <div style="border: 1px solid #fde68a; padding: 14px; border-radius: 4px; background: #fffbeb;">
      <span style="font-weight: 700; color: #92400e; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">Note</span>
      <p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.5;">rf-detr: neural architecture search for real-time detection transformers: <a href="https://huggingface.co/papers/2511.09554" style="color: #92400e; text-decoration: underline;">https://huggingface.co/papers/2511.09554</a></p>
    </div>
  </div>
</div>

## Metrics Loss Map

![metrics_loss_map](https://huggingface.co/prithivMLmods/rf-detr-mobile-gui-detection/resolve/main/metrics_loss_map.png)

## Per Class Metrics

![per_class_metrics](https://huggingface.co/prithivMLmods/rf-detr-mobile-gui-detection/resolve/main/per_class_metrics.png)

## Quick Start with Transformers

```
pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install torchvision==0.23.0 transformers==5.9.0 accelerate gradio==6.19.0
```

```py
import gradio as gr
import torch
from PIL import Image, ImageDraw

from transformers import AutoImageProcessor, RfDetrForObjectDetection

# Load model and processor
model_name = "prithivMLmods/rf-detr-mobile-gui-detection"

processor = AutoImageProcessor.from_pretrained(model_name)
model = RfDetrForObjectDetection.from_pretrained(model_name)

# Detection threshold
THRESHOLD = 0.35


def detect_gui(image):
    image = Image.fromarray(image).convert("RGB")

    inputs = processor(images=image, return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    target_sizes = torch.tensor([image.size[::-1]])
    results = processor.post_process_object_detection(
        outputs,
        target_sizes=target_sizes,
        threshold=THRESHOLD,
    )[0]

    draw = ImageDraw.Draw(image)

    detections = []

    for score, label, box in zip(
        results["scores"],
        results["labels"],
        results["boxes"],
    ):
        box = [round(x, 2) for x in box.tolist()]
        label_name = model.config.id2label[label.item()]
        confidence = round(score.item(), 3)

        # Draw bounding box
        draw.rectangle(box, outline="red", width=3)

        # Draw label
        draw.text(
            (box[0] + 4, max(0, box[1] - 16)),
            f"{label_name} {confidence:.2f}",
            fill="red",
        )

        detections.append(
            {
                "Label": label_name,
                "Confidence": confidence,
                "Bounding Box": box,
            }
        )

    return image, detections


demo = gr.Interface(
    fn=detect_gui,
    inputs=gr.Image(type="numpy", label="Upload Mobile UI Screenshot"),
    outputs=[
        gr.Image(type="pil", label="Detected GUI Elements"),
        gr.JSON(label="Detections"),
    ],
    title="RF-DETR Mobile GUI Detection",
    description="Upload a mobile UI screenshot to detect GUI elements using RF-DETR.",
)

if __name__ == "__main__":
    demo.launch()
```

> e.g., demo screenshot

![screencapture-c959e285e9de4018e9-gradio-live-2026-06-28-21_22_41](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/I0LeSI-y43mKKy9tKMa-0.png)

## Acknowledgements

* **[roboflow/rf-detr-medium](https://huggingface.co/Roboflow/rf-detr-medium)**: rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone (with an rf-detr windowing pattern for efficient attention), a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.

* **[mobile ui design detection[dataset]](https://huggingface.co/datasets/mrtoy/mobile-ui-design)** by **[mrtoy](https://huggingface.co/mrtoy)**: this dataset is designed for object detection tasks focused on detecting elements in mobile ui designs. the target objects include text, images, and groups. the dataset contains mobile ui images with object detection bounding boxes, class labels, and localization information.