thedeoxen commited on
Commit
0ae1ef7
Β·
verified Β·
1 Parent(s): 11d2e4c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +117 -55
README.md CHANGED
@@ -1,66 +1,128 @@
1
  ---
 
 
 
 
2
  tags:
3
- - text-to-image
 
 
 
 
 
 
 
 
4
  - lora
5
- - diffusers
6
- - template:diffusion-lora
7
- widget:
8
- - output:
9
- url: images/1d.png
10
- text: '-'
11
- - output:
12
- url: images/1ref.png
13
- text: '-'
14
- - output:
15
- url: images/1res.png
16
- text: '-'
17
- - output:
18
- url: images/2d.png
19
- text: '-'
20
- - output:
21
- url: images/2ref.png
22
- text: '-'
23
- - output:
24
- url: images/2res.png
25
- text: '-'
26
- - output:
27
- url: images/3d.png
28
- text: '-'
29
- - output:
30
- url: images/3ref.png
31
- text: '-'
32
- - output:
33
- url: images/3res.png
34
- text: '-'
35
- - output:
36
- url: images/4d.png
37
- text: '-'
38
- - output:
39
- url: images/4ref.png
40
- text: '-'
41
- - output:
42
- url: images/4res.png
43
- text: '-'
44
- - output:
45
- url: images/5d.png
46
- text: '-'
47
- - output:
48
- url: images/5ref.png
49
- text: '-'
50
- - output:
51
- url: images/5res.png
52
- text: '-'
53
- base_model: black-forest-labs/FLUX.2-klein-base-4B
54
- instance_prompt: null
55
- license: apache-2.0
56
  ---
57
- # RefControl FLUX.2 Klein 4B – Reference Depth LoRA
 
 
 
 
 
 
 
58
 
59
  <Gallery />
60
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
 
 
62
 
63
- ## Download model
 
 
 
64
 
 
 
 
 
 
 
 
 
 
65
 
66
- [Download](/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/tree/main) them in the Files & versions tab.
 
 
 
 
1
  ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - black-forest-labs/FLUX.2-klein-base-4B
5
+ pipeline_tag: image-to-image
6
  tags:
7
+ - flux
8
+ - flux2
9
+ - klein
10
+ - depth
11
+ - controlnet
12
+ - refcontrol
13
+ - img2img
14
+ - image
15
+ - editing
16
  - lora
17
+ library_name: diffusers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  ---
19
+
20
+ # RefControl FLUX.2 Klein 4B - Reference Depth LoRA
21
+
22
+ ## πŸ“ Short description
23
+ A LoRA for **FLUX.2 Klein 4B Base** that fuses a **reference image** (left) with a **depth map** (right).
24
+ It preserves **identity and style** from the reference while following the **pose and structure** from the depth map.
25
+
26
+ **Trigger word:** `refcontrol`
27
 
28
  <Gallery />
29
 
30
+ ---
31
+
32
+ ## πŸ“Š Examples
33
+
34
+ <table>
35
+ <thead>
36
+ <tr>
37
+ <th>Reference</th>
38
+ <th>Depth</th>
39
+ <th>Result</th>
40
+ </tr>
41
+ </thead>
42
+ <tbody>
43
+ <tr>
44
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1ref.png" width="320" /></td>
45
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1d.png" width="320" /></td>
46
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1res.png" width="320" /></td>
47
+ </tr>
48
+ <tr>
49
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2ref.png" width="320" /></td>
50
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2d.png" width="320" /></td>
51
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2res.png" width="320" /></td>
52
+ </tr>
53
+ <tr>
54
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3ref.png" width="320" /></td>
55
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3d.png" width="320" /></td>
56
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3res.png" width="320" /></td>
57
+ </tr>
58
+ <tr>
59
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4ref.png" width="320" /></td>
60
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4d.png" width="320" /></td>
61
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4res.png" width="320" /></td>
62
+ </tr>
63
+ <tr>
64
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5ref.png" width="320" /></td>
65
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5d.png" width="320" /></td>
66
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5res.png" width="320" /></td>
67
+ </tr>
68
+ </tbody>
69
+ </table>
70
+
71
+ ---
72
+
73
+ ## πŸ“– Extended description
74
+ This LoRA was primarily trained on **humans**, but it also works with **objects** and stylized characters.
75
+ Its main purpose is to **preserve identity** - facial features, clothing, style, or object characteristics - from the reference image, while adapting the subject to the **pose and composition** defined by the depth map.
76
+
77
+ Part of the **RefControl** family: reference + control fusion for consistent, controllable generation on **FLUX.2 Klein 4B Base**.
78
+
79
+ ---
80
+
81
+ ## βš™οΈ How to use
82
+ 1. Use first image as depth reference, second image input.
83
+ 2. Add the trigger word **`refcontrol`** in your prompt.
84
+ 3. Adjust LoRA weight (recommended **0.8-1.0**) depending on how strongly you want to preserve identity.
85
+
86
+ ## ComfyUI requirenments
87
+ For extracting depth you need to install nodes with depth estimation in workflow used:
88
+
89
+ https://github.com/Fannovel16/comfyui_controlnet_aux
90
+
91
+ ### Base model
92
+ Use with [`black-forest-labs/FLUX.2-klein-base-4B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B).
93
+ The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, `guidance_scale` ~4.0).
94
+
95
+ ### βœ… Example prompt
96
+
97
+ ```text
98
+ refcontrol
99
+ ```
100
+
101
+ ---
102
+
103
+ ## 🎯 What it does
104
+ - Preserves **character or object identity** across generations.
105
+ - Embeds the subject into a **new pose or scene** defined by the depth map.
106
+ - Works best when the depth map has **similar proportions and scale** to the reference.
107
 
108
+ ---
109
 
110
+ ## ⚑ Tips
111
+ - Best results when the depth map is not drastically different in object scale from the reference.
112
+ - Combine with **text prompts** to refine background, lighting, or mood.
113
+ - Can be combined with other **RefControl** LoRAs (pose, lineart, canny) on compatible pipelines for multi-constraint generation.
114
 
115
+ ---
116
+
117
+ ## πŸ“Œ Use cases
118
+ - Human portraits in different poses.
119
+ - Consistent **character design** across multiple scenes.
120
+ - Object transformations (cars, furniture, props) with depth-guided placement.
121
+ - Storyboarding, comics, or animation frame generation.
122
+
123
+ ---
124
 
125
+ ## πŸ“¦ Files
126
+ - **Weights:** [`flux2_klein_4b_refcontrol_depth.safetensors`](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/blob/main/flux2_klein_4b_refcontrol_depth.safetensors)
127
+ - **ComfyUI workflow:** [`image_flux2_klein_image_edit_4b_refcontrol_depth.json`](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/blob/main/image_flux2_klein_image_edit_4b_refcontrol_depth.json)
128
+ - **Repo:** [thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora)