| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| base_model: stabilityai/stable-diffusion-xl-base-1.0 |
| library_name: diffusers |
| tags: |
| - Text-to-Image |
| - SDXL |
| - Stable Diffusion |
| - ID Customization |
| pipeline_tag: text-to-image |
| --- |
| <div style="display: flex; justify-content: center; align-items: center;"> |
| <img src="./images/images_alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;"> |
| <img src="./images/images_alimama.png" alt="alimama" style="width: 20%; height: auto;"> |
| </div> |
|
|
| [中文版Readme](./README_ZH.md) |
|
|
| This is a plugin/extension model for SDXL, not a fine-tuned version of stabilityai/stable-diffusion-xl-base-1.0. |
|
|
| EcomID aims to generate customized images from a single reference ID image, ensuring strong semantic consistency while being controlled by keypoints. |
|
|
| This repository provides the EcomID method and model, combining the strengths of [PuLID](https://github.com/ToTheBeginning/PuLID) and [InstantID](https://github.com/instantX-research/InstantID) for better background consistency, facial keypoint control, and realistic facial representation with improved similarity. |
|
|
| # EcomID Overview |
|
|
| ## EcomID Structure |
| <img src="./images/overflow.png" alt="alibaba" style="width: 100%; height: auto; margin-right: 5%;"> |
|
|
|
|
| - **IP-Adapter of PuLID**: EcomID incorporates the ID-Encoder and cross-attention components from PuLID, trained with alignment loss. |
| This method effectively reduces the interference of ID embeddings on text embeddings within the cross-attention part, minimizing disruption to the underlying model's text-to-image capabilities. |
| - **InstantID’s IdentityNet Architecture**: Utilizing **a dataset of 2 million aesthetically pleasing portrait images**, IdentityNet enhances keypoint control, improving ID consistency and facial realism. During training, the IP-adapter is frozen, and only the IdentityNet is trained. Facial landmarks are used as conditional inputs, while face embeddings are integrated into IdentityNet via cross-attention. |
|
|
| # Show Cases |
| ## Comparison with Other Methods |
| ### 1、Preserved Text-to-Image Capability |
|
|
| <table> |
| <tr> |
| <th style="width: 28%;">Prompt</th> |
| <th style="width: 24%;">Reference Image</th> |
| <th style="width: 24%;">EcomID</th> |
| <th style="width: 24%;">InstantID</th> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">girl, white skin, black hair, long wavy hair, <span style="color:red"><strong>in European style living room, Retro tone, decorations</strong></span>, depth of field.</td> |
| <td><img src="images/show_case/50.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/49.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/48.png" alt="InstantID图像" width="100%"></td> |
| </tr> |
| <table> |
| |
| As shown above, EcomID ***preserves background generation abilities while minimizing stylization, greatly enhancing realism***. |
| The visualizations highlight more authentic portraits with improved background semantic consistency, showcasing EcomID's advantage in generating realistic images. |
|
|
| ### 2、Improved Facial Control and Consistency |
| <table> |
| <tr> |
| <th style="width: 24%;">Prompt</th> |
| <th style="width: 19%;">Reference Image</th> |
| <th style="width: 19%;">EcomID</th> |
| <th style="width: 19%;">InstantID</th> |
| <th style="width: 19%;">PuLID</th> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A close-up portrait of a man standing in the library, holding <span style="color:red"><strong>two smiling toddlers</strong></span> next to him.</td> |
| <td><img src="images/show_case/20.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/17.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/18.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/19.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <table> |
| |
| As shown above, EcomID employs keypoints as conditional inputs for training, ***allowing for precise adjustments of facial positions, sizes, and orientations***. This capability ensures that the generated portraits are more controllable while further enhancing facial similarity and the overall quality of the images. |
|
|
| ### More showcases |
| EcomID enhances portrait representation, delivering a more authentic and aesthetically pleasing appearance while ensuring semantic consistency and greater internal ID similarity (i.e., traits that do not vary with age, hairstyle, glasses, or other physical changes). |
|
|
| <table> |
| <tr> |
| <th style="width: 24%;">Prompt</th> |
| <th style="width: 19%;">Reference Image</th> |
| <th style="width: 19%;">EcomID</th> |
| <th style="width: 19%;">InstantID</th> |
| <th style="width: 19%;">PuLID</th> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>little girl with double braids</strong></span>, wearing a white dress, standing on the beach during sunset.</td> |
| <td><img src="images/show_case/21.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/22.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/23.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/24.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td> |
| <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">Agrizzled detective, <span style="color:red"><strong>fedora</strong></span> casting a shadow over his square jaw, a <span style="color:red"><strong>cigar dangling from his lips</strong></span>, his trench coat evocative of film noir, in a <span style="color:red"><strong>rainy alley</strong></span>.</td> |
| <td><img src="images/show_case/25.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/26.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/27.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/28.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A smiling girl with <span style="color:red"><strong>bangs and long hair</strong></span> in a school uniform stands under cherry trees, holding a book.</td> |
| <td><img src="images/show_case/29.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/30.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/31.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/32.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A <span style="color:red"><strong>very old</strong></span> witch, wearing a black cloak, with a pointed hat, holding a magic wand, against a background of a misty forest.</td> |
| <td><img src="images/show_case/33.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/34.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/35.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/36.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td style="font-size: 12px;">A man clad in cyberpunk fashion: <span style="color:red"><strong>neon accents, reflective sunglasses，</strong></span> and a leather jacket with glowing circuit patterns. He stands stoically amidst a soaked cityscape.</td> |
| <td><img src="images/show_case/37.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/38.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/39.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/40.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| |
| </table> |
|
|
| ### More Base Models, Resolutions, and Styles |
| <table> |
| <tr> |
| <th style="width: 12%;">SDXL models</th> |
| <th style="width: 24%;">Prompt</th> |
| <th style="width: 16%;">Reference Image</th> |
| <th style="width: 16%;">EcomID</th> |
| <th style="width: 16%;">InstantID</th> |
| <th style="width: 16%;">PuLID</th> |
| </tr> |
| <tr> |
| <td>sd-xl-base-1.0</td> |
| <td style="font-size: 12px;">girl, solo, brown hair, holding a little teddy bear on her hands, wearing a school uniform, standing in the library, <span style="color:red"><strong>cartoon style</strong></span>.</td> |
| <td><img src="images/show_case/1.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/2.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/3.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/4.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td>EcomXL</td> |
| <td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td> |
| <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td>DreamShaperXL</td> |
| <td style="font-size: 12px;">solo, looking_at_viewer, smile, brown_hair, upper_body, open_clothes, teeth, open_jacket, black_jacket, blurry_background, realistic</td> |
| <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/6.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/7.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/8.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| <tr> |
| <td>leosam_xl_v7</td> |
| <td style="font-size: 12px;">A close-up portrait of a girl, solo, dress, jewelry, beach and sea, pink_dress, realistic.</td> |
| <td><img src="images/show_case/9.png" alt="参考图像" width="100%"></td> |
| <td><img src="images/show_case/15.png" alt="EcomID图像" width="100%"></td> |
| <td><img src="images/show_case/14.png" alt="InstantID图像" width="100%"></td> |
| <td><img src="images/show_case/16.png" alt="PuLID图像" width="100%"></td> |
| </tr> |
| </table> |
| |
| ### Notes |
| - Unless otherwise specified, the showcases are generated using the base model EcomXL, which is also highly compatible with various other SDXL-based models, such as [leosams-helloworld-xl](https://civitai.com/models/43977/leosams-helloworld-xl), [dreamshaper-xl](https://civitai.com/models/112902/dreamshaper-xl), [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and so on. |
| - It works very well with SDXL Turbo/Lighting, [EcomXL Inpainting ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_inpaint) and [EcomXL Softedge ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_softedge). |
|
|
| # How to use |
|
|
| ## ComfyUI |
|
|
| - The EcomID_ComfyUI node has been released: [click here](https://github.com/alimama-creative/SDXL_EcomID_ComfyUI) |
| |
| # Training Details |
| |
| The model is trained on 2M Taobao images, where the proportion of human faces is greater than 3%. The images have a resolution greater than 800, and the aesthetic score is above 5.5. |
| |
| Mixed precision: fp16 |
| |
| Learning rate: 1e-4 |
| |
| Batch size: 2 |
| |
| Image size: 1024x1024 |
