#
InstructPix2Pix

##
Overview

InstructPix2Pix: Learning to Follow Image Editing Instructions is by Tim Brooks, Aleksander Holynski and Alexei A. Efros.

🤗 `Optimum` extends `Diffusers` to support inference on the second generation of Neuron devices(powering Trainium and Inferentia 2). It aims at inheriting the ease of Diffusers on Neuron.

##
Export to Neuron

To deploy models, you will need to compile them to TorchScript optimized for AWS Neuron. In the case of Stable Diffusion, there are four components which need to be exported to the `.neuron` format to boost the performance:

- Text encoder
- U-Net
- VAE encoder
- VAE decoder

You can either compile and export a Stable Diffusion Checkpoint via CLI or `NeuronStableDiffusionInstructPix2PixPipeline` class.

##
Usage Example

With the `NeuronStableDiffusionInstructPix2PixPipeline` class, you can apply instruction-based image editing using both text guidance and image guidance.

    import requests
    import PIL
    from io import BytesIO
    from optimum.neuron import NeuronStableDiffusionInstructPix2PixPipeline

    def download_image(url):
        response = requests.get(url)
        return PIL.Image.open(BytesIO(response.content)).convert("RGB")

    model_id = "timbrooks/instruct-pix2pix"
    input_shapes = {"batch_size": 1, "height": 512, "width": 512}

    pipe = NeuronStableDiffusionInstructPix2PixPipeline.from_pretrained(
      model_id, export=True, dynamic_batch_size=True, **input_shapes,
    )
    pipe.save_pretrained("sd_ip2p/")

    img_url = "https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"
    init_image = download_image(img_url).resize((512, 512))

    prompt = "Add a beautiful sunset"
    image = pipe(prompt=prompt, image=init_image).images[0]
    image.save("sunset_mountain.png")

`image` | `prompt` | output
--- | --- | ---
| Add a beautiful sunset |

####
NeuronStableDiffusionInstructPix2PixPipeline

###
class optimum.neuron.NeuronStableDiffusionInstructPix2PixPipeline

< source >

( config: dict[str, typing.Any] configs: dict[str, 'PretrainedConfig'] neuron_configs: dict[str, 'NeuronDefaultConfig'] data_parallel_mode: typing.Literal['none', 'unet', 'transformer', 'all'] scheduler: diffusers.schedulers.scheduling_utils.SchedulerMixin | None vae_decoder: torch.jit._script.ScriptModule | NeuronModelVaeDecoder text_encoder: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None text_encoder_2: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None unet: torch.jit._script.ScriptModule | NeuronModelUnet | None = None transformer: torch.jit._script.ScriptModule | NeuronModelTransformer | None = None vae_encoder: torch.jit._script.ScriptModule | NeuronModelVaeEncoder | None = None image_encoder: torch.jit._script.ScriptModule | None = None safety_checker: torch.jit._script.ScriptModule | None = None tokenizer: transformers.models.clip.tokenization_clip.CLIPTokenizer | transformers.models.t5.tokenization_t5.T5Tokenizer | None = None tokenizer_2: transformers.models.clip.tokenization_clip.CLIPTokenizer | transformers.models.t5.tokenization_t5.T5Tokenizer | None = None feature_extractor: transformers.models.clip.feature_extraction_clip.CLIPFeatureExtractor | None = None controlnet: torch.jit._script.ScriptModule | list[torch.jit._script.ScriptModule]| NeuronControlNetModel | NeuronMultiControlNetModel | None = None requires_aesthetics_score: bool = False force_zeros_for_empty_prompt: bool = True add_watermarker: bool | None = None model_save_dir: str | pathlib.Path | tempfile.TemporaryDirectory | None = None model_and_config_save_paths: dict[str, tuple[str, pathlib.Path]] | None = None )

####
__call__

< source >

( *args **kwargs )

Are there any other diffusion features that you want us to support in 🤗`Optimum-neuron`? Please file an issue to `Optimum-neuron` Github repo or discuss with us on HuggingFace’s community forum, cheers 🤗 !