# Deploying a LLM Model with Inference Endpoints

If you want to deploy an LLM model for inference to production, an easy way to do it is to use Hugging Face's [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/en/index). Inference Endpoints are a fully managed service that simplifies inference deployment but keeps you at control of your deployment.

It is possible to deploy a model compatible with Inferentia2 devices directly on Inference Endpoints, using vLLM.

## Deploying a Model with vLLM on Inference Endpoints

This guide will walk you through the process of deploying a model with vLLM on Inference Endpoints. These steps are:

1. Create an Inference Endpoint
2. Configure and Deploy the Inference Endpoint
3. Test the Inference Endpoint

### 1. Create an Inference Endpoint

First you need to create an Inference Endpoint on a model compatible with Optimum Neuron. You can do this by going to the [Inference Endpoints](https://endpoints.huggingface.co/) page and click on "Catalog" to see the available models.

![Click on Browse Catalog](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/inference_endpoints/ie-browse-catalog.png)

You will see an interface with many cells, each representing a model configuration. You need to filter the model by the hardware accelerator you want to use. On the left column, select "INF2" to see the models compatible with Inferentia2.

![Filter by Inferentia2](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/inference_endpoints/ie-choose-hw-inf2.png)

You can now select the model configuration that you want to use by clicking on it. This will open a new page with the model configuration details.

### 2. Configure and Deploy the Inference Endpoint

On the model configuration details page, you can see the Inference Endpoint configuration. You can verify that the model is going to be deployed on an Inferentia2 instance and verify the price of the instance. Using vLLM on Neuron does not require any additional hardware setup, so you can just check other settings such as the endpoint name, or the authentication method. For full details on the configuration, you can refer to the [Inference Endpoints documentation](https://huggingface.co/docs/inference-endpoints/en/index).

Once you are happy with the configuration, you can click on "Deploy" to deploy the Inference Endpoint.

![Deploy the Inference Endpoint](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/inference_endpoints/ie-create-endpoint.png)

Once that is done, the Inferentia2 instance will be provisioned and the model will be deployed on it. This can take several minutes, depending on the model size and the instance type. Once the deployment is complete, you will see the Inference Endpoint status showing the message "Endpoint is Running".

![Inference Endpoint deployed](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/inference_endpoints/ie-endpoint-deployed.png)

> **_NOTE:_** If you fail to deploy and the status is showing a "Scheduling failure: unable to schedule" message, it is likely that AWS does not have enough capacity at that moment. You can try to deploy the endpoint again later.

### 3. Test the Inference Endpoint

You can now test the Inference Endpoint by sending a request to the Inference Endpoint URL. There are many ways to do this, but one easy way is to use the playground interface.

> If the endpoint is deployed in "private" mode, you will need to setup a [Hugging Face Token](https://huggingface.co/settings/tokens) able to make calls to your Inference Endpoints.

You can use the `Chat` button on the bottom right of the interface to open a chat interface to interact with the model.
You can also use the playground interface to test the Inference Endpoint, it is easy to use it and it allows to configure the request parameters.

![Test the Inference Endpoint](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/inference_endpoints/ie-test-endpoint.png)

Finally, you can also use the [OpenAI API](https://platform.openai.com/docs/api-reference/chat) to interact with the Inference Endpoint. A simple way to do this is to use the `curl` command.

```bash
# Replace  with the Inference Endpoint URL, and  with your Hugging Face Token
IE_URI=""
HF_TOKEN=""
curl "$IE_URI/v1/completions" \
    -H "Authorization: Bearer $HF_TOKEN" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{"prompt":"Define briefly AWS Trainium", "temperature": 0.0, "stop": ".","max_tokens":128}'
```

You should see the response from the Inference Endpoint that should look like this:

```console
{"id":"cmpl-3yoW-i","object":"text_completion","created":1768475896,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"text":"\nAWS Trainium is a cloud-based deep learning training service that allows users to train large-scale deep learning models quickly and efficiently","logprobs":null,"finish_reason":"stop","stop_reason":".","token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":32,"completion_tokens":26,"prompt_tokens_details":null},"kv_transfer_params":null}
```