Instructions to use tiiuae/falcon-40b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiiuae/falcon-40b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiiuae/falcon-40b-instruct", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b-instruct", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use tiiuae/falcon-40b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiiuae/falcon-40b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-40b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/tiiuae/falcon-40b-instruct

SGLang

How to use tiiuae/falcon-40b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiiuae/falcon-40b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-40b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiiuae/falcon-40b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-40b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use tiiuae/falcon-40b-instruct with Docker Model Runner:
```
docker model run hf.co/tiiuae/falcon-40b-instruct
```

FalconLLM commited on Jun 5, 2023

Commit

4e8f82c

1 Parent(s): b55d7ec

Update citation info

Browse files

Files changed (1) hide show

README.md +33 -1

README.md CHANGED Viewed

@@ -174,7 +174,39 @@ Falcon-40B-Instruct was trained a custom distributed training codebase, Gigatron
 ## Citation
-*Paper coming soon 😊.*
 ## License

 ## Citation
+*Paper coming soon* 😊. In the meanwhile, you can use the following information to cite:
+```
+@article{falcon40b,
+  title={{Falcon-40B}: an open large language model with state-of-the-art performance},
+  author={Almazrouei, Ebtesam and Alobeidli, Hamza and Alshamsi, Abdulaziz and Cappelli, Alessandro and Cojocaru, Ruxandra and Debbah, Merouane and Goffinet, Etienne and Heslow, Daniel and Launay, Julien and Malartic, Quentin and Noune, Badreddine and Pannier, Baptiste and Penedo, Guilherme},
+  year={2023}
+}
+```
+To learn more about the pretraining dataset, see the 📓 [RefinedWeb paper](https://arxiv.org/abs/2306.01116).
+```
+@article{refinedweb,
+  title={The {R}efined{W}eb dataset for {F}alcon {LLM}: outperforming curated corpora with web data, and web data only},
+  author={Guilherme Penedo and Quentin Malartic and Daniel Hesslow and Ruxandra Cojocaru and Alessandro Cappelli and Hamza Alobeidli and Baptiste Pannier and Ebtesam Almazrouei and Julien Launay},
+  journal={arXiv preprint arXiv:2306.01116},
+  eprint={2306.01116},
+  eprinttype = {arXiv},
+  url={https://arxiv.org/abs/2306.01116},
+  year={2023}
+}
+```
+To cite the [Baize](https://github.com/project-baize/baize-chatbot) instruction dataset used for this model:
+```
+@article{xu2023baize,
+  title={Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data},
+  author={Xu, Canwen and Guo, Daya and Duan, Nan and McAuley, Julian},
+  journal={arXiv preprint arXiv:2304.01196},
+  year={2023}
+}
+```
 ## License