Instructions to use InferenceIllusionist/Excalibur-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use InferenceIllusionist/Excalibur-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="InferenceIllusionist/Excalibur-7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("InferenceIllusionist/Excalibur-7b")
model = AutoModelForCausalLM.from_pretrained("InferenceIllusionist/Excalibur-7b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use InferenceIllusionist/Excalibur-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "InferenceIllusionist/Excalibur-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InferenceIllusionist/Excalibur-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/InferenceIllusionist/Excalibur-7b

SGLang

How to use InferenceIllusionist/Excalibur-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "InferenceIllusionist/Excalibur-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InferenceIllusionist/Excalibur-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "InferenceIllusionist/Excalibur-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InferenceIllusionist/Excalibur-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use InferenceIllusionist/Excalibur-7b with Docker Model Runner:
```
docker model run hf.co/InferenceIllusionist/Excalibur-7b
```

Excalibur-7b / README.md

InferenceIllusionist

Update README.md

2d9610a verified over 2 years ago

preview code

Raw

History Blame Contribute Delete

5.68 kB

	---
	base_model:
	- ibm/merlinite-7b
	- InferenceIllusionist/Magic-Dolphin-7b
	- SanjiWatsuki/Kunoichi-DPO-v2-7B
	- mlabonne/Monarch-7B
	- bardsai/jaskier-7b-dpo-v6.1

	library_name: transformers
	tags:
	- mergekit
	- merge
	license: apache-2.0
	---


	# Excalibur-7b

	<img src="https://i.imgur.com/viIO4WT.png" width="550"/>

	<b> Update: A [fine-tuned version](https://huggingface.co/InferenceIllusionist/Excalibur-7b-DPO/) of this model is now publicly available, along with benchmark results. If you're looking for a more conversational, assistant-style exchange you won't want to miss it!</b>

	<i>Image generated with Envoid's [Model9](https://huggingface.co/Envoid/model9) SDXL model </i>

	GGUFs can be found [here](https://huggingface.co/InferenceIllusionist/Excalibur-7b-GGUF)

	Alternative GGUFs from [bartowski](https://huggingface.co/bartowski) can be found [here](https://huggingface.co/bartowski/Excalibur-7b-GGUF).

	EXl2 can also be found [here](https://huggingface.co/bartowski/Excalibur-7b-exl2) again courtesy of [bartowski](https://huggingface.co/bartowski)!


	### Performance Comparison
	\| Name \| Avg. \| ARC \| HellaSwag \| MMLU \| TruthfulQA \| Winogrande \| GSM8K \|
	\| ---- \| ---- \| ---- \| ---- \| ---- \| ---- \| ---- \| ---- \|
	\| <b>Excalibur-7b</b> \| <u><b>73.6</b></u> \| <u><b>69.71</b></u> \| <u><b>87.56</b></u> \| <u><b>65.66</b></u> \| <u><b>67.24</b></u> \| <u><b>82.79</b></u> \| <u><b>68.61</b></u> \|
	\| Magic-Dolphin-7b \| 67.48 \| 65.78 \| 85.61 \| 64.64 \| 58.01 \| 79.64 \| 51.18 \|
	\| merlinite-7b \| 64 \| 63.65 \| 84.52 \| 64.91 \| 50.15 \| 79.72 \| 41.09 \|
	[* Open LLM Leaderboard Dataset](https://huggingface.co/datasets/open-llm-leaderboard/details_InferenceIllusionist__Excalibur-7B)

	### Methodology
	[Magic-Dolphin-7b](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b) was an unexpected surprise. Profoundly satisfied with it as a first attempt. For this follow-up I wanted to target the MMLU benchmark specifically.
	The challenge this time was placing more weight on Merlinite-7b as an unknown quantity that hasn't been in the spotlight despite its novel LAB tuning method.

	<b>Excalibur-7b</b> builds on past success and is the culmination of several learnings:
	* Measuring KL-divergences for new quantization types brought a deeper understanding of benchmarking and assessing model performance
	* This signifcantly sped up the testing process by using MMLU as a base, narrowing down over 10 candidate linear merges to 1: merliniteX-blockB1
	* Reaching the limitations of linear merging necessitated a pivot to reviewing the viability of SLERP, DARE-TIES, and Passthrough methods
	* Thus a competing candidate merge pool was tested between different merge algorithms. Once more the list was narrowed from 10 candidates to 1: merliniteX-blockF2
	* merliniteX-blockF2 (SLERP of Magic-Dolphin-7B and jaskier-7b-dpo in unorthadox proportions) was originally planned for release earlier this week
	* Instead -blockB1 and -blockF2 were merged and the results were placed head to head in a final round of tests. Ultimately a more conventional execution of SLERP showed the best results for the final step.



	# Sample Question

	<img src="https://i.imgur.com/fdFYIhv.jpeg" width="550"/>

	# Bonus Question - Vision Capabilities

	<b>Requires additional [mistral-7b-mmproj-v1.5-Q4_1.gguf](https://huggingface.co/koboldcpp/mmproj/tree/main) file for vision functionality</b>
	<img src="https://i.imgur.com/4wbUrjf.jpeg" width="550"/>

	Select up the gguf file of your choice in Kobold as usual, then make sure to choose the mmproj file above in the LLaVA mmproj field of the model submenu:
	<img src="https://i.imgur.com/x8vqH29.png" width="550"/>

	This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

	## Merge Details
	### Merge Method

	This model was merged using the SLERP merge method.

	### Models Merged

	The following models were included in the merge:
	* [ibm/merlinite-7b](https://huggingface.co/ibm/merlinite-7b)
	* [InferenceIllusionist/Magic-Dolphin-7b](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b)
	* [SanjiWatsuki/Kunoichi-DPO-v2-7B](https://huggingface.co/SanjiWatsuki/Kunoichi-DPO-v2-7B)
	* [mlabonne/Monarch-7B](https://huggingface.co/mlabonne/Monarch-7B)
	* [bardsai/jaskier-7b-dpo-v6.1](https://huggingface.co/bardsai/jaskier-7b-dpo-v6.1)

	### Configuration

	The following YAML configurations were used to produce this model:


	<b>merliniteX-blockB1</b>
	```yaml
	models:
	- model: models/merlinite-7b
	parameters:
	weight: 1.0
	- model: models/Kunoichi-DPO-v2-7B
	parameters:
	weight: 0.2
	- model: models/jaskier-7b-dpo-v6.1
	parameters:
	weight: 0.6
	- model: models/Monarch-7b
	parameters:
	weight: 0.4
	merge_method: linear
	dtype: float16
	```

	<b>merliniteX-blockF2</b>
	```yaml
	slices:
	- sources:
	- model: models/Magic-Dolphin-7b
	layer_range: [0, 32]
	- model: models/jaskier-7b-dpo-v6.1
	layer_range: [0, 32]
	merge_method: slerp
	base_model: models/Magic-Dolphin-7b
	parameters:
	t:
	- filter: self_attn
	value: [0, 0.5, 0.3, 0.7, 0.5, 1]
	- filter: mlp
	value: [1, 0.5, 0.7, 0.3, 0.5, 0]
	- value: 0.5 # fallback for rest of tensors
	dtype: float16
	```

	<b>merliniteX-blockH1 (Excalibur-7b)</b>
	```yaml
	slices:
	- sources:
	- model: models/merliniteX-blockF2
	layer_range: [0, 32]
	- model: models/merliniteX-blockB1
	layer_range: [0, 32]
	merge_method: slerp
	base_model: models/merliniteX-blockF2
	parameters:
	t:
	- filter: self_attn
	value: [1, 0.7, 0.3, 0.5, 0]
	- filter: mlp
	value: [0, 0.3, 0.7, 0.5, 1]
	- value: 0.5 # fallback for rest of tensors
	dtype: float16

	```