Instructions to use Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD") model = AutoModelForCausalLM.from_pretrained("Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD
- SGLang
How to use Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD with Docker Model Runner:
docker model run hf.co/Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD
Qwen-3.5-4B-Auto-ASVD
Qwen-3.5-4B-Auto-ASVD is a structurally factorized and fine-tuned release of Qwen/Qwen3.5-4B. It uses an automated second-order pruning framework that combines empirical Hessian sensitivity profiling (Fisher Information) with Activation-Aware Singular Value Decomposition (ASVD) and a subsequent LoRA healing stage with cosine learning rate scheduling.
All operations were executed and merged in native bfloat16. Unlike standard uniform low-rank matrix factorizations, this variant calculates dynamic layer-by-layer variance retention targets based on loss surface curvature, avoiding arbitrary flat rank truncation across attention heads and MLP blocks.
Engineering Pipeline
1. Second-Order Curvature Profiling (Hessian Approximation)
To prevent the catastrophic degradation common in uniform pruning, gradient energy across all projection modules was sampled over long sequences (1024-token chunks) using gradient checkpointing:
Modules exhibit starkly distinct curvature profilesโespecially with Qwen 3.5's hybrid architecture comprising both standard attention and Gated DeltaNet layers. Rather than applying a linear scaling function dominated by extreme outliers, sensitivity scores were mapped to variance thresholds via log-smoothing:
This automatically allocated higher variance budgets (up to 95.00%) to core routing paths like down_proj, while tolerating tighter truncation (down to 91.60%) on less sensitive projection matrices.
2. Scaled SVD Surgery
Each target linear operator was scaled using activation RMS values collected during forward passes:
Truncated SVD was performed on-device:
where the rank $k$ was resolved per matrix to satisfy the layer-specific variance budget. The factors were then mapped into sequential bottleneck layers (nn.Sequential(Linear(in, k), Linear(k, out))) and converted to bfloat16.
- Target Layers Factored: 64 projection matrices (
q_proj,k_proj,v_proj,o_proj,down_proj). - Parameters Excised: 47,821,312 parameters (6.86% structural reduction across target matrices).
3. LoRA Neural Healing
Following structural factorization, low-rank degradation was resolved via parameter-efficient fine-tuning on Wikitext-2:
- Adapter Topology: Rank 32, Alpha 64 across all linear layers.
- Optimization: AdamW (lr = 2e-4), gradient clipping (1.0), micro-batch size 1, gradient accumulation steps 16 (effective batch size 16).
- Schedule: Cosine decay with 10% warmup over 2 epochs.
- Loss Trajectory: Initial loss converged from 2.2911 (Epoch 1) down to 2.0432 (Epoch 2).
All adapters were permanently merged back into the base weights (merge_and_unload), producing a standalone checkpoint with zero runtime inference overhead or custom kernel dependencies.
Run Metrics
| Stage | Detail | Value |
|---|---|---|
| Base Weights | Architecture | Qwen/Qwen3.5-4B |
| Precision | Runtime / Storage | bfloat16 Native |
| Hardware | Execution Node | NVIDIA A40 (48GB VRAM) |
| Calibration Length | Context Window | 1024 tokens |
| Target Projections | Linear Modules | q_proj, k_proj, v_proj, o_proj, down_proj |
| Observed Variance Bounds | Min / Max Allocated | 91.60% โ 95.00% |
| Net Parameter Reduction | Pruned from Target Layers | 47,821,312 |
| Final Healing Loss | Cross-Entropy (Epoch 2) | 2.0432 |
Quickstart
This model is a drop-in replacement for any pipeline supporting standard causal language models:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = "The most important concept in quantum physics is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.15
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
- Specialized Domain Drift: The healing phase utilized a 1024-token calibration stream from general natural language data (Wikitext-2). For domain-specific production workloads (e.g., medical diagnostics, strict syntactic code generation, legal drafting), task-specific fine-tuning on top of this checkpoint is recommended.
- Kernel Dependencies: When running the base architecture, warnings regarding missing optional acceleration libraries (
causal_conv1d,flash-linear-attention) may appear in vanilla PyTorch environments. The model defaults safely to standard reference operations without loss of accuracy.
- Downloads last month
- 222