Qwen-3.5-4B-Auto-ASVD

Qwen-3.5-4B-Auto-ASVD is a structurally factorized and fine-tuned release of Qwen/Qwen3.5-4B. It uses an automated second-order pruning framework that combines empirical Hessian sensitivity profiling (Fisher Information) with Activation-Aware Singular Value Decomposition (ASVD) and a subsequent LoRA healing stage with cosine learning rate scheduling.

All operations were executed and merged in native bfloat16. Unlike standard uniform low-rank matrix factorizations, this variant calculates dynamic layer-by-layer variance retention targets based on loss surface curvature, avoiding arbitrary flat rank truncation across attention heads and MLP blocks.


Engineering Pipeline

1. Second-Order Curvature Profiling (Hessian Approximation)

To prevent the catastrophic degradation common in uniform pruning, gradient energy across all projection modules was sampled over long sequences (1024-token chunks) using gradient checkpointing:

diag(H)โ‰ˆE[(โˆ‡WL)โˆ˜2]\text{diag}(H) \approx \mathbb{E}\left[ (\nabla_W \mathcal{L})^{\circ 2} \right]

Modules exhibit starkly distinct curvature profilesโ€”especially with Qwen 3.5's hybrid architecture comprising both standard attention and Gated DeltaNet layers. Rather than applying a linear scaling function dominated by extreme outliers, sensitivity scores were mapped to variance thresholds via log-smoothing:

s~=logโก(1+s)โˆ’logโก(1+sminโก)logโก(1+smaxโก)โˆ’logโก(1+sminโก)+ฯต\tilde{s} = \frac{\log(1 + s) - \log(1 + s_{\min})}{\log(1 + s_{\max}) - \log(1 + s_{\min}) + \epsilon}

Target Variance=0.82+0.13โ‹…s~\text{Target Variance} = 0.82 + 0.13 \cdot \tilde{s}

This automatically allocated higher variance budgets (up to 95.00%) to core routing paths like down_proj, while tolerating tighter truncation (down to 91.60%) on less sensitive projection matrices.

2. Scaled SVD Surgery

Each target linear operator was scaled using activation RMS values collected during forward passes:

Wscaled=WโŠ™sRMSW_{\text{scaled}} = W \odot s_{\text{RMS}}

Truncated SVD was performed on-device:

Wscaledโ‰ˆUkฮฃkVkTW_{\text{scaled}} \approx U_k \Sigma_k V_k^T

where the rank $k$ was resolved per matrix to satisfy the layer-specific variance budget. The factors were then mapped into sequential bottleneck layers (nn.Sequential(Linear(in, k), Linear(k, out))) and converted to bfloat16.

  • Target Layers Factored: 64 projection matrices (q_proj, k_proj, v_proj, o_proj, down_proj).
  • Parameters Excised: 47,821,312 parameters (6.86% structural reduction across target matrices).

3. LoRA Neural Healing

Following structural factorization, low-rank degradation was resolved via parameter-efficient fine-tuning on Wikitext-2:

  • Adapter Topology: Rank 32, Alpha 64 across all linear layers.
  • Optimization: AdamW (lr = 2e-4), gradient clipping (1.0), micro-batch size 1, gradient accumulation steps 16 (effective batch size 16).
  • Schedule: Cosine decay with 10% warmup over 2 epochs.
  • Loss Trajectory: Initial loss converged from 2.2911 (Epoch 1) down to 2.0432 (Epoch 2).

All adapters were permanently merged back into the base weights (merge_and_unload), producing a standalone checkpoint with zero runtime inference overhead or custom kernel dependencies.


Run Metrics

Stage Detail Value
Base Weights Architecture Qwen/Qwen3.5-4B
Precision Runtime / Storage bfloat16 Native
Hardware Execution Node NVIDIA A40 (48GB VRAM)
Calibration Length Context Window 1024 tokens
Target Projections Linear Modules q_proj, k_proj, v_proj, o_proj, down_proj
Observed Variance Bounds Min / Max Allocated 91.60% โ€“ 95.00%
Net Parameter Reduction Pruned from Target Layers 47,821,312
Final Healing Loss Cross-Entropy (Epoch 2) 2.0432

Quickstart

This model is a drop-in replacement for any pipeline supporting standard causal language models:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "The most important concept in quantum physics is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

  • Specialized Domain Drift: The healing phase utilized a 1024-token calibration stream from general natural language data (Wikitext-2). For domain-specific production workloads (e.g., medical diagnostics, strict syntactic code generation, legal drafting), task-specific fine-tuning on top of this checkpoint is recommended.
  • Kernel Dependencies: When running the base architecture, warnings regarding missing optional acceleration libraries (causal_conv1d, flash-linear-attention) may appear in vanilla PyTorch environments. The model defaults safely to standard reference operations without loss of accuracy.
Downloads last month
222
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(1,460)
this model
Quantizations
1 model

Collection including Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD