ShubhamRasal/qwen3-8b-code-navigator

Qwen3-8B Code Navigator

Fine-tuned Qwen3-8B to efficiently use grep, find, bash, and file editing tools for code repository navigation.

Training Recipe

Based on SWE-Master (2025) and SWE-Dev (2025) methodologies:

• Base model: Qwen/Qwen3-8B
• Dataset: SWE-bench/SWE-smith-trajectories (tool split, resolved only → ~5K trajectories)
• Method: SFT with LoRA (r=64, α=128) + assistant-only loss masking
• Tools: bash (grep, find, cat, etc.) + str_replace_editor (view/edit files)
• Context: 16K tokens
• Epochs: 2
• Hardware: A100-80GB recommended

Key Design Decisions

1. Assistant-only loss: Only trains on model's reasoning + tool calls, not on bash outputs or system prompts (following SWE-Master §3.3)
2. Proper tool_calls format: Converts SWE-smith's XML <function=bash> format to Qwen3's native <tool_call> format
3. Observation truncation: Long bash outputs are truncated to prevent wasting context on noise
4. LoRA targets: All attention + MLP projections for maximum expressiveness

How to Train

pip install transformers trl torch datasets trackio accelerate peft flash-attn

# Single GPU (A100-80GB)
python train.py

# Multi-GPU with accelerate
accelerate launch train.py

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base_model, "ShubhamRasal/qwen3-8b-code-navigator")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

tools = [
    {"type": "function", "function": {
        "name": "bash",
        "description": "Execute a bash command",
        "parameters": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}
    }}
]

messages = [
    {"role": "system", "content": "You are an expert software engineer that navigates code repositories using bash commands."},
    {"role": "user", "content": "Find all Python files that implement authentication in this Django project at /repo"}
]

text = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:]))

Dataset Preprocessing

The SWE-smith trajectories use an XML function call format:

<function=bash>
<parameter=command>grep -r "auth" /testbed --include="*.py"</parameter>
</function>

This is converted to Qwen3's native tool calling format:

<tool_call>
{"name": "bash", "arguments": {"command": "grep -r \"auth\" /testbed --include=\"*.py\""}}
</tool_call>

References

• SWE-Master: Multi-Turn SFT for SWE Agents (2025)
• SWE-smith: Scaling Data for Software Engineering Agents (2025)
• SWE-Dev: Training LLMs for SWE with Execution Feedback (2025)