# Qwen3-8B Code Navigator

Fine-tuned Qwen3-8B to efficiently use `grep`, `find`, `bash`, and file editing tools for code repository navigation.

## Training Recipe

Based on **SWE-Master (2025)** and **SWE-Dev (2025)** methodologies:

- **Base model**: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)
- **Dataset**: [SWE-bench/SWE-smith-trajectories](https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories) (tool split, resolved only → ~5K trajectories)
- **Method**: SFT with LoRA (r=64, α=128) + assistant-only loss masking
- **Tools**: `bash` (grep, find, cat, etc.) + `str_replace_editor` (view/edit files)
- **Context**: 16K tokens
- **Epochs**: 2
- **Hardware**: A100-80GB recommended

## Key Design Decisions

1. **Assistant-only loss**: Only trains on model's reasoning + tool calls, not on bash outputs or system prompts (following SWE-Master §3.3)
2. **Proper tool_calls format**: Converts SWE-smith's XML `<function=bash>` format to Qwen3's native `<tool_call>` format
3. **Observation truncation**: Long bash outputs are truncated to prevent wasting context on noise
4. **LoRA targets**: All attention + MLP projections for maximum expressiveness

## How to Train

```bash
pip install transformers trl torch datasets trackio accelerate peft flash-attn

# Single GPU (A100-80GB)
python train.py

# Multi-GPU with accelerate
accelerate launch train.py
```

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base_model, "ShubhamRasal/qwen3-8b-code-navigator")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

tools = [
    {"type": "function", "function": {
        "name": "bash",
        "description": "Execute a bash command",
        "parameters": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}
    }}
]

messages = [
    {"role": "system", "content": "You are an expert software engineer that navigates code repositories using bash commands."},
    {"role": "user", "content": "Find all Python files that implement authentication in this Django project at /repo"}
]

text = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:]))
```

## Dataset Preprocessing

The SWE-smith trajectories use an XML function call format:
```
<function=bash>
<parameter=command>grep -r "auth" /testbed --include="*.py"</parameter>
</function>
```

This is converted to Qwen3's native tool calling format:
```
<tool_call>
{"name": "bash", "arguments": {"command": "grep -r \"auth\" /testbed --include=\"*.py\""}}
</tool_call>
```

## References

- [SWE-Master: Multi-Turn SFT for SWE Agents](https://arxiv.org/abs/2504.11862) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://arxiv.org/abs/2504.21798) (2025)
- [SWE-Dev: Training LLMs for SWE with Execution Feedback](https://arxiv.org/abs/2505.06641) (2025)