Nexa_Sci_distilled_Falcon-10B

LoRA-tuned Falcon3‑10B for high-fidelity scientific question answering and methodology drafting.

---

Model Overview

	Details
Base model	tiiuae/Falcon3-10B-Base
Method	QLoRA (4‑bit quantization; rank 64 adapters)
Trainable parameters	26,214,400 (≈0.25 % of the base model)
Training corpus	sft_scientific_v1 (filtered teacher generations across biology, physics, materials science)
Training window	2025‑11‑05 → 2025‑11‑06

Highlights

• Validation loss: 0.410
• Test loss: 0.413
• Generates structured, technically grounded scientific responses
• Rubric (judge) scoring is pending for the final checkpoint

---

Intended Use

• Scientific question answering assistants
• Lab protocol / methodology drafting
• Hypothesis ideation and critique

Not suitable for medical, legal, or other high-stakes decisions without human oversight.

---

Quickstart

1. Load merged weights (single-step deployment)

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "NexaCompute/Nexa_Sci_distilled_Falcon-10B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Explain how superconductors enable near-lossless power transmission in urban grids."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.2,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Load as LoRA adapter (if you prefer to manage the base model separately)

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "tiiuae/Falcon3-10B-Base",
    device_map="auto",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)

adapter = PeftModel.from_pretrained(base, "NexaCompute/Nexa_Sci_distilled_Falcon-10B")
adapter = adapter.merge_and_unload()  # optional
tokenizer = AutoTokenizer.from_pretrained("NexaCompute/Nexa_Sci_distilled_Falcon-10B")

---

Sample Output

Prompt
Design a reproducible lab experiment to measure photosynthesis rate in freshwater algae.

Model output (abridged)

1. Hypothesis: CO₂ enrichment increases dissolved O₂ production in Chlorella vulgaris.
2. Materials: 3× 500 mL photobioreactors, dissolved oxygen probes, LED array (250 µmol·m⁻²·s⁻¹), CO₂ regulator...
3. Procedure:
   - Equilibrate cultures at 25 °C for 48 h.
   - Pressurize headspace to 0.5, 1.0, 1.5 atm CO₂ for 40 min intervals.
   - Record dissolved O₂ every 5 min; compute linear slopes over 30 min windows.
4. Analysis: Fit O₂ flux vs CO₂ concentration; report slope, R², and p-value.

---

Training & Infrastructure

Data

Filtered teacher generations across biology, physics, and materials science. Only examples passing dual-judge quality gates were retained (sft_scientific_v1).

Hyperparameters

• Effective batch size: 32 (QLoRA, gradient accumulation 4, 2 GPUs)
• Epochs: 3 (early stop at step 200)
• Optimizer: AdamW (lr 2e‑4, weight decay 0.01)
• Precision: BF16

Hardware & Cost

• 2 × A100 80 GB (Prime Intellect) for ~$2/hr
• Estimated training cost ≈ USD 18

---

Evaluation Summary

Metric	Value	Notes
Val loss	0.410	QLoRA adapters
Test loss	0.413	Held-out scientific QA
Judge	—	Rubric evaluation pending

Final rubric scoring will be published once inference benchmarking completes.

---

Limitations & Risks

• Knowledge cutoff from Falcon3-10B-Base; recent findings may be missing.
• May produce invented citations or experimental details—verify before use.
• No RLHF/safety fine-tuning; human review is essential.
• Current PyTorch wheels lack CUDA sm_120 kernels, so RTX 5090 inference requires rebuilt binaries or alternative hardware.

---

Responsible Use

• Keep a human in the loop for experimental or safety-critical decisions.
• Do not deploy in clinical, legal, or security domains without additional validation.
• Report issues or unsafe behaviors via NexaCompute support channels.

---

Changelog

• 2025-11-06 — Initial release containing merged weights, tokenizer, and training summary.