Add pipeline tag, library name and link to Github repository

This PR improves the model card by adding the `text-generation` pipeline tag and `transformers` library name. It also adds a link to the Github repository for the project and a link to the paper on Hugging Face Papers.

README.md

@@ -1,51 +1,49 @@
 ---
-license: mit
 datasets:
 - cerebras/SlimPajama-627B
 language:
 - en
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
 ---
 
-# PRRC-Reasoning Language Model (1.3B Parameters, 30B Tokens)
+# Meta-rater Language Model (3.3B Parameters, 100B Tokens)
 
 ## Model Description
 
-This is a 1.3B parameter transformer-based decoder-only language model trained from scratch on 30B tokens selected from SlimPajama dataset using the **Reasoning** dimension of the PRRC framework. The training data was curated by selecting text with high reasoning complexity, focusing on content that requires multi-step logical analysis and critical thinking.
+This is a 3.3B parameter transformer-based decoder-only language model trained from scratch on 100B tokens selected from SlimPajama dataset using the **Meta-rater** framework with all 25 quality scores. This model demonstrates the scalability of Meta-rater's data selection benefits to larger model sizes and training datasets.
 
 ## Model Details
 
 - **Architecture**: Transformer decoder-only
-- **Parameters**: 1.345B (1,345,423,360 parameters)
-- **Training Tokens**: 30B tokens
+- **Parameters**: 3.3B (3,335,989,760 parameters)
+- **Training Tokens**: 100B tokens
 - **Context Window**: 1,024 tokens
 - **Vocabulary Size**: 32,000 (LLaMA tokenizer)
-- **Data Selection Method**: Top-k selection based on Reasoning scores
-- **Rating Model**: ModernBERT-base fine-tuned for Reasoning assessment
+- **Data Selection Method**: Meta-rater with all 25 quality scores
+- **Optimization**: Learned optimal weightings from 1.3B experiments
 
 ## Architecture Specifications
 
-- **Hidden Dimension**: 2,048
-- **Number of Layers**: 24
-- **Attention Heads**: 16
-- **Key-Value Heads**: 16
+- **Hidden Dimension**: 2,560
+- **Number of Layers**: 40
+- **Attention Heads**: 20
+- **Key-Value Heads**: 20
 - **MLP Ratio**: 8/3
 - **Position Encoding**: RoPE (base=10,000)
 
-## Data Selection Criteria
+## Data Selection Framework
+
+The training data was selected using the same Meta-rater framework as the 1.3B models, leveraging:
 
-The training data was selected using the Reasoning rating model, which evaluates:
-- **Logical Structure**: Multi-step reasoning and argument chains
-- **Analytical Depth**: Complex analysis and critical evaluation
-- **Causal Relationships**: Identification and exploration of cause-effect patterns
-- **Problem Solving**: Strategic thinking and solution development
-- **Evidence Integration**: Synthesis of multiple information sources
+### Quality Score Integration (25 total)
+- **Natural Language Quality Signals (11)**: RedPajama rule-based measures
+- **Data Importance Scores (3)**: DSIR similarity to Books, Wikipedia, AutoMathText
+- **Model-based Ratings (11)**: PRRC + QuRating + FineWeb-Edu + WanjuanCC
 
-Selected texts typically include:
-- Analytical essays and research papers
-- Problem-solving discussions and case studies
-- Philosophical and scientific arguments
-- Strategic planning documents
-- Complex technical analyses
+### Optimal Weighting Strategy
+The same learned weights from 1.3B proxy model experiments were applied, ensuring consistent data selection criteria across scales.
 
 ## Training Details
 
@@ -53,34 +51,44 @@ Selected texts typically include:
 - **Global Batch Size**: 4,194,304 tokens
 - **Learning Rate**: 5e-5
 - **Optimizer**: Adam (β₁=0.9, β₂=0.95, ε=1e-8)
-- **Training Time**: ~14 hours
+- **Training Time**: ~129 hours
 
 ## Performance Results
 
 ### Downstream Task Performance (Average Accuracy)
 
-- **General Knowledge**: 55.57% (+2.78% vs Random)
-  - ARC-Easy: 55.35%
-  - ARC-Challenge: 27.05%
-  - SciQ: 84.30%
+- **General Knowledge**: 67.51% (+3.29% vs Random 3.3B)
+  - ARC-Easy: 72.10%
+  - ARC-Challenge: 37.54%
+  - SciQ: 92.90%
 
-- **Commonsense Reasoning**: 44.86% (+0.92% vs Random)
-  - HellaSwag: 41.34%
-  - SIQA: 40.36%
-  - WinoGrande: 52.87%
+- **Commonsense Reasoning**: 54.35% (+0.80% vs Random 3.3B)
+  - HellaSwag: 58.99%
+  - SIQA: 43.91%
+  - WinoGrande: 60.14%
 
-- **Reading Comprehension**: 30.48% (+0.46% vs Random)
-  - RACE: 30.95%
-  - OpenbookQA: 30.00%
+- **Reading Comprehension**: 36.06% (+0.78% vs Random 3.3B)
+  - RACE: 35.12%
+  - OpenbookQA: 37.00%
 
-- **Overall Average**: 45.28% (+1.50% vs Random)
+- **Overall Average**: 54.71% (+1.73% vs Random 3.3B)
 
-## Key Findings
+### Knowledge-Intensive Tasks
+- **MMLU**: 26.21% (+0.73% vs Random 3.3B)
+- **NaturalQuestions**: 6.87% (+0.59% vs Random 3.3B)
 
-- **Reasoning Enhancement**: Improved logical thinking and analysis capabilities
-- **Problem Solving**: Enhanced ability to work through complex problems
-- **Knowledge Application**: Better at applying knowledge to new situations
-- **Analytical Skills**: Stronger performance in tasks requiring multi-step reasoning
+## Scaling Validation
+
+### Benefits Persist at Scale
+Compared to the 1.3B Meta-rater model results:
+- **Consistent Improvements**: Similar relative gains maintained at larger scale
+- **Absolute Performance**: Substantial improvements in all categories
+- **Efficiency**: Data selection remains valuable even with more parameters
+
+### Cross-Scale Comparison
+- **1.3B Meta-rater**: 47.01% overall
+- **3.3B Meta-rater**: 54.71% overall (+7.70% from scaling)
+- **Scale Efficiency**: ~2.5x parameters yield significant performance gains
 
 ## Usage
 
@@ -89,18 +97,18 @@ from transformers import AutoTokenizer, AutoModelForCausalLM
 import torch
 
 # Load model and tokenizer
-model_name = "opendatalab/meta-rater-1b-reasoning"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name)
+model_name = "opendatalab/meta-rater-3b-25raters"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
 
-# Generate text (particularly good for analytical content)
-prompt = "To solve this problem, we need to consider several factors:"
+# Generate text (optimized for high-quality output)
+prompt = "The key principles of sustainable development include"
 inputs = tokenizer(prompt, return_tensors="pt")
 
 with torch.no_grad():
     outputs = model.generate(
         inputs.input_ids,
-        max_length=100,
+        max_length=150,
         temperature=0.7,
         do_sample=True,
         pad_token_id=tokenizer.eos_token_id
@@ -112,47 +120,56 @@ print(generated_text)
 
 ## Applications
 
-This model is particularly well-suited for:
-- **Analytical writing** and problem-solving tasks
-- **Educational content** focused on critical thinking
-- **Research assistance** and hypothesis development
-- **Strategic planning** and decision-making support
-- **Complex reasoning** tasks and logic puzzles
-- **Academic writing** requiring argumentation
-- **Case study** analysis and evaluation
+This model is well-suited for:
+- **Production applications** requiring high-quality text generation
+- **Research** needing stronger baseline performance
+- **Educational platforms** with diverse content requirements
+- **Content creation** at scale with quality assurance
+- **Multi-domain applications** benefiting from balanced capabilities
+- **Scaling studies** for data selection methodologies
+
+## Key Achievements
+
+- **Scalability Validation**: Confirms Meta-rater benefits persist at larger scales
+- **Improved Baselines**: Establishes stronger performance benchmarks
+- **Efficiency Demonstration**: Better results with same computational budget
+- **Quality Consistency**: Maintains data selection advantages across scales
+
+## Research Significance
+
+This model provides crucial evidence for:
+- **Scaling Laws**: Data quality benefits don't diminish with model size
+- **Efficiency**: Quality data selection remains valuable at any scale
+- **Methodology Robustness**: Meta-rater framework generalizes across sizes
+- **Cost-Effectiveness**: Better performance without additional training costs
 
 ## Strengths
 
-- Enhanced logical reasoning and analytical capabilities
-- Improved problem-solving approach and methodology
-- Better at handling complex, multi-step arguments
-- Strong performance on knowledge-intensive reasoning tasks
-- Effective at synthesizing information from multiple sources
-- Good at identifying causal relationships and patterns
+- Enhanced performance across all evaluation categories
+- Scalable data selection methodology
+- Improved knowledge retention and reasoning
+- Consistent quality improvements over random selection
+- Validated framework transferability
 
 ## Limitations
 
-- May generate overly complex reasoning for simple questions
-- Could prioritize analytical depth over accessibility
+- Higher computational requirements for training
 - Limited context window (1,024 tokens)
 - No instruction tuning or safety alignment
-- May struggle with creative or intuitive tasks
+- Requires quality score preprocessing
+- Same data selection overhead as smaller models
 
-## Reasoning Capabilities
+## Comparison Summary
 
-This model demonstrates enhanced abilities in:
-- **Deductive Reasoning**: Drawing logical conclusions from premises
-- **Inductive Reasoning**: Identifying patterns and generalizations
-- **Causal Analysis**: Understanding cause-and-effect relationships
-- **Problem Decomposition**: Breaking complex problems into manageable parts
-- **Evidence Evaluation**: Assessing the strength and relevance of information
-- **Hypothesis Formation**: Developing testable explanations
+### vs Random 3.3B Baseline
+- **Overall**: +1.73% improvement (54.71% vs 52.98%)
+- **General Knowledge**: +3.29% improvement (strongest category)
+- **All Categories**: Consistent improvements across all task types
 
-## Comparison with Baselines
-
-- **vs Random Baseline**: +1.50% overall, with consistent improvements across categories
-- **vs Other PRRC Dimensions**: Competitive performance with focus on analytical tasks
-- **vs Meta-rater All (25)**: Shows specialized improvement in reasoning-heavy applications
+### vs 1.3B Meta-rater
+- **Scale Benefits**: +7.70% improvement from increased parameters
+- **Framework Consistency**: Same data selection principles apply effectively
+- **Efficiency**: Larger models can better utilize high-quality data
 
 ## Citation
 
@@ -167,10 +184,19 @@ If you use this model in your research, please cite:
 }
 ```
 
+## Related Resources
+
+- **1.3B Meta-rater Models**: Smaller-scale versions with detailed analysis
+- **PRRC Rating Models**: Quality assessment models used for data selection
+- **Annotated SlimPajama**: Complete dataset with quality scores
+- **Random Baselines**: Corresponding baseline models for comparison
+- **Project Page**: [Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models](https://huggingface.co/papers/2504.14194)
+- **Github**: [Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models](https://github.com/opendatalab/Meta-rater)
+
 ## License
 
 Please refer to the license terms of the original SlimPajama dataset and follow applicable data licensing requirements.
 
 ## Contact
 
-For questions or issues, please contact the authors or open an issue in the repository. 
+For questions or issues, please contact the authors or open an issue in the repository.