Spaces:
Sleeping
Download RULES_JUDGING.md from goblinasaddy/research_env: direct link, hf CLI and curl.
- Browser
- Download file 4.83 kB
-
https://huggingface.co/spaces/goblinasaddy/research_env/resolve/main/RULES_JUDGING.md
- Command line
-
hf download hf://spaces/goblinasaddy/research_env/RULES_JUDGING.md
-
curl -L -o RULES_JUDGING.md https://huggingface.co/spaces/goblinasaddy/research_env/resolve/main/RULES_JUDGING.md
π Hackathon Rules & Judging Criteria
π Eligibility & Rules
Team Rules
- Team Size: 1β4 members per team
- Submission: Single GitHub repository + Hugging Face Space URL
- Ownership: All work must be original and created during the hackathon
Code Submission Requirements
Repository must be public
Must include:
README.md(clear explanation of environment)openenv.yamlDockerfileinference.py(MANDATORY)
Must deploy successfully on Hugging Face Spaces
Must run within:
- β€ 20 minutes runtime
- 2 vCPU, 8GB RAM constraint
Functional Requirements
Your submission MUST:
Simulate a real-world task (NOT games or toy problems)
Implement full OpenEnv spec:
reset()step(action)state()
Use typed models (Action, Observation, Reward)
Include β₯ 3 tasks:
- Easy β Medium β Hard
Provide deterministic graders:
- Score range: 0.0 β 1.0
Implement a meaningful reward function:
- Must provide partial progress signals
Include a baseline inference script:
- Must produce reproducible scores
Non-Functional Requirements
Must deploy as a Hugging Face Space
Must include a working Dockerfile
Must pass:
openenv validateMust expose working endpoints:
/reset/step
Allowed
- External libraries (NumPy, PyTorch, etc.)
- Pre-trained models
- Open source tools
- LLM APIs (via OpenAI client)
Not Allowed
- Plagiarism or copied environments
- Trivial modifications of existing environments
- Hardcoded grading logic
- Constant reward outputs
- Missing inference script
- Environments that fail to deploy or respond
βοΈ Evaluation Criteria
Scoring Breakdown (Total: 100%)
1. Real-World Utility (30%)
- Does the environment model a genuine real-world task?
- Would this be useful for training or evaluating agents?
| Score Range | Interpretation |
|---|---|
| 0β5 | Toy / unrealistic |
| 6β15 | Valid idea but shallow |
| 16β25 | Good practical value |
| 26β30 | Excellent, high-impact |
2. Task & Grader Quality (25%)
Minimum 3 tasks with increasing difficulty
Clear objectives for each task
Graders:
- Deterministic
- Non-constant
- Produce scores in
[0.0, 1.0]
Hard tasks must challenge strong agents
3. Environment Design (20%)
Clean state transitions (
reset β step β state)Well-defined action and observation spaces
Reward function:
- Dense and meaningful
- Encourages progress
Proper episode termination
4. Code Quality & Spec Compliance (15%)
- Passes
openenv validate - Docker builds successfully
- HF Space deploys and responds
inference.pyruns without errors- Clean, modular, documented code
5. Creativity & Novelty (10%)
- Unique problem domain
- Interesting mechanics
- Innovative reward design
- Not a standard or overused environment
βοΈ Judging Process
Phase 1 β Automated Validation (Pass/Fail)
Submissions must pass ALL:
- HF Space responds to
/reset - Docker builds successfully
- OpenEnv validation passes
inference.pyexecutes successfully- β₯ 3 tasks with graders
- Scores are within
[0.0, 1.0]
β Failure in any step β Disqualification
Phase 2 β Agentic Evaluation
- Baseline agent is executed
- External LLM agent (e.g., Nemotron) is evaluated
Evaluation checks:
- Score differences across agents (variance required)
- Reward signal correctness
- Task difficulty progression
- Agent learning behavior across steps
Phase 3 β Human Review
Top submissions are reviewed by Meta & Hugging Face engineers based on:
- Real-world applicability
- Creativity and innovation
- Robustness (cannot be easily exploited)
- Overall design quality
π« Disqualification Criteria
Projects will be disqualified if:
- Environment does not deploy or respond
- Missing
inference.py - Graders always return same score
- Plagiarized or copied work
- Broken Docker or API
- No meaningful reward function
- Runtime exceeds limits
π¦ Submission Checklist
Before submitting, ensure:
- β HF Space is live and responding
- β
openenv validatepasses - β Docker builds and runs
- β
inference.pyruns successfully - β All tasks return valid scores
- β Runtime < 20 minutes
π Final Note
This hackathon is not about building a UI or product.
It is about building a high-quality evaluation environment where:
- Agents can interact
- Performance can be measured
- Intelligence can be tested
Focus on environment quality β not presentation.