A technical audit of DavidAU and Nightmedia model claims

Scope: This article audits public model-card claims, benchmark methodology, and reproducibility around a sample of models published by DavidAU and Nightmedia on Hugging Face.

Important: I am not claiming that every benchmark number shown on these pages was fabricated. A benchmark output can be real while the conclusion drawn from it is unsupported, misleading, internally contradictory, or not reproducible from the information provided.

---

TL;DR

Across the model cards and public replies reviewed here, I found a recurring pattern:

1. A narrow legacy suite — usually ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, OpenBookQA, PIQA, and WinoGrande — is repeatedly treated as if it were a broad measurement of intelligence or "ability to think."
2. Small improvements on that suite are sometimes escalated into much larger claims about frontier-level intelligence, "strongest/smartest" positioning, or an "OpenAI/Claude/Gemini" intelligence zone.
3. Long-context configurations labeled 1M or 2M are evaluated using short multiple-choice tasks and an MLX perplexity pipeline whose default sequence length is 512 tokens. That does not validate real 1M/2M long-context capability.
4. The default mlx_lm.perplexity pipeline uses allenai/tulu-3-sft-mixture, uses its training split, concatenates tokenized examples before reshaping them into fixed-length blocks, and computes its displayed standard error from token-level losses.
5. "Tokens/sec" from that perplexity script is evaluation throughput, not ordinary autoregressive generation speed.
6. mlx_lm.evaluate does not define a fixed seven-task "standard MLX metrics" suite. It requires the evaluator to explicitly choose --tasks, then delegates evaluation to lm-evaluation-harness.
7. Keeping a fixed historical regression suite is valid, but it does not make that suite sufficient for every modern capability claim. lm-evaluation-harness itself versions task definitions, and modern capabilities require additional capability-matched evaluations.
8. Some model-card claims are directly contradicted by the same card — for example, "Fully uncensored" while the card reports 6 refusals out of 100.
9. Modern official base-model cards report broad capability batteries such as SWE-bench Verified, SWE-bench Pro, Terminal-Bench, GPQA Diamond, LiveCodeBench, HMMT, AIME, and others. If a fine-tune is advertised as a general intelligence improvement, showing only the old seven-task suite does not establish that the base model's modern capabilities were preserved, let alone improved.

The core issue is not whether I "like" these models.

The issue is:

Do the published experiments actually support the capability claims being made?

In many cases below, the answer is no.

---

1. How I classify claims

To keep this technical rather than personal, I use the following labels:

This distinction matters.

Calling something unsupported is not the same as saying the underlying number was fabricated.

---

2. The seven-benchmark problem

A recurring suite across the cards reviewed here is:

• ARC-Challenge
• ARC-Easy
• BoolQ
• HellaSwag
• OpenBookQA
• PIQA
• WinoGrande

These are legitimate benchmarks. The problem is not that they are fake or useless.

The problem is what is being inferred from them.

They were released roughly in the 2018–2020 period and target different narrow abilities:

• BoolQ: yes/no reading comprehension.
• PIQA: physical commonsense.
• HellaSwag: commonsense continuation / multiple-choice inference.
• WinoGrande: Winograd-style commonsense/coreference.
• ARC / OpenBookQA: science-question answering with a mixture of knowledge and reasoning.

That is not the same thing as a complete measurement of modern LLM intelligence.

A seven-task multiple-choice suite does not directly measure:

• repository-level coding,
• terminal autonomy,
• long-horizon planning,
• tool calling,
• function calling,
• browser use,
• agentic recovery from failures,
• software engineering,
• long-context retrieval at hundreds of thousands of tokens,
• modern math competition performance,
• difficult scientific reasoning,
• multimodal agent performance.

Preserving this suite for historical comparison is reasonable.

Using it as the primary evidence for sweeping general-intelligence or frontier-model claims is the problem.

3. ARC-C = 0.700 is not an established "frontier intelligence" threshold

One of the strongest claims appears on DavidAU's Qwen3.6-27B Fable Fusion 711 card, which associates ARC-C around 0.700 with an "OpenAI, Claude and Gemini 'zone of intelligence'" and also uses the phrase "This is the one they fear."

There is no recognized scientific standard where:

ARC-C >= 0.700 = OpenAI / Claude / Gemini intelligence

No such frontier threshold is defined by ARC's creators, by lm-evaluation-harness, or by the major frontier labs.

A 2025 ACL paper by Łukasz Borchmann, In Case You Missed It: ARC "Challenge" Is Not That Challenging, showed that ARC-Challenge's apparent difficulty is highly sensitive to evaluation format. The paper argues that much of the perceived difficulty comes from the evaluation setup rather than only the inherent complexity of the questions.

That means ARC-C can still be useful as a historical benchmark, but treating one particular value such as 0.700 as a universal intelligence boundary is not justified.

Status: UNSUPPORTED / MISLEADING EXTRAPOLATION

📸 Screenshot slot B — "700 intelligence club"

---

4. What the official Qwen3.6-27B card measures instead

This comparison matters because DavidAU's Fable Fusion model is built from Qwen3.6-27B.

The official Qwen3.6-27B card reports a much broader evaluation battery, including results such as:

The point is not that every community fine-tune must rerun every expensive benchmark.

The point is that if a fine-tune is advertised as having improved general intelligence, problem solving, or as being among the strongest models overall, then seven short legacy tasks cannot tell us whether it preserved:

• the base model's SWE-bench ability,
• terminal performance,
• coding,
• hard science,
• math,
• tool use,
• or long-horizon agentic behavior.

Fine-tuning and merging can improve one area while degrading another.

A model card cannot establish "top-to-bottom" capability improvement by only publishing the dimensions that went up.

---

5. DavidAU audit

5.1 Qwen3.6-27B Fable Fusion 711

Model: https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF

Claim: frontier-zone intelligence from ARC-C

The card directly connects its ARC-C result with an OpenAI/Claude/Gemini-level "zone."

As explained above, ARC-C 0.700 is not a recognized frontier-intelligence threshold.

Status: UNSUPPORTED

---

Claim: strongest/smartest consumer open-source fine-tune

The card describes the model in sweeping superlatives.

But the evidence immediately presented is primarily the seven-task legacy suite.

That does not establish superiority across coding, agentic use, math, tools, long-context, or modern reasoning.

A claim this broad requires a benchmark battery equally broad.

Status: UNSUPPORTED

---

Claim: the model exceeds the base Qwen3.6-27B in 6/7 of those benchmarks

This narrower claim is consistent with the table shown on the card:

So this article is not claiming that this specific seven-task comparison is fabricated.

The criticism is the leap from:

better on this seven-task suite

to:

frontier-zone intelligence / strongest overall.

Those are not equivalent statements.

---

Claim: relevant capabilities were preserved while intelligence increased

The card presents the fine-tune as improving general intelligence/problem solving without compromising the original model.

But no complete modern base-model regression suite is shown.

Without rerunning at least a representative sample of the official base capabilities, we do not know whether the fine-tune preserved:

• SWE-bench,
• Terminal-Bench,
• GPQA,
• LiveCodeBench,
• math,
• tool use,
• multimodal ability,
• long-context behavior.

Status: UNSUPPORTED

---

Claim: NEO IMATRIX adds 2–4% accuracy and improves long-context performance

The card states a general 2–4% accuracy improvement over normal GGUFs and claims better long-context performance.

The official llama.cpp documentation says an importance matrix can be used during quantization to enhance quality. It does not provide a universal 2–4% guarantee, and the effect depends on model, quantization, tensors, and calibration data.

A universal numeric improvement should be backed by paired controlled tests:

• same base model,
• same quant type,
• same inference settings,
• with vs. without the imatrix,
• same benchmark items,
• confidence intervals or paired significance.

Those comparisons are not provided next to the broad 2–4% claim.

Status: UNSUPPORTED AS A GENERAL RULE

---

Claim: thinking mode will exceed instruct benchmark scores in most cases

The card says the published benchmark table uses instruct mode, while thinking mode should exceed instruct scores in most cases.

But the corresponding thinking-mode table is not published there.

If that claim is important enough to qualify the published results, the direct evidence should be shown.

Status: NOT DEMONSTRATED

---

Claim: BF16 will be roughly 2–5 points higher than MXFP8 in most metrics

Again, this is a concrete quantitative prediction without the corresponding complete BF16 table being shown for the claim.

If BF16 is intended to represent the full-precision comparison, estimated values should not replace measured values.

Status: NOT DEMONSTRATED IN THE PUBLISHED TABLE

📸 Screenshot slot C — Fable Fusion marketing + benchmark table

---

5.2 Qwen3.5-9B The Defiant

Model: https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF

This card contains one of the clearest internal contradictions.

The card calls the model:

• "Fully uncensored"
• and says it does what the user wants with no questions asked.

But the card's own performance table reports:

Refusals: 6/100

If "fully uncensored" is interpreted literally as zero refusals, the author's own result contradicts the label.

A more defensible description would be "far fewer refusals than the base model."

Status: CONTRADICTED BY OWN DATA

📸 Screenshot slot D — "Fully uncensored" vs 6/100 refusals

---

"Extreme intelligence" and "Superior instruction following"

The evidence directly surrounding these phrases is again the same seven-task benchmark suite.

Those tasks do not directly measure instruction following in the modern sense of benchmarks such as IFEval/IFBench, and they do not establish broad "extreme intelligence."

Status: UNSUPPORTED AS BROAD CLAIMS

---

"640 ARC-C" used as intelligence

The card later describes the 9B as beating many larger models in "intelligence" and directly associates that claim with its ARC-C result.

ARC-C is a benchmark score, not a calibrated IQ scale.

Status: MISLEADING EXTRAPOLATION

---

5.3 Qwen3.6-40B Grand Intelligence

Model: https://huggingface.co/DavidAU/Qwen3.6-40B-Grand-Intelligence-Fable-Fusion-Uncensored-Heretic

The 40B project contains promotional language saying its benchmarks exceed Qwen3.6-27B by a large margin and also connects some releases to OpenAI/Claude/Gemini-level intelligence.

For one published 40B row:

0.687, 0.857, 0.908, 0.825, 0.500, 0.818, 0.771

The related Qwen3.6-27B base row is:

0.647, 0.803, 0.910, 0.773, 0.450, 0.806, 0.742

The 40B improves six values, but BoolQ is lower: 0.908 vs 0.910.

So language equivalent to "far and above" should be read as promotional shorthand for the overall pattern, not literal domination of every metric.

Status: OVERSTATED / MARKETING

---

"SOTA IQ/power at 40B"

No standardized "IQ" metric is defined, and no comprehensive 40B state-of-the-art comparison is supplied.

Status: UNSUPPORTED / NON-STANDARD METRIC LANGUAGE

---

Human testing as evidence

The card invokes human testing to explain strengths that do not appear in the benchmark table.

Human evaluation can be valuable — but only when the protocol is described.

For a reproducible comparison, readers need things such as:

• exact prompts,
• blinded or non-blinded setup,
• number of evaluators,
• rubric,
• pairwise procedure,
• randomization,
• raw preferences or scores,
• failure cases.

Without that, "human testing" is anecdotal evidence rather than independently reproducible evaluation.

Status: NOT REPRODUCIBLE FROM PUBLISHED INFO

📸 Screenshot slot E — 40B claims and human-testing notes

---

6. Nightmedia audit

6.1 Qwen3.5-9B Theseus

Model: https://huggingface.co/nightmedia/Qwen3.5-9B-Theseus

The Theseus card contains some of the strongest scientific-sounding language in this audit.

Claim: "absolute mathematical saturation ceiling for 9B parameter architectures"

No mathematical ceiling is derived.

There is no theorem, scaling-law bound, architecture-specific proof, or optimization argument establishing a maximum score for all 9B models.

A strong empirical result is not the same thing as a mathematical saturation bound.

Status: UNSUPPORTED

📸 Screenshot slot F — mathematical saturation claim

---

Claim: an "Implicit Inner State" caused by merge geometry

The card interprets benchmark fluctuations and merge behavior as evidence of a profound internal shift and describes the model as "thinking in concepts, not in words."

That is a mechanistic claim about internal computation.

To establish it, one would need evidence such as:

• activation analysis,
• representation probes,
• causal interventions / ablations,
• controlled hidden-state comparisons,
• mechanistic interpretability experiments,
• reproducible behavioral signatures tightly linked to those internal states.

A few benchmark movements do not demonstrate a new internal cognitive state.

Status: UNSUPPORTED / SPECULATIVE

---

The "Gemini review"

The card includes a long section labeled as a Gemini review containing highly enthusiastic claims about the model.

An LLM-generated commentary can be interesting qualitative material, but it is not an independent benchmark or peer review.

If one model is asked to interpret another model's behavior, that output should not be treated as mechanistic evidence.

Status: QUALITATIVE / NOT INDEPENDENT VALIDATION

---

6.2 Qwen3.8-27B Brainwaves

Model: https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves

Related metrics post: https://huggingface.co/posts/nightmedia/103998819459939

Claim: the model can be "safely RoPEd to 2M"

The evidence shown next to that claim consists of:

• ARC,
• ARC-Easy,
• BoolQ,
• HellaSwag,
• OpenBookQA,
• PIQA,
• WinoGrande,
• MLX perplexity.

The problem is that these are short-context evaluations.

The default mlx_lm.perplexity configuration uses:

• sequence_length = 512
• num_samples = 256
• batch_size = 8
• dataset allenai/tulu-3-sft-mixture

So a model labeled "2M" being evaluated on 512-token PPL blocks does not establish reliable behavior at 2,000,000 tokens.

Likewise, running ARC or PIQA with a 2M RoPE configuration only shows that the changed RoPE settings did not catastrophically destroy short-context performance.

A real 2M validation would require actual long-context tests at increasing lengths, for example:

• 128K,
• 256K,
• 512K,
• 1M,
• 2M,

with metrics such as:

• passkey retrieval,
• multi-needle retrieval,
• position-dependent recall,
• long-document QA,
• long-context coding/repository tasks,
• degradation vs. depth.

Status: UNSUPPORTED BY THE SHOWN TESTS

---

7. The MLX perplexity table has important hidden assumptions

Nightmedia says the perplexity numbers come from the standard command:

mlx_lm.perplexity --model MODEL

That matters because the default implementation is public.

Source: https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/perplexity.py

7.1 It evaluates the Tulu 3 SFT training mixture by default

The script's default data path is:

allenai/tulu-3-sft-mixture

and its dataset configuration sets:

train_split = "train"

This is an instruction-tuning mixture, not a conventional held-out language-model perplexity corpus.

That does not make the value useless.

But reporting a bare column called Perplexity without emphasizing the dataset and split makes it easy for readers to assume it is directly comparable to standard held-out PPL values from other work.

Status: METHODOLOGICAL CAVEAT

---

7.2 Unrelated examples are concatenated before 512-token chunking

The loader:

1. samples dataset entries,
2. processes each into tokens,
3. appends all tokens into one long list,
4. then reshapes the stream into fixed-length sequences.

That means a 512-token block can cross a boundary like:

end of conversation A -> beginning of unrelated conversation B

The loss across that artificial transition contributes to PPL.

This is not the same as evaluating each independent document/conversation separately.

Status: METHODOLOGICAL CAVEAT

---

7.3 The displayed PPL error bar is token-level

The script:

• collects individual token losses,
• flattens them,
• computes their variance,
• and divides by sqrt(num_tokens) before applying the delta approximation to perplexity.

Tokens inside a sequence are correlated.

Therefore the displayed ± should not be casually interpreted like an error bar from tens of thousands of independent experimental samples.

A more robust uncertainty analysis would resample at a higher-level independent unit, such as documents/conversations, or use a block/bootstrap procedure appropriate for correlated data.

Status: STATISTICAL CAVEAT

---

7.4 "Tokens/sec" is not normal chat-generation speed

The script calculates:

tokens_evaluated / eval_time

during perplexity evaluation.

That is forward-pass evaluation throughput over known tokens.

It is not the same as ordinary autoregressive decode speed where the model generates one new token after another while using a KV cache.

So a PPL table showing 175 tokens/sec should not be read as:

"this model chats at 175 generated tokens/sec."

Status: POTENTIALLY MISLEADING IF INTERPRETED AS GENERATION SPEED

---

7.5 "Peak Memory" is workload-specific

mx.get_peak_memory() in that script reports the peak during this particular PPL workload.

It is not a universal memory requirement.

Memory changes with:

• prompt length,
• batch size,
• sequence length,
• KV cache,
• generation length,
• long-context use,
• model/backend configuration.

This matters even more when the same card discusses 1M/2M contexts, because a short 512-token PPL run says little about memory consumption at real 1M/2M inference.

---

8. "Standard MLX metrics" is not a fixed seven-task standard

In a public reply, Nightmedia described the old suite as standard MLX metrics.

But mlx_lm.evaluate does not define these seven tasks as a mandatory standard suite.

Source: https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/evaluate.py

Its command-line interface explicitly requires:

--tasks

and sends the selected task names to:

lm_eval.simple_evaluate(..., tasks=args.tasks, ...)

So the evaluator chooses the tasks.

MLX supplies the model adapter/inference implementation; the benchmark tasks come from lm-evaluation-harness.

Calling the seven tests a personal fixed regression suite is fair.

Calling them a fixed "standard MLX metrics" set is misleading if readers understand that to mean MLX itself standardizes those seven.

Status: FACTUALLY MISLEADING DESCRIPTION

📸 Screenshot slot H — "standard MLX metrics" reply

---

9. A fixed regression suite is valid — but it is not a complete modern evaluation

Keeping the same benchmark suite over time is useful for longitudinal comparison.

In fact, this is one of the strongest arguments for retaining ARC, BoolQ, HellaSwag, PIQA, OpenBookQA and WinoGrande: they provide a historical regression baseline.

But there are two separate questions:

1. Should an author keep a stable historical benchmark suite?

Yes. This is methodologically useful.

2. Is that historical suite sufficient forever to support new and increasingly broad capability claims?

No.

lm-evaluation-harness itself versions task definitions because prompts, scoring, extraction logic and bugs can change. More importantly, preserving an old regression suite does not prevent adding modern evaluations alongside it.

The appropriate methodology is therefore not:

remove the old tests.

It is:

preserve the historical tests and add capability-matched modern tests when making claims about modern capabilities.

For example, claims about coding, software engineering, tool use, agentic behavior, or million-token context cannot be established by ARC, BoolQ, HellaSwag, PIQA, OpenBookQA and WinoGrande alone.

Status: VALID AS A FIXED REGRESSION SUITE; INSUFFICIENT AS A COMPLETE MODERN EVALUATION

---

10. These seven tasks do not measure only "the ability to think"

Nightmedia wrote that the suite measures the model's ability to think and that coding is not part of it.

The second half is correct: these are not coding-agent benchmarks.

The first half is too broad.

The original benchmark descriptions themselves show different targets:

• BoolQ: yes/no reading comprehension,
• PIQA: physical commonsense,
• HellaSwag: commonsense continuation,
• WinoGrande: commonsense/coreference,
• ARC/OpenBookQA: science QA / knowledge + reasoning.

Reasoning matters in all of them to different degrees, but they are not a pure scalar measurement of "thinking ability."

A model can improve on these tests while getting worse at:

• long-horizon planning,
• coding,
• tool use,
• terminal control,
• instruction following,
• long-context retrieval,
• calibrated scientific reasoning.

Status: MISLEADING EXTRAPOLATION

---

11. Metric-labeling inconsistency: not every column is acc_norm

In the public discussion around the Brainwaves table, it was pointed out that the seven columns are not all the same metric.

For example, WinoGrande's current lm-eval task configuration uses:

metric: acc

rather than acc_norm.

Source: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/winogrande/default.yaml

Nightmedia later acknowledged that two of the tasks were not acc_norm.

This is not a catastrophic benchmark failure, but it matters because a table caption that makes all seven columns look like one uniform normalized metric is technically inaccurate.

Status: ACKNOWLEDGED METRIC-DESCRIPTION ERROR

---

12. Toolchain version inconsistency

Nightmedia publicly said he uses the latest available MLX/tests.

But published evaluation filenames in the discussion included lm-eval versions such as 0.4.9 / 0.4.10.

The current MLX evaluator constructs its output filename using:

version("lm_eval")

so this part of the filename corresponds to the installed lm_eval package version.

lm-eval 0.4.12 was released on May 11, 2026.

Therefore, for runs made after that point, a filename generated with 0.4.9 or 0.4.10 would not correspond to the latest lm-eval release.

Possible explanations exist — for example, an older archived run may have been reused — so this is best treated as something requiring clarification, not as proof of fabrication.

Status: INTERNALLY INCONSISTENT / REQUIRES CLARIFICATION

Sources

• MLX evaluator: https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/evaluate.py
• lm-eval 0.4.12: https://pypi.org/project/lm-eval/0.4.12/

📸 Screenshot slot I — version filename + "latest version" reply

---

13. Qwen3.6-35B-A3B Brainwaves-Nex

Model: https://huggingface.co/nightmedia/Qwen3.6-35B-A3B-Brainwaves-Nex

This model includes or derives from agent-oriented components, including Qwen-AgentWorld-related material, and carries coding/research/agent-adjacent positioning.

Yet the numerical evaluation shown is again predominantly:

• ARC,
• ARC-E,
• BoolQ,
• HellaSwag,
• OpenBookQA,
• PIQA,
• WinoGrande,
• perplexity.

There is no corresponding published:

• Terminal-Bench,
• SWE-bench,
• BFCL,
• tool-use benchmark,
• agent-trajectory benchmark.

This does not prove the model is bad at agents.

It means the card does not establish whether agentic capability improved, stayed constant, or regressed.

Status: NOT EVALUATED FOR THE CAPABILITY IMPLIED BY ITS COMPONENTS/TAGS

---

"Gemini review" as evidence of self-inspection

The card includes a Gemini-generated review praising the model for being able to "map its own tensor geometry."

A language model can generate a plausible-sounding description of neural-network internals without having direct introspective access to its weights or tensor geometry during ordinary inference.

Unless the system actually exposes those tensors or diagnostic tools to the model, the output is not proof of mechanistic self-inspection.

Status: UNSUPPORTED IF INTERPRETED LITERALLY

📸 Screenshot slot J — Gemini review

---

14. Quantization claims need paired tests, not impressions

Both authors discuss certain quantizations as having distinct cognitive or social characteristics.

Quantization can absolutely change outputs.

The problem is making a stable causal claim without controlled tests.

If the claim is:

quantization method X improves cognition / social flow / reasoning

then a strong experiment would compare:

• same exact source checkpoint,
• same tokenizer/template,
• same prompts,
• same decoding,
• same context,
• multiple random seeds where applicable,
• blinded pairwise human evaluation,
• paired benchmark-item analysis,
• a sufficient number of examples.

Otherwise observations like "this quant feels more social" remain qualitative impressions.

Qualitative testing can be valuable.

It should be labeled as qualitative testing.

---

15. Small score deltas need paired analysis

The same benchmark items are evaluated across quantizations, so comparisons are naturally paired.

If quant A scores 0.604 and quant B scores 0.591, looking only at separate aggregate standard errors is not the most informative test.

For binary correctness tasks, the key data are the disagreements:

• both correct,
• both wrong,
• A correct / B wrong,
• A wrong / B correct.

A McNemar-style paired test can determine whether one model consistently flips more items in the right direction.

Publishing per-item outputs would make these comparisons much stronger.

---

16. The moderation / criticism problem

This section is intentionally separate from the benchmark audit.

A technical claim can be weak even if the author is polite.

A technical claim can be strong even if the author blocks someone.

Moderation behavior does not prove benchmark fraud.

But it matters for community scrutiny when criticism disappears or critics lose the ability to respond.

My documented timeline

I previously posted a detailed methodological criticism on a DavidAU model discussion.

After that:

• the discussion was no longer publicly available to me,
• my account became unable to interact with DavidAU,
• Hugging Face returned a 403 stating that I was not allowed to interact with that user.

I am documenting the observable sequence.

I am not claiming that the 403 error alone proves DavidAU's internal motive.

📸 Screenshot slot L — Hugging Face 403

📸 Screenshot slot M — unanswered SWE-bench request

Discussion: https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/discussions/1

---

Nightmedia replies

After I raised methodology concerns, Nightmedia defended the fixed suite as his preferred methodology and later asked me to stop continuing the discussion.

Again, the fact that someone wants a conversation to end does not prove their model is fraudulent.

But ending the interpersonal argument also does not answer the technical questions in this article.

So I am no longer trying to force either author to debate me directly.

The evidence can be read independently.

📸 Screenshot slot N — "old metrics / standard MLX metrics" response

📸 Screenshot slot O — seniority / respect response

📸 Screenshot slot P — "badgering" / block warning

---

17. What I am NOT claiming

To make this absolutely clear:

I am not claiming that:

• every benchmark output is fabricated,
• every DavidAU model is bad,
• every Nightmedia model is bad,
• legacy benchmarks have zero value,
• fine-tuning cannot improve old benchmarks,
• quantization cannot affect behavior,
• human evaluation is useless,
• blocking a user proves guilt.

I am claiming that:

• several capability claims are broader than the evidence shown,
• some statements are directly contradicted by the authors' own tables,
• several methodological details are easy for readers to misinterpret,
• long-context claims are not validated by short-context evaluations,
• the default MLX PPL pipeline has important assumptions that should be disclosed,
• 0.700 ARC-C = frontier intelligence is not an established scientific threshold,
• modern model claims require modern, capability-matched evaluation.

---

18. What a convincing evaluation would look like

This criticism is constructive because the problems are fixable.

For each major release, publish:

Reproducibility

• exact commit/hash of the model,
• exact tokenizer files/hash,
• exact chat template,
• exact mlx-lm version,
• exact lm-eval version,
• exact task versions,
• exact command lines,
• seed,
• sampling settings,
• raw JSON outputs,
• per-item predictions where licensing permits.

General reasoning / knowledge

• GPQA Diamond,
• MMLU-Pro,
• a current hard reasoning suite,
• contemporary math benchmarks.

Coding

• LiveCodeBench,
• repository-level coding benchmark,
• code-generation tests appropriate to the model.

Agentic

• Terminal-Bench,
• SWE-bench / SWE-bench Pro where feasible,
• BFCL or another function/tool-calling benchmark,
• standardized tool-use tasks.

Long context

If claiming 1M or 2M:

• actual runs near those lengths,
• retrieval accuracy by depth,
• multiple needles,
• long-document QA,
• latency,
• peak memory,
• failure rate,
• degradation curve from short context to max context.

Quantization

For every claimed quant advantage:

• paired source checkpoint,
• with/without technique,
• same task items,
• PPL/KL,
• paired downstream tests,
• speed measured separately as prefill and decode,
• memory measured at stated context lengths.

Human evaluation

• publish prompts,
• blind model identities,
• define rubric,
• publish evaluator count,
• publish pairwise results,
• include negative examples.

That would turn marketing claims into actual reproducible evidence.

---

19. The biggest issue: benchmark outputs are being converted into a story they do not prove

This is the central pattern.

A benchmark can be real.

The interpretation can still be wrong.

Examples:

ARC-C improves
→ "general intelligence increased"

Not automatically.

ARC-C crosses 0.700
→ "OpenAI/Claude/Gemini zone"

Not established.

A model configured for 2M retains short-task scores
→ "safe at 2M"

Not demonstrated.

An LLM writes an enthusiastic review of another LLM
→ internal cognition / tensor self-awareness

Not demonstrated.

An importance matrix can improve quantization quality
→ universal +2–4% accuracy and better long context

Not established without paired tests.

Seven legacy tasks improve
→ strongest/smartest overall model

Not established.

This is why modern evaluation has become broader, not narrower.

---

20. Conclusion

Open-source model development depends heavily on trust.

Not blind trust in authors.

Trust that claims can be checked.

DavidAU and Nightmedia have clearly invested significant time into fine-tuning, merging, quantization, and testing. The volume of experimentation is not the issue.

The issue is the gap between:

what was measured

and

what is being claimed.

If the claim is:

"this quant scores 0.711 on ARC-C under this exact harness"

then publish the harness details and the result can be reproduced.

If the claim becomes:

"this is frontier-level intelligence"

then the required evidence becomes much larger.

If the claim is:

"safe at 2M"

then actually test close to 2M.

If the claim is:

"fully uncensored"

then a published refusal count above zero needs to be reconciled with that wording.

If the claim is:

"strongest/smartest"

then evaluate the capabilities that strong modern models are actually expected to have.

That is not hostility toward open source.

That is what open science is supposed to look like:

claims proportional to evidence, reproducibility, raw results, and criticism that can be checked by anyone.

---

Sources / Evidence Index

Model cards can change. For anything important, preserve screenshots and, when possible, link a commit-specific revision.

DavidAU

1. Qwen3.6-27B Fable Fusion 711
   https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF

2. Qwen3.5-9B The Defiant
   https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF

3. Qwen3.6-40B Grand Intelligence
   https://huggingface.co/DavidAU/Qwen3.6-40B-Grand-Intelligence-Fable-Fusion-Uncensored-Heretic

4. Fable Fusion Discussion #1
   https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/discussions/1

Nightmedia

5. Qwen3.5-9B Theseus
   https://huggingface.co/nightmedia/Qwen3.5-9B-Theseus

6. Qwen3.8-27B Brainwaves
   https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves

7. Qwen3.6-35B-A3B Brainwaves-Nex
   https://huggingface.co/nightmedia/Qwen3.6-35B-A3B-Brainwaves-Nex

8. Qwen3.8-27B metrics post / technical discussion
   https://huggingface.co/posts/nightmedia/103998819459939

Evaluation implementation

9. MLX-LM evaluate.py
   https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/evaluate.py

10. MLX-LM perplexity.py
    https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/perplexity.py

11. lm-evaluation-harness task guide / versioning
    https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md

12. lm-evaluation-harness releases
    https://github.com/EleutherAI/lm-evaluation-harness/releases

13. lm-eval 0.4.12 release
    https://pypi.org/project/lm-eval/0.4.12/

14. WinoGrande lm-eval task configuration
    https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/winogrande/default.yaml

Benchmark literature

15. Łukasz Borchmann — In Case You Missed It: ARC "Challenge" Is Not That Challenging
    https://arxiv.org/abs/2412.17758

16. BoolQ — Clark et al.
    https://aclanthology.org/N19-1300/

17. HellaSwag — Zellers et al.
    https://aclanthology.org/P19-1472/

18. PIQA — Bisk et al.
    https://ojs.aaai.org/index.php/AAAI/article/view/6239

Quantization

19. llama.cpp importance matrix documentation
    https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md

20. llama.cpp quantization documentation
    https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md

Official base-model comparison

21. Qwen3.6-27B official model card
    https://huggingface.co/Qwen/Qwen3.6-27B

---

Author note

This article is a technical audit, not an invitation to harass either author.

Please verify the linked evidence yourself, keep criticism focused on public technical claims, and do not contact anyone on my behalf.