Ddavidich commited on
Commit
888fb62
·
verified ·
1 Parent(s): c9b8123

The position map is not the published method it resembles, and we measured that

Browse files

The band of frequency pairs we compress was chosen because those pairs complete between one and thirty-two turns over the trained window. The method that inspired the search, DPE (arXiv:2504.18857), prescribes something else: rank dimensions by their measured 2-norm contribution to attention and remap the top half. Until now we had not measured that ranking, so we could not say whether we were applying the method or merely borrowing its name.

Measured: 16 attention layers, real activations, with a check that the per-pair split reproduces the whole rotary-part norm on every layer. The ranking is nearly flat — 1.4% to 7.0% across 32 pairs — it bears no relation to turn count, and the four pairs at the centre of our band rank 32nd, 31st, 27th and 20th in it.

Then tested directly, one corpus, only the pair set differing: control 10/14, our analytic band 13/14, the paper's top-half-by-norm 11/14. The published criterion loses two records to a band found by trial.

So the card now says it plainly: this is an empirical find named after the paper that prompted the search, not an application of it. That fits what we already knew about the mechanism — compressing a stretch containing no records at all still changes which records are found, so the effect is global rather than targeted.

Nothing about the shipped configuration changes. The gain is reproducible, it was validated on a corpus that played no part in choosing it, and it costs nothing at a million. It ships on that evidence, not on the paper's.

README.md CHANGED
@@ -284,15 +284,27 @@ now have their positions halved above 196,608, while the other twenty-five keep
284
  true positions until 262,144. On the same host and the same corpus, the old map
285
  scores 10/14 and the new one 13/14, gaining three records and losing none.
286
 
287
- Two things about that are worth your scepticism, and we share it. The gain was
288
- found by trying six prefixes and reporting the best, which is a selection effect —
289
- so it was re-tested on a **different corpus that played no part in choosing it**,
290
- and there it gained three again. And the mechanism is not understood: compressing
291
- a stretch of context containing **no records at all** still changes which records
292
- are found, so this is not "we fixed the angles for the needles". Four of the
293
- fourteen records sit near the decision boundary and any perturbation moves them.
294
- What we can say is that the perturbation has moved them the same way twice, on two
295
- corpora, and that the shipped configuration is measured end to end.
 
 
 
 
 
 
 
 
 
 
 
 
296
 
297
  **A million is unchanged at 2/14.** The map costs nothing there and gains nothing;
298
  past the trained window the limit appears to be architectural.
@@ -853,15 +865,26 @@ RTX 4090: 32 768 токенов за 38 с (861 т/с).
853
  262 144. На одном хосте и одном корпусе прежняя карта даёт 10/14, новая 13/14:
854
  три записи приобретены, ни одна не потеряна.
855
 
856
- Две вещи здесь заслуживают вашего скепсиса, и мы его разделяем. Прибавка найдена
857
- перебором шести префиксов с показом лучшего, а это отбор по результату — поэтому
858
- её переснимали на **другом корпусе, который в выборе не участвовал**, и там она
859
- дала те же три записи. И механизм не понят: сжатие участка, где **нет ни одной
860
- записи**, всё равно меняет, какие записи находятся, так что это не «мы починили
861
- углы закладкам». Четыре записи из четырнадцати стоят на грани решения, и любое
862
- возмущение их двигает. Сказать можно лишь то, что это возмущение сдвинуло их в
863
- одну сторону дважды, на двух корпусах, и что отгружаемая конфигурация проверена
864
- сквозным прогоном.
 
 
 
 
 
 
 
 
 
 
 
865
 
866
  **Миллион не изменился, 2/14.** Карта там ничего не стоит и ничего не даёт; за
867
  обученным окном предел, судя по всему, архитектурный.
 
284
  true positions until 262,144. On the same host and the same corpus, the old map
285
  scores 10/14 and the new one 13/14, gaining three records and losing none.
286
 
287
+ Three things about that are worth your scepticism, and we share all three.
288
+
289
+ The gain was found by trying six prefixes and reporting the best, which is a
290
+ selection effect — so it was re-tested on a **different corpus that played no part
291
+ in choosing it**, and there it gained three again.
292
+
293
+ The mechanism is not understood. Compressing a stretch of context containing **no
294
+ records at all** still changes which records are found, so this is not "we fixed
295
+ the angles for the needles". Four of the fourteen records sit near a decision
296
+ boundary and any perturbation moves them; this one has moved them the same way
297
+ twice, on two corpora.
298
+
299
+ And **this is not the published method it resembles.** The search was inspired by
300
+ DPE (arXiv:2504.18857), which prescribes ranking dimensions by their measured
301
+ 2-norm contribution to attention and remapping the top half. We measured that
302
+ ranking — 16 attention layers, real activations, with a check that the per-pair
303
+ split reproduces the whole rotary norm on every layer. It is nearly flat, 1.4% to
304
+ 7.0% across 32 pairs, it has no relation to turn count, and the pairs we compress
305
+ sit 32nd, 31st, 27th and 20th in it. Selecting by the paper's criterion instead
306
+ scores **11/14 against our 13/14.** So the band we ship is an empirical find named
307
+ after the paper that prompted the search, not an application of it.
308
 
309
  **A million is unchanged at 2/14.** The map costs nothing there and gains nothing;
310
  past the trained window the limit appears to be architectural.
 
865
  262 144. На одном хосте и одном корпусе прежняя карта даёт 10/14, новая 13/14:
866
  три записи приобретены, ни одна не потеряна.
867
 
868
+ Три вещи здесь заслуживают вашего скепсиса, и мы разделяем все три.
869
+
870
+ Прибавка найдена перебором шести префиксов с показом лучшего, а это отбор по
871
+ результату поэтому её переснимали на **другом корпусе, который в выборе не
872
+ участвовал**, и там она дала те же три записи.
873
+
874
+ Механизм не понят. Сжатие участка, где **нет ни одной записи**, всё равно меняет,
875
+ какие записи находятся, так что это не «мы починили углы закладкам». Четыре записи
876
+ из четырнадцати стоят на грани решения, и любое возмущение их двигает; это
877
+ сдвинуло их в одну сторону дважды, на двух корпусах.
878
+
879
+ И **это не тот опубликованный метод, на который похоже.** ��оиск подсказан DPE
880
+ (arXiv:2504.18857), где предписано ранжировать измерения по замеренной 2-норме
881
+ вклада во внимание и переназначать позиции верхней половине. Мы это ранжирование
882
+ замерили — 16 слоёв внимания, реальные активации, с проверкой, что понарная
883
+ нарезка воспроизводит норму всей вращаемой части на каждом слое. Оно почти
884
+ плоское, от 1.4% до 7.0% на 32 пары, с числом оборотов не связано, а сжимаемые
885
+ нами пары стоят в нём 32-м, 31-м, 27-м и 20-м. Отбор по критерию статьи даёт
886
+ **11/14 против наших 13/14.** Значит отгружаемый диапазон — эмпирическая находка,
887
+ названная по статье, которая натолкнула на поиск, а не её применение.
888
 
889
  **Миллион не изменился, 2/14.** Карта там ничего не стоит и ничего не даёт; за
890
  обученным окном предел, судя по всему, архитектурный.
receipts/pair_energy/band_measured.json ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "lomonosov_zenit_retention_probe_v1",
3
+ "label": "band_measured",
4
+ "model": "Ddavidich/LOMONOSOV-ZENIT-27B-1M-INDEV",
5
+ "corpus_sha256": "66a5b0f2eed04fce33318cd4f295c4ea7bc25f3116b00798284c20576d61bb6e",
6
+ "corpus_books": 3,
7
+ "prompt_tokens": 250334,
8
+ "needles_scored": 14,
9
+ "correct": 11,
10
+ "accuracy": 0.7857142857142857,
11
+ "kv_cache_dtype": "turboquant_3bit_nc",
12
+ "lora": null,
13
+ "native_rope_requested": true,
14
+ "rope_effective": {
15
+ "top": {
16
+ "rope_type": "default",
17
+ "factor": null,
18
+ "original_max_position_embeddings": null,
19
+ "partial_rotary_factor": 0.25,
20
+ "attention_factor": null
21
+ },
22
+ "text_config": {
23
+ "rope_type": "default",
24
+ "factor": null,
25
+ "original_max_position_embeddings": null,
26
+ "partial_rotary_factor": 0.25,
27
+ "attention_factor": null
28
+ }
29
+ },
30
+ "position_plan": {
31
+ "ZENIT_POSITION_STAGES": "[{\"from\":196608,\"group\":2,\"first_pair\":4,\"last_pair\":7},{\"from\":196608,\"group\":2,\"first_pair\":10,\"last_pair\":10},{\"from\":196608,\"group\":2,\"first_pair\":12,\"last_pair\":13},{\"from\":196608,\"group\":2,\"first_pair\":19,\"last_pair\":26},{\"from\":196608,\"group\":2,\"first_pair\":28,\"last_pair\":28},{\"from\":262144,\"group\":4,\"first_pair\":0,\"last_pair\":31}]",
32
+ "ZENIT_NOOSPHERE_MODE": "raw"
33
+ },
34
+ "load_seconds": 51.10397109598853,
35
+ "first_question_seconds": 131.1878986560041,
36
+ "later_question_seconds_mean": 2.750335492457872,
37
+ "results": [
38
+ {
39
+ "index": 1,
40
+ "depth_fraction": 0.06666666666666667,
41
+ "token_position": 16666,
42
+ "expected": "BQ-13682",
43
+ "answer_text": "BQ-13682\n\n<think>\n\n</think>\n\nBQ-13682",
44
+ "correct": true,
45
+ "seconds": 131.1878986560041,
46
+ "was_first_question": true
47
+ },
48
+ {
49
+ "index": 2,
50
+ "depth_fraction": 0.13333333333333333,
51
+ "token_position": 33357,
52
+ "expected": "HB-24619",
53
+ "answer_text": "<think>\n\n</think>\n\nLOMONOSOV-ZENIT-216-10000000",
54
+ "correct": false,
55
+ "seconds": 3.9299850009847432,
56
+ "was_first_question": false
57
+ },
58
+ {
59
+ "index": 3,
60
+ "depth_fraction": 0.2,
61
+ "token_position": 50047,
62
+ "expected": "RJ-46395",
63
+ "answer_text": "RJ-46395\n\n<think>\n\n</think>\n\nRJ-46395",
64
+ "correct": true,
65
+ "seconds": 3.509081119002076,
66
+ "was_first_question": false
67
+ },
68
+ {
69
+ "index": 4,
70
+ "depth_fraction": 0.26666666666666666,
71
+ "token_position": 66736,
72
+ "expected": "ZK-83394",
73
+ "answer_text": "<think>\n\n</think>\n\nLOMONOSOV-ZENIT-LOMONOSOV-1761-11",
74
+ "correct": false,
75
+ "seconds": 3.9410403729998507,
76
+ "was_first_question": false
77
+ },
78
+ {
79
+ "index": 5,
80
+ "depth_fraction": 0.3333333333333333,
81
+ "token_position": 83428,
82
+ "expected": "TR-15255",
83
+ "answer_text": "<think>\n\n</think>\n\nTR-15255",
84
+ "correct": true,
85
+ "seconds": 2.3776552269991953,
86
+ "was_first_question": false
87
+ },
88
+ {
89
+ "index": 6,
90
+ "depth_fraction": 0.4,
91
+ "token_position": 100120,
92
+ "expected": "MK-13191",
93
+ "answer_text": "<think>\n\n</think>\n\nMK-13191",
94
+ "correct": true,
95
+ "seconds": 2.3586293300031684,
96
+ "was_first_question": false
97
+ },
98
+ {
99
+ "index": 7,
100
+ "depth_fraction": 0.4666666666666667,
101
+ "token_position": 116810,
102
+ "expected": "PW-31226",
103
+ "answer_text": "<think>\n\n</think>\n\nPW-31226",
104
+ "correct": true,
105
+ "seconds": 2.361637265974423,
106
+ "was_first_question": false
107
+ },
108
+ {
109
+ "index": 8,
110
+ "depth_fraction": 0.5333333333333333,
111
+ "token_position": 133500,
112
+ "expected": "CY-84017",
113
+ "answer_text": "<think>\n\n</think>\n\nCY-84017",
114
+ "correct": true,
115
+ "seconds": 2.375112056004582,
116
+ "was_first_question": false
117
+ },
118
+ {
119
+ "index": 9,
120
+ "depth_fraction": 0.6,
121
+ "token_position": 150191,
122
+ "expected": "GD-41058",
123
+ "answer_text": "<think>\n\n</think>\n\nNT-82593",
124
+ "correct": false,
125
+ "seconds": 2.3593893960060086,
126
+ "was_first_question": false
127
+ },
128
+ {
129
+ "index": 10,
130
+ "depth_fraction": 0.6666666666666666,
131
+ "token_position": 166882,
132
+ "expected": "QX-74073",
133
+ "answer_text": "<think>\n\n</think>\n\nQX-74073",
134
+ "correct": true,
135
+ "seconds": 2.504759554984048,
136
+ "was_first_question": false
137
+ },
138
+ {
139
+ "index": 11,
140
+ "depth_fraction": 0.7333333333333333,
141
+ "token_position": 183573,
142
+ "expected": "NT-82593",
143
+ "answer_text": "<think>\n\n</think>\n\nNT-82593",
144
+ "correct": true,
145
+ "seconds": 2.368501618009759,
146
+ "was_first_question": false
147
+ },
148
+ {
149
+ "index": 12,
150
+ "depth_fraction": 0.8,
151
+ "token_position": 200264,
152
+ "expected": "VM-48579",
153
+ "answer_text": "VM-48579",
154
+ "correct": true,
155
+ "seconds": 1.7961700089799706,
156
+ "was_first_question": false
157
+ },
158
+ {
159
+ "index": 13,
160
+ "depth_fraction": 0.8666666666666667,
161
+ "token_position": 216954,
162
+ "expected": "LF-80058",
163
+ "answer_text": "LF-80058\n\n<think>\n\n</think>\n\nLF-80058",
164
+ "correct": true,
165
+ "seconds": 3.49979748899932,
166
+ "was_first_question": false
167
+ },
168
+ {
169
+ "index": 14,
170
+ "depth_fraction": 0.9333333333333333,
171
+ "token_position": 233644,
172
+ "expected": "XS-27918",
173
+ "answer_text": "<think>\n\n</think>\n\nXS-27918",
174
+ "correct": true,
175
+ "seconds": 2.3726029630051926,
176
+ "was_first_question": false
177
+ }
178
+ ]
179
+ }
receipts/pair_energy/pair_energy.json ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "lomonosov_zenit_pair_energy_v1",
3
+ "status": "PASS",
4
+ "captured_layers": [
5
+ 3,
6
+ 7,
7
+ 11,
8
+ 15,
9
+ 19,
10
+ 23,
11
+ 27,
12
+ 31,
13
+ 35,
14
+ 39,
15
+ 43,
16
+ 47,
17
+ 51,
18
+ 55,
19
+ 59,
20
+ 63
21
+ ],
22
+ "pairs": 32,
23
+ "tokens": 8192,
24
+ "share_by_pair": [
25
+ 0.016963258385658264,
26
+ 0.023762475699186325,
27
+ 0.023220213130116463,
28
+ 0.02519119717180729,
29
+ 0.03438106179237366,
30
+ 0.03531483933329582,
31
+ 0.037397172302007675,
32
+ 0.04328432306647301,
33
+ 0.027296338230371475,
34
+ 0.03136613965034485,
35
+ 0.03467932343482971,
36
+ 0.022985415533185005,
37
+ 0.05376176908612251,
38
+ 0.037242356687784195,
39
+ 0.022824469953775406,
40
+ 0.020885437726974487,
41
+ 0.013989882543683052,
42
+ 0.01596434786915779,
43
+ 0.026654137298464775,
44
+ 0.03354493901133537,
45
+ 0.0316782146692276,
46
+ 0.03538951277732849,
47
+ 0.036563269793987274,
48
+ 0.03921252116560936,
49
+ 0.0377679280936718,
50
+ 0.04123658314347267,
51
+ 0.0336410216987133,
52
+ 0.02321423962712288,
53
+ 0.06980059295892715,
54
+ 0.029939157888293266,
55
+ 0.020683670416474342,
56
+ 0.020164238288998604
57
+ ],
58
+ "ranking_desc": [
59
+ 28,
60
+ 12,
61
+ 7,
62
+ 25,
63
+ 23,
64
+ 24,
65
+ 6,
66
+ 13,
67
+ 22,
68
+ 21,
69
+ 5,
70
+ 10,
71
+ 4,
72
+ 26,
73
+ 19,
74
+ 20,
75
+ 9,
76
+ 29,
77
+ 8,
78
+ 18,
79
+ 3,
80
+ 1,
81
+ 2,
82
+ 27,
83
+ 11,
84
+ 14,
85
+ 15,
86
+ 30,
87
+ 31,
88
+ 0,
89
+ 17,
90
+ 16
91
+ ],
92
+ "top_half_by_norm": [
93
+ 4,
94
+ 5,
95
+ 6,
96
+ 7,
97
+ 10,
98
+ 12,
99
+ 13,
100
+ 19,
101
+ 20,
102
+ 21,
103
+ 22,
104
+ 23,
105
+ 24,
106
+ 25,
107
+ 26,
108
+ 28
109
+ ],
110
+ "shipped_map_compresses": [
111
+ 15,
112
+ 16,
113
+ 17,
114
+ 18,
115
+ 19,
116
+ 20,
117
+ 21
118
+ ],
119
+ "overlap": [
120
+ 19,
121
+ 20,
122
+ 21
123
+ ],
124
+ "split_check": "pair split reproduces the rotary-part norm on every layer"
125
+ }