Who this is for: anyone with an index built in several goes, or moved from one provider to another, with no field saying which model wrote which chunk. Short name: it is countermeasure (e) of Rule 4.
Where it comes from. trpg, 28/9 at 22:3x, after measuring their own index:
«Our index is built with two providers — 442 chunks up to 28/8 with Voyage, 29 since 18/9 with you — and both give 1024 values, so no guard on the dimension tells them apart. And our chunk table does not record the model anywhere: the information needed to notice has never existed. We have no way of knowing whether our mixed index is healthy, and we never will for those chunks.»
The corollary they drew from it — write the model name next to the chunk, today — is right, and it is countermeasure (d). The giving-up, on the other hand, was one step too far, and the fault is ours: that sentence («impossible to recover tomorrow») we had written in the rules page too, without having measured it. If the text is still next to the vector, the provenance can be derived. This page is the measurement.
Take a chunk from the index, re-embed its text with us, and compute the cosine between the new vector and the stored one.
No new field is needed and nobody needs to have declared anything: like (a), it applies to the index you already have. The price is one call per chunk.
Taken on 28/9 with the real model in a memory cage, on 64 chunks of ours (26,183 characters),
zero requests to production. Bench: bench/misura-provenienza.py, result committed in
bench/risultati/provenienza-28-9.json. 13 scenes, 0 failures.
| How the stored vector was made | Minimum cosine | Median | Verdict |
|---|---|---|---|
| ours, same request | 1.00000000 | 1.000000 | ours |
| ours, written in another batch (different padding) | 1.00000000 | 1.000000 | ours |
| ours, float16 storage (half precision) | 0.99999998 | 1.000000 | ours |
| ours, int8 storage (scalar compression) | 0.99983766 | 0.999881 | ours |
| ours, one-bit storage (sign only) | 0.75374678 | 0.765632 | ours |
| ours, but the text was edited afterwards | 0.97554815 | 0.989328 | ours |
| ours, but the text was prepared by the indexer (title, prefix, cut) | 0.75475971 | 0.833223 | ours |
| ours, but of another chunk (a similar one) | 0.78578071 | 0.871493 | ours |
| ours, but of a text with nothing in common | 0.23538498 | 0.383763 | worth a look |
| another space: orthogonal rotation | −0.09801596 | −0.020217 | stranger |
| another space: permuted coordinates | −0.04225830 | −0.001143 | stranger |
| another space: flipped signs | −0.10105777 | −0.046048 | stranger |
| another space + int8 storage | −0.09793994 | −0.020290 | stranger |
The worst of ours is 0.7537, the best of the strangers 0.1011: between the two families there is a factor of 7, and nothing in between. The test is decidable.
What «another space» means here. We use no external provider — it is a rule of the service — and we do not have Voyage's vectors. In their place we used isometries of our own: random orthogonal rotation, permutation of the coordinates, flipping of the signs. These are spaces with the same dimension, the same norm and the same internal geometry as ours — that is, of the same quality — written in a different basis. A real model, trained on its own, can only be less aligned with us than that, not more, unless it is distilled from ours. It is the worst case for us, not the most convenient one.
| Cosine | Verdict | What to do |
|---|---|---|
1 − cosine ≤ 1e−6 | ours, and of this text | nothing. It is the normal case on a healthy index stored in float32. |
cosine ≥ 0.50 | our space | nothing, but if it is not 1 then the text or the storage has changed: see point 4. |
0.15 < cosine < 0.50 | worth a look | look at the text of that chunk before concluding: a text that has nothing to do with the vector lands here. |
cosine ≤ 0.15 | another space | we did not write that chunk. |
The threshold that counts is the first one, and the margin is enormous: between the re-made vector and the stored one, on the
same text, we measured a maximum deviation of 2.2·10⁻¹² from cosine 1 — six orders of magnitude
below the 1e−6 threshold. If you store float32 (which is what you do with pack('f*') or a
float BLOB), your verdict is exact: either the cosine is 1, or that vector is not ours.
(1) The stored vector comes from a different batch. It does not matter: measured, cosine 1 even
rewriting the same texts one at a time instead of sixteen at a time. And our own deploy does not matter
either: between 1.1 and 1.2, which group the texts differently, the worst 1 − cosine measured
is 3.9·10⁻¹², still below 1e−6. This test can be done before or after the deploy with the
same verdict.
(2) The storage compresses. It depends how much. At half precision
(float16) the exact verdict still holds: cosine 0.99999998, that is a 1 − cosine of
2·10⁻⁸, fifty times below the 1e−6 threshold — if you halve the space you lose nothing of this
test. With int8 the cosine drops to 0.9998 and with one bit per coordinate to 0.75: it is no longer «1»,
so the first threshold does not apply, but the verdict on the space holds comfortably. If you compress more than half
precision, use 0.50 and not 1e−6.
(3) The text changed afterwards. One different word brings the cosine to 0.9755, a neighbouring chunk to
0.7858: they all stay above 0.50. A text with nothing in common with the vector drops
to 0.2354 — above the stranger threshold, but in the «worth a look» band. So: a wrong text
cannot make you condemn a chunk of ours as a stranger; at worst it makes you stop and look.
(4) You re-embed a text different from the one you sent. It is the easiest mistake, and
the only one that throws the whole index off at once. If, when building the index, your code
prepares the text — puts the document title in front, cuts it at N characters, adds
an instruction like «query: », lower-cases it — then the stored vector is of that text,
not of the one you read in the column. Measured (title + prefix + cut at 200 characters): cosine
0.7548 in the worst case, median 0.8332. So it does not make you say «stranger» — the verdict
on the space stays ours, and the separation from the strangers holds — but the exact
1e−6 threshold no longer holds on any chunk: whoever uses only that condemns a healthy index
entirely. Re-embed through the same function the indexer uses. The tell-tale sign:
the cosines are all odd in the same way, and none of them is 1.
You do not need to redo the whole index to know whether it is mixed. Sampling at random, the probability of drawing at least one
stranger chunk is 1 − (1 − share)^n. For 95 % confidence:
| If the strangers are | 50 % | 20 % | 10 % | 5 % | 1 % |
|---|---|---|---|---|---|
| chunks to look at | 5 | 14 | 29 | 59 | 299 |
The cost. On our box a chunk of ~400 characters costs 0.26 s of computation and is billed
for its characters. An index of 471 chunks, in its entirety, fits in two minutes of service.
Send them in batches (16 per request is fine) and not one at a time, so as not to take the 429.
Sampling tells you whether the index is mixed. If you want to label every chunk — and finally write the field of (d) on real data instead of starting over — then you need all of them, and once only: the verdict is stable, it does not change at the next deploy.
We are not sending you a script: on the Agile boxes the harness forbids running code downloaded from outside (trpg taught us that, and it holds for the others too). These are the steps; write the code yourselves, in house, as you did for the on-site test.
POST /v1/embeddings with {"model":"agilev-m3","input":[…]}.
Check the model of the response (countermeasure (b)) and the length of the vector.RIGA: agilev-provenienza <product> <date> frammenti=<n> nostri=<n> estranei=<n> da-guardare=<n> coseno-min-nostri=<x>
A sanity check on your own bench, before believing it: take a chunk you know is ours and any random vector (normalised random numbers). The first must give 1, the second something around zero. If the second does not give you a small number, you are comparing them wrongly — for instance two unnormalised vectors, or bytes read with the wrong endianness.
Nothing, on your data: texts do not leave and are not recorded, and to us a provenance test is indistinguishable from normal indexing — we only see the counters. If you want to tell us the outcome, send the line, not the chunks.
On our side this test changes one thing: the sentence «the information is impossible to recover tomorrow» in countermeasure (d) was wrong, and it is corrected in the rules page. The advice stays identical — write the model name next to the chunk now, it costs nothing — but whoever did not do it has not lost the index: they have two minutes of calls ahead of them.
This page is self-contained, with no scripts and no remote resources: you can read it, print it and copy it wherever you like. The others: the rules for whoever writes the client, the on-site test, the model field.