Who this is for: products that consume AgileV and cannot run code downloaded from outside. Where it comes from: on 28/9 the agent of trpg-01 had that rule in its own harness, did not run our script (it did not even let it see its own key) and rewrote the test bench in house, in its own language, committed to its own repository. It worked better: the reference was theirs. This page is that bench described as a protocol, so that anyone can redo it without taking a single byte from us.
If, on the other hand, you can run our script, use it: it does the same things in one command
(sha256 adb7eeafd59e6e2d608c6d2b5f12ed997afafa0b839c8cffabcf10fc8b707dad, prova-dal-posto.html).
What counts is not the file: it is the line you send back to us.
From our box we can see that the service answers and how long it takes. We cannot see whether the vectors you receive today are still the ones you built your index with. Same model name, different vectors, search that gets worse without a single error in the logs: it is the most silent fault AgileV can have, and it is only visible from there.
contract version read from /health.The expected cosine is 1.00000000, not «close to 1»: the same texts give the same vectors bit for bit (the same day, the same model, the same machine). trpg measured it on three sentences on 28/9: 1.00000000 on all three.
Within the same version the vectors are identical bit for bit. Across an announced deploy they can
move, and you deserve to know by how much before you measure it, otherwise the first person who sees 0.9999999999 instead of
1.00000000 writes to us about a fault that does not exist — as already happened with the comparison of the names.
The reason is known and deliberate: from 1.2 onwards large requests are split into several passes through the model on a token budget, so as not to leave the service out of memory. Grouping the texts differently means different padding, and on a floating-point model the last bits can change.
Measured by us on 28/9 on the real model, putting the two paths (1.1 and 1.2) side by side on the same model object, so that the only variable was the grouping:
| Your request | What changes with the deploy |
|---|---|
| fits in a single pass (up to 16 texts, or short texts) | nothing: bit-for-bit identical vectors, cosine 1.00000000 |
| exceeds the budget and is split (10 texts of 1,822 tokens → 2 passes) | 2 vectors out of 10 move; worst cosine 0.9999999999960902, maximum deviation on a component 3.06·10-7 |
reranking (/v1/rerank), 10 and 40 documents | the order does not change; scores within 4·10-11 |
What to do with it. The pact remains 0.9999, and now we know how much margin it has: the deploy consumes four hundred-millionths of that margin. So the rule becomes stronger, not weaker — if your «after» measurement drops below 0.9999, it is not the change of grouping: it is a real fault, and it is ours. Stop and write to us, as agreed.
A piece of advice that makes your measurement more sensitive: freeze the reference on short sentences, few per request. That way your request always fits in a single pass, the before/after comparison is exact bit for bit, and any movement you see is real at the first useful decimal instead of being buried under the noise of the grouping. It is what trpg did, and it is why their line says 1.00000000 and not «about 1».
| What you ask | Expected | Why |
|---|---|---|
GET /health (no key) | 200, ok:true, loaded:true, dim:1024, contract | the service is there and the models are loaded; contract tells you which version you are measuring |
GET /v1/models (no key) | 200, agilev-m3 and agilev-rerank | it lists only what really exists on this instance |
| a call without the header | 401 with error.code = KEY_MISSING | the errors are the documented ones: your client can branch on error.code |
| a call with an invented key | 401 with error.code = KEY_INVALID | |
GET /v1/usage | 200: owner, scopes, rate_per_min, your usage for the month | who you are and how much you have spent. Free: it consumes no quota and no per-minute request |
POST /v1/embeddings with {"model":"agilev-m3","input":[…]} | 200, data[].embedding of 1024 values, norm 1.0000 | the OpenAI-compatible format: it is the one to use for new clients |
| the same request, twice | identical vectors (cosine 1.000000), the second one nearly instant | determinism and cache |
| the cosine against your frozen reference | > 0.9999 | the check that counts (§1) |
POST /embed with {"texts":[…]} | 200, vectors[][] of 1024, model:"bge-m3" | the historical AgileVoyage format, frozen. Mind the body: §3 |
POST /v1/rerank with query and documents | 200, results sorted, index referring to your array | the reranking. Mind the fields: §3 |
X-RateLimit-Limit/Remaining/Reset headers | present on every authenticated response from contract 1.2 | with 1.1 online they are not there: report it as skipped, not as a failure |
| timings over 10 identical calls | median and maximum from your own box | round trip included: it is the number your user lives with, not ours |
/v1/usage before and after | +N requests, exactly the ones you sent | the invoice counts what you really sent |
trpg's bench, on its first run, had marked /embed as «broken» and the rerank as «it wants another model».
Neither: it was the bodies. Measured on 28/9 on the 1.1 code in production and on 1.2, identical outcome.
| Request | Response |
|---|---|
POST /embed with {"model":…,"input":[…]} (OpenAI body) | 422: /embed wants texts, input does not exist there |
POST /v1/embeddings with {"texts":[…]} | 422: the opposite swap, /v1/embeddings wants input |
POST /v1/rerank without documents (for instance with input) | 422: the required fields are query and documents |
POST /v1/rerank with "model":"agilev-m3" | 200, and it answers "model":"agilev-rerank": today the model name is only counted, never rejected. If a rerank does not go through, it is not the model |
Know three things about the 422: it is the only response without {"error":{"code":…}} (it is FastAPI's body,
detail[], no code: recognise it from the HTTP status); it arrives before authentication, so with a wrong
body and a missing key you receive 422 and not 401; and it sends back what you sent
(a batch of 64 texts, 89,600 characters, comes back as 89,909 bytes of error). In your error log record the status and the
error.code, not the whole body, or your texts end up in there.
ps and in the shell history. Read it from a 0600 file or from an environment variable./health, /v1/models, the two 401s and /v1/usage consume nothing. The rest costs: the full test is 7 requests of a few dozen characters.403 KEY_REVOKED) stop immediately, without spending requests.Send us back only this line — never the key, never the vectors, never your texts. One single line, in this format:
RIGA: agilev-prova-dal-posto <customer> <UTC ISO date> contratto=<1.1|1.2> controlli=<N> KO=<N> coseno=<x.xxxxxxxx> mediana=<t>A real example, trpg, 28/9 (their bench reports the median in milliseconds: that is fine, as long as the unit is there):
RIGA: agilev-prova-dal-posto trpg 2026-09-28T17:53Z contratto=1.1 controlli=22 KO=0 coseno=1.00000000 mediana=33msIf some check does not apply (the X-RateLimit-* on 1.1, for instance) count it as skipped, not as a failure,
and say so in words. If a check fails, send us what you asked and what you received: before declaring an
endpoint broken, compare it with §3 — the first time that happened, and it was not broken.
When to redo it: after every announced deploy in the release notes — the next one is the move to contract 1.2, and we will tell you when it is online — and before every large reindexing. The complete contract is in the API documentation.