Continuously monitoring 357 active endpoints

LLM API outputs are unstable over time.

LLM APIs are opaque black boxes, even for open-weight models. TrackLLM continuously monitors them for undisclosed changes — cheaply — with logprob tracking (LT) where logprobs are exposed, and black-box border input tracking (B3IT) where they aren't. Small changes are fine, but they should be disclosed.

Endpoints monitored
1,101
357 active · 744 retired · 2,070 in the catalog
Models tracked
290
across 57 orgs
Providers
84
234 serving variants
Queries logged
16.0M
since Jun 2025
Changes detected
297
79 LT · 218 B3IT

Inria IRISA CNRS SequoIAQ
Subscribe

Latest changes

how far each endpoint moved from its baseline All 297 changes →
31 Oct
Probe-prompt outputs shifted (total variation 0.29)
B3IT
31 Oct
Probe-prompt outputs shifted (total variation 0.38)
B3IT
30 Oct
Probe-prompt outputs shifted (total variation 0.51)
B3IT
30 Oct
Probe-prompt outputs shifted (total variation 0.27)
B3IT
30 Oct
Probe-prompt outputs shifted (total variation 0.24)
B3IT
29 Oct
Probe-prompt outputs shifted (total variation 0.26)
B3IT
29 Oct
Probe-prompt outputs shifted (total variation 0.28)
B3IT
27 Oct
Probe-prompt outputs shifted (total variation 0.33)
B3IT
27 Oct
Probe-prompt outputs shifted (total variation 0.43)
B3IT
25 Oct
Probe-prompt outputs shifted (total variation 0.65)
B3IT
More…

Providers

detected changes per endpoint-year of LT monitoring All 84 providers →
More…

Endpoints

1,101 endpoints monitored, of 2,070 in the catalog · 297 changes across 222 endpoints All endpoints →
Endpoint Status Changes Methods Last change Drift from baseline
deepseek-v4-flash-0731streamlake/fp8changed 4 LTB3IT 27 Oct
deepseek-v3.2atlas-cloud/fp8changed 3 LTB3IT 30 Oct
hermes-4-405bnebius/fp8changed 3 LT 15 Oct 2025
hy4-previewtencent/fp8changed 3 B3IT 22 Oct
mimo-v2.5xiaomi/fp8changed 3 B3IT 21 Oct
claude-sonnet-4.5claude-on-awschanged 2 B3IT 9 Sep
deepseek-v4-flashparasail/fp8changed 2 B3IT 3 Oct
deepseek-v4.1-flashalibabachanged 2 B3IT 16 Oct
deepseek-v4.1-flashdeepinfra/fp8changed 2 B3IT 31 Oct
deepseek-v4.1-flashnovita/fp8changed 2 LTB3IT 17 Oct
More…

Timothée Chauvin, Erwan Le Merrer, François Taïani, Gilles Tredan (2026), "Log Probability Tracking of LLM APIs". ICLR 2026. https://arxiv.org/abs/2512.03816

Timothée Chauvin, Clément Lalanne, Erwan Le Merrer, Jean-Michel Loubes, François Taïani, Gilles Tredan (2026), "Token-Efficient Change Detection in LLM APIs". ICML 2026. https://arxiv.org/abs/2602.11083

@inproceedings{
chauvin2026log,
title={Log Probability Tracking of {LLM} {API}s},
author={Timothee Chauvin and Erwan Le Merrer and Francois Taiani and Gilles Tredan},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=hFxivbAgVP}
}

@inproceedings{
chauvin2026tokenefficient,
title={Token-Efficient Change Detection in {LLM} {API}s},
author={Timothee Chauvin and Cl{\'e}ment Lalanne and Erwan Le Merrer and Jean-Michel Loubes and Francois Taiani and Gilles Tredan},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=7cMlZZYZT0}
}