# TrackLLM — LLM API outputs are unstable over time - **TrackLLM** — machine-readable twin of [index.html](https://www.trackllm.net/index.html). Every page on the site has a `.md` sibling at the same path; links below point at those. - **Atom feed** of this page's detected changes: [feeds/all.xml](https://www.trackllm.net/feeds/all.xml) - **JSON** this page is rendered from: [data/home.json](https://www.trackllm.net/data/home.json) (home JSON) - **Raw data** (every collected response): https://github.com/timothee-chauvin/trackllm_data · Built 2026-09-09 09:47 UTC LLM APIs are opaque black boxes, even for open-weight models. TrackLLM continuously monitors them for undisclosed changes — cheaply — with logprob tracking (LT) where logprobs are exposed, and black-box border input tracking (B3IT) where they aren't. Small changes are fine, but they should be disclosed. | Stat | Value | | --- | --- | | Endpoints in the catalog | 1707 | | Endpoints ever monitored | 809 | | Active (tracked) endpoints | 259 | | LT endpoints | 466 | | B3IT endpoints | 316 | | B3IT under monitoring | 168 | | Providers / companies | 178 / 77 | | Models / organizations | 262 / 53 | | Changes detected (LT / B3IT) | 186 (77 / 109) | | Endpoints with a change | 147 | | Changes in the last 30 days | 60 | | Queries since Jun 2025 | 15092232 | | Last LT query | 2026-09-09T09:46:28Z | | Last B3IT query | 2026-09-08T23:34:36Z | | Cumulative spend (USD) | 262.14 | | As of | 2026-09-09 | ## Machine access Everything on this site is available without a browser: - **Markdown mirror** — every `X.html` has an `X.md` twin at the same path, with the same sections and the same links (to other `.md` pages). You are reading one. Charts are replaced by a link to the JSON series they draw. - **Atom feeds** — one entry per detected change: [feeds/all.xml](https://www.trackllm.net/feeds/all.xml) (latest 200), and one per endpoint, model, provider and organization at `feeds//.xml`, linked from each page. Subscribe in any feed reader, or point an RSS-to-email service at one. - **JSON** — the same files the HTML pages render from: [data/home.json](https://www.trackllm.net/data/home.json) (this page: stats, hero, latest changes, the plotted providers and the endpoint preview), [data/overview.json](https://www.trackllm.net/data/overview.json) (stats, every endpoint row, every provider row), [data/changes.json](https://www.trackllm.net/data/changes.json) (every detected change, newest first), [data/changes_page.json](https://www.trackllm.net/data/changes_page.json) (the change log with descriptions), [data/spend.json](https://www.trackllm.net/data/spend.json), `data/models/.json` (per-model: every endpoint's full drift / TV series and changes), `data/providers/.json` (per-provider rates, variants, endpoints), `data/b3it//b3it.json` (per-endpoint border-input state). Each page's header names the JSON it was rendered from. - **Raw data** — every collected response, committed hourly: https://github.com/timothee-chauvin/trackllm_data. The generator that turns it into this site: https://github.com/timothee-chauvin/trackllm_website. Endpoint slugs are stable: `#` with every character outside `A-Z a-z 0-9 . _ - + = @ ~ ,` replaced by its two-digit hex code — `/` → `2f`, `#` → `23`, `:` → `3a` (so `moonshotai/kimi-k3:free` at `chutes/fp8` is `moonshotai2fkimi-k33afree23chutes2ffp8`); its page is `endpoints/.md`. ## Latest detected changes [Full changelog](https://www.trackllm.net/changes.md) · [Feed](https://www.trackllm.net/feeds/all.xml) | Date | Endpoint | Method | Magnitude | Description | | --- | --- | --- | --- | --- | | 2026-09-05 | [gemini-3.1-pro-preview @ google-ai-studio/flex](https://www.trackllm.net/endpoints/google2fgemini-3.1-pro-preview23google-ai-studio2fflex.md) | B3IT | TV 0.28 | Probe-prompt outputs shifted (total variation 0.28) | | 2026-09-04 | [mistral-nemo @ deepinfra/fp8](https://www.trackllm.net/endpoints/mistralai2fmistral-nemo23deepinfra2ffp8.md) | B3IT | TV 0.34 | Probe-prompt outputs shifted (total variation 0.34) | | 2026-09-03 | [deepseek-v4-flash-0731 @ digitalocean](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-flash-073123digitalocean.md) | B3IT | TV 0.55 | Probe-prompt outputs shifted (total variation 0.55) | | 2026-09-02 | [claude-sonnet-4.5 @ claude-on-aws](https://www.trackllm.net/endpoints/anthropic2fclaude-sonnet-4.523claude-on-aws.md) | B3IT | TV 0.34 | Probe-prompt outputs shifted (total variation 0.34) | | 2026-09-01 | [mimo-v2.5 @ deepinfra/fp8](https://www.trackllm.net/endpoints/xiaomi2fmimo-v2.523deepinfra2ffp8.md) | B3IT | TV 0.34 | Probe-prompt outputs shifted (total variation 0.34) | | 2026-09-01 | [glm-5.3 @ modal](https://www.trackllm.net/endpoints/z-ai2fglm-5.323modal.md) | B3IT | TV 0.75 | Probe-prompt outputs shifted (total variation 0.75) | | 2026-09-01 | [deepseek-v4-flash @ atlas-cloud/fp4](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-flash23atlas-cloud2ffp4.md) | B3IT | TV 0.55 | Probe-prompt outputs shifted (total variation 0.55) | | 2026-09-01 | [mythomax-l2-13b @ parasail/fp16](https://www.trackllm.net/endpoints/gryphe2fmythomax-l2-13b23parasail2ffp16.md) | LT | shift 7.18 | Logprob averages moved 7.18 nats from baseline | | 2026-09-01 | [deepseek-v4-flash-0731 @ streamlake/fp8](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-flash-073123streamlake2ffp8.md) | B3IT | TV 0.34 | Probe-prompt outputs shifted (total variation 0.34) | | 2026-08-29 | [deepseek-v4-flash-0731 @ akashml/fp8](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-flash-073123akashml2ffp8.md) | B3IT | TV 0.52 | Probe-prompt outputs shifted (total variation 0.52) | ## Providers Detected changes per endpoint-year of LT monitoring, with a 95% Poisson interval, for the 10 most drift-prone provider companies. The full table, including providers without enough monitoring for a rate yet, is on [providers.md](https://www.trackllm.net/providers.md). | Provider | Endpoints | LT endpoint-years | LT changes | LT rate | LT 95% interval | | --- | --- | --- | --- | --- | --- | | [Alibaba Cloud](https://www.trackllm.net/providers/alibaba.md) | 53 | 2.23 | 5 | 2.24 | 0.28–4.20 | | [Chutes](https://www.trackllm.net/providers/chutes.md) | 88 | 15.62 | 27 | 1.73 | 1.08–2.38 | | [kluster.ai](https://www.trackllm.net/providers/klusterai.md) | 11 | 0.62 | 1 | 1.62 | 0.00–4.78 | | [StreamLake](https://www.trackllm.net/providers/streamlake.md) | 19 | 1.39 | 2 | 1.44 | 0.00–3.44 | | [Crusoe](https://www.trackllm.net/providers/crusoe.md) | 10 | 2.14 | 3 | 1.40 | 0.00–2.99 | | [Fireworks AI](https://www.trackllm.net/providers/fireworks.md) | 35 | 6.46 | 9 | 1.39 | 0.48–2.30 | | [Mancer](https://www.trackllm.net/providers/mancer.md) | 15 | 4.04 | 5 | 1.24 | 0.15–2.32 | | [Lambda](https://www.trackllm.net/providers/lambda.md) | 18 | 3.64 | 4 | 1.10 | 0.02–2.18 | | [CoreWeave](https://www.trackllm.net/providers/coreweave.md) | 14 | 1.12 | 1 | 0.90 | 0.00–2.65 | | [Hyperbolic](https://www.trackllm.net/providers/hyperbolic.md) | 10 | 2.22 | 2 | 0.90 | 0.00–2.15 | ## Endpoints 809 endpoints monitored at some point (259 under active tracking) out of 1707 in the catalog; the 10 tracked ones with the most detected changes are below, every one of them on [endpoints.md](https://www.trackllm.net/endpoints.md). | Model | Provider | Org | Status | Changes | Methods | Last change | | --- | --- | --- | --- | --- | --- | --- | | [hy3](https://www.trackllm.net/endpoints/tencent2fhy323atlas-cloud2ffp8.md) | [atlas-cloud/fp8](https://www.trackllm.net/providers/atlas-cloud.md) | [tencent](https://www.trackllm.net/orgs/tencent.md) | tracked | 3 | `B3IT` | 2026-08-28 | | [deepseek-v4-flash-0731](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-flash-073123streamlake2ffp8.md) | [streamlake/fp8](https://www.trackllm.net/providers/streamlake.md) | [deepseek](https://www.trackllm.net/orgs/deepseek.md) | tracked | 2 | `B3IT` | 2026-09-01 | | [deepseek-v4-pro](https://www.trackllm.net/endpoints/deepseek2fdeepseek-v4-pro23coreweave2ffp8.md) | [coreweave/fp8](https://www.trackllm.net/providers/coreweave.md) | [deepseek](https://www.trackllm.net/orgs/deepseek.md) | tracked | 2 | `LT` `B3IT` | 2026-08-20 | | [glm-5.2](https://www.trackllm.net/endpoints/z-ai2fglm-5.223cloudflare.md) | [cloudflare](https://www.trackllm.net/providers/cloudflare.md) | [z-ai](https://www.trackllm.net/orgs/z-ai.md) | tracked | 2 | `B3IT` | 2026-08-14 | | [glm-5.2](https://www.trackllm.net/endpoints/z-ai2fglm-5.223decart2ffp4.md) | [decart/fp4](https://www.trackllm.net/providers/decart.md) | [z-ai](https://www.trackllm.net/orgs/z-ai.md) | tracked | 2 | `B3IT` | 2026-08-11 | | [glm-5.2](https://www.trackllm.net/endpoints/z-ai2fglm-5.223together.md) | [together](https://www.trackllm.net/providers/together.md) | [z-ai](https://www.trackllm.net/orgs/z-ai.md) | tracked | 2 | `B3IT` | 2026-08-24 | | [glm-5.2](https://www.trackllm.net/endpoints/z-ai2fglm-5.223z-ai2ffp8.md) | [z-ai/fp8](https://www.trackllm.net/providers/z-ai.md) | [z-ai](https://www.trackllm.net/orgs/z-ai.md) | tracked | 2 | `B3IT` | 2026-08-19 | | [gpt-4o-2024-05-13](https://www.trackllm.net/endpoints/openai2fgpt-4o-2024-05-1323azure.md) | [azure](https://www.trackllm.net/providers/azure.md) | [openai](https://www.trackllm.net/orgs/openai.md) | tracked | 2 | `LT` | 2026-05-21 | | [gpt-4o-mini](https://www.trackllm.net/endpoints/openai2fgpt-4o-mini23azure.md) | [azure](https://www.trackllm.net/providers/azure.md) | [openai](https://www.trackllm.net/orgs/openai.md) | tracked | 2 | `LT` `B3IT` | 2026-05-23 | | [hy3](https://www.trackllm.net/endpoints/tencent2fhy323deepinfra2ffp8.md) | [deepinfra/fp8](https://www.trackllm.net/providers/deepinfra.md) | [tencent](https://www.trackllm.net/orgs/tencent.md) | tracked | 2 | `B3IT` | 2026-08-20 | --- Data collected via the OpenRouter API. [Home](https://www.trackllm.net/index.md) · [Changes](https://www.trackllm.net/changes.md) · [Providers](https://www.trackllm.net/providers.md) · [Endpoints](https://www.trackllm.net/endpoints.md) · [Methodology](https://www.trackllm.net/methodology.md) · [About](https://www.trackllm.net/about.md) · [GitHub](https://www.trackllm.net/github.md)