LLM API outputs are unstable over time.

LLM APIs are opaque black boxes, even for open-weight models. TrackLLM continuously monitors them for undisclosed changes — cheaply — with logprob tracking (LT) where logprobs are exposed, and black-box border input tracking (B3IT) where they aren't. Small changes are fine, but they should be disclosed.

Latest detected changes

sparkline is the endpoint's drift around the changepoint (dashed): LT tracks how far the returned logprob averages have moved from the reference period (nats); B3IT the total-variation shift of border-input outputs (0–1) Full changelog →

Providers

detected changes per endpoint-year of monitoring — normalized for fleet size & monitoring length — with a 95% Poisson interval. Providers are grouped by company; their serving variants (fp8, fp4, …) are compared on the provider page
Provider Endpoints LT drift rate (changes / endpoint-year) Monitoring (1 sq = endpoint-month) B3IT Last change

Endpoints

Tracked Retired Untrackable Too expensive Not selected Errors Pending Free LT B3IT Ever changed Recent (60d)
Model Provider Status Changes Methods Stable for Drift from baseline