1.1.0

Breaking changes and migration

Breaking: The Alarms page’s setup is stored as horizon.alert.default instead of horizon.alert.page-setup. A deployment that changed its pinned layers, window or overview alarm limit finds them back at the shipped defaults after upgrading; set them again at Dashboard setup → Alarm pages. The horizon.alert.page-setup record stays on OAP and is no longer read. A deployment that mounts its own bundled templates, in live or readonly mode, renames its alert/page-setup.json to alert/default.json before upgrading; until it does, Horizon cannot load its bundled templates.

Breaking: Each of the six Dashboard setup pages now has its own read/write permission pair, and three permissions are retired. dashboard:read / dashboard:write become layer-template:read / layer-template:write; overview:write splits into overview-template:write, translation:write, alarm-setup:write, infra-3d-setup:write and setup:write; overview:read keeps only its original meaning, the rendered overview dashboards. The built-in roles are updated for you, and a role still naming a retired permission keeps working — the old names, and the dashboard:* / overview:* wildcards, are accepted for this release and expand to what they used to gate. Rename them in your horizon.yaml before the next one; Horizon now logs a warning at startup naming any permission it does not recognise, which it previously ignored in silence. Two consequences worth checking: viewer and maintainer no longer read stored template configuration (they keep every dashboard they could see), and a role granted overview:read alone no longer reaches the template-administration pages.

Breaking: Turning the Cold pill on no longer re-reads the page. It changes what the NEXT read asks for — the following auto-refresh round, or whatever you do first, typically choosing the time range the cold data lives in. Flipping it used to start a round of its own, which on a cold tier routinely ran past the sixty-second cap; the sweep that followed then re-queued exactly the queries the cap had just cancelled, so the cap was at its most useless during the outage it exists for. And because a cold read REPLACES the hot one rather than widening it, an operator who had not yet moved the time range watched the whole page empty on the click. Anything already in flight is cancelled, so no batch is left half hot and half cold — the stage is read when a request goes out, and a call still queued behind the concurrency limiter would otherwise leave under the new stage while belonging to the old one. Nothing to migrate: if you want the page re-read immediately after flipping, press Refresh.

Breaking: An overview widget placed on one service group of a layer is saved as GENERAL[payments] — the way a role grant and an alarm page name a group, GENERAL[-] for the services with no group — and the older GENERAL~PAYMENTS form is no longer read: such a widget shows that its layer is not a layer, in place of its data. In 1.0.0 that form changed the group’s letter case, so a widget read its group only when the name survived the change. Open the overview in Dashboard setup → Overview templates, pick the widget’s layer again, and push.

Breaking: Layer page addresses changed, and old addresses are no longer redirected. A group entry of a layer split by service group moves from /layer/general~payments/… to /layer/general/payments/…, its services with no group from /layer/general~/… to /layer/general/…, and a sub page from …/service/<id> to …/service/page/<id> (the same for Instance and Endpoint). The old tab names that still led to their renamed tabs — profiling, services, dashboards, instances, endpoints, traces — and /setup no longer open anything either. An old address shows Not found: update bookmarks and links kept outside Horizon, and any rbac.landingByRole route that names one. On a layer whose traces come only from Zipkin, such as Mesh, the Traces tab moves from …/trace to …/zipkin-trace, and the old address opens the layer’s first tab.

Breaking: A dashboard template stored on OAP whose content would break the pages built from it is no longer served. Such a template — written to OAP by something other than Horizon, or by a Horizon that checked less — used to render partly, or to break the sidebar or every dashboard along with it; now it is hidden from every menu, settings fall back to their built-in defaults, and the rest of Horizon renders normally. If a layer, overview, alarm page, translation, the 3D map or a global default goes missing after the upgrade, look for the red warning on the Dashboard setup rows in the sidebar: the page lists each invalid template with its problems. Disable it there and configure it again, from the shipped default or from a corrected file — see Invalid Stored Templates.

AI agent conversations

Features

Horizon can now read the AI agent conversations an OAP stores. OAP 11.1.0 keeps the conversations that the SkyWalking AI Sessionizer pushes under the AI_AGENT layer, one row per conversation and one document per conversation. This release wires Horizon to both reads, behind a new permission, ai-conversation:read, that the built-in viewer, maintainer and operator roles carry. It is its own permission rather than part of ai:read, because that one is the AI assistant, where Horizon sends text to a model provider; this one reads stored transcripts of other agents, and a deployment may allow either without the other. Two settings under performance.aiConversation tune the reads: listLimit, how many of the newest rounds the list folds into rows (default the OAP ceiling, 10,000, because a smaller number lets one long conversation hide the short ones), and viewTimeoutMs, how long to wait for a conversation document, whose first byte arrives only once OAP has folded the whole chain (default 120 s, matching OAP). See Configuration File and Roles and Permissions.

A new bundled layer, AI Agents, with a Conversations tab. When OAP reports the AI_AGENT layer, the sidebar gains an AI Agents entry whose one tab lists the conversations of one agent: pick the agent in the layer header and a time range (a day to 90 days), optionally one agent runtime, click Run query, and each row shows the conversation’s title, which Sessionizer sent it, its talks and unresolved references, its span and its last activity, newest first. A title text and a conversation id are query conditions too, applied by OAP. The line above the table states the round budget the list was folded from, because OAP builds the list from the newest rounds rather than from conversations and cannot say whether the budget cut anything. Layer templates gain the aiConversations component flag behind this tab. See AI Agent Conversations and AI Agents.

A conversation opens in its own tab, as one shareable page. Clicking a row on the Conversations tab opens /ai-conversation/<id> in a new browser tab: the conversation’s transcript, with the agent’s work folded under each input; a flow timeline of one execution stream with lanes for input, responses, context, model calls, tools, agent activity and nested child streams, relation lines for the selected step, and a way to dive into a child agent and back; and an inspector with the selected step’s details, relations and landed evidence, where each position the step came from shows what the document carries for it: the step’s text, or on a call’s result position the call’s result. The address carries the conversation, its agent and agent runtime, and the reader’s position (talk, step, stream), updated in place as you move, so it can be shared and lands on the same step. Each read follows the current Cold selection, requesting cold storage only when enabled and otherwise using OAP’s default stages. The page reads the whole document from OAP and shows the wait in phases — OAP assembling the conversation with the seconds counting, the bytes arriving as n of m MB with a percentage, rate and time left, then the parse and the draw — states the document’s integrity (verified, incomplete, mismatch) and the Sessionizer’s recorded problems, and follows your Horizon theme, the kinds of step in their own colours per theme. The renderer behind it is a framework-free module the SkyWalking AI Sessionizer’s own viewer embeds too, built from a pinned Horizon commit, so both show a conversation identically. See AI Agent Conversations.

The AI Agents layer gains metric dashboards for the agent. OAP now keeps per-agent and per-agent runtime metrics under the AI_AGENT layer: the tokens the AI Sessionizer derives from the transcripts it lands, agent.token.usage, and the metrics only Claude Code’s own exporter sends when it is pointed at OAP directly. OAP does not read the exporter’s token metric, so a call is never counted twice. The layer’s landing page lists agents with two columns, Tokens and Cache read share, and the layer gains an Agents dashboard and a Agent runtimes dashboard. Four widgets are always shown, Tokens by type, Main agent and subagents, Tokens by model and Cache read share; seven more, Cost by model, Active time, Sessions started, Lines of code, Commits, Pull requests and Edit permission decisions, appear only when Claude Code’s own exporter reports, because a Sessionizer-only deployment never has them. Every point is the total of its bucket, counted at the end of the minute a call ended, so no title says “per minute”. An OAP that already holds the earlier, conversation-only AI Agents template keeps it: adopt the dashboards from Dashboard setup → Layer dashboards → AI Agents with Reset to → Bundled and Check diff & push. See AI Agents.

The AI Agents layer shows the MCP tools each agent called. OAP turns the AI Sessionizer’s agent.mcp.calls and agent.mcp.duration, derived from the records its Claude Code plugin writes around each call to an MCP server, into one endpoint per server and tool, named <server>/<tool>, under the agent. The layer gains an MCP tools tab, the endpoint dashboard, with the calls, the calls by outcome (returned, failed, interrupted), the average time and the time in calls; the Agents dashboard ranks the agent’s top 20 MCP tools by calls or by time in calls, each averaged over the buckets that had calls. The time is what the runtime measured around the call, waiting included, not the server’s own. An OAP that already holds the AI Agents template keeps it; see AI Agents to adopt the new one.

The conversation page shows which files each tool call changed. When the AI Sessionizer recorded a call’s workspace changes, the call’s card in the transcript wears a mark beside the tool name, 2 files · +3 −1, or an eye reading read-only for a shell command the asz Claude Code plugin classed as read-only and did not scan, which is not the same as no changes. Clicking the mark opens the changed files under the card, each opening to its diff, without moving the selection; the same mark sits on the step’s clip in the flow timeline. The inspector gains a Changes tab with each record’s provenance: who recorded it (the agent’s own patch on Edit, Write and NotebookEdit, or the plugin’s observation of shell commands and subagent edits), the scans it ran between, the command’s outcome, the policy sets, a warning when a scan stopped early, and the landed position it was read from. Two records of one call are kept apart, the agent’s first, and one command that touched several configured workspace roots is listed by root. The status strip counts the conversation’s change records and opens a panel of every changed file by workspace root, each record a link that opens the step that made it on its Changes tab, with changes no observed tool window made listed at the end. See AI Agent Conversations.

The Conversations tab says what each conversation did. Its columns are now Talks, Model calls, Subagents, Bash runs and Changes: the change records the Sessionizer captured as of the newest round, one per observation (a watched tool call per workspace root, the agent’s own patch of an edit, a scan that found nothing, a change seen between calls that no call made), with the plus-over-minus mark and the lines those records added and removed, summed over the records, beside a count above zero. A dash in any of the four means the Sessionizer that pushed the conversation predates the count. They need an OAP that carries the counts in its conversation list.

The conversation page shows the prompt each model call sent, and the answer it got. When the AI Sessionizer captured the bodies a call exchanged with its model provider, the call’s step gains a Prompt tab: the exact request as it was sent and the exact response as it came back, neither of which a transcript holds. The bodies are not part of the conversation document — they are megabytes per call — so the tab opens on an offer stating how many stored files of the session it needs and how many bytes they are, and reads nothing until you press the button. What it reads it keeps while the page is open, so walking from call to call within a session costs one read, and the tab stays chosen as you move. A request opens whole, with its system prompt, its tools, its messages and its settings, each message in a box headed by its role and its place in the request; What it added compares it with the request the call before it sent and shows only what is new, naming the shared history instead of repeating it, and saying when the system prompt, the tools or the settings changed too, or when a compaction rewrote what was shared. A request that names no request before it, such as the first of its chain or any call of a LangChain agent, says so rather than that the one before is not loaded. A response shows its stop reason, the token counts the provider reported and its blocks. Each message, block and tool, and each section that is one document, such as the whole body, has a copy button in its title, which copies the whole text even while the tab shows it cut short or the section is closed. Every body is rebuilt in your browser and checked against the byte count and the SHA-256 digest the Sessionizer recorded, along with the pieces and copies it refers to; one that does not rebuild says so instead of showing a partial body, and one of a shape the reader does not know is shown as the document it is. Reading needs ai-conversation:read, the same permission as the conversation itself, and an OAP that serves the files route (11.1.0 or later). See AI Agent Conversations.

The Prompt tab reads a LangChain or LangGraph agent’s calls. For such an agent the Sessionizer stores what LangChain handed its model, a list of LangChain’s own serialized messages, and the tab drew each of those requests as one empty message. It now shows every message in LangChain’s words, system, human, ai and tool, with the tool calls an ai message made, each drawn once with the arguments LangChain parsed even where the provider repeats it in the message content, and each tool result under the call it answers, marked failed when LangChain marked it. A summary that LangChain’s SummarizationMiddleware put in place of the history is the human message it was sent as. A response shows the answer, the model and the token counts LangChain reported, with the provider’s own metadata under Settings. What it added is not available for a LangChain call, because the Sessionizer does not record which of the stored requests came before it. A body of several prompts at once, or of a shape the tab does not know, is shown as the document it is.

The conversation page shows what each call to an MCP server did. A call to an MCP server is drawn on its own MCP lane of the flow timeline, and its card is named by the server and the tool instead of Claude Code’s mcp__<server>__<tool>. When the asz Claude Code plugin observed the call, the card wears how it ended and the time the runtime measured around it, returned · 380 ms, and the inspector gains an Execution tab: the server by its configured name and where that configuration came from, the outcome, the measured time, and the size and digest of what was sent and what came back. The measured time includes any waiting before the call started, so it is not the server’s own time. The Evidence tab lists the landed position of each execution record and change record after the step’s own, and shows a picked record as the document carries it. Execution records need an OAP that lists them in the conversation document (11.1.0 or later). See AI Agent Conversations.

The inspector says where a step sits in words. Its Details tab opens with the step’s place: the activity window and its number, the agent, the talk by its opening line and its place in the stream, the run’s place in the talk, and the step by kind and name, instead of a chain of clipped identifiers; the identifiers stay in the tooltip.

Tool inputs and results read as fields. On the conversation page, a tool call’s card and its Details tab draw the input the agent recorded as the fields it holds instead of one escaped JSON string: a shell command as the lines that were run, an edit as its path and the text before and after, a result’s output and error streams apart. A long text is cut short with a show all link that opens it in place, and where the document clipped the text a note says how much of it is there. Every field has a copy button beside its name, a result that is one text has one on its result line, an edit’s old_string and new_string are drawn as one diff with removed lines red and added lines green, and the inspector pops out into a panel over the whole page, four fifths of it, with the button in its header whose arrows point outwards, docked again with that same button once its arrows point inwards, or with Escape or a click outside it.

Fixes

A tool call whose change record was made smaller no longer says it changed no file. The AI Sessionizer takes the file list out of a change record too large for its stored file, keeping how many files changed. The tool card’s mark and its Changes tab said “no files changed” for such a record; they now count its files and say the list was taken out to fit. A file whose diff was taken out the same way says its line counts were kept, rather than that it was too large to diff.

The Prompt tab names a body that cites a piece too large to be stored. The AI Sessionizer leaves a piece larger than its file budget, such as a pasted screenshot, out of the stored file, keeping its size and SHA-256, and a later body refers to that piece by its digest. The tab took such a body for one waiting on a file it had not read, so the panel kept saying a file of the session was not stored, and a body that holds or cites such a piece said only that it did not come back whole. It now reads those references, and says a piece of the body was too large to be stored. The renderer the Sessionizer’s own viewer embeds also rebuilds a body across the Sessionizer’s retention, from the bytes the Sessionizer kept, and says when a body was removed; a conversation read from OAP never has one.

GenAI evaluation

Features

• The Virtual GenAI layer gains an Evaluation records tab: the LLM-as-Judge results OAP stores for a provider, one row per judged call with its task, score or verdict, level, reason and judge model, over a level histogram of the page and a detail popout. Provider, model and the calling service are picked in the page with type-to-filter pickers; the conditions cover the value — a score range, a true/false verdict, or a string or JSON result — plus task, calling service, judge model, trace id and a time range up to seven days, and as on the Logs tab they are staged until Run query. The trace link on a row opens the trace the call was judged from — the native waterfall for a SkyWalking-agent trace, the Zipkin renderer for an OTLP one, which finds it only when OAP converts OTLP spans into Zipkin form (otlpTraceStorage: zipkin; OAP 11.1.0 stores them natively by default) — looked up at the record’s own time, so an older trace still opens with the query cold stage on. The trace-id condition narrows to one span of that trace: Pick span… lists the trace’s spans with the ones the judge samples marked LLM and shown first, and picking one fills the segment and span index (native) or the span id (OTLP); the same fields can be typed in from ids copied out of logs. Reading the tab needs logs:read; its provider picker works for a role without metrics:read.

• The Virtual GenAI Models dashboard gains an Evaluation Score chart, and a point on it drills into the Evaluation records tab scoped to that provider, model, task and hour.

Sign-in and access control

Features

A role can be limited to layers, and to service groups within a layer. A data permission can now name the layer it applies to — metrics:read@GENERAL, or metrics:read@GENERAL[payments,risk] for the services whose OAP service group is payments or risk — so one Horizon can serve several teams that each see only their own services. The sidebar follows the grant: such a role sees the layers, and the per-group entries of a layer split by service group, that its grants reach, with service counts to match. Every request is checked against the service it reads, whichever page or URL it arrives through, and so is every read the AI assistant or an MCP agent makes for the role. Links between services are navigation: the topology map, the instance map, the API dependency graph and the hierarchy view still draw a service’s neighbours as OAP returns them, but a neighbour the role may not read shows its metrics as blocked — they are not read — and a call between two services carries its values only when the role reads one end — only its relation metrics when that end is the destination; the page a link opens is checked like any other. The All services map is one hop out from the role’s services, and the Depth control applies to a focused map. A network-profiling graph is the profiling of one instance, so the role that reads that instance sees all of its processes and calls. Profiling is read on the page’s layer and a service the role reads: its tasks, policies and results, and the analysis of a task’s own segments and schedules. Traces cross layers, so they are not narrowed by layer: traces:read on a layer reads its trace tabs, Zipkin and TraceQL included, and opens any trace by id. Evaluation records follow both sides of the call: the provider needs VIRTUAL_GENAI granted and the calling service its own layer, and a side granted only in part must be picked rather than left as “all”. With alarms:read limited to layers, every alarm page, the alarm count, the overview Alarms widget and the 3D map show the alarms of the role’s own services, with no service to pick first, and an alarm’s snapshot comes with it whole. With a layer granted only in part, a query names one of the role’s own services: pickers offer no “all”, and reads that span every service — the SkyWalking trace list, the Logs and Browser Logs tabs without a service picked, overview KPIs that total a whole layer on the server, log tag autocomplete, source maps — need the permission without a layer. A permission without @ means what it always has, so a deployment that grants none sees no change. See Limiting a verb to layers and service groups.

Any permission that opens a layer page can fill its pickers. The service picker at the top of a layer, and the instance and endpoint pickers on its tabs, used to need metrics:read (the instance list also took logs:read), so a role granted only traces:read or logs:read could not pick what its own tab was about. They now open for any of the layer-page permissions — and the instance and endpoint pickers also for alarms:read, since the Alarms page narrows by them — and list only what the role may read. A role without metrics:read on the layer picks its service from the layer’s service list, without the metric columns.

A custom role can now be added without restating the built-in ones. Naming rbac.roles has always replaced the block outright, so adding one role meant copying viewer, maintainer, operator and admin into your config and keeping those copies current forever. Set builtinRoles: keep and the built-ins become the base instead: a role name you list overrides that one role, a new name is added, and everything you did not mention stays as it shipped — landingByRole merges the same way. The default is unchanged (replace), because a deployment that trimmed its block to remove a role must not have it handed back by an upgrade, least of all admin. Under keep a built-in can no longer be dropped by omission; grant it nothing (admin: []) instead. The effective role names are logged at startup so you can confirm what a merge produced. See Access Control Configuration.

The Dashboard setup pages open read-only when you may look but not publish. Each page — Overview templates, Layer dashboards, Translations, Alarm pages, 3D Infra Map, Global defaults — is reached with its own read permission and published with its own write permission. Granting the read half alone shows the page in the sidebar and opens it with the whole configuration visible, every editing control disabled, and a banner naming the permission publishing needs; granting the write half turns the page back on. Previously three of these pages could not be opened at all without edit rights, and the other three rendered every button enabled and failed at the click. Refresh from remote is now treated as the read it is, so it works for a read-only viewer instead of being refused. A read-only page shows what OAP holds (or the shipped default) rather than an unpublished draft saved in your browser: the draft is kept and the page marks it local, but it is not displayed while you have no way to discard or publish it.

The login audit can now be stored in BanyanDB instead of PostgreSQL. Set audit.provider: banyandb and point audit.banyandb.address at a liaison’s gRPC port — including the BanyanDB SkyWalking already stores its telemetry in, since Horizon keeps the records in groups of its own (horizon_audit and horizon_audit_metrics, prefixed by namespace where two deployments share one server). The page is the same on either backend: the same rows, the same three filters, the same hourly summary and token-usage tab. Retention is the groups’ own lifetime rather than a job Horizon runs, so there is no sweep interval to tune; on a cluster, set shards and replicas, because the single-node defaults would keep the whole audit log on one data node. As with PostgreSQL, an address that is not loopback must use TLS unless allowCleartext says otherwise. See Login audit.

Fixes

The Platform monitoring layers are now closed to a role without cluster:read. The OAP, BanyanDB, agent and Satellite self-observability layers have always been listed in the sidebar only for cluster:read, and the Roles page lists them under it, but the layers’ dashboards, instances and metrics answered any role with metrics:read: by opening the layer’s address directly, by naming one of their services under another layer’s address, or through a self-aggregating column on any layer’s service list. They now refuse such a role everywhere — the layer pages, the 3D map’s Self-Observability group, the AI assistant and MCP included — the menu no longer sends those layers to it, and the all-services Logs and Events views, the alarm pages, the alarm count, the overview Alarms widget and the 3D map leave their services’ rows out. Metrics inspect is unchanged: inspect:read is raw access to every metric by design. To open one of these layers to a role without cluster:read, grant it by layer, for example metrics:read@BANYANDB.

A draft template preview now needs layer-template:write. The topology, deployment, instance map, API dependency, process relation and trace routes evaluated a draft template sent with the request for any role that could read the page — the preview is the layer dashboard editor’s, and now takes the editor’s permission, as the TraceQL preview already did.

A layer-wide total is computed only from an expression a stored overview declares. The self-aggregating columns of a layer’s service list, which read a metric across every service with no service in the query, evaluated whatever expression the request carried. They now run only the expressions the stored overview templates, or the layer’s own header, declare; anything else is left out of the answer.

A malformed permission no longer grants more than it says. A grant is at most three segments, but a longer one used to be silently truncated to its first three — so rule:*:typo was read as rule:* and handed over every rule permission, and rule:write:structural:extra matched rule:write:structural. Anything outside the documented forms now matches nothing and is named in the startup warning alongside other unrecognised grants.

The BanyanDB audit configuration is only what a deployment actually differs on — where the server is, how to authenticate to it, a namespace prefix, and how long to keep records. The groups’ layout is bundled rather than exposed: one shard, which a sign-in rate never outgrows. That is also the safe default, because BanyanDB fixes a group’s shard count at creation and does not move data when it changes — a shard count edited after records exist would route new ones away from the old ones rather than resharding. Horizon now reports such a difference instead of applying it. Replication is left to you: Horizon sets no replica count, so a deployment that wants the audit log replicated can create the two groups with the count it wants and Horizon will keep them that way.

The login audit list pages the way every other SkyWalking list does. It asks for a page number rather than carrying a position between requests, which is the arrangement OAP uses for traces, logs, alarms and events alike. The page still shows 50 rows at a time and still reports only whether more exist; paging is bounded at 500 pages deep, and a request past that is refused rather than quietly answered with something else.

The hourly summary and token-usage totals now refuse a window they cannot read completely. Both are sums, so a row that did not fit did not look missing — it looked like a smaller number. A window holding more rows than one read may return now reports that it cannot be shown, instead of drawing a total that is quietly low.

A database outage no longer costs the hourly sign-in counts it spanned. Each Horizon process now records a running total for the hour rather than a count of what happened since its last write, so the first write after the database comes back restores the figures for the hour in progress — and a write that is retried after an uncertain outcome leaves the same number instead of counting twice. The sign-in list is unchanged: rows from an outage are still not recorded.

A service-group entry the role cannot read says “No access to this layer”. Opening such an entry by its address — /layer/general/risk for a role limited to GENERAL[payments] — said the layer was not found.

A link to a service the page cannot show now says so. A ?service= naming a service outside the role’s grant, or one no longer reporting, silently showed the page’s first service under the link; the page now says it is showing another service instead.

The Service dashboard shows a refused read instead of reading for ever. When the dashboard’s read was refused, the grid stayed on “Reading data …”; it now shows the reason.

Refreshing

Features

Refreshing is one coordinated round. A page used to refresh in pieces, on clocks that drifted apart: the header and the service roster followed the topbar timer, while a dashboard’s widgets and the alarms card each polled on intervals of their own. A refresh is now a single round — header, roster, every widget, the alarms card and the maps — all asking about one time window, and the round is not finished until the last of them lands. They still appear as they arrive, so a large comparison fills in over a second or two; what changed is that they are all answering the same question. The countdown measures the gap between rounds rather than between starts: while the readings are out it reads Refreshing, and it begins again when the last of them lands, so time spent loading is never charged against the interval. On a slow backend the interval genuinely stretches and two rounds can no longer overlap. A round that has started always finishes, even if you switch refreshing off or move to a page that pauses it — the next one simply does not begin. One that runs longer than a minute is given up on, so a single wedged screen cannot stop everything else refreshing.

Auto-refresh has its own on/off, separate from the interval. Turning it off and back on returns to the interval you last chose rather than forgetting it, and off means off everywhere: passing through a page that pauses refreshing and coming back no longer triggers a refresh you had switched off. Switching it on while a page is paused or the tab is in the background saves the setting without refreshing then and there, and the interval menu stays usable on pages that pause refreshing — how often to refresh outlives the page you happen to be on. Pages and overlays pause refreshing independently of each other, so opening the Smartscape hierarchy overlay freezes the background while you pan through it and changing the time range no longer unfreezes it underneath you.

The refresh control says whether anything is actually loading. A download arrow beside the countdown appears while requests are in flight and goes when they land — the icon used to spin whenever auto-refresh was merely enabled, which told you nothing about whether data was arriving. While a round is out the refresh buttons are disabled rather than accepting a click that could only ask for what is already being fetched.

A refresh that fails is recorded where you can go and read it. Failures from the timer collect beside the refresh control, newest five first, with a count of the ones you have not seen — nobody asked for that round, so it waits rather than interrupting. Each entry names the screen, what it was trying to do, the request and the server’s answer; secrets in the URL or the response are removed before it is shown. Failures from something you just did — expanding a node, for instance — appear immediately instead, as a message that pauses while you read it and can be opened for the same detail.

Fixes

Overview widget grids and layer dashboards emptied on every refresh. Both treated a moving time window as a new question, so each tick blanked the grid and replaced it with a loading line — over widgets that were usually about to be filled with the same numbers. They keep the previous values while the next reading is out and replace them in one go when it lands.

The Alarms card on a dashboard refreshed on a clock of its own. It polled every minute regardless of the page’s own cadence, so a dashboard set to refresh every fifteen seconds showed an alarm list up to a minute older than the metrics beside it. It moves with the rest of the page now.

A read you walked away from went on costing OAP work. Navigating away mid-load, or a refresh that gave up, stopped the browser waiting but not the query behind it — so an abandoned page went on costing OAP the whole fan-out, multiplied by the node count on a cluster. Those reads are now cancelled all the way through. Anything that CHANGES something — creating a profiling task, pushing a template — still runs to completion, so a closed tab cannot leave it half applied.

Clicking Refresh against a dead backend appeared to do nothing. The failure was filed away and the screen said nothing, though you had just pressed the button and were watching. A refresh you asked for now answers on screen when part of it fails — once, however many widgets failed — and points at the list beside the control, where the detail is.

One outage was listed several times. Two screens sharing the same reading each recorded it, so the failure list showed one problem as several. Each is recorded once now, and a comparison entity that fails on its own is recorded at all, which it was not before.

A failed read was drawn as a page of zeroes. When a layer read could not reach OAP the answer came back empty, and an overview’s KPIs rendered that as 0 and its service counts as none — indistinguishable on screen from a system that genuinely had nothing running. Overviews and the layer service list now keep the last values they read and report the failure instead. Layer dashboards keep their widgets through the same failure rather than emptying them.

Query cold stage could leave hot and cold answers on screen together. Flipping it re-read each screen on its own, so the page showed both stages while it settled, and a screen that had not caught up yet could file its answer under the wrong one. The whole page is re-read as one round now.

A refresh that gave up left its heaviest requests running. The sixty-second cap cancelled the query but not the request behind it for the layer landing, the dashboard batches and the overview — the browser stopped waiting while OAP finished a fan-out nobody would read, multiplied by the node count on a cluster. Those now stop with the round.

Two reads racing could leave every request an hour wrong. The server timezone and the capability probe were each fetched per request, so a round’s dozen reads probed in parallel; behind a load balancer a fast success could be overwritten by a slower timeout falling back to UTC, and that answer then served every request for the next minute. Each is now read once per expiry and shared.

A brief storage hiccup emptied the sidebar for a minute. A failed service-catalog read replaced the known-good roster with an empty one and cached it for the full minute, so service counts fell to zero and group navigation collapsed long after OAP was healthy. The last good roster is kept and marked stale instead, and a failed read is retried within seconds — for as long as the outage lasts, rather than surviving only the first failure.

A landing metric batch that timed out silently reordered the layer. Its services were left without a value, sorted to the bottom as though idle, and could fall out of the top-N — choosing a different busiest service and a different default — while the response still reported success. Services whose metric could not be READ are now ranked above those that genuinely reported nothing, and the layer header says so on screen — naming how many batches failed — rather than leaving the blanks to be read as zeroes. An overview whose widgets aggregate across services says the same thing, because there a lost batch leaves a total quietly low rather than visibly absent. It keeps saying it for as long as the incomplete reading is on screen, not only on the refresh that hit the timeout — and an incomplete reading is re-read once, behind the values already shown, so a momentary backend failure costs a few seconds of a partial header rather than the rest of the hour.

Topology, deployment and dependency maps

Features

Changing what you are looking at says what it is loading. Picking a different service, endpoint, depth or time range names its target — Loading topology for “checkout”… — instead of leaving the previous answer under the new heading while the next one is fetched.

Fixes

The 3D Infra Map follows the theme: its canvas, tier slabs, rims, cube edges, panels and cards took their colours from a fixed dark palette, so a light theme such as Daybreak showed dark panels with unreadable text. They now come from the theme tokens and change with the theme.

The 3D Infra Map opens framed on the tiers that hold services. A deployment with one populated tier had its slab at the top edge of the view, because the default pose aimed between empty planes; the default and Reset views now frame what is there.

An expansion on the API dependency graph could land in a graph you had left. Nothing stopped you switching endpoints while one was loading, and the branch then arrived in whatever graph had replaced it. Expanding now shows its pending state and holds the endpoint picker and the other expand handles until it lands. An expansion started while an admin preview is open also resolves against the draft being previewed, rather than against the published template the rest of the graph is not showing.

A map that failed its FIRST read showed a loading line for ever. With nothing cached to fall back on there was no way to tell a read still in progress from one that had already failed, so the graph sat on “Reading data…” indefinitely. It now says the read failed and offers Retry.

The maps blanked on every refresh. The Topology, Deployment, API dependency and instance-relationship graphs dropped to a loading line and came back on each tick, taking your zoom, pan and any nodes you had dragged with them. They redraw in place now, and are re-framed only when the question changes — a different layer, service, endpoint, focus, depth or time range. A service appearing or disappearing no longer re-frames the canvas or discards placements.

A failed reading erased the map you were looking at. When a round could not reach OAP the graph was replaced by an empty one, so an empty map could mean either that there was nothing to show or that the read had failed. The previous picture stays and the failure is reported beside it. A layer whose template an administrator has disabled is the deliberate exception — that is an answer rather than a failure, so the map is cleared and says so, instead of telling you to check a backend that is fine.

A failed expansion on the API dependency graph was reported as “no further dependencies”. A read that could not be completed marked the branch exhausted and faded its handle — a claim about your system made from a failure to read it. It now says the expansion failed and leaves the handle live, so the click can simply be repeated.

The API dependency columns re-sorted themselves on every refresh. Rows were ranked by the centre metric, which moves every cycle, so an endpoint you were watching had to be found again after each tick. Which endpoints get a row is still decided live — one that becomes busy enough still appears — but an endpoint already on screen keeps its place and an arrival takes the next free one.

Pods drifted on the Deployment map, and dragging could stop working. A placement was held as an offset from the packed position, so a refresh that added a pod carried yours along with the re-pack; and the map only re-armed its drag handles when a count changed, so a refresh that swapped one instance for another silently left them unbound. Placements are absolute now, and a node on the API dependency graph keeps its place the same way.

Open service and API map took a node on a layer split by service group to the wrong page. From the Topology tab of a group entry, both opened the node’s layer without its group: a service of that layer landed on Layer not found, and a node reporting into several OAP layers opened the first of them — often a self-observability layer such as SO11Y_JAVA_AGENT — so a role limited to a service group was told “No access to this layer” even for its own service. They now open the node’s own group entry, and so do Service → and Open endpoint on the API dependency graph and Open in on the service hierarchy.

A graph the role may not read says so. The Topology, Deployment, API dependency and instance map graphs showed “Could not load …” with a Retry that could never succeed when the read was refused; they now show the reason, such as “You do not have access to this service.”

The service hierarchy no longer blames a missing template for a peer it cannot open. A peer in a layer absent from the reader’s menu was labelled “No Horizon layer template configured”, which is also what a role that may not open that layer saw; the label now names both causes.

The 3D Infra Map placed a layer split by service group as a layer nobody had configured. Each of its entries sat on the tier for unknown layers, in grey, with no traffic chips and outside the logic group that names the layer, and Dashboard setup → 3D Infra Map listed every entry as a layer of its own. Every entry now takes the layer’s tier, colour, metric, icon and group, and the setup page lists the layer once.

The 3D Infra Map left out the hierarchy link between two layers split by service group. A service’s link to its twin in another layer was missing when both layers were split.

The Topology node filter did not show the name of a layer split by service group. It showed the layer key in title case, General, instead of the layer’s name, such as General Service.

The instance map no longer calls a service of the same layer “outside this layer”. A callee of another service group, or one outside the reader’s access, was described that way; the note now says the service is not in the page’s service list.

Dashboard templates

Features

Invalid templates stored on OAP are reported, and never served. Horizon checks every template it reads from OAP — layer dashboards, overviews, alarm pages, translations, the 3D map, the theme and the default time window — and one whose content would break its pages is served to nobody, without affecting anything else. The Dashboard setup rows in the sidebar show a red warning with the count of invalid templates each page owns, and each page lists them in a banner with Show problems (the stored template with every problem underlined in red where it sits, beside what is wrong with each), Download and Disable. Pickers mark them INVALID and gain an Invalid filter. An invalid template opens no editor: it is checked or disabled first, and configured again afterwards from the shipped default or a corrected file; publishing the new one writes into the disabled record and turns it on. Whether a template is valid and whether it is enabled are separate: a disabled record shows DISABLED, and INVALID beside it when its content is invalid. A user who opens an invalid layer’s address and can read layer templates is told why it is missing. See Invalid Stored Templates.

A disabled template can be enabled again, on every Dashboard setup page. Each page lists the templates it owns that are disabled on OAP — a deleted overview dashboard or alarm page, a deleted translation, a disabled layer or setting — collapsed to a count under the page’s banner. Enable turns one back on with the content it holds; an invalid one comes back invalid and is reported again. Deleting a dashboard or an alarm page used to be final from the UI. Reactivate on Layer dashboards does the same, and no longer replaces invalid content with the shipped default.

Import checks a file before staging it. Import on every Dashboard setup page runs the checks Check diff & push runs. A file that fails them is not imported: its problems are shown on the file itself, underlined where they are, and nothing becomes a local draft. A file that is not JSON shows where it stops being JSON.

Cluster Status shows what the template store has actually loaded. A new Dashboard templates pane reports how many layer templates, overviews, alert pages and translation overlays are being served from OAP, and when they were last read — and, when a read fails, the message it failed with, so “unreachable” is no longer the only thing you are told about a 404, a 401 and a timeout alike. Its badge separates a store that cannot be read while your dashboards keep rendering from one that has never been read at all.

A template published elsewhere reaches an open browser on its own. Horizon re-reads the template store on a slow cycle of its own, so a dashboard pushed from another Horizon, from swctl, or from anything else writing the same OAP store appears within about a minute — without anyone reloading the page.

Every MQE field in the layer template editor can now be run where you type it. A run button beside each expression opens a panel that picks a service (and, where the metric needs one, an instance, endpoint or process), picks a time range with the same precision-tabbed picker the top of the page uses (custom ranges included), fires the expression against your OAP and shows exactly what came back — a graph of the series it returned, aligned on one time axis with a legend per label, the entity each ranked row belongs to, and OAP’s own error message when the expression is wrong. Until now the editor fired no queries at all, so the only way to find out whether an expression worked was to push the template, open the layer and look. Every field is covered: the service-list columns, the topology, deployment, process and API-dependency metrics, the dashboard widget expressions and their visibleWhen gates. Relation metrics ask for both a source and a destination, since that is what they measure. A blank service-list column is runnable too — it reports the catalog default that will run in its place. An expression run with no service reads the metric across every service, so it needs inspect:read, the permission Metrics inspect uses.

A record widget’s trace icon opens the store the trace is in. The new recordTrace widget field names the store a record widget’s trace ids belong to, with the store names of traces.sources, and the icon opens the trace in that store’s popout over the page — above the AI assistant’s drawer as well, and from the store it names even on a layer’s TraceQL row of another store; the widget editor sets it under Record trace → Trace store, and the AI assistant’s record figures follow it too. The icon used to open every record’s trace in the native popout, which holds no OTLP trace. Absent, it still opens the layer’s own trace popout, so existing templates are unchanged. The OTLP Virtual Database’s Slow Statements widget now opens its traces from the OTLP store through TraceQL. See Layer Dashboard Templates → Open a record’s trace in the store that holds it.

Fixes

One broken template no longer takes down the sidebar or every dashboard. A layer template stored on OAP without the components section made the sidebar say OAP unreachable — with OAP fine — and drop every layer, including ones with live services; other broken shapes made every layer and overview disappear behind the template-store banner. Such a template now loses only itself.

The 3D map called an invalid configuration an unreachable template store. It now says the configuration is invalid, and drops the advice to check that OAP is reachable.

Reactivate on Layer dashboards, and a new translation saved over a deleted one, left the record disabled. On BanyanDB, OAP’s update changes a record’s content but not whether it is disabled, so the new content was stored and never served. Horizon now re-enables the record and waits until OAP reports it enabled.

A disabled layer came back in the sidebar while the template store was unreachable. The sidebar now keeps hiding disabled and invalid layers through an outage, from the last read of the store.

The AI assistant and MCP agents listed layers the sidebar hides. Their layer list included a disabled layer; it now leaves out disabled and invalid layers, as the sidebar does.

A layer whose stored template names its entity terms under aliases opens again. A layer template pushed to OAP with aliases (the documented spelling) and no slots left the sidebar entry without entity names, and opening that layer showed a blank page; the entry now takes the names from aliases.

A template store that could not be read emptied every dashboard, overview and map. They are rendered from the last successful read now, so a brief outage of OAP’s admin port leaves the console up, with the banner saying how stale it is. Horizon still never substitutes the templates bundled in the release — showing shipped defaults in place of your own configuration would misrepresent what is on screen — so a Horizon that has never read the store still blocks those pages rather than inventing content for them.

The unreachable banner never cleared once the store came back. Nothing re-read the store’s status after the page had loaded, so the warning stayed up — and the pages behind it stayed blocked — however healthy OAP had become. Both recover on their own now.

A layer split by service group listed its sidebar entries as layers of their own. The layer picker of Dashboard setup → Layer dashboards showed GENERAL~, GENERAL~PAYMENTS and so on beside GENERAL, each reported as a layer with no template and counted under Not configured. It lists the layer once.

The layer template editor’s Delete button deleted nothing. OAP keeps every template, so the button disabled a published one — which Reactivate brings back — and discarded a draft never published. It is now named for what it does: Disable, or Discard draft.

A sub page’s id can be any name. The editor refused an id equal to a tab name, such as topology, because pages and tabs shared their addresses; a sub page now has an address of its own.

Changing a widget’s type kept the old type’s settings. In the layer widget editor, a card switched to a line kept its value map, a line switched to a table kept its trace drill, and a record switched to a table kept its trace store, after which the push refused the whole layer: hidden, since the editor shows each setting only on its own type, but stored with the template. Changing the type now keeps only what the new type uses — the title, size, expressions and visibility rule carry over, and the old type’s own settings are cleared.

Opening the Topology or API dependency tab of the layer editor took a layer’s map metrics away. On a layer without a topology or API dependency configuration of its own, the tab wrote an empty one as soon as it was opened, and once saved the map showed no metrics instead of the built-in ones. Opening a tab now writes nothing, and the tab says when the built-in metrics apply.

Ordinary edits in the layer editor left a template the push refused. Removing or renaming a metric of a Deployment role pair left the pair’s primary naming it, and an empty ＋ Condition on an Instance page or a cleared Span, Row span or threshold field was saved as it was. A pair’s primary now follows its metrics, and saving drops an empty condition and treats a cleared number as unset.

The layer editor kept settings it no longer shows. A card’s value map and colours stayed after its format left enum; the configuration of Network Profiling stayed after it was switched off (it now asks first, as the other components do); the menu label of a component switched off stayed; and an Instance or Endpoint grid emptied by adding and removing a widget showed nothing instead of the Service grid. From older templates, 24-column widget coordinates, a free-text visibility rule and the single profiling component are converted or dropped when the template is saved.

A widget’s trace drill could not be cleared once Traces was off. The control can now set it back to none.

A single-series line widget’s label, unit and axis were applied but hidden in the widget editor. They are shown for every line widget, and an empty per-series unit no longer hides the widget’s own unit.

Menu labels of a layer template in the bundled file format showed blank in Layer setup, and the first edit dropped the others. Labels kept under aliases are read as the sidebar reads them.

The widget drawer stopped rendering on a widget with an old free-text visibility rule. Such a rule reads as no rule, as it always did on the dashboard.

Renaming or removing a tier on Dashboard setup → 3D Infra Map broke the configuration. Groups and the tier for unknown layers kept the old tier id, so the push refused the whole map. A rename now carries every reference, removing a tier moves them to the first remaining tier, the last tier cannot be removed, and a duplicate or empty tier id is refused as it is typed. A layer in a logic group is listed under its group’s tier, as the map draws it, and joining a group drops its tier pin; the bundled configuration no longer pins the self-observability layers it groups. The stored pipeline block, which the map never read, is dropped.

Clearing the custom window on Dashboard setup → Global defaults half-applied the save. The theme was saved and the empty window refused. Both sections are now checked before either is written, and an empty or out-of-range window is named under the field.

Overview dashboards

Features

A calendar heatmap widget for overview dashboards. The new calendar-heatmap widget type draws a fixed window of the last 7 to 93 days (30 by default), and lays it out to fit: up to 14 days by the hour, one row per day and one column per hour of the OAP’s clock, so ten days read as 240 hour cells over the calendar days they touch; a longer window by the day, one row per week and one column per weekday, Monday first; a Resolution control in the editor forces either. Each cell is shaded by its value, with the window total in its footer; clicking a cell keeps it picked and reads its date and value out beside the total. A line comparing the total to the text of a well-known book can be turned on per widget. It is the one overview widget that does not follow the time picker: its window is part of the template, so the grid reads the same whatever range the rest of the page is on. Give it a plain per-service metric such as meter_ai_agent_tokens, a unit, and whether to sum or average the layer’s busiest services into each cell; the shades are cut at the quantiles of the days that saw traffic, so one heavy day does not flatten the rest, today’s cell is outlined as the day in progress, and each cell’s tooltip carries its date and value. The grid fits its card: cells stretch to the width and height available, a card too narrow for the grid’s long axis turns it the other way, and a day cell wide enough shows its day number. It refreshes with the page and keeps its cells while it reads. Add one from Dashboard setup → Overview templates → Add widget; the drawer edits its metric, unit, aggregation, window and comparison line. See Overview Widgets.

A ranking widget, and range totals for page-side KPIs. The new ranking widget type lists a layer’s services ranked by one per-service metric over the picked time range, busiest first, each with its value and a bar against the top one, in as many columns as its height needs past five rows; it lists up to twenty. Page-side KPI rows and ranking widgets can also ask for a range total (rangeTotal): a service’s buckets are summed over the picked range instead of averaged, so a counter such as tokens reads as the total for the range rather than a value per interval. Add one from Dashboard setup → Overview templates → Add widget; the drawer edits its metric, unit, row count and the range-total switch, and the same switch sits on each KPI row of a page-side tile.

A bundled AI Agent Overview page. When OAP reports the AI_AGENT layer, the sidebar gains an AI Agent Overview with the agents reporting on the layer and their tokens over the picked time range, in total, by type (input, output, read from cache, written to cache), the part spent by subagents, and their cache read share, a Top 20 agents list ranking them by that total in two columns, an Hourly tokens grid of the last 10 days by the hour, and a Daily tokens calendar heatmap of the last 30 days, the busiest agents summed (the footer says when a layer has more than one read covers). It carries only the metrics every metrics source produces; the exporter-only families (cost, sessions, lines of code) stay on the layer’s own dashboards. Like every overview it is editable under Dashboard setup → Overview templates. See AI Agent Overview.

Fixes

A widget can be placed on a layer that has no services yet. The layer list in the overview template editor offered only the layers with services at that moment, plus those the dashboard already used. It now offers every layer Horizon knows, so a dashboard can be prepared before a layer’s first service reports. A layer split by service group is offered with each group as written — it was offered upper-cased (GENERAL~PAYMENTS), so a widget bound to a group found no services.

A KPI tile, ranking or metric widget bound to one service group opens that group’s entry. Its click-through lower-cased the whole stored layer, group included, so a group spelled with capitals opened an entry that does not exist.

An overview disappeared from the sidebar when every layer it covers is split by service group. An overview is listed while one of its layers is in the reader’s menu, and a split layer’s entries did not count as that layer.

An overview’s topology widget on one service group of a layer lost the group’s letter case. A group whose name has upper-case letters read another group, or none: OAP compares group names exactly.

The Overview templates editor left settings the push refused or the page never used. A cleared Columns, Width, Height, Window, Rows, Row limit, Top-N services or Max field was saved as text and refused; Rank by stayed on a KPI row’s position after rows were moved or removed; a KPI switched to service-count kept its query, which was still evaluated layer-wide; a Metric widget showed KPI rows it never draws; a KPI switched from progress bar kept its Max, which still scaled a % value; switching aggregation to server-side kept the page-side rows limit, aggregation and Range total; and a cleared Tip or Description was stored empty. Each edit now updates or drops what it affects, and a save writes only what the dashboard’s settings use. Saving also drops the dashboard’s own layers list, which the editor never showed: the sidebar lists an overview by the layers its widgets use.

Operating Horizon

Fixes

The sidebar uses less left padding and makes layer grouping clearer. AI Agents, Browser and other standalone layers align with group headings. A group with only one visible layer shows that layer directly at the same level; groups with multiple visible layers can be folded and unfolded by clicking their heading. Group headings have the same row height as layer entries. Navigating to a layer opens its group, while a menu refresh preserves groups you folded. Satellite and other direct layer links under Platform monitoring no longer show an extra service-count badge.

Data retention now reads the same on every storage backend, and says which setting governs each figure. The page used to mirror OAP’s wire shape — a Records / Metrics split, a Minute / Hour / Day trio, and a class called Normal — which on ElasticSearch, MySQL, PostgreSQL, H2 and TiDB meant nine numbers that were really two, presented as nine knobs you could turn apart. Those backends now show the five things retention is actually decided for — Metadata, Metrics, Logs, Traces and Others — each naming core.recordDataTTL or core.metricsDataTTL beneath it, with a note saying the two settings are all there is. BanyanDB shows the same five in the same order, expanding Metrics into its minute / hour / day groups and adding Zipkin traces and Browser error logs, because there each one is a separate bydb.yml group that genuinely can be tuned apart.

The retention chart hid data classes whose retention happened to match. Classes sharing a figure were merged into a single All records (5) bar — which reads as a fact about the backend when it is only a fact about today’s configuration, and left an operator unable to see what they could change. Every class now has its own row. Others is the one row named for a group rather than a data type: BanyanDB keeps alarms, events, sampled traces, top-N and every profiling record together in records, so they share one retention and no name for a single one of them would be true.

On the live debugger’s Capture history page, an active session’s start time ran into the rule name beside it. The left column of an Active row carries the DSL badge, the LIVE marker and the start time, and had no room for all three, so the time overlapped the catalog · file · rule text of the same row. The time now sits under the badges.

Copy buttons work when Horizon is served over plain HTTP. A browser offers its clipboard to a page only over HTTPS or on localhost, so on a Horizon opened by an intranet address over plain HTTP most copy buttons did nothing and said nothing: the trace id and link copies on the trace detail and popout of the Traces, Zipkin Traces, TraceQL - Native, TraceQL - Zipkin and OTLP Traces tabs, the Copy of a log and of an evaluation record in their popouts, the click-to-copy statement on the Slow Database Statements, Slow Statements and Slow Commands cards, the Copy of a long value’s full-value dialog, the Copy JSON of an MQE result, and the copy link of a widget’s entities on the Inspect page. They now copy through the browser’s older copy command when the clipboard is not offered, as the layer’s share button and the conversation page already did, and say copied only once the text was copied: the share button and the slow-statement cards said it even when nothing was.

Traces, logs and events

Features

Cluster status reports every trace API in one pane. Operate → Cluster’s Zipkin pane becomes Trace APIs, listing every endpoint behind a layer’s trace rows in one table — API, source, state, endpoint — with a row for the Zipkin v2 REST endpoint and one per TraceQL datasource, SkyWalking-native spans and Zipkin spans, each showing whether it answered and the Tempo API version it reported, plus the error and the exact URL tried when one is configured but silent. A TraceQL datasource with no URL reads not configured rather than broken, and native traces are not listed because the GraphQL pane above already answers for them. See Cluster Status.

Traces can now be queried in TraceQL, over either trace store. OAP can answer for its traces in Grafana Tempo’s query language, and a layer can expose that as its own sidebar row — TraceQL - Native over the SkyWalking-native spans, TraceQL - Zipkin over the Zipkin spans, either or both, renamable per layer. Point Horizon at the service with oap.traceql.url, http://127.0.0.1:3200 by default, and each datasource’s context path defaults to OAP’s own — nativePath, zipkinPath, otlpPath. OAP ships its traceQL module off, so until it is enabled a TraceQL row says it cannot be reached rather than searching something else; url: '' in the configuration file turns TraceQL off. The page is laid out like the native Traces tab — conditions and duration distribution above, results below, the trace opening beside the list, the same Default / Tree / Statistics views and copy buttons, the same dot-picking and box-brushing on the distribution to narrow the list without re-querying, and a tree that pans and zooms — and queries three ways: a Builder whose rows are filled from the store (service, instance or peer service, span name, status, duration range, span tags), a TraceQL editor with highlighting, a schema panel of the fields and tags that store reports, and autocomplete for fields and tag values, and Trace ID for one or more ids. Each result carries its status in the list, which the Tempo search response cannot report on its own, so Horizon settles it from the markers a span carries — an http.status_code, an error tag, an exception type, an RPC or OpenTelemetry status code — and then by reading whatever is still undecided once the list is on screen; the header counts those down. Those markers are read on the spans themselves too, because OAP’s Zipkin converter leaves the OTLP status unset on every span it produces: the waterfall dots, the tree nodes, the span detail’s status and the Statistics error counts all fall back to them where the status field says nothing, so a Zipkin trace of plain 200s reads as succeeded and one carrying a 500 is red on the span that returned it. A span’s own OTLP status still wins where it set one, a span that reports nothing either way stays unset, and a trace of only such spans stays unknown rather than being called a success. A TraceQL trace is shown as the OpenTelemetry trace it is — opaque span ids, span kind, three-valued status, resource attributes apart from span attributes, instrumentation scope — rather than folded into SkyWalking’s span shape, so the tab shows the fields the protocol actually defines; the one reading it adds is the span kind, where a server or consumer span reads Entry and a client or producer span Exit, as on every other trace surface, with OTLP’s own word beside it, and the same direction glyphs the native waterfall draws — an arrow into a bar for an entry, out of one for an exit, a ring for a local span, and an envelope for the two messaging kinds. The tab owns its time range — presets or an absolute Custom range… — and the topbar picker is disabled while you are on it. The editor checks the query against the store’s schema — the intrinsics, that store’s resource attributes, and the span tags it reported for the window — and warns when a query names a field the schema cannot place: an attribute written without a scope (http.method rather than span.http.method or .http.method), a bare name that is not an intrinsic, a resource attribute belonging to the other store, a status outside ok / error / unset, an expression that is not one complete spanset, or a condition left without a value. Every finding is a warning and the query always runs; what a backend does with a form it parsed is the backend’s business. A query the source refuses outright carries its message where the results would be. The Schema reference panel floats over the page and closes on a click outside it, on Escape, or when the query runs. A layer can also carry a regular expression narrowing its TraceQL service picker, because the Tempo API has no notion of a layer and lists every service of the store; result rows are never filtered by it. Needs traces:read. See Traces, OAP Connection and Layer Dashboard Templates.

Traces can be queried in TraceQL over OAP’s natively stored OTLP spans. OAP can now keep OTLP traces exactly as they arrive instead of converting them to Zipkin, and serve them on a TraceQL datasource of their own. A layer can expose that as an OTLP Traces row: point Horizon at it with oap.traceql.otlpPath, tick the store in Dashboard setup → Layer dashboards → (layer) → Trace stores, and it queries and renders like the other TraceQL rows. Because nothing is converted, this row shows what the other two cannot: an attribute keeps its own type, so a numeric http.response.status_code reads as the number it is rather than going missing; events, links, status messages and the instrumentation scope are the ones the SDK exported; and the query can use the kind intrinsic, resource.service.instance.id and Tempo’s span:-prefixed field names, which the Zipkin and SkyWalking stores have no column to match. Each row’s Schema reference lists what that store can filter, so the difference is visible while you write the query. Which of the TraceQL rows holds a deployment’s OTLP traces depends on how its OAP was told to keep them; only one of them will. OAP 11.1.0 keeps them natively by default, so the bundled Mesh, Mesh control plane, Mesh data plane, Kubernetes and Kubernetes service templates list OTLP Traces beside Zipkin Traces: a mesh can send its spans either way. Of these, only Mesh has its Traces tab on. See Traces, OAP Connection and Layer Dashboard Templates.

A trace list shows failures without opening every trace. A TraceQL search response can now carry each span’s status, so a result row is marked as failed or healthy the moment the list lands, instead of the page reading every undecided trace one at a time afterwards. Rows an OAP does not answer for are still read that way — the header counts them down as before — so an older OAP behaves exactly as it did.

Each trace store a layer names gets its own sidebar row, with a name you choose. A layer exposing native and TraceQL traces shows both rows rather than one tab that switches store under you, and Dashboard setup → Layer dashboards → (layer) → Trace stores is where they are ticked, renamed, and — for the TraceQL rows — given the service-picker filter. Renaming a row changes only its label; links and stored menu arrangements name the row itself, so they keep working.

traces.source (native / zipkin / both) becomes traces.sources, a list naming any of native, zipkin, traceql-native and traceql-zipkin. No template has to change for this. One that still carries the old enum is read as the stores it named — both as native plus Zipkin — and one that says nothing at all is read as the native store, which is the row it has always shown. Both are resolved when the editor opens, so the boxes show what the layer actually serves; change one and save, and the resolved set is written out as an explicit sources that from then on says so directly. A layer with no trace rows is spelled "sources": [], which is what unticking every box writes.

A trace can be looked up by its id. The Traces tab, native and Zipkin, and Trace inspect gain a Filter / Trace ID switch at the top of the query form. Trace ID trades the conditions for an id field and a time range that starts at No time range: Run query reads the id within the range you pick — or, with none, wherever the hot and warm stages keep it — with no service and no other condition, and refuses to run without an id. Switching back finds the filter as you left it. The Zipkin side takes several ids at once, each locked in with Enter, and names the ids that matched nothing. This replaces the Zipkin tab’s Open trace ID field, which Run query ignored and Enter turned into a popout, and gives Trace inspect’s Zipkin mode the lookup by id it never had. With no time range the cold stage is not read — pick a range with Cold on for a trace that old — and on BanyanDB with OAP 11.0.0 and earlier the lookup covers the last 24 hours only.

A span’s long tag values are cut, and open in full. A tag whose value runs past a line — a SQL statement, a stack frame, a serialized payload — previews its first 100 characters on a single line, with a full button beside it; the button opens the whole value in a scrollable dialog with its character count and a Copy that takes all of it, never the visible part. Line breaks become spaces in the preview, so a pretty-printed payload — an LLM request body, a JSON document — previews its content rather than the [ it opens on. This is on the span detail of the native Traces tab and of the TraceQL tabs alike.

Trace queries run only from Run query. On the Traces tab, native and Zipkin, and on Trace inspect, neither switching modes nor pressing Enter runs a query — Enter in an id field locks the id in, and Enter in the Zipkin tab’s Annotations or custom-range fields, which used to run the query, now does nothing.

Trace inspect’s Zipkin annotation query suggests keys and values, as the Zipkin Traces tab’s does: the keys OAP is configured to index, and after key=, that key’s values.

Long logs can be read in the list, without opening a popout for each one (apache/skywalking#14118). The log stream on the Logs tab, Log inspect and the AI assistant’s log blocks gains an Inline / Pop out switch for what a click on a row does. Inline, the new default, expands the row in place — the service, instance, endpoint and trace id, a line of its tags, and the whole content with JSON indented — and several rows can be open at once to compare them. One click anywhere on an open row folds it again; selecting text in a row to copy it does not, and a new query or page closes the open rows. An expanded row’s Pop out button opens the full-payload popout, with Copy and the tag table. Pop out keeps the previous behaviour: every click opens the popout. The choice is remembered in the browser and applies to every log stream.

Fixes

The Dashboard templates pane marks readonly mode in yellow, as the module table beside it always has. The same state was drawn green there and yellow in the table, so one of the two said a store Horizon is deliberately not reading looks healthy.

The three trace details now read at one size. The native, Zipkin and TraceQL trace details had drifted apart in type: the trace id, the STARTED / DURATION / SPANS / SERVICES figures, the service chips, the span names and the row heights each differed between them, and the native detail’s Default / Tree / Statistics switch rendered in the browser’s default face rather than the interface font. All three are now measured against the native detail and match it, and the TraceQL waterfall draws its span name on the duration bar as the other two do, with the same depth guides.

The Zipkin Traces tab now disables the topbar time picker, as the native one always has. The tab has owned its own Time range since it shipped, but the global picker stayed live beside it and auto-refresh kept ticking, so the topbar advertised a window that nothing on the page was reading.

A trace’s age reads as days and hours. The trace list on the Traces tab, native and Zipkin, wrote an age past a day as a count of hours, 288h ago; it now reads 12d 0h ago, hours and minutes under a day, minutes under an hour. The Auth status admin page’s time of the last LDAP probe reads the same way.

Annotation suggestions on the Zipkin Traces tab vanished once the query held more than one term. They completed only the last term, while the browser matches and replaces the whole Annotations field, so after error nothing was offered, and picking a value would have dropped error. Each suggestion now carries the terms already typed. Trace inspect’s Zipkin annotation query, which gains the same suggestions in this release, behaves the same way.

Switching the Zipkin service on Trace inspect kept the previous service’s span name and remote service. They were cleared only when the service field was emptied, so going from one service straight to another ran the new service with the old one’s picks and matched nothing. They are cleared on every switch now, as on the Zipkin Traces tab.

Running a new query left the previous trace open on the Traces tab. With a trace open beside the result list, clicking Run query again refreshed the list but kept the old waterfall on screen — so a query that matched nothing still showed a trace, as though it were the answer. A new run now closes the open trace and clears any pick on the Distribution chart, on the native and Zipkin tabs alike, and a list waiting on its first rows says Reading data… instead of reporting an empty window before the answer has arrived.

The Duration distribution was a different size on every trace tab. The chart is drawn beside the conditions that produced it and took whatever height those conditions happened to need, so the native, Zipkin and TraceQL tabs each settled somewhere different, and a Trace ID lookup — two fields — squeezed it to a strip too short to read a duration off. All three tabs now reserve the same height in every query mode. The TraceQL Schema reference opens over the page instead of pushing the card taller — and closes again on a click outside it, on Escape, or when the query runs — so consulting it while writing a query no longer resizes the chart beside it. The TraceQL Limit starts at 30, as the other two tabs do.

A refused TraceQL query was reported twice, in two different places, and one of them contradicted the other. The backend’s own sentence appeared both under the editor and as the result list’s empty state, while the warning above it still read “so the query is not run” — left over from when the editor blocked such queries, which it no longer does. The backend’s answer is now printed once, in the list it stands in for, and a refusal no longer follows you into the Builder or Trace ID view, where it was a verdict on a query those modes are not running. A TraceQL datasource that failed its reachability probe is now re-probed until it answers, instead of being called unreachable for a minute after it came back.

The TraceQL editor judged OAP’s behaviour instead of checking the schema. It warned about a dozen forms — !=, regular expressions, a second spanset, OR, negation, pipelines, duration= and more — on the grounds of what one OAP build was measured to do with each, narrated the mechanism in the message (“parsed and then dropped”), coloured several of them as errors, and had the editor render valid TraceQL operators, > and < among them, in the invalid-syntax colour. None of that is Horizon’s to say: which parts of the language a backend applies is a property of that build, and a query it accepts but answers wrongly is a bug to report upstream. The editor now checks only what it can know — that the query names fields the store’s schema has, and that the expression is a complete spanset — and every finding is a warning in amber, in the notice and in the editor’s gutter alike. Clicking a field in the Schema reference now inserts it spaced as a query is read, span.http.method = "" rather than span.http.method="".

The trace rows are named after the API that answers them. OTel & Zipkin Traces is now Zipkin Traces, and the two TraceQL rows read TraceQL - Native and TraceQL - Zipkin rather than Traces (TraceQL) and OTel & Zipkin (TraceQL). A layer that renamed a row keeps its own name; only the built-in defaults change. Trace stores in the layer editor calls its first field Menu name, which is what it sets, and replaces the free-text Regex flags box with an Ignore case checkbox — i was the only flag worth setting there, and g or y would have carried state from one service name to the next.

A TraceQL service filter that could hang the server is refused. Some regular expressions take exponentially long to match, and this one is run against every service name the store reports — a pattern like (a+)+ or ^a*a*a*a*a*a*a*a*b$ could stall Horizon for everyone. Two shapes are now refused before they run: a repeat applied to a group that itself repeats or branches, and a second variable-length repeat anywhere in the pattern. The service picker then keeps showing every service, as it already did for a pattern that will not compile, and Trace stores in the layer editor says which of the two reasons applies. [a-z]+::.* is refused along with them — the match is a search, so ^[a-z]+:: is what it was trying to say.

A TraceQL trace link and a native one fought over the same URL parameter. A shared TraceQL link used ?traceId=, which is the native and Zipkin popouts’ parameter, so a trace link elsewhere on a TraceQL page — from the AI panel’s log rows, say — was read as a TraceQL id and queried through the Tempo API: the wrong trace, in the wrong renderer, with the timestamp hint an old id needs thrown away. TraceQL links now carry ?traceqlId=, the two no longer collide, and closing the overlay clears only its own parameter instead of the whole query.

On a TraceQL trace, a Zipkin client and server span sharing one span id collapsed into one row. Zipkin gives both records the same id, so the waterfall and the tree drew only one of them — the header counted a span the timeline did not show, selecting either highlighted both, and a reordered pair could leave a stale row behind. Both views now draw every record.

On the TraceQL rows, a trace’s status could be read from the wrong window. Changing only the To end of a custom range and running again within half a minute answered from the previous, narrower read — so a trace that failed inside the widened range was still listed as successful.

A span’s events told you when and what, and hid everything they carried. An event on a TraceQL span — a Zipkin annotation, a logged exception — drew its name with the timestamp pushed to the far right of the row, and its attributes in a block that ran together with the next event’s. An event now reads down the column as a sequence: the time first, the name after it, then an attributes label and the key/value pairs indented under it, each value with the same 100-character preview, full dialog and Copy that a span’s own attributes have. A long payload on an event is as long as one on a span.

A short trace opened as a fragment, and the span dialog crossed its edges. A TraceQL trace popout took whatever height its spans needed, so a trace of three spans was a few rows floating in the middle of a dimmed page. It now has a floor of 640px, yielding to a shorter window rather than overflowing it. The span detail opening over it was centred and up to 82% of the window tall — taller than the card beneath it — so the two sets of edges crossed and read as one crowded shape; it now sits inset within that card on every side.

A Zipkin span’s long tag values had nowhere to go. The Zipkin span detail printed every tag in full, so one long value — a SQL statement, a serialized payload, a URL with a query string — filled the dialog and pushed the rest of the tags out of view. Its tags now preview their first 100 characters with a full button beside them, opening the whole value in a scrollable dialog with its character count and a Copy that takes all of it, exactly as the native and TraceQL span details already did — on the detail beside the result list and on the view a copied trace link opens alike.

A Zipkin trace’s error state was decided by two different rules, and they disagreed on screen. The result row counted a span as failed when it carried an error tag, an http.status_code beginning with 5, or otel.status_code=ERROR, while the waterfall and span detail below it looked only for the error tag. So a span returning 503 with no error tag made the row read ERR over a waterfall in which nothing was marked, a 404 was a failure on neither although the TraceQL - Zipkin row over those very spans called it one, and error="false" — a span reporting success — was read as a failure by both, because the test was for the tag’s presence rather than its value. An empty error still counts as a failure, which is what Zipkin means by it: the tag’s presence is the signal and its value is only the message. All of them now read one shared table, so the row, the waterfall, the span detail, the view a copied trace link opens and the TraceQL rows agree about the same span. See How a span’s status is decided.

Horizon connects to two OAP endpoints, not three. The Metrics inspect page used to resolve its own endpoint for MQE expressions — an optional oap.mqe host/port override, and, when only half of it was given, a discovery step that read OAP’s configuration dump to find which listener served the query port. That endpoint was never a third surface: executing an MQE expression is execExpression in the query protocol, which oap.queryUrl already serves. oap.mqe and HORIZON_OAP_MQE are gone, and the page uses the configured query URL like every other query. A deployment that set the override should point oap.queryUrl at that address instead.

Network profiling’s edge metrics are in the layer template, and a metric says which side it describes. The process-relation panel under a process→process call was drawn from a built-in set of sixteen metrics that no template named — so Dashboard setup → Layer dashboards → (layer) → Network profiling showed an empty configuration for a panel that was rendering. The Mesh and Kubernetes Service layers now carry the full set in their templates: 38 metrics, each with its unit, the TCP families with their connect, close, retransmit, drop, RTT and percentile members, and the HTTP/1.x request and response families that were missing entirely. A metric that answers with several labelled results — a percentile’s ranks, a status-code metric’s codes — draws them all, so the response-code chart names the codes rather than showing one anonymous line. Each metric now names the end of the conversation it describes rather than having it read off its name, which is what let the HTTP/1.x families exist at all — they describe the exchange and belong to neither end, and the panel gives them their own column. A template written as the earlier two side-lists is read as before and rewritten in the new form the first time it is edited.

Every row in the layer menu opens the configuration that governs it. In Layer dashboards, a row whose name the editor did not know was drawn as its raw path — traceql-otlp-trace, evaluation-record, conversations — and several rows could not be clicked at all. The five trace rows now open the Trace stores tab that configures all of them; Pod logs, Continuous profiling, pprof, Evaluation records, Browser logs and Conversations were components with no tab at all and now have one; and Trace, eBPF and Async profiling had a tab the row above was never wired to. The scope tabs also stop title-casing what they are given, so eBPF profiling is no longer rendered as EBPF Profiling.

The Logs tab re-read its window when you came back to the browser tab. Switching to another tab and back after 15 seconds read the stream and the Levels counts again, for a window moved up to the present, so the page you were reading was replaced and the rows you had expanded closed. As the page already promised, the stream now reads only on Run query and the pager.

Log JSON and YAML were faint on the Daybreak theme. The log detail popout drew a JSON payload in cyan and a YAML one in yellow, on a near-white background, and the stream’s YAML chip used the same yellow. The payload is now drawn in the theme’s text colour, and the chip in the theme’s warning colour.

A log row with a service group took three lines. On the Logs tab and Log inspect, a service name with a group prefix (mesh-svr::frontend.sample-services) wrapped inside its column, so every row of a mesh service stood three lines tall. It now stays on one line and ends with …; the full name shows on hover, in the expanded row and in the popout.

Profiling

Fixes

The Continuous profiling service picker showed no armed policies on a layer split by service group. On a group entry’s tab it asked for the policies of a layer that does not exist, so no service showed what it had armed and filtering the picker by armed kind matched nothing. It now reads that group’s services.

Alarms

Features

Named alarm pages. Besides Alarms, a deployment can now set up any number of named alarm pages — for example one per team — each with its own pinned tiles and starting window, shown as its own row in the sidebar and opened at /alarms/<id>. A named page lists only the alarms its pins cover, among those the reader’s alarms:read allows, and its filter row offers only its own layers and the services its pins cover; Alarms keeps listing every alarm the reader may read, with Other for the rest. A named page appears in the sidebar for a role that reaches at least one of its pins, and the pins a role cannot reach are left out of its view. Pages are added, edited and deleted at Dashboard setup → Alarm pages. See Alarm Pages.

A pinned tile can name service groups. A pin is written like a role’s layer limit — GENERAL, or GENERAL[payments, risk] for the GENERAL services of those groups — and its tile counts the alarms of those services, a relation counting under the group at either end. The Alarm pages setup suggests the groups of the layer’s services the editing role reads.

Fixes

A deleted alarm page’s id can be given to a new page. It stayed reserved, because a page saved into the deleted page’s record was stored and never served; Horizon now turns that record back on, so the id is free again and the new page takes over its address.

Instance, endpoint and relation alarms land in their layers, not under Other. The Alarms page placed an alarm by reading a service name out of the alarm’s display name, which works only for a service’s own alarms, so an instance’s (checkout-1 of payments::checkout), an endpoint’s and every relation’s alarm went to Other. An alarm now belongs to the layers of the services it concerns: the service that owns the instance or endpoint, and both ends of a relation. A service in several layers counts under each of them, so the layer tiles can add up to more than Active. OAP reports a relation’s destination by name only, so when the name fits more than one service (a normal and a conjectured service can share one), the alarm counts only under the layers those services have in common.

The Other tile can be clicked. It lists the alarms in no pinned layer, including those no known service owns; a tab above the list picks one unpinned layer.

Unpinned layers are tabs above the list, no longer a row of chips beside the tiles. The chips repeated the tabs, and a layer pinned only for some service groups still got a chip of its own; now a layer any tile names has no separate tab, and its other alarms count under Other.

Filtering by a service lists every alarm about it. The filter asked OAP for the service’s own alarms, which leaves out those of its instances and endpoints and of their relations, although the list and the tiles count them under the service. It now lists every alarm the service is at either end of.

Every rule and every relation is its own incident. Two rules firing on one service, or two relations from the same source, were merged into one row and counted once, on the page and in the top bar’s alarm count, and clicking one of two such rows could open the other in the detail panel.

The detail panel stays on the page. A long alarm message widened the alarm list and pushed the detail panel beside it past the right edge of the page; the list now keeps its share, and long messages and entity names end in “…”.

Filtering by a layer finds the alarms the page counts under it. The filter asked OAP for the layer, which OAP stores once per alarm (the entity’s first layer, or none when it had not resolved one yet), so a layer could come back empty, or miss alarms its tile counted, and a layer together with a service could hide that service’s alarms. A layer now selects the alarms of its services, the way its tile counts them, reading as far into the window as its page needs (up to the window’s first 5,000 alarms), and beside a service the service alone decides.

A layer filter on the Alarms page finds that layer’s alarms however many other alarms come first. It reads as far into the window as its page needs, up to the window’s first 5,000 alarms. On BanyanDB, OAP 11.0.0 and later return nothing for any page after the first of an alarm list read without a service, so every read now starts at the window’s first alarm, and /api/alarms answers page 2 and later correctly there too.

The 3D map marks alarmed services on a layer split by service group. A service on such a layer never turned red, because the map looked its alarms up under the group’s sidebar entry instead of the layer.

A failed alarm read says so. When the alarms could not be read, the Alarms page and the dashboard Alarms widget showed “No alarms in the current window” and an Active count of 0. They now say the read failed and why, and the tiles show no count.

Adding a pin on the Alarm pages setup offers every layer, not only those with services. It listed only the layers with services at that moment, so a layer that was quiet when the page was set up could not be pinned, and a layer split by service group was offered once per group under a key no alarm matches. It now lists every layer the editing role sees in the sidebar, whether or not it has services, each once.

Languages

Fixes

Date pickers follow Horizon’s language. Every From / To field was a native datetime-local control, which browsers draw in their OWN interface language: a Chinese browser rendered 年 / 月 / 日 spinners and a Chinese calendar over an English console, a German one dd.mm.yyyy, and neither the page nor the language picker could reach it. All of them — on Traces, Zipkin traces, TraceQL, Logs, Browser errors, Evaluation records, Alarms, Events, Trace inspect, Log inspect, the audit and token-usage pages, the two profiling task dialogs, and the shared time-range menu — are now Horizon’s own field: the text reads YYYY-MM-DD HH:mm in every language, and the calendar’s month and weekday names come from the language you picked in Horizon, starting the week where that language starts it — Sunday for English, Japanese, Korean and Portuguese, Monday for German, Spanish, French and Chinese. The calendar also opens over the page instead of being cut off by the card it sits in.

A cleared English text kept its old translation. Clearing a widget’s tip or title in English left every other language showing the translation of the old text; an empty English text now stays empty in every language.

Look and feel

Fixes

The topbar’s time-range label pushed the bar out of shape on a narrow window. This page uses its own time range is the longest label the chip carries, and on a triage page in a narrow window it grew past its share and shoved the rest of the topbar along. It now shortens with an ellipsis, and the whole sentence stays in the button’s tooltip.

On a light theme the browser still drew its controls dark. color-scheme was declared once as dark and never turned back for Daybreak, so the calendar icon was white on a white field, native dropdowns and scrollbars stayed dark, and the focus ring was the dark-mode one. It now follows the theme’s own appearance, so any light theme is covered.

The tag suggestions were cut in half by the card around them. On Traces, Logs, Evaluation records and both inspect pages, the tag autocomplete list was clipped at the conditions card’s edge — six suggestions, three reachable — with no way to scroll to the rest. It now opens over the page, as the service picker already did.

One Escape closed every open box at once. Each dismissible thing — a dialog, a popout, a floating panel, a picker’s suggestion list — listened for Escape on its own, so a keypress reached all of them and they all acted on it. Escape now closes one: the innermost, meaning the one opened most recently, so a stack unwinds a layer per keypress. The Traces, Zipkin traces and Trace inspect pages went the other way about it and wrote out the order they expected — span dialog first, then the trace — which worked until something opened above the span dialog that they had no way to know about: opening a long attribute’s full dialog over a span detail and pressing Escape closed the span detail underneath it and left the value hanging. They now say what they have open and let the same rule answer, so a value popout over a span detail over a trace popout takes three keypresses and unwinds in that order. Escape inside a type-ahead or a tag field still clears or closes that field alone and goes no further.

Query cold stage

Features

Cold is asked for only where cold data is kept. Traces, logs and metrics are the classes a deployment is advised to age into cold storage, and they are now the only ones Horizon sends the flag for. Alarms, the instance and endpoint pickers, events and everything from profiling stay hot — they are small, and they are the first things you reach for during an incident, so a Cold toggle emptying them bought nothing. Those pages keep answering while Cold is on.

Fixes

Zipkin traces ignored the Query cold stage toggle. The Zipkin Traces tab and its trace detail read the hot stage whatever the topbar toggle said, and said nothing about it. Both now send the toggle to OAP with the query, so with it on a Zipkin search reads the cold stage like a native one; a detail opened from the list is looked up in the window the list was read with, since a cold trace is older than the day OAP would otherwise search.

The cold-stage warning told you to pick a window that does not exist. On a deployment with no cold stage configured — the BanyanDB default — it still said “pick a window older than N days”, which cannot work, and following it moved the range out of hot+warm so the warning hid itself: a blank page with the one sentence explaining it gone. It now says plainly that no cold stage is configured and stays up until you turn Cold off. Where cold IS configured, the suggested window is the one that clears the deepest class rather than the shallowest — it used to name the records boundary (3 days by default) while every metric widget needed 7, so following it brought traces back and left the metrics exactly as empty. Both the warning and its advice are now decided by the classes Horizon actually reads under Cold — traces, logs and metrics. Alarms, events and profiling retention no longer count: they are usually kept for the shortest time of anything, and reading them here hid the warning for exactly the windows an operator turning Cold on is most likely to be looking at.

Dashboards

Features

The layer header’s KPIs describe one completed hour, and say which one. They used to be read over whatever window the time picker held, which meant reading the sort metric for every service in the layer on every refresh — hundreds of requests on a large layer, repeated every thirty seconds, for figures that only move by the hour. The header now reads one completed hour and holds it, and the hour it covers is named above the table (09:00–10:00) so the numbers are never older than they appear. Ten minutes are left for the backend to finish aggregating that hour, so the header moves on shortly after ten past. While the next hour is being read the previous one stays on screen with a star against each figure, instead of the table emptying. A deployment too new to have any completed hour shows the hour in progress instead — a finished hour is preferred to it whenever one is available. Whenever no hour can answer at all, the header reads the time picker’s window exactly as it always did: on a deployment young enough that the backend has written no hour-level figures yet (they are aggregated on a longer cycle than minute ones), and on the first visit to a large layer while the hour is still being read. Either way the page opens with real numbers rather than a table of dashes. Everything below the header — the trend lines, the dashboards, the maps — still follows the time picker as before.

The BanyanDB layer has a Trace Sampling page. BanyanDB 0.11 can run a chain of sampler plugins during trace merge and finalization, dropping whole traces after their fragments have landed — tail sampling inside the storage tier. The new page is an extension page under Cluster, with its own sidebar row, and it reads the meter_banyandb_trace_sampling_* families OAP collects from the plugin chain: the committed drop ratio and the trace outcomes behind it, per-plugin Decide rate and latency, chain batching, what the first-party samplers proposed by rule, every fail-open guard and drop-set capacity counter, finalization state, and the host’s bounds on a plugin’s own telemetry. It is cluster-wide with the storage group as a series rather than a per-group dashboard, so one page charts every group at once.

The Envoy AI Gateway layer is now called Agent Router. Envoy AI Gateway joined the Agentic AI Foundation as Agent Router, with the same code and APIs, so the bundled layer template now shows Agent Router in the sidebar and Agent Routers as its service label, in every language. Only the display names changed: the layer key ENVOY_AI_GATEWAY, the meter_envoy_ai_gw_* metrics and every widget stay as they were. An OAP that already stores this layer’s template and translations keeps the old name; to adopt the new one, open Dashboard setup → Layer dashboards → Agent Router (listed as Envoy AI Gateway until then), choose Reset to → Bundled and Check diff & push, then do the same for each language on the Translations page. See Agent Router.

The Linux and Windows layers read OpenTelemetry host metrics and show processes. Both layers now take host metrics from the OpenTelemetry Collector hostmetrics receiver as well as from node-exporter, Telegraf or windows_exporter, and each host gains a Processes tab fed by the Collector’s process scraper. The Host dashboard opens with status cards such as CPU Across Cores, Memory Used, and on OpenTelemetry hosts Normalized CPU (0–100% of the whole host) and Logical CPUs, and adds OpenTelemetry-only charts: a Normalized CPU chart on Linux; on Windows CPU Load, Allocated Handles, Windows Pagefile (MB) and Filesystem Usage (%). Three Top Processes panels rank a host’s process groups by CPU, by memory, and by threads, open file descriptors or handles, and process count, and the Process dashboard shows one group’s CPU, memory, threads, process count, open descriptors or handles, and oldest-process uptime. A widget whose source is not reporting for a host is hidden rather than shown empty, so node-exporter, Telegraf and windows_exporter hosts show no empty OpenTelemetry panels; without the process scraper the Processes tab lists nothing and the Top Processes panels are hidden. Several existing widgets were renamed to say what they show: Linux CPU Average Used (%) is now CPU by State, and its Network Status is split into TCP Connections and Linux Socket Counters; Windows CPU Average Used (%) is CPU by State (sum across cores) and Virtual Memory (MB) is Committed Virtual Memory (MB); the host-list columns read CPU total % and, on Windows, Committed %. The OpenTelemetry metrics need an OAP release that includes apache/skywalking#14051. An OAP that already stores these layers’ templates and translations keeps the old dashboards; to adopt the new ones, open Dashboard setup → Layer dashboards → Linux (and Windows), choose Reset to → Bundled and Check diff & push, then do the same for each language on the Translations page. See Linux and Windows.

A layer split by service group has plain addresses. A group’s pages are at /layer/general/payments/topology, and the services whose names have no group are at the layer’s own addresses, /layer/general/topology, instead of /layer/general~payments/… and /layer/general~/…. A sub page of Service, Instance or Endpoint is at …/service/page/<id>. A reader with no entry for the services with no group — the layer has none, or the role’s grant does not reach them — who opens the layer’s own address is taken to the same page of their first group entry.

Four new bundled layers for OpenTelemetry traces. OAP 11.1.0 analyzes the spans arriving over OTLP with its Trace Analysis Language and registers the services under OTLP_SERVICE, with their databases, caches and brokers under OTLP_VIRTUAL_DATABASE, OTLP_VIRTUAL_CACHE and OTLP_VIRTUAL_MQ. When OAP reports those layers the sidebar gains the OTLP Service layer, with the service list, the service, instance and endpoint dashboards, the topology with client-side edge metrics and the Traces tab, and the OTLP Virtual Database, OTLP Virtual MQ and OTLP Virtual Cache layers, each with its service list and dashboard, all four under one OpenTelemetry group; the database one lists the slow statements the OTLP slow-SQL rule records, and the MQ one ranks the broker’s destinations. The metrics are the otlp_* families, so an OAP without the trace analyzer shows nothing here; the Traces tab also needs OAP’s TraceQL module with its OTLP datasource. See OpenTelemetry Service, OpenTelemetry Virtual Database, OpenTelemetry Virtual Cache and OpenTelemetry Virtual MQ.

Fixes

A compared service that failed to refresh was drawn as though it had. In Compare, each entity keeps its last good reading when its own read fails — deliberately, so one failure does not blank its siblings — but the retained series was then plotted across the CURRENT axis and ran to the right-hand edge beside the fresh ones. Ten-minute-old latency sat next to live latency with nothing to tell them apart. Each series is now placed at the buckets it was actually read for, so a stale one stops where its data does and the gap is visible.

A layer whose dashboard an administrator removed no longer reports an outage. The page said the template was disabled and pointed at an admin screen to re-enable it — a control most operators cannot reach, on a page that does not exist under that name — while the refresh history recorded “OAP could not be reached” for what was an administrative decision on a healthy server. The page now simply says it is not available, and the layer leaves the sidebar on the next menu read.

Reactivating a layer brings back its own dashboard, not the shipped default. OAP keeps a disabled template’s configuration; Reactivate was re-pushing the bundled default over it, discarding every edit the layer carried. It now restores what was there.

The sidebar drew a generic icon for a layer split by service group. The General layer’s entries showed the default layer icon instead of the SkyWalking one.

The Events button of the layer header found no events on a layer split by service group. It filtered the events by the group entry instead of the layer, which no event carries.

The service hierarchy named a layer split by service group by its key. It showed GENERAL in the accent colour instead of the layer’s name and colour, and the sidebar did not mark such a layer’s entries when the template OAP stores for it differs from the shipped one. That marker’s tooltip also claimed unpublished local changes; it now says what the marker means.

The “No access to this layer” card named a group entry by an internal key. It says which service group, or that it is the services with no service group, and of which layer.

The AI assistant’s trace figure said a layer split by service group had no traces component. It looked the layer up under a key only a layer that is not split has.

Dependencies

Fastify 5.12.5 and fast-uri 3.1.8 / 4.2.1 clear the security advisories open against the release package. Fastify 5.12 stopped honouring a numeric trustProxy; Horizon keeps server.trustProxy: 1 (and any other hop count) recording the same client address as before, so no configuration change is needed. As before, a hop count believes X-Forwarded-For from a caller that reaches Horizon without going through your proxy; where that is possible, name the proxy’s address or CIDR instead. Fastify’s other published fix, a numeric body schema letting a string through, does not reach Horizon, which validates request bodies itself. fast-uri, which the JSON Schema validators in Fastify and the MCP server use for URI values, had eleven advisories, several of them server-side request forgery through malformed host names.

hono 4.13.11, qs 6.16.0 and ip-address 10.7.2 come with the MCP SDK and belong to parts of it Horizon does not load; they move so the release package no longer carries the flagged versions. Vitest, undici and js-yaml, which only run in tests and linting and never ship, were refreshed too.