Run the E2E Tests

Every change to this chart is validated by an end-to-end suite that installs chart/skywalking into a throwaway kind cluster, pushes real traffic through it, and then asks Horizon UI’s BFF the questions an operator would ask. Nothing in the suite talks to the OAP’s GraphQL endpoint directly — that would test the OAP, not the chart. Five cells run in CI: three that push traffic through OAP 11 on two storage backends, one for configuration overrides, and one that upgrades the published chart. This page explains how they are put together and how to run one on your own machine.

The framework

The tests are driven by skywalking-infra-e2e. Each config file under test/e2e/ is one complete test: the cluster to create, the setup steps to run, the traffic to generate, and the assertions to make. The phases are:

kind.expose-ports is what makes the verify phase possible. Every cell exposes exactly one service:

kind:
  expose-ports:
    - namespace: istio-system
      resource: service/skywalking-ui
      port: 80

infra-e2e port-forwards it and exports two shell variables named after the resource and the port — ${service_skywalking_ui_host} and ${service_skywalking_ui_80} — which every verify case interpolates into a base URL. The OAP’s own ports are not exposed at all.

expected/*.yml are templates, not literals. They use matchers such as {{- contains .services }} and {{ notEmpty .version }}, so a case asserts “these services are present” or “this field has a value”, not an exact payload.

The cells

Three of them run the same fixture — kind kindest/node:v1.28.15 (one control plane, three workers), namespace istio-system, fullnameOverride=skywalking, oap.replicas=1, Satellite enabled, Horizon UI at $UI_REPO:$UI_TAG — and differ in which OAP and which storage they install, plus the flags each of those pairings needs.

All three also switch Zipkin on — oap.ports.zipkin-query=9412 plus SW_RECEIVER_ZIPKIN=default and SW_QUERY_ZIPKIN=default — because the chart only emits oap.zipkinUrl into Horizon’s config when that port is set, and one verify case asserts Horizon can reach it. Each runs seven cases.

The fourth cell: e2e-config-override.yaml

The other three take the default configuration path — Horizon set up purely by environment variable, with no ConfigMap created at all. This cell covers the opt-in third mechanism for both components: a file rendered into a ConfigMap and mounted over the image’s own.

It is deliberately cheap — no Istio, no bookinfo, no traffic generator. What is under test is whether the overrides are read, which needs none of that, so it skips the MESH fixture entirely and never queries a metric.

The mount is the riskier path, and the reason this cell exists. Mounting over Horizon’s /app/horizon.yaml replaces it, so the chart has to merge its computed values back in as ${HORIZON_*:default} tokens. Two of the cases exist to catch that regression rather than to check the ConfigMap: logging in proves the HORIZON_AUTH_LOCAL_USERS token survived the mount (without it the pod is Ready and nobody can sign in), and /api/oap/info proves queryUrl did not revert to 127.0.0.1.

The last two cases check the labels of a fresh install: its Deployments select their pods by the recommended app.kubernetes.io/* labels, nothing in the release carries a Helm 2 release label, its Services select by the recommended labels, and OAP finds its peers by them. The upgrade cell below covers a release installed before 5.1.0.

The fifth cell: e2e-upgrade.yaml

Every other cell installs fresh, so it only ever sees objects this chart created. This one takes the path every existing user takes: it installs the last published chart from oci://docker.io/apache/skywalking-helm (CHART_FROM in the file, 5.0.0), then runs helm upgrade to the working tree with identical values, on BanyanDB standalone with Satellite enabled. A second upgrade then sets only labelStyle: legacy.

An upgrade can fail where a fresh install passes, on fields Kubernetes never lets change. A Deployment’s selector is one: releases installed before 5.1.0 select by app/release/component, and the chart has to keep that selector or the upgrade fails with field is immutable. The init Job’s pod template is the other: while the chart’s default values match CHART_FROM’s, identical values keep the Job’s name, so Helm patches the existing Job, and any chart change to its pod template fails the upgrade the same way. The values are identical in both helm commands on purpose, because different values would rename the Job and hide this.

The live upgrade runs on the bundled BanyanDB, whose subchart defaults changed after 5.0.0, so its Job is renamed and the in-place path is not exercised there. Elasticsearch and PostgreSQL releases still keep their Job’s name on the upgrade, which is why the pod-template case renders all three backends instead of relying on the live one.

CHART_FROM stays at 5.0.0, the last release whose Deployments select by the legacy labels. A release installed with 5.1.0 or later selects by the recommended labels, so a later CHART_FROM would test something else and fail expected/upgrade-selectors.yml. Upgrading from the latest release is a different test; add it as its own cell.

There are no OAP 10.4 cells

CI covers the OAP 11 line only. Version Compatibility still documents running Horizon 1.0.0 against OAP 10.4.0 — oap.ports.admin=null and HORIZON_TEMPLATES_MODE=readonly — but nothing here tests it, so treat that combination as documented rather than verified.

The traffic fixture: everything lands in MESH

This is the single most useful fact for anyone adding a case. There is no Java agent anywhere in this suite. Each cell:

1. installs Istio with the demo profile and Envoy ALS pointed at the chart’s Satellite — meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-satellite.istio-system:11800 and meshConfig.enableEnvoyAccessLogService=true — then labels default with istio-injection=enabled;
2. installs the chart with SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh and SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=k8s-mesh;
3. deploys bookinfo from the $ISTIO_VERSION manifests and runs wrk against http://istio-ingressgateway.istio-system:80/productpage.

So every service the tests can see is synthesized by the OAP from Envoy access logs, and it lands in the MESH layer — never GENERAL. A verify case that queries /api/layer/GENERAL/services will find nothing, no matter how long it retries.

Service names come from two places that must agree: K8S_SERVICE_NAME_RULE='e2e::${service.metadata.name}' on the OAP command line, and oap.config."metadata-service-mapping.yaml" in test/e2e/values.yaml, which maps the Istio canonical-name label to e2e::<name>. That is why the assertions look for e2e::productpage and e2e::reviews.

How an assertion works

Every verify case shells out to test/e2e/script/horizon.sh, which does what an operator does: log in, keep the session cookie, then call an API with it.

horizon.sh <base-url> get  <api-path>
horizon.sh <base-url> post <api-path> <json-body>

It POSTs to /api/auth/login with curl --fail-with-body -c "$JAR", fails loudly unless the response actually set a horizon_sid cookie, and reuses that jar (-b "$JAR") for the real request. Credentials come from HORIZON_USERNAME / HORIZON_PASSWORD and default to admin/admin.

The JSON that comes back is piped through yq into a small projection, and that is what the expected/ template matches. One case verbatim, from test/e2e/e2e-oap11-elasticsearch.yaml:

    - query: |
        bash test/e2e/script/horizon.sh http://${service_skywalking_ui_host}:${service_skywalking_ui_80} get /api/layer/MESH/services | yq -p json -o yaml '{"layer": .layer, "services": ([.services[].name] | sort)}'
      expected: expected/horizon-mesh-services.yml

and the template it matches, test/e2e/expected/horizon-mesh-services.yml (below its license header):

layer: MESH
services:
  {{- contains .services }}
  - e2e::productpage
  - e2e::reviews
  {{- end }}

The yq projection is deliberate: pull out the two or three fields that carry the meaning, sort anything order-dependent, and keep the expected file small enough to read.

Three traps

All of these cost real debugging time, so they are commented in the cell files as well.

/api/layer/:key/services returns reachable: true with an empty list even when the OAP is down. Asserting on .reachable there proves nothing — the case would pass against a dead backend. Assert on the service names, as the case above does.

infra-e2e’s notEmpty matcher rejects numbers. It only accepts nil or a string (notEmpty only supports nil or string type, but was ...), so a numeric assertion has to be turned into a boolean in the yq projection before it reaches the template. That is what the service_cpm case does:

        ... | yq -p json -o yaml '{"reachable": .reachable, "id": .widgets[0].id, "positive": (.widgets[0].value != null and .widgets[0].value > 0)}'

against expected/horizon-service-cpm.yml, which expects positive: true alongside reachable: true and id: cpm. Use notEmpty for strings such as .version; compare numbers yourself.

Do not helm install --wait with the bundled BanyanDB. The BanyanDB subchart’s *-banyandb-http Service is a LoadBalancer, which never gets an address on kind, so Helm 3’s --wait blocks on it until the timeout even though every pod is Ready. Helm 4 does not, so a local run with Helm 4 passes and CI does not. The cells wait with infra-e2e wait: conditions or kubectl wait / kubectl rollout status instead.

Login: the chart ships no users

chart/skywalking/values.yaml sets ui.config: {}, and Horizon has no built-in admin/admin fallback — a fresh install has nobody who can log in until you configure auth, which is exactly what Set Up Logins is about.

test/e2e/values.yaml seeds two local users — admin/admin (role admin) and skywalking/skywalking (roles viewer, maintainer) — as argon2id hashes. Every cell must therefore pass -f test/e2e/values.yaml, or the very first horizon.sh call fails at the login step. That failure is the intended signal, not a flake.

The same overlay carries the other two things the tests need on top of chart defaults:

• oap.config."metadata-service-mapping.yaml" — the e2e:: service naming described above.
• elasticsearch.nodeSets — a single 2Gi ES node with node.store.allow_mmap: false and relaxed disk watermarks, so ES stays green on a kind node with little free disk.

The first verify case in every cell (GET /api/auth/me against expected/horizon-me.yml) asserts username: admin with roles: [admin], which proves the overlay was applied before anything else is tested.

test/e2e/env — the image pin file

Every cell loads this file via init-system-environment, so it is the single place where the versions under test are pinned.

The OAP 11 line moves as a trio: OAP 11.0.0 accepts BanyanDB server API 0.11 only, and Horizon 1.0.0 is the UI tested against it — see Version Compatibility. The BanyanDB chart version is not pinned here; it comes from chart/skywalking/Chart.yaml.

Run one locally

You need Docker and Go — the e2e binary embeds kind as a library, so no separate kind binary is required. The setup steps install yq, kubectl, istioctl and helm into /usr/local/bin and install Istio into the cluster, so run this on a machine you don’t mind changing.

Build the e2e CLI once:

git clone https://github.com/apache/skywalking-infra-e2e.git
cd skywalking-infra-e2e
make install DESTDIR=/usr/local/bin

Then, from the root of this repo (the config files reference chart/skywalking and test/e2e/… relative to the working directory):

export ISTIO_VERSION=1.24.0
e2e run -c test/e2e/e2e-oap11-banyandb-standalone.yaml

ISTIO_VERSION is set by the CI workflow, not by test/e2e/env, and both install-istioctl.sh and the bookinfo manifest URLs read it — export it yourself when running locally.

To iterate without re-creating the cluster, run the phases separately:

e2e setup   -c test/e2e/e2e-oap11-banyandb-standalone.yaml
e2e verify  -c test/e2e/e2e-oap11-banyandb-standalone.yaml   # repeat as you debug
e2e cleanup -c test/e2e/e2e-oap11-banyandb-standalone.yaml   # deletes the kind cluster

While the cluster is up:

kubectl -n istio-system get pods
kubectl -n istio-system logs deploy/skywalking-oap
kubectl -n istio-system logs deploy/skywalking-ui          # Horizon's BFF, where the API calls land
kubectl -n istio-system port-forward svc/skywalking-ui 8080:80   # then log in as admin/admin

With that port-forward running you can also drive the script by hand, exactly as the verify phase does:

bash test/e2e/script/horizon.sh http://localhost:8080 get /api/layer/MESH/services | yq -p json

How CI runs them

.github/workflows/e2e.ci.yaml runs the suite from one matrix job, als, with fail-fast: false and a matrix of five entries — one per cell — each with a 60-minute timeout, so one failing cell does not cancel the others:

strategy:
  fail-fast: false
  matrix:
    test:
      - name: Horizon + OAP 11 + Elasticsearch
        config: test/e2e/e2e-oap11-elasticsearch.yaml
      # …the two OAP 11 BanyanDB cells, e2e-config-override.yaml and e2e-upgrade.yaml

Each entry logs in to ghcr.io (the Satellite and BanyanDB 0.11 images live there), sets up Go 1.24, and hands its config file to the apache/skywalking-infra-e2e action, pinned to SHA 8c21e43e…:

- uses: apache/skywalking-infra-e2e@8c21e43e241a32a54bdf8eeceb9099eb27e5e9b4
  with:
    e2e-file: $GITHUB_WORKSPACE/${{ matrix.test.config }}

The workflow sets ISTIO_VERSION: 1.24.0 in its top-level env:. On failure it dumps disk usage and the local Docker images, then uploads $SW_INFRA_E2E_LOG_DIR as the logs artifact — start there when a CI run fails but a local run passes. A trailing build job depends on als and only runs echo, so one job name aggregates the whole matrix.

The workflow runs on every pull_request and on pushes to master. The paths-ignore: ['**.md'] filter applies only to the push trigger, so a docs-only pull request still runs the full matrix.

Adding a case

1. Add the query: to all three traffic cells, and put the expectation in test/e2e/expected/. Leave e2e-config-override.yaml alone unless the case is about configuration: it installs no Istio and generates no traffic, so anything asking for a metric will never pass there.
2. Go through test/e2e/script/horizon.sh. Anything that calls the OAP directly is testing the OAP.
3. Query the MESH layer, and project the response with yq down to the fields that carry meaning.
4. Keep image references as $OAP_REPO / $OAP_TAG style variables so test/e2e/env stays the only place versions are pinned.

Adding a cell

Copy the closest existing file, change the storage flags, and add a matrix entry in .github/workflows/e2e.ci.yaml — a config file that is not in the matrix never runs. See Elasticsearch and BanyanDB for what those storage flags mean outside the tests.