The OAP Init Job

The chart creates the storage schema with a one-shot *-oap-init-* Job, and the main OAP Deployment refuses to serve until that Job has finished. This page explains that handshake, why the Job is deliberately not a Helm hook, and how to watch, rerun, or clean it up.

The handshake

Two OAP containers from the same image, started with different modes:

Order of events on helm install:

1. Both the Job pod and the OAP pods run the same wait-for-storage init container and block until the storage backend answers. For Elasticsearch and BanyanDB it gives up after 60 attempts 5s apart (at least 5 minutes) and fails; for PostgreSQL it retries pg_isready every 3s indefinitely.
2. The Job’s OAP boots in init mode, writes the schema and exits Completed.
3. The OAP pods, still blocked in no-init mode, see the schema, open 12800, and pass their startup/readiness probes.
4. Helm’s --wait resolves, because the Job and the Deployment run in the same phase.

Because the OAP pods’ readiness depends on the Job, a failing Job shows up as OAP pods stuck in 0/1 Running — see Watching and debugging below.

Why it is not a Helm hook

The Job is a normal release resource, not a post-install/post-upgrade hook. A hook would deadlock helm upgrade --install --wait:

• Helm waits for every release resource to become Ready before it runs post-* hooks.
• The OAP Deployment never becomes Ready until the schema exists.
• The schema is created by the hook, which never runs. Helm waits until timeout.

As a main-phase resource the Job runs alongside the Deployment, and the two unblock each other.

To have Helm surface init-Job failures directly — instead of only reporting that OAP never became Ready — add --wait-for-jobs alongside --wait:

helm upgrade --install skywalking oci://docker.io/apache/skywalking-helm \
  --version 5.0.0 -n skywalking --create-namespace \
  --set oap.image.tag=11.0.0 \
  --set oap.storageType=banyandb \
  --set ui.image.tag=horizon-1.0.0 \
  --set elasticsearch.enabled=false \
  --set banyandb.enabled=true \
  --set banyandb.image.tag=0.11.0 \
  --wait --wait-for-jobs

The value-hashed Job name

A Job’s spec.template is immutable. With a stable name, any helm upgrade that changes the pod template (a new image tag, a new env var) would fail with field is immutable. So the name carries a short hash of the rendered values, without labelStyle, plus the label style for a release installed with 5.1.0 or later:

{{- $hashed := omit .Values "labelStyle" }}
{{- if eq (include "skywalking.labelStyle" .) "modern" }}
{{- $_ := set $hashed "_labelStyle" "modern" }}
{{- end }}
{{ printf "%s-init-%s" (include "skywalking.oap.fullname" . | trunc 40 | trimSuffix "-")
                       ($hashed | toYaml | sha256sum | trunc 8) }}

For a release named skywalking that renders to, for example:

skywalking-skywalking-helm-oap-init-39aeb14b

Consequences worth knowing:

• Any value change produces a new Job name, so helm upgrade creates a fresh Job and re-runs init; Helm prunes the previous one because it is no longer in the manifest.
• The hash covers the whole values tree, including subchart values — an unrelated change (a UI setting, a BanyanDB replica count) also re-runs init. A new chart version whose default values differ, its own or a subchart’s, re-runs init on the upgrade too.
• labelStyle is left out of the hash: it only overrides how the release’s label style is found, so setting or removing it does not re-run init. The style itself is hashed for a release installed with 5.1.0 or later, because the Job’s pod labels follow it: moving an older release to the new labels renames the Job and runs init once more instead of failing. A release installed before 5.1.0 hashes exactly what 5.0.0 hashed.
• The chart itself must never change the Job’s pod template, labels included, without a values change that renames the Job: with an unchanged name, Helm patches the existing Job and the upgrade fails with field is immutable. That is why a release installed before 5.1.0 keeps exactly the pod labels 5.0.0 rendered, with the recommended labels only on the Job’s own metadata. The upgrade E2E cell checks this.
• The OAP name is truncated to 40 characters before the suffix, so the Job name stays inside the 63-character DNS limit.

Configuring the Job

The Job reuses the OAP values for everything that must match the server: oap.image.*, oap.javaOpts, oap.env, oap.resources, oap.securityContext, the OAP service account, the storage env vars derived from oap.storageType, and the same oap.config / oap.secretMounts volumes. Only these oapInit.* keys are Job-specific:

A service mesh sidecar is the common reason to set extraPodLabels — an injected sidecar keeps the pod alive after the OAP container exits, so the Job never reports Complete:

oapInit:
  extraPodLabels:
    sidecar.istio.io/inject: "false"

ttlSecondsAfterFinished and the GitOps caveat

oapInit.ttlSecondsAfterFinished is empty by default, so completed Jobs and their pods stay in the namespace where you can still read their logs. Set it to tidy them up:

--set oapInit.ttlSecondsAfterFinished=600

Leave it empty when the release is managed by a GitOps tool (Argo CD, Flux). The TTL controller deletes the Job, the GitOps reconcile sees a resource missing from the desired state, recreates it, and init runs again on every sync loop. The value-hashed name already makes upgrades work without TTL — the setting is only cosmetic cleanup.

The startup probe budget

Because no-init OAP keeps 12800 closed while it waits, the chart’s default startup probe is generous — from chart/skywalking/values.yaml:

oap:
  startupProbe: {}
  # Boot budget defaults to 30 (failureThreshold) * 10 (periodSeconds) = 300 seconds.
  # In no-init mode OAP keeps port 12800 closed until the OAP init Job has created the storage
  # schema, so the budget must be large enough to cover storage startup + schema creation;
  # otherwise the pod is restarted while it is legitimately waiting for the init Job.

That 300-second budget has to cover storage startup plus schema creation. A cold Elasticsearch cluster, a slow PVC, or a first-time BanyanDB group creation can exceed it, and the pod is then restarted while it was legitimately waiting. Raise it rather than reducing the wait:

oap:
  startupProbe:
    tcpSocket:
      port: 12800
    failureThreshold: 90    # 90 * 10s = 15 minutes
    periodSeconds: 10

Setting oap.startupProbe replaces the default block entirely, so repeat tcpSocket.port: 12800. The liveness and readiness probes default to the same TCP check on 12800 with initialDelaySeconds: 5, periodSeconds: 10.

Watching and debugging

export NS=skywalking
export RELEASE=skywalking

# The chart renders exactly one Job per release, whatever the storage type. Read its name from the
# release's rendered manifest, which is right for any chart version and either label style. With
# `helm template | kubectl apply`, pipe your rendered output into the same awk instead: `kubectl
# apply` never prunes a Job that a values change replaced, so the namespace may hold older ones.
JOB=$(helm get manifest "$RELEASE" -n "$NS" | awk '/^kind: Job$/ {j=1} j && /^  name:/ {gsub(/"/, "", $2); print $2; exit}')
kubectl get job -n "$NS" "$JOB"

# Logs (while running, and after completion until the TTL removes it)
kubectl logs -n "$NS" "job/$JOB" --tail=200

# Block until it finishes
kubectl wait --for=condition=complete -n "$NS" "job/$JOB" --timeout=10m

Common outcomes:

Forcing a rerun

Every helm upgrade with a changed value already reruns init. To rerun it without changing any value, delete the Job and upgrade — Helm recreates the now-missing resource:

# The release's current init Job, read from its rendered manifest: right for any chart version and
# either label style. With `helm template | kubectl apply`, pipe your rendered output into the same
# awk instead; `kubectl apply` never prunes a Job that a values change replaced.
kubectl delete job -n skywalking "$(helm get manifest skywalking -n skywalking | awk '/^kind: Job$/ {j=1} j && /^  name:/ {gsub(/"/, "", $2); print $2; exit}')"
# Keep the chart version the release already runs (see `helm list -n skywalking`): an older
# version would also roll back the release's labels and selectors.
helm upgrade skywalking oci://docker.io/apache/skywalking-helm \
  --version <current chart version> -n skywalking --reuse-values

See also

• Configure OAP — the oap.env, oap.javaOpts and config-override values the Job inherits
• Pick a Backend — what “the schema” means per storage type
• Install and Startup troubleshooting
• Upgrade
• skywalking chart values