Skip to content

Problems ​

The Problems view answers one question across every selected cluster: what is broken right now? It is the second most useful page on a desktop and the landing page on a phone. Open it with :pb, from the sidebar, or from the problems badge in the top bar.

What you see ​

One list, sorted by severity and then by how recently it changed. Each row has a severity tag, the cluster, the problem class, the object (kind, namespace, name), a one-line title such as CrashLoopBackOff: api (restarts 14), and how long it has been going on.

  • A problem appears within about two seconds of the change that causes it, and disappears when it resolves.
  • Rollups. When more than one pod of the same workload has the same problem, v8s shows one row: "Deployment api: 3/5 pods CrashLoopBackOff". Expand it for the pods. This is what keeps the view readable in an incident.
  • A problem that clears within 30 seconds of being raised is held for that window instead of flapping.
  • Time-based checks ("Pending for more than 5 minutes") carry their own thresholds, so a normal rollout does not raise noise.
  • Problems exist only while v8s is open. There is no background alerting or push notification.
  • Press :pb or choose Problems in the sidebar. On a desktop the app opens on Pods; on a phone it opens on Problems.

If you cannot list a kind that a check needs, the view says so ("Some checks are unavailable in prod-eu: no permission to list ingresses") instead of reporting zero problems.

What it finds ​

There are 51 problem classes in 12 families. Each can be switched on or off from the Problems page. Severity is in brackets. "Default" is what a fresh install does; classes whose data source is missing (no Flux, no Kyverno, no Hubble) simply raise nothing.

Pods, workloads, and jobs ​

ClassRaised whenDefault
pod.crashloopA container is in CrashLoopBackOff [error]On
pod.image-pullImagePullBackOff, ErrImagePull, or InvalidImageName [error]On
pod.oomkilledA container was OOMKilled within the last hour [warning]On
pod.pendingPending for more than 5 minutes, with the scheduling reason [warning]On
pod.not-readyRunning but not Ready for more than 5 minutes [warning]On
pod.stuck-terminatingDeleting for more than 10 minutes [warning]On
pod.config-errorCreateContainerConfigError, such as a missing ConfigMap or Secret key [error]On
pod.evictedPod failed with reason Evicted [info]Off
pod.restartingRestarts increased in the last 10 minutes, not yet crash looping [warning]Off
workload.unavailableDeployment, StatefulSet, or DaemonSet below desired beyond its deadline [error]; rolls up podsOn
workload.rollout-stuckProgressDeadlineExceeded [error]On
hpa.unableHPA cannot scale, or has been at max replicas for 30 minutes [warning]Off
job.failedA Job failed (backoff limit or deadline) [error]On
cronjob.last-failedThe last scheduled Job failed [error]On
cronjob.not-schedulingNot suspended, but no successful run in twice its interval [warning]On

Nodes and storage ​

ClassRaised whenDefault
node.not-readyNode Ready is not True [critical]On
node.pressureMemory, disk, or PID pressure [error]On
node.cordonedNode is unschedulable [info]Off
pvc.pendingPVC Pending for more than 5 minutes [error]On
pv.failedPV Failed, or Released with Retain [warning]Off

Networking ​

ClassRaised whenDefault
ingress.no-backendAn Ingress points at a missing Service, or one with no ready endpoints [error]On
service.no-endpointsA Service with a selector has had zero ready endpoints for 2 minutes [warning]On
gateway.route-unacceptedA Gateway API route is Accepted=False or ResolvedRefs=False [error]On
cert.expiringA cert-manager Certificate is not Ready, or expires within 14 days [warning]On

cert.expiring is on by default but only raises anything where cert-manager is installed.

Platform ​

ClassRaised whenDefault
apiservice.unavailableAn APIService is Available=False [error]On
webhook.failingAn admission webhook with failurePolicy=Fail has a Service with no ready endpoints [critical]On
cluster.unreachablev8s cannot reach the API server, or auth has expired [critical]On
events.warning-burstMany Warning events on one object in 10 minutes, not explained by another class [warning]Off

Flux and Helm ​

ClassRaised whenDefault
flux.not-readyReady=False, excluding suspended and progressing within grace [error]On
flux.stalledStalled=True [error]On
flux.staleNo successful reconcile in 3 times the interval [warning]On
flux.source-failedA source (Git, OCI, Helm repository) failed [error]On
flux.controller-downA Flux controller deployment is down [critical]On
flux.dependency-blockeddependsOn chains collapsed to their root cause [warning]On
flux.suspendedAn object is suspended [info]Off
helm.failedThe latest release revision is failed [error]On
helm.stuck-pendingA release has been pending-* for more than 15 minutes [warning]On

Policy and security ​

ClassRaised whenDefault
policy.failA resource has Kyverno policy failures (one problem per resource) [warning]On
policy.errorA resource has Kyverno policy errors [error]On
policy.not-readyA Kyverno policy is not Ready [warning]On
policy.admission-deniedAn enforced Kyverno policy blocked a request in the last hour, from Events [warning]On
vuln.criticalAn image has critical vulnerabilities, one problem per image, not per CVE [error]On
vuln.highAn image has high vulnerabilities [warning]Off
misconfig.criticalCritical configuration audit failures [error]On
misconfig.highHigh configuration audit failures [warning]Off
secret.exposedAn exposed secret was found in an image [error]On
rbac.highCritical or high RBAC assessment findings [warning]Off

These come from Kyverno, Trivy Operator, and Kubescape. See Security. The vulnerability classes apply regardless of which scanner reported them; a source tag tells you which. Misconfiguration classes come from both scanners, secret.exposed and rbac.high from Trivy only.

Network flows ​

ClassRaised whenDefault
network.policy-dropsSustained policy-denied drops between two workloads (about 10 in 5 minutes) [warning]Off
network.l7-errorsA service's HTTP 5xx count over 5 minutes is above a threshold (about 20) [warning]Off
network.dns-errorsA spike of NXDOMAIN or SERVFAIL responses (about 20 in 5 minutes) [warning]Off
hubble.relay-downHubble Relay is unreachable [warning]On

These need Hubble. hubble.relay-down is on by default but only raises where a relay exists. See Network.

Filter and group ​

  • Family chips. One chip per family (Pods, Workloads, Jobs, Nodes, Network, Storage, Flux, Helm, Policy, Vulnerabilities, Network flows, Platform), each with a live count. Click a chip to show only that family. The caret on a chip opens switches for each class in the family (long-press on a phone). A chip whose classes are all off shows off. Class switches take effect in the agent, so a phone never receives problems it would hide, and they are saved in your settings. A class that is overridden for some clusters in the settings file is marked "overridden in N clusters".
  • Severity. All, Error and up, or Critical.
  • Search. Matches the object name. Press / to focus it.
  • Group by. No grouping, class, cluster, or namespace. Namespace grouping is unavailable above 100 namespaces.
  • Order. Newest change first (the default), or oldest first.
  • Across clusters. Merges problems with the same class, kind, namespace, and name across clusters into one row with cluster tags, so "the ingress-nginx webhook is failing in 4 clusters" is one row. Needs at least two clusters and at most 2,000 problems; narrow the filters if the box is disabled.
  • Show acknowledged. Brings acknowledged and snoozed problems back, faded.

Keyboard ​

KeyAction
j / kMove the focus down / up
EnterExpand or collapse the focused row or group
aAcknowledge, or restore if acknowledged
oOpen the problem's object in the drawer

Look into one ​

Expand a row to see the evidence inline: the condition or state that triggered it, container states, the Warning events related to it, related objects, and quick actions that fit the problem. The actions v8s can offer are Logs, Previous logs, Describe, Restart, Roll back, Reconcile, Suspend, Open release, Open policy, and Open image vulns. Only those that apply to the problem and that you are allowed to run are shown. Open the object itself in the drawer with the row's open button or o.

Vulnerability problems are per image: "nginx:1.25 · 4 critical, 31 high · 12 workloads". Expanding shows the workloads, and Open image vulns goes to the full report.

Acknowledge and snooze ​

  • Acknowledge hides a problem for you until it materially changes: its severity rises, or its count grows by 50% (at least one more). A problem that resolves and comes back is a new problem.
  • Snooze hides it for 1 hour, 1 day, or 1 week.
  • Show acknowledged brings them back, faded, with a Restore button.

Acknowledgements belong to you and persist across restarts (stored in your local settings file on desktop, in your per-user settings file on the server in hosted mode). A problem keeps its identity when GitOps deletes and recreates its object, and an acknowledgement is kept while the problem is not live (an object being recreated, a restart, or a class switched off for a while). It is collected only after the problem has gone unseen for 7 days.

On a phone ​

Problems is the landing tab on a phone. It loads a tiny summary first (counts per cluster and class), then a small window of rows, and fetches detail only when you open an entry. Acknowledge and snooze work from the phone. See On a phone.

How it stays cheap ​

When a cluster is selected, v8s keeps a fixed set of cheap watches that feed the default-on checks: projected pods and nodes, workload and Job status, Services, EndpointSlices, Ingresses, PVCs, Warning events only (the Events tab of a resource loads all of its events on demand), and so on. Security and policy report counts come from server-side tables that carry only the counts. Nothing is watched in full that does not need to be. Detectors run in the agent on only what changed. The design is in Problems engine.

Presets ​

Save the current set of enabled classes as a preset for a cluster group (the Presets menu on the Problems page), apply it with one click, or delete it. Presets are stored in your settings, so production can show everything while staging hides noisier classes.

Released under the MIT License.