Monitoring and Alerting Architecture¶

Purpose¶

Alerts from a workload cluster have two audiences with different needs. Servala operations needs alerts about nodes, the control plane, storage and service level indicators, so that incidents get picked up by the on-call team. Customers need alerts about their own service instance, delivered by email if they have configured an address.

Mixing the two causes problems in both directions. Customers would receive alerts about platform components and about other customers' instances, and alerts a customer has already subscribed to would keep paging the on-call team. Platform alerting and customer alerting therefore run as two separate stacks with no overlap in scope.

Architecture Overview¶

flowchart LR
    subgraph collect["Collection"]
        prom["prometheus-infra<br/>syn-infra-monitoring<br/>(the only scraper)"]
    end
    subgraph platform["Platform alerting"]
        amInfra["alertmanager-infra<br/>syn-infra-monitoring"]
        og["OpsGenie<br/>(Schedar on-call)"]
    end
    subgraph user["Customer alerting"]
        ruler["thanosruler-user<br/>syn-user-monitoring"]
        amUser["alertmanager-user<br/>syn-user-monitoring"]
        mail["Customer email"]
    end

    prom -->|"evaluates platform rules"| amInfra
    amInfra --> og
    prom -.->|"queried over HTTP,<br/>no second scrape"| ruler
    ruler -->|"evaluates instance rules"| amUser
    amUser --> mail

Metrics are collected once. prometheus-infra scrapes everything on the cluster, and the customer side has no collection of its own. thanosruler-user evaluates its rules by querying prometheus-infra over the HTTP API and sends the resulting alerts to its own Alertmanager.

The two stacks¶

How an alert is routed¶

Which stack evaluates a rule depends on the namespace the PrometheusRule object lives in. AppCat labels every service instance namespace with appcat.vshn.io/servicename, and the two stacks use inverse selectors on that label:

# prometheus-infra: everything that is not a service instance
ruleNamespaceSelector:
  matchExpressions:
    - key: appcat.vshn.io/servicename
      operator: DoesNotExist

# thanosruler-user: service instances only
ruleNamespaceSelector:
  matchExpressions:
    - key: appcat.vshn.io/servicename
      operator: Exists

Alertmanager configuration is separated the same way. alertmanager-infra has no alertmanagerConfigNamespaceSelector and therefore reads only its own namespace, while alertmanager-user selects instance namespaces and picks up the per-instance AlertmanagerConfig objects. Since the selectors do not overlap, neither stack can see the other's rules or receivers.

What goes where¶

Several rules concern a specific customer instance and still stay on the platform side. Replication, fencing, manual switchover and WAL archiving are all operated by Servala, and a customer has no way to act on them.

The customer-bound rules cover the two persistent volume alerts and a memory alert. AppCat builds the set per service type, so it varies from there.

Customer alerting¶

A customer enables email alerting by setting an address on the claim:

spec:
  parameters:
    monitoring:
      email: user@example.com

AppCat then generates an AlertmanagerConfig in the instance namespace with an email receiver, which alertmanager-user picks up through its namespace selector. The mail body and subject are rendered from a Servala template. See Monitoring for the customer view.