Infrastructure Architecture¶

Purpose¶

Servala is an open platform offering cloud-native services, applications, and DevOps tools as managed services across multiple cloud providers. It provides scalable, secure, and cost-optimized self-services to businesses.

To deliver on this promise, Servala requires underlying infrastructure that is:

• Easy to manage: Consistent tooling and processes across all environments
• Scalable: Able to grow with customer demand
• Secure: Private by default, with controlled access
• Uniform: Same infrastructure model across all cloud service providers

Related pages: Networking covers the network and CIDR model, Secure Access covers cluster and API connectivity, and the Architecture Decision Records document the key decisions behind this design.

Architecture Overview¶

Servala operates two types of Kubernetes clusters:

All clusters run Talos Linux and are deployed using Terraform/OpenTofu across multiple cloud providers. The architecture is designed for secure, private access with no public Kubernetes API exposure. See ADR 002 for the Kubernetes distribution decision. If a CSP offers a managed Kubernetes which is based on Talos Linux we are open to using that instead for simplicity sake.

Production clusters run a highly available control plane consisting of 3 master nodes. Test clusters run a single master node. Worker nodes are currently scaled manually based on capacity needs.

AppCat¶

AppCat is the core component of the Servala service catalog, built on Crossplane. It runs on every cluster and enables the provisioning and management of cloud-native services for customers. See ADR 001 for the converged control plane model.

• AppCat Documentation: User-facing documentation and service guides
• AppCat Architecture Guide: Technical architecture and internal components

Multi-Tenancy¶

Each AppCat service instance is deployed to an isolated namespace with strict network boundaries. The network side of this isolation is described in Networking.

Resource Management¶

AppCat instances require a size selection during creation in the Servala Portal. The selected size translates directly to Kubernetes resource requests and limits, ensuring predictable resource allocation and preventing noisy-neighbor issues.

Security¶

Pod Security¶

All namespaces enforce the Pod Security Standards at the baseline profile by default. This prevents known privilege escalations while maintaining compatibility with most workloads.

Network Security¶

Default-deny network policies isolate customer namespaces, and ingress traffic is only permitted through managed Gateway API endpoints. See Networking for details.

Audit Logging¶

Kubernetes API server audit logging is enabled on all clusters to track who did what and when. Audit logs are collected centrally alongside other cluster logs.

Secrets Management¶

AppCat service credentials are managed natively by Crossplane. Connection secrets (database credentials, API keys, etc.) are stored as Kubernetes secrets on the workload cluster where the service instance runs. No external secrets manager is required.

Naming Conventions¶

Cluster Names¶

Pattern: c-servala-[CSP]-[REGION]-[ID]

Example: c-servala-cls-rma1-9c02

Generate the ID with:

echo -n "$(date +%s%N)$(openssl rand -hex 4)" | sha256sum | cut -c1-4

The cluster name is used as the identifier within Project Syn for cluster inventory, configuration management, and ArgoCD application targeting. Note that DNS names use a shorter format without the c-servala prefix (see below).

DNS Names¶

Node Names¶

Nodes are organized into three groups with a 4-character hex suffix:

Infra nodes are dedicated to platform workloads: ingress and gateway pods (KGateway), monitoring components, and other platform services. They are labeled and tainted with node-role.kubernetes.io/infra, and a mutating admission webhook (Espejote) automatically pins platform pods to them via node selector and toleration. This keeps customer service instances on worker nodes and platform infrastructure on infra nodes.

Observability¶

Centralized Monitoring¶

Metrics and logs from all workload clusters are collected and stored centrally on the management cluster. This provides a single pane of glass for monitoring the entire Servala infrastructure.

Metrics and logs are shipped over the Netbird mesh (see Secure Access).

Alerting¶

Alerts are routed to OpsGenie for incident management and on-call notification.

Backup Strategy¶

Backups are handled at multiple levels to ensure data durability and disaster recovery:

Cluster Backups¶

• etcd snapshots are taken every 6 hours by talos-backup and stored age-encrypted in S3 buckets
• Enables cluster state recovery in case of catastrophic failure

AppCat Instance Backups¶

• Each AppCat service instance manages its own backups using K8up
• Backups are stored in S3 buckets within the same CSP as the workload cluster
• Fault tolerance: Backups are stored in a different region than the running instance

Example: A VSHNPostgreSQL instance running in Cloudscale stores its backups in a different Cloudscale region than where the database is running.

Upgrades¶

Tuppr manages Talos and Kubernetes upgrades automatically on every cluster. Upgrades roll through the nodes with health checks between steps and only run within per-cluster maintenance windows.

Certificate Management¶

Each cluster runs cert-manager with Let's Encrypt for automated TLS certificate provisioning and renewal. This applies to both ingress certificates and internal service certificates.

DNS¶

DNS is managed through VSHN DNS infrastructure for all Servala services and customer endpoints.

Image Management¶

Container images are pulled from public registries (e.g., ghcr.io for AppCat components). Spegel is deployed on each cluster to provide peer-to-peer image sharing between nodes, reducing external registry pulls and improving pull performance.

Storage¶

Each CSP must provide block storage with a corresponding CSI driver. This is a hard requirement for running Servala. CSPs without CSI support are not eligible.

Storage is exposed to workloads via StorageClasses, abstracting the underlying CSP implementation.

Load Balancer¶

Each CSP must provide Load Balancer as a Service (LBaaS) with a corresponding Cloud Controller Manager (CCM). This is a hard requirement for running Servala. CSPs without LBaaS/CCM support are not eligible.

Kubernetes Services of type LoadBalancer are used to expose Gateway API traffic to the internet.

CNI¶

Cilium is the CNI on all clusters, running in kube-proxy replacement mode. Two Talos-specific configuration details are worth knowing:

• KubePrism as API endpoint: Cilium talks to the Kubernetes API via Talos KubePrism on localhost:7445, giving it HA API access on every node without depending on the external VIP.
• Socket load balancer restricted to the host namespace (socketLB.hostNamespaceOnly: true): pods use tc-bpf load balancing at the veth interface instead of the socket-LB. The tc-bpf path also DNATs L3-forwarded traffic (Netbird routing peers) to ClusterIP backends. Without this setting, forwarded packets to a Service IP originating inside a pod bypass the socket-LB and never reach a backend. See the Cilium documentation for details.

Gateway API¶

KGateway implements the Kubernetes Gateway API on all clusters. Each service type gets its own gateway: an HTTP gateway (ports 80/443) plus dedicated TCP gateways per service (PostgreSQL, MariaDB, Redis, SSH). Each gateway is exposed via a LoadBalancer Service.

AppCat contains a port allocation logic that assigns each service instance a unique port on the shared gateway of its service type, so many instances can reuse the same load balancer IP.

DBaaS instances can alternatively be exposed through a dedicated LoadBalancer Service with their own IP instead of the shared gateway.

Cluster Components¶

Project Syn¶

Project Syn is a set of tools for securely managing a fleet of Kubernetes clusters using GitOps principles. It serves as the cluster management layer for Servala:

• ArgoCD Bootstrap: Bootstraps ArgoCD on each cluster during initial setup
• Component Management: Cluster components like AppCat are installed and updated through Project Syn's configuration management
• Hierarchical Configuration: Enables consistent configuration across all clusters while allowing per-cluster customization

Cloud Provider Notes¶

Exoscale¶

Infra nodes are provisioned as an instance pool, since the Exoscale Network Load Balancer (NLB) can only target instance pools. They carry public IPs in addition to the private network so they can serve NLB traffic.

Cloudscale¶

Nodes in private networks require a NAT gateway for external communication. This feature must be requested via Cloudscale support chat.

Required parameters for cluster creation:

• Network UUID
• Subnet UUID
• Gateway IP
• Preferred Gateway Name