ADR 003 Secure Cluster API and Service Access¶

Context and Problem Statement¶

Servala clusters live fully in private networks, there is no direct access to the Kubernetes API and nodes from the Internet. Exposing customers service instances running on the clusters has to be done either via Ingress controller, Gateway API or Kubernetes service type LoadBalancer.

A connectivity and access model is required that:

• Avoids public exposure of any management API, like Kubernetes or Talos
• Works uniformly across different CSPs and environments
• Provides a reliable access path for both automation (Portal) and Platform Engineers
• Scales to many clusters across many providers
• Preserves sovereignty, minimal trust, and strong network isolation
• Provides means to allow customers to access their service instances in a secure way
• Supports multi-tenant network segmentation (organization-based isolation)

This ADR defines the secure cluster connectivity & API access model, including both orchestration and human access paths.

Considered Options¶

Primary Options¶

Netbird: A WireGuard-based peer-to-peer mesh VPN with a central control plane. Fully self-hostable under BSD-3-Clause license. The company (Netbird GmbH) is based in Berlin, Germany. Features include fine-grained access control policies, SSO/OIDC integration, group-based network segmentation, and a dedicated Kubernetes Operator. The self-hosted version has feature parity with the cloud offering. Supports multi-tenancy via MSP Portal for managing multiple tenant accounts.

Tailscale / Headscale: A polished WireGuard-based mesh VPN known for excellent user experience. The official control plane is proprietary SaaS hosted by Tailscale Inc. (US). Headscale is an open-source, community-maintained control server implementation that works with official Tailscale clients. Headscale limitations include: single tailnet scope, no Funnel/Serve features, requires static public IP, and lacks enterprise multi-tenancy features.

Nebula: A mesh VPN developed and open-sourced by Slack, proven at scale (50,000+ production hosts). Fully self-hosted with no SaaS dependency. Uses its own encryption (Noise Protocol Framework, not WireGuard). Requires manual certificate authority management and has a steeper learning curve. No built-in web UI or SSO integration: designed for infrastructure teams comfortable with CLI and automation.

Netmaker: A WireGuard-based mesh VPN with focus on enterprise features like site-to-site connectivity and Kubernetes multi-cluster networking. Self-hostable with SSPL license (similar restrictions to BSL). Offers managed SaaS option. Less mature than alternatives but has good Kubernetes integration via manifests and DaemonSets.

Dismissed Options¶

Pangolin: HTTP/HTTPS reverse proxy only: does not support native TCP protocols (PostgreSQL, Redis, etc.) required for customer service access.

ZeroTier: Licensed under BSL 1.1 which requires commercial licensing for any revenue-generating use. Self-hosted commercial deployments require custom enterprise pricing.

Firezone: The self-hostable version (0.7) reached end-of-life January 2024. Current version focuses on their managed SaaS offering; self-hosting is unsupported and APIs are unstable.

SSH Jumphost / Plain WireGuard / Classic VPN: Manual approaches that don't scale to 20-50+ clusters. Require per-cluster configuration and lack centralized management, SSO integration, and access control policies.

Service Mesh: Solves in-cluster service-to-service communication but doesn't address external access for Portal, engineers, or customers connecting from outside the cluster.

Comparison¶

Decision¶

We go with Netbird.

Reasoning:

• It ticks all the boxes of a modern VPN solution and goes beyond of only allowing access to the Kubernetes API, but can also serve as a "secure entrypoint" for service instances of customers.
• The multi-tenant approach of Netbird (See MSP Portal) matches the idea of Servala quite well.
• The software is fully open source under the BSD-3-Clause license.
• There is no closed source / enterprise version of it, it's possible to fully self-host it.
• Netbird GmbH, the company behind it, is based in Berlin, Germany.
• Pricing of the SaaS version is moderate, starts at $5 user per month with a good feature set (including SSO).
• Integration in Kubernetes is well done with the Kubernetes operator. The operator is only needed when we want to expose Kubernetes Services to the Netbird network for customer service-level access. All infrastructure-related access (Kubernetes API, Talos Machine API) can be solved without the operator.
• A Talos Linux extension for Netbird exists and works very well.
• Works well in restricted environments like APPUiO with user-space WireGuard.

The closest competitor and second choice would be Tailscale. The main drawback of Tailscale is that it relies on a commercial SaaS hosted control-plane which is under US jurisdiction. Self-hosting with Headscale would be possible, but it has a very limited feature set and not really made for a commercial business with enterprise requirements.

Implementation Approach¶

We use the Netbird Talos Extension installed on each cluster node. This approach was chosen over the Kubernetes Operator for infrastructure-level access.

How it works:

1. The Netbird Talos Extension is included in the Talos machine configuration
2. Upon booting, nodes self-register with Netbird and are automatically placed in the cluster group
3. A Netbird network is created with the Kubernetes API resource, using all nodes as routing peers to access the Kubernetes API VIP
4. This provides fault-tolerant access at both the Netbird level (multiple routing peers) and Kubernetes level (HA control plane)

Access configuration via Netbird policies:

• Kubernetes API (port 6443): Clients group can access the Kubernetes API through the node routing peers
• Talos Machine API (port 50000): Clients group can access the Talos Machine API on each node for cluster management
• Private network routing: The nodes' private network can be routed through the peers, giving engineers and the Servala Portal transparent access as if they were in the same network as the cluster

Benefits of this approach:

• No initial bastion host or LBaaS required for cluster bootstrapping
• Machines can be booted with their machine configurations via CloudInit (Netbird is already installed and bootstraps automatically)
• After boot, only talosctl bootstrap needs to be run to initialize the cluster
• Consistent access model across all environments and CSPs
• Provides a reliable access path for both automation (Servala Portal) and Platform Engineers

Consequences¶

We start with the SaaS-hosted control-plane of Netbird to reduce initial operational complexity. When using the SaaS version, we need to factor in the costs in the product pricing. There are some differences between self-hosted and SaaS version documented: Self-hosted vs. Cloud-hosted Netbird.

The Netbird SaaS is currently hosted on AWS, which contradicts our principles regarding cloud provider independence. Additionally, the number of machines that can be registered under their managed plans is limited, which may become a constraint as we scale. For now, using the SaaS offering is acceptable to get started, but in the long term we should evaluate migrating to the self-hosted version to maintain full sovereignty, alignment with our principles, and avoid scaling limitations.

We need to properly learn how Netbird works in detail and make sure to configure it in a secure way.