Security

Securing hosts on both public and private interfaces is an absolute necessity.

This is a tough one. Almost every single guide fails to bring the security topic to the table to the extent it deserves. One of the biggest misconceptions is that private networks are secure, but private does not mean secure. In fact, private networks are more often than not shared between many customers in the same data center. This might not be the case with all providers. It’s generally good advise to gain absolute certainty, what the actual conditions of a private network are.

Firewall

security/ufw

While there are definitely some people out there able to configure iptables reliably, the average mortal will cringe when glancing at the syntax of the most basic rules. Luckily, there are more approachable solutions out there. One of those is UFW, the uncomplicated firewall—a human friendly command line interface offering simple abstractions for managing complex iptables rules.

Assuming the secure public Kubernetes API runs on port 6443, SSH daemon on 22, plus 80 and 443 for serving web traffic, results in the following basic UFW configuration:

ufw allow ssh #  sshd on port 22, be careful to not get locked out!
ufw allow 6443 #  remote, secure Kubernetes API access
ufw allow 80
ufw allow 443
ufw default deny incoming #  deny traffic on every other port, on any interface
ufw enable

This ruleset will get slightly expanded in the upcoming sections.

Secure private networking

Kubernetes cluster members constantly exchange data with each other. A secure network overlay between hosts is not only the simplest, but also the most secure solution for making sure that a third party occupying the same network as our hosts won’t be able to eavesdrop on their private traffic. It’s a tedious job to secure every single service, as this task usually requires creating and distributing certificates across hosts, managing secrets in one way or another and, last but not least, configuring services to actually use encrypted means of communication. That’s why setting up a network overlay using VPN—which itself is a one-time effort requiring very little know how, and which naturally ensures secure inter-host communication for every possible service running now and in the future—is simply the best solution to address this problem.

When talking about VPN, there are generally two types of solutions:

• Traditional VPN services, running in userland, typically providing a tunnel interface
• IPsec, which is part of the Kernel and enables authentication and encryption on any existing interface

VPN software running in userland has in general a huge negative impact on network throughput as opposed to IPsec, which is much faster. Unfortunately, it’s quite a challenge to understand how the latter works. strongSwan is certainly one of the more approachable solutions, but setting it up for even the most basic needs is still accompanied by a steep learning curve.

Complexity is security’s worst contender.

A project called WireGuard supplies the best of both worlds at this point. Running as a Kernel module, it not only offers excellent performance, but is dead simple to set up and provides a tunnel interface out of the box. It may be disputed whether running VPN within the Kernel is a good idea, but then again alternatives running in userland such as tinc or fastd aren’t necessarily more secure. However, they are an order of magnitude slower and typically harder to configure.

WireGuard setup

security/wireguard

As mentioned above, WireGuard runs as a Kernel module and needs to be compiled against the headers of the Kernel running on the host. In most cases it’s enough to follow the simple instructions found here: WireGuard Installation.

Scaleway uses custom Kernel versions which makes the installation process a little more complex. Fortunately, they provide a shell script to download the required headers without much hassle.

Once WireGuard has been compiled, it’s time to create the configuration files. Each host should connect to its peers to create a secure network overlay via a tunnel interface called wg0. Let’s assume the setup consists of three hosts and each one will get a new VPN IP address in the 10.0.1.1/24 range:

In this scenario, a configuration file for kube1 would look like this:

#  /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.1.1
PrivateKey = <PRIVATE_KEY_KUBE1>
ListenPort = 51820
[Peer]
PublicKey = <PUBLIC_KEY_KUBE2>
AllowedIps = 10.0.1.2/32
Endpoint = 10.8.23.94:51820
[Peer]
PublicKey = <PUBLIC_KEY_KUBE3>
AllowedIps = 10.0.1.3/32
Endpoint = 10.8.23.95:51820

To simplify the creation of private and public keys, the following command can be used to generate and print the necessary key-pairs:

for i in 1 2 3; do
  private_key=$(wg genkey)
  public_key=$(echo $private_key | wg pubkey)
  echo "Host $i private key: $private_key"
  echo "Host $i public key:  $public_key"
done

After creating a file named /etc/wireguard/wg0.conf on each host containing the correct IP addresses and public and private keys, configuration is basically done.

What’s left is to add the following firewall rules:

ufw allow in on eth1 to any port 51820 #  open VPN port on private network interface
ufw allow in on wg0 #  allow all traffic on VPN tunnel interface
ufw reload

Executing the command systemctl start wg-quick@wg0 on each host will start the VPN service and, if everything is configured correctly, the hosts should be able to establish connections between each other. Traffic can now be routed securely using the VPN IP addresses (10.0.1.1–10.0.1.3).

In order to check whether the connections are established successfully, wg show comes in handy:

$ wg show
interface: wg0
  public key: 5xKk9...
  private key: (hidden)
  listening port: 51820
peer: HBCwy...
  endpoint: 10.8.23.199:51820
  allowed ips: 10.0.1.1/32
  latest handshake: 25 seconds ago
  transfer: 8.76 GiB received, 25.46 GiB sent
peer: KaRMh...
  endpoint: 10.8.47.93:51820
  allowed ips: 10.0.1.3/32
  latest handshake: 59 seconds ago
  transfer: 41.86 GiB received, 25.09 GiB sent

Last but not least, run systemctl enable wg-quick@wg0 to launch the service whenever the system boots.