TLS Encryption on Alluxio in Kubernetes

Setup TLS for Alluxio Cluster
TLS termination at Proxy

Alluxio supports encryption of the network communication between services with TLS. Refer here for more information about the general usage of TLS.

Enabling TLS on Alluxio in Kubernetes is to some extent different from bare metal machines because of the nature of kubernetes that pods, where the services run, are ephemeral. Since the keystore needs to include the IP address or DNS of the server but a restarted pod doesn’t keep the old IP/DNS, we have to leverage the Kubernetes Service to ensure the restarted pod is reachable/ can reach other components of the system.

Setup TLS for Alluxio Cluster

To enable TLS encryption on Alluxio network traffic on Kubernetes, follow the steps below:

1. Generate keypair

Use keytool to generate a keystore/truststore pair, as described in here and here Note: the generated keystore needs to contain the ip/hostname of all Alluxio masters/workers, otherwise the verification would fail.

The hostnames of Alluxio master pods follow the pattern alluxio-master-x, where x ranges from 0 to # of master pods-1. The hostnames of Alluxio worker pods follow the pattern alluxio-worker-x, where x ranges from 0 to # of worker pods-1.

Therefore, you may need to append the SAN (Subject Alternative Name) extension to the keystore generation command. For example,

$ keytool -genkeypair ... -ext "SAN=DNS:alluxio-master-0,DNS:alluxio-worker-0,DNS:alluxio-worker-1,DNS:alluxio-worker-2"

2. Create Kubernetes secrets of the keypair

Create secrets in Kubernetes with the created keystore and truststore. For example,

$ kubectl create secret generic alluxio-tls-keystore --from-file=/alluxio/keystore.jks
$ kubectl create secret generic alluxio-tls-truststore --from-file=/alluxio/truststore.jks

3. Configure the secrets for the pods

Configure the secrets in helm chart values.yaml file under .secrets section. For example,

secrets:
  master:
    alluxio-tls-keystore: alluxio-tls-keystore
    alluxio-tls-truststore: alluxio-tls-truststore
  worker:
    alluxio-tls-keystore: alluxio-tls-keystore
    alluxio-tls-truststore: alluxio-tls-truststore

4. Configure the keypair for Alluxio

Configure the properties for Alluxio in your values.yaml file, similar to here. By default, the secrets are mounted under /secrets/<secret-name>/<file-name>. Configure the keystore path and the truststore path accordingly. For example,

properties:
  ...
  alluxio.network.tls.keystore.path: "/secrets/alluxio-tls-keystore/keystore.jks"
  alluxio.network.tls.truststore.path: "/secrets/alluxio-tls-truststore/truststore.jks"
  ...

5. Configure Alluxio worker deployment

To make sure the worker pod still works with the existing keystore/truststore after restart, Statefulset with service is required for worker deployment, instead of the default Daemonset deployment type. To use Statefulset, set worker.useStatefulset to true in your values.yaml file, and set the number of worker pods to spawn. For example,

worker:
  useStatefulset: true
  count: 3 # <---- number of worker pods you need

6. Deploy Alluxio and check TLS is enabled

Deploy Alluxio Cluster via helm install. To make sure TLS is enabled, check the logs of the respective services. Similar lines should be logged which indicate the communication iss encrypted by TLS:

2022-10-31 00:53:26,306 INFO  EnterpriseSslContextProvider - TLS server using keystore: /secrets/alluxio-tls-keystore/keystore.jks alias: key
2022-10-31 00:53:26,306 INFO  EnterpriseSslContextProvider - TLS client using truststore: /secrets/alluxio-tls-truststore/truststore.jks alias: key

TLS termination at Proxy

Users are able to do TLS termination at Proxy pod, that is, the point where the client requests get decrypted. The Alluxio proxy acts as the server here, so a keystore is required for the proxy. Similar to configure TLS for Alluxio, a kubernetes secret needs to be created from the keystore file, then configured in the helm chart values.yaml file under .secrets section. For example,

secrets:
  master:
    ...
  worker:
    ...
  proxy:
    client-tls-keystore: client-tls-keystore

Finally, configure the property for Alluxio proxy in the same configuration file, for example,

proxy:
  properties:
    alluxio.network.tls.keystore.path: "/secrets/alluxio-tls-keystore/keystore.jks"
    alluxio.network.tls.keystore.password: "<storepass>"
    alluxio.network.tls.keystore.key.password: "<keypass>"

Categories