Deploy Alluxio on Kubernetes

If hosting a private helm repository or using native Kubernetes specifications, extract the Kubernetes specifications required to deploy Alluxio from the Docker image.

$ id=$(docker create alluxio/alluxio:2.6.2)
$ docker cp $id:/opt/alluxio/integration/kubernetes/ - > kubernetes.tar
$ docker rm -v $id 1>/dev/null
$ tar -xvf kubernetes.tar
$ cd kubernetes

Note: Embedded Journal requires a Persistent Volume for each master Pod to be provisioned and is the preferred HA mechanism for Alluxio on Kubernetes. The volume, once claimed, is persisted across restarts of the master process.

When using the UFS Journal an Alluxio master can also be configured to use a persistent volume for storing the journal. If you are using UFS journal and use an external journal location like HDFS, the rest of this section can be skipped.

There are multiple ways to create a PersistentVolume. This is an example which defines one with hostPath:

# Name the file alluxio-master-journal-pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: alluxio-journal-0
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /tmp/alluxio-journal-0

Note:

• By default each journal volume should be at least 1Gi, because each Alluxio master Pod will have one PersistentVolumeClaim that requests for 1Gi storage. You will see how to configure the journal size in later sections.
• If this hostPath is not already present on the host, Kubernetes can be configured to create it. However the assigned user:group permissions may prevent the Alluxio masters & workers from accessing it. Please ensure the permissions are set to allow the pods to access the directory.
  • See the Kubernetes volume docs for more details
  • From Alluxio v2.1 on, Alluxio Docker containers will run as non-root user alluxio with UID 1000 and GID 1000 by default.

Then create the persistent volume with kubectl:

$ kubectl create -f alluxio-master-journal-pv.yaml

There are other ways to create Persistent Volumes as documented here.

Alluxio workers use host networking with the physical host IP as the hostname. Check the cluster firewall if an error such as the following is encountered:

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Host is unreachable: <host>/<IP>:29999

• Check that <host> matches the physical host address and is not a virtual container hostname. Ping from a remote client to check the address is resolvable.

  $ ping <host>
  

• Verify that a client can connect to the workers on the ports specified in the worker deployment specification. The default ports are [29998, 29999, 29996, 30001, 30002, 30003]. Check access to the given port from a remote client using a network utility such as ncat:

  $ nc -zv <IP> 29999
  

From Alluxio v2.1 on, Alluxio Docker containers will run as non-root user alluxio with UID 1000 and GID 1000 by default. Kubernetes hostPath volumes are only writable by root so you need to update the permission accordingly.

To change the log level for Alluxio servers (master and workers), use the CLI command logLevel as follows:

Access the Alluxio CLI from the master Pod.

$ kubectl exec -ti alluxio-master-0 /bin/bash

From the master Pod, execute the following:

$ alluxio logLevel --level DEBUG --logName alluxio

The Alluxio master and job master run as separate containers of the master Pod. Similarly, the Alluxio worker and job worker run as separate containers of a worker Pod. Logs can be accessed for the individual containers as follows.

Master:

$ kubectl logs -f alluxio-master-0 -c alluxio-master

Worker:

$ kubectl logs -f alluxio-worker-<id> -c alluxio-worker

Job Master:

$ kubectl logs -f alluxio-master-0 -c alluxio-job-master

Job Worker:

$ kubectl logs -f alluxio-worker-<id> -c alluxio-job-worker

In order for an application container to mount the hostPath volume, the node running the container must have the Alluxio FUSE daemon running. The default spec alluxio-fuse.yaml runs as a DaemonSet, launching an Alluxio FUSE daemon on each node of the cluster.

If there are issues accessing Alluxio using the POSIX API:

1. Identify the node the application container ran on using the command kubectl describe pods or the dashboard.
2. Use the command kubectl describe nodes <node> to identify the alluxio-fuse Pod running on that node.
3. Tail logs for the identified Pod to view any errors encountered: kubectl logs -f alluxio-fuse-<id>.

Alluxio workers create a domain socket used for short-circuit access by default. On Mac OS X, Alluxio workers may fail to start if the location for this domain socket is a path which is longer than what the filesystem accepts.

2020-07-27 21:39:06,030 ERROR GrpcDataServer - Alluxio worker gRPC server failed to start on /opt/domain/1d6d7c85-dee0-4ac5-bbd1-86eb496a2a50
java.io.IOException: Failed to bind
	at io.grpc.netty.NettyServer.start(NettyServer.java:252)
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
	at alluxio.grpc.GrpcServer.lambda$start$0(GrpcServer.java:77)
	at alluxio.retry.RetryUtils.retry(RetryUtils.java:39)
	at alluxio.grpc.GrpcServer.start(GrpcServer.java:77)
	at alluxio.worker.grpc.GrpcDataServer.<init>(GrpcDataServer.java:107)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at alluxio.util.CommonUtils.createNewClassInstance(CommonUtils.java:273)
	at alluxio.worker.DataServer$Factory.create(DataServer.java:47)
	at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:162)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:46)
	at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:38)
	at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:72)
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Filename too long

If this is the case, set the following properties to limit the path length:

• alluxio.worker.data.server.domain.socket.as.uuid=false
• alluxio.worker.data.server.domain.socket.address=/opt/domain/d

Note: You may see performance degradation due to lack of node locality.

This is most likely caused due to the Kubernetes configured Pod resource limits having the limits.memory set too low.

Firstly, double check the configured values for your Alluxio worker Pod limits.memory. Note that the Pod consists of two containers, each with their own resource limits.

Check the configured resource requests and limits using kubectl describe pod, kubectl get pod, or equivalent Kube API requests. eg.,

$ kubectl get po -o json alluxio-worker-xxxxx | jq '.spec.containers[].resources'
{
  "limits": {
    "cpu": "4",
    "memory": "4G"
  },
  "requests": {
    "cpu": "1",
    "memory": "2G"
  }
}
{
  "limits": {
    "cpu": "4",
    "memory": "4G"
  },
  "requests": {
    "cpu": "1",
    "memory": "1G"
  }
}

If you used the Helm chart, the default values are:

worker:
  resources:
    limits:
      cpu: "4"
      memory: "4G"
    requests:
      cpu: "1"
      memory: "2G"

jobWorker:
  resources:
    limits:
      cpu: "4"
      memory: "4G"
    requests:
      cpu: "1"
      memory: "1G"

• Even if you did not configure any values with Helm, you may still have resource limits in place due to a LimitRange applied to your namespace

Next, ensure that the nodes that the Alluxio worker pods are running on have sufficient resources matching your configured values. You can check that the nodes you intend to schedule Alluxio worker Pods on have sufficient resources to meet your requests using kubectl describe node, kubectl get node, or equivalent Kube API requests. eg.,

$ kubectl get no -o json k8sworkernode-0 | jq '.status.allocatable'
{
  "cpu": "8",
  "ephemeral-storage": "123684658586",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "64886128Ki",
  "pods": "110"
}

Isolating Alluxio worker Pods from other Pods in your Kubernetes cluster can be accomplished with the help of node selectors and node taints + tolerations.

• Keep in mind that the Alluxio worker Pod definition uses a DaemonSet, so there will be worker Pods assigned to all eligible nodes

Next, verify the Alluxio workers’ configured ramdisk sizes (if any). See the list of Alluxio configuration properties for additional details.

• If you used the Helm chart, the Alluxio site properties are configured using properties. eg.,

properties:
  alluxio.worker.ramdisk.size: 2G
  alluxio.worker.tieredstore.levels: 1
  alluxio.worker.tieredstore.level0.alias: MEM
  alluxio.worker.tieredstore.level0.dirs.mediumtype: MEM
  alluxio.worker.tieredstore.level0.dirs.path: /dev/shm
  alluxio.worker.tieredstore.level0.dirs.quota: 2G

• Otherwise, you can view and modify the site properties in the alluxio-config ConfigMap. eg.,

$ kubectl get cm -o json alluxio-config | jq '.data.ALLUXIO_WORKER_JAVA_OPTS'
"-Dalluxio.worker.ramdisk.size=2G
-Dalluxio.worker.tieredstore.levels=1
-Dalluxio.worker.tieredstore.level0.alias=MEM
-Dalluxio.worker.tieredstore.level0.dirs.mediumtype=MEM
-Dalluxio.worker.tieredstore.level0.dirs.path=/dev/shm
-Dalluxio.worker.tieredstore.level0.dirs.quota=2G "

NOTE: Our DaemonSet uses emptyDir volumes as the Alluxio worker Pod’s ramdisk in Kubernetes.

spec:
  template:
    spec:
      volumes:
        - name: mem
          emptyDir:
            medium: "Memory"
            sizeLimit: 1G

This results in the following nuances:

• sizeLimit has no effect on the size of the allocated ramdisk unless the SizeMemoryBackedVolumes feature gate is enabled
• As stated in the Kubernetes emptyDir documentation, if no size is specified then memory-backed emptyDir volumes will have capacity allocated equal to half the available memory on the host node. This capacity is reflected inside of your containers (for example when running df -u). However if the combined size of your ramdisk and container memory usage exceeds the pod’s limits.memory then the Kubernetes scheduler will trigger an OOMKilled on that pod. This is a very likely overlooked source of memory consumption in Alluxio worker Pods.

Lastly, verify the Alluxio worker JVM heap and off-heap maximum capacities. These are configured with the JVM flags -Xmx/-XX:MaxHeapSize and -XX:MaxDirectMemorySize respectively.

• See the Oracle Java documentation for more details.

To adjust those values, you would have to manually update the (...)_JAVA_OPTS environment variables in the alluxio-config ConfigMap. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: alluxio-config
data:
  ALLUXIO_JAVA_OPTS: |-
    -Xmx2g -Dalluxio.master.hostname=alluxio-master-0 ...
  ALLUXIO_MASTER_JAVA_OPTS: |-
    -Dalluxio.master.hostname=${ALLUXIO_MASTER_HOSTNAME}
  ALLUXIO_JOB_MASTER_JAVA_OPTS: |-
    -Dalluxio.master.hostname=${ALLUXIO_MASTER_HOSTNAME}
  ALLUXIO_WORKER_JAVA_OPTS: |-
    -XX:MaxDirectMemorySize=2g -Dalluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME} ...
  ALLUXIO_JOB_WORKER_JAVA_OPTS: |-
    -XX:MaxDirectMemorySize=1g -Dalluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME} ...
  ALLUXIO_FUSE_JAVA_OPTS: |-
    -Dalluxio.user.hostname=${ALLUXIO_CLIENT_HOSTNAME} -XX:MaxDirectMemorySize=2g
  ALLUXIO_WORKER_TIEREDSTORE_LEVEL0_DIRS_PATH: /dev/shm

Thus to avoid worker Pods running into OOMKilled errors,

1. Verify that the nodes your Alluxio worker Pods are scheduled on have sufficient memory to satisfy all the limits.memory specifications assigned.
2. Ensure you have configured alluxio.worker.ramdisk.size and alluxio.worker.tieredstore.level0.dirs.quota low enough such that the memory consumed by the ramdisk combined with the JVM memory options (-Xmx, -XX:MaxDirectMemorySize) do not exceed the Pod’s limits.memory. It is recommended to allow for some overhead as memory may be consumed by other processes as well.

Aside: There is currently an open issue in Alluxio where Alluxio’s interpretation of byte sizes differs from Kubernetes (due to Kubernetes distinguishing between “-bibytes”). This is unlikely to cause OOMKilled errors unless you are operating on very tight memory margins.