Size and prepare nodes for Lumi Enterprise on OpenShift

info

This information applies to Imply Lumi Enterprise deployments on OpenShift.
Lumi Enterprise on OpenShift is an early access feature.

This topic explains how to size the OpenShift nodes that run Imply Lumi Enterprise, plan for autoscaling, and prepare the nodes with local disks and kernel settings.

In this topic, a Lumi node is an instance of one of the node types in the table, and an OpenShift node is a machine in your cluster. For more information on nodes that Lumi deploys, see Lumi infrastructure.

In this topic, placeholder values are shown in uppercase—for example, NODE_NAME. Replace them with values for your own environment.

Node types and recommended sizes

Lumi uses amd64 node architecture. You can use arm64 nodes for some node types; contact your Imply support representative before planning a mixed-architecture deployment.

You can provide explicit resources for any node type, workerPools for virtual compute pools, and storage volumes (storageVolumes, tmpStorageVolume, storageVolume).

We recommend starting with the following sizes for Lumi on OpenShift bare metal:

Node type	Description	Count	CPU	Memory per node	Local disk
Data	Serves frequently accessed data with low latency and high performance. Data nodes power the hot tier.	2	8 vCPU	64 GiB	Can handle up to 5 TiB, should scale based on requirements
Broker	Routes queries to the appropriate nodes. Lumi brokers are unrelated to Kafka brokers.	2	4 vCPU	16 GiB	—
Coordinator	Manages data availability and distribution, enforces data retention rules, and oversees ingestion tasks.	2	4 vCPU	16 GiB	—
Ingestion	Reads event data from Kafka, then transforms, indexes, and partitions the data into files. Each ingestion task runs in its own pod.	Scales with load	4 vCPU	16 GiB per task pod	50 GiB scratch per task pod
System and infrastructure	Runs the services that support the deployment, such as the gateway, authentication, and event collector.	2 or more (scales with load)	4 vCPU	16 GiB	—

Each node type runs several Lumi services, which share the CPU and memory for that node type. Except for system and infrastructure, we recommend a dedicated node pool for each node type, with one OpenShift node for each Lumi node.

info

Make sure your OpenShift nodes have enough available CPU and memory for the resources you set for each node type. Otherwise, pods for those node types may fail to schedule.

Autoscaling

Lumi creates and removes pods as it operates. Your OpenShift cluster needs to have sufficient nodes to run the pods. If your cluster doesn't have free nodes when these pods need to schedule, the pods stay Pending, and the upgrade or ingestion stalls until you add capacity. We recommend enabling node autoscaling so your cluster provisions nodes on demand.

Consider the following when deciding whether to add autoscaling:

• Ingestion: Every ingestion or compaction task runs as a new pod that needs a free node. As more tasks run concurrently, Lumi spins up more nodes. If none are available, tasks queue and ingestion falls behind.
• Upgrades: Lumi upgrades itself automatically. The data node upgrades using a canary rollout, where new data pods start and load data before old ones are removed, so no data goes offline. By default, the canary is the size of the whole node, so during an upgrade you temporarily need up to twice as many data nodes as usual—for example, 4 instead of 2. Other nodes also briefly run an old and a new pod side by side.
• Infrastructure autoscaling: The gateway, authentication, and event collector replicas grow with traffic.

In OpenShift, you enable node autoscaling by deploying a cluster autoscaler and machine autoscalers for each node pool in your cluster. For more information, see Applying autoscaling to an OpenShift Container Platform cluster in OpenShift documentation.

Node autoscaling is typically available on installer-provisioned infrastructure (IPI), including bare metal. It's typically not available on user-provisioned infrastructure (UPI) or platform: none installs.

If node autoscaling isn't available in your environment, use the following workarounds:

• Keep spare nodes ready at all times. You need to have enough nodes for your peak concurrent ingestion tasks, plus a second set of data nodes to cover upgrades.

• Watch for Pending pods in the Lumi namespace and add nodes when you see them:

  oc get pods -n NAMESPACE --field-selector=status.phase=Pending

• Plan to add ingest capacity ahead of time as your ingestion volume grows.

For more information, see the following OpenShift documentation:

• Overview of machine management
• Applying autoscaling to an OKD cluster
• Overview of nodes
• Postinstallation configuration overview

Local disks for data nodes

Data nodes perform best on local Non-Volatile Memory Express® (NVMe®) storage. Format and mount the disks on each data node before you install Lumi, then pass the mount path to the data pods with hostPath:

data:
  resources:
    requests:
      cpu: "8"
      memory: 32Gi
  count: 3
  storageVolumes:
    - maxSize: 1500Gi
      volume:
        hostPath:
          path: PREPARED_HOST_PATH

Replace PREPARED_HOST_PATH with the directory where you mounted the disk.

Kernel settings

Set the following sysctl values on your Lumi nodes to prevent them from running out of resources. These properties are shared between all workloads on the node.

vm.max_map_count=2000000
kernel.threads-max=999999

You can use the Node Tuning Operator to apply this on a classic OpenShift cluster:

1. Create druid-sysctl-tuned.yaml:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: druid-sysctl
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: druid-sysctl-profile
       data: |
         [main]
         summary=Sysctls required by Lumi's Druid data nodes
         include=openshift-node
         [sysctl]
         vm.max_map_count=2000000
         kernel.threads-max=999999
     recommend:
     - priority: 20
       profile: druid-sysctl-profile
       operand:
         tunedConfig:
           reapply_sysctl: false

2. Optionally add a match entry under recommend to scope to only your Lumi nodes.

3. Apply the file:

   oc apply -f druid-sysctl-tuned.yaml

4. Verify the change was picked up:

   oc get tuned -n openshift-cluster-node-tuning-operator
   
   oc get profile -n openshift-cluster-node-tuning-operator
   
   oc debug node/NODE_NAME -- chroot /host sysctl vm.max_map_count kernel.threads-max

Next step

After you size and prepare your nodes, continue to Configure node labels and taints.