Skip to main content

Provision dependencies for Lumi Enterprise on OpenShift

info

This information applies to Imply Lumi Enterprise deployments on OpenShift. Lumi Enterprise on OpenShift is an early access feature.

Imply Lumi Enterprise deployments depend on some services that you set up and manage outside of Lumi.

This topic describes how to configure the required dependencies for Lumi Enterprise and the corresponding Helm configurations:

For supported versions, see Required infrastructure.

In this topic, placeholder values are shown in uppercase—for example, DATABASE_HOST. Replace these with values for your own environment.

MySQL database​

Provide a MySQL database reachable from the cluster.

Lumi uses the database as its metadata store to store configuration, user information, and other operational data Lumi needs to function.

We recommend using different databases for control plane and data plane since each has different scaling requirements. This also isolates database features to a specific part of the system.

Create the control plane databases:

CREATE DATABASE IF NOT EXISTS manager;
CREATE DATABASE IF NOT EXISTS event_collector;
CREATE DATABASE IF NOT EXISTS iam_keys;
CREATE DATABASE IF NOT EXISTS enterprise;
CREATE DATABASE IF NOT EXISTS auth;
CREATE DATABASE IF NOT EXISTS update_client;
CREATE DATABASE IF NOT EXISTS lumi_ui;

Create the data plane database:

CREATE DATABASE IF NOT EXISTS lumi_data_plane;

Verify all databases were created:

SHOW DATABASES;

Lumi uses a single database user shared across all its services. To define separate parameters for the data plane, add the dataPlane block:

database:
  host: CONTROL_PLANE_DATABASE_HOST
  port: 3306
  username: CONTROL_PLANE_USERNAME
  password: CONTROL_PLANE_PASSWORD
  dataPlane:
    host: DATA_PLANE_DATABASE_HOST
    port: 3306
    username: DATA_PLANE_USERNAME
    password: DATA_PLANE_PASSWORD

We recommend starting with the following default values and adjusting them as needed:

  • Sizing: 2 vCPU, 16 GiB per instance, with headroom to scale up under load.
  • Topology: One writer and one standby replica with automatic failover.
  • Backups: 14-day retention.
  • Server parameters: Raise max_allowed_packet: 
    max_allowed_packet = 67108864  # 64 MiB

Kafka cluster​

Provide a Kafka cluster reachable from the Lumi namespace as follows:

  • Authentication: Use SASL with SCRAM-SHA-512 (recommended), SCRAM-SHA-256, or PLAIN. Unauthenticated PLAINTEXT is supported but not recommended.
  • Encryption: SASL_SSL is the default. The Kafka broker's TLS certificate must be signed by a publicly trusted CA. Use SASL_PLAINTEXT only if your Kafka brokers don't use TLS.
  • User: Create one Kafka user with permission to create topics, produce, consume, and manage consumer groups.
  • Topics: Lumi creates its topics automatically (iow_events, iow_events_slow, iow_dlq_events). Allow topic auto-creation for the Lumi user, or grant it topic-creation rights directly.

Note the Kafka bootstrap server address. You need it to connect Lumi to Kafka.

Example chart values:

kafka:
  bootstrap:
    servers: KAFKA_BROKER_1:9092,KAFKA_BROKER_2:9092
  security:
    protocol: SASL_SSL   # or SASL_PLAINTEXT
  sasl:
    mechanism: SCRAM-SHA-512
    username: KAFKA_USERNAME
    password: KAFKA_PASSWORD

We recommend starting with the following settings and adjusting them as needed:

  • Brokers: 3 brokers, each with 2 vCPU and 8 GiB of memory. Spread across separate failure domains, such as racks or availability zones.
  • Storage: Enough for at least 2 days of event retention at your expected ingest volume.

To estimate storage needs, multiply your ingest rate (in MB/s, after compression) by your retention period (in seconds) and by your replication factor to get total storage. Divide that by your broker count, then multiply by 1.3 to leave headroom for indexes, segment rollover, and spikes. This gives you the storage needed per broker.

info

You can use Streams for Apache Kafka to deploy a Kafka cluster on your OpenShift cluster.

S3-compatible object store​

Provide an S3-compatible object store for Lumi to store event data.

Create one bucket and optionally a key prefix for Lumi. You need a static access key and secret key pair with read, write, list, and delete permission on the bucket. Lumi doesn't support IAM roles and workload identity.

Example chart values:

objectStore:
  s3:
    accessKey: ACCESS_KEY
    secretKey: SECRET_KEY
    bucket: BUCKET_NAME
    prefix: lumi
    endpoint: S3_ENDPOINT_URL                    # omit for AWS S3
    region: us-east-1                            # required if not auto-detectable
    usePathStyle: true                           # required for most S3-compatible stores

Next step​

After you provision your dependencies, continue to Set up networking and ingress.