2023.05

2023.05

  • Imply
  • Ingest
  • Query
  • Visualize
  • Administer
  • Deploy

›Misc

Overview

  • About Imply administration

Manager

  • Using Imply Manager
  • Managing Imply clusters
  • Imply Manager security
  • Extensions

Users

  • Imply Manager users
  • Druid API access
  • Authentication and Authorization

    • Get started with Imply Hybrid Auth
    • Authentication
    • Local users
    • User roles
    • User groups
    • User sessions
    • Brute force attack detection
    • Identity provider integration
    • Okta OIDC integration
    • Okta SAML integration
    • LDAP integration
    • OAuth client authentication

Clarity

  • Monitoring
  • Set up SaaS Clarity
  • Cloudwatch monitoring
  • Metrics

Druid administration

  • Configuration reference
  • Logging
  • API reference
  • Druid design

    • Design
    • Segments
    • Processes and servers
    • Deep storage
    • Metadata storage
    • ZooKeeper

    Data management

    • Overview
    • Data updates
    • Data deletion
    • Schema changes
    • Compaction
    • Automatic compaction

    Security

    • Security overview
    • User authentication and authorization
    • LDAP auth
    • Dynamic Config Providers
    • Password providers
    • Authentication and Authorization
    • TLS support
    • Row and column level security

    Performance tuning

    • Basic cluster tuning
    • Segment size optimization
    • Mixed workloads
    • HTTP compression
    • Automated metadata cleanup

    View Manager

    • View Manager
    • View Manager API
    • Create a view
    • List views
    • Delete a view
    • Inspect view load status
  • Rolling updates
  • Using rules to drop and retain data
  • Alerts
  • Java runtime
  • Working with different versions of Apache Hadoop
  • Misc

    • dump-segment tool
    • reset-cluster tool
    • pull-deps tool
    • Deep storage migration
    • Export Metadata Tool
    • Metadata Migration

Metadata Migration

If you have been running an evaluation Druid cluster using the built-in Derby metadata storage and wish to migrate to a more production-capable metadata store such as MySQL or PostgreSQL, this document describes the necessary steps.

Shut down cluster services

To ensure a clean migration, shut down the non-coordinator services to ensure that metadata state will not change as you do the migration.

When migrating from Derby, the coordinator processes will still need to be up initially, as they host the Derby database.

Exporting metadata

Druid provides an Export Metadata Tool for exporting metadata from Derby into CSV files which can then be imported into your new metadata store.

The tool also provides options for rewriting the deep storage locations of segments; this is useful for deep storage migration.

Run the export-metadata tool on your existing cluster, and save the CSV files it generates. After a successful export, you can shut down the coordinator.

Initializing the new metadata store

Create database

Before importing the existing cluster metadata, you will need to set up the new metadata store.

The MySQL extension and PostgreSQL extension docs have instructions for initial database setup.

Update configuration

Update your Druid runtime properties with the new metadata configuration.

Create Druid tables

Druid provides a metadata-init tool for creating Druid's metadata tables. After initializing the Druid database, you can run the commands shown below from the root of the Druid package to initialize the tables.

In the example commands below:

  • lib is the Druid lib directory
  • extensions is the Druid extensions directory
  • base corresponds to the value of druid.metadata.storage.tables.base in the configuration, druid by default.
  • The --connectURI parameter corresponds to the value of druid.metadata.storage.connector.connectURI.
  • The --user parameter corresponds to the value of druid.metadata.storage.connector.user.
  • The --password parameter corresponds to the value of druid.metadata.storage.connector.password.

MySQL

cd ${DRUID_ROOT}
java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList="[\"mysql-metadata-storage\"]" -Ddruid.metadata.storage.type=mysql -Ddruid.node.type=metadata-init org.apache.druid.cli.Main tools metadata-init --connectURI="<mysql-uri>" --user <user> --password <pass> --base druid

PostgreSQL

cd ${DRUID_ROOT}
java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList="[\"postgresql-metadata-storage\"]" -Ddruid.metadata.storage.type=postgresql -Ddruid.node.type=metadata-init org.apache.druid.cli.Main tools metadata-init --connectURI="<postgresql-uri>" --user <user> --password <pass> --base druid

Import metadata

After initializing the tables, please refer to the import commands for your target database.

Restart cluster

After importing the metadata successfully, you can now restart your cluster.

Last updated on 2/16/2023
← Export Metadata Tool
  • Shut down cluster services
  • Exporting metadata
  • Initializing the new metadata store
    • Create database
    • Update configuration
    • Create Druid tables
    • Import metadata
    • Restart cluster
2023.05
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
Apache Druid forumsBlog
Copyright © 2023 Imply Data, Inc