2022.06

2022.06

  • Imply
  • Ingest
  • Query
  • Visualize
  • Administer
  • Deploy

›Misc

Overview

  • About Imply administration

Manager

  • Using Imply Manager
  • Managing Imply clusters
  • Imply Manager security
  • Extensions

Users

  • Imply Manager users
  • Druid API access
  • Authentication and Authorization

    • Get started with Imply Hybrid Auth
    • Authentication
    • Local users
    • User roles
    • User groups
    • User sessions
    • Brute force attack detection
    • Identity provider integration
    • Okta OIDC integration
    • Okta SAML integration
    • LDAP integration
    • OAuth client authentication

Clarity

  • Monitoring
  • Set up Clarity
  • Cloudwatch monitoring
  • Metrics

Druid administration

  • Configuration reference
  • Logging
  • Druid design

    • Design
    • Segments
    • Processes and servers
    • Deep storage
    • Metadata storage
    • ZooKeeper

    Security

    • Security overview
    • User authentication and authorization
    • LDAP auth
    • Dynamic Config Providers
    • Password providers
    • Authentication and Authorization
    • TLS support
    • Row and column level security

    Performance tuning

    • Basic cluster tuning
    • Segment size optimization
    • Mixed workloads
    • HTTP compression
    • Automated metadata cleanup
  • API reference
  • View Manager

    • View Manager
    • View Manager API
    • Create a view
    • List views
    • Delete a view
    • Inspect view load status
  • Rolling updates
  • Retaining or automatically dropping data
  • Alerts
  • Working with different versions of Apache Hadoop
  • Misc

    • dump-segment tool
    • reset-cluster tool
    • pull-deps tool
    • Deep storage migration
    • Export Metadata Tool
    • Metadata Migration

pull-deps tool

pull-deps is an Apache Druid tool that can pull down dependencies to the local repository and lay dependencies out into the extension directory as needed.

pull-deps has several command line options, they are as follows:

-c or --coordinate (Can be specified multiple times)

Extension coordinate to pull down, followed by a maven coordinate, e.g. org.apache.druid.extensions:mysql-metadata-storage

-h or --hadoop-coordinate (Can be specified multiply times)

Apache Hadoop dependency to pull down, followed by a maven coordinate, e.g. org.apache.hadoop:hadoop-client:2.4.0

--no-default-hadoop

Don't pull down the default hadoop coordinate, i.e., org.apache.hadoop:hadoop-client:2.3.0. If -h option is supplied, then default hadoop coordinate will not be downloaded.

--clean

Remove existing extension and hadoop dependencies directories before pulling down dependencies.

-l or --localRepository

A local repository that Maven will use to put downloaded files. Then pull-deps will lay these files out into the extensions directory as needed.

-r or --remoteRepository

Add a remote repository. Unless --no-default-remote-repositories is provided, these will be used after https://repo1.maven.org/maven2/ and http://metamx.artifactoryonline.com/metamx/pub-libs-releases-local

--no-default-remote-repositories

Don't use the default remote repositories, only use the repositories provided directly via --remoteRepository.

-d or --defaultVersion

Version to use for extension coordinate that doesn't have a version information. For example, if extension coordinate is org.apache.druid.extensions:mysql-metadata-storage, and default version is 2022.06.0-iap, then this coordinate will be treated as org.apache.druid.extensions:mysql-metadata-storage:2022.06.0-iap

--use-proxy

Use http/https proxy to send request to the remote repository servers. --proxy-host and --proxy-port must be set explicitly if this option is enabled.

--proxy-type

Set the proxy type, Should be either http or https, default value is https.

--proxy-host

Set the proxy host. e.g. proxy.com.

--proxy-port

Set the proxy port number. e.g. 8080.

--proxy-username

Set a username to connect to the proxy, this option is only required if the proxy server uses authentication.

--proxy-password

Set a password to connect to the proxy, this option is only required if the proxy server uses authentication.

To run pull-deps, you should

  1. Specify druid.extensions.directory and druid.extensions.hadoopDependenciesDir, these two properties tell pull-deps where to put extensions. If you don't specify them, default values will be used, see Configuration.

  2. Tell pull-deps what to download using -c or -h option, which are followed by a maven coordinate.

Example:

Suppose you want to download mysql-metadata-storage and hadoop-client(both 2.3.0 and 2.4.0) with a specific version, you can run pull-deps command with -c org.apache.druid.extensions:mysql-metadata-storage:2022.06.0-iap, -h org.apache.hadoop:hadoop-client:2.3.0 and -h org.apache.hadoop:hadoop-client:2.4.0, an example command would be:

java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --clean -c org.apache.druid.extensions:mysql-metadata-storage:2022.06.0-iap -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0

Because --clean is supplied, this command will first remove the directories specified at druid.extensions.directory and druid.extensions.hadoopDependenciesDir, then recreate them and start downloading the extensions there. After finishing downloading, if you go to the extension directories you specified, you will see

tree extensions
extensions
└── mysql-metadata-storage
    └── mysql-metadata-storage-2022.06.0-iap.jar
tree hadoop-dependencies
hadoop-dependencies/
└── hadoop-client
    ├── 2.3.0
    │   ├── activation-1.1.jar
    │   ├── avro-1.7.4.jar
    │   ├── commons-beanutils-1.7.0.jar
    │   ├── commons-beanutils-core-1.8.0.jar
    │   ├── commons-cli-1.2.jar
    │   ├── commons-codec-1.4.jar
    ..... lots of jars
    └── 2.4.0
        ├── activation-1.1.jar
        ├── avro-1.7.4.jar
        ├── commons-beanutils-1.7.0.jar
        ├── commons-beanutils-core-1.8.0.jar
        ├── commons-cli-1.2.jar
        ├── commons-codec-1.4.jar
    ..... lots of jars

Note that if you specify --defaultVersion, you don't have to put version information in the coordinate. For example, if you want mysql-metadata-storage to use version 2022.06.0-iap, you can change the command above to

java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --defaultVersion 2022.06.0-iap --clean -c org.apache.druid.extensions:mysql-metadata-storage -h org.apache.hadoop:hadoop-client:2.3.0 -h org.apache.hadoop:hadoop-client:2.4.0

Please note to use the pull-deps tool you must know the Maven groupId, artifactId, and version of your extension.

For Druid community extensions listed here, the groupId is "org.apache.druid.extensions.contrib" and the artifactId is the name of the extension.

Last updated on 3/30/2021
← reset-cluster toolDeep storage migration →
2022.06
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
Apache Druid forumsBlog
Copyright © 2022 Imply Data, Inc