Skip to main content

Manual compaction

In Apache Druid, compaction is a special type of ingestion task that reads data from a Druid datasource and writes it back into the same datasource. A common use case for this is to optimally size segments after ingestion to improve query performance.

You can perform manual compaction where you submit a one-time compaction task for a specific interval. Generally, you don't need to do this if you use automatic compaction, which is recommended for most workloads.

Setting up manual compaction

Compaction tasks merge all segments for the defined interval according to the following syntax:

"type": "compact",
"id": <task_id>,
"dataSource": <task_datasource>,
"ioConfig": <IO config>,
"dimensionsSpec": <custom dimensionsSpec>,
"transformSpec": <custom transformSpec>,
"metricsSpec": <custom metricsSpec>,
"tuningConfig": <parallel indexing task tuningConfig>,
"granularitySpec": <compaction task granularitySpec>,
"context": <task context>
typeTask type. Set the value to compact.Yes
idTask IDNo
dataSourceData source name to compactYes
ioConfigI/O configuration for compaction task. See Compaction I/O configuration for details.Yes
dimensionsSpecWhen set, the compaction task uses the specified dimensionsSpec rather than generating one from existing segments. See Compaction dimensionsSpec for details.No
transformSpecWhen set, the compaction task uses the specified transformSpec rather than using null. See Compaction transformSpec for details.No
metricsSpecWhen set, the compaction task uses the specified metricsSpec rather than generating one from existing segments.No
segmentGranularityDeprecated. Use granularitySpec.No
tuningConfigTuning configuration for parallel indexing. awaitSegmentAvailabilityTimeoutMillis value is not supported for compaction tasks. Leave this parameter at the default value, 0.No
granularitySpecWhen set, the compaction task uses the specified granularitySpec rather than generating one from existing segments. See Compaction granularitySpec for details.No
contextTask contextNo

Note: Use granularitySpec over segmentGranularity and only set one of these values. If you specify different values for these in the same compaction spec, the task fails.

To control the number of result segments per time chunk, you can set maxRowsPerSegment or numShards.


You can run multiple compaction tasks in parallel. For example, if you want to compact the data for a year, you are not limited to running a single task for the entire year. You can run 12 compaction tasks with month-long intervals.

A compaction task internally generates an index or index_parallel task spec for performing compaction work with some fixed parameters. For example, its inputSource is always the druid input source, and dimensionsSpec and metricsSpec include all dimensions and metrics of the input segments by default.

Compaction tasks typically fetch all relevant segments prior to launching any subtasks, unless the following properties are all set to non-null values. It is strongly recommended to set them to non-null values to maximize performance and minimize disk usage of the compact task:

Compaction tasks exit without doing anything and issue a failure status code in either of the following cases:

  • If the interval you specify has no data segments loaded.
  • If the interval you specify is empty.

Note that the metadata between input segments and the resulting compacted segments may differ if the metadata among the input segments differs as well. If all input segments have the same metadata, however, the resulting output segment will have the same metadata as all input segments.

Manual compaction task example

The following JSON illustrates a compaction task to compact all segments within the interval 2020-01-01/2021-01-01 and create new segments:

"type": "compact",
"dataSource": "wikipedia",
"ioConfig": {
"type": "compact",
"inputSpec": {
"type": "interval",
"interval": "2020-01-01/2021-01-01"
"granularitySpec": {
"segmentGranularity": "day",
"queryGranularity": "hour"

granularitySpec is an optional field. If you don't specify granularitySpec, Druid retains the original segment and query granularities when compaction is complete.

Compaction I/O configuration

The compaction ioConfig requires specifying inputSpec as follows:

typeTask type. Set the value to compact.noneYes
inputSpecSpecification of the target interval or segments.noneYes
dropExistingIf true, the task replaces all existing segments fully contained by either of the following:
- the interval in the interval type inputSpec.
- the umbrella interval of the segments in the segment type inputSpec.
If compaction fails, Druid does not change any of the existing segments.
WARNING: dropExisting in ioConfig is a beta feature.
allowNonAlignedIntervalIf true, the task allows an explicit segmentGranularity that is not aligned with the provided interval or segments. This parameter is only used if segmentGranularity is explicitly provided.

This parameter is provided for backwards compatibility. In most scenarios it should not be set, as it can lead to data being accidentally overshadowed. This parameter may be removed in a future release.

The compaction task has two kinds of inputSpec:

Interval inputSpec

typeTask type. Set the value to interval.Yes
intervalInterval to compact.Yes

Segments inputSpec

typeTask type. Set the value to segments.Yes
segmentsA list of segment IDs.Yes

Compaction dimensions spec

dimensionsA list of dimension names or objects. Cannot have the same column in both dimensions and dimensionExclusions. Defaults to null, which preserves the original dimensions.No
dimensionExclusionsThe names of dimensions to exclude from compaction. Only names are supported here, not objects. This list is only used if the dimensions list is null or empty; otherwise it is ignored. Defaults to [].No

Compaction transform spec

filterThe filter conditionally filters input rows during compaction. Only rows that pass the filter will be included in the compacted segments. Any of Druid's standard query filters can be used. Defaults to 'null', which will not filter any row.No

Compaction granularity spec

segmentGranularityTime chunking period for the segment granularity. Defaults to 'null', which preserves the original segment granularity. Accepts all Query granularity values.No
queryGranularityThe resolution of timestamp storage within each segment. Defaults to 'null', which preserves the original query granularity. Accepts all Query granularity values.No
rollupEnables compaction-time rollup. To preserve the original setting, keep the default value. To enable compaction-time rollup, set the value to true. Once the data is rolled up, you can no longer recover individual records.No

Learn more

See the following topics for more information:

  • Compaction for an overview of compaction and how to set up manual compaction in Druid.
  • Segment optimization for guidance on evaluating and optimizing Druid segment size.
  • Coordinator process for details on how the Coordinator plans compaction tasks.