dump-segment tool

The DumpSegment tool can be used to dump the metadata or contents of an Apache Druid segment for debugging purposes. Note that the dump is not necessarily a full-fidelity translation of the segment. In particular, not all metadata is included, and complex metric values may not be complete.

To run the tool, point it at a segment directory and provide a file for writing output:

  1. java -classpath "/my/druid/lib/*" -Ddruid.extensions.loadList="[]" org.apache.druid.cli.Main \
  2.   tools dump-segment \
  3.   --directory /home/druid/path/to/segment/ \
  4.   --out /home/druid/output.txt

Output format

Data dumps

By default, or with --dump rows, this tool dumps rows of the segment as newline-separate JSON objects, with one object per line, using the default serialization for each column. Normally all columns are included, but if you like, you can limit the dump to specific columns with --column name.

For example, one line might look like this when pretty-printed:

  1. {
  2.   "__time": 1442018818771,
  3.   "added": 36,
  4.   "channel": "#en.wikipedia",
  5.   "cityName": null,
  6.   "comment": "added project",
  7.   "count": 1,
  8.   "countryIsoCode": null,
  9.   "countryName": null,
  10.   "deleted": 0,
  11.   "delta": 36,
  12.   "isAnonymous": "false",
  13.   "isMinor": "false",
  14.   "isNew": "false",
  15.   "isRobot": "false",
  16.   "isUnpatrolled": "false",
  17.   "iuser": "00001553",
  18.   "metroCode": null,
  19.   "namespace": "Talk",
  20.   "page": "Talk:Oswald Tilghman",
  21.   "regionIsoCode": null,
  22.   "regionName": null,
  23.   "user": "GELongstreet"
  24. }

Metadata dumps

With --dump metadata, this tool dumps metadata instead of rows. Metadata dumps generated by this tool are in the same format as returned by the SegmentMetadata query.

Bitmap dumps

With --dump bitmaps, this tool dump bitmap indexes instead of rows. Bitmap dumps generated by this tool include dictionary-encoded string columns only. The output contains a field “bitmapSerdeFactory” describing the type of bitmaps used in the segment, and a field “bitmaps” containing the bitmaps for each value of each column. These are base64 encoded by default, but you can also dump them as lists of row numbers with --decompress-bitmaps.

Normally all columns are included, but if you like, you can limit the dump to specific columns with --column name.

Sample output:

  1. {
  2.   "bitmapSerdeFactory": {
  3.     "type": "concise"
  4.   },
  5.   "bitmaps": {
  6.     "isRobot": {
  7.       "false": "//aExfu+Nv3X...",
  8.       "true": "gAl7OoRByQ..."
  9.     }
  10.   }
  11. }

Command line arguments

argumentdescriptionrequired?
—directory fileDirectory containing segment data. This could be generated by unzipping an “index.zip” from deep storage.yes
—output fileFile to write to, or omit to write to stdout.yes
—dump TYPEDump either ‘rows’ (default), ‘metadata’, or ‘bitmaps’no
—column columnNameColumn to include. Specify multiple times for multiple columns, or omit to include all columns.no
—filter jsonJSON-encoded query filter. Omit to include all rows. Only used if dumping rows.no
—time-iso8601Format __time column in ISO8601 format rather than long. Only used if dumping rows.no
—decompress-bitmapsDump bitmaps as arrays rather than base64-encoded compressed bitmaps. Only used if dumping bitmaps.no