Ldb Tool

Introduction

The ldb command line tool offers multiple data access and database admin commands. Some examples are listed below. For more information, please consult the help message displayed when running ldb without any arguments and the unit tests in tools/ldb_test.py.

Example

Example data access sequence:

  1. $./ldb --db=/tmp/test_db --create_if_missing put a1 b1
  2. OK
  3. $ ./ldb --db=/tmp/test_db get a1
  4. b1
  5. $ ./ldb --db=/tmp/test_db get a2
  6. Failed: NotFound:
  7. $ ./ldb --db=/tmp/test_db scan
  8. a1 ==> b1
  9. $ ./ldb --db=/tmp/test_db scan --hex
  10. 0x6131 ==> 0x6231
  11. $ ./ldb --db=/tmp/test_db put --key_hex 0x6132 b2
  12. OK
  13. $ ./ldb --db=/tmp/test_db scan
  14. a1 ==> b1
  15. a2 ==> b2
  16. $ ./ldb --db=/tmp/test_db get --value_hex a2
  17. 0x6232
  18. $ ./ldb --db=/tmp/test_db get --hex 0x6131
  19. 0x6231
  20. $ ./ldb --db=/tmp/test_db batchput a3 b3 a4 b4
  21. OK
  22. $ ./ldb --db=/tmp/test_db scan
  23. a1 ==> b1
  24. a2 ==> b2
  25. a3 ==> b3
  26. a4 ==> b4
  27. $ ./ldb --db=/tmp/test_db batchput "multiple words key" "multiple words value"
  28. OK
  29. $ ./ldb --db=/tmp/test_db scan
  30. Created bg thread 0x7f4a1dbff700
  31. a1 ==> b1
  32. a2 ==> b2
  33. a3 ==> b3
  34. a4 ==> b4
  35. multiple words key : multiple words value

To dump an existing leveldb database in HEX:

  1. $ ./ldb --db=/tmp/test_db dump --hex > /tmp/dbdump

To load the dumped HEX format data to a new leveldb database:

  1. $ cat /tmp/dbdump | ./ldb --db=/tmp/test_db_new load --hex --compression_type=bzip2 --block_size=65536 --create_if_missing --disable_wal

To compact an existing leveldb database:

  1. $ ./ldb --db=/tmp/test_db_new compact --compression_type=bzip2 --block_size=65536

You can specify command line --column_family=<string> for which column family your query will be against.

--try_load_options will try to load the options file in the DB to open the DB. It is a good idea to always try to have this option on when you operate the DB. If you open the DB with default options, it may mess up LSM-tree structure which can’t be recovered automatically.

Open as secondary

By default, ldb can only used against a DB that is offline. Operating the DB, even for read-only operations, might make changes to the DB directory, e.g. info logs. An option --secondary_path=<secondary_path> would open the DB as a Secondary instance, which can be used to open a running DB and/or to minimize impacts to the DB directory. This argument can be used for any ldb command, but since secondary instance not all operations can be done against secondary instance, some operations will fail. Besides write operations which would definitely fail with secondary instance, some read operations might also fail.

SST dump tool

sst_dump tool can be used to gain insights about a specific SST file. There are multiple operations that sst_dump can execute on a SST file.

  1. $ ./sst_dump
  2. file or directory must be specified.
  3. sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw]
  4. --file=<data_dir_OR_sst_file>
  5. Path to SST file or directory containing SST files
  6. --command=check|scan|raw|verify
  7. check: Iterate over entries in files but dont print anything except if an error is encounterd (default command)
  8. scan: Iterate over entries in files and print them to screen
  9. raw: Dump all the table contents to <file_name>_dump.txt
  10. verify: Iterate all the blocks in files verifying checksum to detect possible coruption but dont print anything except if a corruption is encountered
  11. recompress: reports the SST file size if recompressed with different
  12. compression types
  13. --output_hex
  14. Can be combined with scan command to print the keys and values in Hex
  15. --from=<user_key>
  16. Key to start reading from when executing check|scan
  17. --to=<user_key>
  18. Key to stop reading at when executing check|scan
  19. --prefix=<user_key>
  20. Returns all keys with this prefix when executing check|scan
  21. Cannot be used in conjunction with --from
  22. --read_num=<num>
  23. Maximum number of entries to read when executing check|scan
  24. --verify_checksum
  25. Verify file checksum when executing check|scan
  26. --input_key_hex
  27. Can be combined with --from and --to to indicate that these values are encoded in Hex
  28. --show_properties
  29. Print table properties after iterating over the file when executing
  30. check|scan|raw
  31. --set_block_size=<block_size>
  32. Can be combined with --command=recompress to set the block size that will
  33. be used when trying different compression algorithms
  34. --compression_types=<comma-separated list of CompressionType members, e.g.,
  35. kSnappyCompression>
  36. Can be combined with --command=recompress to run recompression for this
  37. list of compression types
  38. --parse_internal_key=<0xKEY>
  39. Convenience option to parse an internal key on the command line. Dumps the
  40. internal key in hex format {'key' @ SN: type}
Dumping SST file blocks
  1. ./sst_dump --file=/path/to/sst/000829.sst --command=raw

This command will generate a txt file named /path/to/sst/000829_dump.txt. This file will contain all index blocks and data blocks encoded in Hex. It will also contain information like table properties, footer details and meta index details.

Printing entries in SST file
  1. ./sst_dump --file=/path/to/sst/000829.sst --command=scan --read_num=5

This command will print the first 5 keys in the SST file to the screen. the output may look like this

  1. 'Key1' @ 5: 1 => Value1
  2. 'Key2' @ 2: 1 => Value2
  3. 'Key3' @ 4: 1 => Value3
  4. 'Key4' @ 3: 1 => Value4
  5. 'Key5' @ 1: 1 => Value5

The output can be interpreted like this

  1. '<key>' @ <sequence number>: <type> => <value>

Please notice that if your key has non-ascii characters, it will be hard to print it on screen, in this case it’s a good idea to use —output_hex like this

  1. ./sst_dump --file=/path/to/sst/000829.sst --command=scan --read_num=5 --output_hex

You can also specify where do you want to start reading from and where do you want to stop by using —from and —to like this

  1. ./sst_dump --file=/path/to/sst/000829.sst --command=scan --from="key2" --to="key4"

You can pass —from and —to using hexadecimal as well by using —input_key_hex

  1. ./sst_dump --file=/path/to/sst/000829.sst --command=scan --from="0x6B657932" --to="0x6B657934" --input_key_hex
Checking SST file
  1. ./sst_dump --file=/path/to/sst/000829.sst --command=check --verify_checksum

This command will Iterate over all entries in the SST file but wont print any thing except if it encountered a problem in the SST file. It will also verify the checksum.

Printing SST file properties
  1. ./sst_dump --file=/path/to/sst/000829.sst --show_properties

This command will read the SST file properties and print them, output may look like this

  1. from [] to []
  2. Process /path/to/sst/000829.sst
  3. Sst file format: block-based
  4. Table Properties:
  5. ------------------------------
  6. # data blocks: 26541
  7. # entries: 2283572
  8. raw key size: 264639191
  9. raw average key size: 115.888262
  10. raw value size: 26378342
  11. raw average value size: 11.551351
  12. data block size: 67110160
  13. index block size: 3620969
  14. filter block size: 0
  15. (estimated) table size: 70731129
  16. filter policy name: N/A
  17. # deleted keys: 571272
Trying different compression algorithms

sst_dump can be used to check the size of the file under different compression algorithms.

  1. ./sst_dump --file=/path/to/sst/000829.sst --show_compression_sizes

By using —show_compression_sizes sst_dump will recreate the SST file in memory using different compression algorithms and report the size, output may look like this

  1. from [] to []
  2. Process /path/to/sst/000829.sst
  3. Sst file format: block-based
  4. Block Size: 16384
  5. Compression: kNoCompression Size: 103974700
  6. Compression: kSnappyCompression Size: 103906223
  7. Compression: kZlibCompression Size: 80602892
  8. Compression: kBZip2Compression Size: 76250777
  9. Compression: kLZ4Compression Size: 103905572
  10. Compression: kLZ4HCCompression Size: 97234828
  11. Compression: kZSTDNotFinalCompression Size: 79821573

These files are created in memory and they are generated with block size of 16KB, the block size can be change by using —set_block_size.