Importing data from the file system

The tools restore command creates data schema objects in the DB and imports to them the data from the file system that was previously dumped there with the tools dump command or prepared manually following the rules described in the File structure article:

  1. ydb [connection options] tools restore -p PATH -i PATH [options]

Importing data from the file system - 图1

where [connection options] are database connection options

If a table already exists in the database, no changes are made to its schema. This may prevent the data import operation from being performed if some columns of the imported files are missing in the DB table or have an incorrect type.

Data import to DB tables is performed using the YQL REPLACE statement. If before the start of the import operation, the table contained any entries, those of them whose keys exist in the imported files are replaced with data from the files. The entries with keys that are missing in the imported files remain unchanged.

Required parameters

-p PATH or --path PATH: Path to the DB directory that data will be imported to. To import data to the root directory, specify .. Any missing directories specified in the path will be created.

-i PATH or --input PATH: Path to the directory in the client file system that data will be imported from.

Optional parameters

[options]: Optional command parameters:

--restore-data VAL: Data import flag, 1 (yes) or 0 (no), defaults to 1. If 0, the import operation will only create schema objects without data being restored to them. If there is no data in the file system (only the data schema is exported), it doesn’t matter if you change the flag value.

--restore-indexes VAL: Index import flag, 1 (yes) or 0 (no), defaults to 1. If 0, when running the import operation, secondary indexes will neither be registered in the data schema nor populated with data.

--dry-run: Mode for checking if the data schema in the DB and file system match without making any changes to the DB, 1 (yes) or 0 (no), defaults to 0. If this mode is enabled, it is checked that:

  • All tables in the file system are present in the DB.
  • The object data schema in the DB and file system match.

--save-partial-result: Save the result of an incomplete import operation. If this option isn’t enabled, in the event of an error during the import operation, the state of the DB is restored to the point before the operation is started.

Load limit parameters

The following parameters let you limit the load on the DB generated by data import processes.

Attention

Some of the parameters listed below have valid default values. This means that even if none of them is specified in the tools restore command call, the load will still be limited.

--bandwidth VAL: Limits the amount of data that can be imported per second, defaults to 0 (not set). VAL stands for the data volume that is specified as a prefixed number like 2MiB.
--rps VAL: Limit on the number of requests per second for importing data packets to the DB, defaults to 30.
--in-flight VAL: Limit on the number of concurrently executed requests, defaults to 10.
--upload-batch-rows VAL: Limit on the number of rows in the imported batch, defaults to 0 (unlimited). VAL stands for the amount of rows, specified as a number with an optional decimal prefix, such as 1K.
--upload-batch-bytes VAL: Limit on the size of an imported batch, defaults to 512KB. VAL stands for the data volume that is specified as a prefixed number like 1MiB.
--upload-batch-rus VAL: Only applies to Serverless databases, limits the use of Request Units (RU) per import of a single batch, defaults 30 RUs. The batch size is selected for the specified value. VAL stands for the amount of RUs, specified as a number with an optional decimal prefix, such as 100 or 1K.

Examples

The examples use a profile named db1. For information about how to create it, see the Getting started with the YDB CLI article in the “Getting started “ section.

Importing data to the DB root

From the current directory of the file system:

  1. ydb -p db1 tools restore -p . -i .

Importing data from the file system - 图2

From the specified directory of the file system:

  1. ydb -p db1 tools restore -p . -i ~/backup_db1

Importing data from the file system - 图3

Importing data to the specified DB directory

From the current directory of the file system:

  1. ydb -p db1 tools restore -p dir1/dir2 -i .

Importing data from the file system - 图4

From the specified directory of the file system:

  1. ydb -p db1 tools restore -p dir1/dir2 -i ~/backup_db1

Importing data from the file system - 图5

Checking if the data schema in the DB and file system match:

  1. ydb -p db1 tools restore -p dir1/dir2 -i ~/backup_db1 --dry-run

Importing data from the file system - 图6