Data Import

Introduction

The Sync Tool is an IoTDB suite tool that periodically uploads persistent tsfiles from the sender disk to the receiver and loads them.

On the sender side of the sync, the sync module is a separate process, independent of the IoTDB process. It can be started and closed through a separate script (see Sections Usage for details). The frequency cycle of the sync can be set by the user.

On the receiver side of the sync, the sync module is embedded in the engine of IoTDB and is in the same process with IoTDB. The receiver module listens for a separate port, which can be set by the user (see Section Configuration for details). Before using it, it needs to set up a whitelist at the sync receiver, which is expressed as a network segment. The receiver only accepts the data transferred from the sender located in the whitelist segment, as detailed in Section Configuration.

The sync tool has a many-to-one sender-receiver mode - that is, one sync receiver can receive data from multiple sync senders simultaneously while one sync sender can only send data to one sync receiver.

Note: Before using the sync tool, the client and server need to be configured separately. The configuration is detailed in Sections Configuration.

Configuration

Sync Receiver

The parameter configuration of the sync receiver is located in the configuration file iotdb-engine.properties of IoTDB, and its directory is $IOTDB_HOME/conf/iotdb-engine.properties. In this configuration file, there are four parameters related to the sync receiver. The configuration instructions are as follows:

parameter: is_sync_enable
DescriptionSync function switch, which is configured as true to indicate that the receiver is allowed to receive the data from the sender and load it. When set to false, it means that the receiver is not allowed to receive the data from any sender.
TypeBoolean
Defaultfalse
Modalities for Entry into Force after ModificationRestart receiver
parameter: IP_white_list
DescriptionSet up a white list of sender IP addresses, which is expressed in the form of network segments and separated by commas between multiple network segments. When the sender transfers data to the receiver, only when the IP address of the sender is within the network segment set by the whitelist can the receiver allow the sync operation. If the whitelist is empty, the receiver does not allow any sender to sync data. The default receiver accepts all IP sync requests.
TypeString
Default0.0.0.0/0
Modalities for Entry into Force after ModificationRestart receiver
parameter: update_historical_data_possibility
DescriptionThe processing strategy chosen by the sync receiver when merging the sync data.
1. If the sync data accounts for more than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended this parameter be set to TRUE, which has a greater impact on the write performance and reduce CPU usage.
2. If the sync data accounts for less than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended this parameter be set to FALSE,which has little impact on the write performance and takes up a large amount of CPU power.
TypeBoolean
Defaultfalse
Modalities for Entry into Force after ModificationRestart receiver
parameter: sync_server_port
DescriptionSync receiver port to listen. Make sure that the port is not a system reserved port and is not occupied. This paramter is valid only when the parameter is_sync_enable is set to TRUE.
TypeShort Int : [0,65535]
Default5555
Modalities for Entry into Force after ModificationRestart receiver

Sync Sender

The parameters of the sync sender are configured in a separate configuration file iotdb-sync-client.properties with the installation directory of $IOTDB_HOME/conf/iotdb-sync-client.properties. In this configuration file, there are five parameters related to the sync sender. The configuration instructions are as follows:

parameter: server_ip
DescriptionIP address of sync receiver.
TypeString
Default127.0.0.1
Modalities for Entry into Force after ModificationRestart client
parameter: server_port
DescriptionListening port of sync receiver, it is necessary to ensure that the port is consistent with the configuration of the listening port set in receiver.
TypeShort Int : [0,65535]
Default5555
Modalities for Entry into Force after ModificationRestart client
parameter: sync_period_in_second
DescriptionThe period time of sync process, the time unit is second.
TypeInt : [0,2147483647]
Default600
Modalities for Entry into Force after ModificationRestart client
parameter: iotdb_schema_directory
DescriptionThe absolute path of the sender’s IoTDB schema file, such as $IOTDB_HOME/data/system/schema/mlog.txt (if the user does not manually set the path of schema metadata, the path is the default path of IoTDB engine). This parameter is not valid by default and is set manually when the user needs it.
TypeString
Modalities for Entry into Force after ModificationRestart client
parameter: iotdb_bufferWrite_directory
DescriptionThe absolute path of the buffer write data (tsfile file) directory of the IoTDB at the sender, such as: $IOTDB_HOME/data/data/settled (if the user does not set the data path manually, the path is the default path of IoTDB engine). This parameter is not valid by default, and is set manually when the user needs it. This parameter needs to be guaranteed to belong to the same IoTDB as the parameter iotdb_schema_directory.
TypeString
Modalities for Entry into Force after ModificationRestart client

Usage

Start Sync Receiver

  1. Set up parameters of sync receiver. For example: Sync - 图1
  2. Start IoTDB engine, and the sync receiver will start at the same time, and the LOG log will start with the sentence IoTDB: start SYNC ServerService successfully indicating the successful start of the return receiver. Sync - 图2

Stop Sync Receiver

Stop IoTDB and the sync receiver will be closed at the same time.

Start Sync Sender

  1. Set up parameters of sync sender. For example: Sync - 图3
  2. Start sync sender Users can use the scripts under the $IOTDB_HOME/bin folder to start the sync sender. For Linux and Mac OS X users:
  1.   Shell >$IOTDB_HOME/bin/start-sync-client.sh

For Windows users:

  1.   Shell >$IOTDB_HOME\bin\start-sync-client.bat

Sync - 图4

Stop Sync Sender

Users can use the scripts under the $IOTDB_HOME/bin folder to stop the sync sender. For Linux and Mac OS X users:

  1.   Shell >$IOTDB_HOME/bin/stop-sync-client.sh

For Windows users:

  1.   Shell >$IOTDB_HOME\bin\stop-sync-client.bat