Dumping Data from an ArangoDB database
- Advanced cluster options
  - Encryption
- Data Maskings

Dumping Data from an ArangoDB database

To dump data from an ArangoDB server instance, you will need to invoke arangodump.Dumps can be re-imported with arangorestore. arangodump can be invoked by executingthe following command:

unix> arangodump --output-directory "dump"

This will connect to an ArangoDB server and dump all non-system collections fromthe default database (_system) into an output directory named dump.Invoking arangodump will fail if the output directory already exists. This isan intentional security measure to prevent you from accidentally overwriting alreadydumped data. If you are positive that you want to overwrite data in the output directory, you can use the parameter —overwrite true to confirm this:

unix> arangodump --output-directory "dump" --overwrite true

arangodump will by default connect to the _system database using the defaultendpoint. If you want to connect to a different database or a different endpoint, or use authentication, you can use the following command-line options:

—server.database <string>: name of the database to connect to
—server.endpoint <string>: endpoint to connect to
—server.username <string>: username
—server.password <string>: password to use (omit this and you’ll be prompted for thepassword)
—server.authentication <bool>: whether or not to use authenticationHere’s an example of dumping data from a non-standard endpoint, using a dedicateddatabase name:

unix> arangodump --server.endpoint tcp://192.168.173.13:8531 --server.username backup --server.database mydb --output-directory "dump"

When finished, arangodump will print out a summary line with some aggregate statistics about what it did, e.g.:

Processed 43 collection(s), wrote 408173500 byte(s) into datafiles, sent 88 batch(es)

By default, arangodump will dump both structural information and documents from allnon-system collections. To adjust this, there are the following command-line arguments:

—dump-data <bool>: set to true to include documents in the dump. Set to false to exclude documents. The default value is true.
—include-system-collections <bool>: whether or not to include system collectionsin the dump. The default value is false.For example, to only dump structural information of all collections (including systemcollections), use:

unix> arangodump --dump-data false --include-system-collections true --output-directory "dump"

To restrict the dump to just specific collections, there is is the —collection option.It can be specified multiple times if required:

unix> arangodump --collection myusers --collection myvalues --output-directory "dump"

Structural information for a collection will be saved in files with name pattern <collection-name>.structure.json. Each structure file will contains a JSON object with these attributes:

parameters: contains the collection properties
indexes: contains the collection indexesDocument data for a collection will be saved in files with name pattern <collection-name>.data.json. Each line in a data file is a document insertion/update ordeletion marker, alongside with some meta data.

Starting with Version 2.1 of ArangoDB, the arangodump tool alsosupports sharding. Simply point it to one of the coordinators and itwill behave exactly as described above, working on sharded collectionsin the cluster.

However, as opposed to the single instance situation, this operation does not guarantee to dump a consistent snapshot if write operations happen during the dump operation. It is therefore recommended not to perform any data-modification operations on the cluster whilst arangodump is running.

As above, the output will be one structure description file and one datafile per sharded collection. Note that the data in the data file issorted first by shards and within each shard by ascending timestamp. Thestructural information of the collection contains the number of shardsand the shard keys.

Note that the version of the arangodump client tool needs to match the version of the ArangoDB server it connects to.

Advanced cluster options

Starting with version 3.1.17, collections may be created with sharddistribution identical to an existing prototypical collection;i.e. shards are distributed in the very same pattern as in theprototype collection. Such collections cannot be dumped without thereference collection or arangodump with yield an error.

unix> arangodump --collection clonedCollection --output-directory "dump"
ERROR Collection clonedCollection's shard distribution is based on a that of collection prototypeCollection, which is not dumped along. You may dump the collection regardless of the missing prototype collection by using the --ignore-distribute-shards-like-errors parameter.

There are two ways to approach that problem: Solve it, i.e. dump theprototype collection along:

unix> arangodump --collection clonedCollection --collection prototypeCollection --output-directory "dump"
Processed 2 collection(s), wrote 81920 byte(s) into datafiles, sent 1 batch(es)

Or override that behavior to be able to dump the collectionindividually.

unix> arangodump --collection B clonedCollection --output-directory "dump" --ignore-distribute-shards-like-errors
Processed 1 collection(s), wrote 34217 byte(s) into datafiles, sent 1 batch(es)

No that in consequence, restoring such a collection without itsprototype is affected. arangorestore

Encryption

In the ArangoDB Enterprise Edition there are the additional parameters:

Encryption key stored in file

—encryption.keyfile path-of-keyfile

The file path-to-keyfile must contain the encryption key. Thisfile must be secured, so that only arangod can access it. You shouldalso ensure that in case some-one steals the hardware, he will not beable to read the file. For example, by encryption /mytmpfs orcreating a in-memory file-system under /mytmpfs.

Encryption key generated by a program

—encryption.key-generator path-to-my-generator

The program path-to-my-generator must output the encryption onstandard output and exit.

Creating keys

The encryption keyfile must contain 32 bytes of random data.

You can create it with a command line this.

dd if=/dev/random bs=1 count=32 of=yourSecretKeyFile

For security, it is best to create these keys offline (away from yourdatabase servers) and directly store them in you secret managementtool.

Data Maskings

—maskings path-of-config

Introduced in: v3.3.22, v3.4.2

This feature allows you to define how sensitive data shall be dumped.It is possible to exclude collections entirely, limit the dump to thestructural information of a collection (name, indexes, sharding etc.)or to obfuscate certain fields for a dump. A JSON configuration file isused to define which collections and fields to mask and how.

The general structure of the configuration file looks like this:

{
  "collection-name": {
    "type": MASKING_TYPE,
    "maskings": [
      MASKING1,
      MASKING2,
      ...
    ]
  },
  ...
}

At the top level, there is an object with collection names and the maskingsettings to be applied to them. Using "*" as collection name defines adefault behavior for collections not listed explicitly.

Masking Types

type is a string describing how to mask the given collection.Possible values are:

"exclude": the collection is ignored completely and not even thestructure data is dumped.
"structure": only the collection structure is dumped, but no data at all
"masked": the collection structure and all data is dumped. However, the datais subject to obfuscation defined in the attribute maskings. It is an arrayof objects, with one object per field to mask. Each object needs at least apath and a type attribute to define which field to mask and whichmasking function to apply. Depending on themasking type, there may exist additional attributes.
"full": the collection structure and all data is dumped. No masking isapplied to this collection at all.

Example

{
  "private": {
    "type": "exclude"
  },
  "log": {
    "type": "structure"
  },
  "person": {
    "type": "masked",
    "maskings": [
      {
        "path": "name",
        "type": "xifyFront",
        "unmaskedLength": 2
      },
      {
        "path": ".security_id",
        "type": "xifyFront",
        "unmaskedLength": 2
      }
    ]
  }
}

The collection called private is completely ignored.
Only the structure of the collection log is dumped, but not the data itself.
The collection person is dumped completely but with maskings applied:
- The name field is masked if it occurs on the top-level.
- It also masks fields with the name security_id anywhere in the document.
- The masking function is of type xifyFront in both cases.The additional setting unmaskedLength is specific so xifyFront.

Masking vs. dump-data option

arangodump also supports a very coarse masking with the option—dump-data false. This basically removes all data from the dump.

You can either use —masking or —dump-data false, but not both.

Masking vs. include-collection option

arangodump also supports a very coarse masking with the option—include-collection. This will restrict the collections that aredumped to the ones explicitly listed.

It is possible to combine —masking and —include-collection.This will take the intersection of exportable collections.

Path

path defines which field to obfuscate. There can only be a singlepath per masking, but an unlimited amount of maskings per collection.

To mask a top-level attribute value, the path is simply the attributename, for instance "name" to mask the value "foobar":

{
  "_key": "1234",
  "name": "foobar"
}

The path to a nested attribute name with a top-level attribute personas its parent is "person.name":

{
  "_key": "1234",
  "person": {
    "name": "foobar"
  }
}

If the path starts with a . then it matches any path ending in name.For example, .name will match the field name of all leaf attributesin the document. Leaf attributes are attributes whose value is null,true, false, or of data type string, number or array.That means, it matches name at the top levelas well as at any nested level (e.g. foo.bar.name), but not nestedobjects themselves.

On the other hand, name will only match leaf attributesat top level. person.name will match the attribute name of a leafin the top-level object person. If person was itself an object,then the masking settings for this path would be ignored, because itis not a leaf attribute.

If the attribute value is an array then the masking is applied toall array elements individually.

If you have an attribute name that contains a dot, you need to quote thename with either a tick or a backtick. For example:

"path": "´name.with.dots´"

"path": "`name.with.dots`"

Example

The following configuration will replace the value of the nameattribute with an “xxxx”-masked string:

{
  "type": "xifyFront",
  "path": ".name",
  "unmaskedLength": 2
}

The document:

{
  "name": "top-level-name",
  "age": 42,
  "nicknames" : [ { "name": "hugo" }, "egon" ],
  "other": {
    "name": [ "emil", { "secret": "superman" } ]
  }
}

… will be changed as follows:

{
  "name": "xxxxxxxxxxxxme",
  "age": 42,
  "nicknames" : [ { "name": "xxgo" }, "egon" ],
  "other": {
    "name": [ "xxil", { "secret": "superman" } ]
  }
}

The values "egon" and "superman" are not replaced, because theyare not contained in an attribute value of which the attribute name isname.

Nested objects and arrays

If you specify a path and the attribute value is an array then themasking decision is applied to each element of the array as if thiswas the value of the attribute. This applies to arrays inside the array too.

If the attribute value is an object, then it is ignored and the attributedoes not get masked. To mask nested fields, specify the full path for eachleaf attribute.

If some documents have an attribute email with a string as value, but otherdocuments store a nested object under the same attribute name, then make sureto set up proper masking for the latter case, in which sub-attributes will notget masked if there is only a masking configured for the attribute emailbut not its nested attributes.

Examples

Masking email with the Xify Front function will convert:

{
  "email" : "email address"
}

… into:

{
  "email" : "xxil xxxxxxss"
}

because email is a leaf attribute. The document:

{
  "email" : [
    "address one",
    "address two",
    [
      "address three"
    ]
  ]
}

… will be converted into:

{
  "email" : [
    "xxxxxss xne",
    "xxxxxss xwo",
    [
      "xxxxxss xxxee"
    ]
  ]
}

… because the masking is applied to each array element individuallyincluding the elements of the sub-array. The document:

{
  "email" : {
    "address" : "email address"
  }
}

… will not be changed because email is not a leaf attribute.To mask the email address, you could use the paths email.addressor .address.

Masking Functions

The following masking functions are only available in theEnterprise Edition.

Xify Front
Zip
Datetime
Integral Number
Decimal Number
Credit Card Number
Phone Number
Email AddressThe masking function:
Random String… is available in the Community Edition as well as the Enterprise Edition.

Random String

This masking type will replace all values of attributes with keyname with an anonymized string. It is not guaranteed that the stringwill be of the same length.

A hash of the original string is computed. If the original string isshorter then the hash will be used. This will result in a longerreplacement string. If the string is longer than the hash thencharacters will be repeated as many times as needed to reach the fulloriginal string length.

Masking settings:

path (string): which field to mask
type (string): masking function name "randomString"Example

{
  "path": ".name",
  "type": "randomString"
}

Above masking setting applies to all leaf attributes with name .name.A document like:

{
  "_key" : "1234",
  "name" : [
    "My Name",
    {
      "other" : "Hallo Name"
    },
    [
      "Name One",
      "Name Two"
    ],
    true,
    false,
    null,
    1.0,
    1234,
    "This is a very long name"
  ],
  "deeply": {
    "nested": {
      "name": "John Doe",
      "not-a-name": "Pizza"
    }
  }
}

… will be converted to:

{
  "_key": "1234",
  "name": [
    "+y5OQiYmp/o=",
    {
      "other": "Hallo Name"
    },
    [
      "ihCTrlsKKdk=",
      "yo/55hfla0U="
    ],
    true,
    false,
    null,
    1.0,
    1234,
    "hwjAfNe5BGw=hwjAfNe5BGw="
  ],
  "deeply": {
    "nested": {
      "name": "55fHctEM/wY=",
      "not-a-name": "Pizza"
    }
  }
}

Xify Front

This masking type replaces the front characters with x andblanks. Alphanumeric characters, _ and - are replaced by x,everything else is replaced by a blank.

Masking settings:

path (string): which field to mask
type (string): masking function name "xifyFront"
unmaskedLength (number, default: 2): how many characters toleave as-is on the right-hand side of each word as integer value
hash (bool, default: false): whether to append a hash value to themasked string to avoid possible unique constraint violations caused bythe obfuscation
seed (integer, default: 0): used as secret for computing the hash.A value of 0 means a random seedExamples

{
  "path": ".name",
  "type": "xifyFront",
  "unmaskedLength": 2
}

This will mask all alphanumeric characters of a word except the lasttwo characters. Words of length 1 and 2 are unmasked. If theattribute value is not a string the result will be xxxx.

"This is a test!Do you agree?"

… will become:

"xxis is a xxst Do xou xxxee "

There is a catch. If you have an index on the attribute the maskingmight distort the index efficiency or even cause errors in case of aunique index.

{
  "type": "xifyFront",
  "path": ".name",
  "unmaskedLength": 2,
  "hash": true
}

This will add a hash at the end of the string.

"This is a test!Do you agree?"

… will become

"xxis is a xxst Do xou xxxee  NAATm8c9hVQ="

Note that the hash is based on a random secret that is different foreach run. This avoids dictionary attacks which can be used to guessvalues based pre-computations on dictionaries.

If you need reproducible results, i.e. hashes that do not change betweendifferent runs of arangodump, you need to specify a secret as seed,a number which must not be 0.

{
  "type": "xifyFront",
  "path": ".name",
  "unmaskedLength": 2,
  "hash": true,
  "seed": 246781478647
}

Zip

This masking type replaces a zip code with a random one.It uses the following rules:

If a character of the original zip code is a digit it will be replacedby a random digit.
If a character of the original zip code is a letter itwill be replaced by a random letter keeping the case.
If the attribute value is not a string then the default value is used.Note that this will generate random zip codes. Therefore there is achance that the same zip code value is generated multiple times, which cancause unique constraint violations if a unique index is or will beused on the zip code attribute.

Masking settings:

path (string): which field to mask
type (string): masking function name "zip"
default (string, default: "12345"): if the input field is not ofdata type string, then this value is usedExamples

{
  "path": ".code",
  "type": "zip",
}

This replaces real zip codes stored in fields called code at any levelwith random ones. "12345" is used as fallback value.

{
  "path": ".code",
  "type": "zip",
  "default": "abcdef"
}

If the original zip code is:

… it will be replaced by e.g.:

If the original zip code is:

SA34-EA

… it will be replaced by e.g.:

OW91-JI

If the original zip code is null, true, false or a number, then theuser-defined default value of "abcdef" will be used.

Datetime

This masking type replaces the value of the attribute with a randomdate between two configured dates in a customizable format.

Masking settings:

path (string): which field to mask
type (string): masking function name "datetime"
begin (string, default: "1970-01-01T00:00:00.000"):earliest point in time to return. Date time string in ISO 8601 format.
end (string, default: now):latest point in time to return. Date time string in ISO 8601 format.In case a partial date time string is provided (e.g. 2010-06 without dayand time) the earliest date and time is assumed (2010-06-01T00:00:00.000).The default value is the current system date and time.
format (string, default: ""): the formatting string format isdescribed in DATE_FORMAT().If no format is specified, then the result will be an empty string.Example

{
  "path": "eventDate",
  "type": "datetime",
  "begin" : "2019-01-01",
  "end": "2019-12-31",
  "format": "%yyyy-%mm-%dd",
}

Above example masks the field eventDate by returning a random date timestring in the range of January 1st and December 31st in 2019 using a formatlike 2019-06-17.

Integral Number

This masking type replaces the value of the attribute with a randomintegral number. It will replace the value even if it is a string,Boolean, or null.

Masking settings:

path (string): which field to mask
type (string): masking function name "integer"
lower (number, default: -100): smallest integer value to return
upper (number, default: 100): largest integer value to returnExample

{
  "path": "count",
  "type": "integer",
  "lower" : -100,
  "upper": 100
}

This masks the field count with a random number between-100 and 100 (inclusive).

Decimal Number

This masking type replaces the value of the attribute with a randomfloating point number. It will replace the value even if it is a string,Boolean, or null.

Masking settings:

path (string): which field to mask
type (string): masking function name "decimal"
lower (number, default: -1): smallest floating point value to return
upper (number, default: 1): largest floating point value to return
scale (number, default: 2): maximal amount of digits in thedecimal fraction partExamples

{
  "path": "rating",
  "type": "decimal",
  "lower" : -0.3,
  "upper": 0.3
}

This masks the field rating with a random floating point number between-0.3 and +0.3 (inclusive). By default, the decimal has a scale of 2.That means, it has at most 2 digits after the dot.

The configuration:

{
  "path": "rating",
  "type": "decimal",
  "lower" : -0.3,
  "upper": 0.3,
  "scale": 3
}

… will generate numbers with at most 3 decimal digits.

Credit Card Number

This masking type replaces the value of the attribute with a randomcredit card number (as integer number).See Luhn algorithmfor details.

Masking settings:

path (string): which field to mask
type (string): masking function name "creditCard"Example

{
  "path": "ccNumber",
  "type": "creditCard"
}

This generates a random credit card number to mask field ccNumber,e.g. 4111111414443302.

Phone Number

This masking type replaces a phone number with a random one.It uses the following rule:

If a character of the original number is a digitit will be replaced by a random digit.
If it is a letter it is replaced by a random letter.
All other characters are left unchanged.
If the attribute value is not a string it is replaced by thedefault value.Masking settings:
path (string): which field to mask
type (string): masking function name "phone"
default (string, default: "+1234567890"): if the input fieldis not of data type string, then this value is usedExamples

{
  "path": "phone.landline",
  "type": "phone"
}

This will replace an existing phone number with a random one, for instance"+31 66-77-88-xx" might get substituted by "+75 10-79-52-sb".

{
  "path": "phone.landline",
  "type": "phone",
  "default": "+49 12345 123456789"
}

This masks a phone number as before, but falls back to a different defaultphone number in case the input value is not a string.

Email Address

This masking type takes an email address, computes a hash value andsplits it into three equal parts AAAA, BBBB, and CCCC. Theresulting email address is in the format AAAA.BBBB@CCCC.invalid.The hash is based on a random secret that is different for each run.

Masking settings:

path (string): which field to mask
type (string): masking function name "email"Example

{
  "path": ".email",
  "type": "email"
}

This masks every leaf attribute email with a random email addresssimilar to "EHwG.3AOg@hGU=.invalid".

Arangodump

Dumping Data from an ArangoDB database

Advanced cluster options

Encryption

Encryption key stored in file

Encryption key generated by a program

Creating keys

Data Maskings

Masking Types

Masking vs. dump-data option

Masking vs. include-collection option

Path

Nested objects and arrays

Masking Functions

Random String

Xify Front

Zip

Datetime

Integral Number

Decimal Number

Credit Card Number

Phone Number

Email Address