Combining data

  • For combining datasets or data arrays along a single dimension, see concatenate.

  • For combining datasets with different variables, see merge.

  • For combining datasets or data arrays with different indexes or missing values, see combine.

  • For combining datasets or data arrays along multiple dimensions see combining.multi.

Concatenate

To combine arrays along existing or new dimension into a larger array, youcan use concat(). concat takes an iterable of DataArrayor Dataset objects, as well as a dimension name, and concatenates alongthat dimension:

  1. In [1]: arr = xr.DataArray(np.random.randn(2, 3),
  2.    ...:                    [('x', ['a', 'b']), ('y', [10, 20, 30])])
  3.    ...: 
  4.  
  5. In [2]: arr[:, :1]
  6. Out[2]: 
  7. <xarray.DataArray (x: 2, y: 1)>
  8. array([[ 0.469112],
  9.        [-1.135632]])
  10. Coordinates:
  11.   * x        (x) <U1 'a' 'b'
  12.   * y        (y) int64 10
  13.  
  14. # this resembles how you would use np.concatenate
  15. In [3]: xr.concat([arr[:, :1], arr[:, 1:]], dim='y')
  16. Out[3]: 
  17. <xarray.DataArray (x: 2, y: 3)>
  18. array([[ 0.469112, -0.282863, -1.509059],
  19.        [-1.135632,  1.212112, -0.173215]])
  20. Coordinates:
  21.   * x        (x) <U1 'a' 'b'
  22.   * y        (y) int64 10 20 30

In addition to combining along an existing dimension, concat can create anew dimension by stacking lower dimensional arrays together:

  1. In [4]: arr[0]
  2. Out[4]: 
  3. <xarray.DataArray (y: 3)>
  4. array([ 0.469112, -0.282863, -1.509059])
  5. Coordinates:
  6.     x        <U1 'a'
  7.   * y        (y) int64 10 20 30
  8.  
  9. # to combine these 1d arrays into a 2d array in numpy, you would use np.array
  10. In [5]: xr.concat([arr[0], arr[1]], 'x')
  11. Out[5]: 
  12. <xarray.DataArray (x: 2, y: 3)>
  13. array([[ 0.469112, -0.282863, -1.509059],
  14.        [-1.135632,  1.212112, -0.173215]])
  15. Coordinates:
  16.   * y        (y) int64 10 20 30
  17.   * x        (x) <U1 'a' 'b'

If the second argument to concat is a new dimension name, the arrays willbe concatenated along that new dimension, which is always inserted as the firstdimension:

  1. In [6]: xr.concat([arr[0], arr[1]], 'new_dim')
  2. Out[6]: 
  3. <xarray.DataArray (new_dim: 2, y: 3)>
  4. array([[ 0.469112, -0.282863, -1.509059],
  5.        [-1.135632,  1.212112, -0.173215]])
  6. Coordinates:
  7.   * y        (y) int64 10 20 30
  8.     x        (new_dim) <U1 'a' 'b'
  9. Dimensions without coordinates: new_dim

The second argument to concat can also be an Index orDataArray object as well as a string, in which case it isused to label the values along the new dimension:

  1. In [7]: xr.concat([arr[0], arr[1]], pd.Index([-90, -100], name='new_dim'))
  2. Out[7]: 
  3. <xarray.DataArray (new_dim: 2, y: 3)>
  4. array([[ 0.469112, -0.282863, -1.509059],
  5.        [-1.135632,  1.212112, -0.173215]])
  6. Coordinates:
  7.   * y        (y) int64 10 20 30
  8.     x        (new_dim) <U1 'a' 'b'
  9.   * new_dim  (new_dim) int64 -90 -100

Of course, concat also works on Dataset objects:

  1. In [8]: ds = arr.to_dataset(name='foo')
  2.  
  3. In [9]: xr.concat([ds.sel(x='a'), ds.sel(x='b')], 'x')
  4. Out[9]: 
  5. <xarray.Dataset>
  6. Dimensions:  (x: 2, y: 3)
  7. Coordinates:
  8.   * y        (y) int64 10 20 30
  9.   * x        (x) <U1 'a' 'b'
  10. Data variables:
  11.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732

concat() has a number of options which provide deeper controlover which variables are concatenated and how it handles conflicting variablesbetween datasets. With the default parameters, xarray will load some coordinatevariables into memory to compare them between datasets. This may be prohibitivelyexpensive if you are manipulating your dataset lazily using Parallel computing with Dask.

Merge

To combine variables and coordinates between multiple DataArray and/orDataset objects, use merge(). It can merge a list ofDataset, DataArray or dictionaries of objects convertible toDataArray objects:

  1. In [10]: xr.merge([ds, ds.rename({'foo': 'bar'})])
  2. Out[10]: 
  3. <xarray.Dataset>
  4. Dimensions:  (x: 2, y: 3)
  5. Coordinates:
  6.   * x        (x) <U1 'a' 'b'
  7.   * y        (y) int64 10 20 30
  8. Data variables:
  9.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
  10.     bar      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
  11.  
  12. In [11]: xr.merge([xr.DataArray(n, name='var%d' % n) for n in range(5)])
  13. Out[11]: 
  14. <xarray.Dataset>
  15. Dimensions:  ()
  16. Data variables:
  17.     var0     int64 0
  18.     var1     int64 1
  19.     var2     int64 2
  20.     var3     int64 3
  21.     var4     int64 4

If you merge another dataset (or a dictionary including data array objects), bydefault the resulting dataset will be aligned on the union of all indexcoordinates:

  1. In [12]: other = xr.Dataset({'bar': ('x', [1, 2, 3, 4]), 'x': list('abcd')})
  2.  
  3. In [13]: xr.merge([ds, other])
  4. Out[13]: 
  5. <xarray.Dataset>
  6. Dimensions:  (x: 4, y: 3)
  7. Coordinates:
  8.   * x        (x) object 'a' 'b' 'c' 'd'
  9.   * y        (y) int64 10 20 30
  10. Data variables:
  11.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 ... nan nan nan nan
  12.     bar      (x) int64 1 2 3 4

This ensures that merge is non-destructive. xarray.MergeError is raisedif you attempt to merge two variables with the same name but different values:

  1. In [14]: xr.merge([ds, ds + 1])
  2. MergeError: conflicting values for variable 'foo' on objects to be combined:
  3. first value: <xarray.Variable (x: 2, y: 3)>
  4. array([[ 0.4691123 , -0.28286334, -1.5090585 ],
  5.        [-1.13563237,  1.21211203, -0.17321465]])
  6. second value: <xarray.Variable (x: 2, y: 3)>
  7. array([[ 1.4691123 ,  0.71713666, -0.5090585 ],
  8.        [-0.13563237,  2.21211203,  0.82678535]])

The same non-destructive merging between DataArray index coordinates isused in the Dataset constructor:

  1. In [15]: xr.Dataset({'a': arr[:-1], 'b': arr[1:]})
  2. Out[15]: 
  3. <xarray.Dataset>
  4. Dimensions:  (x: 2, y: 3)
  5. Coordinates:
  6.   * x        (x) object 'a' 'b'
  7.   * y        (y) int64 10 20 30
  8. Data variables:
  9.     a        (x, y) float64 0.4691 -0.2829 -1.509 nan nan nan
  10.     b        (x, y) float64 nan nan nan -1.136 1.212 -0.1732

Combine

The instance method combine_first() combines twodatasets/data arrays and defaults to non-null values in the calling object,using values from the called object to fill holes. The resulting coordinatesare the union of coordinate labels. Vacant cells as a result of the outer-joinare filled with NaN. For example:

  1. In [16]: ar0 = xr.DataArray([[0, 0], [0, 0]], [('x', ['a', 'b']), ('y', [-1, 0])])
  2.  
  3. In [17]: ar1 = xr.DataArray([[1, 1], [1, 1]], [('x', ['b', 'c']), ('y', [0, 1])])
  4.  
  5. In [18]: ar0.combine_first(ar1)
  6. Out[18]: 
  7. <xarray.DataArray (x: 3, y: 3)>
  8. array([[ 0.,  0., nan],
  9.        [ 0.,  0.,  1.],
  10.        [nan,  1.,  1.]])
  11. Coordinates:
  12.   * x        (x) object 'a' 'b' 'c'
  13.   * y        (y) int64 -1 0 1
  14.  
  15. In [19]: ar1.combine_first(ar0)
  16. Out[19]: 
  17. <xarray.DataArray (x: 3, y: 3)>
  18. array([[ 0.,  0., nan],
  19.        [ 0.,  1.,  1.],
  20.        [nan,  1.,  1.]])
  21. Coordinates:
  22.   * x        (x) object 'a' 'b' 'c'
  23.   * y        (y) int64 -1 0 1

For datasets, ds0.combine_first(ds1) works similarly toxr.merge([ds0, ds1]), except that xr.merge raises MergeError whenthere are conflicting values in variables to be merged, whereas.combine_first defaults to the calling object’s values.

Update

In contrast to merge, update() modifies a datasetin-place without checking for conflicts, and will overwrite any existingvariables with new values:

  1. In [20]: ds.update({'space': ('space', [10.2, 9.4, 3.9])})
  2. Out[20]: 
  3. <xarray.Dataset>
  4. Dimensions:  (space: 3, x: 2, y: 3)
  5. Coordinates:
  6.   * x        (x) <U1 'a' 'b'
  7.   * y        (y) int64 10 20 30
  8.   * space    (space) float64 10.2 9.4 3.9
  9. Data variables:
  10.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732

However, dimensions are still required to be consistent between differentDataset variables, so you cannot change the size of a dimension unless youreplace all dataset variables that use it.

update also performs automatic alignment if necessary. Unlike merge, itmaintains the alignment of the original array instead of merging indexes:

  1. In [21]: ds.update(other)
  2. Out[21]: 
  3. <xarray.Dataset>
  4. Dimensions:  (space: 3, x: 2, y: 3)
  5. Coordinates:
  6.   * x        (x) object 'a' 'b'
  7.   * y        (y) int64 10 20 30
  8.   * space    (space) float64 10.2 9.4 3.9
  9. Data variables:
  10.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
  11.     bar      (x) int64 1 2

The exact same alignment logic when setting a variable with setitemsyntax:

  1. In [22]: ds['baz'] = xr.DataArray([9, 9, 9, 9, 9], coords=[('x', list('abcde'))])
  2.  
  3. In [23]: ds.baz
  4. Out[23]: 
  5. <xarray.DataArray 'baz' (x: 2)>
  6. array([9, 9])
  7. Coordinates:
  8.   * x        (x) object 'a' 'b'

Equals and identical

xarray objects can be compared by using the equals(),identical() andbroadcast_equals() methods. These methods are used bythe optional compat argument on concat and merge.

equals checks dimension names, indexes and arrayvalues:

  1. In [24]: arr.equals(arr.copy())
  2. Out[24]: True

identical also checks attributes, and the name of eachobject:

  1. In [25]: arr.identical(arr.rename('bar'))
  2. Out[25]: False

broadcast_equals does a more relaxed form of equalitycheck that allows variables to have different dimensions, as long as valuesare constant along those new dimensions:

  1. In [26]: left = xr.Dataset(coords={'x': 0})
  2.  
  3. In [27]: right = xr.Dataset({'x': [0, 0, 0]})
  4.  
  5. In [28]: left.broadcast_equals(right)
  6. Out[28]: True

Like pandas objects, two xarray objects are still equal or identical if they havemissing values marked by NaN in the same locations.

In contrast, the == operation performs element-wise comparison (likenumpy):

  1. In [29]: arr == arr.copy()
  2. Out[29]: 
  3. <xarray.DataArray (x: 2, y: 3)>
  4. array([[ True,  True,  True],
  5.        [ True,  True,  True]])
  6. Coordinates:
  7.   * x        (x) <U1 'a' 'b'
  8.   * y        (y) int64 10 20 30

Note that NaN does not compare equal to NaN in element-wise comparison;you may need to deal with missing values explicitly.

Merging with ‘no_conflicts’

The compat argument 'no_conflicts' is only available whencombining xarray objects with merge. In addition to the above comparisonmethods it allows the merging of xarray objects with locations where _either_have NaN values. This can be used to combine data with overlappingcoordinates as long as any non-missing values agree or are disjoint:

  1. In [30]: ds1 = xr.Dataset({'a': ('x', [10, 20, 30, np.nan])}, {'x': [1, 2, 3, 4]})
  2.  
  3. In [31]: ds2 = xr.Dataset({'a': ('x', [np.nan, 30, 40, 50])}, {'x': [2, 3, 4, 5]})
  4.  
  5. In [32]: xr.merge([ds1, ds2], compat='no_conflicts')
  6. Out[32]: 
  7. <xarray.Dataset>
  8. Dimensions:  (x: 5)
  9. Coordinates:
  10.   * x        (x) int64 1 2 3 4 5
  11. Data variables:
  12.     a        (x) float64 10.0 20.0 30.0 40.0 50.0

Note that due to the underlying representation of missing values as floatingpoint numbers (NaN), variable data type is not always preserved when mergingin this manner.

Combining along multiple dimensions

Note

There are currently three combining functions with similar names:auto_combine(), combine_by_coords(), andcombine_nested(). This is becauseauto_combine is in the process of being deprecated in favour of the othertwo functions, which are more general. If your code currently relies onauto_combine, then you will be able to get similar functionality by usingcombine_nested.

For combining many objects along multiple dimensions xarray providescombine_nested`() and combine_by_coords(). Thesefunctions use a combination of concat and merge across differentvariables to combine many objects into one.

combine_nested`() requires specifying the order in which theobjects should be combined, while combine_by_coords() attempts toinfer this ordering automatically from the coordinates in the data.

combine_nested() is useful when you know the spatialrelationship between each object in advance. The datasets must be provided inthe form of a nested list, which specifies their relative position andordering. A common task is collecting data from a parallelized simulation whereeach processor wrote out data to a separate file. A domain which was decomposedinto 4 parts, 2 each along both the x and y axes, requires organising thedatasets into a doubly-nested list, e.g:

  1. In [33]: arr = xr.DataArray(name='temperature', data=np.random.randint(5, size=(2, 2)), dims=['x', 'y'])
  2.  
  3. In [34]: arr
  4. Out[34]: 
  5. <xarray.DataArray 'temperature' (x: 2, y: 2)>
  6. array([[3, 4],
  7.        [3, 2]])
  8. Dimensions without coordinates: x, y
  9.  
  10. In [35]: ds_grid = [[arr, arr], [arr, arr]]
  11.  
  12. In [36]: xr.combine_nested(ds_grid, concat_dim=['x', 'y'])
  13. Out[36]: 
  14. <xarray.DataArray 'temperature' (x: 4, y: 4)>
  15. array([[3, 4, 3, 4],
  16.        [3, 2, 3, 2],
  17.        [3, 4, 3, 4],
  18.        [3, 2, 3, 2]])
  19. Dimensions without coordinates: x, y

combine_nested() can also be used to explicitly merge datasetswith different variables. For example if we have 4 datasets, which are dividedalong two times, and contain two different variables, we can pass Noneto 'concat_dim' to specify the dimension of the nested list over whichwe wish to use merge instead of concat:

  1. In [37]: temp = xr.DataArray(name='temperature', data=np.random.randn(2), dims=['t'])
  2.  
  3. In [38]: precip = xr.DataArray(name='precipitation', data=np.random.randn(2), dims=['t'])
  4.  
  5. In [39]: ds_grid = [[temp, precip], [temp, precip]]
  6.  
  7. In [40]: xr.combine_nested(ds_grid, concat_dim=['t', None])
  8. Out[40]: 
  9. <xarray.Dataset>
  10. Dimensions:        (t: 4)
  11. Dimensions without coordinates: t
  12. Data variables:
  13.     temperature    (t) float64 0.4432 -0.1102 0.4432 -0.1102
  14.     precipitation  (t) float64 -0.1668 0.5011 -0.1668 0.5011

combine_by_coords() is for combining objects which have dimensioncoordinates which specify their relationship to and order relative to oneanother, for example a linearly-increasing ‘time’ dimension coordinate.

Here we combine two datasets using their common dimension coordinates. Noticethey are concatenated in order based on the values in their dimensioncoordinates, not on their position in the list passed to combine_by_coords.

  1. In [41]: x1 = xr.DataArray(name='foo', data=np.random.randn(3), coords=[('x', [0, 1, 2])])
  2.  
  3. In [42]: x2 = xr.DataArray(name='foo', data=np.random.randn(3), coords=[('x', [3, 4, 5])])
  4.  
  5. In [43]: xr.combine_by_coords([x2, x1])
  6. Out[43]: 
  7. <xarray.Dataset>
  8. Dimensions:  (x: 6)
  9. Coordinates:
  10.   * x        (x) int64 0 1 2 3 4 5
  11. Data variables:
  12.     foo      (x) float64 -0.3553 -0.3379 0.581 0.9838 0.0578 0.7619

These functions can be used by open_mfdataset() to open manyfiles as one dataset. The particular function used is specified by setting theargument 'combine' to 'by_coords' or 'nested'. This is useful forsituations where your data is split across many files in multiple locations,which have some known relationship between one another.