Configuration
Taking full advantage of Dask sometimes requires user configuration.This might be to control logging verbosity, specify cluster configuration,provide credentials for security, or any of several other options that arise inproduction.
Configuration is specified in one of the following ways:
- YAML files in
~/.config/dask/
or/etc/dask/
- Environment variables like
DASK_DISTRIBUTEDSCHEDULERWORK_STEALING=True
- Default settings within sub-librariesThis combination makes it easy to specify configuration in a variety ofsettings ranging from personal workstations, to IT-mandated configuration, todocker images.
Access Configuration
dask.config.get (key[, default, config]) | Get elements from global config |
Configuration is usually read by using the dask.config
module, either withthe config
dictionary or the get
function:
- >>> import dask
- >>> import dask.distributed # populate config with distributed defaults
- >>> dask.config.config
- {
- "array": {
- "chunk-size": "128 MiB",
- }
- "distributed": {
- "logging": {
- "distributed": "info",
- "bokeh": "critical",
- "tornado": "critical"
- },
- "admin": {
- "log-format": "%(name)s - %(levelname)s - %(message)s"
- }
- }
- }
- >>> dask.config.get("distributed.logging")
- {
- 'distributed': 'info',
- 'bokeh': 'critical',
- 'tornado': 'critical'
- }
- >>> dask.config.get('distributed.logging.bokeh') # use `.` for nested access
- 'critical'
You may wish to inspect the dask.config.config
dictionary to get a sensefor what configuration is being used by your current system.
Note that the get
function treats underscores and hyphens identically.For example, dask.config.get('num_workers')
is equivalent todask.config.get('num-workers')
.
Values like "128 MiB"
and "10s"
are parsed using the functions inUtilities.
Specify Configuration
YAML files
You can specify configuration values in YAML files like the following:
- array:
- chunk-size: 128 MiB
- distributed:
- logging:
- distributed: info
- bokeh: critical
- tornado: critical
- scheduler:
- work-stealing: True
- allowed-failures: 5
- admin:
- log-format: '%(name)s - %(levelname)s - %(message)s'
These files can live in any of the following locations:
- The
~/.config/dask
directory in the user’s home directory - The
{sys.prefix}/etc/dask
directory local to Python - The root directory (specified by the
DASKROOT_CONFIG
environmentvariable or/etc/dask/
by default)Dask searches for _all YAML files within each of these directories and mergesthem together, preferring configuration files closer to the user over systemconfiguration files (preference follows the order in the list above).Additionally, users can specify a path with theDASK_CONFIG
environmentvariable, which takes precedence at the top of the list above.
The contents of these YAML files are merged together, allowing differentDask subprojects like dask-kubernetes
or dask-ml
to manage configurationfiles separately, but have them merge into the same global configuration.
Note: for historical reasons we also look in the ~/.dask
directory forconfig files. This is deprecated and will soon be removed.
Environment Variables
You can also specify configuration values with environment variables likethe following:
- export DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True
- export DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES=5
resulting in configuration values like the following:
- {
- 'distributed': {
- 'scheduler': {
- 'work-stealing': True,
- 'allowed-failures': 5
- }
- }
- }
Dask searches for all environment variables that start with DASK_
, thentransforms keys by converting to lower case and changing double-underscores tonested structures.
Dask tries to parse all values with ast.literal_eval, letting userspass numeric and boolean values (such as True
in the example above) as wellas lists, dictionaries, and so on with normal Python syntax.
Environment variables take precedence over configuration values found in YAMLfiles.
Defaults
Additionally, individual subprojects may add their own default values when theyare imported. These are always added with lower priority than the YAML filesor environment variables mentioned above:
- >>> import dask.config
- >>> dask.config.config # no configuration by default
- {}
- >>> import dask.distributed
- >>> dask.config.config # New values have been added
- {
- 'scheduler': ...,
- 'worker': ...,
- 'tls': ...
- }
Directly within Python
dask.config.set ([arg, config, lock]) | Temporarily set configuration values within a context manager |
Configuration is stored within a normal Python dictionary indask.config.config
and can be modified using normal Python operations.
Additionally, you can temporarily set a configuration value using thedask.config.set
function. This function accepts a dictionary as an inputand interprets "."
as nested access:
- >>> dask.config.set({'scheduler.work-stealing': True})
This function can also be used as a context manager for consistent cleanup:
- with dask.config.set({'scheduler.work-stealing': True}):
- ...
Note that the set
function treats underscores and hyphens identically.For example, dask.config.set({'scheduler.work-stealing': True})
isequivalent to dask.config.set({'scheduler.work_stealing': True})
.
Updating Configuration
Manipulating configuration dictionaries
dask.config.merge (*dicts) | Update a sequence of nested dictionaries |
dask.config.update (old, new[, priority]) | Update a nested dictionary with values from another |
dask.config.expand_environment_variables (config) | Expand environment variables in a nested config dictionary |
As described above, configuration can come from many places, including severalYAML files, environment variables, and project defaults. Each of theseprovides a configuration that is possibly nested like the following:
- x = {'a': 0, 'c': {'d': 4}}
- y = {'a': 1, 'b': 2, 'c': {'e': 5}}
Dask will merge these configurations respecting nested data structures, andrespecting order:
- >>> dask.config.merge(x, y)
- {'a': 1, 'b': 2, 'c': {'d': 4, 'e': 5}}
You can also use the update
function to update the existing configurationin place with a new configuration. This can be done with priority being givento either config. This is often used to update the global configuration indask.config.config
:
- dask.config.update(dask.config, new, priority='new') # Give priority to new values
- dask.config.update(dask.config, new, priority='old') # Give priority to old values
Sometimes it is useful to expand environment variables stored within aconfiguration. This can be done with the expand_environment_variables
function:
- dask.config.config = dask.config.expand_environment_variables(dask.config.config)
Refreshing Configuration
dask.config.collect ([paths, env]) | Collect configuration from paths and environment variables |
dask.config.refresh ([config, defaults]) | Update configuration by re-reading yaml files and env variables |
If you change your environment variables or YAML files, Dask will notimmediately see the changes. Instead, you can call refresh
to go throughthe configuration collection process and update the default configuration:
- >>> dask.config.config
- {}
- >>> # make some changes to yaml files
- >>> dask.config.refresh()
- >>> dask.config.config
- {...}
This function uses dask.config.collect
, which returns the configurationwithout modifying the global configuration. You might use this to determinethe configuration of particular paths not yet on the config path:
- >>> dask.config.collect(paths=[...])
- {...}
Downstream Libraries
dask.config.ensure_file (source[, …]) | Copy file to default location if it does not already exist |
dask.config.update (old, new[, priority]) | Update a nested dictionary with values from another |
dask.config.update_defaults (new[, config, …]) | Add a new set of defaults to the configuration |
Downstream Dask libraries often follow a standard convention to use the centralDask configuration. This section provides recommendations for integrationusing a fictional project, dask-foo
, as an example.
Downstream projects typically follow the following convention:
- Maintain default configuration in a YAML file within their sourcedirectory:
- setup.py
- dask_foo/__init__.py
- dask_foo/config.py
- dask_foo/core.py
- dask_foo/foo.yaml # <---
- Place configuration in that file within a namespace for the project:
- # dask_foo/foo.yaml
- foo:
- color: red
- admin:
- a: 1
- b: 2
- Within a config.py file (or anywhere) load that default config file andupdate it into the global configuration:
- # dask_foo/config.py
- import os
- import yaml
- import dask.config
- fn = os.path.join(os.path.dirname(__file__), 'foo.yaml')
- with open(fn) as f:
- defaults = yaml.load(f)
- dask.config.update_defaults(defaults)
- Within that same config.py file, copy the
'foo.yaml'
file to the user’sconfiguration directory if it doesn’t already exist.
We also comment the file to make it easier for us to change defaults in thefuture.
- # ... continued from above
- dask.config.ensure_file(source=fn, comment=True)
The user can investigate ~/.config/dask/*.yaml
to see all of thecommented out configuration files to which they have access.
- Ensure that this file is run on import by including it in
init.py
:
- # dask_foo/__init__.py
- from . import config
- Within
dask_foo
code, use thedask.config.get
function to accessconfiguration values:
- # dask_foo/core.py
- def process(fn, color=dask.config.get('foo.color')):
- ...
- You may also want to ensure that your yaml configuration files are includedin your package. This can be accomplished by including the following linein your MANIFEST.in:
- recursive-include <PACKAGE_NAME> *.yaml
and the following in your setup.py setup
call:
- from setuptools import setup
- setup(...,
- include_package_data=True,
- ...)
This process keeps configuration in a central place, but also keeps it safewithin namespaces. It places config files in an easy to access locationby default (~/.config/dask/*.yaml
), so that users can easily discover whatthey can change, but maintains the actual defaults within the source code, sothat they more closely track changes in the library.
However, downstream libraries may choose alternative solutions, such asisolating their configuration within their library, rather than using theglobal dask.config system. All functions in the dask.config
module alsowork with parameters, and do not need to mutate global state.
API
dask.config.
get
(key, default='no_default', config={'temporary-directory': None, 'array': {'svg': {'size': 120}, 'chunk-size': '128MiB', 'rechunk-threshold': 4}})- Get elements from global config
Use ‘.’ for nested access
See also
Examples
- >>> from dask import config
- >>> config.get('foo') # doctest: +SKIP
- {'x': 1, 'y': 2}
- >>> config.get('foo.x') # doctest: +SKIP
- 1
- >>> config.get('foo.x.y', default=123) # doctest: +SKIP
- 123
dask.config.
set
(arg=None, config={'array': {'chunk-size': '128MiB', 'rechunk-threshold': 4, 'svg': {'size': 120}}, 'temporary-directory': None}, lock=, **kwargs) - Temporarily set configuration values within a context manager
Parameters:
- arg:mapping or None, optional
A mapping of configuration key-value pairs to set.
**kwargs :
- Additional key-value pairs to set. If
arg
is provided, values setinarg
will be applied before those inkwargs
.Double-underscores (__
) in keyword arguments will be replaced with.
, allowing nested values to be easily set.
See also
Examples
- >>> import dask
Set 'foo.bar'
in a context, by providing a mapping.
- >>> with dask.config.set({'foo.bar': 123}):
- ... pass
Set 'foo.bar'
in a context, by providing a keyword argument.
- >>> with dask.config.set(foo__bar=123):
- ... pass
Set 'foo.bar'
globally.
- >>> dask.config.set(foo__bar=123) # doctest: +SKIP
This prefers the values in the latter dictionaries to those in the former
See also
Examples
- >>> a = {'x': 1, 'y': {'a': 2}}
- >>> b = {'y': {'b': 3}}
- >>> merge(a, b) # doctest: +SKIP
- {'x': 1, 'y': {'a': 2, 'b': 3}}
This is like dict.update except that it smoothly merges nested values
This operates in-place and modifies old
Parameters:
- priority: string {‘old’, ‘new’}
- If new (default) then the new dictionary has preference.Otherwise the old dictionary does.
See also
Examples
- >>> a = {'x': 1, 'y': {'a': 2}}
- >>> b = {'x': 2, 'y': {'b': 3}}
- >>> update(a, b) # doctest: +SKIP
- {'x': 2, 'y': {'a': 2, 'b': 3}}
- >>> a = {'x': 1, 'y': {'a': 2}}
- >>> b = {'x': 2, 'y': {'b': 3}}
- >>> update(a, b, priority='old') # doctest: +SKIP
- {'x': 1, 'y': {'a': 2, 'b': 3}}
dask.config.
collect
(paths=['/etc/dask', '/home/docs/checkouts/readthedocs.org/user_builds/dask/envs/latest/etc/dask', '/home/docs/.config/dask', '/home/docs/.dask'], env=None)- Collect configuration from paths and environment variables
Parameters:
- paths:List[str]
A list of paths to search for yaml config files
env:dict
- The system environment variablesReturns:
- config: dict
See also
dask.config.refresh
- collect configuration and update into primary config
dask.config.
refresh
(config={'temporary-directory': None, 'array': {'svg': {'size': 120}, 'chunk-size': '128MiB', 'rechunk-threshold': 4}}, defaults=[{'temporary-directory': None, 'array': {'svg': {'size': 120}}}, {'array': {'chunk-size': '128MiB', 'rechunk-threshold': 4}}], **kwargs)- Update configuration by re-reading yaml files and env variables
This mutates the global dask.config.config, or the config parameter ifpassed in.
This goes through the following stages:
- Clearing out all old configuration
- Updating from the stored defaults from downstream libraries(see update_defaults)
- Updating from yaml files and environment variablesNote that some functionality only checks configuration once at startup andmay not change behavior, even if configuration changes. It is recommendedto restart your python process if convenient to ensure that newconfiguration changes take place.
See also
dask.config.collect
- for parameters
dask.config.update_defaults
dask.config.
ensurefile
(_source, destination=None, comment=True)- Copy file to default location if it does not already exist
This tries to move a default configuration file to a default location ifif does not already exist. It also comments out that file by default.
This is to be used by downstream modules (like dask.distributed) that mayhave default configuration files that they wish to include in the defaultconfiguration path.
Parameters:
- source:string, filename
Source configuration file, typically within a source directory.
destination:string, directory
Destination directory. Configurable by
DASK_CONFIG
environmentvariable, falling back to ~/.config/dask.comment:bool, True by default
- Whether or not to comment out the config file when copying.
dask.config.
expandenvironment_variables
(_config)- Expand environment variables in a nested config dictionary
This function will recursively search through any nested dictionariesand/or lists.
Parameters:
- config:dict, iterable, or str
- Input object to search for environment variablesReturns:
- config:same type as input
Examples
- >>> expand_environment_variables({'x': [1, 2, '$USER']}) # doctest: +SKIP
- {'x': [1, 2, 'my-username']}