Authoring and Scheduling

Here you can find detailed documentation about advanced authoring and scheduling airflow DAGs. It’s recommended that you first review the pages in core concepts

Authoring

• Plugins
  • What for?
  • Why build on top of Airflow?
  • When are plugins (re)loaded?
  • Interface
  • Example
  • Exclude views from CSRF protection
  • Plugins as Python packages
  • Automatic reloading webserver
  • Troubleshooting
• Deferrable Operators & Triggers
  • Using Deferrable Operators
  • Writing Deferrable Operators
  • High Availability
• DAG File Processing
• Serialization
  • Airflow Object
• Connections & Hooks
  • Hooks
  • Custom connections
• Dynamic Task Mapping
  • Simple mapping
  • Mapping with non-TaskFlow operators
  • Assigning multiple parameters to a non-TaskFlow operator
  • Mapping over a task group
  • Filtering items from a mapped task
  • Transforming expanding data
  • Combining upstream data (aka “zipping”)
  • What data types can be expanded?
  • How do templated fields and mapped arguments interact?
  • Placing limits on mapped tasks
  • Automatically skipping zero-length maps

Scheduling

• Time Zones
  • Web UI
  • Concepts
  • Time zone aware DAGs
• Data-aware scheduling
  • Quickstart
  • What is a “dataset”?
  • How to use datasets in your DAGs
  • Multiple Datasets
  • Notes on schedules
• Timetables
  • Built-in Timetables
  • Differences between the two cron timetables