From out of the box to totally custom
In project, where I’m partially responsible for integration Airflow with other components, was assigned to me simple at first glance task “Sending custom emails with tasks status”. It seems pretty easy because Airflow already have built in mechanism for such task:
Out of the box
Simply you have to set up your smtp server in email section of Airflow configuration file(default airflow.cfg) :
If you would like to receive just generic email information about errors or task retries in your workflow you just add to your dag definition email_on_failure or email_on_retry respectively. In bellow example in case of failure of ‘hello_task’ airflow will send email with notification.
If you would like to receive emails in case of failure of any of the task you could pass mail parameters in defalut_arg dictionary as follows:
Generic message could be modified we are going back to airflow config file and add:
in email section:
where example template could looks like (in airflow jinja is default template engine):
(all dynamic fields are taken from ‘context’ dictionary)
How to succeed
After couple of more troublesome workflow mailbox could look a little bit sad and depressing What about If there is need to get information about successfully ended tasks? By default there no mechanism for success emails like email_on_failure or email_on_retry. But we could use on_success_callback and on_failure_callback
How Airflow can help your business? Visit our Data Pipeline Automation page and find solution suited your needs
These two methods will trigger any custom python function and pass context dictionary as a argument. Let’s take a look on example:
TODO description
Visit our blog for more in-depth articles on Airflow:
Powerful REST API in Airflow 2.0 — what do you need to know?
- Differences between Airflow 1.10.x and 2.0
- Powerful REST API in Airflow 2.0 — what do you need to know?
- How to improve Airlfow 2.0 performance with Smart Sensors?
.webp)
