Airflow exploration

airflow python script is just a configuration file specifying the DAG's structure as code. The script's purpose is to define a DAG object. 

STEPS

import modules.

default arguments: pass a set of arguments to each task's constructor.

Instantiate a DAG:

how to pass parameters as well to this function using the PythonOperator.

There are actually two ways of passing parameters.

• First, we can use the op_args parameter which is a list of positional arguments that will get unpacked when calling the callable function.
• Second, we can use the op_kwargs parameter which is a dictionary of keyword arguments that will get unpacked in the callable function.

Return result from PythonOperator;

使用XComs in airflow. 

 References:

1. https://marclamberti.com/blog/airflow-pythonoperator/
2. https://stackoverflow.com/questions/50149085/python-airflow-return-result-from-pythonoperator