The foregoing first DAG Airflow's already run up our first task. In this paper, to enrich this task.
Review our task content
- We define a name for the DAG
Hello-World, this is calleddag_id, - Supplement description
- Defines the scheduling interval schedule_interval, this is a cron expression
- It introduces a bash task
- There is an important parameter default_args, which is defined by parameters dag
How to perform different tasks
. Airflow inside by introducing a different operator to perform different operations at present, a number of built:
https://github.com/apache/airflow/tree/master/airflow/operators
Some third parties also contributed:
https://github.com/apache/airflow/tree/master/airflow/contrib/operators
You can also write your own plugin, making their task type plug.
When you want to use these plug-ins, as long as the introduction of
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import BranchPythonOperator
from operators.rdbms_to_redis_operator import RDBMS2RedisOperator
from operators.rdbms_to_hive_operator import RDBMS2HiveOperator
from operators.hive_to_rdbms_operator import Hive2RDBMSOperatorThen fill the required parameters:
t1 = BashOperator(task_id="hello",
bash_command="echo 'Hello World, today is {{ ds }}'",
dag=dag)Https://github.com/apache/airflow/tree/master/airflow/example_dags can refer to the source as well as plug-ins to use these tasks.
How to get the task execution date
This alone is worth pulling an article with this simple look can get through the task jinja date template variables.
The following user variables basic enough
templated_command = """
echo "current bizdate is: {{ ds }} "
echo "current bizdate in number: {{ ds_nodash }} "
echo "7days after: {{ macros.ds_add(ds, 7)}} "
echo "5 days ago: {{ macros.ds_add(ds, -5) }} "
echo "bizdate iso8601 {{ ts }} "
echo "bizdate format: {{ execution_date.strftime("%d-%m-%Y") }} "
echo "bizdate 5 days ago format: {{ (execution_date - macros.timedelta(days=5)).strftime("%Y-%m-%d") }} "
"""
t1 = BashOperator(
task_id='print_date1',
bash_command=templated_command,
# on_success_callback=compass_utils.success_callback(dingding_conn_id='dingding_bigdata', receivers="ryanmiao"),
dag=dag)Execution results log is:
echo "current bizdate is: 2019-09-28 "
echo "current bizdate in number: 20190928 "
echo "7days after: 2019-10-05 "
echo "5 days ago: 2019-09-23 "
echo "bizdate iso8601 2019-09-28T01:00:00+08:00 "
echo "bizdate format: 28-09-2019 "
echo "bizdate 5 days ago format: 2019-09-23 "Warning
Task run their own, how to run a result, we need a notification. I can tell when successful, can I tell when the failure.
default_args = {
'owner': 'ryanmiao',
'depends_on_past': False,
'start_date': datetime(2019, 5, 1, 9),
'email': ['[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
# 'on_failure_callback': compass_utils.ding_failure_callback('dingding_bigdata'),
# 'on_success_callback': compass_utils.ding_success_callback('dingding_bigdata')
}The default comes with email on failure mail notification, you need to set up email in the configuration file. Of course, we usually have their own notification service, but also integrate their own certification and the like. So, Airflow provides a notification callback.
on_failure_callback A Python function, when failure to perform
on_success_callback A Python function, when the successful execution
For example, I need to add nail notice.
from airflow.contrib.operators.dingding_operator import DingdingOperator
def failure_callback(context):
"""
The function that will be executed on failure.
:param context: The context of the executed task.
:type context: dict
"""
message = 'AIRFLOW TASK FAILURE TIPS:\n' \
'DAG: {}\n' \
'TASKS: {}\n' \
'Reason: {}\n' \
.format(context['task_instance'].dag_id,
context['task_instance'].task_id,
context['exception'])
return DingdingOperator(
task_id='dingding_success_callback',
dingding_conn_id='dingding_default',
message_type='text',
message=message,
at_all=True,
).execute(context)
args['on_failure_callback'] = failure_callbackBackground admin-connections to configure a group token nails, then there can be cited connId.
Similarly, we can call our own use http request notification service ah, used to send e-mail, phone calls or something, can be customized. The back cover custom plug-ins to achieve this custom notification.
DAG's task dependencies
Task dependencies defined dag is simple:
a >> b b依赖a
a << b a依赖b
a >> b >> c 依赖可以串起来
[a,b] >> c 可以依赖多个Wrap each dependency statement by division, eventually assembling a complete dependency.
Some parameters of the DAG
Take a look at the source code comments
"""
A dag (directed acyclic graph) is a collection of tasks with directional
dependencies. A dag also has a schedule, a start date and an end date
(optional). For each schedule, (say daily or hourly), the DAG needs to run
each individual tasks as their dependencies are met. Certain tasks have
the property of depending on their own past, meaning that they can't run
until their previous schedule (and upstream tasks) are completed.
DAGs essentially act as namespaces for tasks. A task_id can only be
added once to a DAG.
:param dag_id: The id of the DAG
:type dag_id: str
:param description: The description for the DAG to e.g. be shown on the webserver
:type description: str
:param schedule_interval: Defines how often that DAG runs, this
timedelta object gets added to your latest task instance's
execution_date to figure out the next schedule
:type schedule_interval: datetime.timedelta or
dateutil.relativedelta.relativedelta or str that acts as a cron
expression
:param start_date: The timestamp from which the scheduler will
attempt to backfill
:type start_date: datetime.datetime
:param end_date: A date beyond which your DAG won't run, leave to None
for open ended scheduling
:type end_date: datetime.datetime
:param template_searchpath: This list of folders (non relative)
defines where jinja will look for your templates. Order matters.
Note that jinja/airflow includes the path of your DAG file by
default
:type template_searchpath: str or list[str]
:param template_undefined: Template undefined type.
:type template_undefined: jinja2.Undefined
:param user_defined_macros: a dictionary of macros that will be exposed
in your jinja templates. For example, passing ``dict(foo='bar')``
to this argument allows you to ``{{ foo }}`` in all jinja
templates related to this DAG. Note that you can pass any
type of object here.
:type user_defined_macros: dict
:param user_defined_filters: a dictionary of filters that will be exposed
in your jinja templates. For example, passing
``dict(hello=lambda name: 'Hello %s' % name)`` to this argument allows
you to ``{{ 'world' | hello }}`` in all jinja templates related to
this DAG.
:type user_defined_filters: dict
:param default_args: A dictionary of default parameters to be used
as constructor keyword parameters when initialising operators.
Note that operators have the same hook, and precede those defined
here, meaning that if your dict contains `'depends_on_past': True`
here and `'depends_on_past': False` in the operator's call
`default_args`, the actual value will be `False`.
:type default_args: dict
:param params: a dictionary of DAG level parameters that are made
accessible in templates, namespaced under `params`. These
params can be overridden at the task level.
:type params: dict
:param concurrency: the number of task instances allowed to run
concurrently
:type concurrency: int
:param max_active_runs: maximum number of active DAG runs, beyond this
number of DAG runs in a running state, the scheduler won't create
new active DAG runs
:type max_active_runs: int
:param dagrun_timeout: specify how long a DagRun should be up before
timing out / failing, so that new DagRuns can be created. The timeout
is only enforced for scheduled DagRuns, and only once the
# of active DagRuns == max_active_runs.
:type dagrun_timeout: datetime.timedelta
:param sla_miss_callback: specify a function to call when reporting SLA
timeouts.
:type sla_miss_callback: types.FunctionType
:param default_view: Specify DAG default view (tree, graph, duration,
gantt, landing_times)
:type default_view: str
:param orientation: Specify DAG orientation in graph view (LR, TB, RL, BT)
:type orientation: str
:param catchup: Perform scheduler catchup (or only run latest)? Defaults to True
:type catchup: bool
:param on_failure_callback: A function to be called when a DagRun of this dag fails.
A context dictionary is passed as a single parameter to this function.
:type on_failure_callback: callable
:param on_success_callback: Much like the ``on_failure_callback`` except
that it is executed when the dag succeeds.
:type on_success_callback: callable
:param access_control: Specify optional DAG-level permissions, e.g.,
"{'role1': {'can_dag_read'}, 'role2': {'can_dag_read', 'can_dag_edit'}}"
:type access_control: dict
:param is_paused_upon_creation: Specifies if the dag is paused when created for the first time.
If the dag exists already, this flag will be ignored. If this optional parameter
is not specified, the global config setting will be used.
:type is_paused_upon_creation: bool or None
"""emmm, here it is not disassembled, I tend to use a an understanding. When used against watch.
summary
dag composition is simple, Python syntax style statement than property and yaml configuration it is easier to organize and understand.
Dag defined the parameters, define the task type Operator, defined task depends on the bin.