AWS Data Pipeline is a Web service that you can use to move data processing and conversion automatically. Use AWS Data Pipeline, you can define data-driven workflow, this task can be dependent on the successful implementation of the previous task. You can define data conversion parameter, AWS Data Pipeline logic you set the embodiments.
The following components AWS Data Pipeline to work together to manage your data:
-
Channel definitions designated service data management logic. For more information, see the pipeline definition file syntax .
-
Pipeline by creating an Amazon EC2 instance to perform defined work activities, to plan and run the task. You will define upload pipe to the pipe, and then activate the pipeline. You can edit the pipeline custom pipeline running and reactivate the pipeline to take effect. You can disable the pipeline, modify the data source, and then reactivate the pipeline. After the completion of the use of the pipeline can be deleted.
-
Task Runner polling task, then perform these tasks. For example, Task Runner can copy the log files to Amazon S3, and then start the Amazon EMR cluster. Task Runner is installed and run automatically on a defined resource pipeline created. You can write custom tasks to run program applications, you can also use the Task Runner application AWS Data Pipeline to offer. For more information, see the task to run the program .
For example, you can use every day AWS Data Pipeline the Web server log archive to Amazon Simple Storage Service (Amazon S3), then these weekly logs run Amazon EMR (Amazon EMR) cluster to generate traffic reports. AWS Data Pipeline plan daily task to copy the data, and plans to launch a weekly task Amazon EMR cluster. AWS Data Pipeline also make sure that Amazon EMR after waiting for the last day of data uploaded to Amazon S3, and then began its analysis, even if there is unpredictable log upload delay.
