Open Source Summit Japan

Learn how to leverage new workflow management tools to simplify complex data pipelines and ETL jobs spanning heterogeneous systems. In this technical deep dive from Treasure Data, company founder and chief architect walks through the codebase of Digdag, our recently open-sourced workflow management project. I’ll show how workflows can break large, error-prone SQL statements into smaller blocks that are easier to maintain and reuse. I also demonstrate how a system using ‘last good’ checkpoints can save hours of computation when restarting failed jobs and how to use the workflows to automate data lifecycle management across Apache Hadoop, PostgreSQL, Amazon S3 and Apache Spark. You'll see a few examples where SQL-as-pipeline-code gives data scientists both the right level of ownership over production processes and a comfortable abstraction from the underlying execution engines.