Resource Bank for Google Dataflow

This post is a resource bank for those who would love to learn more about Google Dataflow – Cloud Dataflow is a fully-managed Google Cloud Platform service for running batch and streaming Apache Beam data processing pipelines. The product team initially created this list, and I updated it with public-facing links. Hopefully, this is useful for you. Also, please feel free to comment below if you see other helpful links.

Coursera

• Specialization Home Page: https://www.coursera.org/specializations/serverless-data-processing-with-dataflow 
  • Foundations: https://www.coursera.org/learn/serverless-data-processing-with-dataflow-foundations 
  • Develop Pipelines:

https://www.coursera.org/learn/developing-pipelines-on-dataflow

• Operations:  https://www.coursera.org/learn/serverless-data-processing-with-dataflow-operations 

Dataflow Academy

Complete Learning Resources on Apache Beam site: https://beam.apache.org/documentation/resources/learning-resources/#getting-started 

Hands-on labs provided by qwiklabs. You can register a new account : https://beam.apache.org/documentation/resources/learning-resources/#interactive-labs 

Getting Started

• What is Dataflow: https://www.youtube.com/watch?v=KalJ0VuEM7s 
• Beam Katas
  • Java: https://beam.apache.org/documentation/resources/learning-resources/#java-1 
  • Python: https://beam.apache.org/documentation/resources/learning-resources/#python-1 
• Transform Catalog
  • Java: https://beam.apache.org/documentation/transforms/java/overview/ 
  • Python: https://beam.apache.org/documentation/transforms/python/overview/ 

Code Samples

• Retail Application
  • Demo: https://showcase.withgoogle.com/streaming-analytics 
  • Walkthrough: https://www.youtube.com/watch?v=JRHy8fUQLYQ 
  • Repo: https://github.com/GoogleCloudPlatform/dataflow-sample-applications/tree/master/retail/retail-java-applications 
• Patterns
  • Patterns Library: https://beam.apache.org/documentation/patterns/overview/ 
  • Common Dataflow use-case patterns, Part 1: https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-1 
  • Common Dataflow use-case patterns, Part 2: https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-2 
• Templates
  • Docs: https://cloud.google.com/dataflow/docs/guides/templates/provided-templates 
  • Repo: https://github.com/GoogleCloudPlatform/DataflowTemplates 
• Code Samples
  • Java: https://beam.apache.org/documentation/resources/learning-resources/#java-2 
  • Python: https://beam.apache.org/documentation/resources/learning-resources/#python-2 

Guides

• Production Readiness Guide
  • Planning: https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-planning https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-developing-and-testing 
  • Deploying: https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-deploying 
  • Monitoring: https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-monitoring 

Community

• Beam 
  • Dev Mailing List: https://lists.apache.org/list.html?dev@beam.apache.org 
  • User Mailing List: https://lists.apache.org/list.html?user@beam.apache.org