GCP Contains several examples and solutions to common use cases observed in various scenarios The solutions below become more complex as we incorporate more Dataflow features

Answer:
Contains several examples and solutions to common use cases observed in various scenarios

- [Ingesting data from a file into BigQuery](#ingesting-data-from-a-file-into-bigquery)
- [Transforming data in Dataflow](#transforming-data-in-dataflow)
- [Joining file and BigQuery datasets in Dataflow](#joining-file-and-bigquery-datasets-in-dataflow)
- [Ingest data from files into Bigquery reading the file structure from Datastore](#ingest-data-from-files-into-bigquery-reading-the-file-structure-from-datastore)
- [Data lake to data mart](#data-lake-to-data-mart)

The solutions below become more complex as we incorporate more Dataflow features.

## Ingesting data from a file into BigQuery
![Alt text](img/csv_file_to_bigquery.png?raw=true "CSV file to BigQuery")

This example shows how to ingest a raw CSV file into BigQuery with minimal transformation.  It is the simplest example
and a great one to start with in order to become familiar with Dataflow.

There are three main steps:
1. [Read in the file](pipelines/data_ingestion.py#L100-L106).
2. [Transform the CSV format into a dictionary format](pipelines/data_ingestion.py#L107-L113).
3. [Write the data to BigQuery](pipelines/data_ingestion.py#L114-L126).

### Read data in from the file.
![Alt text](img/csv_file.png?raw=true "CSV file")

Using the built in TextIO connector allows beam to have several workers read the file in parallel.  This allows larger
 file sizes and large number of input files to scale well within beam.

Dataflow will read in each row of data from the file and distribute the data to the next stage of the data pipeline.

### Transform the CSV format into a dictionary format.
![Alt text](img/custom_python_code.png?raw=true "Custom Python code")

This is the stage of the code where you would typically put your business logic.  In this example we are simply transforming the
data from a CSV format into a python dictionary.  The dictionary maps column names to the values we want to store in
BigQuery.

### Write the data to BigQuery.
![Alt text](img/output_to_bigquery.png?raw=true "Output to BigQuery")

Writing the data to BigQuery does not require custom code.  Passing the table name and a few other optional arguments
into BigQueryIO sets up the final stage of the pipeline.

This stage of the pipeline is typically referred to as our sink.  The sink is the final destination of data.  No more
processing will occur in the pipeline after this stage.

### Full code examples

Ready to dive deeper?  Check out the complete code [here](pipelines/data_ingestion.py).

## Transforming data in Dataflow
![Alt text](img/csv_file_to_bigquery.png?raw=true "CSV file to BigQuery")

This example builds upon simple ingestion, and demonstrates some basic data type transformations.

In line with the previous example there are 3 steps.  The transformation step is made more useful by translating the
date format from the source data into a date format BigQuery accepts.

1. [Read in the file](pipelines/data_transformation.py#L136-L142).
2. [Transform the CSV format into a dictionary format and translate the date format](pipelines/data_transformation.py#L143-L149).
3. [Write the data to BigQuery](pipelines/data_transformation.py#L150-L161).

### Read data in from the file.
![Alt text](img/csv_file.png?raw=true "CSV file")

Similar to the previous example, this example uses TextIO to read the file from Google Cloud Storage.

### Transform the CSV format into a dictionary format.
![Alt text](img/custom_python_code.png?raw=true "Custom Python code")

This example builds upon the simpler ingestion example by introducing data type transformations.

### Write the data to BigQuery.
![Alt text](img/output_to_bigquery.png?raw=true "Output to BigQuery")

Just as in our previous example, this example uses BigQuery IO to write out to BigQuery.

### Full code examples

Ready to dive deeper?  Check out the complete code [here](pipelines/data_transformation.py).

## Joining file and BigQuery datasets in Dataflow
![Alt text](img/csv_join_bigquery_to_bigquery.png?raw=true "CSV file joined with BigQuery data to BigQuery")

This example demonstrates how to work with two datasets.  A primary dataset is read from a file, and another dataset
containing reference data is read from BigQuery.  The two datasets are then joined in Dataflow before writing the joined
dataset to BigQuery.

This pipeline contains 4 steps:
1. [Read in the primary dataset from a file](pipelines/data_enrichment.py#L165-L176).
2. [Read in the reference data from BigQuery](pipelines/data_enrichment.py#L155-L163).
3. [Custom Python code](pipelines/data_enrichment.py#L138-L143) is used to [join the two datasets](pipelines/data_enrichment.py#L177-L180).
4. [The joined dataset is written out to BigQuery](pipelines/data_enrichment.py#L181-L194).

### Read in the primary dataset from a file
![Alt text](img/csv_file.png?raw=true "CSV file")

Similar to previous examples, we use TextIO to read the dataset from a CSV file.

### Read in the reference data from BigQuery
![Alt text](img/import_state_name_from_bigquery.png?raw=true "Import state name data from BigQuery")

Using BigQueryIO, we can specify a query to read data from.  Dataflow then is able to distribute the data
from BigQuery to the next stages in the pipeline.

In this example the additional dataset is represented as a side input.  Side inputs in Dataflow are typically reference
datasets that fit into memory.  Other examples will explore alternative methods for joining datasets which work well for
datasets that do not fit into memory.

### Custom Python code is used to join the two datasets
![Alt text](img/3_custom_python_code.png?raw=true "Custom python code")

Using custom python code, we join the two datasets together.  Because the two datasets are dictionaries,
the python code is the same as it would be for unioning any two python dictionaries.

### The joined dataset is written out to BigQuery
![Alt text](img/4_output_to_bigquery.png?raw=true "Custom python code")

Finally the joined dataset is written out to BigQuery.  This uses the same BigQueryIO API which is used in previous
examples.

### Full code examples

Ready to dive deeper?  Check out the complete code [here](pipelines/data_enrichment.py).

## Ingest data from files into Bigquery reading the file structure from Datastore

In this example we create a Python [Apache Beam](https://beam.apache.org/) pipeline running on [Google Cloud Dataflow](https://cloud.google.com/dataflow/) to import CSV files into BigQuery using the following architecture:

![Apache Beam pipeline to import CSV into BQ](img/data_ingestion_configurable.jpg)

The architecture uses:
* [Google Cloud Storage]() to store CSV source files
* [Google Cloud Datastore](https://cloud.google.com/datastore/docs/concepts/overview) to store CSV file structure and field type
* [Google Cloud Dataflow](https://cloud.google.com/dataflow/) to read files from Google Cloud Storage, Transform data base on the structure of the file and import the data into Google BigQuery
* [Google BigQuery](https://cloud.google.com/bigquery/) to store data in a Data Lake.

You can use this script as a starting point to import your files into Google BigQuery. You'll probably need to adapt the script logic to your file name structure or to your peculiar needs.

### 1. Prerequisites
 - Up and running GCP project with enabled billing account
 - gcloud installed and initiated to your project
 - Google Cloud Datastore enabled
 - Google Cloud Dataflow API enabled
 - Google Cloud Storage Bucket containing the file to import (CSV format) using the following naming convention: `TABLENAME_*.csv`
 - Google Cloud Storage Bucket for tem and staging Google Dataflow files
 - Google BigQuery dataset
 - [Python](https://www.python.org/) >= 2.7 and python-dev module
 - gcc
 - Google Cloud [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)

### 2. Create virtual environment
Create a new virtual environment (recommended) and install requirements:

```
virtualenv env
source ./env/bin/activate
pip install -r requirements.txt
```

### 3. Configure Table schema
Create a file that contains the structure of the CSVs to be imported. Filename needs to follow convention: `TABLENAME.csv`.

Example:
```
name,STRING
surname,STRING
age,INTEGER
```

You can check parameters accepted by the `datastore_schema_import.py` script with the following command:
```
python pipelines/datastore_schema_import.py --help
```

Run the `datastore_schema_import.py` script to create the entry in Google Cloud Datastore using the following command:
```
python pipelines/datastore_schema_import.py --schema-file=<path_to_TABLENAME.csv>
```

### 4. Upload files into Google Cloud Storage
Upload files to be imported into Google Bigquery in a Google Cloud Storage Bucket. You can use `gsutil` using a command like:
```
gsutil cp [LOCAL_OBJECT_LOCATION] gs://[DESTINATION_BUCKET_NAME]/
```
To optimize upload of big files see the [documentation](https://cloud.google.com/solutions/transferring-big-data-sets-to-gcp).
Files need to be in CSV format, with the name of the column as first row. For example:
```
name,surname,age
test_1,test_1,30
test_2,test_2,40
"test_3, jr",surname,50
```

### 5. Run pipeline
You can check parameters accepted by the `data_ingestion_configurable.py` script with the following command:
```
python pipelines/data_ingestion_configurable --help
```

You can run the pipeline locally with the following command:
```
python pipelines/data_ingestion_configurable.py \
--project=###PUT HERE PROJECT ID### \
--input-bucket=###PUT HERE GCS BUCKET NAME: gs://bucket_name ### \
--input-path=###PUT HERE INPUT FOLDER### \
--input-files=###PUT HERE FILE NAMES### \
--bq-dataset=###PUT HERE BQ DATASET NAME###
```

or you can run the pipeline on Google Dataflow using the following command:

```
python pipelines/data_ingestion_configurable.py \
--runner=DataflowRunner \
--max_num_workers=100 \
--autoscaling_algorithm=THROUGHPUT_BASED \
--region=###PUT HERE REGION### \
--staging_location=###PUT HERE GCS STAGING LOCATION### \
--temp_location=###PUT HERE GCS TMP LOCATION###\
--project=###PUT HERE PROJECT ID### \
--input-bucket=###PUT HERE GCS BUCKET NAME### \
--input-path=###PUT HERE INPUT FOLDER### \
--input-files=###PUT HERE FILE NAMES### \
--bq-dataset=###PUT HERE BQ DATASET NAME###
```

### 6. Check results
You can check data imported into Google BigQuery from the Google Cloud Console UI.

## Data lake to data mart
![Alt text](img/data_lake_to_data_mart.png?raw=true "Data lake to data mart")

This example demonstrates joining data from two different datasets in BigQuery, applying transformations to
the joined dataset before uploading to BigQuery.

Joining two datasets from BigQuery is a common use case when a data lake has been implemented in BigQuery.
Creating a data mart with denormalized datasets facilitates better performance when using visualization tools.

This pipeline contains 4 steps:
1. [Read in the primary dataset from BigQuery](pipelines/data_lake_to_mart.py#L278-L283).
2. [Read in the reference data from BigQuery](pipelines/data_lake_to_mart.py#L248-L276).
3. [Custom Python code](pipelines/data_lake_to_mart.py#L210-L224) is used to [join the two datasets](pipelines/data_lake_to_mart.py#L284-L287).
Alternatively, [CoGroupByKey can be used to join the two datasets](pipelines/data_lake_to_mart_cogroupbykey.py#L300-L310).
4. [The joined dataset is written out to BigQuery](pipelines/data_lake_to_mart.py#L288-L301).

### Read in the primary dataset from BigQuery
![Alt text](img/1_query_orders.png?raw=true "Read from BigQuery")

Similar to previous examples, we use BigQueryIO to read the dataset from the results of a query.  In this case our
main dataset is a fake orders dataset, containing a history of orders and associated data like quantity.

### Read in the reference data from BigQuery
![Alt text](img/2_query_account_details.png?raw=true "Import state name data from BigQuery")

In this example we use a fake account details dataset.  This represents a common use case for denormalizing a dataset.
The account details information contains attributes linked to the accounts in the orders dataset.  For example the
address and city of the account.

### Custom Python code is used to join the two datasets
![Alt text](img/3_custom_python_code.png?raw=true "Custom python code")

Using custom python code, we join the two datasets together.  We provide two examples of joining these datasets.  The
first example uses side inputs, which require the dataset fit into memory.  The second example demonstrates how to use
CoGroupByKey to join the datasets.

CoGroupByKey will facilitate joins between two datesets even if neither fit into memory.  Explore the comments in the
two code examples for a more in depth explanation.

### The joined dataset is written out to BigQuery
![Alt text](img/4_output_to_bigquery.png?raw=true "Custom python code")

Finally the joined dataset is written out to BigQuery.  This uses the same BigQueryIO API which is used in previous
examples.

### Full code examples

Ready to dive deeper?  Check out the complete code.
The example using side inputs is [here](pipelines/data_lake_to_mart.py) and the example using CoGroupByKey is
[here](pipelines/data_lake_to_mart_cogroupbykey.py).

Contains several examples and solutions to common use cases observed in various scenarios      Ingesting data from a file into BigQuery   ingesting data from a file into bigquery     Transforming data in Dataflow   transforming data in dataflow     Joining file and BigQuery datasets in Dataflow   joining file and bigquery datasets in dataflow     Ingest data from files into Bigquery reading the file structure from Datastore   ingest data from files into bigquery reading the file structure from datastore     Data lake to data mart   data lake to data mart   The solutions below become more complex as we incorporate more Dataflow features      Ingesting data from a file into BigQuery   Alt text  img csv file to bigquery png raw true  CSV file to BigQuery    This example shows how to ingest a raw CSV file into BigQuery with minimal transformation   It is the simplest example and a great one to start with in order to become familiar with Dataflow   There are three main steps  1   Read in the file  pipelines data ingestion py L100 L106   2   Transform the CSV format into a dictionary format  pipelines data ingestion py L107 L113   3   Write the data to BigQuery  pipelines data ingestion py L114 L126         Read data in from the file    Alt text  img csv file png raw true  CSV file    Using the built in TextIO connector allows beam to have several workers read the file in parallel   This allows larger  file sizes and large number of input files to scale well within beam   Dataflow will read in each row of data from the file and distribute the data to the next stage of the data pipeline       Transform the CSV format into a dictionary format    Alt text  img custom python code png raw true  Custom Python code    This is the stage of the code where you would typically put your business logic   In this example we are simply transforming the data from a CSV format into a python dictionary   The dictionary maps column names to the values we want to store in BigQuery       Write the data to BigQuery    Alt text  img output to bigquery png raw true  Output to BigQuery    Writing the data to BigQuery does not require custom code   Passing the table name and a few other optional arguments into BigQueryIO sets up the final stage of the pipeline   This stage of the pipeline is typically referred to as our sink   The sink is the final destination of data   No more processing will occur in the pipeline after this stage       Full code examples  Ready to dive deeper   Check out the complete code  here  pipelines data ingestion py       Transforming data in Dataflow   Alt text  img csv file to bigquery png raw true  CSV file to BigQuery    This example builds upon simple ingestion  and demonstrates some basic data type transformations   In line with the previous example there are 3 steps   The transformation step is made more useful by translating the date format from the source data into a date format BigQuery accepts   1   Read in the file  pipelines data transformation py L136 L142   2   Transform the CSV format into a dictionary format and translate the date format  pipelines data transformation py L143 L149   3   Write the data to BigQuery  pipelines data transformation py L150 L161         Read data in from the file    Alt text  img csv file png raw true  CSV file    Similar to the previous example  this example uses TextIO to read the file from Google Cloud Storage       Transform the CSV format into a dictionary format    Alt text  img custom python code png raw true  Custom Python code    This example builds upon the simpler ingestion example by introducing data type transformations       Write the data to BigQuery    Alt text  img output to bigquery png raw true  Output to BigQuery    Just as in our previous example  this example uses BigQuery IO to write out to BigQuery       Full code examples  Ready to dive deeper   Check out the complete code  here  pipelines data transformation py       Joining file and BigQuery datasets in Dataflow   Alt text  img csv join bigquery to bigquery png raw true  CSV file joined with BigQuery data to BigQuery    This example demonstrates how to work with two datasets   A primary dataset is read from a file  and another dataset containing reference data is read from BigQuery   The two datasets are then joined in Dataflow before writing the joined dataset to BigQuery   This pipeline contains 4 steps  1   Read in the primary dataset from a file  pipelines data enrichment py L165 L176   2   Read in the reference data from BigQuery  pipelines data enrichment py L155 L163   3   Custom Python code  pipelines data enrichment py L138 L143  is used to  join the two datasets  pipelines data enrichment py L177 L180   4   The joined dataset is written out to BigQuery  pipelines data enrichment py L181 L194         Read in the primary dataset from a file   Alt text  img csv file png raw true  CSV file    Similar to previous examples  we use TextIO to read the dataset from a CSV file       Read in the reference data from BigQuery   Alt text  img import state name from bigquery png raw true  Import state name data from BigQuery    Using BigQueryIO  we can specify a query to read data from   Dataflow then is able to distribute the data from BigQuery to the next stages in the pipeline   In this example the additional dataset is represented as a side input   Side inputs in Dataflow are typically reference datasets that fit into memory   Other examples will explore alternative methods for joining datasets which work well for datasets that do not fit into memory       Custom Python code is used to join the two datasets   Alt text  img 3 custom python code png raw true  Custom python code    Using custom python code  we join the two datasets together   Because the two datasets are dictionaries  the python code is the same as it would be for unioning any two python dictionaries       The joined dataset is written out to BigQuery   Alt text  img 4 output to bigquery png raw true  Custom python code    Finally the joined dataset is written out to BigQuery   This uses the same BigQueryIO API which is used in previous examples       Full code examples  Ready to dive deeper   Check out the complete code  here  pipelines data enrichment py       Ingest data from files into Bigquery reading the file structure from Datastore  In this example we create a Python  Apache Beam  https   beam apache org   pipeline running on  Google Cloud Dataflow  https   cloud google com dataflow   to import CSV files into BigQuery using the following architecture     Apache Beam pipeline to import CSV into BQ  img data ingestion configurable jpg   The architecture uses     Google Cloud Storage    to store CSV source files    Google Cloud Datastore  https   cloud google com datastore docs concepts overview  to store CSV file structure and field type    Google Cloud Dataflow  https   cloud google com dataflow   to read files from Google Cloud Storage  Transform data base on the structure of the file and import the data into Google BigQuery    Google BigQuery  https   cloud google com bigquery   to store data in a Data Lake   You can use this script as a starting point to import your files into Google BigQuery  You ll probably need to adapt the script logic to your file name structure or to your peculiar needs       1  Prerequisites    Up and running GCP project with enabled billing account    gcloud installed and initiated to your project    Google Cloud Datastore enabled    Google Cloud Dataflow API enabled    Google Cloud Storage Bucket containing the file to import  CSV format  using the following naming convention   TABLENAME   csv     Google Cloud Storage Bucket for tem and staging Google Dataflow files    Google BigQuery dataset     Python  https   www python org      2 7 and python dev module    gcc    Google Cloud  Application Default Credentials  https   cloud google com sdk gcloud reference auth application default login       2  Create virtual environment Create a new virtual environment  recommended  and install requirements       virtualenv env source   env bin activate pip install  r requirements txt          3  Configure Table schema Create a file that contains the structure of the CSVs to be imported  Filename needs to follow convention   TABLENAME csv    Example      name STRING surname STRING age INTEGER      You can check parameters accepted by the  datastore schema import py  script with the following command      python pipelines datastore schema import py   help      Run the  datastore schema import py  script to create the entry in Google Cloud Datastore using the following command      python pipelines datastore schema import py   schema file  path to TABLENAME csv           4  Upload files into Google Cloud Storage Upload files to be imported into Google Bigquery in a Google Cloud Storage Bucket  You can use  gsutil  using a command like      gsutil cp  LOCAL OBJECT LOCATION  gs    DESTINATION BUCKET NAME       To optimize upload of big files see the  documentation  https   cloud google com solutions transferring big data sets to gcp   Files need to be in CSV format  with the name of the column as first row  For example      name surname age test 1 test 1 30 test 2 test 2 40  test 3  jr  surname 50          5  Run pipeline You can check parameters accepted by the  data ingestion configurable py  script with the following command      python pipelines data ingestion configurable   help      You can run the pipeline locally with the following command      python pipelines data ingestion configurable py     project    PUT HERE PROJECT ID        input bucket    PUT HERE GCS BUCKET NAME  gs   bucket name         input path    PUT HERE INPUT FOLDER        input files    PUT HERE FILE NAMES        bq dataset    PUT HERE BQ DATASET NAME         or you can run the pipeline on Google Dataflow using the following command       python pipelines data ingestion configurable py     runner DataflowRunner     max num workers 100     autoscaling algorithm THROUGHPUT BASED     region    PUT HERE REGION        staging location    PUT HERE GCS STAGING LOCATION        temp location    PUT HERE GCS TMP LOCATION       project    PUT HERE PROJECT ID        input bucket    PUT HERE GCS BUCKET NAME        input path    PUT HERE INPUT FOLDER        input files    PUT HERE FILE NAMES        bq dataset    PUT HERE BQ DATASET NAME             6  Check results You can check data imported into Google BigQuery from the Google Cloud Console UI      Data lake to data mart   Alt text  img data lake to data mart png raw true  Data lake to data mart    This example demonstrates joining data from two different datasets in BigQuery  applying transformations to the joined dataset before uploading to BigQuery   Joining two datasets from BigQuery is a common use case when a data lake has been implemented in BigQuery  Creating a data mart with denormalized datasets facilitates better performance when using visualization tools   This pipeline contains 4 steps  1   Read in the primary dataset from BigQuery  pipelines data lake to mart py L278 L283   2   Read in the reference data from BigQuery  pipelines data lake to mart py L248 L276   3   Custom Python code  pipelines data lake to mart py L210 L224  is used to  join the two datasets  pipelines data lake to mart py L284 L287   Alternatively   CoGroupByKey can be used to join the two datasets  pipelines data lake to mart cogroupbykey py L300 L310   4   The joined dataset is written out to BigQuery  pipelines data lake to mart py L288 L301         Read in the primary dataset from BigQuery   Alt text  img 1 query orders png raw true  Read from BigQuery    Similar to previous examples  we use BigQueryIO to read the dataset from the results of a query   In this case our main dataset is a fake orders dataset  containing a history of orders and associated data like quantity       Read in the reference data from BigQuery   Alt text  img 2 query account details png raw true  Import state name data from BigQuery    In this example we use a fake account details dataset   This represents a common use case for denormalizing a dataset  The account details information contains attributes linked to the accounts in the orders dataset   For example the address and city of the account       Custom Python code is used to join the two datasets   Alt text  img 3 custom python code png raw true  Custom python code    Using custom python code  we join the two datasets together   We provide two examples of joining these datasets   The first example uses side inputs  which require the dataset fit into memory   The second example demonstrates how to use CoGroupByKey to join the datasets   CoGroupByKey will facilitate joins between two datesets even if neither fit into memory   Explore the comments in the two code examples for a more in depth explanation       The joined dataset is written out to BigQuery   Alt text  img 4 output to bigquery png raw true  Custom python code    Finally the joined dataset is written out to BigQuery   This uses the same BigQueryIO API which is used in previous examples       Full code examples  Ready to dive deeper   Check out the complete code  The example using side inputs is  here  pipelines data lake to mart py  and the example using CoGroupByKey is  here  pipelines data lake to mart cogroupbykey py


GCP The test app generates time series with a custom metric called This example demonstrates creating alerts for missing monitoring data with Stackdriver alerts for missing timeseries them is missing If one or two timeseries are missing you want exactly one alert When there is a total outage you still want to get one alert not 100 suppose that you have 100 timeseries and you want to find out when any one of Stackdriver in a way that duplicate alerts are not generated For example alerts

Answer:
# Stackdriver alerts for missing timeseries

This example demonstrates creating alerts for missing monitoring data with
Stackdriver in a way that duplicate alerts are not generated. For example,
suppose that you have 100 timeseries and you want to find out when any one of
them is missing. If one or two timeseries are missing, you want exactly one
alert. When there is a total outage you still want to get one alert, not 100
alerts.

The test app generates time series with a custom metric called
task_latency_distribution with tags based on a partition label. Each
partition generates its own time series.

The example assumes that you are familiar with Stackdriver Monitoring and
Alerting. It builds on the discussion in
[Alerting policies in depth](https://cloud.google.com/monitoring/alerts/concepts-indepth).

## Setup
Clone this repo and change to this working directory. Enable the Stackdriver
Monitoring API

```shell
gcloud services enable monitoring.googleapis.com
```

In the GCP Console, go to
[Monitoring](https://console.cloud.google.com/monitoring).
If you have not already created a workspace for this project before, click New
workspace, and then click Add. It takes a few minutes to create the workspace.
Click Alerting | Policies overview. The list should be empty at this point
unless you have created policies previously.

## Deploy the app

The example code is based on the Go code in
[Custom metrics with OpenCensus](https://cloud.google.com/monitoring/custom-metrics/open-census).

[Download](https://golang.org/dl/) and install the latest version of Go.

Build the test app

```shell
go build
```

The instructions here are based on the article
[Setting up authentication](https://cloud.google.com/monitoring/docs/reference/libraries#setting_up_authentication)
for the Stackdriver client library. Note that if you run the test app on a
Google Cloud Compute Engine instance, on Google Kubernetes Engine, on App
Engine, or on Cloud Run, then you will not need to create a service account
or download the credentials.

First, set the project id in a shell variable

```shell
export GOOGLE_CLOUD_PROJECT=[your project]
```

Create a service account:

```shell
SA_NAME=stackdriver-metrics-writer
gcloud iam service-accounts create $SA_NAME \
 --display-name="Stackdriver Metrics Writer"
```

```shell
SA_ID="$SA_NAME@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
  --member "serviceAccount:$SA_ID" --role "roles/monitoring.metricWriter"
```

Generate a credentials file with an exported variable
GOOGLE_APPLICATION_CREDENTIALS referring to it.

```shell
mkdir -p ~/.auth
chmod go-rwx ~/.auth
export GOOGLE_APPLICATION_CREDENTIALS=~/.auth/stackdriver_demo_credentials.json
gcloud iam service-accounts keys create $GOOGLE_APPLICATION_CREDENTIALS \
 --iam-account $SA_ID
```

Run the program with three partitions, labeled "1", "2", and "3"

```shell
./alert-absence-demo --labels "1,2,3"
```

This will write three time series with the given labels. A few minutes after
starting the app you should be able to see the time series data in the
Stackdriver Metric explorer.

## Create the policy

Create a notification channel with the command

```shell
EMAIL="your email"
CHANNEL=$(gcloud alpha monitoring channels create \
  --channel-labels=email_address=$EMAIL \
  --display-name="Email to project owner" \
  --type=email \
  --format='value(name)')
```

Create an alert policy with the command

```shell
gcloud alpha monitoring policies create \
--notification-channels=$CHANNEL \
--documentation-from-file=policy_doc.md \
--policy-from-file=alert_policy.json
```

At this point no alerts should be firing. You can check that in the Stackdriver
Monitoring console alert policy detail.

## Testing

Kill the processes with CTL-C and restart it with only two partitions:

```shell
./alert-absence-demo --labels "1,2"
```

An alert should be generated in about 5 minutes with a subject like ‘One of the
timeseries is absent.’ You should be able to see this in the Stackdriver console
and you should also receive an email notification.

Restart the process with three partitions:

```shell
./alert-absence-demo --labels "1,2,3"
```

After a few minutes the alert should resolve itself.

Stop the process and restart it with only one partition:

```shell
./alert-absence-demo --labels "1"
```

Check that only one alert is fired.

Stop the process and do not restart it. You should see an alert that indicates
all time series are absent.

Stackdriver alerts for missing timeseries  This example demonstrates creating alerts for missing monitoring data with Stackdriver in a way that duplicate alerts are not generated  For example  suppose that you have 100 timeseries and you want to find out when any one of them is missing  If one or two timeseries are missing  you want exactly one alert  When there is a total outage you still want to get one alert  not 100 alerts   The test app generates time series with a custom metric called task latency distribution with tags based on a partition label  Each partition generates its own time series   The example assumes that you are familiar with Stackdriver Monitoring and Alerting  It builds on the discussion in  Alerting policies in depth  https   cloud google com monitoring alerts concepts indepth       Setup Clone this repo and change to this working directory  Enable the Stackdriver Monitoring API     shell gcloud services enable monitoring googleapis com      In the GCP Console  go to  Monitoring  https   console cloud google com monitoring   If you have not already created a workspace for this project before  click New workspace  and then click Add  It takes a few minutes to create the workspace  Click Alerting   Policies overview  The list should be empty at this point unless you have created policies previously      Deploy the app  The example code is based on the Go code in  Custom metrics with OpenCensus  https   cloud google com monitoring custom metrics open census     Download  https   golang org dl   and install the latest version of Go   Build the test app     shell go build      The instructions here are based on the article  Setting up authentication  https   cloud google com monitoring docs reference libraries setting up authentication  for the Stackdriver client library  Note that if you run the test app on a Google Cloud Compute Engine instance  on Google Kubernetes Engine  on App Engine  or on Cloud Run  then you will not need to create a service account or download the credentials   First  set the project id in a shell variable     shell export GOOGLE CLOUD PROJECT  your project       Create a service account      shell SA NAME stackdriver metrics writer gcloud iam service accounts create  SA NAME      display name  Stackdriver Metrics Writer          shell SA ID   SA NAME  GOOGLE CLOUD PROJECT iam gserviceaccount com  gcloud projects add iam policy binding  GOOGLE CLOUD PROJECT       member  serviceAccount  SA ID    role  roles monitoring metricWriter       Generate a credentials file with an exported variable GOOGLE APPLICATION CREDENTIALS referring to it      shell mkdir  p    auth chmod go rwx    auth export GOOGLE APPLICATION CREDENTIALS    auth stackdriver demo credentials json gcloud iam service accounts keys create  GOOGLE APPLICATION CREDENTIALS      iam account  SA ID      Run the program with three partitions  labeled  1    2   and  3      shell   alert absence demo   labels  1 2 3       This will write three time series with the given labels  A few minutes after starting the app you should be able to see the time series data in the Stackdriver Metric explorer      Create the policy  Create a notification channel with the command     shell EMAIL  your email  CHANNEL   gcloud alpha monitoring channels create       channel labels email address  EMAIL       display name  Email to project owner        type email       format  value name         Create an alert policy with the command     shell gcloud alpha monitoring policies create     notification channels  CHANNEL     documentation from file policy doc md     policy from file alert policy json      At this point no alerts should be firing  You can check that in the Stackdriver Monitoring console alert policy detail      Testing  Kill the processes with CTL C and restart it with only two partitions      shell   alert absence demo   labels  1 2       An alert should be generated in about 5 minutes with a subject like  One of the timeseries is absent   You should be able to see this in the Stackdriver console and you should also receive an email notification   Restart the process with three partitions      shell   alert absence demo   labels  1 2 3       After a few minutes the alert should resolve itself   Stop the process and restart it with only one partition      shell   alert absence demo   labels  1       Check that only one alert is fired   Stop the process and do not restart it  You should see an alert that indicates all time series are absent


GCP An HTTP POST to the airflow endpoint from an on prem system is used as a trigger to initiate the workflow Cloud Composer Ephemeral Dataproc Cluster for Spark Job Workflow Overview

Answer:
## Cloud Composer: Ephemeral Dataproc Cluster for Spark Job
### Workflow Overview

***

![Alt text](../img/composer-http-post-arch.png "A diagram illustrating the workflow described below.")

An HTTP POST to the airflow endpoint from an on-prem system is used as a trigger to initiate the workflow.

At a high level the Cloud Composer workflow performs the following steps:
1. Extracts some metadata from the HTTP POST that triggered the workflow.
1. Spins up a Dataproc Cluster.
1. Submits a Spark job that performs the following:
    * Reads newline delimited json data generated by an export from the [nyc-tlc:yellow.trips public
     BigQuery table](https://bigquery.cloud.google.com/table/nyc-tlc:yellow.trips?pli=1).
    * Enhances the data with an average_speed column.
    * Writes the enhanced data as in CSV format to a temporary location in Google Cloud storage.
1. Tear down the Dataproc Cluster Load these files to BigQuery.
1. Clean up the temporary path of enhanced data in GCS.

##### 1. Extract metadata from POST:
When there is an HTTP POST to the airflow endpoint it should contain a paylaod of the following structure.
```
    payload = {
        'run_id': 'post-triggered-run-%s' % datetime.now().strftime('%Y%m%d%H%M%s'),
        'conf':  "{'raw_path': raw_path, 'transformed_path': transformed_path}"

    }
```
Where raw_path is a timestamped path to the existing raw files in gcs in newline delimited json format and
transformed path is a path with matching time stamp to stage the enhanced file before loading to BigQuery.

##### 2 & 3. Spins up a Dataproc Cluster and submit Spark Job

The workflow then provisions a Dataproc Cluster and submits a spark job to enhance the data.

##### 4. Move to processed bucket

Based on the status of the Spark job, the workflow will then move the processed files to a Cloud Storage bucket setup to store processed data. A separate folder is created along with a processed date field to hold the files in this bucket.

##### Full code examples

Ready to dive deeper? Check out the [complete code](ephemeral_dataproc_spark_dag.py)

***

#### Setup and Pre-requisites
It is recommended that virtualenv be used to keep everything tidy. The [requirements.txt](requirements.txt) describes the dependencies needed for the code used in this repo.
```bash
virtualenv composer-env
source composer-env/bin/activate
```
The POST will need to be authenticated with [Identity Aware Proxy](https://cloud.google.com/iap/docs/).
We reccomend doing this by copying the latest version of [make_iap_request.py](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py)
from the Google Cloud python-docs-samples repo and using the provided [dag_trigger.py](dag_trigger.py).
```bash
pip install -r ~/professional-services/examples/cloud-composer-examples/requirements.txt
wget https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/iap/requirements.txt -O ~/professional-services/examples/cloud-composer-examples/iap_requirements.txt
pip install -r iap_requirements.txt
wget https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/iap/make_iap_request.py -O ~/professional-services/examples/cloud-composer-examples/composer_http_post_example/make_iap_request.py
```
(Or if your are on a Mac you can use curl.)
```bash
# From the cloud-composer-examples directory
pip install -r ~/professional-services/examples/cloud-composer-examples/requirements.txt
curl https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/iap/requirements.txt >> ~/professional-services/examples/cloud-composer-examples/iap_requirements.txt
pip install -r iap-requirements.txt
curl https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/iap/make_iap_request.py >> ~/professional-services/examples/cloud-composer-examples/composer_http_post_example/make_iap_request.py
```

Note that we skipped install pyspark for the purposes of being lighter weight to stand up this example. If you have the need to test pyspark locally you should additionally run:
```bash
pip install pyspark>=2.3.1
```

The following high-level steps describe the setup needed to run this example:
0. set your project information.
```bash
export PROJECT=<REPLACE-THIS-WITH-YOUR-PROJECT-ID>
gcloud config set project $PROJECT
```
1. Create a Cloud Storage (GCS) bucket for receiving input files (*input-gcs-bucket*).
```bash
gsutil mb -c regional -l us-central1 gs://$PROJECT
```
2. Export the public BigQuery Table to a new dataset.
```bash
bq mk ComposerDemo
export EXPORT_TS=`date "+%Y-%m-%dT%H%M%S"`&& bq extract \
--destination_format=NEWLINE_DELIMITED_JSON \
nyc-tlc:yellow.trips \
gs://$PROJECT/cloud-composer-lab/raw-$EXPORT_TS/nyc-tlc-yellow-*.json
```
3. Create a Cloud Composer environment - Follow [these](https://cloud.google.com/composer/docs/quickstart) steps to create a Cloud Composer environment if needed (*cloud-composer-env*).
We will set these variables in the composer environment.

| Key                   | Value                                           |Example                                   |
| :--------------------- |:---------------------------------------------- |:---------------------------              |
| gcp_project           | *your-gcp-project-id*                           |cloud-comp-http-demo                        |
| gcp_bucket            | *gcs-bucket-with-raw-files*                     |cloud-comp-http-demo          |
| gce_zone              | *compute-engine-zone*                           |us-central1-b                              |

```bash
gcloud beta composer environments create demo-ephemeral-dataproc \
    --location us-central1 \
    --zone us-central1-b \
    --machine-type n1-standard-2 \
    --disk-size 20

# Set Airflow Variables in the Composer Environment we just created.
gcloud composer environments run \
demo-ephemeral-dataproc \
--location=us-central1 variables -- \
--set gcp_project $PROJECT
gcloud composer environments run demo-ephemeral-dataproc \
--location=us-central1 variables -- \
--set gce_zone us-central1-b
gcloud composer environments run demo-ephemeral-dataproc \
--location=us-central1 variables -- \
--set gcs_bucket $PROJECT
gcloud composer environments run demo-ephemeral-dataproc \
--location=us-central1 variables -- \
--set bq_output_table $PROJECT:ComposerDemo.nyc-tlc-yellow-trips
```

6. Browse to the Cloud Composer widget in Cloud Console and click on the DAG folder icon as shown below:
![Alt text](../img/dag-folder-example.png "Screen shot showing where to find the DAG folder in the console.")

7. Upload the PySpark code [spark_avg_speed.py](composer_http_examples/spark_avg_speed.py) into a *spark-jobs* folder in GCS.
```bash
gsutil cp ~/professional-services/examples/cloud-composer-examples/composer_http_post_example/spark_avg_speed.py gs://$PROJECT/spark-jobs/
```

8. The DAG folder is essentially a Cloud Storage bucket. Upload the [ephemeral_dataproc_spark_dag.py](composer_http_examples/ephemeral_dataproc_spark_dag.py) file into the folder:

```bash
gsutil cp ~/professional-services/examples/cloud-composer-examples/composer_http_post_example/ephemeral_dataproc_spark_dag.py gs://<dag-folder>/dags
```
***

##### Triggering the workflow

Make sure that you have installed the `iap_requirements.txt` in the steps above.
You will need to create a service account with the necessary permissions to create an IAP request and use Cloud Composer to use the `dag_trigger.py` script. This can be achieved with the following commands:
```bash
gcloud iam service-accounts create dag-trigger

# Give service account permissions to create tokens for
# iap requests.
gcloud projects add-iam-policy-binding $PROJECT \
--member \
serviceAccount:dag-trigger@$PROJECT.iam.gserviceaccount.com \
--role roles/iam.serviceAccountTokenCreator

gcloud projects add-iam-policy-binding $PROJECT \
--member \
serviceAccount:dag-trigger@$PROJECT.iam.gserviceaccount.com \
--role roles/iam.serviceAccountActor

# Service account also needs to be authorized to use Composer.
gcloud projects add-iam-policy-binding $PROJECT \
--member \
serviceAccount:dag-trigger@$PROJECT.iam.gserviceaccount.com \
--role roles/composer.user

# We need a service account key to trigger the dag.
gcloud iam service-accounts keys create ~/$PROJECT-dag-trigger-key.json \
--iam-account=dag-trigger@$PROJECT.iam.gserviceaccount.com

# Finally use this as your application credentials by setting the environment variable on the machine you will run `dag_trigger.py`
export GOOGLE_APPLICATION_CREDENTIALS=~/$PROJECT-dag-trigger-key.json
```
To trigger this workflow use [dag_trigger.py](dag_trigger.py) takes 3 arguments as shown below
```bash
python dag_trigger.py \
--url=<airflow endpoint url> \
--iapClientId=<client id> \
--raw_path=<path to raw files for enhancement in GCS>
```
The endpoint of triggering the dag had the following structure `https://<airflow web server url>/api/experimental/dags/<dag-id>/dag_runs` in this case our dag-id is average-speed.
The airflow webserver can be found once your composer environment is set up by clicking on your environment in the console and checking here:

![Alt text](../img/airflow-ui.png "Screen Shot showing how to get the airflow URL")

In oder to obtain your `--iapClientId`
Visit the Airflow URL https://YOUR_UNIQUE_ID.appspot.com (which you noted in the last step) in an incognito window, *don't* authenticate or login, and first landing page for IAP Auth has client Id in the url in the address bar:
https://accounts.google.com/signin/oauth/identifier?**client_id=00000000000-xxxx0x0xx0xx00xxxx0x00xxx0xxxxx.apps.googleusercontent.com**&as=a6VGEPwFpCL1qIwusi49IQ&destination=https%3A%2F%2Fh0b798498b93687a6-tp.appspot.com&approval_state=!ChRKSmd1TVc1VlQzMDB3MHI2UGI4SxIfWXhaRjJLcWdwcndRVUU3MWpGWk5XazFEbUp6N05SWQ%E2%88%99AB8iHBUAAAAAWvsaqTGCmRazWx9NqQtnYVOllz0r2x_i&xsrfsig=AHgIfE_o0kxXt6N3ch1JH4Fb19CB7wdbMg&flowName=GeneralOAuthFlow

***

Cloud Composer  Ephemeral Dataproc Cluster for Spark Job     Workflow Overview          Alt text     img composer http post arch png  A diagram illustrating the workflow described below     An HTTP POST to the airflow endpoint from an on prem system is used as a trigger to initiate the workflow   At a high level the Cloud Composer workflow performs the following steps  1  Extracts some metadata from the HTTP POST that triggered the workflow  1  Spins up a Dataproc Cluster  1  Submits a Spark job that performs the following        Reads newline delimited json data generated by an export from the  nyc tlc yellow trips public      BigQuery table  https   bigquery cloud google com table nyc tlc yellow trips pli 1         Enhances the data with an average speed column        Writes the enhanced data as in CSV format to a temporary location in Google Cloud storage  1  Tear down the Dataproc Cluster Load these files to BigQuery  1  Clean up the temporary path of enhanced data in GCS         1  Extract metadata from POST  When there is an HTTP POST to the airflow endpoint it should contain a paylaod of the following structure          payload              run id    post triggered run  s    datetime now   strftime   Y m d H M s             conf       raw path   raw path   transformed path   transformed path              Where raw path is a timestamped path to the existing raw files in gcs in newline delimited json format and transformed path is a path with matching time stamp to stage the enhanced file before loading to BigQuery           2   3  Spins up a Dataproc Cluster and submit Spark Job  The workflow then provisions a Dataproc Cluster and submits a spark job to enhance the data         4  Move to processed bucket  Based on the status of the Spark job  the workflow will then move the processed files to a Cloud Storage bucket setup to store processed data  A separate folder is created along with a processed date field to hold the files in this bucket         Full code examples  Ready to dive deeper  Check out the  complete code  ephemeral dataproc spark dag py             Setup and Pre requisites It is recommended that virtualenv be used to keep everything tidy  The  requirements txt  requirements txt  describes the dependencies needed for the code used in this repo     bash virtualenv composer env source composer env bin activate     The POST will need to be authenticated with  Identity Aware Proxy  https   cloud google com iap docs    We reccomend doing this by copying the latest version of  make iap request py  https   github com GoogleCloudPlatform python docs samples blob master iap make iap request py  from the Google Cloud python docs samples repo and using the provided  dag trigger py  dag trigger py      bash pip install  r   professional services examples cloud composer examples requirements txt wget https   raw githubusercontent com GoogleCloudPlatform python docs samples master iap requirements txt  O   professional services examples cloud composer examples iap requirements txt pip install  r iap requirements txt wget https   raw githubusercontent com GoogleCloudPlatform python docs samples master iap make iap request py  O   professional services examples cloud composer examples composer http post example make iap request py      Or if your are on a Mac you can use curl      bash   From the cloud composer examples directory pip install  r   professional services examples cloud composer examples requirements txt curl https   raw githubusercontent com GoogleCloudPlatform python docs samples master iap requirements txt      professional services examples cloud composer examples iap requirements txt pip install  r iap requirements txt curl https   raw githubusercontent com GoogleCloudPlatform python docs samples master iap make iap request py      professional services examples cloud composer examples composer http post example make iap request py      Note that we skipped install pyspark for the purposes of being lighter weight to stand up this example  If you have the need to test pyspark locally you should additionally run     bash pip install pyspark  2 3 1      The following high level steps describe the setup needed to run this example  0  set your project information     bash export PROJECT  REPLACE THIS WITH YOUR PROJECT ID  gcloud config set project  PROJECT     1  Create a Cloud Storage  GCS  bucket for receiving input files   input gcs bucket       bash gsutil mb  c regional  l us central1 gs    PROJECT     2  Export the public BigQuery Table to a new dataset     bash bq mk ComposerDemo export EXPORT TS  date    Y  m  dT H M S     bq extract     destination format NEWLINE DELIMITED JSON   nyc tlc yellow trips   gs    PROJECT cloud composer lab raw  EXPORT TS nyc tlc yellow   json     3  Create a Cloud Composer environment   Follow  these  https   cloud google com composer docs quickstart  steps to create a Cloud Composer environment if needed   cloud composer env    We will set these variables in the composer environment     Key                     Value                                            Example                                                                                gcp project              your gcp project id                             cloud comp http demo                            gcp bucket               gcs bucket with raw files                       cloud comp http demo              gce zone                 compute engine zone                             us central1 b                                    bash gcloud beta composer environments create demo ephemeral dataproc         location us central1         zone us central1 b         machine type n1 standard 2         disk size 20    Set Airflow Variables in the Composer Environment we just created  gcloud composer environments run   demo ephemeral dataproc     location us central1 variables        set gcp project  PROJECT gcloud composer environments run demo ephemeral dataproc     location us central1 variables        set gce zone us central1 b gcloud composer environments run demo ephemeral dataproc     location us central1 variables        set gcs bucket  PROJECT gcloud composer environments run demo ephemeral dataproc     location us central1 variables        set bq output table  PROJECT ComposerDemo nyc tlc yellow trips      6  Browse to the Cloud Composer widget in Cloud Console and click on the DAG folder icon as shown below    Alt text     img dag folder example png  Screen shot showing where to find the DAG folder in the console     7  Upload the PySpark code  spark avg speed py  composer http examples spark avg speed py  into a  spark jobs  folder in GCS     bash gsutil cp   professional services examples cloud composer examples composer http post example spark avg speed py gs    PROJECT spark jobs       8  The DAG folder is essentially a Cloud Storage bucket  Upload the  ephemeral dataproc spark dag py  composer http examples ephemeral dataproc spark dag py  file into the folder      bash gsutil cp   professional services examples cloud composer examples composer http post example ephemeral dataproc spark dag py gs    dag folder  dags                Triggering the workflow  Make sure that you have installed the  iap requirements txt  in the steps above  You will need to create a service account with the necessary permissions to create an IAP request and use Cloud Composer to use the  dag trigger py  script  This can be achieved with the following commands     bash gcloud iam service accounts create dag trigger    Give service account permissions to create tokens for   iap requests  gcloud projects add iam policy binding  PROJECT     member   serviceAccount dag trigger  PROJECT iam gserviceaccount com     role roles iam serviceAccountTokenCreator  gcloud projects add iam policy binding  PROJECT     member   serviceAccount dag trigger  PROJECT iam gserviceaccount com     role roles iam serviceAccountActor    Service account also needs to be authorized to use Composer  gcloud projects add iam policy binding  PROJECT     member   serviceAccount dag trigger  PROJECT iam gserviceaccount com     role roles composer user    We need a service account key to trigger the dag  gcloud iam service accounts keys create    PROJECT dag trigger key json     iam account dag trigger  PROJECT iam gserviceaccount com    Finally use this as your application credentials by setting the environment variable on the machine you will run  dag trigger py  export GOOGLE APPLICATION CREDENTIALS    PROJECT dag trigger key json     To trigger this workflow use  dag trigger py  dag trigger py  takes 3 arguments as shown below    bash python dag trigger py     url  airflow endpoint url      iapClientId  client id      raw path  path to raw files for enhancement in GCS      The endpoint of triggering the dag had the following structure  https    airflow web server url  api experimental dags  dag id  dag runs  in this case our dag id is average speed  The airflow webserver can be found once your composer environment is set up by clicking on your environment in the console and checking here     Alt text     img airflow ui png  Screen Shot showing how to get the airflow URL    In oder to obtain your    iapClientId  Visit the Airflow URL https   YOUR UNIQUE ID appspot com  which you noted in the last step  in an incognito window   don t  authenticate or login  and first landing page for IAP Auth has client Id in the url in the address bar  https   accounts google com signin oauth identifier   client id 00000000000 xxxx0x0xx0xx00xxxx0x00xxx0xxxxx apps googleusercontent com   as a6VGEPwFpCL1qIwusi49IQ destination https 3A 2F 2Fh0b798498b93687a6 tp appspot com approval state  ChRKSmd1TVc1VlQzMDB3MHI2UGI4SxIfWXhaRjJLcWdwcndRVUU3MWpGWk5XazFEbUp6N05SWQ E2 88 99AB8iHBUAAAAAWvsaqTGCmRazWx9NqQtnYVOllz0r2x i xsrfsig AHgIfE o0kxXt6N3ch1JH4Fb19CB7wdbMg flowName GeneralOAuthFlow


GCP This repo contains an example Cloud Composer workflow that triggers Cloud Dataflow to transform enrich and load a delimited text file into Cloud BigQuery Workflow Overview Cloud Composer workflow using Cloud Dataflow A Cloud Function with a Cloud Storage trigger is used to initiate the workflow when a file is uploaded for processing The goal of this example is to provide a common pattern to automatically trigger via Google Cloud Function a Dataflow job when a file arrives in Google Cloud Storage process the data and load it into BigQuery

Answer:
## Cloud Composer workflow using Cloud Dataflow
##### This repo contains an example Cloud Composer workflow that triggers Cloud Dataflow to transform, enrich and load a delimited text file into Cloud BigQuery.
The goal of this example is to provide a common pattern to automatically trigger, via Google Cloud Function, a Dataflow job when a file arrives in Google Cloud Storage, process the data and load it into BigQuery.

### Workflow Overview

***

![Alt text](../img/workflow-overview.png "Workflow Overview")
A Cloud Function with a Cloud Storage trigger is used to initiate the workflow when a file is uploaded for processing.

At a high-level the Cloud Composer workflow performs the following steps:
1. Extracts the location of the input file that triggered the workflow.
2. Executes a Cloud Dataflow job that performs the following:
    - Parses the delimited input file and adds some useful 'metadata'
        - 'filename': The name of the file that is proceeded by the Cloud Dataflow job
        - 'load_dt': The date in YYYY-MM-DD format when the file is processed
    - Loads the data into an existing Cloud BigQuery table (any existing data is truncated)
3. Moves the input file to a Cloud Storage bucket that is setup for storing processed files.

##### 1. Extract the input file location:
When a file is uploaded to the Cloud Storage bucket, a Cloud Function is triggered. This invocation wraps the event information (_bucket and object details_) that triggered this event and passes it to the the Cloud Composer workflow that gets triggered. The workflow extracts this information and passes it to the Cloud Dataflow job.
```
job_args = {
        'input': 'gs:///',
        'output': models.Variable.get('bq_output_table'),
        'fields': models.Variable.get('input_field_names'),
        'load_dt': ds_tag
    }
```

##### 2. Executes the Cloud Dataflow job

The workflow then executes a [Cloud Dataflow job](dataflow/process_delimited.py) to process the delimited file, adds filename and load_dt fields and loads the data into a Cloud BigQuery table.

##### 3. Move to processed bucket

![Alt text](../img/sample-dag.png "DAG Overview")

Based on the status of the Cloud Dataflow job, the workflow will then move the processed files to a Cloud Storage bucket setup to store processed data. A separate folder is created along with a processed date field to hold the files in this bucket.

##### Full code examples

Ready to dive deeper? Check out the complete code [here](simple_load_dag.py)

***

#### Setup and Pre-requisites
It is recommended that virtualenv be used to keep everything tidy. The [requirements.txt](requirements.txt) describes the dependencies needed for the code used in this repo.

The following high-level steps describe the setup needed to run this example:

1. Create a Cloud Storage (GCS) bucket for receiving input files (*input-gcs-bucket*).
2. Create a GCS bucket for storing processed files (*output-gcs-bucket*).
3. Create a Cloud Composer environment - Follow [these](https://cloud.google.com/composer/docs/quickstart) steps to create a Cloud Composer environment if needed (*cloud-composer-env*).
4. Create a Cloud BigQuery table for the processed output. The following schema is used for this example:

|Column Name | Column Type|
|:-----------|:-----------|
|state	     |STRING      |
|gender	     |STRING      |
|year	     |STRING      |
|name	     |STRING      |
|number	     |STRING      |
|created_date|STRING      |
|filename	 |STRING      |
|load_dt	 |DATE        |

5. Set the following [Airflow variables](https://airflow.apache.org/docs/stable/concepts.html#variables) needed for this example:

| Key                   | Value                                           |Example                                   |
| :--------------------- |:---------------------------------------------- |:---------------------------              |
| gcp_project           | *your-gcp-project-id*                           |cloud-comp-df-demo                        |
| gcp_temp_location     | *gcs-bucket-for-dataflow-temp-files*            |gs://my-comp-df-demo-temp/tmp             |
| gcs_completion_bucket | *output-gcs-bucket*                             |my-comp-df-demp-output                    |
| input_field_names     | *comma-separated-field-names-for-delimited-file*|state,gender,year,name,number,created_date|
| bq_output_table       | *bigquery-output-table*                         |my_dataset.usa_names                      |
| email                 | *some-email@mycompany.com*                      |some-email@mycompany.com                  |

 The variables can be set as follows:

 `gcloud composer environments run` **_cloud-composer-env-name_** `variables -- --set` **_key val_**

6. Browse to the Cloud Composer widget in Cloud Console and click on the DAG folder icon as shown below:
![Alt text](../img/dag-folder-example.png "Workflow Overview")

7. The DAG folder is essentially a Cloud Storage bucket. Upload the [simple_load_dag.py](simple_load_dag.py) file into the folder:
![Alt text](../img/bucket-example.png "DAG Bucket")
8. Upload the Python Dataflow code [process_delimited.py](dataflow/process_delimited.py) into a *dataflow* folder created in the base DAG folder.
9. Finally follow [these](https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf) instructions to create a Cloud Function.
    - Ensure that the **DAG_NAME** property is set to _**GcsToBigQueryTriggered**_ i.e. The DAG name defined in [simple_load_dag.py](simple_load_dag.py).

***

##### Triggering the workflow

The workflow is automatically triggered by Cloud Function that gets invoked when a new file is uploaded into the *input-gcs-bucket*
For this example workflow, the [usa_names.csv](resources/usa_names.csv) file can be uploaded into the  *input-gcs-bucket*

`gsutil cp resources/usa_names.csv gs://` **_input-gcs-bucket_**

***

Cloud Composer workflow using Cloud Dataflow       This repo contains an example Cloud Composer workflow that triggers Cloud Dataflow to transform  enrich and load a delimited text file into Cloud BigQuery  The goal of this example is to provide a common pattern to automatically trigger  via Google Cloud Function  a Dataflow job when a file arrives in Google Cloud Storage  process the data and load it into BigQuery       Workflow Overview         Alt text     img workflow overview png  Workflow Overview   A Cloud Function with a Cloud Storage trigger is used to initiate the workflow when a file is uploaded for processing   At a high level the Cloud Composer workflow performs the following steps  1  Extracts the location of the input file that triggered the workflow  2  Executes a Cloud Dataflow job that performs the following        Parses the delimited input file and adds some useful  metadata             filename   The name of the file that is proceeded by the Cloud Dataflow job            load dt   The date in YYYY MM DD format when the file is processed       Loads the data into an existing Cloud BigQuery table  any existing data is truncated  3  Moves the input file to a Cloud Storage bucket that is setup for storing processed files         1  Extract the input file location  When a file is uploaded to the Cloud Storage bucket  a Cloud Function is triggered  This invocation wraps the event information   bucket and object details   that triggered this event and passes it to the the Cloud Composer workflow that gets triggered  The workflow extracts this information and passes it to the Cloud Dataflow job      job args              input    gs                output   models Variable get  bq output table             fields   models Variable get  input field names             load dt   ds tag                  2  Executes the Cloud Dataflow job  The workflow then executes a  Cloud Dataflow job  dataflow process delimited py  to process the delimited file  adds filename and load dt fields and loads the data into a Cloud BigQuery table         3  Move to processed bucket     Alt text     img sample dag png  DAG Overview    Based on the status of the Cloud Dataflow job  the workflow will then move the processed files to a Cloud Storage bucket setup to store processed data  A separate folder is created along with a processed date field to hold the files in this bucket         Full code examples  Ready to dive deeper  Check out the complete code  here  simple load dag py             Setup and Pre requisites It is recommended that virtualenv be used to keep everything tidy  The  requirements txt  requirements txt  describes the dependencies needed for the code used in this repo   The following high level steps describe the setup needed to run this example   1  Create a Cloud Storage  GCS  bucket for receiving input files   input gcs bucket    2  Create a GCS bucket for storing processed files   output gcs bucket    3  Create a Cloud Composer environment   Follow  these  https   cloud google com composer docs quickstart  steps to create a Cloud Composer environment if needed   cloud composer env    4  Create a Cloud BigQuery table for the processed output  The following schema is used for this example    Column Name   Column Type                               state       STRING         gender       STRING         year       STRING         name       STRING         number       STRING         created date STRING         filename   STRING         load dt   DATE           5  Set the following  Airflow variables  https   airflow apache org docs stable concepts html variables  needed for this example     Key                     Value                                            Example                                                                                gcp project              your gcp project id                             cloud comp df demo                            gcp temp location        gcs bucket for dataflow temp files              gs   my comp df demo temp tmp                 gcs completion bucket    output gcs bucket                               my comp df demp output                        input field names        comma separated field names for delimited file  state gender year name number created date    bq output table          bigquery output table                           my dataset usa names                          email                    some email mycompany com                        some email mycompany com                      The variables can be set as follows     gcloud composer environments run     cloud composer env name     variables      set     key val     6  Browse to the Cloud Composer widget in Cloud Console and click on the DAG folder icon as shown below    Alt text     img dag folder example png  Workflow Overview    7  The DAG folder is essentially a Cloud Storage bucket  Upload the  simple load dag py  simple load dag py  file into the folder    Alt text     img bucket example png  DAG Bucket   8  Upload the Python Dataflow code  process delimited py  dataflow process delimited py  into a  dataflow  folder created in the base DAG folder  9  Finally follow  these  https   cloud google com composer docs how to using triggering with gcf  instructions to create a Cloud Function        Ensure that the   DAG NAME   property is set to    GcsToBigQueryTriggered    i e  The DAG name defined in  simple load dag py  simple load dag py               Triggering the workflow  The workflow is automatically triggered by Cloud Function that gets invoked when a new file is uploaded into the  input gcs bucket  For this example workflow  the  usa names csv  resources usa names csv  file can be uploaded into the   input gcs bucket    gsutil cp resources usa names csv gs        input gcs bucket


GCP bigquery analyze realtime reddit data 3 1 5 4 2 Table Of Contents

Answer:
# bigquery-analyze-realtime-reddit-data

## Table Of Contents

1. [Use Case](#use-case)
2. [About](#about)
3. [Architecture](#architecture)
4. [Guide](#guide)
5. [Sample](#sample)

----

## Use-case

Simple deployment of a ([reddit](https://www.reddit.com)) social media data collection architecture on Google Cloud Platform.

----

## About

This repository contains the resources necessary to deploy a basic data stream and data lake on Google Cloud Platform.  Terraform templates deploy the entire infrastructure, which includes a [Google Compute Engine](https://cloud.google.com/compute) VM with a initialization script that clones a [reddit streaming application repository](https://github.com/CYarros10/reddit-streaming-application). The GCE VM executes a python script from that repo.  The python script accesses a user's [reddit developer client](https://www.reddit.com/prefs/apps/) and begins to collect reddit comments from a specified list of the top 50 subreddits.

As the GCE VM collects reddit comments, it cleans, censors, and analyzes sentiment of each comment. Finally, it pushes the comment to a [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) topic.  A [Cloud Dataflow](https://cloud.google.com/dataflow) job subscribes to the PubSub topic, reads the comments, and writes them to a [Cloud Bigquery](https://cloud.google.com/bigquery) table.

The user now has access to an ever-increasing dataset of reddit comments + sentiment analysis.

----

## Architecture

![Stack-Resources](images/architecture.png)

----

## Guide

### 1. Create your reddit bot account

1. [Register a reddit account](https://www.reddit.com/register/)

2. Follow prompts to create new reddit account:
    * Provide email address
    * Choose username and password
    * Click `Finish`

3. Once your account is created, go to [reddit developer console.](https://www.reddit.com/prefs/apps/)

4. Select **“are you a developer? Create an app...”**

5. Give it a name.

6. Select script.  <--- **This is important!**

7. For about url and redirect uri, use http://127.0.0.1

8. You will now get a client_id (underneath web app) and secret

9. Keep track of your reddit account username, password, app client_id (in blue box), and app secret (in red box). These will be used in tutorial Step 11

#### Further Learning / References: PRAW

* [PRAW Quick start](https://praw.readthedocs.io/en/latest/getting_started/quick_start.html)

### 2. Run setup.sh

If you need to allow externalIPs, run this command (or similar) in your project:

```bash
echo "{
  \"constraint\": \"constraints/compute.vmExternalIpAccess\",
	\"listPolicy\": {
	    \"allValues\": \"ALLOW\"
	  }
}" > external_ip_policy.json

gcloud resource-manager org-policies set-policy external_ip_policy.json --project="$projectId"
```

```bash
./scripts/setup.sh -i <project-id> -r <region> -c <reddit-client-id> -u <reddit-user>
```

### 4. Wait for data collection

The VM will take a minute or two to setup. Then comments will start to flow into Bigquery in near-realtime!

### 5. Query your data using Bigquery

**example:**

```sql
    select subreddit, author, comment_text, sentiment_score
    from reddit.comments_raw
    order by sentiment_score desc
    limit 25;
```

----

## Sample

### Example of a Collected+Analyzed reddit Comment:

```json
{
    "comment_id": "fx3wgci",
    "subreddit": "Fitness",
    "author": "silverbird666",
    "comment_text": "well, i dont exactly count my calories, but i run on a competitive base and do kickboxing, that stuff burns quite much calories. i just stick to my established diet, and supplement with protein bars and shakes whenever i fail to hit my daily intake of protein. works for me.",
    "distinguished": null,
    "submitter": false,
    "total_words": 50,
    "reading_ease_score": 71.44,
    "reading_ease": "standard",
    "reading_grade_level": "7th and 8th grade",
    "sentiment_score": -0.17,
    "censored": 0,
    "positive": 0,
    "neutral": 1,
    "negative": 0,
    "subjectivity_score": 0.35,
    "subjective": 0,
    "url": "https://reddit.com/r/Fitness/comments/hlk84h/victory_sunday/fx3wgci/",
    "comment_date": "2020-07-06 15:41:15",
    "comment_timestamp": "2020/07/06 15:41:15",
    "comment_hour": 15,
    "comment_year": 2020,
    "comment_month": 7,
    "comment_day": 6
}
```

bigquery analyze realtime reddit data     Table Of Contents  1   Use Case   use case  2   About   about  3   Architecture   architecture  4   Guide   guide  5   Sample   sample            Use case  Simple deployment of a   reddit  https   www reddit com   social media data collection architecture on Google Cloud Platform            About  This repository contains the resources necessary to deploy a basic data stream and data lake on Google Cloud Platform   Terraform templates deploy the entire infrastructure  which includes a  Google Compute Engine  https   cloud google com compute  VM with a initialization script that clones a  reddit streaming application repository  https   github com CYarros10 reddit streaming application   The GCE VM executes a python script from that repo   The python script accesses a user s  reddit developer client  https   www reddit com prefs apps   and begins to collect reddit comments from a specified list of the top 50 subreddits    As the GCE VM collects reddit comments  it cleans  censors  and analyzes sentiment of each comment  Finally  it pushes the comment to a  Google Cloud Pub Sub  https   cloud google com pubsub  topic   A  Cloud Dataflow  https   cloud google com dataflow  job subscribes to the PubSub topic  reads the comments  and writes them to a  Cloud Bigquery  https   cloud google com bigquery  table   The user now has access to an ever increasing dataset of reddit comments   sentiment analysis            Architecture    Stack Resources  images architecture png            Guide      1  Create your reddit bot account  1   Register a reddit account  https   www reddit com register    2  Follow prompts to create new reddit account        Provide email address       Choose username and password       Click  Finish   3  Once your account is created  go to  reddit developer console   https   www reddit com prefs apps    4  Select    are you a developer  Create an app        5  Give it a name   6  Select script          This is important     7  For about url and redirect uri  use http   127 0 0 1  8  You will now get a client id  underneath web app  and secret  9  Keep track of your reddit account username  password  app client id  in blue box   and app secret  in red box   These will be used in tutorial Step 11       Further Learning   References  PRAW     PRAW Quick start  https   praw readthedocs io en latest getting started quick start html       2  Run setup sh   If you need to allow externalIPs  run this command  or similar  in your project      bash echo        constraint      constraints compute vmExternalIpAccess       listPolicy             allValues      ALLOW             external ip policy json  gcloud resource manager org policies set policy external ip policy json   project   projectId           bash   scripts setup sh  i  project id   r  region   c  reddit client id   u  reddit user           4  Wait for data collection  The VM will take a minute or two to setup  Then comments will start to flow into Bigquery in near realtime       5  Query your data using Bigquery    example        sql     select subreddit  author  comment text  sentiment score     from reddit comments raw     order by sentiment score desc     limit 25                Sample      Example of a Collected Analyzed reddit Comment      json        comment id    fx3wgci        subreddit    Fitness        author    silverbird666        comment text    well  i dont exactly count my calories  but i run on a competitive base and do kickboxing  that stuff burns quite much calories  i just stick to my established diet  and supplement with protein bars and shakes whenever i fail to hit my daily intake of protein  works for me         distinguished   null       submitter   false       total words   50       reading ease score   71 44       reading ease    standard        reading grade level    7th and 8th grade        sentiment score    0 17       censored   0       positive   0       neutral   1       negative   0       subjectivity score   0 35       subjective   0       url    https   reddit com r Fitness comments hlk84h victory sunday fx3wgci         comment date    2020 07 06 15 41 15        comment timestamp    2020 07 06 15 41 15        comment hour   15       comment year   2020       comment month   7       comment day   6


GCP This example provides guidance on how to use to batch records that are published to a Pub Sub topic Using correctly allows us to better utilize the available resources cpu memory network bandwidth on the client machine and to improve throughput In order to demonstrate this process end to end we also provided a simple Dataflow pipeline that reads these Avro messages from a Pub Sub topic and streams those records into a BigQuery table Using Batching in Cloud Pub Sub Java client API In addition to BatchingSettings this code sample also demonstrates the use of to encode the messages instead of using JSON strings Components

Answer:
# Using Batching in Cloud Pub/Sub Java client API.

This example provides guidance on how to use [Pub/Sub's Java client API](https://cloud.google.com/pubsub/docs/reference/libraries) to batch records that are published to a Pub/Sub topic.
Using [BatchingSettings](http://googleapis.github.io/gax-java/1.4.1/apidocs/com/google/api/gax/batching/BatchingSettings.html) correctly allows us to better utilize the available resources (cpu, memory, network bandwidth) on the client machine and to improve throughput.
In addition to BatchingSettings, this code sample also demonstrates the use of [Avro](http://avro.apache.org/docs/current/) to encode the messages instead of using JSON strings.

In order to demonstrate this process end-to-end, we also provided a simple Dataflow pipeline that reads these Avro messages from a Pub/Sub topic and streams those records into a BigQuery table.

## Components

[ObjectPublisher](src/main/java/com/google/cloud/pso/pubsub/common/ObjectPublisher.java) - A generic publisher class that can be used to publish any object as a payload to Cloud Pub/Sub.
This publisher class provides various configurable parameters for controlling the [BatchingSettings](http://googleapis.github.io/gax-java/1.4.1/apidocs/com/google/api/gax/batching/BatchingSettings.html) for the publishing client.

[EmployeePublisher](src/main/java/com/google/cloud/pso/pubsub/EmployeePublisherMain.java) -
An implementation of the [ObjectPublisher](src/main/java/com/google/cloud/pso/pubsub/common/ObjectPublisher.java) to publish Avro encoded test records to Cloud Pub/Sub.
This will generate random test records using the sample [Employee Avro Schema](src/main/avro/employee.avsc).

[IngestionMain](src/main/java/com/google/cloud/pso/pipeline/IngestionMain.java) - A sample Dataflow pipeline to read the Avro encoded test records and stream them into BigQuery.

### Requirements

* Java 8
* Maven 3
* Cloud Pub/Sub topic and subscription
    * The Pub/Sub topic will be used by the client to publish messages.
    * The Pub/Sub subscription on the same topic will be used by the Dataflow job to read the messages.
* BigQuery table to stream records into - The table schema should match the Avro schema of the messages.

### Building the Project

Build the entire project using the maven compile command.
```sh
mvn clean compile
```

### Running unit tests
Run the unit tests using the maven test command.
```sh
mvn clean test
```

### Publishing sample records to Cloud Pub/Sub
Publish sample [Employee](src/main/avro/employee.avsc) records using the maven exec command.
```sh
mvn compile exec:java \
  -Dexec.mainClass=com.google.cloud.pso.pubsub.EmployeePublisherMain \
  -Dexec.cleanupDaemonThreads=false \
  -Dexec.args=" \
  --topic <output-pubsub-topic> --numberOfMessages <number-of-messages>"
```

There are several other optional parameters related to batch settings. These parameters can be viewed by passing the help flag.
```sh
mvn compile exec:java \
  -Dexec.mainClass=com.google.cloud.pso.pubsub.EmployeePublisherMain \
  -Dexec.cleanupDaemonThreads=false \
  -Dexec.args="--help"

Usage: com.google.cloud.pso.pubsub.EmployeePublisherMain [options]
  * - Required parameters
  Options:
    --bytesThreshold, -b
      Batch threshold bytes.
      Default: 1024
    --delayThreshold, -d
      Delay threshold in milliseconds.
      Default: PT0.5S
    --elementCount, -c
      The number of elements to be batched in each request.
      Default: 500
    --help, -h

    --numberOfMessages, -n
      Number of sample messages to publish to Pub/Sub
      Default: 100000
  * --topic, -t
      The Pub/Sub topic to write messages to
```

### Streaming Cloud Pub/Sub messages into BigQuery using Dataflow
The [IngestionMain](src/main/java/com/google/cloud/pso/pipeline/IngestionMain.java) pipeline provides a simple example of consuming the Avro records that were published into Cloud Pub/Sub and streaming those records into a BigQuery table.
This pipeline example can be compiled and executed using the maven exec command.
```sh
mvn compile exec:java \
  -Dexec.mainClass=com.google.cloud.pso.pipeline.IngestionMain \
  -Dexec.cleanupDaemonThreads=false \
  -Dexec.args=" \
  --project=<my-gcp-project> \
  --stagingLocation=<my-gcs-staging-bucket> \
  --tempLocation=<my-gcs-temp-bucket> \
  --runner=DataflowRunner \
  --autoscalingAlgorithm=THROUGHPUT_BASED \
  --maxNumWorkers=<max-num-workers> \
  --subscription=<my-input-pubsub-subscription> \
  --tableId=<my-output-bigquery-table>"
```

### Authentication
These examples use the [Google client libraries to implicitly determine the credentials used][1]. It is strongly recommended that a Service Account with appropriate permissions be used for accessing the resources in Google Cloud Platform Project.

[1]: https://cloud.google.com/docs/authentication/getting-started

Using Batching in Cloud Pub Sub Java client API   This example provides guidance on how to use  Pub Sub s Java client API  https   cloud google com pubsub docs reference libraries  to batch records that are published to a Pub Sub topic  Using  BatchingSettings  http   googleapis github io gax java 1 4 1 apidocs com google api gax batching BatchingSettings html  correctly allows us to better utilize the available resources  cpu  memory  network bandwidth  on the client machine and to improve throughput  In addition to BatchingSettings  this code sample also demonstrates the use of  Avro  http   avro apache org docs current   to encode the messages instead of using JSON strings   In order to demonstrate this process end to end  we also provided a simple Dataflow pipeline that reads these Avro messages from a Pub Sub topic and streams those records into a BigQuery table      Components   ObjectPublisher  src main java com google cloud pso pubsub common ObjectPublisher java    A generic publisher class that can be used to publish any object as a payload to Cloud Pub Sub  This publisher class provides various configurable parameters for controlling the  BatchingSettings  http   googleapis github io gax java 1 4 1 apidocs com google api gax batching BatchingSettings html  for the publishing client    EmployeePublisher  src main java com google cloud pso pubsub EmployeePublisherMain java    An implementation of the  ObjectPublisher  src main java com google cloud pso pubsub common ObjectPublisher java  to publish Avro encoded test records to Cloud Pub Sub  This will generate random test records using the sample  Employee Avro Schema  src main avro employee avsc     IngestionMain  src main java com google cloud pso pipeline IngestionMain java    A sample Dataflow pipeline to read the Avro encoded test records and stream them into BigQuery       Requirements    Java 8   Maven 3   Cloud Pub Sub topic and subscription       The Pub Sub topic will be used by the client to publish messages        The Pub Sub subscription on the same topic will be used by the Dataflow job to read the messages    BigQuery table to stream records into   The table schema should match the Avro schema of the messages       Building the Project  Build the entire project using the maven compile command     sh mvn clean compile          Running unit tests Run the unit tests using the maven test command     sh mvn clean test          Publishing sample records to Cloud Pub Sub Publish sample  Employee  src main avro employee avsc  records using the maven exec command     sh mvn compile exec java      Dexec mainClass com google cloud pso pubsub EmployeePublisherMain      Dexec cleanupDaemonThreads false      Dexec args         topic  output pubsub topic    numberOfMessages  number of messages        There are several other optional parameters related to batch settings  These parameters can be viewed by passing the help flag     sh mvn compile exec java      Dexec mainClass com google cloud pso pubsub EmployeePublisherMain      Dexec cleanupDaemonThreads false      Dexec args    help   Usage  com google cloud pso pubsub EmployeePublisherMain  options        Required parameters   Options        bytesThreshold   b       Batch threshold bytes        Default  1024       delayThreshold   d       Delay threshold in milliseconds        Default  PT0 5S       elementCount   c       The number of elements to be batched in each request        Default  500       help   h        numberOfMessages   n       Number of sample messages to publish to Pub Sub       Default  100000       topic   t       The Pub Sub topic to write messages to          Streaming Cloud Pub Sub messages into BigQuery using Dataflow The  IngestionMain  src main java com google cloud pso pipeline IngestionMain java  pipeline provides a simple example of consuming the Avro records that were published into Cloud Pub Sub and streaming those records into a BigQuery table  This pipeline example can be compiled and executed using the maven exec command     sh mvn compile exec java      Dexec mainClass com google cloud pso pipeline IngestionMain      Dexec cleanupDaemonThreads false      Dexec args         project  my gcp project        stagingLocation  my gcs staging bucket        tempLocation  my gcs temp bucket        runner DataflowRunner       autoscalingAlgorithm THROUGHPUT BASED       maxNumWorkers  max num workers        subscription  my input pubsub subscription        tableId  my output bigquery table            Authentication These examples use the  Google client libraries to implicitly determine the credentials used  1   It is strongly recommended that a Service Account with appropriate permissions be used for accessing the resources in Google Cloud Platform Project    1   https   cloud google com docs authentication getting started


GCP The Dataproc cluster lifecycle management will be done by the automatically generated Airflow DAGs to reuse or create clusters accordingly cluster proposed configuration includes a scalability policy usage Each DAG will execute a single Dataproc cluster Job referenced in a DAG configuration file and that Job will be executed in a Dataproc Cluster that can be reused for multiple Jobs DAGs Writing DAGs isn t practical when having multiple DAGs that run similar Dataproc Jobs and want to share clusters efficiently with just some parameters changing between them Here makes sense to dynamically generate DAGs Using this project you can deploy multiple Composer Airflow DAGs from a single configuration it means you will create configuration files for DAGs to be automatically generated during deployment you can also use it to generate and upload DAGs manually Dataproc lifecycle management orchestrated by Composer

Answer:
# Dataproc lifecycle management orchestrated by Composer

Writing DAGs isn’t practical when having multiple DAGs that run similar Dataproc Jobs, and want to share clusters efficiently, with just some parameters changing between them. Here makes sense to dynamically generate DAGs.

Using this project you can deploy multiple Composer/Airflow DAGs from a single configuration, it means you will create configuration files for DAGs to be automatically generated during deployment _(you can also use it to generate and upload DAGs manually)_.

Each DAG will execute a single Dataproc cluster Job referenced in a DAG configuration file, and that Job will be executed in a Dataproc Cluster that can be reused for multiple Jobs/DAGs.

The Dataproc cluster lifecycle management will be done by the automatically generated Airflow DAGs, to reuse or create clusters accordingly _(cluster proposed configuration includes a scalability policy usage)_.

This approach aims to use resources efficiently meanwhile minimize provisioning and execution time.

This is the high level diagram:
![](images/dataproc_lifecycle.png)

## Prerequisites
1. This blueprint will deploy all its resources into the project defined by the `project_id` variable. Please note that we assume this project already exists.
2. The user deploying the project _(executing terraform plan/apply)_ should have admin permissions in the selected project, or permission to create all the resources defined in the Terraform scripts.

## Project Folder Structure
 ```bash
main.tf
...
dags/          (Autogenerated on Terraform Plan/Apply from /dag_config/ files)
├── ephemeral_cluster_job_1.py
├── ephemeral_cluster_job_2.py
jobs/
├── hello_world_spark.py
├── ...        (Add your dataproc jobs here)
include/
├── dag_template.py
├── generate_dag_files.py
└── dag_config
    ├── dag1_config.json
    └── dag2_config.json
    └── ...     (Add your Composer/Airflow DAGs configuration here)
...
 ```
## Adding Jobs
#### Prepare Dataproc Jobs to be executed
1. Clone this repository
2. Locate your Dataproc jobs in the **/jobs/** folder in your local environment

#### Prepare Composer Dags to be deployed
3. Locate your DAG configuration files in the **/include/dag_config/** folder in your local environment. DAG configuration files have the following variables:
```json
{
    "DagId": "ephemeral_cluster_job_1",     --DAG name you will see in Airflow environment
    "Schedule": "'@daily'",                 --DAG Schedule
    "ClusterName":"ephemeral-cluster-test", --Dataproc Cluster to be Used/created for this DAG/Job to be executed in
    "StartYear":"2022",                     --DAG start year
    "StartMonth":"9",                       --DAG start month
    "StartDay":"13",                        --DAG start day
    "Catchup":"False",                      --DAG backfill catchup
    "ClusterMachineType":"n1-standard-4",   --Dataproc machine type to be used by master and worker cluster nodes
    "ClusterIdleDeleteTtl":"300",           --Time in seconds to delete unused Dataproc cluster
    "SparkJob":"hello_world_spark.py"       --Spark Job to be executed by DAG, should be placed in /jobs/ folder of this project. (if other type of Dataproc jobs modify dag_template.py)
}
```
4. (Optional) You can run `python3 include/generate_dag_files.py` in your local environment, if you want to review generated DAGs before deploying(TF plan/apply) those.

## Deployment

1. Set Google Cloud Platform credentials on local environment:
https://cloud.google.com/source-repositories/docs/authentication

2. You must supply the `project_id` variable as minimum in order to deploy the project. Default Terraform variables and example values in `varibles.tf` file.

3. Run Terraform Plan/Apply
   ```bash
    $ cd terraform/
    $ terraform init
    $ terraform plan
    $ terraform apply
   ##Optionally variables could be used
    $ terraform apply -var 'project_id=<PROJECT_ID>' \
   -var 'region=<REGION>'
    ```

Once you deploy terraform plan for the first time, and composer environment is running, you can _terraform plan/apply_ after adding new DAG configuration files, to generate and deploy DAGs, to the existing environment.

First time it is deployed, resource creation will take several minutes(up to 40) because of Composer Environment provisioning, you should expect for successful completion along with a list of the created resources.

## Running DAGs
DAGs will run as per **Schedule**, **StartDate**, and **Catchup** configuration in DAG configuration file, or can be triggered manually trough Airflow web console after deployment.
![](images/dag_execution_example.png)

## Shared VPC
The example supports the configuration of a Shared VPC as an input variable.
To deploy the solution on a Shared VPC, you have to configure the `network_config` variable:
```
network_config = {
    host_project       = "PROJECT_ID"
    network_self_link  = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/VPC_NAME"
    subnet_self_link   = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/$REGION/subnetworks/SUBNET_NAME"
    name               = "VPC_NAME"
  }
```
If Shared VPC is used [firewall rules](https://cloud.google.com/composer/docs/composer-2/create-environments) would be needed in the host project.

### TODO
- Add support for CMEK (Composer and dataproc)

Dataproc lifecycle management orchestrated by Composer  Writing DAGs isn t practical when having multiple DAGs that run similar Dataproc Jobs  and want to share clusters efficiently  with just some parameters changing between them  Here makes sense to dynamically generate DAGs   Using this project you can deploy multiple Composer Airflow DAGs from a single configuration  it means you will create configuration files for DAGs to be automatically generated during deployment   you can also use it to generate and upload DAGs manually     Each DAG will execute a single Dataproc cluster Job referenced in a DAG configuration file  and that Job will be executed in a Dataproc Cluster that can be reused for multiple Jobs DAGs    The Dataproc cluster lifecycle management will be done by the automatically generated Airflow DAGs  to reuse or create clusters accordingly   cluster proposed configuration includes a scalability policy usage      This approach aims to use resources efficiently meanwhile minimize provisioning and execution time   This is the high level diagram      images dataproc lifecycle png       Prerequisites 1  This blueprint will deploy all its resources into the project defined by the  project id  variable  Please note that we assume this project already exists  2  The user deploying the project   executing terraform plan apply   should have admin permissions in the selected project  or permission to create all the resources defined in the Terraform scripts       Project Folder Structure     bash main tf     dags            Autogenerated on Terraform Plan Apply from  dag config  files      ephemeral cluster job 1 py     ephemeral cluster job 2 py jobs      hello world spark py                 Add your dataproc jobs here  include      dag template py     generate dag files py     dag config         dag1 config json         dag2 config json                  Add your Composer Airflow DAGs configuration here              Adding Jobs      Prepare Dataproc Jobs to be executed 1  Clone this repository 2  Locate your Dataproc jobs in the    jobs    folder in your local environment       Prepare Composer Dags to be deployed 3  Locate your DAG configuration files in the    include dag config    folder in your local environment  DAG configuration files have the following variables     json        DagId    ephemeral cluster job 1         DAG name you will see in Airflow environment      Schedule      daily                      DAG Schedule      ClusterName   ephemeral cluster test     Dataproc Cluster to be Used created for this DAG Job to be executed in      StartYear   2022                         DAG start year      StartMonth   9                           DAG start month      StartDay   13                            DAG start day      Catchup   False                          DAG backfill catchup      ClusterMachineType   n1 standard 4       Dataproc machine type to be used by master and worker cluster nodes      ClusterIdleDeleteTtl   300               Time in seconds to delete unused Dataproc cluster      SparkJob   hello world spark py          Spark Job to be executed by DAG  should be placed in  jobs  folder of this project   if other type of Dataproc jobs modify dag template py        4   Optional  You can run  python3 include generate dag files py  in your local environment  if you want to review generated DAGs before deploying TF plan apply  those      Deployment  1  Set Google Cloud Platform credentials on local environment   https   cloud google com source repositories docs authentication   2  You must supply the  project id  variable as minimum in order to deploy the project  Default Terraform variables and example values in  varibles tf  file   3  Run Terraform Plan Apply       bash       cd terraform        terraform init       terraform plan       terraform apply      Optionally variables could be used       terraform apply  var  project id  PROJECT ID         var  region  REGION            Once you deploy terraform plan for the first time  and composer environment is running  you can  terraform plan apply  after adding new DAG configuration files  to generate and deploy DAGs  to the existing environment    First time it is deployed  resource creation will take several minutes up to 40  because of Composer Environment provisioning  you should expect for successful completion along with a list of the created resources      Running DAGs DAGs will run as per   Schedule      StartDate    and   Catchup   configuration in DAG configuration file  or can be triggered manually trough Airflow web console after deployment       images dag execution example png      Shared VPC The example supports the configuration of a Shared VPC as an input variable   To deploy the solution on a Shared VPC  you have to configure the  network config  variable      network config         host project          PROJECT ID      network self link     https   www googleapis com compute v1 projects PROJECT ID global networks VPC NAME      subnet self link      https   www googleapis com compute v1 projects PROJECT ID regions  REGION subnetworks SUBNET NAME      name                  VPC NAME          If Shared VPC is used  firewall rules  https   cloud google com composer docs composer 2 create environments  would be needed in the host project       TODO   Add support for CMEK  Composer and dataproc