GCP Scikit learn pipeline trainer for AI Platform This is a example for building a scikit learn based machine learning pipeline trainer 3 Estimator exact machine learning model e g RandomForestClassifier 2 Pre processer handle typical standard pre processing e g scaling imputation one hot encoding and etc to serve online traffic The entire pipeline includes three major components The pipeline can be trained locally or remotely on AI platform The trained model can be further deployed on AI platform that can be run on AI Platform which is built on top of the 1 Transformer generate new features from rawdata with user defined logic function

Answer:
# Scikit-learn pipeline trainer for AI Platform

This is a example for building a scikit-learn-based machine learning pipeline trainer
that can be run on AI Platform, which is built on top of the [scikit-learn template](https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/sklearn/sklearn-template/template).
The pipeline can be trained locally or remotely on AI platform. The trained model can be further deployed on AI platform
to serve online traffic. The entire pipeline includes three major components:

1. Transformer: generate new features from raw_data with user defined logic (function).
2. Pre-processer: handle typical standard pre-processing e.g. scaling, imputation, one-hot-encoding and etc.
3. Estimator: exact machine learning model e.g. RandomForestClassifier.

Compared with the [scikit-learn template](https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/sklearn/sklearn-template/template),
this example has the following additional feature:

1. Support both Classification and Regression, which can be specified in the configuration
2. Support serving for both JSON and List of Value formats
3. Support additional custom transformation logics besides typical pre-processing provided by scikit-learn

Google Cloud tools used:
- [Google Cloud Platform](https://cloud.google.com/) (GCP) lets you build and
host applications and websites, store data, and analyze data on Google's
scalable infrastructure.
- [Cloud ML Engine](https://cloud.google.com/ml-engine/) is a managed service
that enables you to easily build machine learning models that work on any type
of data, of any size. This is now part of
[AI Platform](https://cloud.google.com/ai-platform/).
- [Google Cloud Storage](https://cloud.google.com/storage/) (GCS) is a unified
object storage for developers and enterprises, from live data serving to data
analytics/ML to data archiving.
- [Cloud SDK](https://cloud.google.com/sdk/) is a set of tools for Google Cloud
Platform, which contains e.g. gcloud, gsutil, and bq command-line tools to
interact with Google Cloud products and services.
- [Google BigQuery](https://cloud.google.com/bigquery/) A fast, highly scalable,
cost-effective, and fully managed cloud data warehouse for analytics, with even
built-in machine learning.

## Pipeline overview
The overall flow of the pipeline can be summarized as follows and illustrated in the flowchart:

**Raw Data -> Transformer -> Pre-processor -> Estimator -> Trained Pipeline**

![Flowchart](./sklearn.png)

## Repository structure
```
template
    |__ config
        |__ config.yaml             # for running normal training job on AI Platform
        |__ hptuning_config.yaml    # for running hyperparameter tunning job on AI Platform
    |__ scripts
        |__ train.sh                # convenience script for running machine learning training jobs
        |__ deploy.sh               # convenience script for deploying trained scikit-learn model
        |__ predict.sh              # convenience script for requesting online prediction
        |__ predict.py              # helper function for requesting online prediction using python
    |__ trainer                     # trainer package
        |__ util                    # utility functions
            |__ utils.py            # utility functions including e.g. loading data from bigquery and cloud storage
            |__ preprocess_utils.py # utility functions for constructing preprocessing pipeline
            |__ transform_utils.py  # utility functions for constructing transform pipeline
        |__ metadata.py             # dataset metadata and feature columns definitions
        |__ constants.py            # constants used in the project
        |__ model.py                # pre-processing and machine learning model pipeline definition
        |__ task.py                 # training job entry point, handling the parameters passed from command line
        |__ transform_config.py     # configuration for transform pipeline construction"
    |__ predictor.py                # define custom prediction behavior
    |__ setup.py                    # specify necessary dependency for running job on AI Platform
    |__ requirements.txt            # specify necessary dependency, helper for setup environment for local development
```

## Using the template
### Step 0. Prerequisites
Before you follow the instructions below to adapt the template to your machine learning job,
you need a Google cloud project if you don't have one. You can find detailed instructions
[here](https://cloud.google.com/dataproc/docs/guides/setup-project).

- Make sure the following API & Services are enabled.
    * Cloud Storage
    * Cloud Machine Learning Engine
    * BigQuery API
    * Cloud Build API (for CI/CD integration)
    * Cloud Source Repositories API (for CI/CD integration)

- Configure project id and bucket id as environment variable.
  ```bash
  $ export PROJECT_ID=[your-google-project-id]
  $ export BUCKET_ID=[your-google-cloud-storage-bucket-name]
  ```

- Set up a service account for calls to GCP APIs.
  More information on setting up a service account can be found
  [here](https://cloud.google.com/docs/authentication/getting-started).

### Step 1. Tailor the scikit-learn trainer to your data
`metadata.py` is where the dataset's metadata is defined.
By default, the file is configured to train on the Census dataset, which can be found at
[`bigquery-public-data.ml_datasets.census_adult_income`](https://bigquery.cloud.google.com/table/bigquery-public-data:ml_datasets.census_adult_income).

```python
# Usage: Modify below based on the dataset used.
CSV_COLUMNS = None  # Schema of the data. Necessary for data stored in GCS

# In the following, I provided an example based on census dataset.
NUMERIC_FEATURES = [
    'age',
    'hours_per_week',
]

CATEGORICAL_FEATURES = [
    'workclass',
    'education',
    'marital_status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'native_country'
]

FEATURE_COLUMNS = NUMERIC_FEATURES + CATEGORICAL_FEATURES

LABEL = 'income_bracket'
PROBLEM_TYPE = 'classification'  # 'regression' or 'classification'
```

In most cases, only the following items need to be modified, in order to adapt to the target dataset.
- **COLUMNS**: the schema of ths data, only required for data stored in GCS
- **NUMERIC_FEATURES**: columns those will be treated as numerical features
- **CATEGORICAL_FEATURES**: columns those will be treated as categorical features
- **LABEL**: column that will be treated as label

### Step 2. Add new features with domain knowledge
`transform_config.py` is where the logic of generating new features out of raw dataset is defined.
There are two parts need to be provided for each new feature generating logic:

* User defined function that handle the generation of new feature. There would be four cases in terms of the
combinations of the dimensions of input and output as listed below:
    * ()->(): scalar to scalar
    * (n) -> (): multi-inputs to scalar
    * () -> (n): scalar to multi-outputs
    * (n) -> (n): multi-inputs to multi-outputs

The example below takes in `age` and converts it into age bucket, which is an example of "scalar to scalar" function.

```python
def _age_class(age):
  """Example scalar processing function

  Args:
    age: (int), age in integer

  Returns:

  """
  if age < 10:
    return 1
  elif 10 <= age < 18:
    return 2
  elif 18 <= age < 30:
    return 3
  elif 30 <= age < 50:
    return 4
  else:
    return 5
```

* An entry in `TRANSFORM_CONFIG`. After the user defined function is done, to incorporate the transformation into the
entire pipeline, an additional entry need to be added to `TRANSFORM_CONFIG` with
    * input_columns: name of columns needed for as inputs to the transform function
    * process_functions: transform function
    * output_columns: names assigned to the output columns, data type indicator (N: for numerical, C: for categorical)

The example below is the counter part of the user defined function in previous section.

```python
# this is an example for generating new categorical feature using single
# column from the raw data
{
    'input_columns': ['age'],
    'process_function': _age_class,
    'output_columns': [('age_class', constants.CATEGORICAL_INDICATOR)]
},
```

For more examples, please refer to the [Appendix](#appendix).

### Step 3. Modify YAML config files for training on AI Platform
The files are located in `config`:
- `config.yaml`: for running normal training job on AI Platform.
- `hptuning_config.yaml`: for running hyperparameter tuning job on AI Platform.

The YAML files share some configuration parameters. In particular, `runtimeVersion` and `pythonVersion` should
correspond in both files. Note that both Python 2.7 and Python 3.5 are supported, but Python 3.5 is the recommended
one since Python 2.7 is [deprecated](https://pythonclock.org/) soon.

```yaml
trainingInput:
  scaleTier: STANDARD_1   # Machine type
  runtimeVersion: "1.13"  # Scikit-learn version
  # Note that both Python 2.7 and Python 3.5 are supported, but Python 3.5 is the
  # recommended one since 2.7 is deprecated soon
  pythonVersion: "3.5"
```

More information on supported runtime version can be found
[here](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list).

### Step 4. Submit scikit-learn training job

You can run ML training jobs through the `train.sh` Bash script.

```shell
bash scripts/train.sh [INPUT] [RUN_ENV] [RUN_TYPE] [EXTRA_TRAINER_ARGS]
```
- INPUT: Dataset to use for training and evaluation, which can be BigQuery table or a file (CSV).
         BigQuery table should be specified as `PROJECT_ID.DATASET.TABLE_NAME`.
- RUN_ENV: (Optional), whether to run `local` (on-prem) or `remote` (GCP). Default value is `local`.
- RUN_TYPE: (Optional), whether to run `train` or `hptuning`. Default value is `train`.
- EXTRA_TRAINER_ARGS: (Optional), additional arguments to pass to the trainer.

**Note**: Please make sure the REGION is set to a supported Cloud region for your project in `train.sh`
```shell
REGION=us-central1
```

### Step 5. Deploy the trained model

The trained model can then be deployed to AI Platform for online serving using the `deploy.sh` script.

```shell
bash scripts/deploy.sh [MODEL_NAME] [VERSION_NAME] [MODEL_DIR]
```

where:

- MODEL_NAME: Name of the model to be deployed.
- VERSION_NAME: Version of the model to be deployed.
- MODEL_DIR: Path to directory containing trained and exported scikit-learn model.

**Note**: Please make sure the following parameters are properly set in deploy.sh
```shell
REGION=us-central1

# The following two parameters should be aligned with those used during
# training job, i.e., specified in the yaml files under config/
RUN_TIME=1.13
# Note that both Python 2.7 and Python 3.5 are supported,
# but Python 3.5 is the recommended one since 2.7 is deprecated soon
PYTHON_VERSION=3.5
```

### Step 6. Run predictions using the deployed model

After the model is successfully deployed, you can send small samples of new data to the API associated with the model,
and it would return predictions in the response.
There are two helper scripts available, `predict.sh` and `predict.py`, which use gcloud and Python API for
requesting predictions respectively.

```shell
bash scripts/predict.sh [INPUT_DATA_FILE] [MODEL_NAME] [VERSION_NAME]
```

where:

- INPUT_DATA_FILE: Path to sample file contained data in line-delimited JSON format.
  See `sample_data/sample_list.txt` or `sample_data/sample_json.txt` for an example. More information can be found
  [here](https://cloud.google.com/ml-engine/docs/scikit/online-predict#formatting_instances_as_lists).
- MODEL_NAME: Name of the deployed model to use.
- VERSION_NAME: Version of the deployed model to use.

Note that two data formats are supported for online prediction:
* List of values:
```python
[39,34," Private"," 9th"," Married-civ-spouse"," Other-service"," Wife"," Black"," Female"," United-States"]
```
* JSON:
```python
{
    "age": 39,
    "hours_per_week": 34,
    "workclass": " Private",
    "education": " 9th",
    "marital_status": " Married-civ-spouse",
    "occupation": " Other-service",
    "relationship": " Wife",
    "race": " Black",
    "sex": " Female",
    "native_country": " United-States"
  }
```

### Appendix
In this section, I have provided a complete example for iris dataset and demonstrate all four cases of user
defined functions.
```python

import numpy as np

def _numeric_square(num):
  """Example scalar processing function

  Args:
    num: (float)

  Returns:
    float
  """
  return np.power(num, 2)

def _numeric_square_root(num):
  """Example scalar processing function

  Args:
    num: (float)

  Returns:
    float
  """
  return np.sqrt(num)

def _numeric_sq_sr(num):
  """Example function that take scala and return an array

  Args:
    num: (float)

  Returns:
    numpy.array
  """
  return np.array([_numeric_square(num), _numeric_square_root(num)])

def _area(args):
  """Examples function that take an array and return a scalar

  Args:
    args: (numpy.array), args[0] -> length, args[1] -> width

  Returns:
    float
  """
  return args[0] * args[1]

def _area_class(args):
  """Examples function that take an array and return a scalar

  Args:
    args: (numpy.array), args[0] -> length, args[1] -> width

  Returns:
    int
  """
  area = args[0] * args[1]
  cl = 1 if area > 2 else 0
  return cl

def _area_and_class(args):
  """Examples function that take an array and return an array

  Args:
    args: (numpy.array), args[0] -> length, args[1] -> width

  Returns:
    numpy.array
  """
  area = args[0] * args[1]
  cl = 1 if area > 2 else 0
  return np.array([area, cl])

TRANSFORM_CONFIG = [
    # this is an example for pass through features,
    # i.e., those doesn't need any processing
    {
        'input_columns': ['sepal_length', 'sepal_width', 'petal_length',
                          'petal_width'],
        # the raw feature types are defined in the metadata,
        # no need to do it here
        'process_function': None,
        'output_columns': ['sepal_length', 'sepal_width', 'petal_length',
                           'petal_width']
    },
    # this is an example for generating new numerical feature using a single
    # column from the raw data
    {
        'input_columns': ['sepal_length'],
        'process_function': _numeric_square,
        'output_columns': [('sepal_length_sq', constants.NUMERICAL_INDICATOR)]
        # 'N' stands for numerical feature
    },
    # this is an example for generating new numerical feature using a single
    # column from the raw data
    {
        'input_columns': ['sepal_width'],
        'process_function': _numeric_square_root,
        'output_columns': [('sepal_width_sr', constants.NUMERICAL_INDICATOR)]
    },
    # this is an example for generating new numerical feature using multiple
    # columns from the raw data
    {
        'input_columns': ['petal_length', 'petal_width'],
        'process_function': _area,
        'output_columns': [('petal_area', constants.NUMERICAL_INDICATOR)]
    },
    # this is an example for generating new categorical feature using multiple
    # columns from the raw data
    {
        'input_columns': ['petal_length', 'petal_width'],
        'process_function': _area_class,
        'output_columns': [('petal_area_cl', constants.CATEGORICAL_INDICATOR)]
        # 'C' stands for categorical feature
    },
    # this is an example for generating multiple features using a single column
    # from the raw data
    {
        'input_columns': ['petal_length'],
        'process_function': _numeric_sq_sr,
        'output_columns': [('petal_length_sq', constants.NUMERICAL_INDICATOR),
                           ('petal_width_sr', constants.NUMERICAL_INDICATOR)]
    },
    # this is an example for generating multiple features using multiple columns
    # from the raw data
    {
        'input_columns': ['sepal_length', 'sepal_width'],
        'process_function': _area_and_class,
        'output_columns': [('sepal_area', constants.NUMERICAL_INDICATOR),
                           ('sepal_area_cl', constants.CATEGORICAL_INDICATOR)]
    },

]

``

Scikit learn pipeline trainer for AI Platform  This is a example for building a scikit learn based machine learning pipeline trainer that can be run on AI Platform  which is built on top of the  scikit learn template  https   github com GoogleCloudPlatform cloudml samples tree master sklearn sklearn template template   The pipeline can be trained locally or remotely on AI platform  The trained model can be further deployed on AI platform to serve online traffic  The entire pipeline includes three major components   1  Transformer  generate new features from raw data with user defined logic  function   2  Pre processer  handle typical standard pre processing e g  scaling  imputation  one hot encoding and etc  3  Estimator  exact machine learning model e g  RandomForestClassifier   Compared with the  scikit learn template  https   github com GoogleCloudPlatform cloudml samples tree master sklearn sklearn template template   this example has the following additional feature   1  Support both Classification and Regression  which can be specified in the configuration 2  Support serving for both JSON and List of Value formats 3  Support additional custom transformation logics besides typical pre processing provided by scikit learn  Google Cloud tools used     Google Cloud Platform  https   cloud google com    GCP  lets you build and host applications and websites  store data  and analyze data on Google s scalable infrastructure     Cloud ML Engine  https   cloud google com ml engine   is a managed service that enables you to easily build machine learning models that work on any type of data  of any size  This is now part of  AI Platform  https   cloud google com ai platform       Google Cloud Storage  https   cloud google com storage    GCS  is a unified object storage for developers and enterprises  from live data serving to data analytics ML to data archiving     Cloud SDK  https   cloud google com sdk   is a set of tools for Google Cloud Platform  which contains e g  gcloud  gsutil  and bq command line tools to interact with Google Cloud products and services     Google BigQuery  https   cloud google com bigquery   A fast  highly scalable  cost effective  and fully managed cloud data warehouse for analytics  with even built in machine learning      Pipeline overview The overall flow of the pipeline can be summarized as follows and illustrated in the flowchart     Raw Data    Transformer    Pre processor    Estimator    Trained Pipeline      Flowchart    sklearn png      Repository structure     template         config             config yaml               for running normal training job on AI Platform             hptuning config yaml      for running hyperparameter tunning job on AI Platform         scripts             train sh                  convenience script for running machine learning training jobs             deploy sh                 convenience script for deploying trained scikit learn model             predict sh                convenience script for requesting online prediction             predict py                helper function for requesting online prediction using python         trainer                       trainer package             util                      utility functions                 utils py              utility functions including e g  loading data from bigquery and cloud storage                 preprocess utils py   utility functions for constructing preprocessing pipeline                 transform utils py    utility functions for constructing transform pipeline             metadata py               dataset metadata and feature columns definitions             constants py              constants used in the project             model py                  pre processing and machine learning model pipeline definition             task py                   training job entry point  handling the parameters passed from command line             transform config py       configuration for transform pipeline construction          predictor py                  define custom prediction behavior         setup py                      specify necessary dependency for running job on AI Platform         requirements txt              specify necessary dependency  helper for setup environment for local development         Using the template     Step 0  Prerequisites Before you follow the instructions below to adapt the template to your machine learning job  you need a Google cloud project if you don t have one  You can find detailed instructions  here  https   cloud google com dataproc docs guides setup project      Make sure the following API   Services are enabled        Cloud Storage       Cloud Machine Learning Engine       BigQuery API       Cloud Build API  for CI CD integration        Cloud Source Repositories API  for CI CD integration     Configure project id and bucket id as environment variable       bash     export PROJECT ID  your google project id      export BUCKET ID  your google cloud storage bucket name           Set up a service account for calls to GCP APIs    More information on setting up a service account can be found    here  https   cloud google com docs authentication getting started        Step 1  Tailor the scikit learn trainer to your data  metadata py  is where the dataset s metadata is defined  By default  the file is configured to train on the Census dataset  which can be found at   bigquery public data ml datasets census adult income   https   bigquery cloud google com table bigquery public data ml datasets census adult income       python   Usage  Modify below based on the dataset used  CSV COLUMNS   None    Schema of the data  Necessary for data stored in GCS    In the following  I provided an example based on census dataset  NUMERIC FEATURES          age        hours per week      CATEGORICAL FEATURES          workclass        education        marital status        occupation        relationship        race        sex        native country     FEATURE COLUMNS   NUMERIC FEATURES   CATEGORICAL FEATURES  LABEL    income bracket  PROBLEM TYPE    classification      regression  or  classification       In most cases  only the following items need to be modified  in order to adapt to the target dataset      COLUMNS    the schema of ths data  only required for data stored in GCS     NUMERIC FEATURES    columns those will be treated as numerical features     CATEGORICAL FEATURES    columns those will be treated as categorical features     LABEL    column that will be treated as label      Step 2  Add new features with domain knowledge  transform config py  is where the logic of generating new features out of raw dataset is defined  There are two parts need to be provided for each new feature generating logic     User defined function that handle the generation of new feature  There would be four cases in terms of the combinations of the dimensions of input and output as listed below                scalar to scalar        n         multi inputs to scalar              n   scalar to multi outputs        n      n   multi inputs to multi outputs  The example below takes in  age  and converts it into age bucket  which is an example of  scalar to scalar  function      python def  age class age        Example scalar processing function    Args      age   int   age in integer    Returns           if age   10      return 1   elif 10    age   18      return 2   elif 18    age   30      return 3   elif 30    age   50      return 4   else      return 5        An entry in  TRANSFORM CONFIG   After the user defined function is done  to incorporate the transformation into the entire pipeline  an additional entry need to be added to  TRANSFORM CONFIG  with       input columns  name of columns needed for as inputs to the transform function       process functions  transform function       output columns  names assigned to the output columns  data type indicator  N  for numerical  C  for categorical   The example below is the counter part of the user defined function in previous section      python   this is an example for generating new categorical feature using single   column from the raw data        input columns     age         process function    age class       output columns      age class   constants CATEGORICAL INDICATOR           For more examples  please refer to the  Appendix   appendix        Step 3  Modify YAML config files for training on AI Platform The files are located in  config      config yaml   for running normal training job on AI Platform     hptuning config yaml   for running hyperparameter tuning job on AI Platform   The YAML files share some configuration parameters  In particular   runtimeVersion  and  pythonVersion  should correspond in both files  Note that both Python 2 7 and Python 3 5 are supported  but Python 3 5 is the recommended one since Python 2 7 is  deprecated  https   pythonclock org   soon      yaml trainingInput    scaleTier  STANDARD 1     Machine type   runtimeVersion   1 13     Scikit learn version     Note that both Python 2 7 and Python 3 5 are supported  but Python 3 5 is the     recommended one since 2 7 is deprecated soon   pythonVersion   3 5       More information on supported runtime version can be found  here  https   cloud google com ml engine docs tensorflow runtime version list        Step 4  Submit scikit learn training job  You can run ML training jobs through the  train sh  Bash script      shell bash scripts train sh  INPUT   RUN ENV   RUN TYPE   EXTRA TRAINER ARGS        INPUT  Dataset to use for training and evaluation  which can be BigQuery table or a file  CSV            BigQuery table should be specified as  PROJECT ID DATASET TABLE NAME     RUN ENV   Optional   whether to run  local   on prem  or  remote   GCP   Default value is  local     RUN TYPE   Optional   whether to run  train  or  hptuning   Default value is  train     EXTRA TRAINER ARGS   Optional   additional arguments to pass to the trainer     Note    Please make sure the REGION is set to a supported Cloud region for your project in  train sh     shell REGION us central1          Step 5  Deploy the trained model  The trained model can then be deployed to AI Platform for online serving using the  deploy sh  script      shell bash scripts deploy sh  MODEL NAME   VERSION NAME   MODEL DIR       where     MODEL NAME  Name of the model to be deployed    VERSION NAME  Version of the model to be deployed    MODEL DIR  Path to directory containing trained and exported scikit learn model     Note    Please make sure the following parameters are properly set in deploy sh    shell REGION us central1    The following two parameters should be aligned with those used during   training job  i e   specified in the yaml files under config  RUN TIME 1 13   Note that both Python 2 7 and Python 3 5 are supported    but Python 3 5 is the recommended one since 2 7 is deprecated soon PYTHON VERSION 3 5          Step 6  Run predictions using the deployed model  After the model is successfully deployed  you can send small samples of new data to the API associated with the model  and it would return predictions in the response  There are two helper scripts available   predict sh  and  predict py   which use gcloud and Python API for requesting predictions respectively      shell bash scripts predict sh  INPUT DATA FILE   MODEL NAME   VERSION NAME       where     INPUT DATA FILE  Path to sample file contained data in line delimited JSON format    See  sample data sample list txt  or  sample data sample json txt  for an example  More information can be found    here  https   cloud google com ml engine docs scikit online predict formatting instances as lists     MODEL NAME  Name of the deployed model to use    VERSION NAME  Version of the deployed model to use   Note that two data formats are supported for online prediction    List of values     python  39 34   Private    9th    Married civ spouse    Other service    Wife    Black    Female    United States         JSON     python        age   39       hours per week   34       workclass     Private        education     9th        marital status     Married civ spouse        occupation     Other service        relationship     Wife        race     Black        sex     Female        native country     United States               Appendix In this section  I have provided a complete example for iris dataset and demonstrate all four cases of user defined functions     python  import numpy as np  def  numeric square num        Example scalar processing function    Args      num   float     Returns      float         return np power num  2    def  numeric square root num        Example scalar processing function    Args      num   float     Returns      float         return np sqrt num    def  numeric sq sr num        Example function that take scala and return an array    Args      num   float     Returns      numpy array         return np array   numeric square num    numeric square root num      def  area args        Examples function that take an array and return a scalar    Args      args   numpy array   args 0     length  args 1     width    Returns      float         return args 0    args 1    def  area class args        Examples function that take an array and return a scalar    Args      args   numpy array   args 0     length  args 1     width    Returns      int         area   args 0    args 1    cl   1 if area   2 else 0   return cl   def  area and class args        Examples function that take an array and return an array    Args      args   numpy array   args 0     length  args 1     width    Returns      numpy array         area   args 0    args 1    cl   1 if area   2 else 0   return np array  area  cl     TRANSFORM CONFIG           this is an example for pass through features        i e   those doesn t need any processing                input columns     sepal length    sepal width    petal length                              petal width              the raw feature types are defined in the metadata            no need to do it here          process function   None           output columns     sepal length    sepal width    petal length                               petal width                this is an example for generating new numerical feature using a single       column from the raw data                input columns     sepal length             process function    numeric square           output columns      sepal length sq   constants NUMERICAL INDICATOR              N  stands for numerical feature              this is an example for generating new numerical feature using a single       column from the raw data                input columns     sepal width             process function    numeric square root           output columns      sepal width sr   constants NUMERICAL INDICATOR                this is an example for generating new numerical feature using multiple       columns from the raw data                input columns     petal length    petal width             process function    area           output columns      petal area   constants NUMERICAL INDICATOR                this is an example for generating new categorical feature using multiple       columns from the raw data                input columns     petal length    petal width             process function    area class           output columns      petal area cl   constants CATEGORICAL INDICATOR              C  stands for categorical feature              this is an example for generating multiple features using a single column       from the raw data                input columns     petal length             process function    numeric sq sr           output columns      petal length sq   constants NUMERICAL INDICATOR                                petal width sr   constants NUMERICAL INDICATOR                this is an example for generating multiple features using multiple columns       from the raw data                input columns     sepal length    sepal width             process function    area and class           output columns      sepal area   constants NUMERICAL INDICATOR                                sepal area cl   constants CATEGORICAL INDICATOR


GCP Transpose a BigQuery table using Dataflow img src img simplesqlbasedpivot png alt Simple SQL based pivot height 150 width 650 sql Transposing Pivoting Rotating the orientation of a table is a very common task that is performed as part of a standard report generation workflow While some relational databases provide a built in pivot function of some sort it can also be done via standard SQL As an example the following table can be pivoted using

Answer:
# Transpose a BigQuery table using Dataflow

Transposing/Pivoting/Rotating the orientation of a table is a very common task that is performed as part of a standard report generation workflow. While some relational databases provide a built-in *pivot* function of some sort, it can also be done via standard SQL.

As an example, the following table can be pivoted using [BigQuery Standard SQL](https://cloud.google.com/bigquery/docs/reference/standard-sql/):

```sql
SELECT
  id,
  MAX(CASE
      WHEN class = 'HVAC' THEN SALES END) AS HVAC_SALES,  MAX(CASE
      WHEN class = 'GENERATORS' THEN SALES END) AS GENERATORS_SALES
FROM
  `project-id.dataset_id.table_id`
GROUP BY
  id;
```

However, this can get significantly more complicated as:

* Number of pivot fields increase (single pivot field _**class**_ in the above example).
* Number of distinct values in pivot fields increase (two distinct values _**HVAC**_ and _**GENERATORS**_ in the above example).
* Number of pivot values increase (single pivot value _**sales**_ in the above example).

The most common approach to pivoting a complex table would be a two step approach:

1. Run a custom script to analyze the table and generate a SQL statement such as the one above.
2. Run the dynamically generated SQL to pivot the table and write the output to another table.

This could also be done using a convenient Dataflow pipeline as described below.

## [Pivot Dataflow Pipeline](src/main/java/com/google/cloud/pso/pipeline/Pivot.java)

[Pivot](src/main/java/com/google/cloud/pso/pipeline/Pivot.java) -
A Dataflow pipeline that can be used to pivot a BigQuery table across any number of pivot fields and values.
This pipeline allows the user to specify a comma separated list of fields across which the table should be rotated in addition to a comma separated list of fields that are rotated.
i.e. The user can specify:

* Key fields along which the table is rotated (_**id**_ in the above example).
* Pivot fields that should be rotated (_**class**_ in the above example).
* Pivot values that should be rotated (_**sales**_ in the above example).

The pipeline will perform various steps to complete the pivot process:

1. Validate that the fields are valid and have the correct datatypes.
2. Read the data from an input BigQuery table.
3. Analyze the pivot fields and dynamically generate the correct schema.
4. Pivot every record based on the dynamically generated schema.
5. Write the pivoted records into a target BigQuery table.

## Getting Started

### Requirements

* Java 8
* Maven 3

### Building the Project

Build the entire project using the maven compile command.
```sh
mvn clean compile
```

### Running unit tests

Run all unit tests.
```sh
mvn clean test
```

### Running the Pipeline

The above input table shows a slightly more complex example. In order to pivot this table, we have the following inputs:

* keyFields = id,locid
* pivotFields = class,on_sale,state
* pivotValues = sale_price,count

The _**desc**_ field is ignored and will not be in the output table.

The [Pivot](src/main/java/com/google/cloud/pso/pipeline/Pivot.java) pipeline will create a new pivot table based on the inputs.

Execute the pipeline using the maven exec:java command.
```sh
MY_PROJECT=my-project-id
MY_STAGING_BUCKET=my-staging-bucket-name
MY_DATASET_ID=my-dataset-id
MY_SOURCE_TABLE_ID=my-source-table-id
MY_TARGET_TABLE_ID=my-target-table-id

mvn compile exec:java -Dexec.mainClass=com.google.cloud.pso.pipeline.Pivot -Dexec.cleanupDaemonThreads=false -Dexec.args=" \
--project=$MY_PROJECT \
--runner=DataflowRunner \
--stagingLocation=gs://${MY_STAGING_BUCKET}/staging \
--tempLocation=gs://${MY_STAGING_BUCKET}/tmp \
--inputTableSpec=${MY_PROJECT}:${MY_DATASET_ID}.${MY_SOURCE_TABLE_ID} \
--outputTableSpec=${MY_PROJECT}:${MY_DATASET_ID}.${MY_TARGET_TABLE_ID} \
--keyFields=id,locid \
--pivotFields=class,on_sale,state \
--valueFields=sale_price,count"
``

Transpose a BigQuery table using Dataflow  Transposing Pivoting Rotating the orientation of a table is a very common task that is performed as part of a standard report generation workflow  While some relational databases provide a built in  pivot  function of some sort  it can also be done via standard SQL   As an example  the following table can be pivoted using  BigQuery Standard SQL  https   cloud google com bigquery docs reference standard sql      img src  img simple sql based pivot png  alt  Simple SQL based pivot  height 150 width 650        sql SELECT   id    MAX CASE       WHEN class    HVAC  THEN SALES END  AS HVAC SALES   MAX CASE       WHEN class    GENERATORS  THEN SALES END  AS GENERATORS SALES FROM    project id dataset id table id  GROUP BY   id       However  this can get significantly more complicated as     Number of pivot fields increase  single pivot field    class    in the above example     Number of distinct values in pivot fields increase  two distinct values    HVAC    and    GENERATORS    in the above example     Number of pivot values increase  single pivot value    sales    in the above example    The most common approach to pivoting a complex table would be a two step approach   1  Run a custom script to analyze the table and generate a SQL statement such as the one above  2  Run the dynamically generated SQL to pivot the table and write the output to another table   This could also be done using a convenient Dataflow pipeline as described below       Pivot Dataflow Pipeline  src main java com google cloud pso pipeline Pivot java    Pivot  src main java com google cloud pso pipeline Pivot java    A Dataflow pipeline that can be used to pivot a BigQuery table across any number of pivot fields and values  This pipeline allows the user to specify a comma separated list of fields across which the table should be rotated in addition to a comma separated list of fields that are rotated  i e  The user can specify     Key fields along which the table is rotated     id    in the above example     Pivot fields that should be rotated     class    in the above example     Pivot values that should be rotated     sales    in the above example    The pipeline will perform various steps to complete the pivot process   1  Validate that the fields are valid and have the correct datatypes  2  Read the data from an input BigQuery table  3  Analyze the pivot fields and dynamically generate the correct schema  4  Pivot every record based on the dynamically generated schema  5  Write the pivoted records into a target BigQuery table    img src  img pipeline graph png  alt  Pipeline Graph  height 650 width 500       Getting Started      Requirements    Java 8   Maven 3      Building the Project  Build the entire project using the maven compile command     sh mvn clean compile          Running unit tests  Run all unit tests     sh mvn clean test          Running the Pipeline   img src  img example raw table png  alt  Raw input table  height 200 width 550    The above input table shows a slightly more complex example  In order to pivot this table  we have the following inputs     keyFields   id locid   pivotFields   class on sale state   pivotValues   sale price count  The    desc    field is ignored and will not be in the output table   The  Pivot  src main java com google cloud pso pipeline Pivot java  pipeline will create a new pivot table based on the inputs    img src  img example pivoted table png  alt  Raw input table  height 175 width 850    Execute the pipeline using the maven exec java command     sh MY PROJECT my project id MY STAGING BUCKET my staging bucket name MY DATASET ID my dataset id MY SOURCE TABLE ID my source table id MY TARGET TABLE ID my target table id  mvn compile exec java  Dexec mainClass com google cloud pso pipeline Pivot  Dexec cleanupDaemonThreads false  Dexec args       project  MY PROJECT     runner DataflowRunner     stagingLocation gs     MY STAGING BUCKET  staging     tempLocation gs     MY STAGING BUCKET  tmp     inputTableSpec   MY PROJECT    MY DATASET ID    MY SOURCE TABLE ID      outputTableSpec   MY PROJECT    MY DATASET ID    MY TARGET TABLE ID      keyFields id locid     pivotFields class on sale state     valueFields sale price count


GCP Assumptions Through the use of machine learning you can build an automated categorization pipeline This solution describes an approach to automating the review process for audio files using machine learning APIs Introduction Many consumer facing applications allow creators to upload audio files as a part of the creative experience If you re running an application with a similar use case you may want to extract the text from the audio file and then classify based on the content For example you may want to categorize content or add appropriate tags for search indexing The process of having humans listening to content is problematic if you have a large volume of content Having users supplying their own tags may also be problematic because they may not include all useful tags or they may tag inaccurately

Answer:
Introduction
============

Many consumer-facing applications allow creators to upload audio files as a part of the creative experience. If you’re running an application with a similar use case, you may want to extract the text from the audio file and then classify based on the content. For example, you may want to categorize content or add appropriate tags for search indexing. The process of having humans listening to content is problematic if you have a large volume of content. Having users supplying their own tags may also be problematic because they may not include all useful tags or they may tag inaccurately.

Through the use of machine learning, you can build an automated categorization pipeline. This solution describes an approach to automating the review process for audio files using machine learning APIs.

Assumptions
============

This example only supports the encoding formats currently supported by the
[Speech API](https://cloud.google.com/speech-to-text/docs/encoding).

If you try to use .mp3 or another file type, then you may need to perform preprocessing to convert
your file into an accepted encoding type.

Background on Serverless Data Processing Pipelines

The solution involves creating five GCS buckets using default configuration settings. Because of
this, no [object lifecycle management](https://cloud.google.com/storage/docs/lifecycle) policies are
configured. If you would like to specify different retention policies you can [enable](https://cloud.google.com/storage/docs/managing-lifecycles#enable)
this using `gsutil` while following the deployment process.

During processing, audio files are moved between buckets as they progress
through various stages of the pipeline. Specifically, the audio file should first be moved to the
`staging` bucket. After the Speech API completes processing the file, the audio file is moved from
the `staging` bucket to either the `processed` or `error` bucket, depending on whether the Speech
API returned a success or error response.

Installation/Set-up

1. [Install and initialize the Cloud SDK](https://cloud.google.com/sdk/docs/how-to)

Create a GCP Project

1. Open a Linux terminal window or [Cloud Shell](https://console.cloud.google.com/home/dashboard?cloudshell=true), and enter the following to configure your desired project id.

````
export PROJECT=[project_id]
````

2. Create a new GCP project

````
gcloud config create project $PROJECT
````

3. Set your terminal to use that project

````
gcloud config set project $PROJECT
````

Deployment
==========

Step 1: [Deploy the App Engine frontend](#step-1)

Step 2a: [Use gcloud for remaining resources](#step-2a)

Step 2b: [Use terraform for remaining resources](#step2-b)

Step 3: [View Results](#view-results)

### Step 1

Clone the repository and change your directory into the root folder

1. In your terminal type the following to clone the professional-services repository:

````
git clone https://github.com/GoogleCloudPlatform/professional-services
````

2. Change directories into this project

`````
cd examples/ml-audio-content-profiling/
`````

Create Project Resources

1. Change directories into the frontend App Engine application folder.

````
cd app
````

2a. Compile the angular frontend application. Note that this requires [installing Angular](https://angular.io/cli) on
your device and compiling the output.

````
npm install -g @angular/cli
````

````
cd angular/
````

````
npm install
````

````
ng build --prod
````

````
cd ..
````

2b. There is also an [Open Sourced moderator UI](https://github.com/conversationai/conversationai-moderator)
which contains detailed features for sorting through results from the Perspective API. Note that
this will not display any results from the NLP API or a transcript itself, but you may add in these
additional features if you wish. This is an alternative to the frontend in the `app` folder in this
repository.

3. Deploy the application.

````
gcloud app deploy
````

You will be prompted to use a region to serve the location from. You may pick any region, but you
cannot change this value later.

You can verify that the app was deployed correctly by navigating to
https://`$PROJECT`.appspot.google.com. You should see the following UI:

![text](img/app_home.png?raw=true)

### Step 2a

Enable APIs for your Project

1. Enable APIs

````
gcloud services enable language.googleapis.com
gcloud services enable speech.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable commentanalyzer.googleapis.com
gcloud services enable cloudscheduler.googleapis.com
````

2. Change directories to be at the root directory again.

````
cd ..
````

Create PubSub resources

1. Create PubSub topic named stt_queue

````
export TOPIC_NAME=stt_queue

gcloud pubsub topics create $TOPIC_NAME
````

2. Create subscription to stt_queue topic

````
export SUBSCRIPTION_NAME=pull_stt_ids

gcloud pubsub subscriptions create $SUBSCRIPTION_NAME --topic=$TOPIC_NAME
````

3. Generate a static UUID that you will need in each of your bucket names to ensure that they are
unique.

3a. First install uuidgen. If it is already installed or if you are using Cloud Shell, skip this step.

````
sudo apt-get install uuid-runtime
````

3b. Then generate the random UUID

````
export STATIC_UUID=$(echo $(uuidgen | tr '[:upper:]' '[:lower:]') | cut -c1-20)
````

4. Create five GCP buckets to hold the output files

````
export staging_audio_bucket=staging-audio-files-$STATIC_UUID

gsutil mb gs://$staging_audio_bucket
````

````
export processed_audio_bucket=processed-audio-files-$STATIC_UUID
gsutil mb gs://$processed_audio_bucket
````

````
export error_audio_bucket=error-audio-files-$STATIC_UUID
gsutil mb gs://$error_audio_bucket
````

````
export transcription_bucket=transcription-files-$STATIC_UUID
gsutil mb gs://$transcription_bucket
````

````
export output_bucket=output-files-$STATIC_UUID
gsutil mb gs://$output_bucket
````

4. Deploy first Cloud Function to Send STT API

Change directories into the send_stt_api_function directory

````
cd send_stt_api_function/
````

Deploy function

````
gcloud functions deploy send_stt_api \
  --entry-point main \
  --runtime python37 \
  --trigger-resource $staging_audio_bucket \
  --trigger-event google.storage.object.finalize \
  --timeout 540s \
  --set-env-vars topic_name=$TOPIC_NAME,error_bucket=$error_audio_bucket
````

5. Deploy second Cloud Function to Read STT API Output

Change directories into the read_stt_api_function

````
cd ../read_stt_api_function/
````

````
gcloud functions deploy read_stt_api \
  --entry-point main \
   --runtime python37 \
   --trigger-resource cron_topic \
   --trigger-event google.pubsub.topic.publish \
   --timeout 540s \
   --set-env-vars topic_name=$TOPIC_NAME,subscription_name=$SUBSCRIPTION_NAME,transcription_bucket=$transcription_bucket,staging_audio_bucket=$staging_audio_bucket,processed_audio_bucket=$processed_audio_bucket,error_audio_bucket=$error_audio_bucket
````

6. Deploy Cloud Scheduler Job

````
gcloud scheduler jobs create pubsub check_stt_job \
  --schedule "*/10 * * * *" \
  --topic cron_topic \
  --message-body "Check Speech-to-text results"
````

Note that you can edit the `schedule` flag to be any interval in UNIX cron. By default, our solution
uses every 10 minutes.

7. Deploy Perspective API Function

Change directories into the perspective_api_function directory

````
cd ../perspective_api_function/
````

````
gcloud functions deploy perspective_api \
  --entry-point main \
  --runtime python37 \
  --trigger-resource $transcription_bucket \
  --trigger-event google.storage.object.finalize \
  --timeout 540s \
  --set-env-vars output_bucket=$output_bucket
````

8. Deploy NLP Function

Change directories into the nlp_api_function directory

````
cd ../nlp_api_function/
````

````
gcloud functions deploy nlp_api \
  --entry-point main \
  --runtime python37 \
  --trigger-resource $transcription_bucket \
  --trigger-event google.storage.object.finalize \
  --timeout 540s \
  --set-env-vars output_bucket=$output_bucket
````

### Step 2b

1. [Install Terraform](https://learn.hashicorp.com/terraform/getting-started/install)

2. Copy `terraform.tfvars.sample` to create the `terraform.tfvars` file.
You must input the `project_id` inside of the quotes. If you wish to edit any of the other
default values for the other variables specified in `variables.tf`, you may add them in your
`terraform.tfvars`.

3. In your terminal, cd into the `terraform/` directory.

````
cd terraform/
````

4. Enter the following commands, ensuring that there are no errors:

  ````
  terraform init
  ````

  ````
  terraform plan
  ````

  ````
  terraform apply
  ````

  ````
  yes
  ````

All of the resources should be deployed.

### View Results

Test it out

1. You can start by trying to upload an audio file in GCS. You can do this using `gsutil` or in the
UI under the staging bucket. This will trigger `send_stt_api_function`. This submits the
request to the Speech API and publishes the job id to PubSub.

2. By default, `read_stt_api_function` is scheduled to run every ten minutes, as configured by Cloud
Scheduler. If you want to test it earlier, you can navigate to Cloud Scheduler in the console and
click 'Run Now'. This will pull from the PubSub topic to grab any job ids. It then calls
the Speech API to see if the job is complete. If it is not complete, it repushes the id back to
PubSub. If it is complete, it extracts the transcript from the Speech API response. Finally, it then saves this
transcript in GCS in the transcription files bucket. It also moves the audio file from the staging
staging bucket to the processed audio bucket. If there were any errors, it moves the audio file
instead to the error audio bucket.

![](img/cloud_scheduler.png)

3. The upload in the previous step to the transcription files bucket then triggers the other two
functions: `perpsective_api_function` and `nlp_api_function`. Each of these downloads the
transcription file from GCS and then calls its respective API with it to receive insight about the
probability of toxic content in the file as well as entities mentioned. Each then publishes the
response into its respective GCS bucket.

4. You can view the result of the entire pipeline by using the deployed App Engine app. Navigate
to: https://`[PROJECT_ID]`.appspot.com. This will present a table of all files that have been
uploaded and which already have completed processing through the pipeline. You can click through
each file to view the resulting transcript, toxicity, and entity/sentiment analysis.

![](img/app_home_files.png)

Introduction               Many consumer facing applications allow creators to upload audio files as a part of the creative experience  If you re running an application with a similar use case  you may want to extract the text from the audio file and then classify based on the content  For example  you may want to categorize content or add appropriate tags for search indexing  The process of having humans listening to content is problematic if you have a large volume of content  Having users supplying their own tags may also be problematic because they may not include all useful tags or they may tag inaccurately   Through the use of machine learning  you can build an automated categorization pipeline  This solution describes an approach to automating the review process for audio files using machine learning APIs    Assumptions               This example only supports the encoding formats currently supported by the  Speech API  https   cloud google com speech to text docs encoding    If you try to use  mp3 or another file type  then you may need to perform preprocessing to convert your file into an accepted encoding type     h2 Background on Serverless Data Processing Pipelines  h2   The solution involves creating five GCS buckets using default configuration settings  Because of this  no  object lifecycle management  https   cloud google com storage docs lifecycle  policies are configured  If you would like to specify different retention policies you can  enable  https   cloud google com storage docs managing lifecycles enable  this using  gsutil  while following the deployment process   During processing  audio files are moved between buckets as they progress through various stages of the pipeline  Specifically  the audio file should first be moved to the  staging  bucket  After the Speech API completes processing the file  the audio file is moved from the  staging  bucket to either the  processed  or  error  bucket  depending on whether the Speech API returned a success or error response    h2 Installation Set up  h2   1   Install and initialize the Cloud SDK  https   cloud google com sdk docs how to    h3 Create a GCP Project  h3   1  Open a Linux terminal window or  Cloud Shell  https   console cloud google com home dashboard cloudshell true   and enter the following to configure your desired project id        export PROJECT  project id        2  Create a new GCP project       gcloud config create project  PROJECT       3  Set your terminal to use that project       gcloud config set project  PROJECT       Deployment             Step 1   Deploy the App Engine frontend   step 1   Step 2a   Use gcloud for remaining resources   step 2a   Step 2b   Use terraform for remaining resources   step2 b   Step 3   View Results   view results       Step 1  h3 Clone the repository and change your directory into the root folder  h3   1  In your terminal type the following to clone the professional services repository        git clone https   github com GoogleCloudPlatform professional services       2  Change directories into this project        cd examples ml audio content profiling          h3 Create Project Resources  h3  1  Change directories into the frontend App Engine application folder        cd app       2a  Compile the angular frontend application  Note that this requires  installing Angular  https   angular io cli  on your device and compiling the output        npm install  g  angular cli            cd angular             npm install            ng build   prod            cd          2b  There is also an  Open Sourced moderator UI  https   github com conversationai conversationai moderator  which contains detailed features for sorting through results from the Perspective API  Note that this will not display any results from the NLP API or a transcript itself  but you may add in these additional features if you wish  This is an alternative to the frontend in the  app  folder in this repository    3  Deploy the application        gcloud app deploy       You will be prompted to use a region to serve the location from  You may pick any region  but you cannot change this value later   You can verify that the app was deployed correctly by navigating to https     PROJECT  appspot google com  You should see the following UI     text  img app home png raw true        Step 2a  h3 Enable APIs for your Project  h3  1  Enable APIs       gcloud services enable language googleapis com gcloud services enable speech googleapis com gcloud services enable cloudfunctions googleapis com gcloud services enable commentanalyzer googleapis com gcloud services enable cloudscheduler googleapis com        2  Change directories to be at the root directory again        cd           h3 Create PubSub resources  h3   1  Create PubSub topic named stt queue       export TOPIC NAME stt queue  gcloud pubsub topics create  TOPIC NAME       2  Create subscription to stt queue topic       export SUBSCRIPTION NAME pull stt ids  gcloud pubsub subscriptions create  SUBSCRIPTION NAME   topic  TOPIC NAME       3  Generate a static UUID that you will need in each of your bucket names to ensure that they are unique   3a  First install uuidgen  If it is already installed or if you are using Cloud Shell  skip this step        sudo apt get install uuid runtime       3b  Then generate the random UUID       export STATIC UUID   echo   uuidgen   tr    upper       lower       cut  c1 20         4  Create five GCP buckets to hold the output files       export staging audio bucket staging audio files  STATIC UUID  gsutil mb gs    staging audio bucket            export processed audio bucket processed audio files  STATIC UUID gsutil mb gs    processed audio bucket            export error audio bucket error audio files  STATIC UUID gsutil mb gs    error audio bucket             export transcription bucket transcription files  STATIC UUID gsutil mb gs    transcription bucket            export output bucket output files  STATIC UUID gsutil mb gs    output bucket        4  Deploy first Cloud Function to Send STT API  Change directories into the send stt api function directory       cd send stt api function        Deploy function       gcloud functions deploy send stt api       entry point main       runtime python37       trigger resource  staging audio bucket       trigger event google storage object finalize       timeout 540s       set env vars topic name  TOPIC NAME error bucket  error audio bucket       5  Deploy second Cloud Function to Read STT API Output  Change directories into the read stt api function       cd    read stt api function             gcloud functions deploy read stt api       entry point main        runtime python37        trigger resource cron topic        trigger event google pubsub topic publish        timeout 540s        set env vars topic name  TOPIC NAME subscription name  SUBSCRIPTION NAME transcription bucket  transcription bucket staging audio bucket  staging audio bucket processed audio bucket  processed audio bucket error audio bucket  error audio bucket       6  Deploy Cloud Scheduler Job       gcloud scheduler jobs create pubsub check stt job       schedule    10                topic cron topic       message body  Check Speech to text results        Note that you can edit the  schedule  flag to be any interval in UNIX cron  By default  our solution uses every 10 minutes    7  Deploy Perspective API Function  Change directories into the perspective api function directory       cd    perspective api function             gcloud functions deploy perspective api       entry point main       runtime python37       trigger resource  transcription bucket       trigger event google storage object finalize       timeout 540s       set env vars output bucket  output bucket       8  Deploy NLP Function  Change directories into the nlp api function directory       cd    nlp api function             gcloud functions deploy nlp api       entry point main       runtime python37       trigger resource  transcription bucket       trigger event google storage object finalize       timeout 540s       set env vars output bucket  output bucket            Step 2b  1   Install Terraform  https   learn hashicorp com terraform getting started install   2  Copy  terraform tfvars sample  to create the  terraform tfvars  file  You must input the  project id  inside of the quotes  If you wish to edit any of the other default values for the other variables specified in  variables tf   you may add them in your  terraform tfvars    3  In your terminal  cd into the  terraform   directory        cd terraform        4  Enter the following commands  ensuring that there are no errors            terraform init                  terraform plan                  terraform apply                  yes          All of the resources should be deployed        View Results  h3 Test it out  h3   1  You can start by trying to upload an audio file in GCS  You can do this using  gsutil  or in the UI under the  b staging bucket  b   This will trigger  send stt api function   This submits the request to the Speech API and publishes the job id to PubSub    2  By default   read stt api function  is scheduled to run every ten minutes  as configured by Cloud Scheduler  If you want to test it earlier  you can navigate to Cloud Scheduler in the console and click  Run Now   This will pull from the PubSub topic to grab any job ids  It then calls the Speech API to see if the job is complete  If it is not complete  it repushes the id back to PubSub  If it is complete  it extracts the transcript from the Speech API response  Finally  it then saves this transcript in GCS in the transcription files bucket  It also moves the audio file from the staging staging bucket to the processed audio bucket  If there were any errors  it moves the audio file instead to the error audio bucket       img cloud scheduler png   3  The upload in the previous step to the transcription files bucket then triggers the other two functions   perpsective api function  and  nlp api function   Each of these downloads the transcription file from GCS and then calls its respective API with it to receive insight about the probability of toxic content in the file as well as entities mentioned  Each then publishes the response into its respective GCS bucket    4  You can view the result of the entire pipeline by using the deployed App Engine app  Navigate to  https     PROJECT ID   appspot com  This will present a table of all files that have been uploaded and which already have completed processing through the pipeline  You can click through each file to view the resulting transcript  toxicity  and entity sentiment analysis       img app home files png


GCP Prepare the infrastructure e g datasets tables etc needed by the pipeline by referring to the Creating infrastructure components Note the BigQuery dataset name that you crate for late steps Usage dataflow production ready Python

Answer:
# dataflow-production-ready (Python)

## Usage

### Creating infrastructure components

Prepare the infrastructure (e.g. datasets, tables, etc) needed by the pipeline by referring to the
[Terraform module](/terraform/README.MD)

Note the BigQuery dataset name that you crate for late steps.

### Creating Python Virtual Environment for development

In the module root directory, run the following:

```
python3 -m venv /tmp/venv/dataflow-production-ready-env
source /tmp/venv/dataflow-production-ready-env/bin/activate
pip install -r python/requirements.txt
```

### Setting the GCP Project

In the repo root directory, set the environment variables

```
export GCP_PROJECT=<PROJECT_ID>
export REGION=<GCP_REGION>
export BUCKET_NAME=<DEMO_BUCKET_NAME>
```

Then set the GCP project

```
gcloud config set project $GCP_PROJECT
```

Then, create a GCS bucket for this demo
```
gsutil mb -l $REGION -p $GCP_PROJECT gs://$BUCKET_NAME
```

### Running a full build

The build is defined by [cloudbuild.yaml](cloudbuild.yaml) and runs on Cloud Build. It applies the following steps:
* Run unit tests
* Build a container image as defined in [Dockerfile](Dockerfile)
* Create a Dataflow flex template based on the container image
* Run automated system integration test using the Flex template (including test resources provisioning)

Set the following variables:
```
export TARGET_GCR_IMAGE="dataflow_flex_ml_preproc"
export TARGET_GCR_IMAGE_TAG="python"
export TEMPLATE_GCS_LOCATION="gs://$BUCKET_NAME/template/spec.json"
```

Run the following command in the root folder

```
gcloud builds submit --config=python/cloudbuild.yaml --substitutions=_IMAGE_NAME=${TARGET_GCR_IMAGE},_IMAGE_TAG=${TARGET_GCR_IMAGE_TAG},_TEMPLATE_GCS_LOCATION=${TEMPLATE_GCS_LOCATION},_REGION=${REGION}
```

### Manual Commands

#### Prerequisites

* Create an input file similar to [integration_test_input.csv](/data/integration_test_input.csv) (or copy it to GCS and use it as input)

* Set extra variables (use same dataset name as created by the Terraform module)
```
export INPUT_CSV="gs://$BUCKET_NAME/input/path_to_CSV"
export BQ_RESULTS="project:dataset.ml_preproc_results"
export BQ_ERRORS="project:dataset.ml_preproc_errors"
export TEMP_LOCATION="gs://$BUCKET_NAME/tmp"
export SETUP_FILE="/dataflow/template/ml_preproc/setup.py"
```

#### Running pipeline locally

Export this extra variables and run the script
```
chmod +x run_direct_runner.sh
./run_direct_runner.sh
```

#### Running pipeline on Dataflow service

Export this extra variables and run the script
```
chmod +x run_dataflow_runner.sh
./run_dataflow_runner.sh
```

#### Running Flex Templates

Even if the job runs successfully on Dataflow service when submitted locally, the template has to be tested as well since
it might contain errors in the Docker file that prevents the job from running.

To run the flex template after deploying it, run:

```
chmod +x run_dataflow_template.sh
./run_dataflow_template.sh
```

Note that the parameter setup_file must be included in [metadata.json](ml_preproc/spec/metadata.json) and passed to the pipeline. It enables working with multiple Python modules/files and it's set to the path of
[setup.py](ml_preproc/setup.py) inside the docker container.

#### Running Unit Tests

To run all unit tests

```
python -m unittest discover
```

To run particular test file

```
python -m unittest ml_preproc.pipeline.ml_preproc_test
```

#### Debug flex-template container image
In cloud shell, run the deployed container image using the bash endpoint
```
docker run -it --entrypoint /bin/bash gcr.io/$PROJECT_ID/$TARGET_GCR_IMAGE
```

## Dataflow Pipeline
* [main.py](ml_preproc/main.py) - The entry point of the pipeline
* [setup.py](ml_preproc/setup.py) - To package the pipeline and distribute it to the workers. Without this file, main.py won't be able to import modules at runtime. [[source]](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#multiple-file-dependencies)

## Flex Templates Overview
The pipeline demonstrates how to use Flex Templates in Dataflow to create a template out of practically any Dataflow pipeline. This pipeline
does not use any [ValueProvider](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/value_provider.py) to accept user inputs and is built like any other non-templated
Dataflow pipeline. This pipeline also allows the user to change the job
graph depending on the value provided for an option at runtime

We make the pipeline ready for reuse by "packaging" the pipeline artifacts
in a Docker container. In order to simplify the process of packaging the pipeline into a container we
utilize [Google Cloud Build](https://cloud.google.com/cloud-build/).

We preinstall all the dependencies needed to *compile and execute* the pipeline
into a container using a custom [Dockerfile](ml_preproc/Dockerfile).

In this example, we are using the following base image for Python 3:

`gcr.io/dataflow-templates-base/python3-template-launcher-base`

We will utilize Google Cloud Builds ability to build a container using a Dockerfile as documented in the [quickstart](https://cloud.google.com/cloud-build/docs/quickstart-docker).

In addition, we will use a CD pipeline on Cloud Build to update the flex template automatically.

## Continues deployment
The CD pipeline is defined in [cloudbuild.yaml](ml_preproc/cloudbuild.yaml) to be executed by Cloud Build. It follows the following steps:
1. Run unit tests
2. Build and register a container image via Cloud Build as defined in the [Dockerfile](ml_preproc/Dockerfile). The container packages the Dataflow pipeline and its dependencies and acts as the Dataflow Flex Template
3. Build the Dataflow template by creating a spec.json file on GCS including the container image ID and the pipeline metadata based on [metadata.json](ml_preproc/spec/metadata.json). The template could be run later on by pointing to this spec.json file
4. Running system integration test using the deployed Flex-template and waiting for it's results

### Substitution variables
Cloud Build provides default variables such as `$PROJECT_ID` that could be used in the build YAML file. User defined variables could also be used in the form of `$_USER_VARIABLE`.

In this project the following variables are used:
- `$_TARGET_GCR_IMAGE`: The GCR image name to be submitted to Cloud Build (not URI) (e.g wordcount-flex-template)
- `$_TEMPLATE_GCS_LOCATION`: GCS location to store the template spec file (e.g. gs://bucket/dir/). The spec file path is required later on to submit run commands to Dataflow
- `$_REGION`: GCP region to deploy and run the dataflow flex template
- `$_IMAGE_TAG`: Image tag

These variables must be set during manual build execution or via a build trigger

### Triggering builds automatically
To trigger a build on certain actions (e.g. commits to master)
1. Go to Cloud Build > Triggers > Create Trigger. If you're using Github, choose the "Connect Repository" option.
2. Configure the trigger
3. Point the trigger to the [cloudbuild.yaml](ml_preproc/cloudbuild.yaml) file in the repository
4. Add the substitution variables as explained in the [Substitution variables](#substitution-variables) section.

dataflow production ready  Python      Usage      Creating infrastructure components   Prepare the infrastructure  e g  datasets  tables  etc  needed by the pipeline by referring to the  Terraform module   terraform README MD   Note the BigQuery dataset name that you crate for late steps       Creating Python Virtual Environment for development  In the module root directory  run the following       python3  m venv  tmp venv dataflow production ready env source  tmp venv dataflow production ready env bin activate pip install  r python requirements txt          Setting the GCP Project  In the repo root directory  set the environment variables      export GCP PROJECT  PROJECT ID  export REGION  GCP REGION  export BUCKET NAME  DEMO BUCKET NAME       Then set the GCP project      gcloud config set project  GCP PROJECT      Then  create a GCS bucket for this demo     gsutil mb  l  REGION  p  GCP PROJECT gs    BUCKET NAME           Running a full build  The build is defined by  cloudbuild yaml  cloudbuild yaml  and runs on Cloud Build  It applies the following steps    Run unit tests   Build a container image as defined in  Dockerfile  Dockerfile    Create a Dataflow flex template based on the container image   Run automated system integration test using the Flex template  including test resources provisioning   Set the following variables      export TARGET GCR IMAGE  dataflow flex ml preproc  export TARGET GCR IMAGE TAG  python  export TEMPLATE GCS LOCATION  gs    BUCKET NAME template spec json       Run the following command in the root folder      gcloud builds submit   config python cloudbuild yaml   substitutions  IMAGE NAME   TARGET GCR IMAGE   IMAGE TAG   TARGET GCR IMAGE TAG   TEMPLATE GCS LOCATION   TEMPLATE GCS LOCATION   REGION   REGION           Manual Commands       Prerequisites    Create an input file similar to  integration test input csv   data integration test input csv   or copy it to GCS and use it as input     Set extra variables  use same dataset name as created by the Terraform module      export INPUT CSV  gs    BUCKET NAME input path to CSV  export BQ RESULTS  project dataset ml preproc results  export BQ ERRORS  project dataset ml preproc errors  export TEMP LOCATION  gs    BUCKET NAME tmp  export SETUP FILE   dataflow template ml preproc setup py            Running pipeline locally   Export this extra variables and run the script     chmod  x run direct runner sh   run direct runner sh            Running pipeline on Dataflow service  Export this extra variables and run the script     chmod  x run dataflow runner sh   run dataflow runner sh            Running Flex Templates  Even if the job runs successfully on Dataflow service when submitted locally  the template has to be tested as well since it might contain errors in the Docker file that prevents the job from running    To run the flex template after deploying it  run        chmod  x run dataflow template sh   run dataflow template sh       Note that the parameter setup file must be included in  metadata json  ml preproc spec metadata json  and passed to the pipeline  It enables working with multiple Python modules files and it s set to the path of   setup py  ml preproc setup py  inside the docker container         Running Unit Tests  To run all unit tests      python  m unittest discover      To run particular test file      python  m unittest ml preproc pipeline ml preproc test           Debug flex template container image In cloud shell  run the deployed container image using the bash endpoint      docker run  it   entrypoint  bin bash gcr io  PROJECT ID  TARGET GCR IMAGE           Dataflow Pipeline    main py  ml preproc main py    The entry point of the pipeline    setup py  ml preproc setup py    To package the pipeline and distribute it to the workers  Without this file  main py won t be able to import modules at runtime    source   https   beam apache org documentation sdks python pipeline dependencies  multiple file dependencies       Flex Templates Overview The pipeline demonstrates how to use Flex Templates in Dataflow to create a template out of practically any Dataflow pipeline  This pipeline does not use any  ValueProvider  https   github com apache beam blob master sdks python apache beam options value provider py  to accept user inputs and is built like any other non templated Dataflow pipeline  This pipeline also allows the user to change the job graph depending on the value provided for an option at runtime  We make the pipeline ready for reuse by  packaging  the pipeline artifacts in a Docker container  In order to simplify the process of packaging the pipeline into a container we utilize  Google Cloud Build  https   cloud google com cloud build     We preinstall all the dependencies needed to  compile and execute  the pipeline into a container using a custom  Dockerfile  ml preproc Dockerfile    In this example  we are using the following base image for Python 3    gcr io dataflow templates base python3 template launcher base   We will utilize Google Cloud Builds ability to build a container using a Dockerfile as documented in the  quickstart  https   cloud google com cloud build docs quickstart docker    In addition  we will use a CD pipeline on Cloud Build to update the flex template automatically       Continues deployment The CD pipeline is defined in  cloudbuild yaml  ml preproc cloudbuild yaml  to be executed by Cloud Build  It follows the following steps  1  Run unit tests 2  Build and register a container image via Cloud Build as defined in the  Dockerfile  ml preproc Dockerfile   The container packages the Dataflow pipeline and its dependencies and acts as the Dataflow Flex Template 3  Build the Dataflow template by creating a spec json file on GCS including the container image ID and the pipeline metadata based on  metadata json  ml preproc spec metadata json   The template could be run later on by pointing to this spec json file 4  Running system integration test using the deployed Flex template and waiting for it s results       Substitution variables Cloud Build provides default variables such as   PROJECT ID  that could be used in the build YAML file  User defined variables could also be used in the form of    USER VARIABLE    In this project the following variables are used       TARGET GCR IMAGE   The GCR image name to be submitted to Cloud Build  not URI   e g wordcount flex template       TEMPLATE GCS LOCATION   GCS location to store the template spec file  e g  gs   bucket dir    The spec file path is required later on to submit run commands to Dataflow      REGION   GCP region to deploy and run the dataflow flex template      IMAGE TAG   Image tag  These variables must be set during manual build execution or via a build trigger       Triggering builds automatically To trigger a build on certain actions  e g  commits to master  1  Go to Cloud Build   Triggers   Create Trigger  If you re using Github  choose the  Connect Repository  option       2  Configure the trigger 3  Point the trigger to the  cloudbuild yaml  ml preproc cloudbuild yaml  file in the repository 4  Add the substitution variables as explained in the  Substitution variables   substitution variables  section


GCP In this solution we build an approch to ingestion flat files in GCS to BigQuery using serverless technology This solution might be not be performanct if you have frequent small files that lands to GCP We use data in this example Figure below shows an overall approach Once the object is uploaded in the GCS bucket a object notification is recevied by the Pub Sub Topic Pub Sub Topic triggers a cloud function which then invokes the serverless spark Any error during the cloud function invocation and serverless spark execution is send to dead letter topic Step 1 Create a bucket the bucket holds the data to be ingested in GCP Once the object is upload in a bucket the notification is created in Pub Sub topic Ingesting GCS files to BigQuery using Cloud Functions and Serverless Spark

Answer:
# Ingesting GCS files to BigQuery using Cloud Functions and Serverless Spark

In this solution, we build an approch to ingestion flat files (in GCS) to BigQuery using serverless technology. This solution might be not be performanct if you have frequent small files that lands to GCP. We use [Daily Shelter Occupancy](https://open.toronto.ca/dataset/daily-shelter-occupancy/) data in this example. Figure below shows an overall approach. Once the object is uploaded in the GCS bucket, a object notification is recevied by the Pub/Sub Topic. Pub/Sub Topic triggers a cloud function which then invokes the serverless spark. Any error during the cloud function invocation and serverless spark execution is send to dead letter topic.

![](docs/gcs2bq_serverless_spark.jpg)

- **Step 1:**  Create a bucket, the bucket holds the data to be ingested in GCP. Once the object is upload in a bucket, the notification is created in Pub/Sub topic.

    ```
    PROJECT_ID=<<project_id>>
    GCS_BUCKET_NAME=<<Bucket name>>
    gsutil mb gs://${GCS_BUCKET_NAME}
    gsutil notification create \
        -t projects/${PROJECT_ID}/topics/create_notification_${GCS_BUCKET_NAME} \
        -e OBJECT_FINALIZE \
        -f json gs://${GCS_BUCKET_NAME}
    ```
- **Step 2:** Build and copy jar to a GCS bucket(Create a GCS bucket to store the jar if you dont have one). There are number of dataproce templates that are avaliable to [use](https://github.com/GoogleCloudPlatform/dataproc-templates).

    ```
    GCS_ARTIFACT_REPO=<<artifact repo name>>
    gsutil mb gs://${GCS_ARTIFACT_REPO}
    cd gcs2bq-spark
    mvn clean install
    gsutil cp target/GCS2BQWithSpark-1.0-SNAPSHOT.jar gs://${GCS_ARTIFACT_REPO}/
    ```

- **Step 3:** [The page](https://cloud.google.com/dataproc-serverless/docs/concepts/network) describe the network configuration required to run serverless spark

  - **Open subnet connectivity:** The subnet must allow subnet communication on all ports. The following gcloud command attaches a network firewall to a subnet that allows ingress communications using all protocols on all ports if the source and destination are tagged with "serverless-spark"

      ```
      gcloud compute firewall-rules create allow-internal-ingress \
      --network="default" \
      --source-tags="serverless-spark" \
      --target-tags="serverless-spark" \
      --direction="ingress" \
      --action="allow" \
      --rules="all"
      ````

  - **Private Google Access:** The subnet must have [Private Google Access](https://cloud.google.com/vpc/docs/configure-private-google-access) enabled.
      - External network access. Drivers and executors have internal IP addresses. You can set up [Cloud NAT](https://cloud.google.com/nat/docs/overview) to allow outbound traffic using internal IPs on your VPC network.

- **Step 4:** Create necessary GCP resources required by Serverless Spark
    - **Create BQ Dataset** Create a dataset to load GCS files.
        ```
        DATASET_NAME=<<dataset_name>>
        bq --location=US mk -d \
            ${DATASET_NAME}
        ```

    -  **Create BQ table** Create a table using the schema in `schema/schema.json`
        ```
        TABLE_NAME=<<table_name>>
        bq mk --table ${PROJECT_ID}:${DATASET_NAME}.${TABLE_NAME} \
            ./schema/schema.json
        ```

    - **Create service account** Create service acccount used run the service account. We also create the permission required to read from GCS bucket, write to BigQuery table and publish error message in deadletter queue. The service account is used to run the serverless spark, so it needs dataproc worker role as well.

        ```
        SERVICE_ACCOUNT_ID="gcs-to-bq-sa"
        gcloud iam service-accounts create ${SERVICE_ACCOUNT_ID} \
            --description="GCS to BQ service account for Serverless Spark" \
            --display-name="GCS2BQ-SA"

        roles=("roles/dataproc.worker" "roles/bigquery.dataEditor" "roles/bigquery.jobUser" "roles/storage.objectViewer" "roles/pubsub.publisher")
        for role in ${roles[@]}; do
            gcloud projects add-iam-policy-binding ${PROJECT_ID} \
                --member="serviceAccount:${SERVICE_ACCOUNT_ID}@${PROJECT_ID}.iam.gserviceaccount.com" \
                --role="$role"
        done
        ```
    - **Create BQ temp Bucket** GCS to BigQuery requires a temporary bucket. Lets create a temporary bucket
        ```
        GCS_TEMP_BUCKET=<<temp_bucket>>
        gsutil mb gs://${GCS_TEMP_BUCKET}
        ```
    - **Create Deadletter Topic and Subscription** Lets create a dead letter topic and subscription

        ```
        ERROR_TOPIC=err_gcs2bq_${GCS_BUCKET_NAME}
        gcloud pubsub topics create $ERROR_TOPIC
        gcloud pubsub subscriptions create err_sub_${GCS_BUCKET_NAME}} \
        --topic=${ERROR_TOPIC}
        ```

    Once all resources are create, please change the varaibles value () in `trigger-serverless-spark-fxn/main.py` from line 25 to 29

        ```
        bq_temp_bucket = <<GCS_TEMP_BUCKET>>
        gcs_artifact_rep = <<GCS_ARTIFACT_REPO>>
        dataset= <<DATASET_NAME>>
        bq_table = <<TABLE_NAME>>
        error_topic=<<ERROR_TOPIC>>
        ```

- **Step 5:** The cloud function is triggered once the object is copied to bucket.  The cloud function triggers the Servereless spark

  Deploy the function.

    ```
    cd trigger-serverless-spark-fxn
    gcloud functions deploy trigger-serverless-spark-fxn --entry-point \
    invoke_sreverless_spark --runtime python37 \
    --trigger-resource ${GCS_BUCKET_NAME}_create_notification \
    --trigger-event google.pubsub.topic.publish
    ```

- **Step 6:** Invoke the end-to-end pipeline. Download [2020 Daily Center Data](https://ckan0.cf.opendata.inter.prod-toronto.ca/download_resource/800cc97f-34b3-4d4d-9bc1-6e2ce2d6f44a?format=csv) and upload to the GCS bucket(<<GCS_BUCKET_NAME>>) in Step 1.

**Debugging Pipelines**

  Error message for the failed data pipelines are publish to Pub/Sub topic (ERROR_TOPIC) created in Step 4(Create Deadletter Topic and Subscription). The errors from both cloud function and spark are forwarded to Pub/Sub. Pub/Sub topic might have multiple entry for the same data-pipeline instance. Messages in Pub/Sub topic can be filtered using "oid" attribute. The attribute(oid) is unique for each pipeline run and holds full object name with the [generation id](https://cloud.google.com/storage/docs/metadata#generation-number).

Ingesting GCS files to BigQuery using Cloud Functions and Serverless Spark  In this solution  we build an approch to ingestion flat files  in GCS  to BigQuery using serverless technology  This solution might be not be performanct if you have frequent small files that lands to GCP  We use  Daily Shelter Occupancy  https   open toronto ca dataset daily shelter occupancy   data in this example  Figure below shows an overall approach  Once the object is uploaded in the GCS bucket  a object notification is recevied by the Pub Sub Topic  Pub Sub Topic triggers a cloud function which then invokes the serverless spark  Any error during the cloud function invocation and serverless spark execution is send to dead letter topic       docs gcs2bq serverless spark jpg        Step 1     Create a bucket  the bucket holds the data to be ingested in GCP  Once the object is upload in a bucket  the notification is created in Pub Sub topic               PROJECT ID   project id       GCS BUCKET NAME   Bucket name       gsutil mb gs     GCS BUCKET NAME      gsutil notification create            t projects   PROJECT ID  topics create notification   GCS BUCKET NAME             e OBJECT FINALIZE            f json gs     GCS BUCKET NAME              Step 2    Build and copy jar to a GCS bucket Create a GCS bucket to store the jar if you dont have one   There are number of dataproce templates that are avaliable to  use  https   github com GoogleCloudPlatform dataproc templates                   GCS ARTIFACT REPO   artifact repo name       gsutil mb gs     GCS ARTIFACT REPO      cd gcs2bq spark     mvn clean install     gsutil cp target GCS2BQWithSpark 1 0 SNAPSHOT jar gs     GCS ARTIFACT REPO                Step 3     The page  https   cloud google com dataproc serverless docs concepts network  describe the network configuration required to run serverless spark          Open subnet connectivity    The subnet must allow subnet communication on all ports  The following gcloud command attaches a network firewall to a subnet that allows ingress communications using all protocols on all ports if the source and destination are tagged with  serverless spark                   gcloud compute firewall rules create allow internal ingress           network  default            source tags  serverless spark            target tags  serverless spark            direction  ingress            action  allow            rules  all                    Private Google Access    The subnet must have  Private Google Access  https   cloud google com vpc docs configure private google access  enabled          External network access  Drivers and executors have internal IP addresses  You can set up  Cloud NAT  https   cloud google com nat docs overview  to allow outbound traffic using internal IPs on your VPC network       Step 4    Create necessary GCP resources required by Serverless Spark         Create BQ Dataset   Create a dataset to load GCS files                       DATASET NAME   dataset name           bq   location US mk  d                 DATASET NAME                        Create BQ table   Create a table using the schema in  schema schema json                      TABLE NAME   table name           bq mk   table   PROJECT ID    DATASET NAME    TABLE NAME                  schema schema json                      Create service account   Create service acccount used run the service account  We also create the permission required to read from GCS bucket  write to BigQuery table and publish error message in deadletter queue  The service account is used to run the serverless spark  so it needs dataproc worker role as well                         SERVICE ACCOUNT ID  gcs to bq sa          gcloud iam service accounts create   SERVICE ACCOUNT ID                  description  GCS to BQ service account for Serverless Spark                  display name  GCS2BQ SA                   roles   roles dataproc worker   roles bigquery dataEditor   roles bigquery jobUser   roles storage objectViewer   roles pubsub publisher           for role in   roles      do             gcloud projects add iam policy binding   PROJECT ID                      member  serviceAccount   SERVICE ACCOUNT ID    PROJECT ID  iam gserviceaccount com                      role   role          done                     Create BQ temp Bucket   GCS to BigQuery requires a temporary bucket  Lets create a temporary bucket                     GCS TEMP BUCKET   temp bucket           gsutil mb gs     GCS TEMP BUCKET                      Create Deadletter Topic and Subscription   Lets create a dead letter topic and subscription                      ERROR TOPIC err gcs2bq   GCS BUCKET NAME          gcloud pubsub topics create  ERROR TOPIC         gcloud pubsub subscriptions create err sub   GCS BUCKET NAME               topic   ERROR TOPIC                   Once all resources are create  please change the varaibles value    in  trigger serverless spark fxn main py  from line 25 to 29                      bq temp bucket     GCS TEMP BUCKET           gcs artifact rep     GCS ARTIFACT REPO           dataset    DATASET NAME           bq table     TABLE NAME           error topic   ERROR TOPIC                    Step 5    The cloud function is triggered once the object is copied to bucket   The cloud function triggers the Servereless spark      Deploy the function               cd trigger serverless spark fxn     gcloud functions deploy trigger serverless spark fxn   entry point       invoke sreverless spark   runtime python37         trigger resource   GCS BUCKET NAME  create notification         trigger event google pubsub topic publish              Step 6    Invoke the end to end pipeline  Download  2020 Daily Center Data  https   ckan0 cf opendata inter prod toronto ca download resource 800cc97f 34b3 4d4d 9bc1 6e2ce2d6f44a format csv  and upload to the GCS bucket   GCS BUCKET NAME    in Step 1          Debugging Pipelines      Error message for the failed data pipelines are publish to Pub Sub topic  ERROR TOPIC  created in Step 4 Create Deadletter Topic and Subscription   The errors from both cloud function and spark are forwarded to Pub Sub  Pub Sub topic might have multiple entry for the same data pipeline instance  Messages in Pub Sub topic can be filtered using  oid  attribute  The attribute oid  is unique for each pipeline run and holds full object name with the  generation id  https   cloud google com storage docs metadata generation number