GCP GCP Project to host all the resources A terraform script is provided to setup all the required resources Introduction GCP resources This example implements the infrastructure required to deploy an end to end using platform MLOps with Vertex AI Infra setup

Answer:
# MLOps with Vertex AI - Infra setup

## Introduction
This example implements the infrastructure required to deploy an end-to-end [MLOps process](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf) using [Vertex AI](https://cloud.google.com/vertex-ai) platform.

##  GCP resources
A terraform script is provided to setup all the required resources:

- GCP Project  to host all the resources
- Isolated VPC network and a subnet to be used by Vertex and Dataflow (using a Shared VPC is also possible).
- Firewall rule to allow the internal subnet communication required by Dataflow
- Cloud NAT required to reach the internet from the different computing resources (Vertex and Dataflow)
- GCS buckets to host Vertex AI and Cloud Build Artifacts.
- BigQuery Dataset where the training data will be stored
- Service account `mlops-[env]@` with the minimum permissions required by Vertex and Dataflow
- Service account `github` to be used by Workload Identity Federation, to federate Github identity.
- Secret to store the Github SSH key to get access the CI/CD code repo (you will set the secret value later, so it can be used).

![MLOps project description](./images/mlops_projects.png "MLOps project description")

## Pre-requirements

### User groups

User groups provide a stable frame of reference that allows decoupling the final set of permissions from the stage where entities and resources are created, and their IAM bindings defined. These groups should be created before launching Terraform.

We use the following groups to control access to resources:

- *Data Scientits* (gcp-ml-ds@<company.org>). They create ML pipelines in the experimentation environment.
- *ML Engineers* (gcp-ml-eng@<company.org>). They handle and run the different environments, with access to all resources in order to troubleshoot possible issues with pipelines.

These groups are not suitable for production grade environments. You can configure the group names through the `groups`variable.

### Git environment for the ML Pipelines

Clone the Google Cloud Professional services [repo](https://github.com/GoogleCloudPlatform/professional-services) to a temp directory:
```
git clone https://github.com/GoogleCloudPlatform/professional-services.git
cd professional-services/
```

Setup your new Github repo using the Github web console or CLI.

Copy the `vertex_mlops_enterprise` folder to your local folder, including the Github actions, hidden dirs and files:

```
cp -r ./examples/vertex_mlops_enterprise/  <YOUR LOCAL FOLDER>
```

Commit the files in the main branch (`main`):
```
git init
git add *
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/<ORG>/<REPO>.git
git push -u origin main
```
You will need to configure the Github organization and repo name in the `github` variable.

### Branches
Create the additional branches in Github (`dev`, `staging`, `prod`). This can be also done from the UI (`Create branch: dev from main`).

Pull the remote repo with `git pull`.

Checkout the staging branch with `git checkout dev`.

Review the files `*.yml` files in the `.github/workflows` and modify them if needed. These files should be automatically updated when launched terraform.

Review the files `*.yaml` files in the `build` folder and modify them if needed. These files should be automatically updated when launched terraform.

##  Instructions
###  Deploy the different environments

You will need to repeat this process for each one of the different environments (01-development, 02-staging, 03-production):

- Go to the environment folder: I.e. `cd ../terraform/01-dev`
- It is recommended to have a remote state file. In this case, make sure to create the right `providers.tf` file, set the name of a bucket that you want to use as the storage for your Terraform state. This should be an existing bucket that your user has access to.
- Create a `terraform.tfvars` file and specify the required variables. You can use the `terraform.tfvars.sample` an an starting point

```tfm
project_create = {
    billing_account_id = "000000-123456-123456"
    parent             = "folders/111111111111"
}
project_id          = "creditcards-dev"
```
- Make sure you fill in the following parameters:
  - `project_create.billing_account_id`: Billing account
  - `project_create.parent `: Parent folder where the project will be created.
  - `project_id`:  Project id, references existing project if `project_create` is null.
- Make sure you have the right authentication setup (application default credentials, or a service account key)
- Run `terraform init` and `terraform apply`
- It is possible that some errors like `googleapi: Error 400: Service account xxxx does not exist.` appears. This is due to some dependencies with the Project IAM authoritative bindings of the service accounts. In this case, re-run again the process with `terraform apply`

##  What's next?
Continue [configuring the GIT integration with Cloud Build](./02-GIT_SETUP.md) and [launching the MLOps pipeline](./03-MLOPS.md).

<!-- BEGIN TFDOC -->
<!-- END TFDOC -->

MLOps with Vertex AI   Infra setup     Introduction This example implements the infrastructure required to deploy an end to end  MLOps process  https   services google com fh files misc practitioners guide to mlops whitepaper pdf  using  Vertex AI  https   cloud google com vertex ai  platform        GCP resources A terraform script is provided to setup all the required resources     GCP Project  to host all the resources   Isolated VPC network and a subnet to be used by Vertex and Dataflow  using a Shared VPC is also possible      Firewall rule to allow the internal subnet communication required by Dataflow   Cloud NAT required to reach the internet from the different computing resources  Vertex and Dataflow    GCS buckets to host Vertex AI and Cloud Build Artifacts    BigQuery Dataset where the training data will be stored   Service account  mlops  env    with the minimum permissions required by Vertex and Dataflow   Service account  github  to be used by Workload Identity Federation  to federate Github identity    Secret to store the Github SSH key to get access the CI CD code repo  you will set the secret value later  so it can be used      MLOps project description    images mlops projects png  MLOps project description       Pre requirements      User groups  User groups provide a stable frame of reference that allows decoupling the final set of permissions from the stage where entities and resources are created  and their IAM bindings defined  These groups should be created before launching Terraform   We use the following groups to control access to resources      Data Scientits   gcp ml ds  company org    They create ML pipelines in the experimentation environment     ML Engineers   gcp ml eng  company org    They handle and run the different environments  with access to all resources in order to troubleshoot possible issues with pipelines    These groups are not suitable for production grade environments  You can configure the group names through the  groups variable        Git environment for the ML Pipelines  Clone the Google Cloud Professional services  repo  https   github com GoogleCloudPlatform professional services  to a temp directory       git clone https   github com GoogleCloudPlatform professional services git cd professional services       Setup your new Github repo using the Github web console or CLI   Copy the  vertex mlops enterprise  folder to your local folder  including the Github actions  hidden dirs and files       cp  r   examples vertex mlops enterprise    YOUR LOCAL FOLDER       Commit the files in the main branch   main        git init git add   git commit  m  first commit  git branch  M main git remote add origin https   github com  ORG   REPO  git git push  u origin main     You will need to configure the Github organization and repo name in the  github  variable       Branches Create the additional branches in Github   dev    staging    prod    This can be also done from the UI   Create branch  dev from main     Pull the remote repo with  git pull    Checkout the staging branch with  git checkout dev    Review the files    yml  files in the   github workflows  and modify them if needed  These files should be automatically updated when launched terraform   Review the files    yaml  files in the  build  folder and modify them if needed  These files should be automatically updated when launched terraform       Instructions      Deploy the different environments  You will need to repeat this process for each one of the different environments  01 development  02 staging  03 production      Go to the environment folder  I e   cd    terraform 01 dev    It is recommended to have a remote state file  In this case  make sure to create the right  providers tf  file  set the name of a bucket that you want to use as the storage for your Terraform state  This should be an existing bucket that your user has access to    Create a  terraform tfvars  file and specify the required variables  You can use the  terraform tfvars sample  an an starting point     tfm project create         billing account id    000000 123456 123456      parent                folders 111111111111    project id             creditcards dev        Make sure you fill in the following parameters       project create billing account id   Billing account      project create parent    Parent folder where the project will be created       project id    Project id  references existing project if  project create  is null    Make sure you have the right authentication setup  application default credentials  or a service account key    Run  terraform init  and  terraform apply    It is possible that some errors like  googleapi  Error 400  Service account xxxx does not exist   appears  This is due to some dependencies with the Project IAM authoritative bindings of the service accounts  In this case  re run again the process with  terraform apply       What s next  Continue  configuring the GIT integration with Cloud Build    02 GIT SETUP md  and  launching the MLOps pipeline    03 MLOPS md         BEGIN TFDOC          END TFDOC


GCP Using dbt and Cloud Composer for managing BigQuery example code Cloud Composer is a fully managed data workflow orchestration service that empowers you to author schedule and monitor pipelines Code Examples 1 Basic This repository demonstrate using the dbt to manage tables in BigQuery and using Cloud Composer for schedule the dbt run DBT Data Building Tool is a command line tool that enables data analysts and engineers to transform data in their warehouses simply by writing select statements There are two sets of example

Answer:
# Using dbt and Cloud Composer for managing BigQuery example code

DBT (Data Building Tool) is a command-line tool that enables data analysts and engineers to transform data in their warehouses simply by writing select statements.
Cloud Composer is a fully managed data workflow orchestration service that empowers you to author, schedule, and monitor pipelines.

This repository demonstrate using the dbt to manage tables in BigQuery and using Cloud Composer for schedule the dbt run.

## Code Examples
There are two sets of example:
1. Basic
    The basic example is demonstrating the minimum configuration that you need to run dbt on Cloud Composer
2. Optimized
    The optimized example is demonstrating optimization on splitting the dbt run for each models,
    implementing incremental in the dbt model, and using Airflow execution date to handle backfill.

## Technical Requirements
These GCP services will be used in the example code:
- Cloud Composer
- BigQuery
- Google Cloud Storage (GCS)
- Cloud Build
- Google Container Repository (GCR)
- Cloud Source Repository (CSR)

## High Level Flow
This diagram explains the example solution's flow:
<img src="img/dbt-on-cloud-composer-diagram.PNG" width="700">

1. The code starts from a dbt project stored in a repository. (The example is under [basic or optimized]/dbt-project folder)
2. Any changes from the dbt project will trigger Cloud Build run
3. The Cloud Build will create/update an image to GCR; and export dbt docs to GCS
4. The Airflow DAG deployed to Cloud Composer (The example is under [basic or optimized]/dag folder)
5. The dbt run triggered using KubernetesPodOperator that pulls image from the step \#3
6. At the end of the process the BigQuery objects will be created/updated (i.e datasets and tables)

## How to run

### Prerequisites
1. Cloud Composer environment
    https://cloud.google.com/composer/docs/how-to/managing/creating
2. Set 3 ENVIRONMENT VARIABLES in the Cloud Composer (AIRFLOW_VAR_BIGQUERY_LOCATION, AIRFLOW_VAR_RUN_ENVIRONMENT, AIRFLOW_VAR_SOURCE_DATA_PROJECT)
    https://cloud.google.com/composer/docs/how-to/managing/environment-variables
3. Cloud Source Repository (or any git provider)
    Store the code from dbt-project in this dedicated repository
    The repository should contain dbt_project.yml file (Check the example code under [basic or optimized]/dbt-project] folder)
    Note that the dedicated dbt-project repository is not this example code repository (github repo)
4. Cloud Build triggers
    Trigger build from the dbt project repository
    https://cloud.google.com/build/docs/automating-builds/create-manage-triggers

    Set Trigger's substitution variables : _GCS_BUCKET and _DBT_SERVICE_ACCOUNT
    _GCS_BUCKET : A GCS bucket id for storing dbt documentation files.
    _DBT_SERVICE_ACCOUNT : A service account to run dbt from Cloud Build.
5. BigQuery API enabled
6. Service account to run dbt commands
7. Kubernetes Secret to be bound with the service account
    https://cloud.google.com/kubernetes-engine/docs/concepts/secret

    Alternatively, instead of using Kubernetes Secret, Workload Identity federation can be used (recommended approach). More details in **Authentication** section below.

### Profiles for running the dbt project
Check in the /dbt-project/.dbt/profiles.yml, you will find 2 options to run the dbt:
1. local
    You can run the dbt project using your local machine or Cloud Shell.
    To do that, run
    ```
    gcloud auth application-default login
    ```

    Trigger dbt run by using this command:
    ```
    dbt run --vars '{"project_id": [Your Project id], "bigquery_location": "us", "execution_date": "1970-01-01","source_data_project": "bigquery-public-data"}' --profiles-dir .dbt
    ```
2. remote
    This option is for running dbt using service account
    For example from Cloud build and Cloud Composer
    Check cloudbuild.yaml and dag/dbt_with_kubernetes.py to see how to use this option

### Run the code
After all the Prerequisites are prepared. You will have:
1. A dbt-project repository
2. Airflow DAG to run the dbt

Here are the follow up steps for running the code:
1. Push the code in dbt-project repository and make sure the Cloud Build triggered; and successfully create the docker image
2. In the Cloud Composer UI, run the DAG (e.g dbt_with_kubernetes.py)
3. If successfull, check the BigQuery console to check the tables

With this mechanism, you have 2 independent runs.
Updating the dbt-project, including models, schema and configurations will run the Cloud Build to create the docker image.
The DAG as dbt scheduler will run the dbt-project from the latest docker image available.

### Passing variables from Cloud Composer to dbt run
You can pass variables from Cloud Composer to the dbt run.
As an example, in this code we configure the BigQuery dataset location in the US as part of the DAG.
```
default_dbt_vars = {
        "project_id": project,

        # Example on using Cloud Composer's variable to be passed to dbt
        "bigquery_location": Variable.get("bigquery_location"),

        "key_file_dir": '/var/secrets/google/key.json',
        "source_data_project": Variable.get("source_data_project")
    }
```

In the dbt script, you can use the variable like this:
```
location: ""
```

### Authentication
When provisioning DBT runtime environment using KubernetesPodOperator there are two available options for authentication of the DBT process. To achieve better separation of concerns and follow good security practices, the identity of the DBT process (Service Account) should be different than the Cloud Composer Service Account.

Authentication options are:

- Workload Identity federation [**Recommended**]

A better way to manage the identity and authentication for K8s workloads is to avoid using SA Keys as Secrets and use Workload Identity federation mechanism [[documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)]

Depending on the Composer version you might need to enable Workload Identity in the cluster, configure the node pool to use the GKE_METADATA metadata server to request a short-lived auth tokens.

- Service Account key stored as Kubernetes Secret

Create a the SA key using the command (Note: SA keys are very sensitive and easy to misuse which can be a security risk. They should be kept protected and only be used under special circumstances)
```bash
gcloud iam service-accounts keys create key-file \
    --iam-account=sa-name@project-id.iam.gserviceaccount.com
```

Then save the key json file as *key.json* and configure *kubectl* command line tool to access the GKE cluster used by the Cloud Composer environment.

```bash
gcloud container clusters get-credentials gke-cluster-name --zone cluster-zone --project project-name
```

Onced authenticated, create dbt secret in the default namespace by running
```
kubectl create secret generic dbt-sa-secret --from-file key.json=./key.json
```

Since then, in the DAG code, when creating a container, the service account key will extracted from K8s Secret and then be be mounted under /var/secrets/google path in the container filesystem and available for DBT in the runtime.

- Composer 1

To enable Workload Identity on a new cluster, run the following command:
```bash
gcloud container clusters create CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --workload-pool=PROJECT_ID.svc.id.goog
```
Create new node pool (recommended) or update the existing one (might break the airflow setup and require extra steps):

```bash
gcloud container node-pools create NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --workload-metadata=GKE_METADATA
```

- Composer 2

No further actions required, as the GKE is already using Workload Identity and thanks to the Autopilot mode there's no need to manage the node pool manually.

To let the DAG to use the Workload Identity the following steps are required:

1) Create a namespace for the Kubernetes service account:
```bash
kubectl create namespace NAMESPACE
```
2) Create a Kubernetes service account for your application to use
```bash
kubectl create serviceaccount KSA_NAME \
    --namespace NAMESPACE
```
3) Assuming that the dbt-sa already exists and has the right permissions to trigger BigQuery jobs, the special binding has to be added to allow the Kubernetes service account as the IAM service account:

```bash
gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
```

4) Using *kubectl* tool annotate the Kubernetes service account with the email address of the IAM service account.

```bash
kubectl annotate serviceaccount KSA_NAME \
    --namespace NAMESPACE \
    iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com
```

To make use of the Workload Identity in our DAG, replace the existing KubernetesPodOperator call with the one that uses the Workload Identity.

1) Composer 1
Use example configuration from the snippet below:

```python
KubernetesPodOperator(
(...)
    namespace='dbt-namespace',
    service_account_name="dbt-k8s-sa",
    affinity={
        'nodeAffinity': {
            'requiredDuringSchedulingIgnoredDuringExecution': {
                'nodeSelectorTerms': [{
                    'matchExpressions': [{
                        'key': 'cloud.google.com/gke-nodepool',
                        'operator': 'In',
                        'values': [
                            'dbt-pool',
                        ]
                    }]
                }]
            }
        }
    }
(...)
).execute(context)
```

The affinity configuration lets GKE to schedule the pod in one of the specific node-pools that are set up to use Workload Identity.

2) Composer 2

In case of Composer 2 (Autopilot), the configuration is simpler, example snippet:

```python
KubernetesPodOperator(
(...)
    namespace='dbt-tasks',
    service_account_name="dbt-k8s-sa"
(...)
).execute(context)
```

When using Workload Identity option there is no need to store the IAM SA key as a Secret in GKE which massively improves the maintenance efforts and is generally considered more secure, as there is no need to generate and export the SA Key.

Using dbt and Cloud Composer for managing BigQuery example code     DBT  Data Building Tool  is a command line tool that enables data analysts and engineers to transform data in their warehouses simply by writing select statements     Cloud Composer is a fully managed data workflow orchestration service that empowers you to author  schedule  and monitor pipelines          This repository demonstrate using the dbt to manage tables in BigQuery and using Cloud Composer for schedule the dbt run         Code Examples There are two sets of example     1  Basic        The basic example is demonstrating the minimum configuration that you need to run dbt on Cloud Composer 2  Optimized        The optimized example is demonstrating optimization on splitting the dbt run for each models         implementing incremental in the dbt model  and using Airflow execution date to handle backfill      Technical Requirements These GCP services will be used in the example code       Cloud Composer   BigQuery   Google Cloud Storage  GCS     Cloud Build   Google Container Repository  GCR    Cloud Source Repository  CSR      High Level Flow This diagram explains the example solution s flow      img src  img dbt on cloud composer diagram PNG  width  700    1  The code starts from a dbt project stored in a repository   The example is under  basic or optimized  dbt project folder  2  Any changes from the dbt project will trigger Cloud Build run 3  The Cloud Build will create update an image to GCR  and export dbt docs to GCS 4  The Airflow DAG deployed to Cloud Composer  The example is under  basic or optimized  dag folder  5  The dbt run triggered using KubernetesPodOperator that pulls image from the step   3 6  At the end of the process the BigQuery objects will be created updated  i e datasets and tables      How to run      Prerequisites 1  Cloud Composer environment        https   cloud google com composer docs how to managing creating 2  Set 3 ENVIRONMENT VARIABLES in the Cloud Composer  AIRFLOW VAR BIGQUERY LOCATION  AIRFLOW VAR RUN ENVIRONMENT  AIRFLOW VAR SOURCE DATA PROJECT         https   cloud google com composer docs how to managing environment variables 3  Cloud Source Repository  or any git provider         Store the code from dbt project in this dedicated repository        The repository should contain dbt project yml file  Check the example code under  basic or optimized  dbt project  folder          Note that the dedicated dbt project repository is not this example code repository  github repo  4  Cloud Build triggers        Trigger build from the dbt project repository        https   cloud google com build docs automating builds create manage triggers         Set Trigger s substitution variables    GCS BUCKET and  DBT SERVICE ACCOUNT         GCS BUCKET   A GCS bucket id for storing dbt documentation files          DBT SERVICE ACCOUNT   A service account to run dbt from Cloud Build       5  BigQuery API enabled 6  Service account to run dbt commands 7  Kubernetes Secret to be bound with the service account        https   cloud google com kubernetes engine docs concepts secret      Alternatively  instead of using Kubernetes Secret  Workload Identity federation can be used  recommended approach   More details in   Authentication   section below       Profiles for running the dbt project Check in the  dbt project  dbt profiles yml  you will find 2 options to run the dbt     1  local         You can run the dbt project using your local machine or Cloud Shell      To do that  run                  gcloud auth application default login                 Trigger dbt run by using this command                 dbt run   vars    project id    Your Project id    bigquery location    us    execution date    1970 01 01   source data project    bigquery public data      profiles dir  dbt         2  remote        This option is for running dbt using service account        For example from Cloud build and Cloud Composer          Check cloudbuild yaml and dag dbt with kubernetes py to see how to use this option      Run the code After all the Prerequisites are prepared  You will have     1  A dbt project repository 2  Airflow DAG to run the dbt  Here are the follow up steps for running the code  1  Push the code in dbt project repository and make sure the Cloud Build triggered  and successfully create the docker image 2  In the Cloud Composer UI  run the DAG  e g dbt with kubernetes py  3  If successfull  check the BigQuery console to check the tables  With this mechanism  you have 2 independent runs     Updating the dbt project  including models  schema and configurations will run the Cloud Build to create the docker image     The DAG as dbt scheduler will run the dbt project from the latest docker image available       Passing variables from Cloud Composer to dbt run You can pass variables from Cloud Composer to the dbt run      As an example  in this code we configure the BigQuery dataset location in the US as part of the DAG         default dbt vars              project id   project             Example on using Cloud Composer s variable to be passed to dbt          bigquery location   Variable get  bigquery location                      key file dir     var secrets google key json            source data project   Variable get  source data project              In the dbt script  you can use the variable like this      location              Authentication When provisioning DBT runtime environment using KubernetesPodOperator there are two available options for authentication of the DBT process  To achieve better separation of concerns and follow good security practices  the identity of the DBT process  Service Account  should be different than the Cloud Composer Service Account   Authentication options are     Workload Identity federation    Recommended     A better way to manage the identity and authentication for K8s workloads is to avoid using SA Keys as Secrets and use Workload Identity federation mechanism   documentation  https   cloud google com kubernetes engine docs how to workload identity    Depending on the Composer version you might need to enable Workload Identity in the cluster  configure the node pool to use the GKE METADATA metadata server to request a short lived auth tokens      Service Account key stored as Kubernetes Secret  Create a the SA key using the command  Note  SA keys are very sensitive and easy to misuse which can be a security risk  They should be kept protected and only be used under special circumstances     bash gcloud iam service accounts keys create key file         iam account sa name project id iam gserviceaccount com      Then save the key json file as  key json  and configure  kubectl  command line tool to access the GKE cluster used by the Cloud Composer environment      bash gcloud container clusters get credentials gke cluster name   zone cluster zone   project project name      Onced authenticated  create dbt secret in the default namespace by running     kubectl create secret generic dbt sa secret   from file key json   key json      Since then  in the DAG code  when creating a container  the service account key will extracted from K8s Secret and then be be mounted under  var secrets google path in the container filesystem and available for DBT in the runtime     Composer 1  To enable Workload Identity on a new cluster  run the following command     bash gcloud container clusters create CLUSTER NAME         region COMPUTE REGION         workload pool PROJECT ID svc id goog     Create new node pool  recommended  or update the existing one  might break the airflow setup and require extra steps       bash gcloud container node pools create NODEPOOL NAME         cluster CLUSTER NAME         workload metadata GKE METADATA        Composer 2  No further actions required  as the GKE is already using Workload Identity and thanks to the Autopilot mode there s no need to manage the node pool manually   To let the DAG to use the Workload Identity the following steps are required   1  Create a namespace for the Kubernetes service account     bash kubectl create namespace NAMESPACE     2  Create a Kubernetes service account for your application to use    bash kubectl create serviceaccount KSA NAME         namespace NAMESPACE     3  Assuming that the dbt sa already exists and has the right permissions to trigger BigQuery jobs  the special binding has to be added to allow the Kubernetes service account as the IAM service account      bash gcloud iam service accounts add iam policy binding GSA NAME GSA PROJECT iam gserviceaccount com         role roles iam workloadIdentityUser         member  serviceAccount PROJECT ID svc id goog NAMESPACE KSA NAME        4  Using  kubectl  tool annotate the Kubernetes service account with the email address of the IAM service account      bash kubectl annotate serviceaccount KSA NAME         namespace NAMESPACE       iam gke io gcp service account GSA NAME GSA PROJECT iam gserviceaccount com      To make use of the Workload Identity in our DAG  replace the existing KubernetesPodOperator call with the one that uses the Workload Identity   1  Composer 1 Use example configuration from the snippet below      python KubernetesPodOperator            namespace  dbt namespace       service account name  dbt k8s sa       affinity            nodeAffinity                  requiredDuringSchedulingIgnoredDuringExecution                      nodeSelectorTerms                           matchExpressions                               key    cloud google com gke nodepool                            operator    In                            values                                  dbt pool                                                                                execute context       The affinity configuration lets GKE to schedule the pod in one of the specific node pools that are set up to use Workload Identity   2  Composer 2  In case of Composer 2  Autopilot   the configuration is simpler  example snippet      python KubernetesPodOperator            namespace  dbt tasks       service account name  dbt k8s sa          execute context       When using Workload Identity option there is no need to store the IAM SA key as a Secret in GKE which massively improves the maintenance efforts and is generally considered more secure  as there is no need to generate and export the SA Key


GCP FairingXGBoost this notebook demonstrate how to complete you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint Kubeflow Fairing Examples Train an XGBoost model in a local notebook Train an XGBoost model remotely on Kubeflow cluster with Kubeflow Fairing ML models in a hybrid cloud environment By using Kubeflow Fairing and adding a few lines of code you can run your ML In the repo we provided three notebooks to demonstrate the usage of Kubeflow Faring training job locally or in the cloud directly from Python code or a Jupyter notebook After your training job is is a Python package that streamlines the process of building training and deploying machine learning

Answer:
# Kubeflow Fairing Examples
`Kubeflow Fairing` is a Python package that streamlines the process of building, training, and deploying machine learning
(ML) models in a hybrid cloud environment. By using Kubeflow Fairing and adding a few lines of code, you can run your ML
training job locally or in the cloud, directly from Python code or a Jupyter notebook. After your training job is
complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint.

In the repo, we provided three notebooks to demonstrate the usage of Kubeflow Faring:
- Fairing_XGBoost: this notebook demonstrate how to
    * Train an XGBoost model in a local notebook,
    * Train an XGBoost model remotely on Kubeflow cluster, with Kubeflow Fairing
    * Train an XGBoost model remotely on AI Platform training, with Kubeflow Fairing
    * Deploy a trained model to Kubeflow, and call the deployed endpoint for predictions, with Kubeflow Fairing

- Fairing_Tensorflow_Keras: this notebook demonstrate how to
    * Train an Keras model in a local notebook,
    * Train an Keras model remotely on Kubeflow cluster (distributed), with Kubeflow Fairing
    * Train an Keras model remotely on AI Platform training, with Kubeflow Fairing
    * Deploy a trained model to Kubeflow, with Kubeflow Fairing

- Fairing_Py_File: this notebook introduces you to using Kubeflow Fairing to train the model, which is developed
using tensorflow or keras and enclosed in python files
    * Train an Tensorflow model remotely on Kubeflow cluster (distributed), with Kubeflow Fairing
    * Train an Tensorflow model remotely on AI Platform training, with Kubeflow Fairing

**Note that Kubeflow Fairing doesn't require kubeflow cluster as pre-requisite.
Kubeflow Fairing + AI platform is a valid combination**

## Setups:
### Prerequisites
Before you follow the instructions below to deploy your own kubeflow cluster,
you need a Google cloud project if you don't have one. You can find detailed instructions
[here](https://cloud.google.com/dataproc/docs/guides/setup-project).

- Make sure the following API & Services are enabled.
    * Cloud Storage
    * Cloud Machine Learning Engine
    * Cloud Source Repositories API (for CI/CD integration)
    * Compute Engine API
    * GKE API
    * IAM API
    * Deployment Manager API

- Configure project id and bucket id as environment variable.
  ```bash
  $ export PROJECT_ID=[your-google-project-id]
  $ export GCP_BUCKET=[your-google-cloud-storage-bucket-name]
  $ export DEPLOYMENT_NAME=[your-deployment-name]
  ```

- Deploy Kubeflow Cluster on GCP.
The running of training and serving jobs on kubeflow will require a kubeflow deployment. Please refer the link
[here](https://www.kubeflow.org/docs/gke/deploy/) to set up your Kubeflow deployment in your environment.

### Setup Environment
Please refer the link [here](https://www.kubeflow.org/docs/fairing/gcp-local-notebook/) to properly
setup the environments. The key steps are summarized as follows
- Create service account
    ```bash
    export SA_NAME = [service account name]
    gcloud iam service-accounts create ${SA_NAME}
    gcloud projects add-iam-policy-binding ${PROJECT_ID} \
      --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
      --role 'roles/editor'
    gcloud iam service-accounts keys create ~/key.json \
      --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
    ```

- Authorize for Source Repository
    ```bash
    gcloud auth configure-docker
    ```

- Update local kubeconfig (for submitting job to kubeflow cluster)
    ```bash
    export CLUSTER_NAME=${DEPLOYMENT_NAME} # this is the deployment name or the kubernetes cluster name
    export ZONE=us-central1-c
    gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${ZONE}
    ```

- Set the environmental variable: GOOGLE_APPLICATION_CREDENTIALS
    ```bash
    export GOOGLE_APPLICATION_CREDENTIALS = ~/key.json
    ```

- Install the latest version of fairing
    ```bash
    pip install git+https://github.com/kubeflow/fairing@master
    ```
### Running Notebook
Please not that the above configuration is required for notebook service running outside Kubeflow environment.
And the examples demonstrated are fully tested on notebook service outside Kubeflow cluster also, which
means it could be
- Notebook running on your personal computer
- Notebook on AI Platform, Google Cloud Platform
- Essentially notebook on any environment outside Kubeflow cluster

For notebook running inside Kubeflow cluster, for example JupytHub will be deployed together with kubeflow, the
environment variables, e.g. service account, projects and etc, should have been pre-configured while
setting up the cluster. The fairing package will also be pre-installed together with the deployment. **The only thing
need to be aware is that docker is usually not installed, which would require `cluster` as the builder option as
explained in the following section**

## Concepts of Kubeflow Fairing
There are three major concepts in Kubeflow Fairing: preprocessor, builder and deployer

### Preprocessor
The preprocessor defines how Kubeflow Fairing will map a set of inputs to a context when building the container image for your training job. The preprocessor can convert input files, exclude some files, and change the entrypoint for the training job.

* **python**: Copies the input files directly into the container image.
* **notebook**: Converts a notebook into a runnable python file. Strips out the non-python code.
* **full_notebook**: Runs a full notebook as-is, including bash scripts or non-Python code.
* **function**: FunctionPreProcessor preprocesses a single function. It sets as the command a function_shim that calls the function directly.

### Builder
The builder defines how Kubeflow Fairing will build the container image for your training job, and location of the container registry to store the container image in. There are different strategies that will make sense for different environments and use cases.

* **append**: Creates a Dockerfile by appending the your code as a new layer on an existing docker image. This builder requires less to time to create a container image for your training job, because the base image is not pulled to create the image and only the differences are pushed to the container image registry.
* **cluster**: Builds the container image for your training job in the Kubernetes cluster. This option is useful for building jobs in environments where a Docker daemon is not present, for example a hosted notebook.
* **docker**: Uses a local docker daemon to build and push the container image for your training job to your container image registry.

### Deployer
The deployer defines where Kubeflow Fairing will deploy and run your training job. The deployer uses the image produced by the builder to deploy and run your training job on Kubeflow or Kubernetes.

* **Job**: Uses a Kubernetes Job resource to launch your training job.
* **TfJob**: Uses the TFJob component of Kubeflow to launch your Tensorflow training job.
* **GCPJob**: Handle submitting training job to GCP.
* **Serving**: Serves a prediction endpoint using Kubernetes deployments and services

Kubeflow Fairing Examples  Kubeflow Fairing  is a Python package that streamlines the process of building  training  and deploying machine learning  ML  models in a hybrid cloud environment  By using Kubeflow Fairing and adding a few lines of code  you can run your ML training job locally or in the cloud  directly from Python code or a Jupyter notebook  After your training job is complete  you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint   In the repo  we provided three notebooks to demonstrate the usage of Kubeflow Faring    Fairing XGBoost  this notebook demonstrate how to       Train an XGBoost model in a local notebook        Train an XGBoost model remotely on Kubeflow cluster  with Kubeflow Fairing       Train an XGBoost model remotely on AI Platform training  with Kubeflow Fairing       Deploy a trained model to Kubeflow  and call the deployed endpoint for predictions  with Kubeflow Fairing    Fairing Tensorflow Keras  this notebook demonstrate how to       Train an Keras model in a local notebook        Train an Keras model remotely on Kubeflow cluster  distributed   with Kubeflow Fairing       Train an Keras model remotely on AI Platform training  with Kubeflow Fairing       Deploy a trained model to Kubeflow  with Kubeflow Fairing    Fairing Py File  this notebook introduces you to using Kubeflow Fairing to train the model  which is developed using tensorflow or keras and enclosed in python files       Train an Tensorflow model remotely on Kubeflow cluster  distributed   with Kubeflow Fairing       Train an Tensorflow model remotely on AI Platform training  with Kubeflow Fairing    Note that Kubeflow Fairing doesn t require kubeflow cluster as pre requisite  Kubeflow Fairing   AI platform is a valid combination       Setups      Prerequisites Before you follow the instructions below to deploy your own kubeflow cluster  you need a Google cloud project if you don t have one  You can find detailed instructions  here  https   cloud google com dataproc docs guides setup project      Make sure the following API   Services are enabled        Cloud Storage       Cloud Machine Learning Engine       Cloud Source Repositories API  for CI CD integration        Compute Engine API       GKE API       IAM API       Deployment Manager API    Configure project id and bucket id as environment variable       bash     export PROJECT ID  your google project id      export GCP BUCKET  your google cloud storage bucket name      export DEPLOYMENT NAME  your deployment name           Deploy Kubeflow Cluster on GCP  The running of training and serving jobs on kubeflow will require a kubeflow deployment  Please refer the link  here  https   www kubeflow org docs gke deploy   to set up your Kubeflow deployment in your environment       Setup Environment Please refer the link  here  https   www kubeflow org docs fairing gcp local notebook   to properly setup the environments  The key steps are summarized as follows   Create service account        bash     export SA NAME    service account name      gcloud iam service accounts create   SA NAME      gcloud projects add iam policy binding   PROJECT ID            member serviceAccount   SA NAME    PROJECT ID  iam gserviceaccount com           role  roles editor      gcloud iam service accounts keys create   key json           iam account   SA NAME    PROJECT ID  iam gserviceaccount com            Authorize for Source Repository        bash     gcloud auth configure docker            Update local kubeconfig  for submitting job to kubeflow cluster         bash     export CLUSTER NAME   DEPLOYMENT NAME    this is the deployment name or the kubernetes cluster name     export ZONE us central1 c     gcloud container clusters get credentials   CLUSTER NAME    region   ZONE             Set the environmental variable  GOOGLE APPLICATION CREDENTIALS        bash     export GOOGLE APPLICATION CREDENTIALS     key json            Install the latest version of fairing        bash     pip install git https   github com kubeflow fairing master             Running Notebook Please not that the above configuration is required for notebook service running outside Kubeflow environment  And the examples demonstrated are fully tested on notebook service outside Kubeflow cluster also  which means it could be   Notebook running on your personal computer   Notebook on AI Platform  Google Cloud Platform   Essentially notebook on any environment outside Kubeflow cluster  For notebook running inside Kubeflow cluster  for example JupytHub will be deployed together with kubeflow  the environment variables  e g  service account  projects and etc  should have been pre configured while setting up the cluster  The fairing package will also be pre installed together with the deployment    The only thing need to be aware is that docker is usually not installed  which would require  cluster  as the builder option as explained in the following section       Concepts of Kubeflow Fairing There are three major concepts in Kubeflow Fairing  preprocessor  builder and deployer      Preprocessor The preprocessor defines how Kubeflow Fairing will map a set of inputs to a context when building the container image for your training job  The preprocessor can convert input files  exclude some files  and change the entrypoint for the training job       python    Copies the input files directly into the container image      notebook    Converts a notebook into a runnable python file  Strips out the non python code      full notebook    Runs a full notebook as is  including bash scripts or non Python code      function    FunctionPreProcessor preprocesses a single function  It sets as the command a function shim that calls the function directly       Builder The builder defines how Kubeflow Fairing will build the container image for your training job  and location of the container registry to store the container image in  There are different strategies that will make sense for different environments and use cases       append    Creates a Dockerfile by appending the your code as a new layer on an existing docker image  This builder requires less to time to create a container image for your training job  because the base image is not pulled to create the image and only the differences are pushed to the container image registry      cluster    Builds the container image for your training job in the Kubernetes cluster  This option is useful for building jobs in environments where a Docker daemon is not present  for example a hosted notebook      docker    Uses a local docker daemon to build and push the container image for your training job to your container image registry       Deployer The deployer defines where Kubeflow Fairing will deploy and run your training job  The deployer uses the image produced by the builder to deploy and run your training job on Kubeflow or Kubernetes       Job    Uses a Kubernetes Job resource to launch your training job      TfJob    Uses the TFJob component of Kubeflow to launch your Tensorflow training job      GCPJob    Handle submitting training job to GCP      Serving    Serves a prediction endpoint using Kubernetes deployments and services


GCP you may not use this file except in compliance with the License https www apache org licenses LICENSE 2 0 Copyright 2022 Google LLC You may obtain a copy of the License at Unless required by applicable law or agreed to in writing software Licensed under the Apache License Version 2 0 the License

Answer:
<!--
Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Deploy the Custom Dataflow template by following these steps

## Overview

The purpose of this walkthrough is to create
[Custom Dataflow templates](https://cloud.google.com/dataflow/docs/concepts/dataflow-templates).

The value of Custom Dataflow templates is that it allows us to execute
Dataflow jobs without installing any code.  This is useful to enable Dataflow
execution using an automated process or to enable others without technical
expertise to run jobs via a user-friendly guided user interface.

## 1. Select or create project

**It is recommended to go through this walkthrough using a new temporary Google
Cloud project, unrelated to any of your existing Google Cloud projects.**

Select or create a project to begin.

<walkthrough-project-setup></walkthrough-project-setup>

## 2. Set default project

```sh
gcloud config set project <walkthrough-project-id/>
```

## 3. Setup environment

Best practice recommends a Dataflow job to:
1) Utilize a worker service account to access the pipeline's files and resources
2) Minimally necessary IAM permissions for the worker service account
3) Minimally required Google cloud services

Therefore, this step will:

- Create service accounts
- Provision IAM credentials
- Enable required Google cloud services

Run the terraform workflow in
the [infrastructure/01.setup](infrastructure/01.setup) directory.

Terraform will ask your permission before provisioning resources.
If you agree with terraform provisioning resources,
type `yes` to proceed.

```sh
DIR=infrastructure/01.setup
terraform -chdir=$DIR init
terraform -chdir=$DIR apply -var='project=<walkthrough-project-id/>'
```

## 4. Provision network

Best practice recommends a Dataflow job to:
1) Utilize a custom network and subnetwork
2) Minimally necessary network firewall rules
3) Building Python custom templates additionally requires the use of a
[Cloud NAT](https://cloud.google.com/nat/docs/overview); per best practice we
execute the Dataflow job using private IPs

Therefore, this step will:

- Provision a custom network and subnetwork
- Provision firewall rules
- Provision a Cloud NAT and its dependent Cloud Router

Run the terraform workflow in
the [infrastructure/02.network](infrastructure/02.network) directory.

Terraform will ask your permission before provisioning resources.
If you agree with terraform provisioning resources,
type `yes` to proceed.

```sh
DIR=infrastructure/02.network
terraform -chdir=$DIR init
terraform -chdir=$DIR apply -var='project=<walkthrough-project-id/>'
```

## 5. Provision data pipeline IO resources

The Apache Beam example that our Dataflow template executes is a derived word
count for both [Java](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java)
and [python](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py).

The word count example requires a source [Google Cloud Storage](https://cloud.google.com/storage) bucket.
To make the example interesting, we copy all the files from
`gs://apache-beam-samples/shakespeare/*` to a custom bucket in our project.

Therefore, this step will:
- Provision a Google Cloud storage bucket
- Create Google Cloud storage objects to read from in the pipeline

Run the terraform workflow in
the [infrastructure/03.io](infrastructure/03.io) directory.

Terraform will ask your permission before provisioning resources.
If you agree with terraform provisioning resources,
type `yes` to proceed.

```sh
DIR=infrastructure/03.io
terraform -chdir=$DIR init
terraform -chdir=$DIR apply -var='project=<walkthrough-project-id/>'
```

## 6. Provision the Dataflow template builder

We will use [Cloud Build](https://cloud.google.com/build) to build the
custom Dataflow template.  There are advantages to using Cloud Build to build
our custom Dataflow template, instead of performing the necessary commands on
our local machine.  Cloud Build connects to our version control,
[GitHub](https://GitHub.com) in this example, so that any changes made to
a specific branch will automatically trigger a new build of our Dataflow
template.

Therefore, this step will:

- Provision cloud build trigger that will:
    1. Run the language specific build process i.e. gradle shadowJar, go build, etc.
    2. Execute the `gcloud dataflow flex-template` command with relevant arguments.

### 6.1. Fork the repository into your own GitHub organization or personal account

In order to benefit from [Cloud Build](https://cloud.google.com/build), the service
requires we own this repository; it will not work with a any repository, even
if it is public.

Therefore, complete these steps before proceeding:
1) [Fork the repository](https://github.com/GoogleCloudPlatform/professional-services/fork)
2) [Connect forked repository to Cloud Build](https://console.cloud.google.com/cloud-build/triggers/connect)

### 6.2 Execute terraform module to provision Cloud Build trigger

First, set your GitHub organization or username:

```sh
GITHUB_REPO_OWNER=<change me>
```

Next, set expected defaults.  (_Note: Normally it makes sense to default terraform
variables instead of doing this._)

```sh
GITHUB_REPO_NAME=professional-services
WORKING_DIR_PREFIX=examples/dataflow-custom-templates
```

Run the terraform workflow in
the [infrastructure/04.template](infrastructure/04.template) directory.

Terraform will ask your permission before provisioning resources.
If you agree with terraform provisioning resources,
type `yes` to proceed.

```sh
DIR=infrastructure/04.template
terraform -chdir=$DIR init
terraform -chdir=$DIR apply -var="project=$(gcloud config get-value project)" -var="github_repository_owner=$GITHUB_REPO_OWNER" -var="github_repository_name=$GITHUB_REPO_NAME" -var="working_dir_prefix=$WORKING_DIR_PREFIX"
```

## 7. Run Cloud Build Trigger

Navigate to [cloud-build/triggers](https://console.cloud.google.com/cloud-build/triggers).
You should see a Cloud Build trigger listed for each language of this example.
Click the `RUN` button next to the created Cloud Build trigger to execute the
custom template Cloud Build trigger for your language of choice manually.

See [Create Manual Triggers](https://cloud.google.com/build/docs/automating-builds/create-manual-triggers?hl=en#running_manual_triggers)
for more information.

This step will take several minutes to complete.

## 8. Execute the Dataflow Template

### 1. Start the Dataflow Job creation form

There are multiple ways to run a Dataflow Job from a custom template.  We will
use the Google Cloud Web UI.

To start the process, navigate to [dataflow/createjob](https://console.cloud.google.com/dataflow/createjob).

### 2. Select Custom Template

Select `Custom Template` from the `Dataflow template` drop down menu.  Then,
click the `BROWSE` button and navigate to the bucket with the name that starts
with `dataflow-templates-`.  Within this bucket, select the json file object
that represents the template details.  You should see a JSON file for each
of the Cloud Build triggers you ran to create the custom template.

### 3. Complete Dataflow Job template UI form

The Google Cloud console will further prompt for required fields such as Job
name and any required fields for the custom Dataflow template.

### 4. Run the template

When you are satisfied by the values provided to the custom Dataflow template,
click the `RUN` button.

### 5. Monitor the Dataflow Job

Navigate to [dataflow/jobs](https://console.cloud.google.com/dataflow/jobs) to locate the job
you just created.  Clicking on the job will let you navigate to the job
monitoring screen.

Copyright 2022 Google LLC  Licensed under the Apache License  Version 2 0  the  License    you may not use this file except in compliance with the License  You may obtain a copy of the License at      https   www apache org licenses LICENSE 2 0  Unless required by applicable law or agreed to in writing  software distributed under the License is distributed on an  AS IS  BASIS  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND  either express or implied  See the License for the specific language governing permissions and limitations under the License         Deploy the Custom Dataflow template by following these steps     Overview  The purpose of this walkthrough is to create  Custom Dataflow templates  https   cloud google com dataflow docs concepts dataflow templates    The value of Custom Dataflow templates is that it allows us to execute Dataflow jobs without installing any code   This is useful to enable Dataflow execution using an automated process or to enable others without technical expertise to run jobs via a user friendly guided user interface      1  Select or create project    It is recommended to go through this walkthrough using a new temporary Google Cloud project  unrelated to any of your existing Google Cloud projects     Select or create a project to begin    walkthrough project setup   walkthrough project setup      2  Set default project     sh gcloud config set project  walkthrough project id           3  Setup environment  Best practice recommends a Dataflow job to  1  Utilize a worker service account to access the pipeline s files and resources 2  Minimally necessary IAM permissions for the worker service account 3  Minimally required Google cloud services  Therefore  this step will     Create service accounts   Provision IAM credentials   Enable required Google cloud services  Run the terraform workflow in the  infrastructure 01 setup  infrastructure 01 setup  directory   Terraform will ask your permission before provisioning resources  If you agree with terraform provisioning resources  type  yes  to proceed      sh DIR infrastructure 01 setup terraform  chdir  DIR init terraform  chdir  DIR apply  var  project  walkthrough project id            4  Provision network  Best practice recommends a Dataflow job to  1  Utilize a custom network and subnetwork 2  Minimally necessary network firewall rules 3  Building Python custom templates additionally requires the use of a  Cloud NAT  https   cloud google com nat docs overview   per best practice we execute the Dataflow job using private IPs  Therefore  this step will     Provision a custom network and subnetwork   Provision firewall rules   Provision a Cloud NAT and its dependent Cloud Router  Run the terraform workflow in the  infrastructure 02 network  infrastructure 02 network  directory   Terraform will ask your permission before provisioning resources  If you agree with terraform provisioning resources  type  yes  to proceed      sh DIR infrastructure 02 network terraform  chdir  DIR init terraform  chdir  DIR apply  var  project  walkthrough project id            5  Provision data pipeline IO resources  The Apache Beam example that our Dataflow template executes is a derived word count for both  Java  https   github com apache beam blob master examples java src main java org apache beam examples MinimalWordCount java  and  python  https   github com apache beam blob master sdks python apache beam examples wordcount minimal py    The word count example requires a source  Google Cloud Storage  https   cloud google com storage  bucket  To make the example interesting  we copy all the files from  gs   apache beam samples shakespeare    to a custom bucket in our project   Therefore  this step will    Provision a Google Cloud storage bucket   Create Google Cloud storage objects to read from in the pipeline  Run the terraform workflow in the  infrastructure 03 io  infrastructure 03 io  directory   Terraform will ask your permission before provisioning resources  If you agree with terraform provisioning resources  type  yes  to proceed      sh DIR infrastructure 03 io terraform  chdir  DIR init terraform  chdir  DIR apply  var  project  walkthrough project id            6  Provision the Dataflow template builder  We will use  Cloud Build  https   cloud google com build  to build the custom Dataflow template   There are advantages to using Cloud Build to build our custom Dataflow template  instead of performing the necessary commands on our local machine   Cloud Build connects to our version control   GitHub  https   GitHub com  in this example  so that any changes made to a specific branch will automatically trigger a new build of our Dataflow template   Therefore  this step will     Provision cloud build trigger that will      1  Run the language specific build process i e  gradle shadowJar  go build  etc      2  Execute the  gcloud dataflow flex template  command with relevant arguments       6 1  Fork the repository into your own GitHub organization or personal account  In order to benefit from  Cloud Build  https   cloud google com build   the service requires we own this repository  it will not work with a any repository  even if it is public   Therefore  complete these steps before proceeding  1   Fork the repository  https   github com GoogleCloudPlatform professional services fork  2   Connect forked repository to Cloud Build  https   console cloud google com cloud build triggers connect       6 2 Execute terraform module to provision Cloud Build trigger  First  set your GitHub organization or username      sh GITHUB REPO OWNER  change me       Next  set expected defaults     Note  Normally it makes sense to default terraform variables instead of doing this        sh GITHUB REPO NAME professional services WORKING DIR PREFIX examples dataflow custom templates      Run the terraform workflow in the  infrastructure 04 template  infrastructure 04 template  directory   Terraform will ask your permission before provisioning resources  If you agree with terraform provisioning resources  type  yes  to proceed      sh DIR infrastructure 04 template terraform  chdir  DIR init terraform  chdir  DIR apply  var  project   gcloud config get value project    var  github repository owner  GITHUB REPO OWNER   var  github repository name  GITHUB REPO NAME   var  working dir prefix  WORKING DIR PREFIX          7  Run Cloud Build Trigger  Navigate to  cloud build triggers  https   console cloud google com cloud build triggers   You should see a Cloud Build trigger listed for each language of this example  Click the  RUN  button next to the created Cloud Build trigger to execute the custom template Cloud Build trigger for your language of choice manually   See  Create Manual Triggers  https   cloud google com build docs automating builds create manual triggers hl en running manual triggers  for more information   This step will take several minutes to complete      8  Execute the Dataflow Template      1  Start the Dataflow Job creation form  There are multiple ways to run a Dataflow Job from a custom template   We will use the Google Cloud Web UI   To start the process  navigate to  dataflow createjob  https   console cloud google com dataflow createjob        2  Select Custom Template  Select  Custom Template  from the  Dataflow template  drop down menu   Then  click the  BROWSE  button and navigate to the bucket with the name that starts with  dataflow templates     Within this bucket  select the json file object that represents the template details   You should see a JSON file for each of the Cloud Build triggers you ran to create the custom template       3  Complete Dataflow Job template UI form  The Google Cloud console will further prompt for required fields such as Job name and any required fields for the custom Dataflow template       4  Run the template  When you are satisfied by the values provided to the custom Dataflow template  click the  RUN  button       5  Monitor the Dataflow Job  Navigate to  dataflow jobs  https   console cloud google com dataflow jobs  to locate the job you just created   Clicking on the job will let you navigate to the job monitoring screen


GCP 1 Cloud Storage Entities creation and update for Dialogflow This module is an example how to create and update entities for Dialogflow Recommended Reading Technology Stack 1 Dialogflow 1 Cloud Functions

Answer:
# Entities creation and update for Dialogflow
This module is an example how to create and update entities for Dialogflow.

## Recommended Reading
[Entities Options](https://cloud.google.com/dialogflow/docs/entities-options)

## Technology Stack
1. Cloud Storage
1. Cloud Functions
1. Dialogflow

## Programming Language
Python 3

## Project Structure
```
.
└── dialogflow_webhook_bank_example
 ├── main.py # Implementation of examples how to load entities in Dialogflow
 ├── entities.json # file with entities to be load in a json format
 ├── requirements.txt # Required libraries for this example
```
## Setup Instructions
 ### Project Setup
 How to setup your project for this example can be found [here](https://cloud.google.com/dialogflow/docs/quick/setup).

 ### Dialogflow Agent Setup
 Build an agent by following the instructions [here](https://cloud.google.com/dialogflow/docs/quick/build-agent).

 ### Cloud Storage Setup
 Upload the entities.json file to a bucket by following the instructions [here](https://cloud.google.com/storage/docs/quickstart-console#create_a_bucket).

 ### Cloud Functions Setup
 This implementation is deployed on GCP using Cloud Functions.
 More info [here](https://cloud.google.com/functions/docs/concepts/overview).

 To run the Python scripts on GCP, the `gcloud` command-line tool from the Google Cloud SDK is needed.
 Refer to the [installation](https://cloud.google.com/sdk/install) page for the appropriate
 instructions depending on your platform.

 Note that this project has been tested on a Unix-based environment.

 After installing, make sure to initialize your Cloud project:
 ```
 `$ gcloud init`
```

## Usage

### Create entities one by one
Use the EntityTypesClient.create_entity_type to create entities one by one.

#### More Info
[EntityType proto](https://github.com/googleapis/googleapis/blob/551cf1e6e3addcc63740427c4f9b40dedd3dac27/google/cloud/dialogflow/v2/entity_type.proto#L200)

[Client for Dialogflow API - EntityTypeClient.create_entity_type](https://dialogflow-python-client-v2.readthedocs.io/en/latest/_modules/dialogflow_v2/gapic/entity_types_client.html#EntityTypesClient.create_entity_type)

### Example
Run the sample using gcloud util as followed:

```
      $ gcloud functions call entities_builder --data '{
          "entities": [{
             "display_name":
                 "saving-account-types",
             "kind": "KIND_MAP",
             "entities": [{
                 "value": "saving-account-types",
                 "synonyms": [
                     "saving",
                     "saving account",
                     "child saving",
                     "IRA",
                     "CD",
                     "student saving"]
             }]
         }, {
             "display_name":
                 "checking-account-types",
             "kind": "KIND_MAP",
             "entities": [{
                 "value":
                     "checking-account-types",
                 "synonyms": [
                     "checking", "checking account", "student checking account",
                     "student account", "business checking account", "business account"
                 ]
             }]
         }, {
             "display_name": "account_types",
             "kind": "KIND_LIST",
             "entities": [
                 {
                     "value": "@saving-account-types:saving-account-types",
                     "synonyms": [
                         "@saving-account-types:saving-account-types"
                     ]
                 },
                 {
                     "value": "@checking-account-types:checking-account-types",
                     "synonyms": [
                         "@checking-account-types:checking-account-types"
                     ]
                 },
                 {
                     "value": "@sys.date-period:date-period @saving-account-types:saving-account-types",
                     "synonyms": [
                         "@sys.date-period:date-period @saving-account-types:saving-account-types"
                     ]
                 },
                 {
                     "value": "@sys.date-period:date-period @checking-account-types:checking-account-types",
                     "synonyms": [
                         "@sys.date-period:date-period @checking-account-types:checking-account-types"
                     ]
                 }
             ]
         }]
    }'
```

### Create entities in batch
Use the EntityTypesClient.batch_update_entity_types to create or update entities in batch.

#### More Info
[Client for Dialogflow API - EntityTypeClient.batch_update_entity_types](https://dialogflow-python-client-v2.readthedocs.io/en/latest/_modules/dialogflow_v2/gapic/entity_types_client.html#EntityTypesClient.batch_update_entity_types)

[EntityTypeBatch proto](https://github.com/googleapis/googleapis/blob/551cf1e6e3addcc63740427c4f9b40dedd3dac27/google/cloud/dialogflow/v2/entity_type.proto#L533)

[BatchUpdateEntityTypesRequest proto](https://github.com/googleapis/googleapis/blob/master/google/cloud/dialogflow/v2/entity_type.proto#L397)

#### Examples
##### Using entity_type_batch_uri
The URI to a Google Cloud Storage file containing entity types to update or create.
The URI must start with "gs://".
The entities.json file is an example of a json format file that can be uploaded to gcs and passed to the function.
```
    $ gcloud functions call entities_builder --data '{ "bucket": "gs://<bucket_name>/entities.json"}'

```
##### Using entity_type_batch_inline
For each entity type in the batch:
- The `name` is the the unique identifier of the entity type
- If `name` is specified, we update an existing entity type.
- If `name` is not specified, we create a new entity type.
```
   $ gcloud functions call entities_builder --data '{
       "entities_batch": {
           "entity_types":[
               {
                   "name": "5201cee0-ddfb-4f7c-ae94-fff87189d13c",
                   "display_name":
                     "saving-account-types",
                   "kind": "KIND_MAP",
                   "entities": [{
                       "value": "saving-account-types",
                       "synonyms": [
                           "saving",
                           "saving account",
                           "child saving",
                           "IRA",
                           "CD",
                           "student saving",
                           "senior saving"]
                   }]
               },
               {
                   "display_name":
                     "checking-account-types",
                   "kind": "KIND_MAP",
                   "entities": [{
                       "value":
                         "checking-account-types",
                       "synonyms": [
                           "checking", "checking account", "student checking account",
                           "student account", "business checking account", "business account"
                       ]
                   }]
               },
               {
                   "display_name": "account_types",
                   "kind": "KIND_LIST",
                   "entities": [
                       {
                           "value": "@saving-account-types:saving-account-types",
                           "synonyms": [
                               "@saving-account-types:saving-account-types"
                           ]
                       },
                       {
                           "value": "@checking-account-types:checking-account-types",
                           "synonyms": [
                               "@checking-account-types:checking-account-types"
                           ]
                       },
                       {
                           "value": "@sys.date-period:date-period @saving-account-types:saving-account-types",
                           "synonyms": [
                               "@sys.date-period:date-period @saving-account-types:saving-account-types"
                           ]
                       },
                       {
                           "value": "@sys.date-period:date-period @checking-account-types:checking-account-types",
                           "synonyms": [
                               "@sys.date-period:date-period @checking-account-types:checking-account-types"
                           ]
                       }
                   ]
               }
           ]
       }
}'
```

# Entities Definition
```
    └──  main
       ├── creates a map entity
       ├── create a composite entity
       ├── updates a map entity
```

Below the definition of the entities.

## Map entities
#### entity name: saving-accounts-types

Define synonyms: true
```
      {
        "value": "saving-account-types",
        "synonyms": [
          "saving",
          "saving account",
          "child saving",
          "IRA",
          "CD"
        ]
      }
```

#### entity name: checking-account-types

Define synonyms: true
```
      {
        "value": "checking-account-types",
        "synonyms": [
          "checking",
          "checking account",
          "student checking account",
          "student account",
          "business checking account",
          "business account"
        ]
      }
```
## Composite entities
### entity name: account-types
```
      {
        "value": "@saving-account-type:saving-account-type",
        "synonyms": [
          "@saving-account-type:saving-account-type"
        ]
      },

      {
        "value": "@checking-account-type:checking-account-type",
        "synonyms": [
          "@checking-account-type:checking-account-type"
        ]
      },

      {
        "value": "@sys.date-period:date-period @saving-account-type:saving-account-type",
        "synonyms": [
          "@sys.date-period:date-period @saving-account-type:saving-account-type"
        ]
      },

      {
        "value": "@sys.date-period:date-period @checking-account-type:checking-account-type",
        "synonyms": [
          "@sys.date-period:date-period @checking-account-type:checking-account-type"
        ]
      }
```

# References
[Client for Dialogflow API ](https://dialogflow-python-client-v2.readthedocs.io/en/latest/gapic/v2/api.html#dialogflow_v2.EntityTypesClient)

[EntityType proto](https://github.com/googleapis/googleapis/blob/master/google/cloud/dialogflow/v2/entity_type.proto)

[Protocol Buffers Tutorial](https://developers.google.com/protocol-buffers/docs/pythontutorial

Entities creation and update for Dialogflow This module is an example how to create and update entities for Dialogflow      Recommended Reading  Entities Options  https   cloud google com dialogflow docs entities options      Technology Stack 1  Cloud Storage 1  Cloud Functions 1  Dialogflow     Programming Language Python 3     Project Structure           dialogflow webhook bank example      main py   Implementation of examples how to load entities in Dialogflow      entities json   file with entities to be load in a json format      requirements txt   Required libraries for this example        Setup Instructions      Project Setup  How to setup your project for this example can be found  here  https   cloud google com dialogflow docs quick setup         Dialogflow Agent Setup  Build an agent by following the instructions  here  https   cloud google com dialogflow docs quick build agent         Cloud Storage Setup  Upload the entities json file to a bucket by following the instructions  here  https   cloud google com storage docs quickstart console create a bucket         Cloud Functions Setup  This implementation is deployed on GCP using Cloud Functions   More info  here  https   cloud google com functions docs concepts overview     To run the Python scripts on GCP  the  gcloud  command line tool from the Google Cloud SDK is needed   Refer to the  installation  https   cloud google com sdk install  page for the appropriate  instructions depending on your platform    Note that this project has been tested on a Unix based environment    After installing  make sure to initialize your Cloud project           gcloud init          Usage      Create entities one by one Use the EntityTypesClient create entity type to create entities one by one        More Info  EntityType proto  https   github com googleapis googleapis blob 551cf1e6e3addcc63740427c4f9b40dedd3dac27 google cloud dialogflow v2 entity type proto L200    Client for Dialogflow API   EntityTypeClient create entity type  https   dialogflow python client v2 readthedocs io en latest  modules dialogflow v2 gapic entity types client html EntityTypesClient create entity type       Example Run the sample using gcloud util as followed               gcloud functions call entities builder   data               entities                    display name                     saving account types                 kind    KIND MAP                 entities                        value    saving account types                     synonyms                           saving                         saving account                         child saving                         IRA                         CD                         student saving                                               display name                     checking account types                 kind    KIND MAP                 entities                        value                         checking account types                     synonyms                           checking    checking account    student checking account                         student account    business checking account    business account                                                                 display name    account types                 kind    KIND LIST                 entities                                              value     saving account types saving account types                         synonyms                                saving account types saving account types                                                                                value     checking account types checking account types                         synonyms                                checking account types checking account types                                                                                value     sys date period date period  saving account types saving account types                         synonyms                                sys date period date period  saving account types saving account types                                                                                value     sys date period date period  checking account types checking account types                         synonyms                                sys date period date period  checking account types checking account types                                                                                Create entities in batch Use the EntityTypesClient batch update entity types to create or update entities in batch        More Info  Client for Dialogflow API   EntityTypeClient batch update entity types  https   dialogflow python client v2 readthedocs io en latest  modules dialogflow v2 gapic entity types client html EntityTypesClient batch update entity types    EntityTypeBatch proto  https   github com googleapis googleapis blob 551cf1e6e3addcc63740427c4f9b40dedd3dac27 google cloud dialogflow v2 entity type proto L533    BatchUpdateEntityTypesRequest proto  https   github com googleapis googleapis blob master google cloud dialogflow v2 entity type proto L397        Examples       Using entity type batch uri The URI to a Google Cloud Storage file containing entity types to update or create  The URI must start with  gs      The entities json file is an example of a json format file that can be uploaded to gcs and passed to the function            gcloud functions call entities builder   data     bucket    gs    bucket name  entities json               Using entity type batch inline For each entity type in the batch    The  name  is the the unique identifier of the entity type   If  name  is specified  we update an existing entity type    If  name  is not specified  we create a new entity type           gcloud functions call entities builder   data            entities batch                 entity types                                         name    5201cee0 ddfb 4f7c ae94 fff87189d13c                       display name                         saving account types                       kind    KIND MAP                       entities                              value    saving account types                           synonyms                                 saving                               saving account                               child saving                               IRA                               CD                               student saving                               senior saving                                                                                display name                         checking account types                       kind    KIND MAP                       entities                              value                             checking account types                           synonyms                                 checking    checking account    student checking account                               student account    business checking account    business account                                                                                display name    account types                       kind    KIND LIST                       entities                                                          value     saving account types saving account types                               synonyms                                      saving account types saving account types                                                                                value     checking account types checking account types                               synonyms                                      checking account types checking account types                                                                                value     sys date period date period  saving account types saving account types                               synonyms                                      sys date period date period  saving account types saving account types                                                                                value     sys date period date period  checking account types checking account types                               synonyms                                      sys date period date period  checking account types checking account types                                                                                Entities Definition              main            creates a map entity            create a composite entity            updates a map entity      Below the definition of the entities      Map entities      entity name  saving accounts types  Define synonyms  true                      value    saving account types            synonyms                saving              saving account              child saving              IRA              CD                               entity name  checking account types  Define synonyms  true                      value    checking account types            synonyms                checking              checking account              student checking account              student account              business checking account              business account                           Composite entities     entity name  account types                      value     saving account type saving account type            synonyms                 saving account type saving account type                                       value     checking account type checking account type            synonyms                 checking account type checking account type                                       value     sys date period date period  saving account type saving account type            synonyms                 sys date period date period  saving account type saving account type                                       value     sys date period date period  checking account type checking account type            synonyms                 sys date period date period  checking account type checking account type                           References  Client for Dialogflow API   https   dialogflow python client v2 readthedocs io en latest gapic v2 api html dialogflow v2 EntityTypesClient    EntityType proto  https   github com googleapis googleapis blob master google cloud dialogflow v2 entity type proto    Protocol Buffers Tutorial  https   developers google com protocol buffers docs pythontutorial