GCP the entire solution on Google Cloud Platform GCP IoT Nirvana distributed all over the world and to follow temperature evolution by city in Architecture real time This document will guide you through the necessary steps to set up Internet of Things architecture running on Google Cloud Platform The purpose of This solution was built with the purpose of demonstrating an end to end the solution is to simulate the collection of temperature measures from sensors

Answer:
# IoT Nirvana

This solution was built with the purpose of demonstrating an end-to-end
Internet of Things architecture running on Google Cloud Platform. The purpose of
the solution is to simulate the collection of temperature measures from sensors
distributed all over the world and to follow temperature evolution by city in
real time. This document will guide you through the necessary steps to set up
the entire solution on Google Cloud Platform (GCP).

## Architecture

The image below contains a high level diagram of the solution.
![](img/architecture.png)

The following components are represented on the diagram:

1. Temperature sensors are simulated by running IoT Java clients on Google Compute
   Engine
2. The sensors send temperature data to an IoT Core registry running on GCP
3. The IoT Core registry publishes it into a PubSub topic
4. A streaming Dataflow pipeline is capturing the temperature data in real time
   by subscribing to and reading from the PubSub topic
5. Temperature data is pushed into BigQuery for analytics purposes
6. Temperature data is also saved to Datastore for real time querying
7. Temperature is displayed in real time in a Web AppEngine application
8. All components are logging data to Stackdriver

## Bootstrapping

As a pre-requisite you will need a GCP project to which you have owner rights
in order to facilitate the setup of the solution. In the remainder of this guide
this project's identifier will be referred to as **[PROJECT_ID]**.

Enable the following APIs in your project:

* [Cloud Pub/Sub API](https://console.cloud.google.com/apis/api/pubsub.googleapis.com)
* [DataFlow API](https://console.cloud.google.com/apis/api/dataflow.googleapis.com)
* [Google Cloud IoT API](https://console.cloud.google.com/apis/library/cloudiot.googleapis.com)

In order to run the simulation in ideal conditions, with 10 virtual machines,
please request an increase of your CPU quota to 80 vCPU. This is however
optional.

Create the environment variables that will be used throughout this tutorial. You can edit the default values provided below, however please note that not all products may be available in all regions:

```
export PROJECT_ID=<PROJECT_ID>
export BUCKET_NAME=<BUCKET_NAME>
export REGION=us-central1
export ZONE=us-central1-a
export PUBSUB_TOPIC=telemetry
export PUBSUB_SUBSCRIPTION=telemetry-sub
export IOT_REGISTRY=devices
export BIGQUERY_DATASET=warehouse
export BIGQUERY_TABLE=device_data
```
Run the **setup_gcp_environment.sh** script with the following parameters, in
this order, to create the corresponding resources in your GCP project:

Following arguments must be provided:

    1) Project Id
    2) Region where the Cloud IoT Core registry will be created
    3) Zone where a temporary VM to generate the Java image will be created
    4) Cloud IoT Core registry name
    5) PubSub telemetry topic name
    6) PubSub subscription name
    7) BigQuery dataset name

In addition, the script also creates an Debian image with Java pre-installed,
called **debian9-java8-img** that will be used to run the Java programs
simulating temperature sensors.

Example:

`setup_gcp_environment.sh $PROJECT_ID $REGION $ZONE $IOT_REGISTRY $PUBSUB_TOPIC $PUBSUB_SUBSCRIPTION  $BIGQUERY_DATASET`

## Build the solution

The first action is to compile and package all the modules of the solution: the
client simulating the temperature sensor, the Dataflow pipeline and the frontend
AppEngine application displaying temperatures in real time. To to this, run the
following command at the root of the project:

`mvn clean install`

## Dataflow pipeline

In order to run the Dataflow pipeline execute the `run_oncloud.sh` script at the
root of the project with the following parameters:

* **[PROJECT_ID]** - your project's identifier
* **[BUCKET_NAME]** - the name of the bucket created by the bootstrapping
  script, identical to your project's identifier, **[PROJECT_ID]**, where the
  Dataflow pipeline's binary package will be stored
* **[PUBSUB_TOPIC]** - the name of the PubSub topic created by the bootstrapping
  script, from which the Dataflow pipeline will read the temperature data;
  please note that this isn't the topic's canonical name, but instead the name
  relative to your project
* **[BIGQUERY_TABLE]** - a name for the BigQuery table where Dataflow will save
  the temperature data; the format of this parameter must follow the rule
  **[BIGQUERY_DATASET].[TABLE_NAME]**

Example:

`run_oncloud.sh $PROJECT_ID $BUCKET_NAME $PUBSUB_TOPIC $BIGQUERY_DATASET.$BIGQUERY_TABLE`

## Temperature sensor

Copy the JAR package containing the client binaries to Google Cloud Storage in
the bucket previously created. Run the following command in the `/client`
folder:

`gsutil cp target/google-cloud-demo-iot-nirvana-client-jar-with-dependencies.jar gs://$BUCKET_NAME/client/`

Check that the JAR file has been correctly copied in the Google Cloud Storage
bucket with the following command:

`gsutil ls gs://$BUCKET_NAME/client/google-cloud-demo-iot-nirvana-client-jar-with-dependencies.jar`

## AppEngine Web frontend

The following steps will allow you to set up and run on AppEngine the Web
frontend that allows to visualize in real time the temperature data captured
from the temperature sensors:
1. Modify the `src/main/webapp/startup.sh`file in the `/app-engine` folder by
   updating the variables below. This is the startup script of the Virtual
   Machines that will be created from the image **debian9-java8-img** and it
   creates 10 instances of the Java client simulating a temperature sensor.
   * PROJECT_ID - your GCP project's identifier, **[PROJECT_ID]**
   * BUCKET_NAME - name of the Google Cloud Storage bucket created by the
     bootstrapping script
   * REGISTRY_NAME - name of the IoT Core registry created by the bootstrapping
     script
   * REGION - region in which the IoT Core registry was created by the
     bootstrapping script
2. Copy the `startup.sh` file in the Google Cloud Storage bucket by running the
   following command in the `/app-engine` folder:
   `gsutil cp src/main/webapp/startup.sh gs://$BUCKET_NAME/`
3. Modify the `/pom.xml` file in the `/app-engine` folder:
   * Update the `<app.id/>` node with the **[PROJECT_ID]** of your GCP project
   * Update the `<app.version/>` with the desired version of the application
4. Modify the `src/main/webapp/config/client.properties` file in the
   `/app-engine` folder by updating the values of the following parameters:
   * GCS_BUCKET- name of the Google Cloud Storage bucket created by the
     bootstrapping script
   * GCE_METADATA_STARTUP_VALUE - path on Google Cloud Storage to the startup
     script edited at the previous step, gs://[BUCKET_NAME]/startup.sh
   * GCP_CLOUD_IOT_CORE_REGISTRY_NAME - name of the IoT Core registry created by
     the bootstrapping script
   * GCP_CLOUD_IOT_CORE_REGION - region in which the IoT Core registry was
     created by the bootstrapping script
5. Enable the [Maps Javascript API](https://console.cloud.google.com/apis/library/maps-backend.googleapis.com)
6. In the *Credentials* section of the Maps Javascript API generate the API key
   that will be used by the Web frontend to call Google Maps. This key will be
   referred to as **[MAPS_API_KEY]** further in the document. Make sure to:
   * Select HTTP in the "Application restrictions" list
   * Enter the URLs of the application, `https://[YOUR_PROJECT_ID].appspot.com/*`
     and `http://[YOUR_PROJECT_ID].appspot.com/*`, in the "Accept requests for
     these HTTP referrers (web sites)" input zone
7. Update the `src/main/webapp/index.html` file in the `/app-engine` folder by
   replacing the **[MAPS_API_KEY]** text with the actual value of the Google
   Maps API key generated at step 2.
8. Run the `gcloud app create` command to create the Google AppEngine
   application
9. Deploy the frontend Web application on AppEngine by running the following
   command in the at the root of the project:
   `mvn -pl app-engine appengine:update`

## Testing

In order to test the end to end solution, it is necessary first to start the
temperature sensors simulation. Follow the steps below to achieve this:

* Go to the following address in your web browser, which will display the map
  of the Earth with 3 buttons at the bottom: **Start**, **Update**, **Stop**
  `https://[YOUR_PROJECT_ID].appspot.com/index.html`
* Click on the **Start** button at the bottom left of the page (this also
  enables the buttons **Update** and **Stop**)
* The VM instances being launched are visible in the Google Cloud Console under
  [Compute Engine](https://console.cloud.google.com/compute/instances)

In order to visualize temperature data in real time on Google Maps do the
following:

* Click on the **Update** button at the bottom center of the page
  `https://[YOUR_PROJECT_ID].appspot.com/index.html`. This will display on the
  map test cities used for simulating the temperature sensors.
* Run the following SQL query in BigQuery to retrieve the most recent cities for
  which data is available:
  `SELECT City, Time FROM ``[BIGQUERY_DATASET].[TABLE_NAME]`` ORDER BY 2 DESC LIMIT 10`
* Locate on the map one of the cities returned by the query and click on the
  city icon to visualise the temperatures graph.

To stop the simulation click on the **Stop** button at the bottom right of the
page `https://[YOUR_PROJECT_ID].appspot.com/index.html`.

IoT Nirvana  This solution was built with the purpose of demonstrating an end to end Internet of Things architecture running on Google Cloud Platform  The purpose of the solution is to simulate the collection of temperature measures from sensors distributed all over the world and to follow temperature evolution by city in real time  This document will guide you through the necessary steps to set up the entire solution on Google Cloud Platform  GCP       Architecture  The image below contains a high level diagram of the solution      img architecture png   The following components are represented on the diagram   1  Temperature sensors are simulated by running IoT Java clients on Google Compute    Engine 2  The sensors send temperature data to an IoT Core registry running on GCP 3  The IoT Core registry publishes it into a PubSub topic 4  A streaming Dataflow pipeline is capturing the temperature data in real time    by subscribing to and reading from the PubSub topic 5  Temperature data is pushed into BigQuery for analytics purposes 6  Temperature data is also saved to Datastore for real time querying 7  Temperature is displayed in real time in a Web AppEngine application 8  All components are logging data to Stackdriver     Bootstrapping  As a pre requisite you will need a GCP project to which you have owner rights in order to facilitate the setup of the solution  In the remainder of this guide this project s identifier will be referred to as    PROJECT ID      Enable the following APIs in your project      Cloud Pub Sub API  https   console cloud google com apis api pubsub googleapis com     DataFlow API  https   console cloud google com apis api dataflow googleapis com     Google Cloud IoT API  https   console cloud google com apis library cloudiot googleapis com   In order to run the simulation in ideal conditions  with 10 virtual machines  please request an increase of your CPU quota to 80 vCPU  This is however optional   Create the environment variables that will be used throughout this tutorial  You can edit the default values provided below  however please note that not all products may be available in all regions       export PROJECT ID  PROJECT ID  export BUCKET NAME  BUCKET NAME  export REGION us central1 export ZONE us central1 a export PUBSUB TOPIC telemetry export PUBSUB SUBSCRIPTION telemetry sub export IOT REGISTRY devices export BIGQUERY DATASET warehouse export BIGQUERY TABLE device data     Run the   setup gcp environment sh   script with the following parameters  in this order  to create the corresponding resources in your GCP project   Following arguments must be provided       1  Project Id     2  Region where the Cloud IoT Core registry will be created     3  Zone where a temporary VM to generate the Java image will be created     4  Cloud IoT Core registry name     5  PubSub telemetry topic name     6  PubSub subscription name     7  BigQuery dataset name  In addition  the script also creates an Debian image with Java pre installed  called   debian9 java8 img   that will be used to run the Java programs simulating temperature sensors   Example    setup gcp environment sh  PROJECT ID  REGION  ZONE  IOT REGISTRY  PUBSUB TOPIC  PUBSUB SUBSCRIPTION   BIGQUERY DATASET      Build the solution  The first action is to compile and package all the modules of the solution  the client simulating the temperature sensor  the Dataflow pipeline and the frontend AppEngine application displaying temperatures in real time  To to this  run the following command at the root of the project    mvn clean install      Dataflow pipeline  In order to run the Dataflow pipeline execute the  run oncloud sh  script at the root of the project with the following parameters        PROJECT ID      your project s identifier      BUCKET NAME      the name of the bucket created by the bootstrapping   script  identical to your project s identifier     PROJECT ID     where the   Dataflow pipeline s binary package will be stored      PUBSUB TOPIC      the name of the PubSub topic created by the bootstrapping   script  from which the Dataflow pipeline will read the temperature data    please note that this isn t the topic s canonical name  but instead the name   relative to your project      BIGQUERY TABLE      a name for the BigQuery table where Dataflow will save   the temperature data  the format of this parameter must follow the rule      BIGQUERY DATASET   TABLE NAME     Example    run oncloud sh  PROJECT ID  BUCKET NAME  PUBSUB TOPIC  BIGQUERY DATASET  BIGQUERY TABLE      Temperature sensor  Copy the JAR package containing the client binaries to Google Cloud Storage in the bucket previously created  Run the following command in the   client  folder    gsutil cp target google cloud demo iot nirvana client jar with dependencies jar gs    BUCKET NAME client    Check that the JAR file has been correctly copied in the Google Cloud Storage bucket with the following command    gsutil ls gs    BUCKET NAME client google cloud demo iot nirvana client jar with dependencies jar      AppEngine Web frontend  The following steps will allow you to set up and run on AppEngine the Web frontend that allows to visualize in real time the temperature data captured from the temperature sensors  1  Modify the  src main webapp startup sh file in the   app engine  folder by    updating the variables below  This is the startup script of the Virtual    Machines that will be created from the image   debian9 java8 img   and it    creates 10 instances of the Java client simulating a temperature sensor       PROJECT ID   your GCP project s identifier     PROJECT ID         BUCKET NAME   name of the Google Cloud Storage bucket created by the      bootstrapping script      REGISTRY NAME   name of the IoT Core registry created by the bootstrapping      script      REGION   region in which the IoT Core registry was created by the      bootstrapping script 2  Copy the  startup sh  file in the Google Cloud Storage bucket by running the    following command in the   app engine  folder      gsutil cp src main webapp startup sh gs    BUCKET NAME   3  Modify the   pom xml  file in the   app engine  folder       Update the   app id    node with the    PROJECT ID    of your GCP project      Update the   app version    with the desired version of the application 4  Modify the  src main webapp config client properties  file in the      app engine  folder by updating the values of the following parameters       GCS BUCKET  name of the Google Cloud Storage bucket created by the      bootstrapping script      GCE METADATA STARTUP VALUE   path on Google Cloud Storage to the startup      script edited at the previous step  gs    BUCKET NAME  startup sh      GCP CLOUD IOT CORE REGISTRY NAME   name of the IoT Core registry created by      the bootstrapping script      GCP CLOUD IOT CORE REGION   region in which the IoT Core registry was      created by the bootstrapping script 5  Enable the  Maps Javascript API  https   console cloud google com apis library maps backend googleapis com  6  In the  Credentials  section of the Maps Javascript API generate the API key    that will be used by the Web frontend to call Google Maps  This key will be    referred to as    MAPS API KEY    further in the document  Make sure to       Select HTTP in the  Application restrictions  list      Enter the URLs of the application   https    YOUR PROJECT ID  appspot com         and  http    YOUR PROJECT ID  appspot com     in the  Accept requests for      these HTTP referrers  web sites   input zone 7  Update the  src main webapp index html  file in the   app engine  folder by    replacing the    MAPS API KEY    text with the actual value of the Google    Maps API key generated at step 2  8  Run the  gcloud app create  command to create the Google AppEngine    application 9  Deploy the frontend Web application on AppEngine by running the following    command in the at the root of the project      mvn  pl app engine appengine update      Testing  In order to test the end to end solution  it is necessary first to start the temperature sensors simulation  Follow the steps below to achieve this     Go to the following address in your web browser  which will display the map   of the Earth with 3 buttons at the bottom    Start      Update      Stop      https    YOUR PROJECT ID  appspot com index html    Click on the   Start   button at the bottom left of the page  this also   enables the buttons   Update   and   Stop      The VM instances being launched are visible in the Google Cloud Console under    Compute Engine  https   console cloud google com compute instances   In order to visualize temperature data in real time on Google Maps do the following     Click on the   Update   button at the bottom center of the page    https    YOUR PROJECT ID  appspot com index html   This will display on the   map test cities used for simulating the temperature sensors    Run the following SQL query in BigQuery to retrieve the most recent cities for   which data is available     SELECT City  Time FROM    BIGQUERY DATASET   TABLE NAME    ORDER BY 2 DESC LIMIT 10    Locate on the map one of the cities returned by the query and click on the   city icon to visualise the temperatures graph   To stop the simulation click on the   Stop   button at the bottom right of the page  https    YOUR PROJECT ID  appspot com index html


Answer:
- [Reusable Plugins](#reusable-plugins-for-cloud-data-fusion-cdf--cdap)
    - [Overview](#overview)
    - [CheckPointReadAction, CheckPointUpdateAction](#checkpointreadaction-checkpointupdateaction)
        - [Dependencies](#dependencies)
            - [Setting up Firestore](#setting-up-firestore)
            - [Set Runtime Arguments](#set-runtime-arguments)
    - [CopyTableAction](#copytableaction)
    - [DropTableAction](#droptableaction)
    - [TruncateTableAction](#truncatetableaction)
- [Putting it all together into a Pipeline](#putting-it-all-together-into-a-pipeline)
  - [CheckPointReadAction](#checkpointreadaction)
  - [TruncateTableAction](#truncatetableaction)
  - [Database source](#database-source)
  - [BigQuery sink](#bigquery-sink)
  - [MergeLastUpdateTSAction](#mergelastupdatetsaction)
  - [CheckPointUpdateAction](#checkpointupdateaction)
- [Building the CDF/CDAP Plugin (JAR file / JSON file) and deploying into CDF/CDAP](#building-the-cdfcdap-plugin-jar-file--json-file-and-deploying-into-cdfcdap)

# Reusable Plugins for Cloud Data Fusion (CDF) / CDAP

## Overview
The CDF/CDAP plugins detailed below can be reused in the context of data pipelines.

Let's say you run your incremental pipeline once every 5 minutes. When running an incremental pipeline, you have to filter the records by a specific field (e.g., `lastUpdateDateTime` of records > latest watermark value - buffer time) so it will sync the records that were updated since your last incremental sync. Subsequently, a merge and dedupe step is done to make sure only new/updated are synced into the destination table.

## `CheckPointReadAction`, `CheckPointUpdateAction`

**Plugin Description**
Creates, reads, and updates checkpoints in incremental pull pipelines.

`CheckPointReadAction` - reads checkpoints in Firestore DB and provides the data during runtime as environment variable

`CheckPointUpdateAction` - updates checkpoints in Firestore DB (i.e., creates a new document and stores maximum update date / time  from BQ so the next run it can use this checkpoint value to filter records that were added since then)

For now these plugins only support timestamp values - in the future, integer values can potentially be added.
### Dependencies

#### Setting up Firestore

1. Setup Firestore DB
    1. Firestore is used to store / read checkpoints, which is used in your incremental pipelines

1. Create a collection with a document from the parent path /
    1. Collection ID: `PIPELINE_CHECKPOINTS`
    1. Document ID: `INCREMENTAL_DEMO`

![image](img/1-create_pipeline_checkpoint_collection.png)

1. Create a collection under Parent path `/PIPELINE_CHECKPOINTS/INCREMENTAL_DEMO`
    1. Collection ID: `CHECKPOINT`
    1. Document ID: just accept what was provided initially
        1. Field #1
            1. Note:
                1. Set to maximum timestamp from destination (BQ table)
                1. Set to minimum timestamp if running for the first time from source (e.g., SQL server table)

            1. Field name: `CREATED_TIMESTAMP`
            1. Field type: `string`
            1. Date and time: `2020-05-08 17:21:01`

        1. Field #2
            1. Note: enter the current time in timestamp format
            1. Field name: CREATED_TIMESTAMP
            1. Field type: timestamp
            1. Date and time: 25/08/2020, 15:49

![image](img/2-create_check_point_collection.png)

#### Set Runtime Arguments

Before running the pipeline, add the `lastWatermarkValue` variable as runtime argument (on Pipeline Studio view, click on drop-down arrow for Run button) and set the value = 0 :

![image](img/3-runtime_arguments.png)

CheckpointReadAction will populate lastWatermarkValue with the CHECKPOINT_VALUE  from Firestore. lastWatermarkValue runtime argument will be used as parameter of the import query of the Database Source in a subsequent step:
```sql
SELECT * FROM test WHERE last_update_datetime > '${latestWatermarkValue}'
```

BigQuery - actual destination table name (this is where max checkpoint is taken from - i.e., max timestamp)

**Use Case**
This plugin can be used at the beginning of an incremental CDAP data pipeline to read the checkpoint value from the last sync.

Let's say you run your pipeline once every 5 minutes. When running an incremental pipeline, you have to filter the records by a specific field (timestamp - current date > current date -3) - it is doing merge and dedupe even though we are processing the same records to make sure duplicate records are not in the destination table.

`CheckPointReadAction` - reads checkpoints in Firestore DB and provides the data during runtime as environment variable

`CheckPointUpdateAction` - updates checkpoints in Firestore DB (i.e., creates a new document and stores maximum update date / time  from BQ so the next run it can use this checkpoint value to filter records that were added since then)

For now these plugins only support timestamp values - in the future, integer values can potentially be added.

**`CheckpointReadAction` plugin requires the following config properties:**

-  Label : plugin label name.
-  Specify the collection name in firestore DB: Name of the Collection.
-  Specify the document name to read the checkpoint details: Provide the document name specified in the Collection.
-  Buffer time to add to checkpoint value. (Note: in Minutes): Number of minutes that need to be subtracted from the Firestore collection value.
-  Project: project ID.
-  Key path: Service account key file path to communicate with the Firestore DB.

**Please see the following screenshot for example.**

![image](img/4-checkpoint_read_action_plugin_ui.png)

**`CheckpointUpdateAction` plugin requires the following configuration:**

-  Label : plugin label name.
-  Specify the collection name in firestore DB: Name of the Collection.
-  Specify the document name to read the checkpoint details: Provide the document name specified in the Collection.
-  Dataset name where incremental pull table exists: Big Query Dataset name.
-  Table name that needs incremental pull: Big Query table name.
-  Specify the checkpoint column from incremental pull table:
-  Project: project ID.
-  Key path: Service account key file path to communicate with the Firestore DB.

**Please see the below screenshot for example:**

![image](img/5-checkpoint_update_action_plugin_ui.png)

## `CopyTableAction`

**Plugin description**
Copies the BigQuery table from staging to destination at the end of the pipeline run. A new table is created if it doesn't exist. Otherwise, if the table exists, the plugin replaces the existing BigQuery destination table with data from staging.

**Use case**
This is applicable in the CDAP data pipelines which do the full import/scan the data from source system to BigQuery.

**Dependencies**
Destination dataset : `bq_dataset`
Destination table : `bq_table`
Source dataset : `bq_dataset_batch_staging`
Source table : `bq_table`

**`CopyTableAction` plugin requires the following configuration:**

-  Label: plugin label name.
-  Key path: Service account key file path to call the Big Query API.
-  Project ID: GCP project ID.
-  Dataset: Big Query dataset name.
-  Table Name: Big Query table name.

**Please see the following screenshot for example:**

![image](img/6-copy_table_action_ui.png)

## `DropTableAction`

**Plugin Description**
Drops a BigQuery table in the beginning of the pipeline runs.

**Use Case**
Useful to drop staging tables.

**Dependencies**
Requires BQ table to drop to exist.

**Drop table action plugin requires the following configuration:**

-  Label : plugin label name.
-  Key path: Service account key file path to call the Big Query API.
-  Project ID: GCP project ID.
-  Dataset: Big Query dataset name.
-  Table Name: Big Query table name.

Please see the following screenshot for example configuration:

![image](img/7-drop_table_action_ui.png)

## `TruncateTableAction`

**Plugin Description**
Truncates a BigQuery table when we set pipelines to restore the data from source.

**Use Case**
Applicable in restoring data pipelines from source.

**TruncateTable action plugin requires the following configuration:**

-  Label : plugin label name.
-  Key path: Service account key file path to call the Big Query API.
-  Project ID: GCP project ID.
-  Dataset: Big Query dataset name.
-  Table Name: Big Query table name.

**Please see the following screenshot for example configuration:**

![image](img/8-truncate_table_action_ui.png)

# Putting it all together into a Pipeline

`CheckPointReadAction` → `TruncateTableAction` → Database → BigQuery → `MergeLastUpdateTSAction` → `CheckPointUpdateAction`

What does the pipeline do?

1. `CheckPointReadAction` - reads latest checkpoint from Firestore
1. `TruncateTableAction` - truncate the records in the log table
1. Database Source- imports data from the source
1. BigQuery Sink - exports data into BigQuery from previous step (database source)
1. `MergeLastUpdateTSAction` -  merge based on timestamp and the update column list (columns to keep in the merge).
    -  Note: Alternatively, you can use [`BigQueryExecute`](https://github.com/data-integrations/google-cloud/blob/develop/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java) action to do a Merge.
1. `CheckPointUpdateAction` - update checkpoint in Firestore from the max record lastUpdateTimestamp in BigQuery

## Successful run of Incremental Pipeline

![image](img/9-successful_run_incremental_v2_pipeline.png)

### Runtime arguments (set latestWatermarkValue to 0)
![image](img/9-set_runtime_arugments_latestWatermarkValue.png)

### `CheckPointReadAction`

**Label:**
`CheckPointReadAction`

**Specify the document name to read the checkpoint details\*:**
INCREMENTAL_DEMO

**Buffer time to add to checkpoint value. (Note: in Minutes):**
1

**project:**
`pso-cdf-plugins-287518`

**serviceFilePath:**
auto-detect

**Screenshot:**

![image](img/10-checkpoint_read_action_ui_pipeline_parameters.png)

### `TruncateTableAction`

**Label:**
`TruncateTableAction`

**Key path*:**
auto-detect

**ProjectId* :**
`pso-cdf-plugins-287518`

**Dataset** *
`bq_dataset`

**Table name***
`bq_table_LOG`

![image](img/11-truncate_table_action_ui_pipeline_parameters.png)

### Database source

**Label** *
Database

**Reference Name***
test

**Plugin Name***
sqlserver42

**Plugin Type**
jdbc

**Connection String**
jdbc:sqlserver://<fill in IP address of database server>:<db port>;databaseName=main;user=<fill in user>;password=<fill in password>;

**Import Query:**
```sql
SELECT * FROM test WHERE last_update_datetime > '${latestWatermarkValue}'
```

![image](img/12-database_source_ui_pipeline_parameters.png)

### BigQuery sink

**Label** *
BigQuery

**Reference Name***
bq_table_sink

**Project ID**
`pso-cdf-plugins-287518`

**Dataset***
`bq_dataset`

**Table***  (write to a temporary table, e.g., bq_table_LOG)
`bq_table_LOG`

**Service Account File Path**
auto-detect

**Schema**

![image](img/13-bigquery_sink_ui_pipeline_parameters.png)

### `MergeLastUpdateTSAction`

**Label***
`MergeLastUpdateTSAction`

**Key path***
auto-detect

**Project ID***
`pso-cdf-plugins-287518`

**Dataset name**
`bq_dataset`

**Table name***
`bq_table`

**Primary key list***
id

**Update columns list***
id,name,last_update_datetime

![image](img/14-merge_last_update_ts_action.png)

### `CheckPointUpdateAction`

**Label***
`CheckPointUpdateAction`

**Specify the collection name in firestore DB***
PIPELINE_CHECKPOINTS

**Specify the document name to read the checkpoint details***
INCREMENTAL_DEMO

**Dataset name where incremental pull table exists***
`bq_dataset`

**Table name that needs incremental pull***
`bq_table`

**Specify the checkpoint column from incremental pull table***
last_update_datetime

**serviceFilePath**
auto-detect

**project**
`pso-cdf-plugins-287518`

![image](img/15-checkpoint_update_action_ui_pipeline_parameters.png)

## Building the CDF/CDAP Plugin (JAR file / JSON file) and deploying into CDF/CDAP

This plugin requires Java JDK1.8 and maven.

1. To build the CDAP / CDF plugin jar, execute the following command on the root.
```bash
mvn clean compile package
```

2. You will find the generated JAR file and JSON file under target folder:
   1. `GoogleFunctions-1.6.jar`
   1. `GoogleFunctions-1.6.json`

1. Deploy `GoogleFunctions-1.6.jar` and `GoogleFunctions-1.6.json` into CDF/CDAP (note that if you have the same version already deployed then you’ll get an error that it already exists):
    1. Go to Control Center
    1. Delete `GoogleFunctions` artifact if the same version already exists.
    1. Upload plugin by clicking on the circled green + button
    1. Pick the JAR file / JSON file created under target folder
    1. You’ll see a confirmation of the successful plugin upload

Reusable Plugins   reusable plugins for cloud data fusion cdf  cdap         Overview   overview         CheckPointReadAction  CheckPointUpdateAction   checkpointreadaction checkpointupdateaction             Dependencies   dependencies                 Setting up Firestore   setting up firestore                 Set Runtime Arguments   set runtime arguments         CopyTableAction   copytableaction         DropTableAction   droptableaction         TruncateTableAction   truncatetableaction     Putting it all together into a Pipeline   putting it all together into a pipeline       CheckPointReadAction   checkpointreadaction       TruncateTableAction   truncatetableaction       Database source   database source       BigQuery sink   bigquery sink       MergeLastUpdateTSAction   mergelastupdatetsaction       CheckPointUpdateAction   checkpointupdateaction     Building the CDF CDAP Plugin  JAR file   JSON file  and deploying into CDF CDAP   building the cdfcdap plugin jar file  json file and deploying into cdfcdap     Reusable Plugins for Cloud Data Fusion  CDF    CDAP     Overview The CDF CDAP plugins detailed below can be reused in the context of data pipelines   Let s say you run your incremental pipeline once every 5 minutes  When running an incremental pipeline  you have to filter the records by a specific field  e g    lastUpdateDateTime  of records   latest watermark value   buffer time  so it will sync the records that were updated since your last incremental sync  Subsequently  a merge and dedupe step is done to make sure only new updated are synced into the destination table       CheckPointReadAction    CheckPointUpdateAction     Plugin Description     Creates  reads  and updates checkpoints in incremental pull pipelines     CheckPointReadAction    reads checkpoints in Firestore DB and provides the data during runtime as environment variable   CheckPointUpdateAction    updates checkpoints in Firestore DB  i e   creates a new document and stores maximum update date   time  from BQ so the next run it can use this checkpoint value to filter records that were added since then   For now these plugins only support timestamp values   in the future  integer values can potentially be added      Dependencies       Setting up Firestore  1  Setup Firestore DB     1  Firestore is used to store   read checkpoints  which is used in your incremental pipelines  1  Create a collection with a document from the parent path       1  Collection ID   PIPELINE CHECKPOINTS      1  Document ID   INCREMENTAL DEMO     image  img 1 create pipeline checkpoint collection png   1  Create a collection under Parent path   PIPELINE CHECKPOINTS INCREMENTAL DEMO      1  Collection ID   CHECKPOINT      1  Document ID  just accept what was provided initially         1  Field  1             1  Note                  1  Set to maximum timestamp from destination  BQ table                  1  Set to minimum timestamp if running for the first time from source  e g   SQL server table               1  Field name   CREATED TIMESTAMP              1  Field type   string              1  Date and time   2020 05 08 17 21 01           1  Field  2             1  Note  enter the current time in timestamp format             1  Field name  CREATED TIMESTAMP             1  Field type  timestamp             1  Date and time  25 08 2020  15 49    image  img 2 create check point collection png        Set Runtime Arguments  Before running the pipeline  add the  lastWatermarkValue  variable as runtime argument  on Pipeline Studio view  click on drop down arrow for Run button  and set the value   0      image  img 3 runtime arguments png   CheckpointReadAction will populate lastWatermarkValue with the CHECKPOINT VALUE  from Firestore  lastWatermarkValue runtime argument will be used as parameter of the import query of the Database Source in a subsequent step       sql SELECT   FROM test WHERE last update datetime      latestWatermarkValue        BigQuery   actual destination table name  this is where max checkpoint is taken from   i e   max timestamp     Use Case     This plugin can be used at the beginning of an incremental CDAP data pipeline to read the checkpoint value from the last sync    Let s say you run your pipeline once every 5 minutes  When running an incremental pipeline  you have to filter the records by a specific field  timestamp   current date   current date  3    it is doing merge and dedupe even though we are processing the same records to make sure duplicate records are not in the destination table    CheckPointReadAction    reads checkpoints in Firestore DB and provides the data during runtime as environment variable   CheckPointUpdateAction    updates checkpoints in Firestore DB  i e   creates a new document and stores maximum update date   time  from BQ so the next run it can use this checkpoint value to filter records that were added since then   For now these plugins only support timestamp values   in the future  integer values can potentially be added      CheckpointReadAction  plugin requires the following config properties        Label   plugin label name     Specify the collection name in firestore DB  Name of the Collection     Specify the document name to read the checkpoint details  Provide the document name specified in the Collection     Buffer time to add to checkpoint value   Note  in Minutes   Number of minutes that need to be subtracted from the Firestore collection value     Project  project ID     Key path  Service account key file path to communicate with the Firestore DB     Please see the following screenshot for example       image  img 4 checkpoint read action plugin ui png      CheckpointUpdateAction  plugin requires the following configuration        Label   plugin label name     Specify the collection name in firestore DB  Name of the Collection     Specify the document name to read the checkpoint details  Provide the document name specified in the Collection     Dataset name where incremental pull table exists  Big Query Dataset name     Table name that needs incremental pull  Big Query table name     Specify the checkpoint column from incremental pull table     Project  project ID     Key path  Service account key file path to communicate with the Firestore DB     Please see the below screenshot for example       image  img 5 checkpoint update action plugin ui png       CopyTableAction     Plugin description     Copies the BigQuery table from staging to destination at the end of the pipeline run  A new table is created if it doesn t exist  Otherwise  if the table exists  the plugin replaces the existing BigQuery destination table with data from staging     Use case     This is applicable in the CDAP data pipelines which do the full import scan the data from source system to BigQuery     Dependencies     Destination dataset    bq dataset    Destination table    bq table    Source dataset    bq dataset batch staging  Source table    bq table      CopyTableAction  plugin requires the following configuration        Label  plugin label name     Key path  Service account key file path to call the Big Query API     Project ID  GCP project ID     Dataset  Big Query dataset name     Table Name  Big Query table name     Please see the following screenshot for example       image  img 6 copy table action ui png       DropTableAction     Plugin Description     Drops a BigQuery table in the beginning of the pipeline runs     Use Case     Useful to drop staging tables     Dependencies     Requires BQ table to drop to exist     Drop table action plugin requires the following configuration        Label   plugin label name     Key path  Service account key file path to call the Big Query API     Project ID  GCP project ID     Dataset  Big Query dataset name     Table Name  Big Query table name   Please see the following screenshot for example configuration     image  img 7 drop table action ui png       TruncateTableAction     Plugin Description     Truncates a BigQuery table when we set pipelines to restore the data from source     Use Case     Applicable in restoring data pipelines from source     TruncateTable action plugin requires the following configuration        Label   plugin label name     Key path  Service account key file path to call the Big Query API     Project ID  GCP project ID     Dataset  Big Query dataset name     Table Name  Big Query table name     Please see the following screenshot for example configuration       image  img 8 truncate table action ui png      Putting it all together into a Pipeline   CheckPointReadAction     TruncateTableAction    Database   BigQuery    MergeLastUpdateTSAction     CheckPointUpdateAction   What does the pipeline do   1   CheckPointReadAction    reads latest checkpoint from Firestore 1   TruncateTableAction    truncate the records in the log table 1  Database Source  imports data from the source 1  BigQuery Sink   exports data into BigQuery from previous step  database source  1   MergeLastUpdateTSAction     merge based on timestamp and the update column list  columns to keep in the merge          Note  Alternatively  you can use   BigQueryExecute   https   github com data integrations google cloud blob develop src main java io cdap plugin gcp bigquery action BigQueryExecute java  action to do a Merge  1   CheckPointUpdateAction    update checkpoint in Firestore from the max record lastUpdateTimestamp in BigQuery      Successful run of Incremental Pipeline    image  img 9 successful run incremental v2 pipeline png       Runtime arguments  set latestWatermarkValue to 0    image  img 9 set runtime arugments latestWatermarkValue png        CheckPointReadAction     Label       CheckPointReadAction     Specify the document name to read the checkpoint details        INCREMENTAL DEMO    Buffer time to add to checkpoint value   Note  in Minutes       1    project       pso cdf plugins 287518     serviceFilePath      auto detect    Screenshot       image  img 10 checkpoint read action ui pipeline parameters png        TruncateTableAction     Label     TruncateTableAction     Key path       auto detect    ProjectId         pso cdf plugins 287518     Dataset      bq dataset     Table name       bq table LOG     image  img 11 truncate table action ui pipeline parameters png       Database source    Label     Database    Reference Name      test    Plugin Name      sqlserver42    Plugin Type     jdbc    Connection String     jdbc sqlserver    fill in IP address of database server   db port  databaseName main user  fill in user  password  fill in password      Import Query       sql SELECT   FROM test WHERE last update datetime      latestWatermarkValue          image  img 12 database source ui pipeline parameters png        BigQuery sink    Label     BigQuery    Reference Name      bq table sink    Project ID      pso cdf plugins 287518     Dataset       bq dataset     Table      write to a temporary table  e g   bq table LOG   bq table LOG     Service Account File Path     auto detect    Schema      image  img 13 bigquery sink ui pipeline parameters png        MergeLastUpdateTSAction     Label       MergeLastUpdateTSAction     Key path      auto detect    Project ID       pso cdf plugins 287518     Dataset name      bq dataset     Table name       bq table     Primary key list      id    Update columns list      id name last update datetime    image  img 14 merge last update ts action png        CheckPointUpdateAction     Label       CheckPointUpdateAction     Specify the collection name in firestore DB      PIPELINE CHECKPOINTS    Specify the document name to read the checkpoint details      INCREMENTAL DEMO    Dataset name where incremental pull table exists       bq dataset     Table name that needs incremental pull       bq table     Specify the checkpoint column from incremental pull table      last update datetime    serviceFilePath     auto detect    project      pso cdf plugins 287518     image  img 15 checkpoint update action ui pipeline parameters png      Building the CDF CDAP Plugin  JAR file   JSON file  and deploying into CDF CDAP  This plugin requires Java JDK1 8 and maven   1  To build the CDAP   CDF plugin jar  execute the following command on the root     bash mvn clean compile package      2  You will find the generated JAR file and JSON file under target folder     1   GoogleFunctions 1 6 jar     1   GoogleFunctions 1 6 json       1  Deploy  GoogleFunctions 1 6 jar  and  GoogleFunctions 1 6 json  into CDF CDAP  note that if you have the same version already deployed then you ll get an error that it already exists       1  Go to Control Center     1  Delete  GoogleFunctions  artifact if the same version already exists      1  Upload plugin by clicking on the circled green   button       1  Pick the JAR file   JSON file created under target folder     1  You ll see a confirmation of the successful plugin upload


GCP The GitLab Kubernetes Agent KAS is a tool that helps you deploy your code to a Google Kubernetes Engine GKE cluster using GitOps practices It does this by using a YAML configuration file that you create in your repository It handles automated deployment of services after the images have been built by a CI CD pipeline from source code The flow diagram below illustrate the relationship between the two processes Note that in this example and is the same in production it can be hosted in different GitLab repos This example deploys a dummy image to decouple the Application CI CD step Brief explanation Terraform for Deploying a KAS agent in a GKE cluster This repository provides Terraform code for deploying a KAS agent in a GKE cluster to connect it with a Gitlab repository to automatically deploy manage and monitor your cloud native solutions using GitOps practices This creates resources in your cluster to deploy an agent that communicates with Gitlab to synchronize deployments Here is a to a guide that explains the manual steps for making this configuration and an overview of the solution More resource links are provided in section The Gitlab agent created in the Gitlab project is connected to the agentk service running in the cluster When there is a change in the manifest file describing the deployment the agent in Gitlab pulls the changes and invokes the connected service running in the GKE cluster to apply the declarative changes to the Kubernetes resources The Kubernetes Service Account federating the deployment is governed by the RBAC policies with limited permissions to the cluster configmap and the product namespaces

Answer:
# Terraform for Deploying a KAS agent in a GKE cluster
This repository provides Terraform code for deploying a KAS agent in a GKE cluster, to connect it with a Gitlab repository to automatically deploy, manage, and monitor your cloud-native solutions using GitOps practices. This creates resources in your cluster to deploy an agent that communicates with Gitlab to synchronize deployments. Here is a [link](https://about.gitlab.com/blog/2021/09/10/setting-up-the-k-agent/) to a guide that explains the manual steps for making this configuration and an overview of the solution. More resource links are provided in [this](#references-and-public-docs) section.

## Brief explanation
The GitLab Kubernetes Agent (KAS) is a tool that helps you deploy your code to a Google Kubernetes Engine (GKE) cluster using GitOps practices. It does this by using a YAML configuration file that you create in your repository. It handles automated deployment of services after the images have been built by a CI/CD pipeline from source code. The flow diagram below illustrate the relationship between the two processes. Note that in this example `Manifest Repository` and `Agent configuration repository` is the same -- in production it can be hosted in different GitLab repos. This example deploys a dummy `nginx` image to decouple the Application CI/CD step.

![Gitlab KAS flow diagram](docs/gitlab-kas-flow-diag.png)

The Gitlab agent created in the Gitlab project is connected to the agentk service running in the cluster. When there is a change in the manifest file describing the deployment, the agent in Gitlab pulls the changes and invokes the connected service running in the GKE cluster to apply the declarative changes to the Kubernetes resources. The Kubernetes Service Account federating the deployment is governed by the RBAC policies with limited permissions to the cluster configmap and the product namespaces.

![KAS Agent connection](docs/kas-agent-connection.png)

## How to setup
- Create a new project in Gitlab
- [Create a personal token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) in Gitlab, and set environment variable `GITLAB_TOKEN` to be used my Terraform
```
export GITLAB_TOKEN=<token value>
```
Note that in production you'd want to use a [*Runner authentication token* or a *CI/CD job token*](https://docs.gitlab.com/ee/security/token_overview.html#runner-authentication-tokens-also-called-runner-tokens) depending on your IaC pipeline strategy.
- Setup a [GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster) that can egress to public internet to communicate with a Gitlab agent.
- Use a personal or service account that has `roles/container.admin` permission.
- Create directory `manifests/` in your Gitlab project
- Create a tfvars file from the [sample](./terraform.tfvars.sample) file
- Run terraform init and apply to deploy resources
```
terraform init
terraform apply
```
- Add yaml file in `/manifests` directory (sample is provided below) to add a deployment, and commit the change.
- Ensure your namespace for the product has the correct deployment

## Useful commands
```
# Get namespaces. Check that 2 namespaces for product and gitlab-agent exists
kubectl get ns

# Check pods gitlab kas namespace
kubectl get pods -n <gitlab kas namespace name>

# Check pods product namespace
kubectl get pods -n <product namespace name>

# Check logs in kas agent pod for synchronization
kubectl logs <kas agent pod name> -n <gitlab kas namespace name>
```

## Resources created
See [terraform-docs.md](./terraform-docs.md) for details.
Here is a summary:

**Gitlab Resources**
- Agent instance in Gitlab project to poll configuration changes in deployment manifests
- Agent config file in the Gitlab project based on template in `templates/agent-config.yaml.tpl`

**Product K8s Resources**
- Sample prouct namespace

**Agentk K8s Resources**
- Namespace for agentk image in the cluster
- Deploy KAS agent client through Helm chart
- Kubernetes Service Account used by agent to manage deployments in product namespace
- RBAC roles for KSA to read/write configMap of cluster
- RBAC roles for KSA to read/write any k8s resources in product namespace

## Considerations
- Setup remote backend in provider
- Configure service account with appropriate permissions in providers.tf
- This example assumes SaaS offering of Gitlab KAS server is used "wss://kas.gitlab.com". For a self-managed server, the endpoint will be different
- The product namespace is created here for giving an example. Remove it for implementation.
- Networking and proxy settings between cluster and Gitlab agent can be configured in the helm chart value. Documentation reference is provided in `main.tf`.
- This solution assumes that the product namespaces are created separately.Before adding a manifest YAML file for a new namespace ensure that the namespace is created.
- Host the agentk image in an Artifact Registry and set the reference to that in the variables rather than pulling it from Gitlab's registry.

## Variables to configure (Example)
```
project_id = "gitlab-kas-gke"
cluster_name = "gitlab-kas-agent-cluster"
cluster_location = "us-central1-c"
gitlab_repo_name = "<user/org>/test-gitlab-kas-gke"
product_name = "test-kas"
agentk_image_tag = "v15.9.0-rc1"
```

## Sample deployment.yaml
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-test
  namespace: test-kas # Make sure this matches the product's namespace
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
```

## [References and public docs](#references-and-public-docs)
- [Using GitOps with a Kubernetes cluster](https://docs.gitlab.com/ee/user/clusters/agent/gitops.html)
- [How to deploy the GitLab Agent for Kubernetes with limited permissions](https://about.gitlab.com/blog/2021/09/10/setting-up-the-k-agent/)<a name="helpful">
- [Troubleshooting](https://docs.gitlab.com/ee/user/clusters/agent/troubleshooting.html)
- [Installing the agent for Kubernetes](https://docs.gitlab.com/ee/user/clusters/agent/install)
- [Working with the agent for Kubernetes](https://docs.gitlab.com/ee/user/clusters/agent/work_with_agent.html)

Terraform for Deploying a KAS agent in a GKE cluster This repository provides Terraform code for deploying a KAS agent in a GKE cluster  to connect it with a Gitlab repository to automatically deploy  manage  and monitor your cloud native solutions using GitOps practices  This creates resources in your cluster to deploy an agent that communicates with Gitlab to synchronize deployments  Here is a  link  https   about gitlab com blog 2021 09 10 setting up the k agent   to a guide that explains the manual steps for making this configuration and an overview of the solution  More resource links are provided in  this   references and public docs  section      Brief explanation The GitLab Kubernetes Agent  KAS  is a tool that helps you deploy your code to a Google Kubernetes Engine  GKE  cluster using GitOps practices  It does this by using a YAML configuration file that you create in your repository  It handles automated deployment of services after the images have been built by a CI CD pipeline from source code  The flow diagram below illustrate the relationship between the two processes  Note that in this example  Manifest Repository  and  Agent configuration repository  is the same    in production it can be hosted in different GitLab repos  This example deploys a dummy  nginx  image to decouple the Application CI CD step     Gitlab KAS flow diagram  docs gitlab kas flow diag png   The Gitlab agent created in the Gitlab project is connected to the agentk service running in the cluster  When there is a change in the manifest file describing the deployment  the agent in Gitlab pulls the changes and invokes the connected service running in the GKE cluster to apply the declarative changes to the Kubernetes resources  The Kubernetes Service Account federating the deployment is governed by the RBAC policies with limited permissions to the cluster configmap and the product namespaces     KAS Agent connection  docs kas agent connection png      How to setup   Create a new project in Gitlab    Create a personal token  https   docs gitlab com ee user profile personal access tokens html  in Gitlab  and set environment variable  GITLAB TOKEN  to be used my Terraform     export GITLAB TOKEN  token value      Note that in production you d want to use a   Runner authentication token  or a  CI CD job token   https   docs gitlab com ee security token overview html runner authentication tokens also called runner tokens  depending on your IaC pipeline strategy    Setup a  GKE cluster  https   cloud google com kubernetes engine docs how to creating a zonal cluster  that can egress to public internet to communicate with a Gitlab agent     Use a personal or service account that has  roles container admin  permission    Create directory  manifests   in your Gitlab project   Create a tfvars file from the  sample    terraform tfvars sample  file   Run terraform init and apply to deploy resources     terraform init terraform apply       Add yaml file in   manifests  directory  sample is provided below  to add a deployment  and commit the change    Ensure your namespace for the product has the correct deployment     Useful commands       Get namespaces  Check that 2 namespaces for product and gitlab agent exists kubectl get ns    Check pods gitlab kas namespace kubectl get pods  n  gitlab kas namespace name     Check pods product namespace kubectl get pods  n  product namespace name     Check logs in kas agent pod for synchronization kubectl logs  kas agent pod name   n  gitlab kas namespace name          Resources created See  terraform docs md    terraform docs md  for details  Here is a summary     Gitlab Resources     Agent instance in Gitlab project to poll configuration changes in deployment manifests   Agent config file in the Gitlab project based on template in  templates agent config yaml tpl     Product K8s Resources     Sample prouct namespace    Agentk K8s Resources     Namespace for agentk image in the cluster   Deploy KAS agent client through Helm chart   Kubernetes Service Account used by agent to manage deployments in product namespace   RBAC roles for KSA to read write configMap of cluster   RBAC roles for KSA to read write any k8s resources in product namespace     Considerations   Setup remote backend in provider   Configure service account with appropriate permissions in providers tf   This example assumes SaaS offering of Gitlab KAS server is used  wss   kas gitlab com   For a self managed server  the endpoint will be different   The product namespace is created here for giving an example  Remove it for implementation    Networking and proxy settings between cluster and Gitlab agent can be configured in the helm chart value  Documentation reference is provided in  main tf     This solution assumes that the product namespaces are created separately Before adding a manifest YAML file for a new namespace ensure that the namespace is created    Host the agentk image in an Artifact Registry and set the reference to that in the variables rather than pulling it from Gitlab s registry      Variables to configure  Example      project id    gitlab kas gke  cluster name    gitlab kas agent cluster  cluster location    us central1 c  gitlab repo name     user org  test gitlab kas gke  product name    test kas  agentk image tag    v15 9 0 rc1          Sample deployment yaml     apiVersion  apps v1 kind  Deployment metadata    name  nginx deployment test   namespace  test kas   Make sure this matches the product s namespace   labels      app  nginx spec    replicas  3   selector      matchLabels        app  nginx   template      metadata        labels          app  nginx     spec        containers          name  nginx         image  nginx 1 14 2         ports            containerPort  80          References and public docs   references and public docs     Using GitOps with a Kubernetes cluster  https   docs gitlab com ee user clusters agent gitops html     How to deploy the GitLab Agent for Kubernetes with limited permissions  https   about gitlab com blog 2021 09 10 setting up the k agent   a name  helpful      Troubleshooting  https   docs gitlab com ee user clusters agent troubleshooting html     Installing the agent for Kubernetes  https   docs gitlab com ee user clusters agent install     Working with the agent for Kubernetes  https   docs gitlab com ee user clusters agent work with agent html


GCP Getting Started Pub Sub topic to be ingested elsewhere This example illustrates how to use the DLP api in a Cloud Function to redact Redacting Sensitive Data Using the DLP API following These instructions will walk you through setting up your environment to do the sensitive data from log exports The scrubbed logs will then be posted to a

Answer:
# Redacting Sensitive Data Using the DLP API

This example illustrates how to use the DLP api in a Cloud Function to redact
sensitive data from log exports. The scrubbed logs will then be posted to a
Pub/Sub topic to be ingested elsewhere.

## Getting Started

These instructions will walk you through setting up your environment to do the
following:

* Export logs to a Pub/Sub log export
* Deploy a Cloud Function that subscribes to the log export
* Write scrubbed logs to a Pub/Sub topic.

### Prerequisites

Ensure that you have the [Google Cloud SDK](https://cloud.google.com/sdk/install) installed and authenticated to the project you want
to deploy the example to.

### Enable Required APIs

The Cloud Functions, Pub/Sub, and DLP APIs will all need to be enabled for this
example to work properly.

```
gcloud services enable cloudfunctions pubsub dlp.googleapis.com
```

### Pub/Sub

Pub/Sub will be used to facilitate the transfer of logs from Stackdriver to the
Cloud Function for processing.  Once the logs are scrubbed, they will be sent to
another Pub/Sub topic for final consumption.

Define the Pub/Sub topic and subscription names
```
export LOG_EXPORT_TOPIC_NAME=log-export

export DLP_SCRUBBED_TOPIC_NAME=scrubbed-log-export

export LOG_EXPORT_NAME=log-export-destination

export DLP_SCRUBBED_SUBSCRIPTION_NAME=scrubbed-log-export-subscription
```

Create the Pub/Sub topics and subscription
```
gcloud pubsub topics create $LOG_EXPORT_TOPIC_NAME

gcloud pubsub topics create $DLP_SCRUBBED_TOPIC_NAME

gcloud pubsub subscriptions create $DLP_SCRUBBED_SUBSCRIPTION_NAME \
 --topic $DLP_SCRUBBED_TOPIC_NAME
```

### Stackdriver Log Export

A log export will be created in Stackdriver that sends all "global" logs to the
first Pub/Sub topic we created.

Create the export.
```
export PROJECT_ID=$(gcloud config get-value project)

gcloud logging sinks create $LOG_EXPORT_NAME \
  $LOG_EXPORT_TOPIC_NAME pubsub.googleapis.com/projects/$PROJECT_ID/topics/$LOG_EXPORT_TOPIC_NAME \
  --log-filter resource.type="global"
```

Give Pub/Sub topic writer permissions to the service account being used for the
log export.
```
export SERVICE_ACCOUNT=$(gcloud logging sinks describe $LOG_EXPORT_NAME --format="value(writerIdentity)")

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member $SERVICE_ACCOUNT \
  --role roles/pubsub.publisher
```

### Cloud Function

Cloud Functions will be used to call the DLP API to scrub the log content, then post the output to a new Pub/Sub topic.

Clone the professional services repo.
```
git clone https://github.com/GoogleCloudPlatform/professional-services.git
```

Deploy the Cloud Function.
```
gcloud functions deploy dlp-log-scrubber \
  --runtime python37 \
  --trigger-topic $LOG_EXPORT_TOPIC_NAME \
  --entry-point process_log_entry \
  --set-env-vars OUTPUT_TOPIC_NAME=$DLP_SCRUBBED_TOPIC_NAME \
  --source professional-services/examples/dlp/cloud_function_example/cloud_function/.
```

Wait a few minutes for the Cloud Function to deploy, then you can proceed to test.

### Testing the DLP API

Now that we have set up our Stackdriver log exports, Pub/Sub, and Cloud Function we can proceed to test the DLP API.

First, write a log entry to Stackdriver that will be picked up by the log export we created.
```
gcloud logging write my-test-log \
  '{ "message": "user: test_email@test.com, visa: 1111-2222-3333-4444, DOB: 01/22/2019"}' \
  --payload-type=json
```

You will see the log entry under Logging > Logs if you view the "Global" log type

![Stackdriver Log Screenshot](img/stackdriver_log_img.png)

Now that we have written the log to Stackdriver, the log export to Pub/Sub should have triggered our Cloud Function. To verify, we can change the log viewer to filter for our Cloud Function logs.

![Stackdriver Cloud Function Screenshot](img/stackdriver_cloud_function_log.png)

Finally, lets grab the Cloud Function output from the Pub/Sub subscription we created
```
gcloud pubsub subscriptions pull --auto-ack $DLP_SCRUBBED_SUBSCRIPTION_NAME
```

The output will show the same log message with the email address, visa card number, and date of birth removed.

``"jsonPayload": {"message": "user: [EMAIL_ADDRESS], visa: [CREDIT_CARD_NUMBER], DOB: [DATE_OF_BIRTH]"}``

This example demonstrates the identification and replacement of a small number of data types (EMAIL_ADDRESS, CREDIT_CARD_NUMBER, and DATE_OF_BIRTH), however, the DLP API supports many more which can be found [here](https://cloud.google.com/dlp/docs/infotypes-reference)

Redacting Sensitive Data Using the DLP API  This example illustrates how to use the DLP api in a Cloud Function to redact sensitive data from log exports  The scrubbed logs will then be posted to a Pub Sub topic to be ingested elsewhere      Getting Started  These instructions will walk you through setting up your environment to do the following     Export logs to a Pub Sub log export   Deploy a Cloud Function that subscribes to the log export   Write scrubbed logs to a Pub Sub topic       Prerequisites  Ensure that you have the  Google Cloud SDK  https   cloud google com sdk install  installed and authenticated to the project you want to deploy the example to       Enable Required APIs  The Cloud Functions  Pub Sub  and DLP APIs will all need to be enabled for this example to work properly       gcloud services enable cloudfunctions pubsub dlp googleapis com          Pub Sub  Pub Sub will be used to facilitate the transfer of logs from Stackdriver to the Cloud Function for processing   Once the logs are scrubbed  they will be sent to another Pub Sub topic for final consumption   Define the Pub Sub topic and subscription names     export LOG EXPORT TOPIC NAME log export  export DLP SCRUBBED TOPIC NAME scrubbed log export  export LOG EXPORT NAME log export destination  export DLP SCRUBBED SUBSCRIPTION NAME scrubbed log export subscription      Create the Pub Sub topics and subscription     gcloud pubsub topics create  LOG EXPORT TOPIC NAME  gcloud pubsub topics create  DLP SCRUBBED TOPIC NAME  gcloud pubsub subscriptions create  DLP SCRUBBED SUBSCRIPTION NAME      topic  DLP SCRUBBED TOPIC NAME          Stackdriver Log Export  A log export will be created in Stackdriver that sends all  global  logs to the first Pub Sub topic we created   Create the export      export PROJECT ID   gcloud config get value project   gcloud logging sinks create  LOG EXPORT NAME      LOG EXPORT TOPIC NAME pubsub googleapis com projects  PROJECT ID topics  LOG EXPORT TOPIC NAME       log filter resource type  global       Give Pub Sub topic writer permissions to the service account being used for the log export      export SERVICE ACCOUNT   gcloud logging sinks describe  LOG EXPORT NAME   format  value writerIdentity     gcloud projects add iam policy binding  PROJECT ID       member  SERVICE ACCOUNT       role roles pubsub publisher          Cloud Function  Cloud Functions will be used to call the DLP API to scrub the log content  then post the output to a new Pub Sub topic   Clone the professional services repo      git clone https   github com GoogleCloudPlatform professional services git      Deploy the Cloud Function      gcloud functions deploy dlp log scrubber       runtime python37       trigger topic  LOG EXPORT TOPIC NAME       entry point process log entry       set env vars OUTPUT TOPIC NAME  DLP SCRUBBED TOPIC NAME       source professional services examples dlp cloud function example cloud function        Wait a few minutes for the Cloud Function to deploy  then you can proceed to test       Testing the DLP API  Now that we have set up our Stackdriver log exports  Pub Sub  and Cloud Function we can proceed to test the DLP API   First  write a log entry to Stackdriver that will be picked up by the log export we created      gcloud logging write my test log         message    user  test email test com  visa  1111 2222 3333 4444  DOB  01 22 2019          payload type json      You will see the log entry under Logging   Logs if you view the  Global  log type    Stackdriver Log Screenshot  img stackdriver log img png   Now that we have written the log to Stackdriver  the log export to Pub Sub should have triggered our Cloud Function  To verify  we can change the log viewer to filter for our Cloud Function logs     Stackdriver Cloud Function Screenshot  img stackdriver cloud function log png   Finally  lets grab the Cloud Function output from the Pub Sub subscription we created     gcloud pubsub subscriptions pull   auto ack  DLP SCRUBBED SUBSCRIPTION NAME      The output will show the same log message with the email address  visa card number  and date of birth removed      jsonPayload     message    user   EMAIL ADDRESS   visa   CREDIT CARD NUMBER   DOB   DATE OF BIRTH       This example demonstrates the identification and replacement of a small number of data types  EMAIL ADDRESS  CREDIT CARD NUMBER  and DATE OF BIRTH   however  the DLP API supports many more which can be found  here  https   cloud google com dlp docs infotypes reference


GCP Anthos Service Mesh ASM for multiple GKE clusters using Terraform Here are several reference documents if you encounter an issue when following the instructions below Pod traffic security scanning using ASM Docker and Google Artifact Registry GAR Documentation Multi Cluster ASM on Private Clusters Contents Connecting GKE clusters and ASM to an external database

Answer:
# Contents
- [Multi-Cluster ASM on Private Clusters](./infrastructure): Anthos Service Mesh (ASM) for multiple GKE clusters, using Terraform
- [Twistlock PoC](./twistlock): Pod traffic security scanning, using ASM, Docker and Google Artifact Registry (GAR)
- [Cloud SQL for PostgreSQL PoC](./postgres): Connecting GKE clusters and ASM to an external database

# Multi-Cluster ASM on Private Clusters

## Documentation

Here are several reference documents if you encounter an issue when following the instructions below:

- [Installing ASM using Anthos CLI](https://cloud.google.com/service-mesh/docs/gke-anthos-cli-existing-cluster)
- [Installing ASM using IstioCtl](https://cloud.google.com/service-mesh/docs/gke-install-existing-cluster)
- [Adding clusters to an Athos Service Mesh](https://cloud.google.com/service-mesh/docs/gke-install-multi-cluster)

## Description

In [Adding clusters to an Athos Service Mesh](https://cloud.google.com/service-mesh/docs/gke-install-multi-cluster), it shows how to federate service meshes of two Anthos **public** clusters. However, it misses a key instruction to open the firewall for the service port to the remote cluster. So, your final test of HelloWorld might not work.

This sample builds on the topic of Google's Anthos Service Mesh official installation documents, and adds instructions on how to federate two private clusters, which is more likely in real world environments.

As illustrated in the diagram below, we will create a VPC with three subnets. Two subnets are for private clusters, and one for GCE servers. So, we illustrate using a bastion server to access private clusters as in a real environment.

![NetworkImage](./asm-private-multiclusters-intranet.png)

The clusters are not accessible from an external network. Users can only log into the bastion server via an IAP tunnel to gain access to this VPC. A firewall rule is built to allow IAP tunneling into the GCE subnet (Subnet C) only. For the bastion server in Subnet C to access Kubernetes APIs of both private clusters, Subnet C's CIDR range is added to the "_GKE Control Plane Authorized Network_" of both clusters. This is illustrated as blue lines and yellow underscore lines in the diagram above.

Also, in order for both clusters to access the service mesh (Istiod) and service deployed on the other cluster, we need to do the following:
- The pod CIDR range of one cluster must be added to the "_GKE Control Plane Authorized Network_" of the other cluster. This enables one cluster to ping _istiod_ on the other cluster.
- The firewall needs to be open for one cluster's pod CIDR to access the service port on the other cluster. In this sample, it is port 5000 used by the HelloWord testing application. Because the invocation of service is bidirectional in HelloWorld testing application, we will add firewall rules for each direction.

The infrastructure used in this sample is coded in Terraform scripts. The ASM installation steps are coded in a Shell script.

## Prerequisites

As mentioned in [Add GKE clusters to Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/gke-install-multi-cluster), there are several prerequisites.

This guide assumes that you have:

- [A Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).
- [A Cloud Billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account).

Also, the multi-cluster configuration has these requirements for the clusters in it:

- All clusters must be on the same network.

  **NOTE:** ASM 1.7 does [not support multiple networks](https://cloud.google.com/service-mesh/docs/supported-features#platform_environment), even peered ones.

- If you join clusters that are not in the same project, they must be installed using the `asm-gcp-multiproject` profile and the clusters must be in a shared VPC configuration together on the same network. In addition, we recommend that you have one project to host the shared VPC, and two service projects for creating clusters. For more information, see [Setting up clusters with Shared VPC](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc).

In this sample, we create two private clusters in different subnets of the same VPC in the same project, and enable clusters to communicate to each other's API server.

## How to set up and run this sample

### Build Infrastructure

1. Create a GCP project.
2. Create a VPC in GCP project.
3. Create a subnet in the VPC.
4. Create a VM in the subnet. This will be the bastion server to simulate an intranet access to GKE clusters.
   - This step is now done by Terraform, in file [infrastructure/bastion.tf](./infrastructure/bastion.tf)
   - The Bastion host is used for interaction with the GKE clusters
   - For this demo, we ran Terraform from a local machine, not from the Bastion host
   - **Note:** you will have to manually [create a Google Cloud firewall rule](https://cloud.google.com/vpc/docs/using-firewalls), to allow connection to the bastion server via SSH (port 22). We did not automate this for security reasons.

5. Set up Git on your local machine, then clone this Github sample. Also clone this Github sample onto the bastion server

6. Set up [Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html) on your local machine, so you will be able to build infrastructure.

7. On your local machine, update the corresponding parameters for your project.
   - In ``vars.sh``, check to see whether you need to update `CLUSTER1_LOCATION`,`CLUSTER1_CLUSTER_NAME`, `CLUSTER1_CLUSTER_CTX`, `CLUSTER2_LOCATION`, `CLUSTER2_CLUSTER_NAME`, `CLUSTER2_CLUSTER_CTX`.
   - In [infrastructure/terraform.example.tfvars](./infrastructure/terraform.example.tfvars), rename this file to terraform.tfvars and update "project_id" and "billing_account".
   - In [infrastructure/shared.tf](./infrastructure/shared.tf), check whether you need to update "project_prefix" and "region".
   - **[OPTIONAL]** In the locals section of _infrastructure/shared.tf_, update CIDR ranges for bastion_cidr and existing_vpc if you need to.
   - Source _vars.sh_ to set up basic environment variables.

     ```
     source vars.sh
     ```

8. If you want to run Terraform in your own workspace, create a `backend.tf` file from _infrastructure/backend.tf_tmpl_, and update your Terraform workspace information in this file.

9. Under "_infrastructure_" directory, run
   - terraform init

     ```
     terraform init
     ```

   - terraform plan

     ```
     terraform plan -out output.tftxt
     ```

     **NOTE:** You may get an error that the Compute Engine API has not been used before in the project. In this case please [manually enable the Compute Engine API](https://console.cloud.google.com/apis/library/compute.googleapis.com)

     ```
     Error: Error when reading or editing GCE default service account: googleapi: Error 403: Compute Engine API has not been used in project XXXXXXXXXXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
     ```

   - terraform apply

     ```
     terraform apply output.tftxt
     ```

   - If Terraform completes without error, you should have VPC, NAT, a bastion server, two private clusters and firewall rules. Please check all artifacts in GCP Console.

10. SSH onto the bastion server.
11. Make sure you have the following tools installed:
    - The Cloud SDK (the gcloud command-line tool)
    - The standard command-line tools: awk, curl, grep, sed, sha256sum, and tr
    - git
    - kpt
    - kubectl
    - jq

### Install ASM

1. On bastion server, go to this source code directory, then source _vars.sh_

   **NOTE:** Make sure you manually [create a Google Cloud firewall rule](https://cloud.google.com/vpc/docs/using-firewalls) to allow SSH connections to your bastion server over port 22

   ```
   cd asm-private-multiclusters-intranet
   source vars.sh
   ```

2. Source _scripts/main.sh_
   ```
   source scripts/main.sh
   ```
3. Run install_asm_mesh
   ```
   install_asm_mesh
   ```

   or, you can run the commands in install_asm_mesh step by step manually
   ```
   # Navigate to your working directory. Binaries will be downloaded to this directory.
   cd ${WORK_DIR}

   # Set up K8s config and context
   set_up_credential ${CLUSTER1_CLUSTER_NAME} ${CLUSTER1_LOCATION} ${CLUSTER1_CLUSTER_CTX} ${TF_VAR_project_id}
   set_up_credential ${CLUSTER2_CLUSTER_NAME} ${CLUSTER2_LOCATION} ${CLUSTER2_CLUSTER_CTX} ${TF_VAR_project_id}

   # Download ASM Installer
   download_asm_installer ${ASM_MAJOR_VER} ${ASM_MINOR_VER}

   #Install ASM
   install_asm ${CLUSTER1_CLUSTER_NAME} ${CLUSTER1_LOCATION} ${TF_VAR_project_id}
   install_asm ${CLUSTER2_CLUSTER_NAME} ${CLUSTER2_LOCATION} ${TF_VAR_project_id}

   # Register clusters
   grant_role_to_connect_agent ${TF_VAR_project_id}
   register_cluster ${CLUSTER1_CLUSTER_CTX} ${CLUSTER1_LOCATION}
   register_cluster ${CLUSTER2_CLUSTER_CTX} ${CLUSTER2_LOCATION}

   # Add clusters to mesh
   cross_cluster_service_secret ${CLUSTER1_CLUSTER_NAME} ${CLUSTER1_CLUSTER_CTX} ${CLUSTER2_CLUSTER_CTX}
   cross_cluster_service_secret ${CLUSTER2_CLUSTER_NAME} ${CLUSTER2_CLUSTER_CTX} ${CLUSTER1_CLUSTER_CTX}

   ```

### Deploy test helloworld application

Run install_test_app
```
install_test_app
```

### Prepare for verification
```
export CTX1=$CLUSTER1_CLUSTER_CTX
export CTX2=$CLUSTER2_CLUSTER_CTX
```

Follow the instruction in "**Verify cross-cluster load balancing**" section of [Add clusters to an Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/gke-install-multi-cluster) to verify.

**Please Note:** You don't need to install `Helloworld` application, it has been installed for you already.

## Internal Load Balancer

Anthos ASM deploys ingress gateway using external load balancer by default. If we need to change the ingress gateway to be internal load balancer, we can use `--option` or `--custom-overlay` parameter along with out load balancer yaml (./istio-profiles/internal-load-balancer.yaml).

Please note that we need to specify out "targetPort" for https and http2 ports for current ASM version.

# Twistlock PoC
- Pod traffic security scanning, using ASM, Docker and Google Artifact Registry (GAR)
- Please see the [twistlock folder readme](./twistlock)

# Cloud SQL for PostgreSQL PoC
- Connecting GKE clusters and ASM to a database that is external database to the Kubernetes clusters
- Please see the [postgres folder readme](./postgres)

Contents    Multi Cluster ASM on Private Clusters    infrastructure   Anthos Service Mesh  ASM  for multiple GKE clusters  using Terraform    Twistlock PoC    twistlock   Pod traffic security scanning  using ASM  Docker and Google Artifact Registry  GAR     Cloud SQL for PostgreSQL PoC    postgres   Connecting GKE clusters and ASM to an external database    Multi Cluster ASM on Private Clusters     Documentation  Here are several reference documents if you encounter an issue when following the instructions below      Installing ASM using Anthos CLI  https   cloud google com service mesh docs gke anthos cli existing cluster     Installing ASM using IstioCtl  https   cloud google com service mesh docs gke install existing cluster     Adding clusters to an Athos Service Mesh  https   cloud google com service mesh docs gke install multi cluster      Description  In  Adding clusters to an Athos Service Mesh  https   cloud google com service mesh docs gke install multi cluster   it shows how to federate service meshes of two Anthos   public   clusters  However  it misses a key instruction to open the firewall for the service port to the remote cluster  So  your final test of HelloWorld might not work    This sample builds on the topic of Google s Anthos Service Mesh official installation documents  and adds instructions on how to federate two private clusters  which is more likely in real world environments    As illustrated in the diagram below  we will create a VPC with three subnets  Two subnets are for private clusters  and one for GCE servers  So  we illustrate using a bastion server to access private clusters as in a real environment      NetworkImage    asm private multiclusters intranet png   The clusters are not accessible from an external network  Users can only log into the bastion server via an IAP tunnel to gain access to this VPC  A firewall rule is built to allow IAP tunneling into the GCE subnet  Subnet C  only  For the bastion server in Subnet C to access Kubernetes APIs of both private clusters  Subnet C s CIDR range is added to the   GKE Control Plane Authorized Network   of both clusters  This is illustrated as blue lines and yellow underscore lines in the diagram above   Also  in order for both clusters to access the service mesh  Istiod  and service deployed on the other cluster  we need to do the following    The pod CIDR range of one cluster must be added to the   GKE Control Plane Authorized Network   of the other cluster  This enables one cluster to ping  istiod  on the other cluster     The firewall needs to be open for one cluster s pod CIDR to access the service port on the other cluster  In this sample  it is port 5000 used by the HelloWord testing application  Because the invocation of service is bidirectional in HelloWorld testing application  we will add firewall rules for each direction    The infrastructure used in this sample is coded in Terraform scripts  The ASM installation steps are coded in a Shell script           Prerequisites  As mentioned in  Add GKE clusters to Anthos Service Mesh  https   cloud google com service mesh docs gke install multi cluster   there are several prerequisites    This guide assumes that you have      A Cloud project  https   cloud google com resource manager docs creating managing projects      A Cloud Billing account  https   cloud google com billing docs how to manage billing account    Also  the multi cluster configuration has these requirements for the clusters in it     All clusters must be on the same network       NOTE    ASM 1 7 does  not support multiple networks  https   cloud google com service mesh docs supported features platform environment   even peered ones     If you join clusters that are not in the same project  they must be installed using the  asm gcp multiproject  profile and the clusters must be in a shared VPC configuration together on the same network  In addition  we recommend that you have one project to host the shared VPC  and two service projects for creating clusters  For more information  see  Setting up clusters with Shared VPC  https   cloud google com kubernetes engine docs how to cluster shared vpc    In this sample  we create two private clusters in different subnets of the same VPC in the same project  and enable clusters to communicate to each other s API server         How to set up and run this sample      Build Infrastructure  1  Create a GCP project  2  Create a VPC in GCP project  3  Create a subnet in the VPC   4  Create a VM in the subnet  This will be the bastion server to simulate an intranet access to GKE clusters       This step is now done by Terraform  in file  infrastructure bastion tf    infrastructure bastion tf       The Bastion host is used for interaction with the GKE clusters      For this demo  we ran Terraform from a local machine  not from the Bastion host        Note    you will have to manually  create a Google Cloud firewall rule  https   cloud google com vpc docs using firewalls   to allow connection to the bastion server via SSH  port 22   We did not automate this for security reasons   5  Set up Git on your local machine  then clone this Github sample  Also clone this Github sample onto the bastion server  6  Set up  Terraform  https   learn hashicorp com terraform getting started install html  on your local machine  so you will be able to build infrastructure    7  On your local machine  update the corresponding parameters for your project        In   vars sh    check to see whether you need to update  CLUSTER1 LOCATION   CLUSTER1 CLUSTER NAME    CLUSTER1 CLUSTER CTX    CLUSTER2 LOCATION    CLUSTER2 CLUSTER NAME    CLUSTER2 CLUSTER CTX        In  infrastructure terraform example tfvars    infrastructure terraform example tfvars   rename this file to terraform tfvars and update  project id  and  billing account        In  infrastructure shared tf    infrastructure shared tf   check whether you need to update  project prefix  and  region            OPTIONAL    In the locals section of  infrastructure shared tf   update CIDR ranges for bastion cidr and existing vpc if you need to        Source  vars sh  to set up basic environment variables                       source vars sh           8  If you want to run Terraform in your own workspace  create a  backend tf  file from  infrastructure backend tf tmpl   and update your Terraform workspace information in this file   9  Under   infrastructure   directory  run      terraform init                      terraform init                      terraform plan                       terraform plan  out output tftxt                         NOTE    You may get an error that the Compute Engine API has not been used before in the project  In this case please  manually enable the Compute Engine API  https   console cloud google com apis library compute googleapis com                 Error  Error when reading or editing GCE default service account  googleapi  Error 403  Compute Engine API has not been used in project XXXXXXXXXXX before or it is disabled  Enable it by visiting https   console developers google com apis api compute googleapis com overview then retry  If you enabled this API recently  wait a few minutes for the action to propagate to our systems and retry   accessNotConfigured                       terraform apply                       terraform apply output tftxt                If Terraform completes without error  you should have VPC  NAT  a bastion server  two private clusters and firewall rules  Please check all artifacts in GCP Console   10  SSH onto the bastion server  11  Make sure you have the following tools installed        The Cloud SDK  the gcloud command line tool        The standard command line tools  awk  curl  grep  sed  sha256sum  and tr       git       kpt       kubectl       jq      Install ASM  1  On bastion server  go to this source code directory  then source  vars sh           NOTE    Make sure you manually  create a Google Cloud firewall rule  https   cloud google com vpc docs using firewalls  to allow SSH connections to your bastion server over port 22            cd asm private multiclusters intranet    source vars sh         2  Source  scripts main sh            source scripts main sh        3  Run install asm mesh           install asm mesh            or  you can run the commands in install asm mesh step by step manually             Navigate to your working directory  Binaries will be downloaded to this directory     cd   WORK DIR        Set up K8s config and context    set up credential   CLUSTER1 CLUSTER NAME    CLUSTER1 LOCATION    CLUSTER1 CLUSTER CTX    TF VAR project id     set up credential   CLUSTER2 CLUSTER NAME    CLUSTER2 LOCATION    CLUSTER2 CLUSTER CTX    TF VAR project id        Download ASM Installer    download asm installer   ASM MAJOR VER    ASM MINOR VER       Install ASM    install asm   CLUSTER1 CLUSTER NAME    CLUSTER1 LOCATION    TF VAR project id     install asm   CLUSTER2 CLUSTER NAME    CLUSTER2 LOCATION    TF VAR project id        Register clusters    grant role to connect agent   TF VAR project id     register cluster   CLUSTER1 CLUSTER CTX    CLUSTER1 LOCATION     register cluster   CLUSTER2 CLUSTER CTX    CLUSTER2 LOCATION        Add clusters to mesh    cross cluster service secret   CLUSTER1 CLUSTER NAME    CLUSTER1 CLUSTER CTX    CLUSTER2 CLUSTER CTX     cross cluster service secret   CLUSTER2 CLUSTER NAME    CLUSTER2 CLUSTER CTX    CLUSTER1 CLUSTER CTX               Deploy test helloworld application  Run install test app     install test app          Prepare for verification     export CTX1  CLUSTER1 CLUSTER CTX export CTX2  CLUSTER2 CLUSTER CTX      Follow the instruction in    Verify cross cluster load balancing    section of  Add clusters to an Anthos Service Mesh  https   cloud google com service mesh docs gke install multi cluster  to verify     Please Note    You don t need to install  Helloworld  application  it has been installed for you already      Internal Load Balancer  Anthos ASM deploys ingress gateway using external load balancer by default  If we need to change the ingress gateway to be internal load balancer  we can use    option  or    custom overlay  parameter along with out load balancer yaml    istio profiles internal load balancer yaml    Please note that we need to specify out  targetPort  for https and http2 ports for current ASM version      Twistlock PoC   Pod traffic security scanning  using ASM  Docker and Google Artifact Registry  GAR    Please see the  twistlock folder readme    twistlock     Cloud SQL for PostgreSQL PoC   Connecting GKE clusters and ASM to a database that is external database to the Kubernetes clusters   Please see the  postgres folder readme    postgres