GCP Summary PostgreSQL uses application level protocol negotiation for SSL connection Istio Proxy currently uses TCP level protocol negotiation so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL In this article we document how to reproduce this issue Auto Encrypt PostgreSQL SSL Connection Using Istio Proxy Sidecar Enforce SSL connection on Cloud SQL PostgreSQL instance Prerequisites Create client certificate and download client certificate client key and server certificate We will use them in the client container for testing without sidecar auto encryption and mount them into Istio Proxy sidecar for sidecar auto encryption

Answer:
# Auto Encrypt PostgreSQL SSL Connection Using Istio Proxy Sidecar

### Summary

PostgreSQL uses application-level protocol negotiation for SSL connection. Istio Proxy currently uses TCP-level protocol negotiation, so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL. In this article, we document how to reproduce this issue.

### Prerequisites

* Enforce SSL connection on Cloud SQL PostgreSQL instance.
* Create client certificate and download client certificate, client key and server certificate. We will use them in the client container for testing without sidecar auto-encryption, and mount them into Istio Proxy sidecar for sidecar auto-encryption.
* Add K8s node IPs to the Authorized Networks of PostgreSQL instance. Or, we can add "0.0.0.0/0" to allow client connection from any IP address for testing purpose.

### Build Container

Use the Dockerfile to build a testing PostgreSQL client container image. We package the certificates into Docker image just for initial connection testing. **They are not needed in client container for sidecar auto-encryption**.

### Test Direct SSL Connection

Deploy a PostgreSQL client without any sidecar. Please note that we turn off the sidecar inject in the YAML file even though we have label our namespace to have Istio sidecar auto inject.

```
kubectl apply -f postgres-plain.yaml -n <YOUR_NAMESPACE>
```

Run the follow command to make sure SSL connection works.
```
# Enter into postgres-plain Pod
kubectl exec -it deploy/postgres-plain -n <YOUR_NAMESPACE> -- /bin/bash

# Once in the Pod, run this psql command with SSL mode
  psql "sslmode=verify-ca sslrootcert=server-ca.pem \
      sslcert=client-cert.pem sslkey=client-key.pem \
      hostaddr=YOUR_POSTGRESQL_IP \
      port=5432 \
      user=YOUR_USERNAME dbname=YOUR_DB_NAME"

# Enter your password when it is prompted
```

You should see something like this:
```
psql (12.5, server 12.4)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
```

Now, run `psql` command in Non-SSL mode
```
psql "hostaddr=YOUR_POSTGRESQL_IP port=5432 user=YOUR_USERNAME dbname=YOUR_DB_NAME"
```

You should see the error message as below. This proves that Non-SSL connection doesn't work.
```
psql: error: FATAL:  connection requires a valid client certificate
FATAL:  pg_hba.conf rejects connection for host "35.235.65.143", user "postgres", database "postgres", SSL off
```

### Deploy Istio Proxy Sidecar

#### Create K8s secret for the certificates

We will mount our PostgreSQL certificates into Istio Proxy sidecar. In order to achieve this, we need to upload certificates as K8s secret.

```
kubectl create secret generic postgres-cert --from-file=certs/client-cert.pem  --from-file=certs/client-key.pem  --from-file=certs/server-ca.pem
```

#### Mount the certificates into Istio Proxy sidecar

We mount our PostgreSQL certificates into Istio Proxy sidecar via the following annotations.
```
        sidecar.istio.io/userVolume: '[{"name":"postgres-cert", "secret":{"secretName":"postgres-cert"}}]'
        sidecar.istio.io/userVolumeMount: '[{"name":"postgres-cert", "mountPath":"/etc/certs/postgres-cert", "re
adonly":true}]'
```

#### Configure Sidecar certificates

We use [Service Entry](https://istio.io/latest/docs/reference/config/networking/service-entry/) and [Destination
 Rule](https://istio.io/latest/docs/reference/config/networking/destination-rule/) to instruct Istio Proxy sidec
ar to auto encrypt network traffic with specified Redis host and port with **SIMPLE** TLS. The detailed comments
 can be found in `destination-rule.yaml` and `service-entry.yaml` source code.

Also, here is how we instruct Istio Proxy sidecar to use our certificates for encryption.

```
      clientCertificate: /etc/certs/postgres-cert/client-cert.pem
      privateKey: /etc/certs/postgres-cert/client-key.pem
      caCertificates: /etc/certs/postgres-cert/server-ca.pem
```

Deploy both YAML files to your namespace.
```
kubectl apply -f destination-rule.yaml -n <YOUR_NAMESPACE>

kubectl apply -f service-entry.yaml -n <YOUR_NAMESPACE>
```

#### Deploy PostgresSQL client with sidecar injection

Run this command to deploy PostgreSQL client with Istio Proxy sidecar inject
``
kubectl apply -f postgres-istio.yaml -n <YOUR_NAMESPACE>
``

#### Run `psql` command in Non-SSL mode
```
psql "hostaddr=YOUR_POSTGRESQL_IP port=5432 user=YOUR_USERNAME dbname=YOUR_DB_NAME"
```

You should be prompted for password.

You will see errors.

#### Look into the Istio Proxy log

You can read the logs in Cloud Logging. However, you may want to view sidecar log messages for the detailed network traffic information and errors with the following command.
```
kubectl logs deploy/postgres-istio -c istio-proxy -n <YOUR_NAMESPACE>
```

Auto Encrypt PostgreSQL SSL Connection Using Istio Proxy Sidecar      Summary  PostgreSQL uses application level protocol negotiation for SSL connection  Istio Proxy currently uses TCP level protocol negotiation  so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL  In this article  we document how to reproduce this issue       Prerequisites    Enforce SSL connection on Cloud SQL PostgreSQL instance    Create client certificate and download client certificate  client key and server certificate  We will use them in the client container for testing without sidecar auto encryption  and mount them into Istio Proxy sidecar for sidecar auto encryption    Add K8s node IPs to the Authorized Networks of PostgreSQL instance  Or  we can add  0 0 0 0 0  to allow client connection from any IP address for testing purpose       Build Container  Use the Dockerfile to build a testing PostgreSQL client container image  We package the certificates into Docker image just for initial connection testing    They are not needed in client container for sidecar auto encryption         Test Direct SSL Connection  Deploy a PostgreSQL client without any sidecar  Please note that we turn off the sidecar inject in the YAML file even though we have label our namespace to have Istio sidecar auto inject        kubectl apply  f postgres plain yaml  n  YOUR NAMESPACE       Run the follow command to make sure SSL connection works        Enter into postgres plain Pod kubectl exec  it deploy postgres plain  n  YOUR NAMESPACE      bin bash    Once in the Pod  run this psql command with SSL mode   psql  sslmode verify ca sslrootcert server ca pem         sslcert client cert pem sslkey client key pem         hostaddr YOUR POSTGRESQL IP         port 5432         user YOUR USERNAME dbname YOUR DB NAME     Enter your password when it is prompted      You should see something like this      psql  12 5  server 12 4  SSL connection  protocol  TLSv1 3  cipher  TLS AES 256 GCM SHA384  bits  256  compression  off  Type  help  for help       Now  run  psql  command in Non SSL mode     psql  hostaddr YOUR POSTGRESQL IP port 5432 user YOUR USERNAME dbname YOUR DB NAME       You should see the error message as below  This proves that Non SSL connection doesn t work      psql  error  FATAL   connection requires a valid client certificate FATAL   pg hba conf rejects connection for host  35 235 65 143   user  postgres   database  postgres   SSL off          Deploy Istio Proxy Sidecar       Create K8s secret for the certificates  We will mount our PostgreSQL certificates into Istio Proxy sidecar  In order to achieve this  we need to upload certificates as K8s secret       kubectl create secret generic postgres cert   from file certs client cert pem    from file certs client key pem    from file certs server ca pem           Mount the certificates into Istio Proxy sidecar  We mount our PostgreSQL certificates into Istio Proxy sidecar via the following annotations              sidecar istio io userVolume      name   postgres cert    secret    secretName   postgres cert              sidecar istio io userVolumeMount      name   postgres cert    mountPath    etc certs postgres cert    re adonly  true              Configure Sidecar certificates   We use  Service Entry  https   istio io latest docs reference config networking service entry   and  Destination  Rule  https   istio io latest docs reference config networking destination rule   to instruct Istio Proxy sidec ar to auto encrypt network traffic with specified Redis host and port with   SIMPLE   TLS  The detailed comments  can be found in  destination rule yaml  and  service entry yaml  source code    Also  here is how we instruct Istio Proxy sidecar to use our certificates for encryption              clientCertificate   etc certs postgres cert client cert pem       privateKey   etc certs postgres cert client key pem       caCertificates   etc certs postgres cert server ca pem      Deploy both YAML files to your namespace       kubectl apply  f destination rule yaml  n  YOUR NAMESPACE   kubectl apply  f service entry yaml  n  YOUR NAMESPACE            Deploy PostgresSQL client with sidecar injection  Run this command to deploy PostgreSQL client with Istio Proxy sidecar inject    kubectl apply  f postgres istio yaml  n  YOUR NAMESPACE           Run  psql  command in Non SSL mode     psql  hostaddr YOUR POSTGRESQL IP port 5432 user YOUR USERNAME dbname YOUR DB NAME       You should be prompted for password    You will see errors          Look into the Istio Proxy log  You can read the logs in Cloud Logging  However  you may want to view sidecar log messages for the detailed network traffic information and errors with the following command      kubectl logs deploy postgres istio  c istio proxy  n  YOUR NAMESPACE


GCP Summary PostgreSQL Auto SSL Connection Using Cloud SQL Proxy Because ASM Istio Proxy sidecar doesn t work with PostgreSQL SSL auto encryption we demonstrate how to use Cloud SQL Proxy to auto encrypt SSL connection with Cloud SQL PostgreSQL database in this article PostgreSQL uses application level protocol negotiation for SSL connection Istio Proxy currently uses TCP level protocol negotiation so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL Please follow the steps in to see the details of this issue Prerequisites

Answer:
# PostgreSQL Auto SSL Connection Using Cloud SQL Proxy

## Summary

PostgreSQL uses application-level protocol negotiation for SSL connection. Istio Proxy currently uses TCP-level protocol negotiation, so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL. Please follow the steps in [PostgreSQL Auto SSL Connection Problem Using Istio Sidecar](./Istio-Sidecar.md) to see the details of this issue.

Because ASM Istio Proxy sidecar doesn't work with PostgreSQL SSL auto encryption, we demonstrate how to use Cloud SQL Proxy to auto encrypt SSL connection with Cloud SQL PostgreSQL database in this article.

## Prerequisites

* Enforce SSL connection on Cloud SQL PostgreSQL instance.
* **We don't need certificates for Cloud SQL Proxy connection.** However, we will create client certificate and download client certificate, client key and server certificate for the purpose of initial SSL connection without sidecar auto-encryption. Instructions for downloading Cloud SQL for PostgreSQL certificates is on this page: [Configuring SSL/TLS certificates](https://cloud.google.com/sql/docs/postgres/configure-ssl-instance)
* Add K8s node IPs to the Authorized Networks of PostgreSQL instance. Or, we can add "0.0.0.0/0" to allow client connection from any IP address for testing purpose.

## Build Container

1. Download the `client-cert.pem`, `client-key.pem` and `server-ca.pem` certificates, using the instructions on [Configuring SSL/TLS certificates](https://cloud.google.com/sql/docs/postgres/configure-ssl-instance)

    **NOTE:** These certificates are not needed for connecting via the Cloud SQL Proxy

2. Use the [Dockerfile](./Dockerfile) to build a test PostgreSQL client container image.

    The certificates are packaged into the Docker image just for initial connection testing.

## Test Direct SSL Connection

1. Deploy a PostgreSQL client without any sidecar.

    ```
    kubectl apply -f postgres-plain.yaml -n sample
    ```

2. Run the follow command to make sure the SSL connection works.

    ```
    # Enter into postgres-plain Pod
    kubectl exec -it deploy/postgres-plain -n sample -- /bin/bash

    # Once in the Pod, run this psql command with SSL mode
      psql "sslmode=verify-ca sslrootcert=server-ca.pem \
          sslcert=client-cert.pem sslkey=client-key.pem \
          hostaddr=YOUR_POSTGRESQL_IP \
          port=5432 \
          user=YOUR_USERNAME dbname=YOUR_DB_NAME"

    # Enter your password when it is prompted
    ```

    You should see something like this:

    ```
    psql (12.5, server 12.4)
    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
    Type "help" for help.
    ```

3. Now, run `psql` command in Non-SSL mode

    ```
    psql "hostaddr=YOUR_POSTGRESQL_IP port=5432 user=YOUR_USERNAME dbname=YOUR_DB_NAME"
    ```

    You should see the below error message. This proves that a Non-SSL connection doesn't work.

    ```
    psql: error: FATAL:  connection requires a valid client certificate
    FATAL:  pg_hba.conf rejects connection for host "35.235.65.143", user "postgres", database "postgres", SSL off
    ```

## Deploy the Cloud SQL Proxy Sidecar

1. Create a Kubernetes Service Account

    - We have already deployed our GKE cluster with [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) enabled.
    - We will use a Kubernetes Service Account (KSA) binding to a Google Cloud Service Account (GSA) to simplify our Cloud Proxy sidecar deployment.
    - Create a KSA named "ksa-sqlproxy"

      ```
      kubectl apply -f service-account.yaml -n sample
      ```

2. Set up a Google Cloud Service Account

    - Set up a Google Cloud Service Account (or use an existing GSA).
    - Make sure that [Cloud SQL Client predefined role](https://cloud.google.com/sql/docs/mysql/project-access-control#roles) (roles/cloudsql.client) is granted to this GSA.
    - In the next step, we will create a new GSA `sql-client@${PROJECT_ID}.iam.gserviceaccount.com`.

3. Bind KSA to GSA

    ```
    export PROJECT_ID="$(gcloud config get-value project || ${GOOGLE_CLOUD_PROJECT})"

    gcloud iam service-accounts add-iam-policy-binding \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:${PROJECT_ID}.svc.id.goog[YOUR_NAMESPACE/ksa-sqlproxy]" \
      sql-client@${PROJECT_ID}.iam.gserviceaccount.com
    ```

4. Add annotation to the service account

    ```
    kubectl annotate serviceaccount \
      ksa-sqlproxy \
      iam.gke.io/gcp-service-account="sql-client@${PROJECT_ID}.iam.gserviceaccount.com" \
      -n YOUR_NAMESPACE
    ```

5. Deploy PostgreSQL client with Cloud SQL Proxy sidecar

    Take a look at the deployment YAML file, [postgres-cloudproxy.yaml](./postgres-cloudproxy.yaml). Please note the following two items:

      i. The "serviceAccountName: ksa-sqlproxy" entry for pod.

      - This pod will use this KSA to authenticate itself through Google Cloud IAM.
      - Remember that we don't need the certificate files.

      ii. The container entry for Cloud SQL Proxy.

    ```
    kubectl apply -f postgres-cloudproxy.yaml -n sample
    ```

6. Test out Cloud SQL Proxy sidecar

    - Run the following command to get into Postgres client container

      ```
      kubectl exec -it deploy/postgres-check -c postgres-check -n sample -- /bin/bash
      ```

    - Run `psql` command in Non-SSL mode

      ```
      psql "hostaddr=YOUR_POSTGRESQL_IP port=5432 user=YOUR_USERNAME dbname=YOUR_DB_NAME"
      ```

    You should be prompted for password, then you should be connected to your PostgreSQL database.

PostgreSQL Auto SSL Connection Using Cloud SQL Proxy     Summary  PostgreSQL uses application level protocol negotiation for SSL connection  Istio Proxy currently uses TCP level protocol negotiation  so Istio Proxy sidecar errors out during SSL handshake when it tries to auto encrypt connection with PostgreSQL  Please follow the steps in  PostgreSQL Auto SSL Connection Problem Using Istio Sidecar    Istio Sidecar md  to see the details of this issue   Because ASM Istio Proxy sidecar doesn t work with PostgreSQL SSL auto encryption  we demonstrate how to use Cloud SQL Proxy to auto encrypt SSL connection with Cloud SQL PostgreSQL database in this article      Prerequisites    Enforce SSL connection on Cloud SQL PostgreSQL instance      We don t need certificates for Cloud SQL Proxy connection    However  we will create client certificate and download client certificate  client key and server certificate for the purpose of initial SSL connection without sidecar auto encryption  Instructions for downloading Cloud SQL for PostgreSQL certificates is on this page   Configuring SSL TLS certificates  https   cloud google com sql docs postgres configure ssl instance    Add K8s node IPs to the Authorized Networks of PostgreSQL instance  Or  we can add  0 0 0 0 0  to allow client connection from any IP address for testing purpose       Build Container  1  Download the  client cert pem    client key pem  and  server ca pem  certificates  using the instructions on  Configuring SSL TLS certificates  https   cloud google com sql docs postgres configure ssl instance         NOTE    These certificates are not needed for connecting via the Cloud SQL Proxy  2  Use the  Dockerfile    Dockerfile  to build a test PostgreSQL client container image       The certificates are packaged into the Docker image just for initial connection testing      Test Direct SSL Connection  1  Deploy a PostgreSQL client without any sidecar               kubectl apply  f postgres plain yaml  n sample          2  Run the follow command to make sure the SSL connection works                 Enter into postgres plain Pod     kubectl exec  it deploy postgres plain  n sample     bin bash        Once in the Pod  run this psql command with SSL mode       psql  sslmode verify ca sslrootcert server ca pem             sslcert client cert pem sslkey client key pem             hostaddr YOUR POSTGRESQL IP             port 5432             user YOUR USERNAME dbname YOUR DB NAME         Enter your password when it is prompted              You should see something like this               psql  12 5  server 12 4      SSL connection  protocol  TLSv1 3  cipher  TLS AES 256 GCM SHA384  bits  256  compression  off      Type  help  for help           3  Now  run  psql  command in Non SSL mode              psql  hostaddr YOUR POSTGRESQL IP port 5432 user YOUR USERNAME dbname YOUR DB NAME               You should see the below error message  This proves that a Non SSL connection doesn t work               psql  error  FATAL   connection requires a valid client certificate     FATAL   pg hba conf rejects connection for host  35 235 65 143   user  postgres   database  postgres   SSL off             Deploy the Cloud SQL Proxy Sidecar  1  Create a Kubernetes Service Account        We have already deployed our GKE cluster with  Workload Identity  https   cloud google com kubernetes engine docs how to workload identity  enabled        We will use a Kubernetes Service Account  KSA  binding to a Google Cloud Service Account  GSA  to simplify our Cloud Proxy sidecar deployment        Create a KSA named  ksa sqlproxy                   kubectl apply  f service account yaml  n sample            2  Set up a Google Cloud Service Account         Set up a Google Cloud Service Account  or use an existing GSA          Make sure that  Cloud SQL Client predefined role  https   cloud google com sql docs mysql project access control roles   roles cloudsql client  is granted to this GSA        In the next step  we will create a new GSA  sql client   PROJECT ID  iam gserviceaccount com    3  Bind KSA to GSA              export PROJECT ID    gcloud config get value project      GOOGLE CLOUD PROJECT         gcloud iam service accounts add iam policy binding           role roles iam workloadIdentityUser           member  serviceAccount   PROJECT ID  svc id goog YOUR NAMESPACE ksa sqlproxy           sql client   PROJECT ID  iam gserviceaccount com          4  Add annotation to the service account              kubectl annotate serviceaccount         ksa sqlproxy         iam gke io gcp service account  sql client   PROJECT ID  iam gserviceaccount com           n YOUR NAMESPACE          5  Deploy PostgreSQL client with Cloud SQL Proxy sidecar      Take a look at the deployment YAML file   postgres cloudproxy yaml    postgres cloudproxy yaml   Please note the following two items         i  The  serviceAccountName  ksa sqlproxy  entry for pod           This pod will use this KSA to authenticate itself through Google Cloud IAM          Remember that we don t need the certificate files         ii  The container entry for Cloud SQL Proxy                kubectl apply  f postgres cloudproxy yaml  n sample          6  Test out Cloud SQL Proxy sidecar        Run the following command to get into Postgres client container                  kubectl exec  it deploy postgres check  c postgres check  n sample     bin bash                  Run  psql  command in Non SSL mode                      psql  hostaddr YOUR POSTGRESQL IP port 5432 user YOUR USERNAME dbname YOUR DB NAME                 You should be prompted for password  then you should be connected to your PostgreSQL database


GCP Cost Optimization Dashboard This repo contains SQL scripts for analyzing GCP Billing Recommendations data and also a guide to setup the Cost Optimization dashboard Introduction For sample dashboard The Cost Optimization dashboard builds on top of existing and adds following additional insights to the dashboard Compute Engine Insights Cloud Storage Insights

Answer:
# Cost Optimization Dashboard

This repo contains SQL scripts for analyzing GCP Billing, Recommendations data and also a guide to setup the Cost Optimization dashboard.
For sample dashboard [see here](https://datastudio.google.com/c/u/0/reporting/6cf564a4-9c94-4cfd-becd-b9c770ee7aa2/page/r34iB).

## Introduction
The Cost Optimization dashboard builds on top of existing [GCP billing dashboard](https://cloud.google.com/billing/docs/how-to/visualize-data) and adds following additional insights to the dashboard.
* Compute Engine Insights
* Cloud Storage Insights
* BigQuery Insights
* Cost Optimization Recommendations
* Etc.

Few key things to keep in mind before starting.
* [Recommendations data export](https://cloud.google.com/recommender/docs/bq-export/export-recommendations-to-bq) to bigquery is still in Preview(as of June 2021).
* Currently, no automation is available for this setup.

> *NOTE: Implementing this dashboard will incur additional BigQuery charges.*

## Prerequisites
The user running the steps in this guide should have ability to
* Create a GCP project and BQ datasets.
* Schedule BQ queries and data transfer jobs.
* Provision BQ access to the Datastudio dashboard.
* Fully qualified names of BQ tables. Example format: ```<project-id>.<dataset-name>.<table-name>```.

Also, Make sure the following resources are accessible.
* [billing_dashboard_view](https://datastudio.google.com/datasources/c7f4b535-3dd0-4aaa-bd2b-78f9718df9e2)
* [co_dashboard_view](https://datastudio.google.com/datasources/78dc2597-d8e7-40db-8fbb-f3b2c8271b6d)
* [co_recommendations_view](https://datastudio.google.com/datasources/c972e0f6-51e1-483c-8947-214b300d26a6)
* [GCP Cost Optimization Dashboard](https://datastudio.google.com/reporting/6cf564a4-9c94-4cfd-becd-b9c770ee7aa2)

> *NOTE: In case of permission issues with above links, please reach out to gcp-co-dashboard@google.com*

## Setup
The overall process of the setup is as follows, each step is outlined in detail below,
* Create a project and the required BigQuery datasets.
* Create data transfers jobs, and scheduled queries.
  * Initiate data transfer for billing data export into ```billing``` dataset.
  * Initiate data transfer for recommendations data export into ```recommender``` dataset.
  * Setup dashboard related functions, views and scheduled queries in ```dashboard``` dataset.
* Use pre-existing templates as a starting point to set up the DataStudio dashboard.

### Project and datasets
* Create a project to hold the BigQuery datasets for the dashboard
* Create the following datasets in the project. Please make sure that all datasets are created in the same region (eg: US). [See instructions](https://cloud.google.com/bigquery/docs/datasets#create-dataset) on how to create a dataset in BigQuery
  * ```billing```
  * ```recommender```
  * ```dashboard```

### Billing data exports to Bigquery
* In the project created above, enable export for both ‘Daily cost detail’ and ‘Pricing’ data tables to the ```billing``` dataset, by following the instructions [here](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup).
* Data availability

  > Your [BigQuery dataset](https://cloud.google.com/bigquery/docs/datasets-intro) only reflects Google Cloud usage and cost data incurred from the date you set up Cloud Billing export, and after. That is, Google Cloud billing data is not added retroactively, so you won't see Cloud Billing data from before you enable export.

### Recommendations data exports to Bigquery
* Export recommendations data to the ```recommender``` dataset, by following the instructions [here](https://cloud.google.com/recommender/docs/bq-export/export-recommendations-to-bq).
* Data availability

  > There is a one day delay after a transfer is first created before your requested organization is opted-in to exports, and recommendations for your organization are available for export. In the interim, you will see the message “Transfer deferred due to source data not being available”. This message may also be shown in case the data is not ready for export on future days - the transfers will automatically be rescheduled to check for source data at a later time.

* At this point the datasets would look something like below.

  ![](docs/image1.png)

### Cost Optimization data analysis scripts
This step involves setting up the following data analysis components.
* Required SQL functions
* Daily scheduled scripts for data analysis and aggregation

#### Common Functions
* Compose a new query and copy the SQL at [common_functions.sql](scripts/common_functions.sql).
* Execute the query to create some required functions in the ```dashboard``` dataset.
* This is how the dataset will look like after the above step.

  ![](docs/image2.png)

#### CO Billing Data Table
* Compose a new query and copy the SQL at [co_billing_data.sql](scripts/co_billing_data.sql).
* Replace ```<BILLING_EXPORT_TABLE>``` with the correct table name created at "Billing data export to Bigquery" step.
* Run the query and ensure it’s completed without errors
* Click ‘Schedule query -> Create new scheduled query’.

  ![](docs/image3.png)

* Fill the details as seen in the below screenshot.

  ![](docs/image4.png)

* Click ‘Schedule’ to create a scheduled query.

#### CO Pricing Data Table
* Compose a new query and copy the SQL at [co_pricing_data.sql](scripts/co_pricing_data.sql).
* Replace ```<PRICING_EXPORT_TABLE>``` with the correct table name created at "Billing data export to Bigquery" step.
* Click ‘Schedule query -> Create new scheduled query’.

  ![](docs/image3.png)

* Fill the details as seen in the below screenshot.

  [ ![](docs/image5.png)](docs/image5.png)

* Click ‘Schedule’ to create a scheduled query.

#### CO Recommendations Data Table
* Compose a new query and copy the SQL at [co_recommendations_data.sql](scripts/co_recommendations_data.sql).
* Click ‘Schedule query -> Create new scheduled query’.

  ![](docs/image3.png)

* Fill the details as seen in the below screenshot.

  ![](docs/image6.png)

* Click ‘Schedule’ to create a scheduled query.

#### Verify
* This is how the BQ scheduled queries screen will look like for CO queries after the above steps.

  ![](docs/image7.png)

* This is how the dataset will look like after the above step.

  ![](docs/image8.png)

## Dashboard
This step involves copying the template data sources and template dashboard report and making necessary changes.

### Data Sources
Below are the template data sources that are of interest.
* [billing_dashboard_view](https://datastudio.google.com/datasources/c7f4b535-3dd0-4aaa-bd2b-78f9718df9e2)
* [co_dashboard_view](https://datastudio.google.com/datasources/78dc2597-d8e7-40db-8fbb-f3b2c8271b6d)
* [co_recommendations_view](https://datastudio.google.com/datasources/c972e0f6-51e1-483c-8947-214b300d26a6)

For each of the above data sources
* Copy the data source

  ![](docs/image9.png)

* Change the name at the top left corner.
  * For Example, from "Copy of [EXTERNAL] xyz_view" to “xyz_view”.
* Click the ‘Edit Connection’ button if the Custom Query editor window does not show up after copying the data source in the above step.
* In the Billing Project Selector panel (middle panel), select the project created in the beginning of this guide.
* In the Query panel to the right
  * Wherever applicable, replace all occurrences of project name(anilgcp-co-dev) to the project created in the beginning of this guide.
    * Example: `anilgcp-co-dev.dashboard.co_billing_data` to `REPLACE_WITH_PROJECT_ID.dashboard.co_billing_data`
  * Wherever applicable, replace all occurrences of billing export table name to the correct table name.
    * Example: `anilgcp-co-dev.billing.gcp_billing_export_v1_xxxxxxxxxxxx` to `REPLACE_WITH_PROJECT_ID.billing.billing_export_table_name`
* Make sure all occurrences of fully qualified table names are enclosed within the backticks (`).
* Make sure ‘Enable date parameters’ is selected for both data sources from above steps.

  ![](docs/image10.png)

* Click "Reconnect".
* Review the confirmation, which should say "Allow parameter sharing?", and click "Allow".

  ![](docs/image11.png)

* Review the confirmation, which should say "No fields changed due to configuration change.", and click "Apply".

### Report
* Copy the [dashboard template](https://datastudio.google.com/reporting/6cf564a4-9c94-4cfd-becd-b9c770ee7aa2) by clicking the "Make a copy" button at the top right hand side, as shown below.

  ![](docs/image12.png)

* Point the "New Data Source" to the newly created data sources from the above steps, and click "Copy Report".

  ![](docs/image13.png)

* Change the name of the dashboard from “Copy of [EXTERNAL] GCP Cost Optimization Dashboard” to “GCP Cost Optimization Dashboard” or something similar.

## References and support
* For feedback and support reach out to your TAM

Cost Optimization Dashboard  This repo contains SQL scripts for analyzing GCP Billing  Recommendations data and also a guide to setup the Cost Optimization dashboard  For sample dashboard  see here  https   datastudio google com c u 0 reporting 6cf564a4 9c94 4cfd becd b9c770ee7aa2 page r34iB        Introduction The Cost Optimization dashboard builds on top of existing  GCP billing dashboard  https   cloud google com billing docs how to visualize data  and adds following additional insights to the dashboard    Compute Engine Insights   Cloud Storage Insights   BigQuery Insights   Cost Optimization Recommendations   Etc    Few key things to keep in mind before starting     Recommendations data export  https   cloud google com recommender docs bq export export recommendations to bq  to bigquery is still in Preview as of June 2021     Currently  no automation is available for this setup      span style  color red   NOTE  Implementing this dashboard will incur additional BigQuery charges    span       Prerequisites The user running the steps in this guide should have ability to    Create a GCP project and BQ datasets    Schedule BQ queries and data transfer jobs    Provision BQ access to the Datastudio dashboard    Fully qualified names of BQ tables  Example format      project id   dataset name   table name        Also  Make sure the following resources are accessible     billing dashboard view  https   datastudio google com datasources c7f4b535 3dd0 4aaa bd2b 78f9718df9e2     co dashboard view  https   datastudio google com datasources 78dc2597 d8e7 40db 8fbb f3b2c8271b6d     co recommendations view  https   datastudio google com datasources c972e0f6 51e1 483c 8947 214b300d26a6     GCP Cost Optimization Dashboard  https   datastudio google com reporting 6cf564a4 9c94 4cfd becd b9c770ee7aa2      span style  color red   NOTE  In case of permission issues with above links  please reach out to gcp co dashboard google com   span       Setup The overall process of the setup is as follows  each step is outlined in detail below    Create a project and the required BigQuery datasets    Create data transfers jobs  and scheduled queries      Initiate data transfer for billing data export into    billing    dataset      Initiate data transfer for recommendations data export into    recommender    dataset      Setup dashboard related functions  views and scheduled queries in    dashboard    dataset    Use pre existing templates as a starting point to set up the DataStudio dashboard       Project and datasets   Create a project to hold the BigQuery datasets for the dashboard   Create the following datasets in the project  Please make sure that all datasets are created in the same region  eg  US    See instructions  https   cloud google com bigquery docs datasets create dataset  on how to create a dataset in BigQuery        billing           recommender           dashboard         Billing data exports to Bigquery   In the project created above  enable export for both  Daily cost detail  and  Pricing  data tables to the    billing    dataset  by following the instructions  here  https   cloud google com billing docs how to export data bigquery setup     Data availability      Your  BigQuery dataset  https   cloud google com bigquery docs datasets intro  only reflects Google Cloud usage and cost data incurred from the date you set up Cloud Billing export  and after  That is  Google Cloud billing data is not added retroactively  so you won t see Cloud Billing data from before you enable export       Recommendations data exports to Bigquery   Export recommendations data to the    recommender    dataset  by following the instructions  here  https   cloud google com recommender docs bq export export recommendations to bq     Data availability      There is a one day delay after a transfer is first created before your requested organization is opted in to exports  and recommendations for your organization are available for export  In the interim  you will see the message  Transfer deferred due to source data not being available   This message may also be shown in case the data is not ready for export on future days   the transfers will automatically be rescheduled to check for source data at a later time     At this point the datasets would look something like below         docs image1 png       Cost Optimization data analysis scripts This step involves setting up the following data analysis components    Required SQL functions   Daily scheduled scripts for data analysis and aggregation        Common Functions   Compose a new query and copy the SQL at  common functions sql  scripts common functions sql     Execute the query to create some required functions in the    dashboard    dataset    This is how the dataset will look like after the above step         docs image2 png         CO Billing Data Table   Compose a new query and copy the SQL at  co billing data sql  scripts co billing data sql     Replace     BILLING EXPORT TABLE     with the correct table name created at  Billing data export to Bigquery  step    Run the query and ensure it s completed without errors   Click  Schedule query    Create new scheduled query          docs image3 png     Fill the details as seen in the below screenshot         docs image4 png     Click  Schedule  to create a scheduled query         CO Pricing Data Table   Compose a new query and copy the SQL at  co pricing data sql  scripts co pricing data sql      Replace     PRICING EXPORT TABLE     with the correct table name created at  Billing data export to Bigquery  step    Click  Schedule query    Create new scheduled query          docs image3 png     Fill the details as seen in the below screenshot           docs image5 png   docs image5 png     Click  Schedule  to create a scheduled query         CO Recommendations Data Table   Compose a new query and copy the SQL at  co recommendations data sql  scripts co recommendations data sql     Click  Schedule query    Create new scheduled query          docs image3 png     Fill the details as seen in the below screenshot         docs image6 png     Click  Schedule  to create a scheduled query        Verify   This is how the BQ scheduled queries screen will look like for CO queries after the above steps         docs image7 png     This is how the dataset will look like after the above step         docs image8 png       Dashboard This step involves copying the template data sources and template dashboard report and making necessary changes       Data Sources Below are the template data sources that are of interest     billing dashboard view  https   datastudio google com datasources c7f4b535 3dd0 4aaa bd2b 78f9718df9e2     co dashboard view  https   datastudio google com datasources 78dc2597 d8e7 40db 8fbb f3b2c8271b6d     co recommendations view  https   datastudio google com datasources c972e0f6 51e1 483c 8947 214b300d26a6   For each of the above data sources    Copy the data source        docs image9 png     Change the name at the top left corner      For Example  from  Copy of  EXTERNAL  xyz view  to  xyz view     Click the  Edit Connection  button if the Custom Query editor window does not show up after copying the data source in the above step    In the Billing Project Selector panel  middle panel   select the project created in the beginning of this guide    In the Query panel to the right     Wherever applicable  replace all occurrences of project name anilgcp co dev  to the project created in the beginning of this guide        Example   anilgcp co dev dashboard co billing data  to  REPLACE WITH PROJECT ID dashboard co billing data      Wherever applicable  replace all occurrences of billing export table name to the correct table name        Example   anilgcp co dev billing gcp billing export v1 xxxxxxxxxxxx  to  REPLACE WITH PROJECT ID billing billing export table name    Make sure all occurrences of fully qualified table names are enclosed within the backticks        Make sure  Enable date parameters  is selected for both data sources from above steps         docs image10 png     Click  Reconnect     Review the confirmation  which should say  Allow parameter sharing    and click  Allow          docs image11 png     Review the confirmation  which should say  No fields changed due to configuration change    and click  Apply         Report   Copy the  dashboard template  https   datastudio google com reporting 6cf564a4 9c94 4cfd becd b9c770ee7aa2  by clicking the  Make a copy  button at the top right hand side  as shown below         docs image12 png     Point the  New Data Source  to the newly created data sources from the above steps  and click  Copy Report          docs image13 png     Change the name of the dashboard from  Copy of  EXTERNAL  GCP Cost Optimization Dashboard  to  GCP Cost Optimization Dashboard  or something similar       References and support   For feedback and support reach out to your TAM


GCP Copyright 2021 Google LLC All Rights Reserved you may not use this file except in compliance with the License http www apache org licenses LICENSE 2 0 You may obtain a copy of the License at Unless required by applicable law or agreed to in writing software Licensed under the Apache License Version 2 0 the License

Answer:
# Vertex AI Pipeline
This repository demonstrates end-to-end [MLOps process](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf)
using [Vertex AI](https://cloud.google.com/vertex-ai) platform
and [Smart Analytics](https://cloud.google.com/solutions/smart-analytics) technology capabilities.

In particular two general [Vertex AI Pipeline](https://cloud.google.com/vertex-ai/docs/pipelines)
templates has been provided:
- Training pipeline including:
  - Data processing
  - Custom model training
  - Model evaluation
  - Endpoint creation
  - Model deployment
  - Deployment testing
  - Model monitoring
- Batch-prediction pipeline including
  - Data processing
  - Batch prediction using deployed model

Note that besides Data processing being done using BigQuery, all other steps are build on top of
[Vertex AI](https://cloud.google.com/vertex-ai) platform capabilities.

### Dataset
The dataset used throughout the demonstration is
[Banknote Authentication Data Set](https://archive.ics.uci.edu/ml/datasets/banknote+authentication).
Data were extracted from images that were taken from genuine and forged banknote-like specimens.
For digitization, an industrial camera usually used for print inspection was used.
The final images have 400x 400 pixels. Due to the object lens and distance to the
investigated object gray-scale pictures with a resolution of about 660 dpi were gained.
Wavelet Transform tool were used to extract features from images.
Attribute Information:
1. variance of Wavelet Transformed image (continuous)
2. skewness of Wavelet Transformed image (continuous)
3. curtosis of Wavelet Transformed image (continuous)
4. entropy of image (continuous)
5. class (integer)

### Machine Learning Problem
Given the Banknote Authentication Data Set, a binary classification problem is adopted where
attribute `class` is chosen as label and the remaining attributes are used as features.

[LightGBM](https://github.com/microsoft/LightGBM), a gradient boosting framework that uses tree based
learning algorithms, is used to train the model for purpose of demonstrating
[custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training) and
[custom serving](https://cloud.google.com/vertex-ai/docs/predictions/use-custom-container)
capabilities of Vertex AI platform, which provide more native support for e.g. Tensorflow,
Pytorch, Scikit-Learn and Pytorch.

## Repository Structure

The repository contains the following:

```
.
├── components    : custom vertex pipeline components
├── images        : custom container images for training and serving
├── pipelines     : vertex ai pipeline definitions and runners
├── configs       : configurations for defining vertex ai pipeline
├── scripts       : scripts for runing local testing
└── notebooks     : notebooks used development and testing of vertex ai pipeline
```
In addition
- `build_components_cb.sh`: build all components defined in `components` folder using Cloud Build
- `build_images_cb.sh`: build custom images (training and serving) defined in `images` folder using Cloud Build
- `build_pipeline_cb.sh`: build training and batch-prediction pipeline defined in `pipelines` folder using Cloud Build

## Get Started

The end-to-end process of creating and running the training pipeline contains the following steps:

1. Setting up [MLOps environment](https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai/tree/main/provision) on Google Cloud.
2. Create an [Artifact Registry](https://cloud.google.com/artifact-registry) for your organization to manage container images
3. Develop the training and serving logic
4. Create the components required to build and run the pipeline
5. Prepare and consolidate the configurations of the various steps of the pipeline
6. Build the pipeline
7. Run and orchestrate the pipeline

### Create Artifact Registry
[Artifact Registry](https://cloud.google.com/artifact-registry)
is a single place for your organization to manage container images and language
packages (such as Maven and npm). It is fully integrated with Google Cloud’s tooling and
runtimes and comes with support for native artifact protocols. More importantly, it supports
regional and multi-regional repositories.

We have provided a helper script: `scripts/create_artifact_registry.sh`

### Develop Training and Serving Logic
Develop your machine learning program and then containerize them as demonstrated in `images`.
The requirements for writing training code can be found
[here](https://cloud.google.com/vertex-ai/docs/training/code-requirements) as well.
Note that custom serving image is not necessary if your choosen framework is supported by
[pre-built-container](https://cloud.google.com/vertex-ai/docs/predictions/pre-built-containers),
which are organized by machine learning (ML) framework and framework version,
provide HTTP prediction servers that you can use to serve predictions with minimal configuration

We have also provided helper scripts:
- `scripts/run_training_local.sh`: test the training program locally
- `scripts/run_serving_local.sh`: test the serving program locally
- `build_images_cb.sh`: build the images using Cloud Build service

#### Environment variables for special Cloud Storage directories
Vertex AI sets the following environment variables when it runs your training code:

- `AIP_MODEL_DIR`: a Cloud Storage URI of a directory intended for saving model artifacts.
- `AIP_CHECKPOINT_DIR`: a Cloud Storage URI of a directory intended for saving checkpoints.
- `AIP_TENSORBOARD_LOG_DIR`: a Cloud Storage URI of a directory intended for saving TensorBoard logs. See Using Vertex TensorBoard with custom training.

### Build Components
The following template custom components are provided:
- `components/data_process`: read BQ table, perform transformation in BQ and export to GCS
- `components/train_model`: launch custom (distributed) training job on Vertex AI platform
- `components/check_model_metrics`: check the metrics of a training job and verify whether it produces better model
- `components/create_endpoint`: create an endpoint on Vertex AI platform
- `components/deploy_model`: deployed a model artifact to a created endpoint on Vertex AI platform
- `components/test_endpoint`: call the endpoint of deployed model for verification
- `components/monitor_model`: track deployed model performance using Vertex Model Monitoring
- `components/batch_prediction`: launch batch prediction job on Vertex AI platform

We have also provided a helper script: `build_components_cb.sh`

### Build and Run Pipeline
The sample definition of pipelines are
- `pipelines/training_pipeline.py`
- `pipelines/batch_prediction_pipeline.py`

After compiled the training or batch-prediction pipeline, you may trigger the pipeline run
using the provided runner
- `pipelines/trainin_pipeline_runner.py`
- `pipelines/batch_prediction_pipeline_runner.py`

An example to run training pipeline using the runner
```shell
python training_pipeline_runner \
  --project_id "$PROJECT_ID" \
  --pipeline_region $PIPELINE_REGION \
  --pipeline_root $PIPELINE_ROOT \
  --pipeline_job_spec_path $PIPELINE_SPEC_PATH \
  --data_pipeline_root $DATA_PIPELINE_ROOT \
  --input_dataset_uri "$DATA_URI" \
  --training_data_schema ${DATA_SCHEMA} \
  --data_region $DATA_REGION \
  --gcs_data_output_folder $GCS_OUTPUT_PATH \
  --training_container_image_uri "$TRAIN_IMAGE_URI" \
  --train_additional_args $TRAIN_ARGS \
  --serving_container_image_uri "$SERVING_IMAGE_URI" \
  --custom_job_service_account $CUSTOM_JOB_SA \
  --hptune_region $PIPELINE_REGION \
  --hp_config_max_trials 30 \
  --hp_config_suggestions_per_request 5 \
  --vpc_network "$VPC_NETWORK" \
  --metrics_name $METRIC_NAME \
  --metrics_threshold $METRIC_THRESHOLD \
  --endpoint_machine_type n1-standard-4 \
  --endpoint_min_replica_count 1 \
  --endpoint_max_replica_count 2 \
  --endpoint_test_instances ${TEST_INSTANCE} \
  --monitoring_user_emails $MONITORING_EMAIL \
  --monitoring_log_sample_rate 0.8 \
  --monitor_interval 3600 \
  --monitoring_default_threshold 0.3 \
  --monitoring_custom_skew_thresholds $MONITORING_CONFIG \
  --monitoring_custom_drift_thresholds $MONITORING_CONFIG \
  --enable_model_monitoring True \
  --pipeline_schedule "0 2 * * *" \
  --pipeline_schedule_timezone "US/Pacific" \
  --enable_pipeline_caching
```

We have also provided helper scripts:
- `scripts/build_pipeline_spec.sh`: compile and build the pipeline specs locally
- `scripts/run_training_pipeline.sh`: create and run training Vertex AI Pipeline based on the specs
- `scripts/run_batch_prediction_pipeline.sh`: create and run batch-prediction Vertex AI Pipeline based on the specs
- `build_pipeline_spec_cb.sh`: compile and build the pipeline specs using Cloud Build service

### Some common parameters
|Field|Explanation|
|-----|-----|
|project_id|Your GCP project|
|pipeline_region|The region to run Vertex AI Pipeline|
|pipeline_root|The GCS buckets used for storing artifacts of your pipeline runs|
|data_pipeline_root|The GCS staging location for custom job|
|input_dataset_uri|Full URI of input dataset|
|data_region|Region of input dataset|

## Contributors
- [Shixin Luo](https://github.com/luotigerlsx)
- [Tommy Siu](https://github.com/tommysiu)
- [Nathan Faggian](https://github.com/nfaggian

Copyright 2021 Google LLC  All Rights Reserved   Licensed under the Apache License  Version 2 0  the  License    you may not use this file except in compliance with the License  You may obtain a copy of the License at        http   www apache org licenses LICENSE 2 0  Unless required by applicable law or agreed to in writing  software distributed under the License is distributed on an  AS IS  BASIS  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND  either express or implied  See the License for the specific language governing permissions and limitations under the License                                                                                Vertex AI Pipeline This repository demonstrates end to end  MLOps process  https   services google com fh files misc practitioners guide to mlops whitepaper pdf   using  Vertex AI  https   cloud google com vertex ai  platform  and  Smart Analytics  https   cloud google com solutions smart analytics  technology capabilities   In particular two general  Vertex AI Pipeline  https   cloud google com vertex ai docs pipelines   templates has been provided    Training pipeline including      Data processing     Custom model training     Model evaluation     Endpoint creation     Model deployment     Deployment testing     Model monitoring   Batch prediction pipeline including     Data processing     Batch prediction using deployed model  Note that besides Data processing being done using BigQuery  all other steps are build on top of  Vertex AI  https   cloud google com vertex ai  platform capabilities    p align  center        img src    training pipeline png  alt  Sample Training Pipeline  width  600      p       Dataset The dataset used throughout the demonstration is  Banknote Authentication Data Set  https   archive ics uci edu ml datasets banknote authentication   Data were extracted from images that were taken from genuine and forged banknote like specimens   For digitization  an industrial camera usually used for print inspection was used   The final images have 400x 400 pixels  Due to the object lens and distance to the  investigated object gray scale pictures with a resolution of about 660 dpi were gained   Wavelet Transform tool were used to extract features from images  Attribute Information  1  variance of Wavelet Transformed image  continuous  2  skewness of Wavelet Transformed image  continuous  3  curtosis of Wavelet Transformed image  continuous  4  entropy of image  continuous  5  class  integer       Machine Learning Problem Given the Banknote Authentication Data Set  a binary classification problem is adopted where  attribute  class  is chosen as label and the remaining attributes are used as features    LightGBM  https   github com microsoft LightGBM   a gradient boosting framework that uses tree based  learning algorithms  is used to train the model for purpose of demonstrating   custom training  https   cloud google com vertex ai docs training custom training  and  custom serving  https   cloud google com vertex ai docs predictions use custom container   capabilities of Vertex AI platform  which provide more native support for e g  Tensorflow  Pytorch  Scikit Learn and Pytorch       Repository Structure  The repository contains the following             components      custom vertex pipeline components     images          custom container images for training and serving     pipelines       vertex ai pipeline definitions and runners     configs         configurations for defining vertex ai pipeline     scripts         scripts for runing local testing      notebooks       notebooks used development and testing of vertex ai pipeline     In addition    build components cb sh   build all components defined in  components  folder using Cloud Build    build images cb sh   build custom images  training and serving  defined in  images  folder using Cloud Build    build pipeline cb sh   build training and batch prediction pipeline defined in  pipelines  folder using Cloud Build     Get Started  The end to end process of creating and running the training pipeline contains the following steps   1  Setting up  MLOps environment  https   github com GoogleCloudPlatform mlops with vertex ai tree main provision  on Google Cloud  2  Create an  Artifact Registry  https   cloud google com artifact registry  for your organization to manage container images 3  Develop the training and serving logic 4  Create the components required to build and run the pipeline 5  Prepare and consolidate the configurations of the various steps of the pipeline 6  Build the pipeline 7  Run and orchestrate the pipeline      Create Artifact Registry  Artifact Registry  https   cloud google com artifact registry  is a single place for your organization to manage container images and language  packages  such as Maven and npm   It is fully integrated with Google Cloud s tooling and  runtimes and comes with support for native artifact protocols  More importantly  it supports  regional and multi regional repositories   We have provided a helper script   scripts create artifact registry sh        Develop Training and Serving Logic Develop your machine learning program and then containerize them as demonstrated in  images    The requirements for writing training code can be found   here  https   cloud google com vertex ai docs training code requirements  as well  Note that custom serving image is not necessary if your choosen framework is supported by   pre built container  https   cloud google com vertex ai docs predictions pre built containers   which are organized by machine learning  ML  framework and framework version   provide HTTP prediction servers that you can use to serve predictions with minimal configuration  We have also provided helper scripts     scripts run training local sh   test the training program locally    scripts run serving local sh   test the serving program locally    build images cb sh   build the images using Cloud Build service       Environment variables for special Cloud Storage directories Vertex AI sets the following environment variables when it runs your training code      AIP MODEL DIR   a Cloud Storage URI of a directory intended for saving model artifacts     AIP CHECKPOINT DIR   a Cloud Storage URI of a directory intended for saving checkpoints     AIP TENSORBOARD LOG DIR   a Cloud Storage URI of a directory intended for saving TensorBoard logs  See Using Vertex TensorBoard with custom training       Build Components The following template custom components are provided     components data process   read BQ table  perform transformation in BQ and export to GCS    components train model   launch custom  distributed  training job on Vertex AI platform     components check model metrics   check the metrics of a training job and verify whether it produces better model    components create endpoint   create an endpoint on Vertex AI platform    components deploy model   deployed a model artifact to a created endpoint on Vertex AI platform    components test endpoint   call the endpoint of deployed model for verification    components monitor model   track deployed model performance using Vertex Model Monitoring    components batch prediction   launch batch prediction job on Vertex AI platform  We have also provided a helper script   build components cb sh       Build and Run Pipeline The sample definition of pipelines are    pipelines training pipeline py     pipelines batch prediction pipeline py   After compiled the training or batch prediction pipeline  you may trigger the pipeline run using the provided runner    pipelines trainin pipeline runner py     pipelines batch prediction pipeline runner py   An example to run training pipeline using the runner    shell python training pipeline runner       project id   PROJECT ID        pipeline region  PIPELINE REGION       pipeline root  PIPELINE ROOT       pipeline job spec path  PIPELINE SPEC PATH       data pipeline root  DATA PIPELINE ROOT       input dataset uri   DATA URI        training data schema   DATA SCHEMA        data region  DATA REGION       gcs data output folder  GCS OUTPUT PATH       training container image uri   TRAIN IMAGE URI        train additional args  TRAIN ARGS       serving container image uri   SERVING IMAGE URI        custom job service account  CUSTOM JOB SA       hptune region  PIPELINE REGION       hp config max trials 30       hp config suggestions per request 5       vpc network   VPC NETWORK        metrics name  METRIC NAME       metrics threshold  METRIC THRESHOLD       endpoint machine type n1 standard 4       endpoint min replica count 1       endpoint max replica count 2       endpoint test instances   TEST INSTANCE        monitoring user emails  MONITORING EMAIL       monitoring log sample rate 0 8       monitor interval 3600       monitoring default threshold 0 3       monitoring custom skew thresholds  MONITORING CONFIG       monitoring custom drift thresholds  MONITORING CONFIG       enable model monitoring True       pipeline schedule  0 2              pipeline schedule timezone  US Pacific        enable pipeline caching      We have also provided helper scripts     scripts build pipeline spec sh   compile and build the pipeline specs locally    scripts run training pipeline sh   create and run training Vertex AI Pipeline based on the specs    scripts run batch prediction pipeline sh   create and run batch prediction Vertex AI Pipeline based on the specs    build pipeline spec cb sh   compile and build the pipeline specs using Cloud Build service       Some common parameters  Field Explanation                 project id Your GCP project   pipeline region The region to run Vertex AI Pipeline   pipeline root The GCS buckets used for storing artifacts of your pipeline runs   data pipeline root The GCS staging location for custom job   input dataset uri Full URI of input dataset   data region Region of input dataset      Contributors    Shixin Luo  https   github com luotigerlsx     Tommy Siu  https   github com tommysiu     Nathan Faggian  https   github com nfaggian


GCP Instrumenting Web Applications End to End with Stackdriver and OpenTelemetry Stackdriver export to BigQuery and analyze the logs with SQL queries This tutorial demonstrates instrumenting a web application end to end from the JavaScript browser code that drives HTTP requests to a Node js backend that with OpenTelemetry and Stackdriver to run for a load test It shows how to can be run anywhere that you can run Node collect the instrumentation data from the browser and the server ship to browser to the backend application including logging monitoring and tracing The app is something like a simple web version of Apache Bench It includes

Answer:
# Instrumenting Web Applications End-to-End with Stackdriver and OpenTelemetry

This tutorial demonstrates instrumenting a web application end-to-end, from the
browser to the backend application, including logging, monitoring, and tracing
with OpenTelemetry and Stackdriver to run for a load test. It shows how to
collect the instrumentation data from the browser and the server, ship to
Stackdriver, export to BigQuery, and analyze the logs with SQL queries.
The app is something like a simple web version of Apache Bench. It includes
JavaScript browser code that drives HTTP requests to a Node.js backend that
can be run anywhere that you can run Node.

## Setup

To work through this example, you will need a GCP project. Follow these steps
1. [Select or create a GCP project](https://console.cloud.google.com/projectselector2/home/dashboard)
2. [Enable billing for your project](https://support.google.com/cloud/answer/6293499#enable-billing)
3. Clone the repo
git clone https://github.com/GoogleCloudPlatform/professional-services.git
4. Install Node.js
5. Install Go
6. Install Docker
7. Install the [Google Cloud SDK](https://cloud.google.com/sdk/install)

Clone this repository to your environment with the command

```shell
git clone https://github.com/GoogleCloudPlatform/professional-services.git
```

Change to this directory and set an environment variable to remember the
location

```shell
cd professional-services/examples/web-instrumentation
WI_HOME=`pwd`
```

Set Google Cloud SDK to the current project

```shell
export GOOGLE_CLOUD_PROJECT=[Your project]
gcloud config set project $GOOGLE_CLOUD_PROJECT
```

Enable the required services

```shell
gcloud services enable bigquery.googleapis.com \
  cloudbuild.googleapis.com \
  cloudtrace.googleapis.com \
  compute.googleapis.com \
  container.googleapis.com \
  containerregistry.googleapis.com \
  logging.googleapis.com \
  monitoring.googleapis.com
```

Install the JavaScript packages required by both the server and the browser:

```shell
npm install
```

## OpenTelemetry collector

Open up a new shell. In a new directory, clone the OpenTelemetry collector
contrib project, which contains the Stackdriver exporter

```shell
git clone https://github.com/open-telemetry/opentelemetry-collector-contrib
cd opentelemetry-collector-contrib
```

Build the binary executable

```shell
make otelcontribcol
```

Build the container

```shell
make docker-otelcontribcol
```

Tag it for Google Container Registry (GCR)

```shell
docker tag otelcontribcol gcr.io/$GOOGLE_CLOUD_PROJECT/otelcontribcol:latest
```

Push to GCR

```shell
docker push gcr.io/$GOOGLE_CLOUD_PROJECT/otelcontribcol
```

### Run the OpenTelemetry collector locally

If you are running on GKE only, you do do not need to do this step.
For running locally, the OpenTelemetry collector needs permissions and
credentials to write to Stackdriver.

Obtain user access credentials and store them for Application Default
Credentials

```shell
gcloud auth application-default login \
  --scopes="https://www.googleapis.com/auth/trace.append"
```

Install Go and run **This bit is not working for OT**

```shell
make otelcontribcol
bin/linux/otelcontribcol --config=$WI_HOME/conf/otservice-config.yaml
```

## Browser code

The browser code refers to ES2015 modules that need to be transpiled and bundled
with the help of webpack. Make sure that the variable `agentURL` in
`src\index.js` refers to localhost if running the OpenCensus agent locally or to
the external IP of the OpenCensus agent if running on Kubernetes.

In the original terminal, change to the browser code directory

```shell
cd browser
```

Install the browser dependencies

```shell
npm install
```

Compile the code

```shell
npm run build
```

## Run app locally

The app can be deployed locally. First change to the top level directory

```shell
cd ..
```

To run the app locally type

```shell
node ./src/app.js
```

Open your browser at http://localhost:8080

Fill in the test form to generate some load. You should see logs from both the
Node.js server and the browser code in the console. You should see traces in
Stackdriver.

## Deploy to Kubernetes

Create a cluster with 1 node and cluster autoscaling enabled

```shell
ZONE=us-central1-a
NAME=web-instrumentation
CHANNEL=regular # choose rapid if you want to live on the edge
gcloud beta container clusters create $NAME \
   --num-nodes 1 \
   --enable-autoscaling --min-nodes 1 --max-nodes 4 \
   --enable-basic-auth \
   --issue-client-certificate \
   --release-channel $CHANNEL \
   --zone $ZONE \
   --enable-stackdriver-kubernetes
```

Change the project id in file `k8s/ot-service.yaml` with the sed command

```shell
sed -i.bak "s//$GOOGLE_CLOUD_PROJECT/" k8s/ot-service.yaml
```

Deploy the OpenTelemetry service to the Kubernetes cluster

```shell
kubectl apply -f k8s/ot-service.yaml
```

Get the external IP. It might take a few minutes for the deployment to complete
and an IP address be allocated. You may have to execute the command below
several times before the EXTERNAL_IP shell variable is successfully set.

```shell
EXTERNAL_IP=$(kubectl get svc ot-service-service \
    -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
echo "External IP: $EXTERNAL_IP"
```

Edit the file `browser/src/index.js` changing the variable `collectorURL` to
refer to the external IP and port (80) of the agent with the following sed
command

```shell
sed -i.bak "s/localhost:55678/${EXTERNAL_IP}:80/" browser/src/index.js
```

Rebuild the web client

```shell
cd browser
npm run build
cd ..
```

### Build the app image

To deploy the image to the Google Container Registry (GCR), use the following
Cloud Build command

```shell
gcloud builds submit
```

Change the project id in file `k8s/deployment.yaml` with the sed command

```shell
sed -i.bak "s//$GOOGLE_CLOUD_PROJECT/" k8s/deployment.yaml
```

Deploy the app

```shell
kubectl apply -f k8s/deployment.yaml
```

Configure a service

```shell
kubectl apply -f k8s/service.yaml
```

Expose a service:

```shell
kubectl apply -f k8s/ingress.yaml
```

Check for the external IP

```shell
kubectl get ingress
```

It may take a few minutes for the service to be exposed through an external IP.

Navigate to the external IP. It should present a form that will allow you to
send a series of XML HTTP requests to the server. That will generate trace and
monitoring data.

## Log exports

Before running tests, consider first setting up
[log exports](https://cloud.google.com/logging/docs/export).
to BigQuery for more targeted log queries to analyse the results of your load
tests or production issues.

Create a BQ dataset to export the container logs to

```shell
bq --location=US mk -d \
  --description "Web instrumentation container log exports" \
  --project_id $GOOGLE_CLOUD_PROJECT \
  web_instr_container
```

Create a log export for the container logs

```shell
LOG_SA=$(gcloud logging sinks create web-instr-container-logs \
  bigquery.googleapis.com/projects/$GOOGLE_CLOUD_PROJECT/datasets/web_instr_container \
  --log-filter='resource.type="k8s_container" AND labels.k8s-pod/app="web-instrumentation"' \
  --format='value("writerIdentity")')
```

The identify of the logs writer service account is captured in the shell
variable `LOG_SA`. Give this service account write access to BigQuery

```shell
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
    --member $LOG_SA \
    --role roles/bigquery.dataEditor
```

Create a BQ dataset for the load balancer logs

```shell
bq --location=US mk -d \
  --description "Web instrumentation load balancer log exports" \
  --project_id $GOOGLE_CLOUD_PROJECT \
  web_instr_load_balancer
```

Repeat creation of the log sink for load balancer logs

```shell
LOG_SA=$(gcloud logging sinks create web-instr-load-balancer-logs \
  bigquery.googleapis.com/projects/$GOOGLE_CLOUD_PROJECT/datasets/web_instr_load_balancer \
  --log-filter='resource.type="http_load_balancer"' \
  --format='value("writerIdentity")')
```

Note that the service account id changes so that you need to note that anre
repeat the step for granting write access BigQuery

```shell
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
  --member $LOG_SA \
  --role roles/bigquery.dataEditor
```

## Running the load test

Now you are ready to run the load test. You might try opening two tabs in a
browser. In one tab generate a steady state load with request of say, 1 second
apart, to give a baseline. Then hit the app with a sudden spike to see how it
responds.

You can send a request from the command line with cURL

```shell
EXTERNAL_IP=[from kubectl get ingress command]
REQUEST_ID=1234567889 # A random number
# See W3C Trace Context for format
TRACE=00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
MILLIS=`date +%s%N | cut -b1-13`
curl "http://$EXTERNAL_IP/data/$REQUEST_ID" \
  -H "traceparent: $TRACE" \
  -H 'Content-Type: application/json' \
  --data-binary "{\"data\":{\"name\":\"Smoke test\",\"reqId\":$REQUEST_ID,\"tSent\":$MILLIS}}"
```

Check that you see the log for the request in the Log Viewer. After the log is
replicated to BigQuery, you should be able to query it with a query like below.
Note that the table name will be something like `requests_20200129`. A shell
variable is used to set the date below.

```shell
DATE=$(date -u +'%Y%m%d')
bq query --use_legacy_sql=false \
"SELECT
  httpRequest.status,
  httpRequest.requestUrl,
  timestamp
FROM web_instr_load_balancer.requests_${DATE}
ORDER BY timestamp DESC
LIMIT 10"
```

There are more queries in the Colab sheet
[load_test_analysis.ipynb](https://colab.research.google.com/github/googlecolab/GoogleCloudPlatform/professional-services/blob/web-instrumentation/examples/web-instrumentation/load_test_analysis.ipynb).

## Troubleshooting

Try the following, depending on where you encounter problems.

### Check project id

Check that you have set your project id in files `k8s/oc-service.yaml`,
`conf\ocagent-config.yaml`, and `deployment.yaml`.

### Tracing issues
You can use zPages to see the trace data sent to the OC agent. Find the
name of the POD running the agent:

```shell
kubectl get pods
```

Start port forwarding

```shell
kubectl port-forward $POD 55679:55679
```

Browse to the URL http://localhost:55679/debug/tracez

### Browser JavaScript
For trouble bundling the web client, see Webpack
[Getting Started](https://webpack.js.org/guides/getting-started/).

Instrumenting Web Applications End to End with Stackdriver and OpenTelemetry  This tutorial demonstrates instrumenting a web application end to end  from the browser to the backend application  including logging  monitoring  and tracing with OpenTelemetry and Stackdriver to run for a load test  It shows how to collect the instrumentation data from the browser and the server  ship to Stackdriver  export to BigQuery  and analyze the logs with SQL queries  The app is something like a simple web version of Apache Bench  It includes JavaScript browser code that drives HTTP requests to a Node js backend that can be run anywhere that you can run Node      Setup  To work through this example  you will need a GCP project  Follow these steps 1   Select or create a GCP project  https   console cloud google com projectselector2 home dashboard  2   Enable billing for your project  https   support google com cloud answer 6293499 enable billing  3  Clone the repo git clone https   github com GoogleCloudPlatform professional services git 4  Install Node js 5  Install Go 6  Install Docker 7  Install the  Google Cloud SDK  https   cloud google com sdk install   Clone this repository to your environment with the command     shell git clone https   github com GoogleCloudPlatform professional services git      Change to this directory and set an environment variable to remember the location     shell cd professional services examples web instrumentation WI HOME  pwd       Set Google Cloud SDK to the current project     shell export GOOGLE CLOUD PROJECT  Your project  gcloud config set project  GOOGLE CLOUD PROJECT      Enable the required services     shell gcloud services enable bigquery googleapis com     cloudbuild googleapis com     cloudtrace googleapis com     compute googleapis com     container googleapis com     containerregistry googleapis com     logging googleapis com     monitoring googleapis com       Install the JavaScript packages required by both the server and the browser      shell npm install         OpenTelemetry collector  Open up a new shell  In a new directory  clone the OpenTelemetry collector contrib project  which contains the Stackdriver exporter     shell git clone https   github com open telemetry opentelemetry collector contrib cd opentelemetry collector contrib      Build the binary executable     shell make otelcontribcol      Build the container     shell make docker otelcontribcol      Tag it for Google Container Registry  GCR      shell docker tag otelcontribcol gcr io  GOOGLE CLOUD PROJECT otelcontribcol latest      Push to GCR     shell docker push gcr io  GOOGLE CLOUD PROJECT otelcontribcol          Run the OpenTelemetry collector locally  If you are running on GKE only  you do do not need to do this step  For running locally  the OpenTelemetry collector needs permissions and credentials to write to Stackdriver   Obtain user access credentials and store them for Application Default Credentials     shell gcloud auth application default login       scopes  https   www googleapis com auth trace append       Install Go and run   This bit is not working for OT       shell make otelcontribcol bin linux otelcontribcol   config  WI HOME conf otservice config yaml         Browser code  The browser code refers to ES2015 modules that need to be transpiled and bundled with the help of webpack  Make sure that the variable  agentURL  in  src index js  refers to localhost if running the OpenCensus agent locally or to the external IP of the OpenCensus agent if running on Kubernetes   In the original terminal  change to the browser code directory     shell cd browser      Install the browser dependencies     shell npm install      Compile the code     shell npm run build         Run app locally  The app can be deployed locally  First change to the top level directory     shell cd         To run the app locally type     shell node   src app js      Open your browser at http   localhost 8080  Fill in the test form to generate some load  You should see logs from both the Node js server and the browser code in the console  You should see traces in Stackdriver      Deploy to Kubernetes  Create a cluster with 1 node and cluster autoscaling enabled     shell ZONE us central1 a NAME web instrumentation CHANNEL regular   choose rapid if you want to live on the edge gcloud beta container clusters create  NAME        num nodes 1        enable autoscaling   min nodes 1   max nodes 4        enable basic auth        issue client certificate        release channel  CHANNEL        zone  ZONE        enable stackdriver kubernetes      Change the project id in file  k8s ot service yaml  with the sed command     shell sed  i bak  s   GOOGLE CLOUD PROJECT   k8s ot service yaml      Deploy the OpenTelemetry service to the Kubernetes cluster     shell kubectl apply  f k8s ot service yaml      Get the external IP  It might take a few minutes for the deployment to complete and an IP address be allocated  You may have to execute the command below several times before the EXTERNAL IP shell variable is successfully set      shell EXTERNAL IP   kubectl get svc ot service service        o jsonpath    status loadBalancer ingress    ip    echo  External IP   EXTERNAL IP       Edit the file  browser src index js  changing the variable  collectorURL  to refer to the external IP and port  80  of the agent with the following sed command     shell sed  i bak  s localhost 55678   EXTERNAL IP  80   browser src index js      Rebuild the web client     shell cd browser npm run build cd             Build the app image  To deploy the image to the Google Container Registry  GCR   use the following Cloud Build command     shell gcloud builds submit      Change the project id in file  k8s deployment yaml  with the sed command     shell sed  i bak  s   GOOGLE CLOUD PROJECT   k8s deployment yaml      Deploy the app     shell kubectl apply  f k8s deployment yaml      Configure a service     shell kubectl apply  f k8s service yaml      Expose a service      shell kubectl apply  f k8s ingress yaml      Check for the external IP     shell kubectl get ingress      It may take a few minutes for the service to be exposed through an external IP   Navigate to the external IP  It should present a form that will allow you to send a series of XML HTTP requests to the server  That will generate trace and monitoring data      Log exports  Before running tests  consider first setting up  log exports  https   cloud google com logging docs export   to BigQuery for more targeted log queries to analyse the results of your load tests or production issues   Create a BQ dataset to export the container logs to     shell bq   location US mk  d       description  Web instrumentation container log exports        project id  GOOGLE CLOUD PROJECT     web instr container      Create a log export for the container logs     shell LOG SA   gcloud logging sinks create web instr container logs     bigquery googleapis com projects  GOOGLE CLOUD PROJECT datasets web instr container       log filter  resource type  k8s container  AND labels k8s pod app  web instrumentation         format  value  writerIdentity          The identify of the logs writer service account is captured in the shell variable  LOG SA   Give this service account write access to BigQuery     shell gcloud projects add iam policy binding  GOOGLE CLOUD PROJECT         member  LOG SA         role roles bigquery dataEditor      Create a BQ dataset for the load balancer logs     shell bq   location US mk  d       description  Web instrumentation load balancer log exports        project id  GOOGLE CLOUD PROJECT     web instr load balancer      Repeat creation of the log sink for load balancer logs     shell LOG SA   gcloud logging sinks create web instr load balancer logs     bigquery googleapis com projects  GOOGLE CLOUD PROJECT datasets web instr load balancer       log filter  resource type  http load balancer         format  value  writerIdentity          Note that the service account id changes so that you need to note that anre repeat the step for granting write access BigQuery     shell gcloud projects add iam policy binding  GOOGLE CLOUD PROJECT       member  LOG SA       role roles bigquery dataEditor         Running the load test  Now you are ready to run the load test  You might try opening two tabs in a browser  In one tab generate a steady state load with request of say  1 second apart  to give a baseline  Then hit the app with a sudden spike to see how it responds   You can send a request from the command line with cURL     shell EXTERNAL IP  from kubectl get ingress command  REQUEST ID 1234567889   A random number   See W3C Trace Context for format TRACE 00 0af7651916cd43dd8448eb211c80319c b7ad6b7169203331 01 MILLIS  date   s N   cut  b1 13  curl  http    EXTERNAL IP data  REQUEST ID       H  traceparent   TRACE       H  Content Type  application json        data binary     data      name     Smoke test     reqId    REQUEST ID   tSent    MILLIS         Check that you see the log for the request in the Log Viewer  After the log is replicated to BigQuery  you should be able to query it with a query like below  Note that the table name will be something like  requests 20200129   A shell variable is used to set the date below      shell DATE   date  u    Y m d   bq query   use legacy sql false    SELECT   httpRequest status    httpRequest requestUrl    timestamp FROM web instr load balancer requests   DATE  ORDER BY timestamp DESC LIMIT 10       There are more queries in the Colab sheet  load test analysis ipynb  https   colab research google com github googlecolab GoogleCloudPlatform professional services blob web instrumentation examples web instrumentation load test analysis ipynb       Troubleshooting  Try the following  depending on where you encounter problems       Check project id  Check that you have set your project id in files  k8s oc service yaml    conf ocagent config yaml   and  deployment yaml        Tracing issues You can use zPages to see the trace data sent to the OC agent  Find the name of the POD running the agent      shell kubectl get pods      Start port forwarding     shell kubectl port forward  POD 55679 55679      Browse to the URL http   localhost 55679 debug tracez      Browser JavaScript For trouble bundling the web client  see Webpack  Getting Started  https   webpack js org guides getting started


GCP Five9 Voicestream Integration with Agent Assist Copyright 2024 Google This software is provided as is without warranty or representation for any use or purpose Your use of it is subject to your agreement with Google Project Structure This is a PoC to integrate Five9 Voicestream with Agent Assist

Answer:
Copyright 2024 Google. This software is provided as-is, without warranty or
representation for any use or purpose. Your use of it is subject to your
agreement with Google.

# Five9 Voicestream Integration with Agent Assist

This is a PoC to integrate Five9 Voicestream with Agent Assist.

## Project Structure

```
.
├── assets
│   └── FAQ.csv
├── client
│   ├── audio
│   │   ├── END_USER.wav
│   │   └── HUMAN_AGENT.wav
│   └── client_voicestream.py
├── .env
├── proto
│   ├── voice_pb2_grpc.py
│   ├── voice_pb2.py
│   └── voice.proto
├── README.md
├── requirements.txt
└── server
    ├── server.py
    ├── services
    │   └── get_suggestions.py
    └── utils
        ├── conversation_management.py
        └── participant_management.py
```

## Components
- Agent Assist
- Five9 with VoiceStream

## Setup Instructions

### GCP Project Setup

#### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
1. In the drop-down menu at the top, select **Create a project**.
1. Give your project a name.
1. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

#### Enabling billing for your project.

If you haven't already enabled billing for your project, [enable
billing][enable-billing] now. Enabling billing allows is required to use Cloud
Bigtable and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

#### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud SDK, [install the Google
Cloud SDK][cloud-sdk] now. The SDK contains tools and libraries that enable you
to create and manage resources on Google Cloud Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

#### Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
   gcloud init
```

Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
command:

```
    gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing

[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

#### Create a Knowledge Base

Agent Assist follows a conversation between a human agent and an end-user and provide the human agent with relevant document suggestions. These suggestions are based on knowledge bases, namely, collections of documents that you upload to Agent Assist. These documents are called knowledge documents and can be either articles (for use with Article Suggestion) or FAQ documents (for use with FAQ Assist).

In this specific implementation, a CSV sheet with FAQ will be used as knowledge document.
> [FAQ CSV file](./assets/FAQ.csv)
> [Create a Knowledge Base](https://cloud.google.com/agent-assist/docs/knowledge-base)

#### Create a Conversation Profile

A conversation profile configures a set of parameters that control the suggestions made to an agent.

> [Create/Edit an Agent Assist Conversation Profile](https://cloud.google.com/agent-assist/docs/conversation-profile#create_and_edit_a_conversation_profile)

While creating the the conversation profile, check the FAQs box. In the "Knowledge bases" input box, select the recently created Knowledge Base. The other values in the section should be set as default.

Once the conversation profile is created, you can find the CONVERSATION_PROFILE_ID (Integration ID) in the following ways:

> Open [Agent Assist](https://agentassist.cloud.google.com/), then Conversation
> Profiles on the left bottom

### Usage Pre-requisites

- FAQs Suggestions should be enabled in the Agent Assist Conversation Profile
- Agent Assist will only give you suggestions to conversations with Human Agents. It will not
  give suggestions if the conversation is being guided by virtual agents.

### Local Development Set Up

This application is designed to run on port 8080. Upon launch, the
application will initialize and bind to port 8080, making it accessible for
incoming connections. This can be changed in the .env file.

#### Protocol Buffer Compiler:

This implementation leverages from Buffer compilers for service definitions and data serialization. In this case, protoc is used to compile Five9's protofile.

 ```
NOTE: The compilation of the Five9's Voicestream protofile was already made, therefore this step can be skipped. But if an update of the protofile is needed, please follow these steps to properly output the required python files.
 ```

> [Protocol Buffer Compiler Installation](https://grpc.io/docs/protoc-installation/)
> [Five9's Voicestream protofile](./proto/voice.proto)

To compile the protofile:
> Open a terminal window
> Go to the root where your proto folder is
> Run the following command:
   ```
   python3 -m grpc_tools.protoc -I proto --python_out=proto --grpc_python_out=proto proto/voice.proto
   ```
> Two python files will be generated inside the proto folder.
   > [voice_pb2_grpc.py](./proto/voice_pb2_grpc.py)
   > [voice_pb2.py](./proto/voice_pb2.py)

#### Set of variables:

The following variables need to be set up in the .env file inside the root folder

```
SERVER_ADDRESS :
   Target server address

PORT :
   Connection Port

PROJECT_ID :
   GCP Project ID where the Agent Assist Conversation Profile is deployed.

CONVERSATION_PROFILE_ID :
   Agent Assist Conversation Profile ID

CHUNK_SIZE :
   Number of bytes of audio to be sent each time

RESTART_TIMEOUT :
   Timeout of one stream

MAX_LOOKBACK :
   Lookback for unprocessed audio data

```

### Steps to follow

## Start gRPC Server

Start the gRPC Server controller. This will start a server on port 8080, where the voicestream client will send the data.

> [Server Controller](./server/server.py)

Inside the server folder, run the following command:

```
python server.py
```

## Start gRPC Client

According to Five9's Self Service Developer Guide:

```
VoiceStream does not support multi-channel streaming. VoiceStream
transmits each distinct audio stream over a separate gRPC session: one
for audio from the agent, and one for audio to the agent.
```

In order to simulate this behaviour using our local environment, the same script should be run simultaneously. One that sends the customer audio (END_USER) and one that sends the agent audio (HUMAN_AGENT)

> [Five9 Voicestream Client](./client/client_voicestream.py)

Inside the client folder, run the following command to send the human agent audio:

```
python client_voicestream.py --role=HUMAN_AGENT --call_id=<CALL_ID>

```
In another terminal, run the following command to send the customer audio:
```
python client_voicestream.py --role=END_USER --call_id=<CALL_ID>

```

In order for both streams to be associated to the same conversation it is fundamental to specify a destination CONVERSATION_ID. For this to happen, the CALL_ID specified in the initial configuration sent by Five9 will be passed to the Agent Assist as the internal CONVERSATION_ID. In this implementation, we are manually defining this CALL_ID for testing purposes.

# References
1.[Agent Assist Documentation](https://cloud.google.com/agent-assist/docs)
2.[Dialogflow](https://cloud.google.com/dialogflow/docs)
3.[Five9 VoiceStream](https://www.five9.com/news/news-releases/five9-announces-five9-voicestream)
4,[Five9 VoiceStream Release Notes](https://releasenotes.five9.com/space/RNA/23143057870/VoiceStream)

Copyright 2024 Google  This software is provided as is  without warranty or representation for any use or purpose  Your use of it is subject to your agreement with Google     Five9 Voicestream Integration with Agent Assist  This is a PoC to integrate Five9 Voicestream with Agent Assist      Project Structure            assets         FAQ csv     client         audio             END USER wav             HUMAN AGENT wav         client voicestream py      env     proto         voice pb2 grpc py         voice pb2 py         voice proto     README md     requirements txt     server         server py         services             get suggestions py         utils             conversation management py             participant management py         Components   Agent Assist   Five9 with VoiceStream     Setup Instructions      GCP Project Setup       Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   1  In the drop down menu at the top  select   Create a project    1  Give your project a name  1  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com        Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now  Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings       Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk        Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command          gcloud init      Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command           gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing   application default credentials   https   developers google com identity protocols application default credentials       Create a Knowledge Base  Agent Assist follows a conversation between a human agent and an end user and provide the human agent with relevant document suggestions  These suggestions are based on knowledge bases  namely  collections of documents that you upload to Agent Assist  These documents are called knowledge documents and can be either articles  for use with Article Suggestion  or FAQ documents  for use with FAQ Assist    In this specific implementation  a CSV sheet with FAQ will be used as knowledge document     FAQ CSV file    assets FAQ csv     Create a Knowledge Base  https   cloud google com agent assist docs knowledge base        Create a Conversation Profile  A conversation profile configures a set of parameters that control the suggestions made to an agent      Create Edit an Agent Assist Conversation Profile  https   cloud google com agent assist docs conversation profile create and edit a conversation profile   While creating the the conversation profile  check the FAQs box  In the  Knowledge bases  input box  select the recently created Knowledge Base  The other values in the section should be set as default   Once the conversation profile is created  you can find the CONVERSATION PROFILE ID  Integration ID  in the following ways     Open  Agent Assist  https   agentassist cloud google com    then Conversation   Profiles on the left bottom      Usage Pre requisites    FAQs Suggestions should be enabled in the Agent Assist Conversation Profile   Agent Assist will only give you suggestions to conversations with Human Agents  It will not   give suggestions if the conversation is being guided by virtual agents        Local Development Set Up  This application is designed to run on port 8080  Upon launch  the application will initialize and bind to port 8080  making it accessible for incoming connections  This can be changed in the  env file        Protocol Buffer Compiler   This implementation leverages from Buffer compilers for service definitions and data serialization  In this case  protoc is used to compile Five9 s protofile        NOTE  The compilation of the Five9 s Voicestream protofile was already made  therefore this step can be skipped  But if an update of the protofile is needed  please follow these steps to properly output the required python files           Protocol Buffer Compiler Installation  https   grpc io docs protoc installation      Five9 s Voicestream protofile    proto voice proto   To compile the protofile    Open a terminal window   Go to the root where your proto folder is   Run the following command            python3  m grpc tools protoc  I proto   python out proto   grpc python out proto proto voice proto          Two python files will be generated inside the proto folder        voice pb2 grpc py    proto voice pb2 grpc py        voice pb2 py    proto voice pb2 py         Set of variables   The following variables need to be set up in the  env file inside the root folder      SERVER ADDRESS       Target server address  PORT      Connection Port     PROJECT ID      GCP Project ID where the Agent Assist Conversation Profile is deployed   CONVERSATION PROFILE ID       Agent Assist Conversation Profile ID  CHUNK SIZE      Number of bytes of audio to be sent each time  RESTART TIMEOUT      Timeout of one stream  MAX LOOKBACK       Lookback for unprocessed audio data              Steps to follow     Start gRPC Server  Start the gRPC Server controller  This will start a server on port 8080  where the voicestream client will send the data      Server Controller    server server py   Inside the server folder  run the following command       python server py         Start gRPC Client   According to Five9 s Self Service Developer Guide       VoiceStream does not support multi channel streaming  VoiceStream transmits each distinct audio stream over a separate gRPC session  one for audio from the agent  and one for audio to the agent       In order to simulate this behaviour using our local environment  the same script should be run simultaneously  One that sends the customer audio  END USER  and one that sends the agent audio  HUMAN AGENT      Five9 Voicestream Client    client client voicestream py   Inside the client folder  run the following command to send the human agent audio       python client voicestream py   role HUMAN AGENT   call id  CALL ID       In another terminal  run the following command to send the customer audio      python client voicestream py   role END USER   call id  CALL ID        In order for both streams to be associated to the same conversation it is fundamental to specify a destination CONVERSATION ID  For this to happen  the CALL ID specified in the initial configuration sent by Five9 will be passed to the Agent Assist as the internal CONVERSATION ID  In this implementation  we are manually defining this CALL ID for testing purposes      References 1  Agent Assist Documentation  https   cloud google com agent assist docs  2  Dialogflow  https   cloud google com dialogflow docs  3  Five9 VoiceStream  https   www five9 com news news releases five9 announces five9 voicestream  4  Five9 VoiceStream Release Notes  https   releasenotes five9 com space RNA 23143057870 VoiceStream