GCP 3 1 4 dataproc job optimization guide 2 Table Of Contents

Answer:
# dataproc-job-optimization-guide

----

## Table Of Contents

1. [About](#About)
2. [Guide](#Guide)
3. [Results](#Results)
4. [Next Steps](#Next-steps)

----

## About

This guide is designed to optimize performance and cost of applications running on Dataproc clusters.  Because Dataproc supports many big data technologies - each with their own intricacies - this guide is intended to be trial-and-error experimentation.  Initially it will begin with a generic dataproc cluster with defaults set. As you proceed through the guide, you’ll increasingly customize Dataproc cluster configurations to fit your specific workload.

Plan to separate Dataproc Jobs into different clusters - they use resources differently and can impact each other’s performances when run simultaneously. Even better, isolating single jobs to single clusters can set you up for ephemeral clusters, where jobs can run in parallel on their own dedicated resources.

Once your job is running successfully, you can safely iterate on the configuration to improve runtime and cost, falling back to the last successful run whenever experimental changes have a negative impact.

----

## Guide

### 1. Getting Started

Fill in your environment variables and run the following code in a terminal to set up your Google Cloud project.

```bash
export PROJECT_ID=""
export REGION=""
export CLUSTER_NAME=""
export BUCKET_NAME=""
TIMESTAMP=$(date "+%Y-%m-%dT%H%M%S")
export TIMESTAMP

./scripts/setup.sh -p $PROJECT_ID -r $REGION -c $CLUSTER_NAME -b $BUCKET_NAME -t $TIMESTAMP
```

This script will:

1. Setup project and enable APIs
2. Remove any old infrastructure related to this project (in case of previous runs)
3. Create a Google Cloud Storage bucket and a BigQuery Dataset
4. Load public data into a personal BigQuery Dataset
5. Import autoscaling policies
6. Create the first Dataproc sizing cluster

**Monitoring Dataproc Jobs**
![Stack-Resources](images/monitoring-jobs.png)

### 2. Calculate Dataproc cluster size

A sizing cluster can help determine the right number of workers for your application.  This cluster will have an autoscaling policy attached.  Set the autoscaling policy min/max values to whatever is allowed in your project. Run your jobs on this cluster. Autoscaling will continue to add nodes until the YARN pending memory metric is zero. A perfectly sized cluster should never have YARN pending memory.

```bash
gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-sizing scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

![Stack-Resources](images/monitoring-nodemanagers.png)

### 3. Optimize Dataproc cluster configuration

Using a non-autoscaling cluster during this experimentation phase can lead to the discovery of more accurate machine-types, persistent disks, application properties, etc. For now, build an isolated non-autoscaling cluster for your job that has the optimized number of primary workers.

Run your job on this appropriately-sized non-autoscaling cluster. If the CPU is maxing out, consider using C2 machine type. If memory is maxing out, consider using N2D-highmem machine types.  Also consider increasing the machine cores (while maintaining a consistent overall core count observed during sizing phase).

**Monitoring CPU Utilization**

![Stack-Resources](images/monitoring-cpu.png)

**Monitoring YARN Memory**

![Stack-Resources](images/monitoring-yarn-memory.png)

**8 x n2-standard-2 = 1 min 53 seconds**

```bash
gcloud dataproc clusters create $CLUSTER_NAME-testing-2x8-standard \
  --master-machine-type=n2-standard-2 \
  --worker-machine-type=n2-standard-2 \
  --num-workers=8 \
  --master-boot-disk-type=pd-standard \
  --master-boot-disk-size=1000GB \
  --worker-boot-disk-type=pd-standard \
  --worker-boot-disk-size=1000GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-2x8-standard scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

**4 x n2-standard-4 = 1 min 48 seconds**

```bash
gcloud dataproc clusters delete $CLUSTER_NAME-testing-2x8-standard \
  --region=$REGION

gcloud dataproc clusters create $CLUSTER_NAME-testing-4x4-standard \
  --master-machine-type=n2-standard-4 \
  --worker-machine-type=n2-standard-4 \
  --num-workers=4 \
  --master-boot-disk-type=pd-standard \
  --master-boot-disk-size=1000GB \
  --worker-boot-disk-type=pd-standard \
  --worker-boot-disk-size=1000GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-4x4-standard scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

**2 x n2-standard-8 = 1 min 31 seconds**

```bash
gcloud dataproc clusters delete $CLUSTER_NAME-testing-4x4-standard \
  --region=$REGION

gcloud dataproc clusters create $CLUSTER_NAME-testing-8x2-standard \
  --master-machine-type=n2-standard-8 \
  --worker-machine-type=n2-standard-8 \
  --num-workers=2 \
  --master-boot-disk-type=pd-standard \
  --master-boot-disk-size=1000GB \
  --worker-boot-disk-type=pd-standard \
  --worker-boot-disk-size=1000GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-8x2-standard scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

If you’re still observing performance issues, consider moving from pd-standard to pd-balanced or pd-ssd.

- Standard persistent disks (pd-standard) are suited for large data processing workloads that primarily use sequential I/Os.

- Balanced persistent disks (pd-balanced) are an alternative to SSD persistent disks that balance performance and cost. With the same maximum IOPS as SSD persistent disks and lower IOPS per GB, a balanced persistent disk offers performance levels suitable for most general-purpose applications at a price point between that of standard and SSD persistent disks.

- SSD persistent disks (pd-ssd) are suited for enterprise applications and high-performance database needs that require lower latency and more IOPS than standard persistent disks provide.

For similar costs, pd-standard 1000GB == pd-balanced 500GB == pd-ssd 250 GB.

![Stack-Resources](images/monitoring-hdfs.png)

**2 x n2-standard-8-balanced = 1 min 26 seconds**

```bash
gcloud dataproc clusters delete $CLUSTER_NAME-testing-8x2-standard \
  --region=$REGION

gcloud dataproc clusters create $CLUSTER_NAME-testing-8x2-balanced \
  --master-machine-type=n2-standard-8 \
  --worker-machine-type=n2-standard-8 \
  --num-workers=2 \
  --master-boot-disk-type=pd-balanced \
  --master-boot-disk-size=500GB \
  --worker-boot-disk-type=pd-balanced \
  --worker-boot-disk-size=500GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-8x2-balanced scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

**2 x n2-standard-8-ssd = 1 min 21 seconds**

```bash
gcloud dataproc clusters delete $CLUSTER_NAME-testing-8x2-balanced \
  --region=$REGION

gcloud dataproc clusters create $CLUSTER_NAME-testing-8x2-ssd \
  --master-machine-type=n2-standard-8 \
  --worker-machine-type=n2-standard-8 \
  --num-workers=2 \
  --master-boot-disk-type=pd-ssd \
  --master-boot-disk-size=250GB \
  --worker-boot-disk-type=pd-ssd \
  --worker-boot-disk-size=250GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-8x2-ssd scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

Monitor HDFS Capacity to determine disk size. If this ever drops to zero, you’ll need to increase the persistent disk size.  If HDFS Capacity is too large for this job, consider lowering the disk size to save on storage costs.

**2 x n2-standard-8-ssd-costop = 1 min 18 seconds**

```bash
gcloud dataproc clusters delete $CLUSTER_NAME-testing-8x2-ssd \
  --region=$REGION

gcloud dataproc clusters create $CLUSTER_NAME-testing-8x2-ssd-costop \
  --master-machine-type=n2-standard-8 \
  --worker-machine-type=n2-standard-8 \
  --num-workers=2 \
  --master-boot-disk-type=pd-ssd \
  --master-boot-disk-size=30GB \
  --worker-boot-disk-type=pd-ssd \
  --worker-boot-disk-size=30GB \
  --region=$REGION

gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-8x2-ssd-costop scripts/spark_average_speed.py -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

### 4. Optimize application-specific properties

If you’re still observing performance issues, you can begin to adjust application properties. Ideally these properties are set on the job submission. This isolates properties to their respective jobs. Since this job runs on Spark, view the [tuning guide here.](https://cloud.google.com/dataproc/docs/support/spark-job-tuning)

Since this guide uses a simple spark application and small amount of data, you may not see job performance improvement.  This section is more applicable for larger use-cases.

sample job submit:

**2 x n2-standard-8-ssd-costop-appop = 1 min 15 seconds**

```bash
gsutil -m rm -r gs://$BUCKET_NAME/transformed-$TIMESTAMP

gcloud dataproc jobs submit pyspark --region=$REGION --cluster=$CLUSTER_NAME-testing-8x2-ssd-costop scripts/spark_average_speed.py --properties='spark.executor.cores=5,spark.driver.cores=5,spark.executor.instances=1,spark.executor.memory=25459m,spark.driver.memory=25459m,spark.executor.memoryOverhead=2829m,spark.default.parallelism=10,spark.sql.shuffle.partitions=10,spark.shuffle.spill.compress=true,spark.checkpoint.compress=true,spark.io.compresion.codex=snappy,spark.dynamicAllocation=true,spark.shuffle.service.enabled=true' -- gs://$BUCKET_NAME/raw-$TIMESTAMP/ gs://$BUCKET_NAME/transformed-$TIMESTAMP/
```

### 5.  Handle edge-case workload spikes via an autoscaling policy

Now that you have an optimally sized, configured, tuned cluster, you can choose to introduce autoscaling.  This should NOT be seen as a cost-optimization technique.  But it can improve performance during the edge-cases that require more worker nodes.

Use ephemeral clusters (see step 6) to allow clusters to scale up, and delete them when the job or workflow is complete. Downscaling may not be necessary on ephemeral, job/workflow scoped clusters.

- Ensure primary workers make up >50% of your cluster.  Do not scale primary workers.

    - This does increase cost versus a smaller number of primary workers, but this is a tradeoff you can make; stability versus cost.

        - Note: Having too many secondary workers can create job instability. This is a tradeoff you can choose to make as you see fit, but best practice is to avoid having the majority of your workers be secondary.

- Prefer ephemeral clusters where possible.

    - Allow these to scale up, but not down, and delete them when jobs are complete.

    - Set scaleDownFactor to 0.0 for ephemeral clusters.

[Autoscaling clusters | Dataproc Documentation | Google Cloud](https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling#how_autoscaling_works)

sample template:

```yaml
workerConfig:
  minInstances: 2
  maxInstances: 2
secondaryWorkerConfig:
  minInstances: 0
  maxInstances: 10
basicAlgorithm:
  cooldownPeriod: 5m
  yarnConfig:
    scaleUpFactor: 1.0
    scaleDownFactor: 0
    gracefulDecommissionTimeout: 0s
```

### 6. Optimize cost and reusability via ephemeral Dataproc clusters

There are several key advantages of using ephemeral clusters:

- You can use different cluster configurations for individual jobs, eliminating the administrative burden of managing tools across jobs.

- You can scale clusters to suit individual jobs or groups of jobs.

- You only pay for resources when your jobs are using them.

- You don't need to maintain clusters over time, because they are freshly configured every time you use them.

- You don't need to maintain separate infrastructure for development, testing, and production. You can use the same definitions to create as many different versions of a cluster as you need when you need them.

sample workflow template:

```yaml
jobs:
- pysparkJob:
    args:
    - "gs://%%BUCKET_NAME%%/raw-%%TIMESTAMP%%/"
    - "gs://%%BUCKET_NAME%%/transformed-%%TIMESTAMP%%/"
    mainPythonFileUri: gs://%%BUCKET_NAME%%/scripts/spark_average_speed.py
  stepId: spark_average_speed
placement:
  managedCluster:
    clusterName: final-cluster-wft
    config:
      gceClusterConfig:
        zoneUri: %%REGION%%-a
      masterConfig:
        diskConfig:
          bootDiskSizeGb: 30
          bootDiskType: pd-ssd
        machineTypeUri: n2-standard-8
        minCpuPlatform: AUTOMATIC
        numInstances: 1
        preemptibility: NON_PREEMPTIBLE
      workerConfig:
        diskConfig:
          bootDiskSizeGb: 30
          bootDiskType: pd-ssd
        machineTypeUri: n2-standard-8
        minCpuPlatform: AUTOMATIC
        numInstances: 2
        preemptibility: NON_PREEMPTIBLE
```

sample workflow template execution

```bash
gcloud dataproc workflow-templates instantiate-from-file \
  --file templates/final-cluster-wft.yml \
  --region $REGION
```

----

## Results

Even in this small scale example. job performance was optimized by 71% (265 seconds -> 75 seconds).  And with a properly sized ephemeral cluster, you only pay for what is necessary.

![Stack-Resources](images/monitoring-job-progress.png)

----

## Next-Steps

To continue striving for maximum optimal performance, please review and consider the guidance laid out in the Google Cloud Blog.

- [Dataproc best practices | Google Cloud Blog](https://cloud.google.com/blog/topics/developers-practitioners/dataproc-best-practices-guide)
- [7 best practices for running Cloud Dataproc in production | Google Cloud Blog](https://cloud.google.com/blog/products/data-analytics/7-best-practices-for-running-cloud-dataproc-in-production)

dataproc job optimization guide           Table Of Contents  1   About   About  2   Guide   Guide  3   Results   Results  4   Next Steps   Next steps            About  This guide is designed to optimize performance and cost of applications running on Dataproc clusters   Because Dataproc supports many big data technologies   each with their own intricacies   this guide is intended to be trial and error experimentation   Initially it will begin with a generic dataproc cluster with defaults set  As you proceed through the guide  you ll increasingly customize Dataproc cluster configurations to fit your specific workload   Plan to separate Dataproc Jobs into different clusters   they use resources differently and can impact each other s performances when run simultaneously  Even better  isolating single jobs to single clusters can set you up for ephemeral clusters  where jobs can run in parallel on their own dedicated resources   Once your job is running successfully  you can safely iterate on the configuration to improve runtime and cost  falling back to the last successful run whenever experimental changes have a negative impact            Guide      1  Getting Started   Fill in your environment variables and run the following code in a terminal to set up your Google Cloud project       bash export PROJECT ID    export REGION    export CLUSTER NAME    export BUCKET NAME    TIMESTAMP   date    Y  m  dT H M S   export TIMESTAMP    scripts setup sh  p  PROJECT ID  r  REGION  c  CLUSTER NAME  b  BUCKET NAME  t  TIMESTAMP      This script will   1  Setup project and enable APIs 2  Remove any old infrastructure related to this project  in case of previous runs  3  Create a Google Cloud Storage bucket and a BigQuery Dataset 4  Load public data into a personal BigQuery Dataset 5  Import autoscaling policies 6  Create the first Dataproc sizing cluster    Monitoring Dataproc Jobs     Stack Resources  images monitoring jobs png        2  Calculate Dataproc cluster size  A sizing cluster can help determine the right number of workers for your application   This cluster will have an autoscaling policy attached   Set the autoscaling policy min max values to whatever is allowed in your project  Run your jobs on this cluster  Autoscaling will continue to add nodes until the YARN pending memory metric is zero  A perfectly sized cluster should never have YARN pending memory       bash gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME sizing scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP         Stack Resources  images monitoring nodemanagers png       3  Optimize Dataproc cluster configuration  Using a non autoscaling cluster during this experimentation phase can lead to the discovery of more accurate machine types  persistent disks  application properties  etc  For now  build an isolated non autoscaling cluster for your job that has the optimized number of primary workers   Run your job on this appropriately sized non autoscaling cluster  If the CPU is maxing out  consider using C2 machine type  If memory is maxing out  consider using N2D highmem machine types   Also consider increasing the machine cores  while maintaining a consistent overall core count observed during sizing phase        Monitoring CPU Utilization      Stack Resources  images monitoring cpu png      Monitoring YARN Memory      Stack Resources  images monitoring yarn memory png     8 x n2 standard 2   1 min 53 seconds       bash gcloud dataproc clusters create  CLUSTER NAME testing 2x8 standard       master machine type n2 standard 2       worker machine type n2 standard 2       num workers 8       master boot disk type pd standard       master boot disk size 1000GB       worker boot disk type pd standard       worker boot disk size 1000GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 2x8 standard scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP         4 x n2 standard 4   1 min 48 seconds       bash gcloud dataproc clusters delete  CLUSTER NAME testing 2x8 standard       region  REGION  gcloud dataproc clusters create  CLUSTER NAME testing 4x4 standard       master machine type n2 standard 4       worker machine type n2 standard 4       num workers 4       master boot disk type pd standard       master boot disk size 1000GB       worker boot disk type pd standard       worker boot disk size 1000GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 4x4 standard scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP         2 x n2 standard 8   1 min 31 seconds       bash gcloud dataproc clusters delete  CLUSTER NAME testing 4x4 standard       region  REGION  gcloud dataproc clusters create  CLUSTER NAME testing 8x2 standard       master machine type n2 standard 8       worker machine type n2 standard 8       num workers 2       master boot disk type pd standard       master boot disk size 1000GB       worker boot disk type pd standard       worker boot disk size 1000GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 8x2 standard scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP       If you re still observing performance issues  consider moving from pd standard to pd balanced or pd ssd     Standard persistent disks  pd standard  are suited for large data processing workloads that primarily use sequential I Os     Balanced persistent disks  pd balanced  are an alternative to SSD persistent disks that balance performance and cost  With the same maximum IOPS as SSD persistent disks and lower IOPS per GB  a balanced persistent disk offers performance levels suitable for most general purpose applications at a price point between that of standard and SSD persistent disks     SSD persistent disks  pd ssd  are suited for enterprise applications and high performance database needs that require lower latency and more IOPS than standard persistent disks provide   For similar costs  pd standard 1000GB    pd balanced 500GB    pd ssd 250 GB      Stack Resources  images monitoring hdfs png      2 x n2 standard 8 balanced   1 min 26 seconds       bash gcloud dataproc clusters delete  CLUSTER NAME testing 8x2 standard       region  REGION  gcloud dataproc clusters create  CLUSTER NAME testing 8x2 balanced       master machine type n2 standard 8       worker machine type n2 standard 8       num workers 2       master boot disk type pd balanced       master boot disk size 500GB       worker boot disk type pd balanced       worker boot disk size 500GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 8x2 balanced scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP         2 x n2 standard 8 ssd   1 min 21 seconds       bash gcloud dataproc clusters delete  CLUSTER NAME testing 8x2 balanced       region  REGION  gcloud dataproc clusters create  CLUSTER NAME testing 8x2 ssd       master machine type n2 standard 8       worker machine type n2 standard 8       num workers 2       master boot disk type pd ssd       master boot disk size 250GB       worker boot disk type pd ssd       worker boot disk size 250GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 8x2 ssd scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP       Monitor HDFS Capacity to determine disk size  If this ever drops to zero  you ll need to increase the persistent disk size   If HDFS Capacity is too large for this job  consider lowering the disk size to save on storage costs     2 x n2 standard 8 ssd costop   1 min 18 seconds       bash gcloud dataproc clusters delete  CLUSTER NAME testing 8x2 ssd       region  REGION  gcloud dataproc clusters create  CLUSTER NAME testing 8x2 ssd costop       master machine type n2 standard 8       worker machine type n2 standard 8       num workers 2       master boot disk type pd ssd       master boot disk size 30GB       worker boot disk type pd ssd       worker boot disk size 30GB       region  REGION  gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 8x2 ssd costop scripts spark average speed py    gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP           4  Optimize application specific properties   If you re still observing performance issues  you can begin to adjust application properties  Ideally these properties are set on the job submission  This isolates properties to their respective jobs  Since this job runs on Spark  view the  tuning guide here   https   cloud google com dataproc docs support spark job tuning   Since this guide uses a simple spark application and small amount of data  you may not see job performance improvement   This section is more applicable for larger use cases   sample job submit     2 x n2 standard 8 ssd costop appop   1 min 15 seconds       bash gsutil  m rm  r gs    BUCKET NAME transformed  TIMESTAMP  gcloud dataproc jobs submit pyspark   region  REGION   cluster  CLUSTER NAME testing 8x2 ssd costop scripts spark average speed py   properties  spark executor cores 5 spark driver cores 5 spark executor instances 1 spark executor memory 25459m spark driver memory 25459m spark executor memoryOverhead 2829m spark default parallelism 10 spark sql shuffle partitions 10 spark shuffle spill compress true spark checkpoint compress true spark io compresion codex snappy spark dynamicAllocation true spark shuffle service enabled true     gs    BUCKET NAME raw  TIMESTAMP  gs    BUCKET NAME transformed  TIMESTAMP           5   Handle edge case workload spikes via an autoscaling policy  Now that you have an optimally sized  configured  tuned cluster  you can choose to introduce autoscaling   This should NOT be seen as a cost optimization technique   But it can improve performance during the edge cases that require more worker nodes   Use ephemeral clusters  see step 6  to allow clusters to scale up  and delete them when the job or workflow is complete  Downscaling may not be necessary on ephemeral  job workflow scoped clusters     Ensure primary workers make up  50  of your cluster   Do not scale primary workers         This does increase cost versus a smaller number of primary workers  but this is a tradeoff you can make  stability versus cost             Note  Having too many secondary workers can create job instability  This is a tradeoff you can choose to make as you see fit  but best practice is to avoid having the majority of your workers be secondary     Prefer ephemeral clusters where possible         Allow these to scale up  but not down  and delete them when jobs are complete         Set scaleDownFactor to 0 0 for ephemeral clusters    Autoscaling clusters   Dataproc Documentation   Google Cloud  https   cloud google com dataproc docs concepts configuring clusters autoscaling how autoscaling works   sample template      yaml workerConfig    minInstances  2   maxInstances  2 secondaryWorkerConfig    minInstances  0   maxInstances  10 basicAlgorithm    cooldownPeriod  5m   yarnConfig      scaleUpFactor  1 0     scaleDownFactor  0     gracefulDecommissionTimeout  0s          6  Optimize cost and reusability via ephemeral Dataproc clusters  There are several key advantages of using ephemeral clusters     You can use different cluster configurations for individual jobs  eliminating the administrative burden of managing tools across jobs     You can scale clusters to suit individual jobs or groups of jobs     You only pay for resources when your jobs are using them     You don t need to maintain clusters over time  because they are freshly configured every time you use them     You don t need to maintain separate infrastructure for development  testing  and production  You can use the same definitions to create as many different versions of a cluster as you need when you need them    sample workflow template      yaml jobs    pysparkJob      args         gs     BUCKET NAME   raw   TIMESTAMP            gs     BUCKET NAME   transformed   TIMESTAMP         mainPythonFileUri  gs     BUCKET NAME   scripts spark average speed py   stepId  spark average speed placement    managedCluster      clusterName  final cluster wft     config        gceClusterConfig          zoneUri    REGION   a       masterConfig          diskConfig            bootDiskSizeGb  30           bootDiskType  pd ssd         machineTypeUri  n2 standard 8         minCpuPlatform  AUTOMATIC         numInstances  1         preemptibility  NON PREEMPTIBLE       workerConfig          diskConfig            bootDiskSizeGb  30           bootDiskType  pd ssd         machineTypeUri  n2 standard 8         minCpuPlatform  AUTOMATIC         numInstances  2         preemptibility  NON PREEMPTIBLE      sample workflow template execution     bash gcloud dataproc workflow templates instantiate from file       file templates final cluster wft yml       region  REGION                Results  Even in this small scale example  job performance was optimized by 71   265 seconds    75 seconds    And with a properly sized ephemeral cluster  you only pay for what is necessary     Stack Resources  images monitoring job progress png            Next Steps  To continue striving for maximum optimal performance  please review and consider the guidance laid out in the Google Cloud Blog      Dataproc best practices   Google Cloud Blog  https   cloud google com blog topics developers practitioners dataproc best practices guide     7 best practices for running Cloud Dataproc in production   Google Cloud Blog  https   cloud google com blog products data analytics 7 best practices for running cloud dataproc in production


GCP Why is Survival Analysis Helpful for Churn Prediction Survival Analysis is used to predict the time to event when the event in question has not necessarily occurred yet In this case the event is a customer churning This model uses Survival Analysis to classify customers into time to churn buckets The model output can be used to calculate each user s churn score for different durations Churn Prediction with Survival Analysis If a customer is still active or is censored using Survival Analysis terminology we do not know their final lifetime or when they will churn If we assume that the customer s lifetime ended at the time of prediction or training the results will be biased underestimating lifetime Throwing out active users will also bias results through information loss The same methodology can be used used to predict customers total lifetime from their birth initial signup or t 0 and from the current state t 0

Answer:
# Churn Prediction with Survival Analysis
This model uses Survival Analysis to classify customers into time-to-churn buckets. The model output can be used to calculate each user's churn score for different durations.

The same methodology can be used used to predict customers' total lifetime from their "birth" (initial signup, or t = 0) and from the current state (t > 0).

## Why is Survival Analysis Helpful for Churn Prediction?
Survival Analysis is used to predict the time-to-event, when the event in question has not necessarily occurred yet. In this case, the event is a customer churning.

If a customer is still active, or is "censored" using Survival Analysis terminology, we do not know their final lifetime or when they will churn. If we assume that the customer's lifetime ended at the time of prediction (or training), the results will be biased (underestimating lifetime). Throwing out active users will also bias results through information loss.

By using a Survival Analysis approach to churn prediction, the entire population (regardless of current tenure or status) can be included.

## Dataset
This example uses the public [Google Analytics Sample Dataset](https://support.google.com/analytics/answer/7586738?hl=en) on BigQuery and artificially generated subscription start and end dates as input.

To create a churn model with real data, omit the 'Generate Data' step in the Beam pipeline in preprocessor/preprocessor/preprocess.py. Instead of randomly generating values, the BigQuery results should include the following fields: start_date, end_date, and active. These values correspond to the user's subscription lifetime and their censorship status.

## Setup
### Set up GCP credentials
```shell
gcloud auth login
gcloud auth application-default login
```

### Set up Python environment
```shell
virtualenv venv
source ./venv/bin/activate
pip install -r requirements.txt
```

## Preprocessing
Using Dataflow, the data preprocessing script reads user data from BigQuery, generates random (fake) time-to-churn labels, creates TFRecords, and adds them to Google Cloud Storage.

Each record should have three labels before preprocessing:
1. **active**: indicator for censorship. It is 0 if user is inactive (uncensored) and 1 if the user is active (censored).
2. **start_date**: Date when user began their lifetime.
3. **end_date**: Date when user ends their lifetime. It is None if the user is still active.
`_generateFakeData` randomly generates these three fields in order to create fake sample data. In practice, these fields should be available in some form in the historical data.

During preprocessing, the aforementioned fields are combined into a single `2*n-dimensional indicator array`, where n is the number of bounded lifetime buckets (i.e. n = 2 for 0-2 months, 2-3 months, 3+ months):
  + indicator array = [survival array | failure array]
  + survival array = 1 if individual has survived interval, 0 otherwise (for each of the n intervals)
  + failure array = 1 if individual failed during interval, 0 otherwise
    + If an individual is censored (still active), their failure array contains only 0s

### Set Constants
```shell
BUCKET="gs://[GCS Bucket]"
NOW="$(date +%Y%m%d%H%M%S)"
OUTPUT_DIR="${BUCKET}/output_data/${NOW}"
PROJECT="[PROJECT ID]"
```

### Run locally with Dataflow
```shell
cd preprocessor

python -m run_preprocessing \
--output_dir "${OUTPUT_DIR}" \
--project_id "${PROJECT}"

cd ..
```

### Run on the Cloud with Dataflow
The top-level preprocessor directory should be the working directory for running the preprocessing script. The setup.py file should be located in the working directory.

```shell
cd preprocessor

python -m run_preprocessing \
--cloud \
--output_dir "${OUTPUT_DIR}" \
--project_id "${PROJECT}"

cd ..
```

## Model Training
Model training minimizes the negative of the log likelihood function for a statistical Survival Analysis model with discrete-time intervals. The loss function is based off the paper [A scalable discrete-time survival model for neural networks](https://peerj.com/articles/6257.pdf).

For each record, the conditional hazard probability is the probability of failure in an interval, given that individual has survived at least to the beginning of the interval. Therefore, the probability that a user survives the given interval, or the likelihood, is the product of (1 - hazard) for all of the earlier (and current) intervals.

So, the log likelihood is: ln(current hazard) + sum(ln(1 - earlier hazards)) summed over all time intervals. Equivalently, each individual's log likelihood is: `ln(1 - (1 if survived 0 if not)*(Prob of failure)) + ln(1 - (1 if failed 0 if not)*(Prob of survival))` summed over all time intervals.

### Set Constants
The TFRecord output of the preprocessing job should be used as input to the training job.

Make sure to navigate back to the top-level directory.

```shell
INPUT_DIR="${OUTPUT_DIR}"
MODEL_DIR="${BUCKET}/model/$(date +%Y%m%d%H%M%S)"
```

### Train locally with AI Platform
```shell
gcloud ai-platform local train \
--module-name trainer.task \
--package-path trainer/trainer \
--job-dir ${MODEL_DIR} \
-- \
--input-dir "${INPUT_DIR}"
```

### Train on the Cloud with AI Platform
```shell
JOB_NAME="train_$(date +%Y%m%d%H%M%S)"

gcloud ai-platform jobs submit training ${JOB_NAME} \
--job-dir ${MODEL_DIR} \
--config trainer/config.yaml \
--module-name trainer.task \
--package-path trainer/trainer \
--region us-east1 \
--python-version 3.5 \
--runtime-version 1.13 \
-- \
--input-dir ${INPUT_DIR}
```

### Hyperparameter Tuning with AI Platform
```shell
JOB_NAME="hptuning_$(date +%Y%m%d%H%M%S)"

gcloud ai-platform jobs submit training ${JOB_NAME} \
--job-dir ${MODEL_DIR} \
--module-name trainer.task \
--package-path trainer/trainer \
--config trainer/hptuning_config.yaml \
--python-version 3.5 \
--runtime-version 1.13 \
-- \
--input-dir ${INPUT_DIR}
```

### Launch Tensorboard
```shell
tensorboard --logdir ${MODEL_DIR}
```

## Predictions
The model predicts the conditional likelihood that a user survived an interval given that the user reached the interval. It outputs an n-dimensional vector, where each element corresponds to predicted conditional probability of surviving through end of time interval (1 - hazard).

In order to determine the predicted class, the cumulative product of the conditional probabilities must be compared to some threshold.

### Deploy model on AI Platform
The SavedModel was saved in a timestamped subdirectory of model_dir.
```shell
MODEL_NAME="survival_model"
VERSION_NAME="demo_version"
SAVED_MODEL_DIR=$(gsutil ls $MODEL_DIR/export/export | tail -1)

gcloud ai-platform models create $MODEL_NAME \
--regions us-east1

gcloud ai-platform versions create $VERSION_NAME \
  --model $MODEL_NAME \
  --origin $SAVED_MODEL_DIR \
  --runtime-version=1.13 \
  --framework TENSORFLOW \
  --python-version=3.5
```
### Running batch predictions
```shell
INPUT_PATHS=$INPUT_DIR/data/test/*
OUTPUT_PATH=<GCS directory for predictions>
JOB_NAME="predict_$(date +%Y%m%d%H%M%S)"

gcloud ai-platform jobs submit prediction $JOB_NAME \
    --model $MODEL_NAME \
    --input-paths $INPUT_PATHS \
    --output-path $OUTPUT_PATH \
    --region us-east1 \
    --data-format TF_RECORD
```

Churn Prediction with Survival Analysis This model uses Survival Analysis to classify customers into time to churn buckets  The model output can be used to calculate each user s churn score for different durations   The same methodology can be used used to predict customers  total lifetime from their  birth   initial signup  or t   0  and from the current state  t   0       Why is Survival Analysis Helpful for Churn Prediction  Survival Analysis is used to predict the time to event  when the event in question has not necessarily occurred yet  In this case  the event is a customer churning   If a customer is still active  or is  censored  using Survival Analysis terminology  we do not know their final lifetime or when they will churn  If we assume that the customer s lifetime ended at the time of prediction  or training   the results will be biased  underestimating lifetime   Throwing out active users will also bias results through information loss   By using a Survival Analysis approach to churn prediction  the entire population  regardless of current tenure or status  can be included      Dataset This example uses the public  Google Analytics Sample Dataset  https   support google com analytics answer 7586738 hl en  on BigQuery and artificially generated subscription start and end dates as input   To create a churn model with real data  omit the  Generate Data  step in the Beam pipeline in preprocessor preprocessor preprocess py  Instead of randomly generating values  the BigQuery results should include the following fields  start date  end date  and active  These values correspond to the user s subscription lifetime and their censorship status      Setup     Set up GCP credentials    shell gcloud auth login gcloud auth application default login          Set up Python environment    shell virtualenv venv source   venv bin activate pip install  r requirements txt          Preprocessing Using Dataflow  the data preprocessing script reads user data from BigQuery  generates random  fake  time to churn labels  creates TFRecords  and adds them to Google Cloud Storage   Each record should have three labels before preprocessing  1    active    indicator for censorship  It is 0 if user is inactive  uncensored  and 1 if the user is active  censored   2    start date    Date when user began their lifetime  3    end date    Date when user ends their lifetime  It is None if the user is still active    generateFakeData  randomly generates these three fields in order to create fake sample data  In practice  these fields should be available in some form in the historical data   During preprocessing  the aforementioned fields are combined into a single  2 n dimensional indicator array   where n is the number of bounded lifetime buckets  i e  n   2 for 0 2 months  2 3 months  3  months       indicator array    survival array   failure array      survival array   1 if individual has survived interval  0 otherwise  for each of the n intervals      failure array   1 if individual failed during interval  0 otherwise       If an individual is censored  still active   their failure array contains only 0s      Set Constants    shell BUCKET  gs    GCS Bucket   NOW    date   Y m d H M S   OUTPUT DIR    BUCKET  output data   NOW   PROJECT   PROJECT ID            Run locally with Dataflow    shell cd preprocessor  python  m run preprocessing     output dir    OUTPUT DIR       project id    PROJECT    cd             Run on the Cloud with Dataflow The top level preprocessor directory should be the working directory for running the preprocessing script  The setup py file should be located in the working directory      shell cd preprocessor  python  m run preprocessing     cloud     output dir    OUTPUT DIR       project id    PROJECT    cd             Model Training Model training minimizes the negative of the log likelihood function for a statistical Survival Analysis model with discrete time intervals  The loss function is based off the paper  A scalable discrete time survival model for neural networks  https   peerj com articles 6257 pdf    For each record  the conditional hazard probability is the probability of failure in an interval  given that individual has survived at least to the beginning of the interval  Therefore  the probability that a user survives the given interval  or the likelihood  is the product of  1   hazard  for all of the earlier  and current  intervals   So  the log likelihood is  ln current hazard    sum ln 1   earlier hazards   summed over all time intervals  Equivalently  each individual s log likelihood is   ln 1    1 if survived 0 if not   Prob of failure     ln 1    1 if failed 0 if not   Prob of survival    summed over all time intervals       Set Constants The TFRecord output of the preprocessing job should be used as input to the training job   Make sure to navigate back to the top level directory      shell INPUT DIR    OUTPUT DIR   MODEL DIR    BUCKET  model   date   Y m d H M S            Train locally with AI Platform    shell gcloud ai platform local train     module name trainer task     package path trainer trainer     job dir   MODEL DIR           input dir    INPUT DIR            Train on the Cloud with AI Platform    shell JOB NAME  train   date   Y m d H M S    gcloud ai platform jobs submit training   JOB NAME      job dir   MODEL DIR      config trainer config yaml     module name trainer task     package path trainer trainer     region us east1     python version 3 5     runtime version 1 13          input dir   INPUT DIR           Hyperparameter Tuning with AI Platform    shell JOB NAME  hptuning   date   Y m d H M S    gcloud ai platform jobs submit training   JOB NAME      job dir   MODEL DIR      module name trainer task     package path trainer trainer     config trainer hptuning config yaml     python version 3 5     runtime version 1 13          input dir   INPUT DIR           Launch Tensorboard    shell tensorboard   logdir   MODEL DIR          Predictions The model predicts the conditional likelihood that a user survived an interval given that the user reached the interval  It outputs an n dimensional vector  where each element corresponds to predicted conditional probability of surviving through end of time interval  1   hazard    In order to determine the predicted class  the cumulative product of the conditional probabilities must be compared to some threshold       Deploy model on AI Platform The SavedModel was saved in a timestamped subdirectory of model dir     shell MODEL NAME  survival model  VERSION NAME  demo version  SAVED MODEL DIR   gsutil ls  MODEL DIR export export   tail  1   gcloud ai platform models create  MODEL NAME     regions us east1  gcloud ai platform versions create  VERSION NAME       model  MODEL NAME       origin  SAVED MODEL DIR       runtime version 1 13       framework TENSORFLOW       python version 3 5         Running batch predictions    shell INPUT PATHS  INPUT DIR data test   OUTPUT PATH  GCS directory for predictions  JOB NAME  predict   date   Y m d H M S    gcloud ai platform jobs submit prediction  JOB NAME         model  MODEL NAME         input paths  INPUT PATHS         output path  OUTPUT PATH         region us east1         data format TF RECORD


GCP The DFDL definitions are stored in a Firestore database The processor service subscribes to the topic processes every message The application send a request with the binary to process to a pubsub topic Project Structure Data Format Description Language Processor Example This module is a example how to process a binary using a DFDL definition applies the definition and publishes the json result to a topic in pubsub

Answer:
# Data Format Description Language ([DFDL](https://en.wikipedia.org/wiki/Data_Format_Description_Language)) Processor Example
This module is a example how to process a binary using a DFDL definition.
The DFDL definitions are stored in a Firestore database.

The application send a request with the binary to process to a pubsub topic.

The processor service subscribes to the topic, processes every message,
applies the definition and publishes the json result to a topic in pubsub.

## Project Structure

```
.
└── dfdl_example
 ├── examples # Contain a binary and dfdl definition to be used to run this example
 └── src
       └── main
           └── java
               └── com.example.dfdl
                   ├── DfdlDef # Embedded entiites
                   ├── DfdlDefRepository # A Firestore Reactive Repository
                   ├── DfdlService # Processes the binary using a dfdl definition and output a json
                   ├── FirestoreService # Reads dfdl definitons from a firestore database
                   ├── MessageController # Publishes message to a topic with a binary to be processed.
                   ├── ProcessorService # Initializes components, configurations and services.
                   ├── PubSubServer # Publishes and subscribes to topics using channels adapters.
                   └── README.md
 └── resources
      └── application.properties
 └── pom.xml
```

## Technology Stack
1. Cloud Firestore
2. Cloud Pubsub

## Frameworks
1. Spring Boot
2. [Spring Data Cloud Firestore](https://docs.spring.io/spring-cloud-gcp/docs/current/reference/html/firestore.html)
   * [Reactive Repository](https://docs.spring.io/spring-cloud-gcp/docs/current/reference/html/firestore.html#_reactive_repositories)

## Libraries
1. [Apache Daffodil](https://daffodil.apache.org/)

## Setup Instructions
### Project Setup
#### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
1. In the drop-down menu at the top, select **Create a project**.
1. Give your project a name = my-dfdl-project
1. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

#### Enabling billing for your project.

If you haven't already enabled billing for your project, [enable
billing][enable-billing] now.  Enabling billing allows is required to use Cloud Bigtable
and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

#### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud SDK, [install the Google
Cloud SDK][cloud-sdk] now. The SDK contains tools and libraries that enable you
to create and manage resources on Google Cloud Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

#### Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
gcloud init
```
Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command:

```
    gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing
[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

### Firestore Setup
How to create a Firestore database instance can be found [here](https://cloud.google.com/firestore/docs/quickstart-servers#create_a_in_native_mode_database)

#### How to add data to firestore
The following doc, [Managing firestore using the console](https://cloud.google.com/firestore/docs/using-console),
can be used to add data to firestore to run the example.

This example connects to a Cloud Firestore with a collection with the
following specification.
The configuration can be changed by changing the application.properties file.
```
    Root collection
     dfdl-schemas =>
         document_id
             binary_example => {
                 'definiton':
                  "<?xml version"1.0" encoding="UTF-8"?>
                     <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
                      xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
                      targetNamespace="http://example.com/dfdl/helloworld/">
                        <xs:include
                      schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
                        <xs:annotation>
                           <xs:appinfo source="http://www.ogf.org/dfdl/">
                              <dfdl:format ref="GeneralFormat" />
                           </xs:appinfo>
                        </xs:annotation>
                        <xs:element name="binary_example">
                           <xs:complexType>
                              <xs:sequence>
                                 <xs:element name="w" type="xs:int">
                                    <xs:annotation>
                                       <xs:appinfo source="http://www.ogf.org/dfdl/">
                                          <dfdl:element representation="binary"
                                              binaryNumberRep="binary" byteOrder="bigEndian" lengthKind="implicit" />
                                       </xs:appinfo>
                                    </xs:annotation>
                                 </xs:element>
                                 <xs:element name="x" type="xs:int">
                                    <xs:annotation>
                                       <xs:appinfo source="http://www.ogf.org/dfdl/">
                                          <dfdl:element representation="binary"
                                                binaryNumberRep="binary" byteOrder="bigEndian" lengthKind="implicit" />
                                       </xs:appinfo>
                                    </xs:annotation>
                                 </xs:element>
                                 <xs:element name="y" type="xs:double">
                                    <xs:annotation>
                                       <xs:appinfo source="http://www.ogf.org/dfdl/">
                                          <dfdl:element representation="binary"
                                                 binaryFloatRep="ieee" byteOrder="bigEndian" lengthKind="implicit" />
                                       </xs:appinfo>
                                    </xs:annotation>
                                 </xs:element>
                                 <xs:element name="z" type="xs:float">
                                    <xs:annotation>
                                       <xs:appinfo source="http://www.ogf.org/dfdl/">
                                          <dfdl:element representation="binary"
                                                byteOrder="bigEndian" lengthKind="implicit" binaryFloatRep="ieee" />
                                       </xs:appinfo>
                                    </xs:annotation>
                                 </xs:element>
                              </xs:sequence>
                           </xs:complexType>
                        </xs:element>
                     </xs:schema>";
                  }
```
This dfdl definition example can be found in the binary_example.dfdl.xsd file.

### Pubsub Setup
The following [doc](https://cloud.google.com/pubsub/docs/quickstart-console)
can be used to set up the topics and subscriptions needed to run this example.

#### Topics
To run this example two topics need to be created:
1. A topic to publish the final json output: "data-output-json-topic"
2. A topic to publish the binary to be processed: "data-input-binary-topic"

#### Subscription
The following subscriptions need to be created:
1. A subscription to pull the binary data: data-input-binary-sub

## Usage
### Initialize the application
Reference: [Building an Application with Spring Boot](https://spring.io/guides/gs/spring-boot/)
```
      ./mvnw spring-boot:run
```
### Send a request
```
    curl --data "message=0000000500779e8c169a54dd0a1b4a3fce2946f6" localhost:8081/publish
``

Data Format Description Language   DFDL  https   en wikipedia org wiki Data Format Description Language   Processor Example This module is a example how to process a binary using a DFDL definition  The DFDL definitions are stored in a Firestore database   The application send a request with the binary to process to a pubsub topic   The processor service subscribes to the topic  processes every message  applies the definition and publishes the json result to a topic in pubsub      Project Structure            dfdl example      examples   Contain a binary and dfdl definition to be used to run this example      src            main                java                    com example dfdl                        DfdlDef   Embedded entiites                        DfdlDefRepository   A Firestore Reactive Repository                        DfdlService   Processes the binary using a dfdl definition and output a json                        FirestoreService   Reads dfdl definitons from a firestore database                        MessageController   Publishes message to a topic with a binary to be processed                         ProcessorService   Initializes components  configurations and services                         PubSubServer   Publishes and subscribes to topics using channels adapters                         README md      resources           application properties      pom xml         Technology Stack 1  Cloud Firestore 2  Cloud Pubsub     Frameworks 1  Spring Boot 2   Spring Data Cloud Firestore  https   docs spring io spring cloud gcp docs current reference html firestore html        Reactive Repository  https   docs spring io spring cloud gcp docs current reference html firestore html  reactive repositories      Libraries 1   Apache Daffodil  https   daffodil apache org       Setup Instructions     Project Setup      Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   1  In the drop down menu at the top  select   Create a project    1  Give your project a name   my dfdl project 1  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com        Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now   Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings       Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk        Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command       gcloud init     Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command              gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing  application default credentials   https   developers google com identity protocols application default credentials      Firestore Setup How to create a Firestore database instance can be found  here  https   cloud google com firestore docs quickstart servers create a in native mode database         How to add data to firestore The following doc   Managing firestore using the console  https   cloud google com firestore docs using console   can be used to add data to firestore to run the example   This example connects to a Cloud Firestore with a collection with the following specification  The configuration can be changed by changing the application properties file          Root collection      dfdl schemas             document id              binary example                        definiton                        xml version 1 0  encoding  UTF 8                          xs schema xmlns xs  http   www w3 org 2001 XMLSchema                        xmlns dfdl  http   www ogf org dfdl dfdl 1 0                         targetNamespace  http   example com dfdl helloworld                             xs include                       schemaLocation  org apache daffodil xsd DFDLGeneralFormat dfdl xsd                              xs annotation                              xs appinfo source  http   www ogf org dfdl                                   dfdl format ref  GeneralFormat                                  xs appinfo                            xs annotation                           xs element name  binary example                               xs complexType                                 xs sequence                                    xs element name  w  type  xs int                                        xs annotation                                          xs appinfo source  http   www ogf org dfdl                                               dfdl element representation  binary                                                binaryNumberRep  binary  byteOrder  bigEndian  lengthKind  implicit                                              xs appinfo                                        xs annotation                                     xs element                                    xs element name  x  type  xs int                                        xs annotation                                          xs appinfo source  http   www ogf org dfdl                                               dfdl element representation  binary                                                  binaryNumberRep  binary  byteOrder  bigEndian  lengthKind  implicit                                              xs appinfo                                        xs annotation                                     xs element                                    xs element name  y  type  xs double                                        xs annotation                                          xs appinfo source  http   www ogf org dfdl                                               dfdl element representation  binary                                                   binaryFloatRep  ieee  byteOrder  bigEndian  lengthKind  implicit                                              xs appinfo                                        xs annotation                                     xs element                                    xs element name  z  type  xs float                                        xs annotation                                          xs appinfo source  http   www ogf org dfdl                                               dfdl element representation  binary                                                  byteOrder  bigEndian  lengthKind  implicit  binaryFloatRep  ieee                                              xs appinfo                                        xs annotation                                     xs element                                  xs sequence                               xs complexType                            xs element                         xs schema                            This dfdl definition example can be found in the binary example dfdl xsd file       Pubsub Setup The following  doc  https   cloud google com pubsub docs quickstart console  can be used to set up the topics and subscriptions needed to run this example        Topics To run this example two topics need to be created  1  A topic to publish the final json output   data output json topic  2  A topic to publish the binary to be processed   data input binary topic        Subscription The following subscriptions need to be created  1  A subscription to pull the binary data  data input binary sub     Usage     Initialize the application Reference   Building an Application with Spring Boot  https   spring io guides gs spring boot               mvnw spring boot run         Send a request         curl   data  message 0000000500779e8c169a54dd0a1b4a3fce2946f6  localhost 8081 publish


GCP tl dr The Terraform project in this repository deploys a single simple Python Cloud Function that executes a simple action against Google Cloud API using identity of the function caller illustrating the full flow described above Often the Google Cloud Functions are used for automation tasks e g cloud infrastrcuture automation In big Google Cloud organizations such automation needs access to all organization tenants and because of that the Cloud Function Service Account needs wide scope IAM permissions at the organization or similar level This example describes one possible solution of reducing the set of Google Cloud IAM permissions required by a Service Account running a Google Cloud Function by executing the Cloud Function on behalf of and with IAM permissions of the caller identity Cloud Function Act As Caller

Answer:
# Cloud Function "Act As" Caller

> tl;dr The Terraform project in this repository deploys a single simple Python Cloud Function that executes a simple action against Google Cloud API using identity of the function caller illustrating the full flow described above.

---

This example describes one possible solution of reducing the set of Google Cloud IAM permissions required by a Service Account running a Google Cloud Function by executing the Cloud Function on behalf of and with IAM permissions of the caller identity.

Often the Google Cloud Functions are used for automation tasks, e.g. cloud infrastrcuture automation. In big Google Cloud organizations such automation needs access to all organization tenants and because of that the Cloud Function Service Account needs wide scope IAM permissions at the organization or similar level.

Following the Principle of Least Privilege the Service Account that the Cloud Function executes under needs to have only IAM permissions required for its successful execution. In the case when a common Cloud Function automation is called by multiple tenants for performing operations on their individual tenant Google Cloud resources it would only require IAM permissions to the GCP resources of that tenant at a time. Those are typically permissions that the tenant is already having or can obtain for its tenant Service Accounts or Workload Identity.

This example illustrates a possible approach of reducing the set of Cloud Function IAM permissions to the temporary set of permissions defined by the context of the Cloud Function caller and its identity in particular.

The example contains a solution for the GitHub based CI/CD workflow caller of a Google Cloud Function based on the Workload Idenitty Federation mechanism supported by GitHub and [google-github-actions/auth](https://github.com/google-github-actions/auth) GitHub Action.

The Terraform project in this repository deploys a single simple Python Cloud Function that executes a simple action against Google Cloud API using identity of the function caller illustrating the full flow described above.

## Caller Workload Identity

One of the typical security concerns in Cloud SaaS based CI/CD pipelines that manage and manipulate Google Cloud resources is the need to securely authenticate the caller process on the Google Cloud side.

A typical way of authenticating is using Google Cloud Serivce Account keys stored on the caller side that effectively are being used as long lived client secrets that need to be protected and kept in secret. The need to manage and protect the long lived Service Account keys is a security challenge for the client.

The recommended way to improve the secruity posture of the CI/CD automation is to remove the need to authenticate with Google Cloud using Service Account keys altogether. The Workload Identity Federation is the mechanism that allows secure authentication with no long lived secrets managed by the client based on the Open ID Connect authentication flow.

The following diagram describes the authentication process that does not require Google Cloud Service Account key management on the client side.

![Workflow Identity Federation Authentication](images/wif-authentication.png?raw=true "Workflow Identity Federation Authentication")

## Solution

The Cloud Function Service Account is not granted any IAM permissions in the tenant GCP project. The only permissions it requires is read access to the ephemeal temporary Google Secret Manager Secret resource in its own project where the Cloud Function is defined and running.

The Caller is the GitHub Runner that executes the GitHub Workflow defined in this source repository in the [.github/workflows/call-function.yml](./.github/workflows/call-function.yml) file. It authenticates to GCP as a Workload Identity using Workflow Identity Fedederation set up by the Terraform project in this repository.

The Service Account "mapped" to the Workload Identity of the GitHub Workflow run has needed read/write permissions to the GCP resources that the Cloud Function needs to manipulate.

There are several variations possible in regards to the location of the ephemeral secrets to be stored (application project vs central automation project).

### Simple Invocation

The following diagram shows the simplest case of the Cloud Function invocation in which the GCP access token gets passed in the call to the Cloud Function, e.g. in its payload or better HTTP header.

![Invocation with the secret in the application project](images/cf-act-as0.png?raw=true "Invocation with the secret in the application project")

1. The GitHub Workflow (the Caller) authenticates to the GCP using the [google-github-actions/auth](https://github.com/google-github-actions/auth) GitHub Action. After this step the GitHub Workflow has short lived GCP access token available to the subsequent workflow steps to execute and connect to the GCP to call the Cloud Function.
2. The Caller passes the GCP access token obtained on the previous step in the Cloud Function invocation HTTPS request payload or header.
3. The Cloud Function authenticates to the Google API using the access token extracted from the Secret Manager Secret on the previous step and accesses the target GCP resource on behalf and with IAM permissions of the Caller.

### Invocation with Secret

In some cases, such as when the logic represented by a Cloud Function is implemented by several components, such as Cloud Functions chained together, it would be needed to pass the caller's access token between those components securely. In that situation it is preferrable to apply a [Claim Check pattern](https://www.enterpriseintegrationpatterns.com/StoreInLibrary.html) and pass the resource name of the secret containing the access token between the solution components as it is illustrated in the following diagram.

![Invocation with the secret in the application project](images/cf-act-as3.png?raw=true "Invocation with the secret in the application project")

1. The GitHub Workflow (the Caller) authenticates to the GCP using the [google-github-actions/auth](https://github.com/google-github-actions/auth) GitHub Action. After this step the GitHub Workflow has short lived GCP access token available to the subsequent workflow steps to execute and connect to the GCP to call the Cloud Function.
2. The Caller passes the GCP access token obtained on the previous step in the Cloud Function invocation HTTPS request payload or header.
3. The first Cloud Function extracts the GCP access token obtained on the previous step from the incoming message payload and stores it in an ephemeral Secret Manager Secret in the central project location.
4. The Cloud Function extracts the access token from the ephemeral Secret Manager Secret
5. The Cloud Function authenticates to the Google API using the access token extracted from the Secret Manager Secret on the previous step and accesses the target GCP resource on behalf and with IAM permissions of the Caller.
6. (Optionally) The Cloud Function drops the ephemeral Secret Manager Secret resource.
7. (Optionally) The Caller double checks and drops the ephemeral Secret Manager Secret resource.

## Service Accounts and IAM Permissions

This project creates two GCP Service Accounts:
* Cloud Function Service Account – replaces the default Cloud Function [runtime service account](https://cloud.google.com/functions/docs/securing/function-identity#runtime_service_account) with an explicit customer managed service account
* Workload Identity Service Account – the GCP service account that represents the external GitHub Workload Identity. When the GitHub workflow authenticates to the GCP it is this service account's IAM permissions that the GitHub Workload Identity is granted.

| Service Account     | Role                                        | Description                                       |
|---------------------|---------------------------------------------|---------------------------------------------------|
| `cf-sample-account` | `roles/secretmanager.secretVersionManager`  | To create the Secret Manager secret for access token for the "Invocation with central secret" case |
|                     | `roles/secretmanager.secretAccessor`        | To read the access token from Secret Manager secret  |
| `wi-sample-account` | `roles/secretmanager.secretVersionManager`  | To create the Secret Manager secret for access token for the "Invocation with application owned secret" case |
|                     |                                             | To store the access token in the Secret Manager secret in the "Invocation with application owned secret" case |
|                     | `roles/iam.workloadIdentityUser`            | Maps the external GitHub Workload Identity to the Workload Identity Service Account |
|                     | `roles/cloudfunctions.developer`            | `cloudfunctions.functions.call` permission to invoke the sample Cloud Function  |
|                     | `roles/viewer`                              | Sample IAM permissions to list GCE VM instances granted to the Workload Identity Service Account but not to the Cloud Function Service Account  |

## Deployment

The Terraform project in this repository defines the following input variables that can either be edited in the `variables.tf` file directly or passed over the Terraform command line.

| Name          | Description                                     |
|---------------|-------------------------------------------------|
| `project_id`  | Target GCP project id. All resources will be deployed here.  |
| `project_number`  | Target GCP project number.                  |
| `location`    | The name of the GCP region to deploy resources  |
| `zone`        | The GCP Zone for the sample Cloud Function to list GCE VM instances in                                     |
| `github_repo`  | The name of the GitHub repository in the format `organization/repository`.                                |
| `github_ref`  | The git reference to the source code repository. Usually a reference to the branch which is getting built  |

To deploy the example with Cloud Function and all required GCP components including Workload Idepntity Pool and Provider use the usual
```
terraform init
terraform plan
terraform apply
```

in the root folder of this repository.

The project deploys the GCP resources by default into the `europe-west3` region. You can change that by passing alternative value to the `location`  input variable by copying the `terraform.tfvars.sample` to the `terraform.tfvars` file and updating values there.

## Call Function

After the example is provisioned through Terraform, you can test and call the deployed function from the command line with gcloud
```
gcloud functions call sample-function --region=${REGION} --data '{}'
```

The sample Cloud Function calls Google Compute Engine API v1 and [lists](https://cloud.google.com/compute/docs/reference/rest/v1/instances/list) Google Compte Engine instances in the specified region.

The Cloud Function deployed by this project runs as `cf-sample-account@${PROJECT_ID}.iam.gserviceaccount.com` service account.
This service account doesn't have any granted permissions in GCP except for the read access to the Secret Manager Secret. Hence, the Cloud Function cannot reach the GCE API and list the VMs by default.

For this action to successfully complete from the command line as illustrated above, the Cloud Function service
account `cf-sample-account@${PROJECT_ID}.iam.gserviceaccount.com` needs to have `compute.instances.list` permission in the target GCP project.

If the execution succeeds, the command line output will be similar to the following
```
$ gcloud functions call sample-function --region=${REGION} --data '{}'

executionId: 84e2bkg5717v
result: ', jumpbox (https://www.googleapis.com/compute/v1/projects/${PROJECT_ID}/zones/europe-west3-c/machineTypes/e2-micro)'
```

## GitHub Workflow

The sample GitHub [workflow](.github/workflows/call-function.yml) in this repository illustrates the way of calling the sample Cloud Function from a GitHub workflow.

For the workflow to succeed, a dedicated service account `wi-sample-account` is mapped to the authenticated GitHub Workload Identity. It needs to have `cloudfunctions.functions.call` permission for the deployed Sample Cloud Function in order to be able to invoke it. The `roles/cloudfunctions.developer` built-in role grants that permission.

## Running Example

Copy `terraform.tfvar.sample` file to `terraform.tfvar` and adjust settings inside for your project, location, etc.

Deploy the GCP resources with Terraform:
```
terraform init
terraform plan
terraform apply
```

To invoke the Cloud Function on behalf of the GitHub workload idenity, it is needed to create GitHub Actions workflow from the `.github/workflows/call-function.yml` file. Copy this file with relative folders to the root of your GitHub repository for GitHub to pick up the workflow.

Please note that the GitHub workflow reads the parameters during the run from the `terraform.tfvars.sample` file in the root repository folder. You'd need to either modify the workflow file or check in correct values to the `terraform.tfvars.sample` file.

After the GCP resources are provisioned, and given that the parameters in the `terraform.tfvars.sample` are correct, the GitHub Actions run should succeed. It is the [last step](.github/workflows/call-function.yml#L68) is the sample Cloud Function call.
This is because the Workload Identity Service Account that the project associates with the GitHub identity has permissions to list GCE VM instances, which is what the sample Python Cloud Function is doing.

At this point it is now possible to ensure that the direct Cloud Function invocation, e.g. using an interactive user account, with no access token supplied, fails because the Cloud Function Service Account itself does not have permissions to list GCE VM instances:
```
gcloud functions call sample-function --region=${REGION} --data '{}'
```
That call should fail no matter which permissions for GCE your current user account is having:
```
result: "Error: <HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/$PROJECT_ID/zones/$ZONE/instances?alt=json\
  \ returned \"Required 'compute.instances.list' permission for 'projects/$PROJECT_ID'\"\
  . Details: \"[{'message': \"Required 'compute.instances.list' permission for 'projects/$PROJECT_ID'\"\
  , 'domain': 'global', 'reason': 'forbidden'}]\">"
```

Now you can try to pass the access token that represents an account that has GCE VM instances list permissions. E.g. if your current user account has that permission:
```
gcloud functions call sample-function --region=${REGION} --data "{ \"access_token\": \"$(gcloud auth print-access-token)\" }"
```
This time the call should succeed and show the list of GCE VMs, e.g.
```
executionId: 76wkh0r8yhjf
result: 'jumpbox'
```

Alternatively, you can pass the access token via Secret Manager Secret in the same way as GitHub workflow does.

Save the access token to the Secret Manager Secret created by this Terraform project and pass the Secret Manager secret resource name in the call to the sample Cloud Function instead of the access token:
```
gcloud functions call sample-function --region=${REGION} \
    --data "{ \"secret_resource\": \"$(echo -n $(gcloud auth print-access-token) | \
    gcloud secrets versions add access-token-secret --data-file=- --format json | \
    jq -r .name)\" }"
```

This call should succeed as well. The sample Cloud Function will extract the access token of the current user account from the Secret Manager secret and call GCE API provding that access token for authentication.

## Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can use Terraform to delete most of the resources. If you
created a new project for deploying the resources, you can also delete the entire project.

To delete resources using Terraform, run the following command:

    terraform destroy

To delete the project, do the following:

1.  In the Cloud Console, go to the [Projects page](https://console.cloud.google.com/iam-admin/projects).
1.  In the project list, select the project you want to delete and click **Delete**.
1.  In the dialog, type the project ID, and then click **Shut down** to delete the project.
---

## Useful Commands

### Read current access token using gcloud

[Getting the access token using gcloud](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/print-access-token)

```
gcloud auth application-default print-access-token
```

### Store access token in Secret Manager
```
echo -n "$(gcloud auth print-access-token)" | \
    gcloud secrets versions add access-token-secret --data-file=-
```

### Develop and Debugg the Cloud Function locally
Within the [function](./function) folder run following commds to start the function framework locally:
```
pip install -r requirements.txt
functions-framework --target main --debug
``

Cloud Function  Act As  Caller    tl dr The Terraform project in this repository deploys a single simple Python Cloud Function that executes a simple action against Google Cloud API using identity of the function caller illustrating the full flow described above        This example describes one possible solution of reducing the set of Google Cloud IAM permissions required by a Service Account running a Google Cloud Function by executing the Cloud Function on behalf of and with IAM permissions of the caller identity   Often the Google Cloud Functions are used for automation tasks  e g  cloud infrastrcuture automation  In big Google Cloud organizations such automation needs access to all organization tenants and because of that the Cloud Function Service Account needs wide scope IAM permissions at the organization or similar level   Following the Principle of Least Privilege the Service Account that the Cloud Function executes under needs to have only IAM permissions required for its successful execution  In the case when a common Cloud Function automation is called by multiple tenants for performing operations on their individual tenant Google Cloud resources it would only require IAM permissions to the GCP resources of that tenant at a time  Those are typically permissions that the tenant is already having or can obtain for its tenant Service Accounts or Workload Identity   This example illustrates a possible approach of reducing the set of Cloud Function IAM permissions to the temporary set of permissions defined by the context of the Cloud Function caller and its identity in particular    The example contains a solution for the GitHub based CI CD workflow caller of a Google Cloud Function based on the Workload Idenitty Federation mechanism supported by GitHub and  google github actions auth  https   github com google github actions auth  GitHub Action   The Terraform project in this repository deploys a single simple Python Cloud Function that executes a simple action against Google Cloud API using identity of the function caller illustrating the full flow described above       Caller Workload Identity  One of the typical security concerns in Cloud SaaS based CI CD pipelines that manage and manipulate Google Cloud resources is the need to securely authenticate the caller process on the Google Cloud side    A typical way of authenticating is using Google Cloud Serivce Account keys stored on the caller side that effectively are being used as long lived client secrets that need to be protected and kept in secret  The need to manage and protect the long lived Service Account keys is a security challenge for the client   The recommended way to improve the secruity posture of the CI CD automation is to remove the need to authenticate with Google Cloud using Service Account keys altogether  The Workload Identity Federation is the mechanism that allows secure authentication with no long lived secrets managed by the client based on the Open ID Connect authentication flow   The following diagram describes the authentication process that does not require Google Cloud Service Account key management on the client side     Workflow Identity Federation Authentication  images wif authentication png raw true  Workflow Identity Federation Authentication       Solution  The Cloud Function Service Account is not granted any IAM permissions in the tenant GCP project  The only permissions it requires is read access to the ephemeal temporary Google Secret Manager Secret resource in its own project where the Cloud Function is defined and running   The Caller is the GitHub Runner that executes the GitHub Workflow defined in this source repository in the   github workflows call function yml     github workflows call function yml  file  It authenticates to GCP as a Workload Identity using Workflow Identity Fedederation set up by the Terraform project in this repository   The Service Account  mapped  to the Workload Identity of the GitHub Workflow run has needed read write permissions to the GCP resources that the Cloud Function needs to manipulate   There are several variations possible in regards to the location of the ephemeral secrets to be stored  application project vs central automation project        Simple Invocation  The following diagram shows the simplest case of the Cloud Function invocation in which the GCP access token gets passed in the call to the Cloud Function  e g  in its payload or better HTTP header     Invocation with the secret in the application project  images cf act as0 png raw true  Invocation with the secret in the application project    1  The GitHub Workflow  the Caller  authenticates to the GCP using the  google github actions auth  https   github com google github actions auth  GitHub Action  After this step the GitHub Workflow has short lived GCP access token available to the subsequent workflow steps to execute and connect to the GCP to call the Cloud Function  2  The Caller passes the GCP access token obtained on the previous step in the Cloud Function invocation HTTPS request payload or header  3  The Cloud Function authenticates to the Google API using the access token extracted from the Secret Manager Secret on the previous step and accesses the target GCP resource on behalf and with IAM permissions of the Caller       Invocation with Secret  In some cases  such as when the logic represented by a Cloud Function is implemented by several components  such as Cloud Functions chained together  it would be needed to pass the caller s access token between those components securely  In that situation it is preferrable to apply a  Claim Check pattern  https   www enterpriseintegrationpatterns com StoreInLibrary html  and pass the resource name of the secret containing the access token between the solution components as it is illustrated in the following diagram     Invocation with the secret in the application project  images cf act as3 png raw true  Invocation with the secret in the application project    1  The GitHub Workflow  the Caller  authenticates to the GCP using the  google github actions auth  https   github com google github actions auth  GitHub Action  After this step the GitHub Workflow has short lived GCP access token available to the subsequent workflow steps to execute and connect to the GCP to call the Cloud Function  2  The Caller passes the GCP access token obtained on the previous step in the Cloud Function invocation HTTPS request payload or header  3  The first Cloud Function extracts the GCP access token obtained on the previous step from the incoming message payload and stores it in an ephemeral Secret Manager Secret in the central project location  4  The Cloud Function extracts the access token from the ephemeral Secret Manager Secret 5  The Cloud Function authenticates to the Google API using the access token extracted from the Secret Manager Secret on the previous step and accesses the target GCP resource on behalf and with IAM permissions of the Caller  6   Optionally  The Cloud Function drops the ephemeral Secret Manager Secret resource  7   Optionally  The Caller double checks and drops the ephemeral Secret Manager Secret resource      Service Accounts and IAM Permissions  This project creates two GCP Service Accounts    Cloud Function Service Account   replaces the default Cloud Function  runtime service account  https   cloud google com functions docs securing function identity runtime service account  with an explicit customer managed service account   Workload Identity Service Account   the GCP service account that represents the external GitHub Workload Identity  When the GitHub workflow authenticates to the GCP it is this service account s IAM permissions that the GitHub Workload Identity is granted     Service Account       Role                                          Description                                                                                cf sample account     roles secretmanager secretVersionManager     To create the Secret Manager secret for access token for the  Invocation with central secret  case                            roles secretmanager secretAccessor           To read the access token from Secret Manager secret       wi sample account     roles secretmanager secretVersionManager     To create the Secret Manager secret for access token for the  Invocation with application owned secret  case                                                                         To store the access token in the Secret Manager secret in the  Invocation with application owned secret  case                            roles iam workloadIdentityUser               Maps the external GitHub Workload Identity to the Workload Identity Service Account                            roles cloudfunctions developer                cloudfunctions functions call  permission to invoke the sample Cloud Function                             roles viewer                                 Sample IAM permissions to list GCE VM instances granted to the Workload Identity Service Account but not to the Cloud Function Service Account         Deployment  The Terraform project in this repository defines the following input variables that can either be edited in the  variables tf  file directly or passed over the Terraform command line     Name            Description                                                                                project id     Target GCP project id  All resources will be deployed here        project number     Target GCP project number                        location       The name of the GCP region to deploy resources       zone           The GCP Zone for the sample Cloud Function to list GCE VM instances in                                          github repo     The name of the GitHub repository in the format  organization repository                                       github ref     The git reference to the source code repository  Usually a reference to the branch which is getting built     To deploy the example with Cloud Function and all required GCP components including Workload Idepntity Pool and Provider use the usual     terraform init terraform plan terraform apply      in the root folder of this repository   The project deploys the GCP resources by default into the  europe west3  region  You can change that by passing alternative value to the  location   input variable by copying the  terraform tfvars sample  to the  terraform tfvars  file and updating values there       Call Function  After the example is provisioned through Terraform  you can test and call the deployed function from the command line with gcloud     gcloud functions call sample function   region   REGION    data           The sample Cloud Function calls Google Compute Engine API v1 and  lists  https   cloud google com compute docs reference rest v1 instances list  Google Compte Engine instances in the specified region   The Cloud Function deployed by this project runs as  cf sample account   PROJECT ID  iam gserviceaccount com  service account  This service account doesn t have any granted permissions in GCP except for the read access to the Secret Manager Secret  Hence  the Cloud Function cannot reach the GCE API and list the VMs by default   For this action to successfully complete from the command line as illustrated above  the Cloud Function service account  cf sample account   PROJECT ID  iam gserviceaccount com  needs to have  compute instances list  permission in the target GCP project   If the execution succeeds  the command line output will be similar to the following       gcloud functions call sample function   region   REGION    data       executionId  84e2bkg5717v result     jumpbox  https   www googleapis com compute v1 projects   PROJECT ID  zones europe west3 c machineTypes e2 micro           GitHub Workflow  The sample GitHub  workflow   github workflows call function yml  in this repository illustrates the way of calling the sample Cloud Function from a GitHub workflow   For the workflow to succeed  a dedicated service account  wi sample account  is mapped to the authenticated GitHub Workload Identity  It needs to have  cloudfunctions functions call  permission for the deployed Sample Cloud Function in order to be able to invoke it  The  roles cloudfunctions developer  built in role grants that permission       Running Example  Copy  terraform tfvar sample  file to  terraform tfvar  and adjust settings inside for your project  location  etc   Deploy the GCP resources with Terraform      terraform init terraform plan terraform apply      To invoke the Cloud Function on behalf of the GitHub workload idenity  it is needed to create GitHub Actions workflow from the   github workflows call function yml  file  Copy this file with relative folders to the root of your GitHub repository for GitHub to pick up the workflow   Please note that the GitHub workflow reads the parameters during the run from the  terraform tfvars sample  file in the root repository folder  You d need to either modify the workflow file or check in correct values to the  terraform tfvars sample  file   After the GCP resources are provisioned  and given that the parameters in the  terraform tfvars sample  are correct  the GitHub Actions run should succeed  It is the  last step   github workflows call function yml L68  is the sample Cloud Function call   This is because the Workload Identity Service Account that the project associates with the GitHub identity has permissions to list GCE VM instances  which is what the sample Python Cloud Function is doing   At this point it is now possible to ensure that the direct Cloud Function invocation  e g  using an interactive user account  with no access token supplied  fails because the Cloud Function Service Account itself does not have permissions to list GCE VM instances      gcloud functions call sample function   region   REGION    data          That call should fail no matter which permissions for GCE your current user account is having      result   Error   HttpError 403 when requesting https   compute googleapis com compute v1 projects  PROJECT ID zones  ZONE instances alt json      returned   Required  compute instances list  permission for  projects  PROJECT ID         Details       message     Required  compute instances list  permission for  projects  PROJECT ID          domain    global    reason    forbidden             Now you can try to pass the access token that represents an account that has GCE VM instances list permissions  E g  if your current user account has that permission      gcloud functions call sample function   region   REGION    data      access token        gcloud auth print access token           This time the call should succeed and show the list of GCE VMs  e g      executionId  76wkh0r8yhjf result   jumpbox       Alternatively  you can pass the access token via Secret Manager Secret in the same way as GitHub workflow does   Save the access token to the Secret Manager Secret created by this Terraform project and pass the Secret Manager secret resource name in the call to the sample Cloud Function instead of the access token      gcloud functions call sample function   region   REGION          data      secret resource        echo  n   gcloud auth print access token          gcloud secrets versions add access token secret   data file     format json         jq  r  name            This call should succeed as well  The sample Cloud Function will extract the access token of the current user account from the Secret Manager secret and call GCE API provding that access token for authentication      Cleaning up  To avoid incurring charges to your Google Cloud account for the resources used in this tutorial  you can use Terraform to delete most of the resources  If you  created a new project for deploying the resources  you can also delete the entire project   To delete resources using Terraform  run the following command       terraform destroy  To delete the project  do the following   1   In the Cloud Console  go to the  Projects page  https   console cloud google com iam admin projects   1   In the project list  select the project you want to delete and click   Delete    1   In the dialog  type the project ID  and then click   Shut down   to delete the project           Useful Commands      Read current access token using gcloud   Getting the access token using gcloud  https   cloud google com sdk gcloud reference auth application default print access token       gcloud auth application default print access token           Store access token in Secret Manager     echo  n    gcloud auth print access token           gcloud secrets versions add access token secret   data file            Develop and Debugg the Cloud Function locally  Within the  function    function  folder run following commds to start the function framework locally      pip install  r requirements txt functions framework   target main   debug