GCP Config Validator Setup User Guide Table of Contents Go from setup to proof of concept in under 1 hour

Answer:
## Config Validator | Setup & User Guide

### Go from setup to proof-of-concept in under 1 hour

**Table of Contents**

* [Overview](#overview)
* [How to set up constraints with Policy Library](#how-to-set-up-constraints-with-policy-library)
  * [Get started with the Policy Library repository](#get-started-with-the-policy-library-repository)
  * [Instantiate constraints](#instantiate-constraints)
* [How to validate policies](#how-to-validate-policies)
  * [Deploy Forseti](#deploy-forseti)
  * [Policy Library Sync from Git Repository](https://forsetisecurity.org/docs/latest/configure/config-validator/policy-library-sync-from-git-repo.html)
  * [Policy Library Sync from GCS](https://forsetisecurity.org/docs/latest/configure/config-validator/policy-library-sync-from-gcs.html)
* [End to end workflow with sample constraint](#end-to-end-workflow-with-sample-constraint)
* [Contact Info](#contact-info)

## Overview

This tool is designed to perform policy validation check on Terraform resource changes. It will not help with ongoing monitoring in your organization heirarchy, so if you're looking for that, please find the [config-validator](https://github.com/GoogleCloudPlatform/config-validator) project and associated [policy-library](https://github.com/GoogleCloudPlatform/policy-library) to get started with Cloud Asset Inventory policies.

Designed as an offshoot from the aforementioned policy-library, we set out to design a similar library that targets resource changes before terraform deployments. By refactoring rego policies in our library, we were able to target `validation.resourcechange.terraform.cloud.google.com` instead of the forsetisecurity target for CAI data. This allows for policy control in cases when the current state of the enviornment would clearly conflict with security policies, but you can't enforce fine-grained control to allow for that state to exist while locking out nearby features from terraform. This would likely be the case in automated IAM role or permission granting in a project with a super-admin. The super-admin may need to be there, and if using CAI policy validation, the pipeline would always fail if you define policies that limit the scope of a user's control.

Keep in mind that this behavior may lead to security vulnerabilities, because the tool does not perform any ongoing monitoring.

## How to set up constraints with Policy Library

### Get started with the Policy Library repository

The Policy Library repository contains the following directories:

* `policies`
  * `constraints`: This is initially empty. You should place your constraint
        files here.
  * `templates`: This directory contains pre-defined constraint templates.
* `validator`: This directory contains the `.rego` files and their associated
    unit tests. You do not need to touch this directory unless you intend to
    modify existing constraint templates or create new ones. Running `make
    build` will inline the Rego content in the corresponding constraint template
    files.

This repository contains a set of pre-defined constraint
templates. You can duplicate this repository into a private repository. First
you should create a new **private** git repository. For example, if you use
GitHub then you can use the [GitHub UI](https://github.com/new). Then follow the
steps below to get everything setup.

This policy library can also be made public, but it is not recommended. By
making your policy library public, it would allow others to see what you
are and **ARE NOT** scanning for.

#### Duplicate Policy Library Repository

To run the following commands, you will need to configure git to connect
securely. It is recommended to connect with SSH. [Here is a helpful resource](https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) for learning about how
this works, including steps to set this up for GitHub repositories; other
providers offer this feature as well.

```
export GIT_REPO_ADDR="git@github.com:${YOUR_GITHUB_USERNAME}/policy-library.git"
git clone --bare https://github.com/tdesrosi/gcp-terraform-config-validator.git
cd policy-library.git
git push --mirror ${GIT_REPO_ADDR}
cd ..
rm -rf policy-library.git
git clone ${GIT_REPO_ADDR}
```

#### Setup Constraints

Then you need to examine the available constraint templates inside the
`templates` directory. Pick the constraint templates that you wish to use,
create constraint YAML files corresponding to those templates, and place them
under `policies/constraints`. Commit the newly created constraint files to
**your** Git repository. For example, assuming you have created a Git repository
named "policy-library" under your GitHub account, you can use the following
commands to perform the initial commit:

```
cd policy-library
# Add new constraints...
git add --all
git commit -m "Initial commit of policy library constraints"
git push -u origin master
```

#### Pull in latest changes from Public Repository

Periodically you should pull any changes from the public repository, which might
contain new templates and Rego files.

```
git remote add public https://github.com/tdesrosi/policy-library-tf-resource-change.git
git pull public main
git push origin main
```

### Instantiate constraints

The constraint template library only contains templates. Templates specify the
constraint logic, and you must create constraints based on those templates in
order to enforce them. Constraint parameters are defined as YAML files in the
following format:

```
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: # place constraint template kind here
metadata:
  name: # place constraint name here
spec:
  severity: # low, medium, or high
  match:
    target: [] # put the constraint application target here
    exclude: [] # optional, default is no exclusions
  parameters: # put the parameters defined in constraint template here
```

The target field is specified in a path-like format. It
specifies where in the GCP resources hierarchy the constraint is to be applied.
For example:

   Target

   Description

   organizations/**

   All organizations

   organizations/123/**

   Everything in organization 123

   organizations/123/folders/**

   Everything in organization 123 that is under a folder

   organizations/123/folders/456

   Everything in folder 456 in organization 123

   organizations/123/folders/456/projects/789

   Everything in project 789 in folder 456 in organization 123

The exclude field follows the same pattern and has
precedence over the target field. If a resource is in
both, it will be excluded.

The schema of the parameters field is defined in the
constraint template, using the
[OpenAPI V3](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject)
schema. This is the same validation schema in Kubernetes's custom resource
definition. Every template contains a validation section
that looks like the following:

```
validation:
  openAPIV3Schema:
    properties:
      mode:
        type: string
      instances:
        type: array
        items: string
```

According to the template above, the parameter field in the constraint file
should contain a string named `mode` and a string array named
instances. For example:

```
parameters:
  mode: allowlist
  instances:
    - //compute.googleapis.com/projects/test-project/zones/us-east1-b/instances/one
    - //compute.googleapis.com/projects/test-project/zones/us-east1-b/instances/two
```

These parameters specify that two VM instances may have external IP addresses.
The are exempt from the constraint since they are allowlisted.

Here is a complete example of a sample external IP address constraint file:

```
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: TFGCPExternalIpAccessConstraintV1
metadata:
  name: forbid-external-ip-allowlist
spec:
  severity: high
  match:
    target: ["organizations/**"]
  parameters:
    mode: "allowlist"
    instances:
    - //compute.googleapis.com/projects/test-project/zones/us-east1-b/instances/one
    - //compute.googleapis.com/projects/test-project/zones/us-east1-b/instances/two
```

## How to validate policies

Follow the [instructions](https://cloud.google.com/docs/terraform/policy-validation/validate-policies)
to validate policies in your local or production environments.

## End to end workflow with sample constraint

In this section, you will apply a constraint that enforces IAM policy member
domain restriction using [Cloud Shell](https://cloud.google.com/shell/).

First click on this
[link](https://console.cloud.google.com/cloudshell/open?cloudshell_image=gcr.io/graphite-cloud-shell-images/terraform:latest&cloudshell_git_repo=https://github.com/tdesrosi/policy-tf-resource-change.git)
to open a new Cloud Shell session. The Cloud Shell session has Terraform
pre-installed and the Policy Library repository cloned. Once you have the
session open, the next step is to copy over the sample IAM domain restriction
constraint:

```
cp samples/constraints/iam_service_accounts_only.yaml policies/constraints
```

Let's take a look at this constraint:

```
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: TFGCPIAMAllowedPolicyMemberDomainsConstraintV2
metadata:
  name: service-accounts-only
  annotations:
    description:
      Checks that members that have been granted IAM roles belong to allowlisted
      domains. Block IAM role bindings for non-service accounts by domain
      (gserviceaccount.com)
spec:
  severity: high
  parameters:
    domains:
      - gserviceaccount.com
```

It specifies that only members from gserviceaccount.com domain can be present in
an IAM policy. To verify that it works, let's attempt to create a project.
Create the following Terraform `main.tf` file:

```
provider "google" {
  version = "~> 1.20"
  project = "your-terraform-provider-project"
}

resource "random_id" "proj" {
  byte_length = 8
}

resource "google_project" "sample_project" {
  project_id      = "validator-${random_id.proj.hex}"
  name            = "config validator test project"
}

resource "google_project_iam_binding" "sample_iam_binding" {
  project = "${google_project.sample_project.project_id}"
  role    = "roles/owner"

  members = [
    "user:your-email@your-domain"
  ]
}

```

Make sure to specify your Terraform
[provider project](https://www.terraform.io/docs/providers/google/getting_started.html)
and email address. Then initialize Terraform and generate a Terraform plan:

```
terraform init
terraform plan -out=test.tfplan
terraform show -json ./test.tfplan > ./tfplan.json
```

Since your email address is in the IAM policy binding, the plan should result in
a violation. Let's try this out:

```
gcloud beta terraform vet tfplan.json --policy-library=policy-library
```

The Terraform validator should return a violation. As a test, you can relax the
constraint to make the violation go away. Edit the
`policy-library/policies/constraints/iam_service_accounts_only.yaml` file and
append your email domain to the domains allowlist:

```
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: TFGCPIAMAllowedPolicyMemberDomainsConstraintV2
metadata:
  name: service-accounts-only
  annotations:
    description:
      Checks that members that have been granted IAM roles belong to allowlisted
      domains. Block IAM role bindings for non-service accounts by domain
      (gserviceaccount.com)
spec:
  severity: high
  parameters:
    domains:
      - gserviceaccount.com
      - your-email-domain.com
```

Then run Terraform plan and validate the output again:

```
terraform plan -out=test.tfplan
terraform show -json ./test.tfplan > ./tfplan.json
gcloud beta terraform vet tfplan.json --policy-library=policy-library
```

The command above should result in no violations found.

## Contact Info

Questions or comments? Please contact tdesrosi@google.com for this project, or validator-support@google.com for information about the terraform-validator project.

Config Validator   Setup   User Guide      Go from setup to proof of concept in under 1 hour    Table of Contents       Overview   overview     How to set up constraints with Policy Library   how to set up constraints with policy library       Get started with the Policy Library repository   get started with the policy library repository       Instantiate constraints   instantiate constraints     How to validate policies   how to validate policies       Deploy Forseti   deploy forseti       Policy Library Sync from Git Repository  https   forsetisecurity org docs latest configure config validator policy library sync from git repo html       Policy Library Sync from GCS  https   forsetisecurity org docs latest configure config validator policy library sync from gcs html     End to end workflow with sample constraint   end to end workflow with sample constraint     Contact Info   contact info      Overview  This tool is designed to perform policy validation check on Terraform resource changes  It will not help with ongoing monitoring in your organization heirarchy  so if you re looking for that  please find the  config validator  https   github com GoogleCloudPlatform config validator  project and associated  policy library  https   github com GoogleCloudPlatform policy library  to get started with Cloud Asset Inventory policies   Designed as an offshoot from the aforementioned policy library  we set out to design a similar library that targets resource changes before terraform deployments  By refactoring rego policies in our library  we were able to target  validation resourcechange terraform cloud google com  instead of the forsetisecurity target for CAI data  This allows for policy control in cases when the current state of the enviornment would clearly conflict with security policies  but you can t enforce fine grained control to allow for that state to exist while locking out nearby features from terraform  This would likely be the case in automated IAM role or permission granting in a project with a super admin  The super admin may need to be there  and if using CAI policy validation  the pipeline would always fail if you define policies that limit the scope of a user s control   Keep in mind that this behavior may lead to security vulnerabilities  because the tool does not perform any ongoing monitoring      How to set up constraints with Policy Library      Get started with the Policy Library repository  The Policy Library repository contains the following directories      policies       constraints   This is initially empty  You should place your constraint         files here       templates   This directory contains pre defined constraint templates     validator   This directory contains the   rego  files and their associated     unit tests  You do not need to touch this directory unless you intend to     modify existing constraint templates or create new ones  Running  make     build  will inline the Rego content in the corresponding constraint template     files   This repository contains a set of pre defined constraint templates  You can duplicate this repository into a private repository  First you should create a new   private   git repository  For example  if you use GitHub then you can use the  GitHub UI  https   github com new   Then follow the steps below to get everything setup   This policy library can also be made public  but it is not recommended  By making your policy library public  it would allow others to see what you are and   ARE NOT   scanning for        Duplicate Policy Library Repository  To run the following commands  you will need to configure git to connect securely  It is recommended to connect with SSH   Here is a helpful resource  https   help github com en github authenticating to github connecting to github with ssh  for learning about how this works  including steps to set this up for GitHub repositories  other providers offer this feature as well       export GIT REPO ADDR  git github com   YOUR GITHUB USERNAME  policy library git  git clone   bare https   github com tdesrosi gcp terraform config validator git cd policy library git git push   mirror   GIT REPO ADDR  cd    rm  rf policy library git git clone   GIT REPO ADDR            Setup Constraints  Then you need to examine the available constraint templates inside the  templates  directory  Pick the constraint templates that you wish to use  create constraint YAML files corresponding to those templates  and place them under  policies constraints   Commit the newly created constraint files to   your   Git repository  For example  assuming you have created a Git repository named  policy library  under your GitHub account  you can use the following commands to perform the initial commit       cd policy library   Add new constraints    git add   all git commit  m  Initial commit of policy library constraints  git push  u origin master           Pull in latest changes from Public Repository  Periodically you should pull any changes from the public repository  which might contain new templates and Rego files       git remote add public https   github com tdesrosi policy library tf resource change git git pull public main git push origin main          Instantiate constraints  The constraint template library only contains templates  Templates specify the constraint logic  and you must create constraints based on those templates in order to enforce them  Constraint parameters are defined as YAML files in the following format       apiVersion  constraints gatekeeper sh v1beta1 kind    place constraint template kind here metadata    name    place constraint name here spec    severity    low  medium  or high   match      target       put the constraint application target here     exclude       optional  default is no exclusions   parameters    put the parameters defined in constraint template here      The  code  em target  em   code  field is specified in a path like format  It specifies where in the GCP resources hierarchy the constraint is to be applied  For example    table     tr      td Target      td      td Description      td      tr     tr      td organizations         td      td All organizations      td      tr     tr      td organizations 123         td      td Everything in organization 123      td      tr     tr      td organizations 123 folders         td      td Everything in organization 123 that is under a folder      td      tr     tr      td organizations 123 folders 456      td      td Everything in folder 456 in organization 123      td      tr     tr      td organizations 123 folders 456 projects 789      td      td Everything in project 789 in folder 456 in organization 123      td      tr    table   The  code  em exclude  em   code  field follows the same pattern and has precedence over the  code  em target  em   code  field  If a resource is in both  it will be excluded   The schema of the  code  em parameters  em   code  field is defined in the constraint template  using the  OpenAPI V3  https   github com OAI OpenAPI Specification blob master versions 3 0 0 md schemaObject  schema  This is the same validation schema in Kubernetes s custom resource definition  Every template contains a  code  em validation  em   code  section that looks like the following       validation    openAPIV3Schema      properties        mode          type  string       instances          type  array         items  string      According to the template above  the parameter field in the constraint file should contain a string named  mode  and a string array named  code  em instances  em   code   For example       parameters    mode  allowlist   instances          compute googleapis com projects test project zones us east1 b instances one         compute googleapis com projects test project zones us east1 b instances two      These parameters specify that two VM instances may have external IP addresses  The are exempt from the constraint since they are allowlisted   Here is a complete example of a sample external IP address constraint file       apiVersion  constraints gatekeeper sh v1beta1 kind  TFGCPExternalIpAccessConstraintV1 metadata    name  forbid external ip allowlist spec    severity  high   match      target    organizations        parameters      mode   allowlist      instances          compute googleapis com projects test project zones us east1 b instances one         compute googleapis com projects test project zones us east1 b instances two         How to validate policies  Follow the  instructions  https   cloud google com docs terraform policy validation validate policies  to validate policies in your local or production environments      End to end workflow with sample constraint  In this section  you will apply a constraint that enforces IAM policy member domain restriction using  Cloud Shell  https   cloud google com shell     First click on this  link  https   console cloud google com cloudshell open cloudshell image gcr io graphite cloud shell images terraform latest cloudshell git repo https   github com tdesrosi policy tf resource change git  to open a new Cloud Shell session  The Cloud Shell session has Terraform pre installed and the Policy Library repository cloned  Once you have the session open  the next step is to copy over the sample IAM domain restriction constraint       cp samples constraints iam service accounts only yaml policies constraints      Let s take a look at this constraint       apiVersion  constraints gatekeeper sh v1beta1 kind  TFGCPIAMAllowedPolicyMemberDomainsConstraintV2 metadata    name  service accounts only   annotations      description        Checks that members that have been granted IAM roles belong to allowlisted       domains  Block IAM role bindings for non service accounts by domain        gserviceaccount com  spec    severity  high   parameters      domains          gserviceaccount com      It specifies that only members from gserviceaccount com domain can be present in an IAM policy  To verify that it works  let s attempt to create a project  Create the following Terraform  main tf  file       provider  google      version       1 20    project    your terraform provider project     resource  random id   proj      byte length   8    resource  google project   sample project      project id         validator   random id proj hex     name               config validator test project     resource  google project iam binding   sample iam binding      project      google project sample project project id     role       roles owner     members          user your email your domain              Make sure to specify your Terraform  provider project  https   www terraform io docs providers google getting started html  and email address  Then initialize Terraform and generate a Terraform plan       terraform init terraform plan  out test tfplan terraform show  json   test tfplan     tfplan json      Since your email address is in the IAM policy binding  the plan should result in a violation  Let s try this out       gcloud beta terraform vet tfplan json   policy library policy library      The Terraform validator should return a violation  As a test  you can relax the constraint to make the violation go away  Edit the  policy library policies constraints iam service accounts only yaml  file and append your email domain to the domains allowlist       apiVersion  constraints gatekeeper sh v1beta1 kind  TFGCPIAMAllowedPolicyMemberDomainsConstraintV2 metadata    name  service accounts only   annotations      description        Checks that members that have been granted IAM roles belong to allowlisted       domains  Block IAM role bindings for non service accounts by domain        gserviceaccount com  spec    severity  high   parameters      domains          gserviceaccount com         your email domain com      Then run Terraform plan and validate the output again       terraform plan  out test tfplan terraform show  json   test tfplan     tfplan json gcloud beta terraform vet tfplan json   policy library policy library      The command above should result in no violations found      Contact Info  Questions or comments  Please contact tdesrosi google com for this project  or validator support google com for information about the terraform validator project


GCP The folder structure below contains a TL DR explanation of each item s purpose We ll go into further detail below bash You ll notice that this repository contains a handful of folders each with different items It s confusing at first so let s dive into it First let s start with how the library is organized policy library tf resource change root Functional Principles of the Constraint Framework Folder Structure

Answer:
# Functional Principles of the Constraint Framework

You'll notice that this repository contains a handful of folders, each with different items. It's confusing at first, so let's dive into it! First, let's start with how the library is organized.

## Folder Structure

The folder structure below contains a TL;DR explanation of each item's purpose. We'll go into further detail below.

```(bash)
policy-library-tf-resource-change (root)/
├── docs/
│   └── *Contains documentation on this library*
├── lib/
│   └── *Contains shared rego functions*
├── policies/
│   ├── constraints/
│   │   └── *Contains constraint yaml files*
│   └── templates/
│       └── *Contains constraint template yaml files*
├── samples/
│   └── constraints/
│       └── *Contains sample constraint yaml files (not checked at runtime)*
├── scripts/
│   └── *Contains helper scripts to assist with policy development*
├── validator/
│   ├── *Contains rego policies (used in constraint template yaml files)*
│   ├── *files ending in `*_test.rego` are base unit testing files for their associated rego policies*
│   └── test/
│       └── *Contains test data/constraints used for unit testing*
└── Makefile *Allows user to use `make ...` commands for policy development*
```

## Basic Operation

When you run `gcloud beta terrafor vet`, a number of things happen. First, the resource being tested (ie. Your terraform plan json file) is translated from its native Terraform schema to Cloud Asset Inventory (CAI) schema. Keep in mind that the data being passed in is exactly the same, it's just translated into a language that the validator can understand. Because the validator is also set up to perform ongoing policy validation on the environment, all resources are cast into the CAI schema to bring all data together. Don't be alarmed, however, our constraints and constraint templates are reworked to tell the validator to *only* look at terraform resource changes. We'll explore how that happens later on.

Once the data is in CAI format, `gcloud beta terraform vet` then initializes the constraints and templates in your policy library. Let's look at what each type of yaml file contains:

| Type | Description |
| -- | -- |
| ConstraintTemplate | Describes how to integrate logic from a rego policy into rules that the validator checks. This file will describe how constraints that use it should be configured, and also provides the rego policy as an inlined yaml string field. You'll also notice the `target` definition, which reads as `validation.resourcechange.terraform.cloud.google.com`. This is the main difference between this policy library and the [existing CAI policy library](https://github.com/GoogleCloudPlatform/policy-library). **This is the part of the library that allows us to target terraform resource changes and skip current environment monitoring.** Ultimately, think of ConstraintTemplates as definitions that our corresponding constraint(s) must abide by. |
| Constraint | Implements single rules that depend on a constraint template. Remember that constraint templates contain a schema that describes how to write constraints that depend on them. The actual constraint will contain the data that you'll be looking to test. For example, if you want to validate that terraform only creates IAM bindings, you would create a constraint that passes in the *mode* `allowlist`, and a list of `members` (domains, in this case) that terraform is allowed to create. The constraint tells the validator to fail the pipeline if terraform tries to bind an IAM role to a user ouside of the domain(s) you've passed in.

 You can create multiple constraints for any given ConstraintTemplate, you just need to make sure that the rules don't conflict with one another. For example, any allowlist/denylist policy would be difficult to create multiple constraints for. The reason is that if one constraint is of type `allowlist` and the other is of type `denylist`, it's very easy to introduce overlaps in the sets that you create. By this logic, if *domain-a.com* is not a part of your policy member domain allowlist or denylist constraints, it'll automatically be denied. This is because allowlists are more inclusive than denylists, and any domain not in an allowlist will be denied. Just because it's not denied in the denylist, it will be denied by the allowlist. |

If this is still confusing to you, the [Gatekeeper docuemntation](https://github.com/GoogleCloudPlatform/policy-library) provides a detailed explanation of how Constraints and ConstraintTemplates work together. Yes, Gatekeeper is generally synonymous with Kubernetes resources, but it's simply an extension of Open Policy Agent. We use gatekeeper's API in the terraform validator, and it's how the tool is able to interpret policies from the policy library.

## Developing Terraform Constraints

In order to develop new policies, you must first understand the process for writing rego policies and how the constraint framework is used. The very first step is to collect some sample data. erraform-based constraints operate on **resource change data**, which comes from the `resource_changes` key of [Terraform plan JSON.](https://developer.hashicorp.com/terraform/internals/json-format) This will be an array of objects that describe changes that terraform will try to make to the target environment. Here's an example that comes directly from one of the data files used in one of the unit tests in this library.

```(json)
 "resource_changes": [
        {
            "address": "google_project_iam_binding.iam-service-account-user-12345",
            "mode": "managed",
            "type": "google_project_iam_binding",
            "name": "iam-service-account-user-12345",
            "provider_name": "registry.terraform.io/hashicorp/google",
            "change": {
                "actions": [
                    "create"
                ],
                "before": null,
                "after": {
                    "condition": [],
                    "members": [
                        "serviceAccount:service-12345@notiam.gserviceaccount.com",
                        "user:bad@notgoogle.com"
                    ],
                    "project": "12345",
                    "role": "roles/iam.serviceAccountUser"
                },
                "after_unknown": {
                    "condition": [],
                    "etag": true,
                    "id": true,
                    "members": [
                        false,
                        false
                    ]
                },
                "before_sensitive": false,
                "after_sensitive": {
                    "condition": [],
                    "members": [
                        false,
                        false
                    ]
                }
            }
        },
        [...]
    ]
```

You can start to see the schema of each resource, which will help us with the next step. It's time to write the Rego policy, which you can put under the `validator/` folder in the project.

Note that Rego policies for terraform resource changes are slightly different than their CAI counterparts. Let's go through a sample policy that allows us to allowlist/denylist certain IAM roles:

```(rego)
package templates.gcp.TFGCPIAMAllowBanRolesConstraintV1

violation[{
 "msg": message,
 "details": metadata,
}] {
 params := input.parameters
 resource := input.review
 resource.type == "google_project_iam_binding"
 not resource.change.actions[0] == "delete"
 role := resource.change.after.role
 matches_found = [r | r := config_pattern(role); glob.match(params.roles[_], [], r)]
 mode := object.get(params, "mode", "allowlist")
 target_match_count(mode, desired_count)
 count(matches_found) != desired_count
 message := output_msg(desired_count, resource.name, role)
 metadata := {
  "resource": resource.name,
  "role": role,
 }
}
target_match_count(mode) = 0 {
 mode == "denylist"
}
target_match_count(mode) = 1 {
 mode == "allowlist"
}
output_msg(0, asset_name, role) = msg {
 msg := sprintf("%v is in the banned list of IAM policy for %v", [role, asset_name])
}
output_msg(1, asset_name, role) = msg {
 msg := sprintf("%v is NOT in the allowed list of IAM policy for %v", [role, asset_name])
}
config_pattern(old_pattern) = "**" {
 old_pattern == "*"
}
config_pattern(old_pattern) = old_pattern {
 old_pattern != "*"
}
```

There are two parts to this policy. We have a `violation` object, which contains line-by-line logic statements. The way rego works is that the policy will run line-by-line, and will `break` if any of the conditions don't pass. For instance, if the `resource.type` is **not** "google_project_iam_binding", the rest of the rule will not be checked, and no violation gets reported. If you're familiar with Golang syntax, a `:=` means *set the variable equal to the value on the right side (declaration, assignment, and redeclaration)*. A `==` means *check the equality of the left and right side*. A `=` is for assignment **only**, meaning that it won't infer the type of the variable. It can be used outside of functions of violations, however.

The violation object will run with every `input.review` object (which is each object in the `resource_changes` array, evaluated one at a time). `input.parameters` will always be the same, as it contains the information passed in the `parameters` section of the constraint (AHA! That's why we use the constraint framework!). In the case of this policy, if the number of matches we get between the input object and the constraint does not match the desired count (determined by whether the mode of the constraint is `allowlist` or `denylist`), the `violation` will spit out a violation message, and an object containing metadata.

The second part of the policy is for functions that the policy uses in the violation. You'll see duplicates, like `target_match_count`, for example. These duplicative functions allow us to return a different value based on the input. In the case of `target_match_count`, if the mode is of type `*denylist`, it will return a `0`. So, if we end up getting one hit between our `input.review` object, the actual count **will not** equal our desired count, and the violation will trigger, and send a violation message to stdout.

### Inlining Rego Policies

This is the easiest part of the development process. Once you've created a new policy, if you've cloned this repo, simply run `make build` to inline the policy into its accompanying constraint template. Remember the first line of the rego policy? `package templates.gcp.TFGCPIAMAllowBanRolesConstraintV1`? You'll need to make sure that you create a constraint template with the `kind` property under spec->crd->spec->names->kind set to `TFGCPIAMAllowBanRolesConstraintV1`. You can see this, and other details, in the ConstraintTemplate that accompanys this policy:

```(yaml)
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: tfgcpiamallowbanrolesconstraintv1
spec:
  crd:
    spec:
      names:
        kind: TFGCPIAMAllowBanRolesConstraintV1
      validation:
        openAPIV3Schema:
          properties:
            mode:
              description: "Enforcement mode, defaults to allow"
              type: string
              enum: [denylist, allowlist]
            roles:
              description: "Roles to be allowed or banned
                ex. roles/owner; Wildcards (*) supported"
              type: array
              items:
                type: string
  targets:
    - target: validation.resourcechange.terraform.cloud.google.com
      rego: |
        #INLINE("validator/my_rego_rule.rego")
        ... (rego code)
        #ENDINLINE
```

You can see that the ConstraintTemplate defines a set of properties that constraints built from this template must abide by. Look at a constraint that uses this template to get a better idea of how the two interract:

```(yaml)
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: TFGCPIAMAllowBanRolesConstraintV1
metadata:
  name: iam-allow-roles
  annotations:
    description: Allow only the listed IAM role bindings to be created. This
      constraint is member-agnostic.
spec:
  severity: high
  match:
    target: # {"$ref":"#/definitions/io.k8s.cli.setters.target"}
      - "organizations/**"
    exclude: [] # optional, default is no exclusions
  parameters:
    mode: "allowlist"
    roles:
      - "roles/logging.viewer"
      - "roles/resourcemanager.projectIamAdmin"
```

Notice that the constraint defines a set of parameters, including `mode` and `roles`. If you look at the ConstraintTempalte, you can see that these two fields are defined and described. `Mode` is a string enumerable that *must* be either denylist or allowlist. `gcloud beta terraform vet` will actually error out if this is not upheld in the associated constraint(s).

## Additional Resources

Documentation on the Constraint Framework and `gcloud beta terraform vet` is pretty widespread, but this document provides some application-specific information for using the `validation.resourcechange.terraform.cloud.google.com` target for validating terraform resource changes. Documentation on this use case does exist, and can be found [here](https://cloud.google.com/docs/terraform/policy-validation/create-terraform-constraints).

When it comes time to create your own rego policies, I also recommend using the [Rego Playground](https://play.openpolicyagent.org/) to first get a hang of the Rego language. Developing Rego in a local environment is really challenging, as debugging and tracing features must be integrated manually, usually with a log statement, like `trace(sprintf("Value: %s", [value]))`. However, the very nature of how Rego runs makes this extremely challenging, becasue if your violation exits before you even get to a log line, the trace will never be printed to stdout, and you won't easily be able to see what's going wrong in your policies.

In my early development of this policy library, I created a couple of playgrounds in order to get a hang of Rego. Here's [an example](https://play.openpolicyagent.org/p/HzzUikhvQ4) of that, where I tested the logic for validating permissions on IAM Custom Roles. Take a look if you're interested! Although keep in mind that this wouldn't work if brought into the policy library, as this policy is unsuported by the Gatekeeper v1beta1 API Version. You would need to change the `deny` rule to `violation` and use updated Rego built-ins, like `object.get` instead of `get_default()`. For me, the output gave me helpful hints on what was wrong with my rules and utility functions.

## Contact Info

Questions or comments? Please contact tdesrosi@google.com for this project, or validator-support@google.com for information about the terraform-validator project.

Functional Principles of the Constraint Framework  You ll notice that this repository contains a handful of folders  each with different items  It s confusing at first  so let s dive into it  First  let s start with how the library is organized      Folder Structure  The folder structure below contains a TL DR explanation of each item s purpose  We ll go into further detail below       bash  policy library tf resource change  root       docs           Contains documentation on this library      lib           Contains shared rego functions      policies          constraints               Contains constraint yaml files          templates               Contains constraint template yaml files      samples          constraints               Contains sample constraint yaml files  not checked at runtime       scripts           Contains helper scripts to assist with policy development      validator           Contains rego policies  used in constraint template yaml files            files ending in    test rego  are base unit testing files for their associated rego policies          test               Contains test data constraints used for unit testing      Makefile  Allows user to use  make      commands for policy development          Basic Operation  When you run  gcloud beta terrafor vet   a number of things happen  First  the resource being tested  ie  Your terraform plan json file  is translated from its native Terraform schema to Cloud Asset Inventory  CAI  schema  Keep in mind that the data being passed in is exactly the same  it s just translated into a language that the validator can understand  Because the validator is also set up to perform ongoing policy validation on the environment  all resources are cast into the CAI schema to bring all data together  Don t be alarmed  however  our constraints and constraint templates are reworked to tell the validator to  only  look at terraform resource changes  We ll explore how that happens later on   Once the data is in CAI format   gcloud beta terraform vet  then initializes the constraints and templates in your policy library  Let s look at what each type of yaml file contains     Type   Description                 ConstraintTemplate   Describes how to integrate logic from a rego policy into rules that the validator checks  This file will describe how constraints that use it should be configured  and also provides the rego policy as an inlined yaml string field  You ll also notice the  target  definition  which reads as  validation resourcechange terraform cloud google com   This is the main difference between this policy library and the  existing CAI policy library  https   github com GoogleCloudPlatform policy library     This is the part of the library that allows us to target terraform resource changes and skip current environment monitoring    Ultimately  think of ConstraintTemplates as definitions that our corresponding constraint s  must abide by      Constraint   Implements single rules that depend on a constraint template  Remember that constraint templates contain a schema that describes how to write constraints that depend on them  The actual constraint will contain the data that you ll be looking to test  For example  if you want to validate that terraform only creates IAM bindings  you would create a constraint that passes in the  mode   allowlist   and a list of  members   domains  in this case  that terraform is allowed to create  The constraint tells the validator to fail the pipeline if terraform tries to bind an IAM role to a user ouside of the domain s  you ve passed in   br    br    You can create multiple constraints for any given ConstraintTemplate  you just need to make sure that the rules don t conflict with one another  For example  any allowlist denylist policy would be difficult to create multiple constraints for  The reason is that if one constraint is of type  allowlist  and the other is of type  denylist   it s very easy to introduce overlaps in the sets that you create  By this logic  if  domain a com  is not a part of your policy member domain allowlist or denylist constraints  it ll automatically be denied  This is because allowlists are more inclusive than denylists  and any domain not in an allowlist will be denied  Just because it s not denied in the denylist  it will be denied by the allowlist     If this is still confusing to you  the  Gatekeeper docuemntation  https   github com GoogleCloudPlatform policy library  provides a detailed explanation of how Constraints and ConstraintTemplates work together  Yes  Gatekeeper is generally synonymous with Kubernetes resources  but it s simply an extension of Open Policy Agent  We use gatekeeper s API in the terraform validator  and it s how the tool is able to interpret policies from the policy library      Developing Terraform Constraints  In order to develop new policies  you must first understand the process for writing rego policies and how the constraint framework is used  The very first step is to collect some sample data  erraform based constraints operate on   resource change data    which comes from the  resource changes  key of  Terraform plan JSON   https   developer hashicorp com terraform internals json format  This will be an array of objects that describe changes that terraform will try to make to the target environment  Here s an example that comes directly from one of the data files used in one of the unit tests in this library       json    resource changes                            address    google project iam binding iam service account user 12345                mode    managed                type    google project iam binding                name    iam service account user 12345                provider name    registry terraform io hashicorp google                change                      actions                          create                                      before   null                   after                          condition                            members                              serviceAccount service 12345 notiam gserviceaccount com                            user bad notgoogle com                                              project    12345                        role    roles iam serviceAccountUser                                      after unknown                          condition                            etag   true                       id   true                       members                             false                          false                                                           before sensitive   false                   after sensitive                          condition                            members                             false                          false                                                                                You can start to see the schema of each resource  which will help us with the next step  It s time to write the Rego policy  which you can put under the  validator   folder in the project   Note that Rego policies for terraform resource changes are slightly different than their CAI counterparts  Let s go through a sample policy that allows us to allowlist denylist certain IAM roles       rego  package templates gcp TFGCPIAMAllowBanRolesConstraintV1  violation     msg   message    details   metadata        params    input parameters  resource    input review  resource type     google project iam binding   not resource change actions 0      delete   role    resource change after role  matches found    r   r    config pattern role   glob match params roles         r    mode    object get params   mode    allowlist    target match count mode  desired count   count matches found     desired count  message    output msg desired count  resource name  role   metadata         resource   resource name     role   role       target match count mode    0    mode     denylist    target match count mode    1    mode     allowlist    output msg 0  asset name  role    msg    msg    sprintf   v is in the banned list of IAM policy for  v    role  asset name     output msg 1  asset name  role    msg    msg    sprintf   v is NOT in the allowed list of IAM policy for  v    role  asset name     config pattern old pattern            old pattern          config pattern old pattern    old pattern    old pattern               There are two parts to this policy  We have a  violation  object  which contains line by line logic statements  The way rego works is that the policy will run line by line  and will  break  if any of the conditions don t pass  For instance  if the  resource type  is   not    google project iam binding   the rest of the rule will not be checked  and no violation gets reported  If you re familiar with Golang syntax  a      means  set the variable equal to the value on the right side  declaration  assignment  and redeclaration    A      means  check the equality of the left and right side   A     is for assignment   only    meaning that it won t infer the type of the variable  It can be used outside of functions of violations  however   The violation object will run with every  input review  object  which is each object in the  resource changes  array  evaluated one at a time    input parameters  will always be the same  as it contains the information passed in the  parameters  section of the constraint  AHA  That s why we use the constraint framework    In the case of this policy  if the number of matches we get between the input object and the constraint does not match the desired count  determined by whether the mode of the constraint is  allowlist  or  denylist    the  violation  will spit out a violation message  and an object containing metadata   The second part of the policy is for functions that the policy uses in the violation  You ll see duplicates  like  target match count   for example  These duplicative functions allow us to return a different value based on the input  In the case of  target match count   if the mode is of type   denylist   it will return a  0   So  if we end up getting one hit between our  input review  object  the actual count   will not   equal our desired count  and the violation will trigger  and send a violation message to stdout       Inlining Rego Policies  This is the easiest part of the development process  Once you ve created a new policy  if you ve cloned this repo  simply run  make build  to inline the policy into its accompanying constraint template  Remember the first line of the rego policy   package templates gcp TFGCPIAMAllowBanRolesConstraintV1   You ll need to make sure that you create a constraint template with the  kind  property under spec  crd  spec  names  kind set to  TFGCPIAMAllowBanRolesConstraintV1   You can see this  and other details  in the ConstraintTemplate that accompanys this policy       yaml  apiVersion  templates gatekeeper sh v1beta1 kind  ConstraintTemplate metadata    name  tfgcpiamallowbanrolesconstraintv1 spec    crd      spec        names          kind  TFGCPIAMAllowBanRolesConstraintV1       validation          openAPIV3Schema            properties              mode                description   Enforcement mode  defaults to allow                type  string               enum   denylist  allowlist              roles                description   Roles to be allowed or banned                 ex  roles owner  Wildcards     supported                type  array               items                  type  string   targets        target  validation resourcechange terraform cloud google com       rego             INLINE  validator my rego rule rego                rego code           ENDINLINE      You can see that the ConstraintTemplate defines a set of properties that constraints built from this template must abide by  Look at a constraint that uses this template to get a better idea of how the two interract       yaml  apiVersion  constraints gatekeeper sh v1beta1 kind  TFGCPIAMAllowBanRolesConstraintV1 metadata    name  iam allow roles   annotations      description  Allow only the listed IAM role bindings to be created  This       constraint is member agnostic  spec    severity  high   match      target       ref     definitions io k8s cli setters target            organizations         exclude       optional  default is no exclusions   parameters      mode   allowlist      roles           roles logging viewer           roles resourcemanager projectIamAdmin       Notice that the constraint defines a set of parameters  including  mode  and  roles   If you look at the ConstraintTempalte  you can see that these two fields are defined and described   Mode  is a string enumerable that  must  be either denylist or allowlist   gcloud beta terraform vet  will actually error out if this is not upheld in the associated constraint s       Additional Resources  Documentation on the Constraint Framework and  gcloud beta terraform vet  is pretty widespread  but this document provides some application specific information for using the  validation resourcechange terraform cloud google com  target for validating terraform resource changes  Documentation on this use case does exist  and can be found  here  https   cloud google com docs terraform policy validation create terraform constraints    When it comes time to create your own rego policies  I also recommend using the  Rego Playground  https   play openpolicyagent org   to first get a hang of the Rego language  Developing Rego in a local environment is really challenging  as debugging and tracing features must be integrated manually  usually with a log statement  like  trace sprintf  Value   s    value      However  the very nature of how Rego runs makes this extremely challenging  becasue if your violation exits before you even get to a log line  the trace will never be printed to stdout  and you won t easily be able to see what s going wrong in your policies   In my early development of this policy library  I created a couple of playgrounds in order to get a hang of Rego  Here s  an example  https   play openpolicyagent org p HzzUikhvQ4  of that  where I tested the logic for validating permissions on IAM Custom Roles  Take a look if you re interested  Although keep in mind that this wouldn t work if brought into the policy library  as this policy is unsuported by the Gatekeeper v1beta1 API Version  You would need to change the  deny  rule to  violation  and use updated Rego built ins  like  object get  instead of  get default     For me  the output gave me helpful hints on what was wrong with my rules and utility functions      Contact Info  Questions or comments  Please contact tdesrosi google com for this project  or validator support google com for information about the terraform validator project


GCP format and compressed using compression Application can be extended to support additional redis operations different encodings e g avro protobuf and Application Framework to execute various operations on redis to evaluate key performance metrics such as CPU Memory compression formats Check below for details Overview utilization Bytes transferred Time per command etc For complete list of metrics refer Framework currently supports few operations set get push pop with sample payload Payload is encoded in

Answer:
## Overview
  Application Framework to execute various operations on redis to evaluate key performance metrics such as CPU & Memory
  utilization, Bytes transferred, Time per command etc. For complete list of metrics refer [MemoryStore Redis Metrics](https://cloud.google.com/memorystore/docs/redis/supported-monitoring-metrics)

  Framework currently supports few operations (set/get, push & pop) with sample payload. Payload is encoded in
  [message pack](https://msgpack.org/index.html) format and compressed using [LZ4](https://lz4.org/) compression.

  Application can be extended to support additional redis operations, different encodings (e.g: avro, protobuf) and
  compression formats. Check below [section](#ExtendApplication) for details.

## Getting Started

### System Requirements

* Java 11
* Maven 3
* [gcloud CLI](https://cloud.google.com/sdk/gcloud)

### Building Jar
- Execute the below script from the project root directory to build uber jar:

```bash
$ ./scripts/build-jar.sh
```
 Successful execution of the script will generate the jar in the path:```{project_dir}/artifacts/redis-benchmarks-${version}.jar```

### Building Docker image
Docker image can be built and pushed to [google cloud artifact registry](https://cloud.google.com/artifact-registry).

Prior to building the docker image, follow the below steps:
- [Enable Artifact Registry](https://cloud.google.com/artifact-registry/docs/enable-service)

- Create a new repository called ```benchmarks``` using the command:
```bash
$ location="us-central1"
$ gcloud artifacts repositories create benchmarks \
      --repository-format=docker \
      --location=${location} \
      --description="Contains docker images to execute benchmark tests."
```

- Execute the below script from the project root directory to build & push docker image:
```bash
$ ./scripts/build-image.sh
```
Successful execution of the script will push the image to artifact registry.

- Image can be pulled using below command:
```bash
$ project_id=<<gcp-project>>
$ docker pull us-central1-docker.pkg.dev/${project_id}/benchmarks/redis-benchmark:latest
```

## Executing Application

### Application options
Following are the options that can be supplied while executing the application:

| Name                 | Description                                                                                | Optional | Default Value |
|----------------------|:----------------------------------------------------------------------------------------------|:---------|:--------------|
| Project              | Cloud Project identifier                                                                      | N        |               |
| hostname             | Redis host name                                                                               | Y        | localhost     |
| port                 | Redis port number                                                                             | Y        | 6379          |
| runduration_minutes  | Amount of time (in minutes) to run the application                                            | Y        | 1             |
| cpu_scaling_factor   | Determines the parallelism (i.e tasks) by multiplying cpu_scaling_factor with available cores | Y        | 1             |
| write_ratio          | Determines the percent of writes compared to reads. Default is 20% writes and 80% reads.      | Y        | 0.2           |
| task_types           | Task types to be executed. Can be supplied 1 or more as comma seperated values.               | N        |               |

### Run application
```bash
$ PROJECT_ID=<<gcp-project>>
$ RUNDURATION_MINUTES=1
$ CPU_SCALING_FACTOR=1
$ WRITE_RATIO=0.2
$ REDIS_HOST=localhost
$ REDIS_PORT=6379
$ TASK_TYPES="SetGet,ListOps"

$ java -jar ./artifacts/redis-benchmarks-1.0.jar \
--project=${PROJECT_ID} \
--runduration_minutes=${RUNDURATION_MINUTES} \
--cpu_scaling_factor=${CPU_SCALING_FACTOR} \
--write_ratio=${WRITE_RATIO} \
--task_types=${TASK_TYPES} \
--hostname=${REDIS_HOST} \
--port=${REDIS_PORT}
```

### Output
Post completion, the application will display the number of writes, reads, cache hits and misses for each task.

![Output.png](img/Output.png)

## Extending Application

Application can be extended to support additional payloads, encodings and compression formats.

### Adding new payload
Create a new class similar to [profile](./src/main/java/com/google/cloud/pso/benchmarks/redis/model/Profile.java) that
implements Payload interface

### Using different encoding
Create a new class similar to [MessagePack](./src/main/java/com/google/cloud/pso/benchmarks/redis/serde/MessagePack.java)
that implements EncDecoder interface

### Using different Compression
Create a new class similar to [LZ4Compression](./src/main/java/com/google/cloud/pso/benchmarks/redis/compression/LZ4Compression.java)
that implements Compression interface

### Adding additional tasks
Create a new class similar to [SetGetTask](./src/main/java/com/google/cloud/pso/benchmarks/redis/tasks/SetGetTask.java)
that implements RedisTask interface

- Payload can be generated using [PayloadGenerator](./src/main/java/com/google/cloud/pso/benchmarks/redis/PayloadGenerator.java)
that accepts payload type, encoding and compression as input parameters.

- Tasks can be created with appropriate payloads and can be supplied to the workload executior. Check **createRedisTask** method
 in [WorkloadExecutor](./src/main/java/com/google/cloud/pso/benchmarks/redis/WorkloadExecutor.java) for reference.

## Disclaimer

This project is not an official Google project. It is not supported by Google and disclaims all warranties as to its
quality, merchantability, or fitness for a particular purpose.

Overview   Application Framework to execute various operations on redis to evaluate key performance metrics such as CPU   Memory   utilization  Bytes transferred  Time per command etc  For complete list of metrics refer  MemoryStore Redis Metrics  https   cloud google com memorystore docs redis supported monitoring metrics     Framework currently supports few operations  set get  push   pop  with sample payload  Payload is encoded in    message pack  https   msgpack org index html  format and compressed using  LZ4  https   lz4 org   compression     Application can be extended to support additional redis operations  different encodings  e g  avro  protobuf  and   compression formats  Check below  section   ExtendApplication  for details      Getting Started      System Requirements    Java 11   Maven 3    gcloud CLI  https   cloud google com sdk gcloud       Building Jar   Execute the below script from the project root directory to build uber jar      bash     scripts build jar sh      Successful execution of the script will generate the jar in the path     project dir  artifacts redis benchmarks   version  jar         Building Docker image Docker image can be built and pushed to  google cloud artifact registry  https   cloud google com artifact registry    Prior to building the docker image  follow the below steps     Enable Artifact Registry  https   cloud google com artifact registry docs enable service     Create a new repository called    benchmarks    using the command     bash   location  us central1    gcloud artifacts repositories create benchmarks           repository format docker           location   location            description  Contains docker images to execute benchmark tests          Execute the below script from the project root directory to build   push docker image     bash     scripts build image sh     Successful execution of the script will push the image to artifact registry     Image can be pulled using below command     bash   project id   gcp project     docker pull us central1 docker pkg dev   project id  benchmarks redis benchmark latest         Executing Application      Application options Following are the options that can be supplied while executing the application     Name                   Description                                                                                Optional   Default Value                                                                                Project                Cloud Project identifier                                                                        N                            hostname               Redis host name                                                                                Y          localhost         port                   Redis port number                                                                               Y          6379              runduration minutes    Amount of time  in minutes  to run the application                                              Y          1                 cpu scaling factor     Determines the parallelism  i e tasks  by multiplying cpu scaling factor with available cores   Y          1                 write ratio            Determines the percent of writes compared to reads  Default is 20  writes and 80  reads         Y          0 2               task types             Task types to be executed  Can be supplied 1 or more as comma seperated values                  N                               Run application    bash   PROJECT ID   gcp project     RUNDURATION MINUTES 1   CPU SCALING FACTOR 1   WRITE RATIO 0 2   REDIS HOST localhost   REDIS PORT 6379   TASK TYPES  SetGet ListOps     java  jar   artifacts redis benchmarks 1 0 jar     project   PROJECT ID      runduration minutes   RUNDURATION MINUTES      cpu scaling factor   CPU SCALING FACTOR      write ratio   WRITE RATIO      task types   TASK TYPES      hostname   REDIS HOST      port   REDIS PORT           Output Post completion  the application will display the number of writes  reads  cache hits and misses for each task     Output png  img Output png       a name  ExtendApplication  Extending Application  a   Application can be extended to support additional payloads  encodings and compression formats       Adding new payload Create a new class similar to  profile    src main java com google cloud pso benchmarks redis model Profile java  that implements Payload interface      Using different encoding Create a new class similar to  MessagePack    src main java com google cloud pso benchmarks redis serde MessagePack java  that implements EncDecoder interface      Using different Compression Create a new class similar to  LZ4Compression    src main java com google cloud pso benchmarks redis compression LZ4Compression java  that implements Compression interface      Adding additional tasks Create a new class similar to  SetGetTask    src main java com google cloud pso benchmarks redis tasks SetGetTask java  that implements RedisTask interface    Payload can be generated using  PayloadGenerator    src main java com google cloud pso benchmarks redis PayloadGenerator java  that accepts payload type  encoding and compression as input parameters     Tasks can be created with appropriate payloads and can be supplied to the workload executior  Check   createRedisTask   method  in  WorkloadExecutor    src main java com google cloud pso benchmarks redis WorkloadExecutor java  for reference       Disclaimer  This project is not an official Google project  It is not supported by Google and disclaims all warranties as to its quality  merchantability  or fitness for a particular purpose


GCP kafka2avro batch mode pipeline This example contains two Dataflow pipelines reads objects from Kafka convert them to Avro and write the generates a set of objects and write them to Kafka This is a output to Google Cloud Storage This is a streaming mode pipeline This example shows how to use Apache Beam and SCIO to read objects from a Kafka topic and serialize them encoded as Avro files in Google Cloud Storage

Answer:
# kafka2avro

This example shows how to use Apache Beam and SCIO to read objects from a Kafka
topic, and serialize them encoded as Avro files in Google Cloud Storage.

This example contains two Dataflow pipelines:
* [Object2Kafka](src/main/scala/com/google/cloud/pso/kafka2avro/Object2Kafka.scala): generates a set of objects and write them to Kafka. This is a
  batch mode pipeline.
* [Kafka2Avro](src/main/scala/com/google/cloud/pso/kafka2avro/Kafka2Avro.scala): reads objects from Kafka, convert them to Avro, and write the
  output to Google Cloud Storage. This is a streaming mode pipeline

## Configuration

Before compiling and generating your package, you need to change some options in
[`src/main/resources/application.conf`](src/main/resources/application.conf):

* `broker`: String with the address of the Kafka brokers.
* `dest-bucket`: The name of the bucket where the Avro files will be written
* `dest-path`: The directories structure where the Avro files will be written
  (e.g. blank to write in the top level dir in the bucket, or anything like
  a/b/c).
* `kafka-topic`: The name of the topic where the objects are written to, or read
  from.
* `num-demo-objects`: Number of objects that will be generated by the Object2Kafka
  pipeline, these objects can be read with the Kafka2Avro pipeline to test that
  everything is working as expected.

The configuration file follows [the HOCON format](https://github.com/lightbend/config/blob/master/README.md#using-hocon-the-json-superset).

Here is a sample configuration file with all the options set:

```bash
broker = "1.2.3.4:9092"
dest-bucket = "my-bucket-in-gcs"
dest-path = "persisted/fromkafka/avro/"
kafka-topic = "my_kafka_topic"
num-demo-objects = 500  # comments are allowed in the config file
```

## Pre-requirements

### Build tool

This example is written in Scala and uses SBT as build tool.

You need to have SBT >= 1.0 installed. You can download SBT from https://www.scala-sbt.org/

The Scala version is 2.12.8. If you have the JDK > 1.8 installed, SBT should automatically download the Scala compiler.

## Compile

Run `sbt` in the top sources folder.

Inside sbt, download all the dependencies:

```
sbt:kafka2avro> update
```

and then compile

```
sbt:kafka2avro> compile
```

## Deploy and run

If you have managed to compile the code, you can generate a JAR package to be
deployed on Dataflow, with:

```
sbt:kafka2avro> pack
```

This will generate a set of JAR files in `target/pack/lib`

### Running the Object2Kafka pipeline

This is batch pipeline, provided just an example to populate Kafka and test the
streaming pipeline.

Once you have generated the JAR file using the `pack` command inside SBT, you
can now launch the job in Dataflow to populate Kafka with some demo
objects. Using Java 1.8, run the following command. Notice that you have to set
the project id, and a location in a GCS bucket to store the JARs imported by Dataflow:

```
CLASSPATH="target/pack/lib/*" java com.google.cloud.pso.kafka2avro.Object2Kafka --exec.mainClass=com.google.cloud.pso.kafka2avro.Object2Kafka --project=YOUR_PROJECT_ID --stagingLocation="gs://YOUR_BUCKET/YOUR_STAGING_LOCATION" --runner=DataflowRunner
```

### Running the Kafka2Avro pipeline

This is a streaming pipeline. It will keep running unless you cancel it. The
default windowing policy is to group messages every 2 minutes, in a fixed
window. To change the policy, please see
[the function `windowIn` in `Kafka2Avro.scala`](src/main/scala/com/google/cloud/pso/kafka2avro/Kafka2Avro.scala#L60-L70).

Once you have generated the JAR file using the `pack` command inside SBT, you
can now launch the job in Dataflow to populate Kafka with some demo
objects. Using Java 1.8, run the following command. Notice that you have to set
the project id, and a location in a GCS bucket to store the JARs imported by
Dataflow:

```
CLASSPATH="target/pack/lib/*" java com.google.cloud.pso.kafka2avro.Kafka2Avro --exec.mainClass=com.google.cloud.pso.kafka2avro.Kafka2Avro --project=YOUR_PROJECT_ID --stagingLocation="gs://YOUR_BUCKET/YOUR_STAGING_LOCATION" --runner=DataflowRunner
```

Please remember that the machine running the JAR may need to have connectivity
to the Kafka cluster in order to retrieve some metadata, prior to launching the
pipeline in Dataflow.

**Remember that this is a streaming pipeline, it will keep running forever until
you cancel or stop it.**

### Wrong filenames for some dependencies

In some cases, some dependencies may be downloaded with wrong filenames. For
instance, containing symbols that need to be escaped. Importing these JARs in
the job in Dataflow will fail.

If when running your Dataflow job, it fails before it is launched because it
cannot copy some dependencies, change the name of the offending files so they
don't contain symbols. For instance:

```
mv target/pack/lib/netty-codec-http2-\[4.1.25.Final,4.1.25.Final\].jar target/pack/lib/netty-codec-http2.jar
```

## Continuous Integration

This example includes [a configuration file for Cloud Build](cloudbuild.yaml),
so you can use it to run the unit tests with every commit done to your repository.
To use this configuration file:

* Add your sources to a Git repository (either in Bitbucket, Github or Google
  Cloud Source).
* Configure a trigger in Google Cloud Build linked to your Git repository.
* Set the path for the configuration file to [`cloudbuild.yaml`](cloudbuild.yaml).

The included configuration file will do the following steps:
* Download a cache for Ivy2 from a Google Cloud Storage bucket named
  YOURPROJECT_cache, where `YOURPROJECT` is your GCP project id.
* Compile and test the Scala code.
* Generate a package.
* Upload the new Ivy2 cache to the same bucket as in the first step.
* Upload the generated package and all its dependencies to a bucket named
  YOURPROJECT_pkgs, where `YOURPROJECT` is your GCP project id.

So these default steps will try to write to and read from two different buckets in
Google Cloud Storage. Please either create these buckets in your GCP project, or
change the configuration.

Please note that you need to build and include the `scala-sbt` Cloud Builder in
order to use this configuration file.

* Make sure you have the Google Cloud SDK configured with your credentials and
  project
* Download the sources from
  [GoogleCloudPlatform/cloud-builders-community/tree/master/scala-sbt](https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/scala-sbt)
* And then in the `scala-sbt` sources dir, run `gcloud builds submit
  . --config=cloudbuild.yaml` to add the builder to your GCP project. You only
  need to do this once.

kafka2avro  This example shows how to use Apache Beam and SCIO to read objects from a Kafka topic  and serialize them encoded as Avro files in Google Cloud Storage   This example contains two Dataflow pipelines     Object2Kafka  src main scala com google cloud pso kafka2avro Object2Kafka scala   generates a set of objects and write them to Kafka  This is a   batch mode pipeline     Kafka2Avro  src main scala com google cloud pso kafka2avro Kafka2Avro scala   reads objects from Kafka  convert them to Avro  and write the   output to Google Cloud Storage  This is a streaming mode pipeline      Configuration  Before compiling and generating your package  you need to change some options in   src main resources application conf   src main resources application conf       broker   String with the address of the Kafka brokers     dest bucket   The name of the bucket where the Avro files will be written    dest path   The directories structure where the Avro files will be written    e g  blank to write in the top level dir in the bucket  or anything like   a b c      kafka topic   The name of the topic where the objects are written to  or read   from     num demo objects   Number of objects that will be generated by the Object2Kafka   pipeline  these objects can be read with the Kafka2Avro pipeline to test that   everything is working as expected   The configuration file follows  the HOCON format  https   github com lightbend config blob master README md using hocon the json superset    Here is a sample configuration file with all the options set      bash broker    1 2 3 4 9092  dest bucket    my bucket in gcs  dest path    persisted fromkafka avro   kafka topic    my kafka topic  num demo objects   500    comments are allowed in the config file         Pre requirements      Build tool  This example is written in Scala and uses SBT as build tool   You need to have SBT    1 0 installed  You can download SBT from https   www scala sbt org   The Scala version is 2 12 8  If you have the JDK   1 8 installed  SBT should automatically download the Scala compiler      Compile  Run  sbt  in the top sources folder   Inside sbt  download all the dependencies       sbt kafka2avro  update      and then compile      sbt kafka2avro  compile         Deploy and run  If you have managed to compile the code  you can generate a JAR package to be deployed on Dataflow  with       sbt kafka2avro  pack      This will generate a set of JAR files in  target pack lib       Running the Object2Kafka pipeline  This is batch pipeline  provided just an example to populate Kafka and test the streaming pipeline   Once you have generated the JAR file using the  pack  command inside SBT  you can now launch the job in Dataflow to populate Kafka with some demo objects  Using Java 1 8  run the following command  Notice that you have to set the project id  and a location in a GCS bucket to store the JARs imported by Dataflow       CLASSPATH  target pack lib    java com google cloud pso kafka2avro Object2Kafka   exec mainClass com google cloud pso kafka2avro Object2Kafka   project YOUR PROJECT ID   stagingLocation  gs   YOUR BUCKET YOUR STAGING LOCATION    runner DataflowRunner          Running the Kafka2Avro pipeline  This is a streaming pipeline  It will keep running unless you cancel it  The default windowing policy is to group messages every 2 minutes  in a fixed window  To change the policy  please see  the function  windowIn  in  Kafka2Avro scala   src main scala com google cloud pso kafka2avro Kafka2Avro scala L60 L70    Once you have generated the JAR file using the  pack  command inside SBT  you can now launch the job in Dataflow to populate Kafka with some demo objects  Using Java 1 8  run the following command  Notice that you have to set the project id  and a location in a GCS bucket to store the JARs imported by Dataflow       CLASSPATH  target pack lib    java com google cloud pso kafka2avro Kafka2Avro   exec mainClass com google cloud pso kafka2avro Kafka2Avro   project YOUR PROJECT ID   stagingLocation  gs   YOUR BUCKET YOUR STAGING LOCATION    runner DataflowRunner      Please remember that the machine running the JAR may need to have connectivity to the Kafka cluster in order to retrieve some metadata  prior to launching the pipeline in Dataflow     Remember that this is a streaming pipeline  it will keep running forever until you cancel or stop it         Wrong filenames for some dependencies  In some cases  some dependencies may be downloaded with wrong filenames  For instance  containing symbols that need to be escaped  Importing these JARs in the job in Dataflow will fail   If when running your Dataflow job  it fails before it is launched because it cannot copy some dependencies  change the name of the offending files so they don t contain symbols  For instance       mv target pack lib netty codec http2   4 1 25 Final 4 1 25 Final   jar target pack lib netty codec http2 jar         Continuous Integration  This example includes  a configuration file for Cloud Build  cloudbuild yaml   so you can use it to run the unit tests with every commit done to your repository  To use this configuration file     Add your sources to a Git repository  either in Bitbucket  Github or Google   Cloud Source     Configure a trigger in Google Cloud Build linked to your Git repository    Set the path for the configuration file to   cloudbuild yaml   cloudbuild yaml    The included configuration file will do the following steps    Download a cache for Ivy2 from a Google Cloud Storage bucket named   YOURPROJECT cache  where  YOURPROJECT  is your GCP project id    Compile and test the Scala code    Generate a package    Upload the new Ivy2 cache to the same bucket as in the first step    Upload the generated package and all its dependencies to a bucket named   YOURPROJECT pkgs  where  YOURPROJECT  is your GCP project id   So these default steps will try to write to and read from two different buckets in Google Cloud Storage  Please either create these buckets in your GCP project  or change the configuration   Please note that you need to build and include the  scala sbt  Cloud Builder in order to use this configuration file     Make sure you have the Google Cloud SDK configured with your credentials and   project   Download the sources from    GoogleCloudPlatform cloud builders community tree master scala sbt  https   github com GoogleCloudPlatform cloud builders community tree master scala sbt    And then in the  scala sbt  sources dir  run  gcloud builds submit       config cloudbuild yaml  to add the builder to your GCP project  You only   need to do this once


GCP This approach genrally has following benefits With multi folder repositories it s possible to combine similar IaC into a single repository and or centralize the mangement of multiple business units code Organizing code across multiple folders within a single version control repositiroy such as github is a very common practice and we re referring this as multi folder repository Selective deployment approach lets you find the folders changed within your repository and only run the logic for changed folders Selective deployment Reduced overhead in managing multiple CI CD pipelines

Answer:
# Selective deployment

Organizing code across multiple folders within a single version control repositiroy such as github is a very common practice, and we're referring this as multi-folder repository.

**Selective deployment** approach lets you find the folders changed within your repository and only run the logic for changed folders.

With multi-folder repositories, it's possible to combine similar IaC into a single repository and/or centralize the mangement of multiple business units code.
This approach genrally has following benefits:

- Reduced overhead in managing multiple CI/CD pipelines
- Better code visibility
- Reduces overhead in managing multiple ACLs for similar code

Example multi-folder repository structure

```txt
└── single-repo
    ├── build-files
    │   └── compile.sh
    ├── build.yaml
    └── user-resources
        ├── business-unit1
        │   ├── dev-env
        │   ├── prod-env
        │   └── qa-env
        └── business-unit2
            ├── dev-env
            ├── prod-env
            └── qa-env
```

``` Note: A Mono repo is always a multi-folder repository, however vice versa is not always true. Checkout https://www.hashicorp.com/blog/terraform-mono-repo-vs-multi-repo-the-great-debate article for more information on mono vs multi repos for IaC.```

## Solution

This can be addressed in multiple different ways, and our approach is as below:

### Step 1: Find the commit associated with last successful build.

```sh
nth_successful_commit() {
  local n=$1  # n=1 --> Last successful commit.
  local trigger_name=$2
  local project=$3

  local trigger_id=$(get_trigger_value $trigger_name $project "id")
  local nth_successful_build=$(gcloud builds list --filter "buildTriggerId=$trigger_id AND STATUS=(SUCCESS)" --format "value(id)" --limit=$build_find_limit --project $project | awk "NR==$n") || exit 1

  local nth_successful_commit=$(gcloud builds describe $nth_successful_build --format "value(substitutions.COMMIT_SHA)" --project $project) || exit 1
  echo $nth_successful_commit
}
```

### Step 2: Find the differece between current commit and last successful commit.

```sh
previous_commit_sha=$(nth_successful_commit 1 $apply_trigger_name $project) || exit 1

git diff --name-only ${previous_commit_sha} ${commit_sha} | sort -u > $logs_dir/diff.log || exit 1
```
This step will give you a list of files/folders that were modified after the commit associated with last successful build.

### Step 3: Iterate over changed folders.

You can now iterate over only the changed folders received from Step 2 in the $logs_dir/diff.log file.

## Implementation steps

### Pre-requisites

- A cloud source repository (or any other source control repositories).
- A cloud build configuration file with basic steps to be executed on a single folder of the repository.

```Note: We are assuming that the pipeline is built on Google cloud build. If the pipeline is built on other platforms, you might need to retrofit this solution accordingly.```

### Setup Clooud build
Add the right values for the below CloudBuild substitution variables in the `cloudbuild.yaml` file.

```sh
  _TF_SA_EMAIL: ''
  _PREVIOUS_COMMIT_SHA: ''
  _RUN_ALL_PROJECTS: 'false'
```

- `_TF_SA_EMAIL` is the GCP service_account with the necessary IAM permissions that the terraform impersonates. Cloudbuild default SA should have [roles/iam.serviceAccountTokenCreator](https://cloud.google.com/iam/docs/service-accounts#token-creator-role) on the `_TF_SA_EMAIL` service_account.
- `_PREVIOUS_COMMIT_SHA` is the github commit_sha that is used for explicitly checking delta changes between this commit and the latest commit instead of automatically detecting last sucessful commit based on a successful cloudbuild execution. Especially this would be required for first time execution when there is not a successful cloudbuild execution.
- `_RUN_ALL_PROJECTS` is to force executing through all folders. Once is a while this is required:
    - for deploying a change that imapcts all folders such as a terraform module commonly used by code in all/multiple folders.
    - for detecting and fixing any configurations drifts, especially when manual changes are performed.

## Important points

Use unshallow copy of git clone.

Cloud build in its default behaviour uses shallow a copy of the repository (i.e. only the code associated with the commit with which the current build was triggered). Shallow copy prevents us from performing git operations like git diff. However, we can use following step in the cloud build to fetch unshallow copy:

```yaml
  - id: 'unshallow'
    name: gcr.io/cloud-builders/git
    args: ['fetch', '--unshallow']
``

Selective deployment  Organizing code across multiple folders within a single version control repositiroy such as github is a very common practice  and we re referring this as multi folder repository       Selective deployment   approach lets you find the folders changed within your repository and only run the logic for changed folders     With multi folder repositories  it s possible to combine similar IaC into a single repository and or centralize the mangement of multiple business units code     This approach genrally has following benefits     Reduced overhead in managing multiple CI CD pipelines   Better code visibility   Reduces overhead in managing multiple ACLs for similar code  Example multi folder repository structure     txt     single repo         build files             compile sh         build yaml         user resources             business unit1                 dev env                 prod env                 qa env             business unit2                 dev env                 prod env                 qa env          Note  A Mono repo is always a multi folder repository  however vice versa is not always true  Checkout https   www hashicorp com blog terraform mono repo vs multi repo the great debate article for more information on mono vs multi repos for IaC          Solution  This can be addressed in multiple different ways  and our approach is as below         Step 1  Find the commit associated with last successful build      sh nth successful commit       local n  1    n 1     Last successful commit    local trigger name  2   local project  3    local trigger id   get trigger value  trigger name  project  id     local nth successful build   gcloud builds list   filter  buildTriggerId  trigger id AND STATUS  SUCCESS     format  value id     limit  build find limit   project  project   awk  NR   n      exit 1    local nth successful commit   gcloud builds describe  nth successful build   format  value substitutions COMMIT SHA     project  project     exit 1   echo  nth successful commit            Step 2  Find the differece between current commit and last successful commit        sh previous commit sha   nth successful commit 1  apply trigger name  project     exit 1  git diff   name only   previous commit sha    commit sha    sort  u    logs dir diff log    exit 1     This step will give you a list of files folders that were modified after the commit associated with last successful build         Step 3  Iterate over changed folders     You can now iterate over only the changed folders received from Step 2 in the  logs dir diff log file      Implementation steps      Pre requisites    A cloud source repository  or any other source control repositories     A cloud build configuration file with basic steps to be executed on a single folder of the repository      Note  We are assuming that the pipeline is built on Google cloud build  If the pipeline is built on other platforms  you might need to retrofit this solution accordingly          Setup Clooud build Add the right values for the below CloudBuild substitution variables in the  cloudbuild yaml  file        sh    TF SA EMAIL        PREVIOUS COMMIT SHA        RUN ALL PROJECTS   false           TF SA EMAIL  is the GCP service account with the necessary IAM permissions that the terraform impersonates  Cloudbuild default SA should have  roles iam serviceAccountTokenCreator  https   cloud google com iam docs service accounts token creator role  on the   TF SA EMAIL  service account      PREVIOUS COMMIT SHA  is the github commit sha that is used for explicitly checking delta changes between this commit and the latest commit instead of automatically detecting last sucessful commit based on a successful cloudbuild execution  Especially this would be required for first time execution when there is not a successful cloudbuild execution        RUN ALL PROJECTS  is to force executing through all folders  Once is a while this is required        for deploying a change that imapcts all folders such as a terraform module commonly used by code in all multiple folders        for detecting and fixing any configurations drifts  especially when manual changes are performed      Important points  Use unshallow copy of git clone     Cloud build in its default behaviour uses shallow a copy of the repository  i e  only the code associated with the commit with which the current build was triggered   Shallow copy prevents us from performing git operations like git diff  However  we can use following step in the cloud build to fetch unshallow copy      yaml     id   unshallow      name  gcr io cloud builders git     args    fetch      unshallow