GCP This pre commit hook uses open source tools to provide developers with a way to validate Kubernetes manifests before changes are committed and pushed to a repository Left Shift Validation at Pre Commit Hook Using left shift validate means you learn if your deployments are going to fail on the order of seconds as it happens right before committing rather than minutes or hours after it has already undergone multiple parts or even most of your CI CD pipeline While these scripts were first designed to work in environemnts using Kustomize they ve been adapted to work for whatever file organization structure you provide uses git diff to find any and all staged yaml files and then will validate each against the Constraints and ConstraintTemplates you provide Policy checks are typically instantiated after code is pushed to the repository as it goes through each environment dev QA production etc and right before administration to the cluster The goal with this script is to validate at the pre commit hook stage applying your security and policy check even earlier shifting left than usual

Answer:
# Left-Shift Validation at Pre-Commit Hook

This pre-commit hook uses open-source tools to provide developers with a way to validate Kubernetes manifests before changes are committed and pushed to a repository.

Policy checks are typically instantiated after code is pushed to the repository, as it goes through each environment (dev, QA, production, etc.), and right before administration to the cluster. The goal with this script is to validate at the pre-commit hook stage, applying your security and policy check even earlier (shifting left) than usual.

Using left-shift validate means you learn *if your deployments are going to fail* on the order of seconds as it happens right before committing, rather than minutes or hours after it has already undergone multiple parts, or even most, of your CI/CD pipeline.

While these scripts were first designed to work in environemnts using Kustomize, they've been adapted to work for whatever file organization structure you provide. `validate.sh` uses git diff to find any and all staged yaml files, and then will validate each against the Constraints and ConstraintTemplates you provide.

---

## Setting Up left-shift validation

Using left-shift validation is simple!

### Initial installation

1. From this repository, you only need the following items:

- `validate.sh`
- `setup.sh`
- `constraints-and-templates/` directory

    *If you don't want to clone the whole project, you can use `wget` to download the specific files. For example:*

    `wget https://raw.githubusercontent.com/GoogleCloudPlatform/professional-services/main/examples/left-shift-validation-pre-commit-hook/validate.sh`

2. Place these items into the same directory as your `.git` folder. `setup.sh` will move `validate.sh` into your hooks from there.

3. Make both files executable, which can be accomplished by:

    `$ chmod +x setup.sh && chmod +x validate.sh`

4. After this step, run `setup.sh`

    `$ ./setup.sh`

5. Answer the prompts, and then you're done!

Now, whenever you go to commit code and updated Kubernetes yaml files are found, the pre-commit hook will test your changes against the policy constraints you've specified. Nifty!

**After initial installation:**

`setup.sh` is primarily for dependency installations and contains a guided walkthrough of obtaining the locations of your Constraint and ConstraintTemplates. If your policy constraints change, you can run this script again, which will automatically update dependencies as well.

The locations you specify for your Constraints, ConstraintTemplates, and (if using Kustomize) base Kustomization folder will be written into a settings file in `.oss_dependencies` for `validate.sh` (pre-commit hook) to use. You can directly change these variables as necessary without having to run `setup.sh`. The pre-commit hook will run every time you attempt a commit!

---

## Technical Overview

Left-shift validation is intended to run as a pre-commit hook, so it has been designed with two distinct Bash scripts, `setup.sh` and `validate.sh`. **Here's what they do:**

### setup.sh

1. Install or update dependencies.
2. Turn `validate.sh` into a pre commit hook.
3. Determine the locations of Constraints/ConstraintTemplates to use, then save the path/remote repository URL to `.env`.

### validate.sh (Pre-commit Hook)

1. When `git commit` runs, check if any yaml files have been updated.
2. (If you're using Kustomize) Run `kustomize` to build a unified yaml file for evaluation.
3. Gather the Constraints and ConstraintTemplates and save them to a local folder. Use `kpt` if from a remote repo.
4. Run `gator test`, which validates the unified yaml file against the policies (Constraints and ConstraintTemplates).
5. Fail the commit if violations are found. If there are no errors, continue the commit.

---

## Purpose

Organizations that deploy applications on Kubernetes clusters often use tools like [Open Policy Agent](https://www.openpolicyagent.org/) (OPA) or [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs) to enforce security and operational policies. These are often essential for a company to meet legal or business requirements, but they have the added benefit of empowering their developers by providing consistent feedback on their work by showcasing if it meets the security standards of their organization.

Typically, validation checks occur at the end of the CI/CD pipeline right before admission to deploy to a cluster, throughout the pipeline, or even in the repository, with automated code reviews. These checks are great, and using many in tandem is benificial for redundancy, but our goal is to reduce the potentially long wait times between when a developer submits their Kubernetes files and when those files either pass or fail policy reviews.
Left-shift validation intends to streamline the development process by providing actionable feedback as quickly as possible at the pre-commit hook stage (which one can consider even before the pipeline), which as early as you could go.

Here's what a typical CI/CD pipeline might look like with OSS policy validations throughout. In this case, we use Gatekeeper to define Constraints and ConstraintTemplates, and they are stored in a Git Repo that can be accessed by the pipeline (Jenkins, Google Cloud Build, etc.).

![A Sample Ci/CD Pipeline with Policy Validation Built-in](img/sample-cicd-pipeline.png)

And what left-shift validation does is extend these redundant validation steps into the developer's local development environment, much earlier in the pipeline, like so:

![Left-shift validation Architectural Diagram](img/leftshift-validate-architecture.png)

---

## Important Things to Note

### Left-shift validation is an **Enhancement**, not a Replacement

We do not intend for left-shift validation to replace other automated policy control systems. Instead, this is a project that can be used to help support developers who work on Kubernetes manifests by enhancing the delivery pipeline. Using left-shift validation means you learn *if your deployments are going to fail* on the order of seconds, rather than minutes or hours.

The only thing that happens if you don't use left-shift validation to shift left on automated policy validation is it takes longer for you to learn if you have a problem. That's all!

### Handling Dependencies

Left-shift validation uses the follwing dependencies:

| Name | Role |
| ----- | -----|
| [kpt](https://kpt.dev/) | Enables us to fetch directories from remote git repositories and use their contents in later steps. Kpt also provides easier integration with Kustomize. |
| [kustomize](https://github.com/kubernetes-sigs/kustomize) | Collates and hydrates raw yaml files into formats that work best with validation steps. |
| [gator](https://open-policy-agent.github.io/gatekeeper/website/docs/gator/) | Allows for evaluating Gatekeeper ConstraintTemplates and Constraints in a local environment. |

In order for left-shift validation to work, these tools must be installed on your system. `setup.sh` will install or update each tool, and they will be accessible to the validation script via a dependency folder. If you'd like to handle installation yourself, you may! As long as the commands are in your `$PATH`, `validate.sh` will recognize and use them.

---

### Deep-Dive

Let's go a bit further into how everything works together. The idea is that you can run `setup.sh` whenever you need to configure your pre-commit script. This can include changes like:

- Updating Dependencies (which will happen automatically anytime you run `setup.sh`)
- Resetting the default behavior of your pre-commit hook, if you make changes that break the code.
- Describing new Constraints and/or ConstraintTemplates to use. We have a collection of samples from the [OPA Gatekeeper Library](https://github.com/open-policy-agent/gatekeeper-library) that you can use, but you can also supply your own repository. If you have separate repositories for your Constraints and Templates, that is also supported.

`validate.sh` depends on `setup.sh` to take care of the more time-consuming steps in order to run as fast as possible. It's for this reason that `setup.sh` handles all aspects of setting up the environment, Then, `validate.sh` only needs to identify changed resources, gather Constraints and ConstraintTemplates with kpt, and finally run `gator test` to produce an outcome.

When configured as a pre-commit script (taken care of by `setup.sh`), `validate.sh` will take the locations of your Constraints and ConstraintTemplates, which you provided in `setup.sh`, and obtain those manifests. Whether they're stored in one or two repositories, stored locally, or if you want to continue with the sample policies in the OPA Gatekeeper Library, the script supports all of those combinations. Here's how that decision flow works:

![validate-decision-tree](img/validate-decision-tree.png)

---

## Cleanup

Done with left-shift validation? Uninstalling is easy! Since we installed dependencies from pre-compiled binaries, all we need to do is delete a few directories and files. You have two options here:

### Automatic Uninstall Using cleanup.sh

You can simply use `cleanup.sh`, which will automatically delete folders like `.oss_dependencies/` and any manifests, Constraints, or ConstraintTemplates that are still lingering around.

### IMPORTANT NOTE

When you install left-shift validation, it becomes `pre-commit` in the `.git/hooks/` directory. This script will delete the file, rather than renaming it by appending `.sample` to the filename.

### Manual Uninstall

To remove left-shift validation, you must delete all of the files that have been created. This includes:

- `setup.sh`
- `.oss_dependencies/*` (which can be found in the root directory of your project)

Then, you must go into your .git/hooks/ folder, and either delete `pre-commit`, or add ".sample" to the end of the filename, which tells Git not to run it in the future.

---

## Contact Info

We're always looking for help, or suggestions for improvement. Please feel free to reach out to us if you've got ideas or feedback!

- [Janine Bariuan](mailto:janinebariuan@google.com)
- [Thomas Desrosiers](mailto:tdesrosi@google.com)

Left Shift Validation at Pre Commit Hook  This pre commit hook uses open source tools to provide developers with a way to validate Kubernetes manifests before changes are committed and pushed to a repository   Policy checks are typically instantiated after code is pushed to the repository  as it goes through each environment  dev  QA  production  etc    and right before administration to the cluster  The goal with this script is to validate at the pre commit hook stage  applying your security and policy check even earlier  shifting left  than usual   Using left shift validate means you learn  if your deployments are going to fail  on the order of seconds as it happens right before committing  rather than minutes or hours after it has already undergone multiple parts  or even most  of your CI CD pipeline   While these scripts were first designed to work in environemnts using Kustomize  they ve been adapted to work for whatever file organization structure you provide   validate sh  uses git diff to find any and all staged yaml files  and then will validate each against the Constraints and ConstraintTemplates you provide           Setting Up left shift validation  Using left shift validation is simple       Initial installation  1  From this repository  you only need the following items      validate sh     setup sh     constraints and templates   directory       If you don t want to clone the whole project  you can use  wget  to download the specific files  For example         wget https   raw githubusercontent com GoogleCloudPlatform professional services main examples left shift validation pre commit hook validate sh   2  Place these items into the same directory as your   git  folder   setup sh  will move  validate sh  into your hooks from there   3  Make both files executable  which can be accomplished by          chmod  x setup sh    chmod  x validate sh   4  After this step  run  setup sh            setup sh   5  Answer the prompts  and then you re done   Now  whenever you go to commit code and updated Kubernetes yaml files are found  the pre commit hook will test your changes against the policy constraints you ve specified  Nifty     After initial installation      setup sh  is primarily for dependency installations and contains a guided walkthrough of obtaining the locations of your Constraint and ConstraintTemplates  If your policy constraints change  you can run this script again  which will automatically update dependencies as well   The locations you specify for your Constraints  ConstraintTemplates  and  if using Kustomize  base Kustomization folder will be written into a settings file in   oss dependencies  for  validate sh   pre commit hook  to use  You can directly change these variables as necessary without having to run  setup sh   The pre commit hook will run every time you attempt a commit           Technical Overview  Left shift validation is intended to run as a pre commit hook  so it has been designed with two distinct Bash scripts   setup sh  and  validate sh     Here s what they do         setup sh  1  Install or update dependencies  2  Turn  validate sh  into a pre commit hook  3  Determine the locations of Constraints ConstraintTemplates to use  then save the path remote repository URL to   env        validate sh  Pre commit Hook   1  When  git commit  runs  check if any yaml files have been updated  2   If you re using Kustomize  Run  kustomize  to build a unified yaml file for evaluation  3  Gather the Constraints and ConstraintTemplates and save them to a local folder  Use  kpt  if from a remote repo  4  Run  gator test   which validates the unified yaml file against the policies  Constraints and ConstraintTemplates   5  Fail the commit if violations are found  If there are no errors  continue the commit           Purpose  Organizations that deploy applications on Kubernetes clusters often use tools like  Open Policy Agent  https   www openpolicyagent org    OPA  or  Gatekeeper  https   open policy agent github io gatekeeper website docs  to enforce security and operational policies  These are often essential for a company to meet legal or business requirements  but they have the added benefit of empowering their developers by providing consistent feedback on their work by showcasing if it meets the security standards of their organization   Typically  validation checks occur at the end of the CI CD pipeline right before admission to deploy to a cluster  throughout the pipeline  or even in the repository  with automated code reviews  These checks are great  and using many in tandem is benificial for redundancy  but our goal is to reduce the potentially long wait times between when a developer submits their Kubernetes files and when those files either pass or fail policy reviews  Left shift validation intends to streamline the development process by providing actionable feedback as quickly as possible at the pre commit hook stage  which one can consider even before the pipeline   which as early as you could go   Here s what a typical CI CD pipeline might look like with OSS policy validations throughout  In this case  we use Gatekeeper to define Constraints and ConstraintTemplates  and they are stored in a Git Repo that can be accessed by the pipeline  Jenkins  Google Cloud Build  etc       A Sample Ci CD Pipeline with Policy Validation Built in  img sample cicd pipeline png   And what left shift validation does is extend these redundant validation steps into the developer s local development environment  much earlier in the pipeline  like so     Left shift validation Architectural Diagram  img leftshift validate architecture png           Important Things to Note      Left shift validation is an   Enhancement    not a Replacement  We do not intend for left shift validation to replace other automated policy control systems  Instead  this is a project that can be used to help support developers who work on Kubernetes manifests by enhancing the delivery pipeline  Using left shift validation means you learn  if your deployments are going to fail  on the order of seconds  rather than minutes or hours   The only thing that happens if you don t use left shift validation to shift left on automated policy validation is it takes longer for you to learn if you have a problem  That s all       Handling Dependencies  Left shift validation uses the follwing dependencies     Name   Role                       kpt  https   kpt dev     Enables us to fetch directories from remote git repositories and use their contents in later steps  Kpt also provides easier integration with Kustomize       kustomize  https   github com kubernetes sigs kustomize    Collates and hydrates raw yaml files into formats that work best with validation steps       gator  https   open policy agent github io gatekeeper website docs gator     Allows for evaluating Gatekeeper ConstraintTemplates and Constraints in a local environment     In order for left shift validation to work  these tools must be installed on your system   setup sh  will install or update each tool  and they will be accessible to the validation script via a dependency folder  If you d like to handle installation yourself  you may  As long as the commands are in your   PATH    validate sh  will recognize and use them            Deep Dive  Let s go a bit further into how everything works together  The idea is that you can run  setup sh  whenever you need to configure your pre commit script  This can include changes like     Updating Dependencies  which will happen automatically anytime you run  setup sh     Resetting the default behavior of your pre commit hook  if you make changes that break the code    Describing new Constraints and or ConstraintTemplates to use  We have a collection of samples from the  OPA Gatekeeper Library  https   github com open policy agent gatekeeper library  that you can use  but you can also supply your own repository  If you have separate repositories for your Constraints and Templates  that is also supported    validate sh  depends on  setup sh  to take care of the more time consuming steps in order to run as fast as possible  It s for this reason that  setup sh  handles all aspects of setting up the environment  Then   validate sh  only needs to identify changed resources  gather Constraints and ConstraintTemplates with kpt  and finally run  gator test  to produce an outcome   When configured as a pre commit script  taken care of by  setup sh     validate sh  will take the locations of your Constraints and ConstraintTemplates  which you provided in  setup sh   and obtain those manifests  Whether they re stored in one or two repositories  stored locally  or if you want to continue with the sample policies in the OPA Gatekeeper Library  the script supports all of those combinations  Here s how that decision flow works     validate decision tree  img validate decision tree png           Cleanup  Done with left shift validation  Uninstalling is easy  Since we installed dependencies from pre compiled binaries  all we need to do is delete a few directories and files  You have two options here       Automatic Uninstall Using cleanup sh  You can simply use  cleanup sh   which will automatically delete folders like   oss dependencies   and any manifests  Constraints  or ConstraintTemplates that are still lingering around       IMPORTANT NOTE  When you install left shift validation  it becomes  pre commit  in the   git hooks   directory  This script will delete the file  rather than renaming it by appending   sample  to the filename       Manual Uninstall  To remove left shift validation  you must delete all of the files that have been created  This includes      setup sh      oss dependencies     which can be found in the root directory of your project   Then  you must go into your  git hooks  folder  and either delete  pre commit   or add   sample  to the end of the filename  which tells Git not to run it in the future           Contact Info  We re always looking for help  or suggestions for improvement  Please feel free to reach out to us if you ve got ideas or feedback      Janine Bariuan  mailto janinebariuan google com     Thomas Desrosiers  mailto tdesrosi google com


GCP Config files included in this repo 3 1 The config files in this repo supports the GCP blogpost on Visualizing Cloud DNS public zone query data using log based metrics and Cloud Monitoring Note For details on configuring Cloud Monitoring to monitor GCP cloud DNS public zones please refer to the blog post 2 Overview Monitoring GCP Cloud DNS public zone

Answer:
# Monitoring GCP Cloud DNS public zone
## Overview
The config files in this repo supports the GCP blogpost on "Visualizing Cloud DNS public zone query data using log-based metrics and Cloud Monitoring".

### Config files included in this repo
1. [config.yaml](config.yaml)
2. [dashboard.json](dashboard.json)
3. [latency-config.yaml](latency-config.yaml)

###### Note: For details on configuring Cloud Monitoring to monitor GCP cloud DNS public zones, please refer to the blog post.

## Creating the log-based metrics
We require the creation of two distinct log-based metrics: a counter metric and a distribution metric.

* Counter metrics count the number of log entries that match a specified filter within a specified period. For example, we can use a counter metric to count the number of log entries for a specific DNS query name, query type, or response code.
* Distribution metrics also count values, but they collect the counts into ranges of values (histogram buckets). For example, we can use a distribution metric to extract the distribution of server latency.

To create log-based metrics, use the `gcloud logging metrics create` command. The configuration for the logging metrics can be passed to `gcloud` using the [config.yaml](./config.yaml) file.

**Note:** All [user-defined log-based](https://cloud.google.com/logging/docs/logs-based-metrics#user-metrics) metrics are a class of Cloud Monitoring custom metrics and are subject to charges. For pricing information, please refer to [Cloud Logging pricing: Log-based metrics](https://cloud.google.com/stackdriver/pricing#log-based-metrics).

**Note:** The retention period for log-based metrics is six weeks. Please refer to the [data retention](https://cloud.google.com/monitoring/quotas#data_retention_policy) documentation for more details.

## **Create the counter metric**

1. Create a file named `config.yaml `with the following content:

    ```
    filter: |-
    resource.type="dns_query"
    resource.labels.target_type="public-zone"
    labelExtractors:
    ProjectID: EXTRACT(resource.labels.project_id)
    QueryName: EXTRACT(jsonPayload.queryName)
    QueryType: EXTRACT(jsonPayload.queryType)
    ResponseCode: EXTRACT(jsonPayload.responseCode)
    TargetName: EXTRACT(resource.labels.target_name)
    metricDescriptor:
    labels:
        - key: QueryName
        - key: TargetName
        - key: ResponseCode
        - key: ProjectID
        - key: QueryType
    metricKind: DELTA
    unit: "1"
    valueType: INT64
    ```

2. To create counter metrics, use the `gcloud logging metrics create` command.

    **Command**
    ```
    gcloud logging metrics create cloud-dns-log-based-metric --config-from-file=config.yaml
    ```

## **Create the distribution metric**

1. Create a file named `latency-config.yaml `with the following content:

    ```
    filter: |
    resource.type="dns_query"
    resource.labels.target_type="public-zone"
    labelExtractors:
    ProjectID: EXTRACT(resource.labels.project_id)
    QueryName: EXTRACT(jsonPayload.queryName)
    QueryType: EXTRACT(jsonPayload.queryType)
    ResponseCode: EXTRACT(jsonPayload.responseCode)
    SourceIP: EXTRACT(jsonPayload.sourceIP)
    TargetName: EXTRACT(resource.labels.target_name)
    metricDescriptor:
    labels:
        - key: ResponseCode
        - key: QueryType
        - key: TargetName
        - key: ProjectID
        - key: SourceIP
        - key: QueryName
    metricKind: DELTA
    unit: "1"
    valueType: DISTRIBUTION
    valueExtractor: EXTRACT(jsonPayload.serverLatency)
    bucketOptions:
    exponentialBuckets:
        growthFactor: 2
        numFiniteBuckets: 64
        scale: 0.01
    ```

 To create counter metrics, use the `gcloud logging metrics create` command.

**Command**

    ```
    gcloud logging metrics create cloud-dns-latency-log-based-metric --config-from-file=latency-config.yaml
    ```

## **Customization options**

The provided customization options are optional and are included for illustrative purposes only. We are not using these options in this blog post. If you decide to use these options in the future, you can edit the log-based metrics to make the desired changes.

### **Include Source IP (Counter Metrics Only)**

To extract the Source IP from the log based metrics, add the following to labelExtractors and metricDescriptor in the `config.yaml` provided above. However, please note that extracting this label comes with risk and would be best suitable for temporary testing or zones where the expected volume of DNS queries is low.

In general, it is best practice to extract labels with a finite set of values. Otherwise, values that come from an infinite set, or are always unique, can lead to [high cardinality of metrics](https://cloud.google.com/monitoring/api/v3/metric-model#cardinality), which can not only increase costs but also result in ingestion errors.

**Example**

```
labelExtractors:
ProjectID: EXTRACT(resource.labels.project_id)
QueryName: EXTRACT(jsonPayload.queryName)
QueryType: EXTRACT(jsonPayload.queryType)
ResponseCode: EXTRACT(jsonPayload.responseCode)
TargetName: EXTRACT(resource.labels.target_name)
SourceIP: EXTRACT(jsonPayload.sourceIP)

metricDescriptor:
labels:
    - key: QueryName
    - key: TargetName
    - key: ResponseCode
    - key: ProjectID
    - key: QueryType
    - key: SourceIP
```

### **Finetune the filters**

The provided `config.yaml` processes all DNS query logs with a target_type of public-zone. Cloud Logging will process logs for all public zones that have logging enabled. To reduce the number of log entries processed, users can update the filter to provide a specific project or public zones.

**Example**

```
resource.type="dns_query"
resource.labels.target_type="public-zone"
resource.labels.project_id="my-project-id"
resource.labels.target_name="my-zone-name"
```

## **Create the custom dashboard**

Use the `gcloud monitoring dashboards create command` to create the dashboard. This command will create a custom dashboard named gcloud-custom-dashboard.

**Command**

```
gcloud monitoring dashboards create --config-from-file=dashboard.json
```

### **Things to consider**

1. Log-based metrics are not suitable for real-time monitoring or highly sensitive alerts because they have higher ingestion delays than other types of metrics. This is because the ingestion time of the log, the metric processing time, and the reporting time must all be taken into account.
2. There may be a delay in your metric counts.  Due to the potential 10 minute delay for log ingestion, the corresponding log-base metric could also have delays in displaying the correct log count.
3. It is recommended that users change the alignment period to at least 5 minutes when configuring alerts for log-based metrics to account for delays. This will ensure that alerts are triggered only when there is a significant change in the metric, rather than being triggered by minor fluctuations.

## References
- [GCP Cloud DNS](https://cloud.google.com/dns/docs/overview)
- [GCP Cloud DNS log schema](https://cloud.google.com/dns/docs/monitoring)
- [GCP Cloud Monitoring](https://cloud.google.com/monitoring)
- [GCP Log based metrics](https://cloud.google.com/logging/docs/logs-based-metrics)
- [High cardinality of metrics](https://cloud.google.com/monitoring/api/v3/metric-model#cardinality)
- [User-defined log-based](https://cloud.google.com/logging/docs/logs-based-metrics#user-metrics)
- [Cloud Logging pricing: Log-based metrics](https://cloud.google.com/stackdriver/pricing#log-based-metrics)
- [Data retention](https://cloud.google.com/monitoring/quotas#data_retention_policy)

Monitoring GCP Cloud DNS public zone    Overview The config files in this repo supports the GCP blogpost on  Visualizing Cloud DNS public zone query data using log based metrics and Cloud Monitoring          Config files included in this repo 1   config yaml  config yaml  2   dashboard json  dashboard json  3   latency config yaml  latency config yaml          Note  For details on configuring Cloud Monitoring to monitor GCP cloud DNS public zones  please refer to the blog post      Creating the log based metrics We require the creation of two distinct log based metrics  a counter metric and a distribution metric     Counter metrics count the number of log entries that match a specified filter within a specified period  For example  we can use a counter metric to count the number of log entries for a specific DNS query name  query type  or response code    Distribution metrics also count values  but they collect the counts into ranges of values  histogram buckets   For example  we can use a distribution metric to extract the distribution of server latency   To create log based metrics  use the  gcloud logging metrics create  command  The configuration for the logging metrics can be passed to  gcloud  using the  config yaml    config yaml  file      Note    All  user defined log based  https   cloud google com logging docs logs based metrics user metrics  metrics are a class of Cloud Monitoring custom metrics and are subject to charges  For pricing information  please refer to  Cloud Logging pricing  Log based metrics  https   cloud google com stackdriver pricing log based metrics      Note    The retention period for log based metrics is six weeks  Please refer to the  data retention  https   cloud google com monitoring quotas data retention policy  documentation for more details          Create the counter metric    1  Create a file named  config yaml  with the following content               filter         resource type  dns query      resource labels target type  public zone      labelExtractors      ProjectID  EXTRACT resource labels project id      QueryName  EXTRACT jsonPayload queryName      QueryType  EXTRACT jsonPayload queryType      ResponseCode  EXTRACT jsonPayload responseCode      TargetName  EXTRACT resource labels target name      metricDescriptor      labels            key  QueryName           key  TargetName           key  ResponseCode           key  ProjectID           key  QueryType     metricKind  DELTA     unit   1      valueType  INT64           2  To create counter metrics  use the  gcloud logging metrics create  command         Command               gcloud logging metrics create cloud dns log based metric   config from file config yaml               Create the distribution metric    1  Create a file named  latency config yaml  with the following content               filter        resource type  dns query      resource labels target type  public zone      labelExtractors      ProjectID  EXTRACT resource labels project id      QueryName  EXTRACT jsonPayload queryName      QueryType  EXTRACT jsonPayload queryType      ResponseCode  EXTRACT jsonPayload responseCode      SourceIP  EXTRACT jsonPayload sourceIP      TargetName  EXTRACT resource labels target name      metricDescriptor      labels            key  ResponseCode           key  QueryType           key  TargetName           key  ProjectID           key  SourceIP           key  QueryName     metricKind  DELTA     unit   1      valueType  DISTRIBUTION     valueExtractor  EXTRACT jsonPayload serverLatency      bucketOptions      exponentialBuckets          growthFactor  2         numFiniteBuckets  64         scale  0 01           To create counter metrics  use the  gcloud logging metrics create  command     Command                gcloud logging metrics create cloud dns latency log based metric   config from file latency config yaml                 Customization options    The provided customization options are optional and are included for illustrative purposes only  We are not using these options in this blog post  If you decide to use these options in the future  you can edit the log based metrics to make the desired changes          Include Source IP  Counter Metrics Only     To extract the Source IP from the log based metrics  add the following to labelExtractors and metricDescriptor in the  config yaml  provided above  However  please note that extracting this label comes with risk and would be best suitable for temporary testing or zones where the expected volume of DNS queries is low   In general  it is best practice to extract labels with a finite set of values  Otherwise  values that come from an infinite set  or are always unique  can lead to  high cardinality of metrics  https   cloud google com monitoring api v3 metric model cardinality   which can not only increase costs but also result in ingestion errors      Example        labelExtractors  ProjectID  EXTRACT resource labels project id  QueryName  EXTRACT jsonPayload queryName  QueryType  EXTRACT jsonPayload queryType  ResponseCode  EXTRACT jsonPayload responseCode  TargetName  EXTRACT resource labels target name  SourceIP  EXTRACT jsonPayload sourceIP   metricDescriptor  labels        key  QueryName       key  TargetName       key  ResponseCode       key  ProjectID       key  QueryType       key  SourceIP            Finetune the filters    The provided  config yaml  processes all DNS query logs with a target type of public zone  Cloud Logging will process logs for all public zones that have logging enabled  To reduce the number of log entries processed  users can update the filter to provide a specific project or public zones     Example        resource type  dns query  resource labels target type  public zone  resource labels project id  my project id  resource labels target name  my zone name            Create the custom dashboard    Use the  gcloud monitoring dashboards create command  to create the dashboard  This command will create a custom dashboard named gcloud custom dashboard     Command        gcloud monitoring dashboards create   config from file dashboard json            Things to consider    1  Log based metrics are not suitable for real time monitoring or highly sensitive alerts because they have higher ingestion delays than other types of metrics  This is because the ingestion time of the log  the metric processing time  and the reporting time must all be taken into account  2  There may be a delay in your metric counts   Due to the potential 10 minute delay for log ingestion  the corresponding log base metric could also have delays in displaying the correct log count  3  It is recommended that users change the alignment period to at least 5 minutes when configuring alerts for log based metrics to account for delays  This will ensure that alerts are triggered only when there is a significant change in the metric  rather than being triggered by minor fluctuations      References    GCP Cloud DNS  https   cloud google com dns docs overview     GCP Cloud DNS log schema  https   cloud google com dns docs monitoring     GCP Cloud Monitoring  https   cloud google com monitoring     GCP Log based metrics  https   cloud google com logging docs logs based metrics     High cardinality of metrics  https   cloud google com monitoring api v3 metric model cardinality     User defined log based  https   cloud google com logging docs logs based metrics user metrics      Cloud Logging pricing  Log based metrics  https   cloud google com stackdriver pricing log based metrics     Data retention  https   cloud google com monitoring quotas data retention policy


GCP Getting user profile from IAP enabled GAE application This setup can be done from This example demonstrates how to retrieve user profile e g name photo from an IAP enabled GAE application GCP project Initial Setup following The following setup assumes that you are setting up your new application GCP project based on the You need permission to run this e g for creating GAE app

Answer:
# Getting user profile from IAP-enabled GAE application
This example demonstrates how to retrieve user profile (e.g. name, photo) from an IAP-enabled GAE application.

## Initial Setup
This setup can be done from `Cloud Shell`.
You need `Project Owner` permission to run this, e.g. for creating GAE app.

The following setup assumes that you are setting up your new application GCP project based on the
following:
* GCP project: `project-id-1234`
* GAE region: `asia-northeast1`

1.  Set up environment variables.
    ```bash
    export PROJECT=project-id-1234
    export REGION=asia-northeast1
    ```

1.  Setup your gcloud
    ```bash
    gcloud config configurations create iap-user-profile
    gcloud config set project $PROJECT
    ```

1.  Create GAE application.
    ```bash
    gcloud app create --region=$REGION
    ```

1.  Enable required APIs
    ```bash
    gcloud services enable \
        iap.googleapis.com \
        secretmanager.googleapis.com \
        cloudresourcemanager.googleapis.com \
        people.googleapis.com
    ```

1.  Deploy this sample application. This will become the `default` service.
    Note: IAP can only be enabled when there is already a service deployed in GAE.
    ```bash
    cd professional-services/examples/iap-user-profile/
    gcloud app deploy --quiet
    ```

1.  Configure `Consent Screen` on the below link
    https://console.cloud.google.com/apis/credentials/consent?project=project-id-1234

    1.  Choose `Internal` for the `User Type`, then click `Create`
    1.  Type in the `Application name`, e.g. IAP User Profile Example
    1.  Choose the appropriate `Support email`. Alternatively, you can leave it to use your own email.
    1.  Fill in the `Authorized domains` based on your GAE application domain, e.g.
        * project-id-1234.an.r.appspot.com
    1.  Click `Save`

1.  Enable IAP on the below link for the `App Engine app`. Toggle on the button and then click `Turn On`.
    https://console.cloud.google.com/security/iap?project=project-id-1234

1.  Add IAM policy binding to the IAP-enabled App Engine application.
    Register your user email to access the application.
    ```bash
    gcloud iap web add-iam-policy-binding --resource-type=app-engine \
          --member='user:your-user@domain.com' \
          --role='roles/iap.httpsResourceAccessor'
    ```

1.  Create new `Credentials` on the below link. This credential will be used by the OAuth2 login
    flow to retrieve the user profile.
    https://console.cloud.google.com/apis/credentials?project=project-id-1234

    1.  Click `Create Credentials`. Choose `OAuth client ID`.
    1.  Choose `Web application` for the `Application type`
    1.  Type in `IAP User Profile Svc` for the `Name`
    1.  Fill in the `Authorized JavaScript origins`
        * https://project-id-1234.an.r.appspot.com
    1.  Fill in the `Authorized redirect URIs`
        * https://project-id-1234.an.r.appspot.com/auth-callback
    1.  Click `Create`
    1.  Click the newly created `IAP User Profile Example` credential and then click the `Download JSON` button.
        You'll need to paste the JSON content later as secret in the Secret Manager.

1.  Create a secret in Secret Manager named `iap-user-profile-svc-oauth2-client` with the client credential JSON file as
    the value.
    ```bash
    gcloud secrets create iap-user-profile-svc-oauth2-client \
        --locations=asia-southeast1 --replication-policy=user-managed \
        --data-file=/path-to/client_secret.json
    ```

1.  Add IAM policy binding to the secret for GAE default service account.
    ```bash
    gcloud secrets add-iam-policy-binding iap-user-profile-svc-oauth2-client \
          --member='serviceAccount:project-id-1234@appspot.gserviceaccount.com' \
          --role='roles/secretmanager.secretAccessor'
    ```

## Accessing the Application

1.  Access the application in your browser.
    Note: If you are accessing it first time, it may take some time before the policy takes effect.
    Retry several times until you are prompted the OAuth login screen.
    https://project-id-1234.an.r.appspot.com/

1.  You will be prompted the OAuth login one more time.
    This is intended since we are going to use this scope to access your People API.

1.  You should be able to see your user profile displayed on the web page.

Getting user profile from IAP enabled GAE application This example demonstrates how to retrieve user profile  e g  name  photo  from an IAP enabled GAE application      Initial Setup This setup can be done from  Cloud Shell     You need  Project Owner  permission to run this  e g  for creating GAE app   The following setup assumes that you are setting up your new application GCP project based on the  following    GCP project   project id 1234    GAE region   asia northeast1   1   Set up environment variables         bash     export PROJECT project id 1234     export REGION asia northeast1          1   Setup your gcloud        bash     gcloud config configurations create iap user profile     gcloud config set project  PROJECT          1   Create GAE application         bash     gcloud app create   region  REGION          1   Enable required APIs        bash     gcloud services enable           iap googleapis com           secretmanager googleapis com           cloudresourcemanager googleapis com           people googleapis com          1   Deploy this sample application  This will become the  default  service        Note  IAP can only be enabled when there is already a service deployed in GAE         bash     cd professional services examples iap user profile      gcloud app deploy   quiet              1   Configure  Consent Screen  on the below link       https   console cloud google com apis credentials consent project project id 1234          1   Choose  Internal  for the  User Type   then click  Create      1   Type in the  Application name   e g  IAP User Profile Example     1   Choose the appropriate  Support email   Alternatively  you can leave it to use your own email      1   Fill in the  Authorized domains  based on your GAE application domain  e g            project id 1234 an r appspot com     1   Click  Save   1   Enable IAP on the below link for the  App Engine app   Toggle on the button and then click  Turn On         https   console cloud google com security iap project project id 1234  1   Add IAM policy binding to the IAP enabled App Engine application      Register your user email to access the application         bash     gcloud iap web add iam policy binding   resource type app engine               member  user your user domain com                role  roles iap httpsResourceAccessor               1   Create new  Credentials  on the below link  This credential will be used by the OAuth2 login     flow to retrieve the user profile        https   console cloud google com apis credentials project project id 1234          1   Click  Create Credentials   Choose  OAuth client ID       1   Choose  Web application  for the  Application type      1   Type in  IAP User Profile Svc  for the  Name      1   Fill in the  Authorized JavaScript origins            https   project id 1234 an r appspot com     1   Fill in the  Authorized redirect URIs            https   project id 1234 an r appspot com auth callback     1   Click  Create      1   Click the newly created  IAP User Profile Example  credential and then click the  Download JSON  button          You ll need to paste the JSON content later as secret in the Secret Manager    1   Create a secret in Secret Manager named  iap user profile svc oauth2 client  with the client credential JSON file as     the value          bash     gcloud secrets create iap user profile svc oauth2 client             locations asia southeast1   replication policy user managed             data file  path to client secret json              1   Add IAM policy binding to the secret for GAE default service account         bash     gcloud secrets add iam policy binding iap user profile svc oauth2 client               member  serviceAccount project id 1234 appspot gserviceaccount com                role  roles secretmanager secretAccessor              Accessing the Application  1   Access the application in your browser        Note  If you are accessing it first time  it may take some time before the policy takes effect      Retry several times until you are prompted the OAuth login screen        https   project id 1234 an r appspot com       1   You will be prompted the OAuth login one more time        This is intended since we are going to use this scope to access your People API   1   You should be able to see your user profile displayed on the web page


GCP Social Security Numbers SSNs While the DLP API in GCP offers the ability to look for SSNs it may not be Hashpipeline accurate especially if there are other items such as account numbers that look similar One solution would Only 5 Million total records be to store SSNs in a Dictionary InfoType in Cloud DLP however that has the following limitations Overview In this solution we are trying to create a way to indicate security teams if there is a file found with US

Answer:
# Hashpipeline

## Overview

In this solution, we are trying to create a way to indicate security teams if there is a file found with US
Social Security Numbers (SSNs). While the DLP API in GCP offers the ability to look for SSNs, it may not be
accurate, especially if there are other items such as account numbers that look similar. One solution would
be to store SSNs in a Dictionary InfoType in Cloud DLP, however that has the following limitations:

* Only 5 Million total records
* SSNs stored in plain text

To avoid those limitations, we built a PoC Dataflow pipeline that will run for every new file
in a specified GCS bucket and determine how many (if any) SSNs are found, triggering a Pubsub Topic. The known
SSNs will be stored in Firestore, a highly scalable key value store, only after being hashed with a salt and
key, which is stored in Secret Manager. This is what the architecture will look like when we're done.

![](./img/arch.png)

## Usage

This repo offers end-to-end deployment of the Hashpipeline solution using [HashiCorp Terraform](https://terraform.io)
given a project and list of buckets to monitor.

### Prerequisites

This has only been tested on Mac OSX but will likely work on Linux as well.

* `terraform` executable is available in `$PATH`
* `gcloud` is installed and up to date
* `python` is version 3.5 or higher

### Step 1: Deploy the Infrastructure
Note that the following APIs will be enabled on your project by Terraform:

* `iam.googleapis.com`
* `dlp.googleapis.com`
* `secretmanager.googleapis.com`
* `firestore.googleapis.com`
* `dataflow.googleapis.com`
* `compute.googleapis.com`

Then deploy the infrastructure to your project

```
cd infrastructure
cp terraform.tfvars.sample terraform.tfvars
# Update with your own values.
terraform apply
```

### Step 2: Generate the Hash Key

This will create a new 64 byte key for use with HMAC and store it in Secret Manager

```
make pip
make create_key
```

### Step 3: Seed Firestore with SSNs

Since SSNs can exist in the data center in many stores, we'll assume the input is
a flat, newline separated file including valid SSNs. How you get them in that format is
up to you. Once you have your input file, simply authenticate to `gcloud` and then run:

```
./scripts/hasher.py upload \
		--project $PROJECT \
		--secret $SECRET \
		--salt $SALT \
		--collection $COLLECTION \
		--infile $SSN_FILE
```

For more information on the input parameters, just run `./bin/hasher.py --help`

### Step 4: Build and Deploy

This uses Dataflow's Templates to build our pipeline and then run it. To use the values we created in terraform, just run:

```
make build
make deploy
```

At this point your Dataflow job will start up, so you can check its progress in the GCP Console.

### Step 5: Subscribe

This pipeline just emits every finding in the file as a separate Pubsub message. We show an
example of how to subscribe to this and consume these messages in Python in the [poller.py](./scripts/poller.py)
script. However since this is specifically a security solution, you will likely want to consume
these notifications in your SIEM such as Splunk, etc.

## Testing/Demo

### Step 1

Follow Step 1 and 2 from above to set up the demo environment

### Step 2: Seed the Firestore with Fake SSNs

This script will do the following:

* Create a list of valid and random Social Security Numbers
* Store the plain text in `scripts/socials.txt`
* Hash the numbers (normalized without dashes) using HMAC-SHA256 and the key generated from `make create_key`
* Store the hashed values in Firestore under the collection specified in the terraform variable: `firestore_collection`

```
make seed_firestore
```

### Step 3: Generate some input files for dataflow to use

This will store the input files under the `inputs/` directory, so we have something to test with.

```
make generate_input_files
```

### Step 5: Test out the pipeline locally

This will run the pipeline against the `small-input.txt` file generated by the previous step. In only has
50 lines so it shouldn't take too long.

```
make run_local
```

### Step 6: Subscribe
In a separate terminal, start the poller from the test subscription and count the findings by filename.
```
$ make subscribe
Successfully subscribed to <subscription>. Messages will print below...
```

Now in a third terminal, run the following command to upload a file to the test bucket.

```
export BUCKET=<dataflow-test-bucket>
gsutil cp inputs/small-input.txt gs://$BUCKET/small.txt
```

After a little while, in your `subscribe` terminal, you should get something that looks like this, after
the files have been uploaded, along with the raw messages printed to standard out:

```
...
-----------------------------------  --------
Filename                             Findings
gs://<dataflow-test-bucket>/small.txt  26
-----------------------------------  --------
```

This number can be verified by looking in the file itself on the first line, which would say `expected_valid_socials = 26`
for this example.

### Step 7: Deploy the pipeline to a template

```
make build
make deploy
```

Now you can try out the same thing as Step 4 to verify it works.

## Disclaimer

While best efforts have been made to make this pipeline hardened from a security perspective, this is meant **only as
a demo and proof of concept** and should not be directly used in a production system without being fully vetted by security
teams and the people who will maintain the code in the organization.

Hashpipeline     Overview  In this solution  we are trying to create a way to indicate security teams if there is a file found with US Social Security Numbers  SSNs   While the DLP API in GCP offers the ability to look for SSNs  it may not be accurate  especially if there are other items such as account numbers that look similar  One solution would be to store SSNs in a Dictionary InfoType in Cloud DLP  however that has the following limitations     Only 5 Million total records   SSNs stored in plain text  To avoid those limitations  we built a PoC Dataflow pipeline that will run for every new file in a specified GCS bucket and determine how many  if any  SSNs are found  triggering a Pubsub Topic  The known SSNs will be stored in Firestore  a highly scalable key value store  only after being hashed with a salt and key  which is stored in Secret Manager  This is what the architecture will look like when we re done         img arch png      Usage  This repo offers end to end deployment of the Hashpipeline solution using  HashiCorp Terraform  https   terraform io  given a project and list of buckets to monitor       Prerequisites  This has only been tested on Mac OSX but will likely work on Linux as well      terraform  executable is available in   PATH     gcloud  is installed and up to date    python  is version 3 5 or higher       Step 1  Deploy the Infrastructure Note that the following APIs will be enabled on your project by Terraform      iam googleapis com     dlp googleapis com     secretmanager googleapis com     firestore googleapis com     dataflow googleapis com     compute googleapis com   Then deploy the infrastructure to your project      cd infrastructure cp terraform tfvars sample terraform tfvars   Update with your own values  terraform apply          Step 2  Generate the Hash Key  This will create a new 64 byte key for use with HMAC and store it in Secret Manager      make pip make create key          Step 3  Seed Firestore with SSNs  Since SSNs can exist in the data center in many stores  we ll assume the input is a flat  newline separated file including valid SSNs  How you get them in that format is up to you  Once you have your input file  simply authenticate to  gcloud  and then run         scripts hasher py upload       project  PROJECT       secret  SECRET       salt  SALT       collection  COLLECTION       infile  SSN FILE      For more information on the input parameters  just run    bin hasher py   help       Step 4  Build and Deploy  This uses Dataflow s Templates to build our pipeline and then run it  To use the values we created in terraform  just run       make build make deploy      At this point your Dataflow job will start up  so you can check its progress in the GCP Console       Step 5  Subscribe  This pipeline just emits every finding in the file as a separate Pubsub message  We show an example of how to subscribe to this and consume these messages in Python in the  poller py    scripts poller py  script  However since this is specifically a security solution  you will likely want to consume these notifications in your SIEM such as Splunk  etc      Testing Demo      Step 1  Follow Step 1 and 2 from above to set up the demo environment      Step 2  Seed the Firestore with Fake SSNs  This script will do the following     Create a list of valid and random Social Security Numbers   Store the plain text in  scripts socials txt    Hash the numbers  normalized without dashes  using HMAC SHA256 and the key generated from  make create key    Store the hashed values in Firestore under the collection specified in the terraform variable   firestore collection       make seed firestore          Step 3  Generate some input files for dataflow to use  This will store the input files under the  inputs   directory  so we have something to test with       make generate input files          Step 5  Test out the pipeline locally  This will run the pipeline against the  small input txt  file generated by the previous step  In only has 50 lines so it shouldn t take too long       make run local           Step 6  Subscribe In a separate terminal  start the poller from the test subscription and count the findings by filename        make subscribe Successfully subscribed to  subscription   Messages will print below         Now in a third terminal  run the following command to upload a file to the test bucket       export BUCKET  dataflow test bucket  gsutil cp inputs small input txt gs    BUCKET small txt      After a little while  in your  subscribe  terminal  you should get something that looks like this  after the files have been uploaded  along with the raw messages printed to standard out                                                         Filename                             Findings gs    dataflow test bucket  small txt  26                                                    This number can be verified by looking in the file itself on the first line  which would say  expected valid socials   26  for this example        Step 7  Deploy the pipeline to a template      make build make deploy      Now you can try out the same thing as Step 4 to verify it works      Disclaimer  While best efforts have been made to make this pipeline hardened from a security perspective  this is meant   only as a demo and proof of concept   and should not be directly used in a production system without being fully vetted by security teams and the people who will maintain the code in the organization


GCP This directory shows a series of pipelines used to generate data in GCS or BigQuery The intention for these pipelines are to be a tool for partners customers and SCEs who want to create a dummy dataset that Testing Data Generators Human Readable Data Generation looks like the schema of their actual data in order to run some queries in BigQuery Data Generator There are two different types of use cases for this kind of tool which we refer to throughout this documentation as Human Readable and Performance These pipelines are a great place to get started when you only have a customer s schema

Answer:
# Data Generator
This directory shows a series of pipelines used to generate data in GCS or BigQuery.
The intention for these pipelines are to be a tool for partners, customers and SCEs who want to create a dummy dataset that
looks like the schema of their actual data in order to run some queries in BigQuery.
There are two different types of use cases for this kind of tool which we refer to throughout this documentation as Human Readable and Performance
Testing Data Generators.

## Human Readable Data Generation
These pipelines are a great place to get started when you only have a customer's schema
and do not have a requirement for your generated dataset to have similar distribution to
the source dataset (this is required for accurately capturing query performance).
 - Human readable / queryable data. This includes smart populating columns with data formatted based on the field name.
 - This can be used in scenarios where there are hurdles to get over in migrating actual data to BigQuery  to unblock integration tests and downstream development.
 - Generate joinable schemas for < 1 Billion distinct keys
 - Generates data from just a schema
 - Numeric columns trend upwards based on a `date` field if it exists.

![Alt text](img/data-generator.png)

 - [Data Generator](data-generator-pipeline/data_generator_pipeline.py): This pipeline should
    can be used to generate a central fact table in snowflake schema.
 - [Data Generator (Joinable Table)](data-generator-pipeline/data_generator_pipeline.py):
    this pipeline should be used to generate data that joins to an existing BigQuery Table
    on a certain key.

## Performance Testing Data Generation
The final pipeline supports the later use case where matching the distribution of the source
dataset for replicating query performance is the goal.
 - Prioritizes speed and distribution matching over human readable data (ie. random strings rather than random sentences w/ english words)
 - Match the distribution of keys in a dataset to benchmark join performance
 - Generate joinable schemas on a larger scale.
 - Generates data based on a schema and a histogram table containing the desired distribution of data across the key columns

![Alt text](img/distribution-matcher.png)

 - [Histogram Tool](bigquery-scripts/bq_histogram_tool.py): This is an example script of what could
    be run on a customer's table to extract the distribution information per key without collecting
    meaningful data. This script would be run by the client and they would share the output table.
    If the customer is not already in BigQuery this histogram tool can serve as boilerplate for a
    histogram tool that reads from their source database and writes to BigQuery.
 - [Distribution Matcher](data-generator-pipeline/data_distribution_matcher.py): This pipeline operates
    on a BigQuery table containing key hashes and counts and will replicate this distribution in the
    generated dataset..

## General Performance Recommendations
A few recommendations when generating large datasets with any of these pipelines:
 - Write to AVRO on GCS then load to BigQuery.
 - Use machines with a lot of CPU. We recommend `n1-highcpu-32`.
 - Run on a private network to avoid using public ip addresses.
 - Request higher quotas for your project to support scaling to 300+ large workers,
   specifically, in the region you wish to run the pipeline:
   - 300+ In-use IP addresses
   - 10,000+ CPUs

### Human Readable Data Generator Usage
This tool has several parameters to specify what kind of data you would like to generate.

#### Schema
The schema may be specified using the `--schema_file` parameter  with a file containing a
list of json objects with `name`,  `type`, `mode` and optionally `description` fields.
This form follows the output of`bq show --format=json --schema <table_reference>`.
This data generator now supports nested types like `RECORD`/`STRUCT`. Note, that the approach
taken was to generate a `REPEATED` `RECORD` (aka `ARRAY<STRUCT>`) and each record generated
will have between 0 and 3 elements in this array.
ie.
```
--schema_file=gs://python-dataflow-example/schemas/lineorder-schema.json
```
lineorder-schema.json:
```
{
    "fields": [
                {"name": "lo_order_key",
                 "type": "STRING",
                 "mode": "REQUIRED"
                },
                {"name": "lo_linenumber",
                 "type": "INTEGER",
                 "mode": "NULLABLE"
                },
                {...}
              ]
}
```
Alternatively, the schema may be specified with a reference to an existing BigQuery table with the
`--input_bq_table` parameter. We suggest using the BigQuery UI to create an empty BigQuery table to
avoid typos when writing your own schema json.

```
--input_bq_table=BigQueryFaker.lineorders
```

Note, if you are generating data that is also being loaded into an RDBMS you can specify the RDMS type
in the `description` field of the schema. The data generator will parse this to extract datasize.
ie. The below field will have strings truncated to be within 36 bytes.
```
[
    {"name": "lo_order_key",
     "type": "STRING",
     "mode": "REQUIRED",
     "description": "VARCHAR(36)"
    },
    {...}
]
```

#### Number of records
To specify the number of records to generate use the `--num_records` parameter. Note we recommend only calling this
pipeline for a maximum of 50 Million records at a time. For generating larger tables you can simply call the pipeline
script several times.

```
--num_records=1000000
```

#### Output Prefix
The output is specified as a GCS prefix. Note that multiple files will be written with
`<prefix>-<this-shard-number>-of-<total-shards>.<suffix>`. The suffix will be the appropriate suffix for the file type
based on if you pass the `--csv_schema_order` or `--avro_schema_file` parameters described later.

--gcs_output_prefix=gs://<BUCKET NAME>/path/to/myprefix

Will create files at:

gs://<BUCKET NAME>/path/to/myprefix-#####-of-#####.<suffix>

#### Output format

Output format is specified by passing one of the `--csv_schema_order`, `--avro_schema_file`, or `--write_to_parquet` parameters.

`--csv_schema_order` should be a comma separated list specifying the order of the fieldnames for writing.
Note that `RECORD` are not supported when writing to CSV, because it is a flat file format.

```
--csv_schema_order=lo_order_key,lo_linenumber,...
```

`--avro_schema_file` should be a file path to the avro schema to write.

```
--avro_schema_file=/path/to/linorders.avsc
```

`--write_to_parquet` is a flag that specifies the output should be parquet. In order for beam to write to parquet,
a pyarrow schema is needed. Therefore, this tool translates the schema in the `--schema_file` to
a pyarrow schema automatically if this flag is included, but pyarrow doesn't support all fields that are supported
by BigQuery. STRING, NUMERIC, INTEGER, FLOAT, NUMERIC, BOOLEAN, TIMESTAMP, DATE, TIME, and DATETIME types are supported.

There is limited support for writing RECORD types to parquet. Due to this [known pyarrow issue](https://jira.apache.org/jira/browse/ARROW-2587?jql=project%20%3D%20ARROW%20AND%20fixVersion%20%3D%200.14.0%20AND%20text%20~%20%22struct%22) this tool does not support writing arrays nested within structs.

However BYTE, and GEOGRAPHY fields are not supported and cannot be included in the `--schema_file` when writing
to parquet.

```
--write_to_parquet
```

Alternatively, you can write directly to a BigQuery table by specifying an `--output_bq_table`. However, if you are generating
more than 100K records, you may run into the limitation of the python SDK where WriteToBigQuery does not orchestrate multiple
load jobs you hit one of the single load job limitations [BEAM-2801](https://issues.apache.org/jira/browse/BEAM-2801). If you
are not concerned with having many duplicates, you can generate an initial BigQuery table with `--num_records=10000000` and
then use [`bq_table_resizer.py`](bigquery-scripts/bq_table_resizer.py) to copy the table into itself until it reaches the
desired size.

```
--output_bq_table=project:dataset.table
```

#### Sparsity (optional)
Data is seldom full for every record so you can specify the probability of a NULLABLE column being null with the `--p_null` parameter.

```
--p_null=0.2
```

#### Keys and IDs (optional)

The data generator will parse your field names and generate keys/ids for fields whose name contains "`_key`" or "`_id`".
The cardinality of such key columns can be controlled with the `--n_keys` parameter.

Additionally, you can parameterize the key-skew by passing` --key_skew_distribution`. By default this is `None`, meaning roughly equal
distribution of rowcount across keys. This also supports `"binomial"` giving a maximum variance bell curve of keys over the range of the
keyset or `"zipf"` giving a distribution across the keyset according to zipf's law.

##### Primary Key (optional)
The data generator can support a primary key columns by passing a comma separated list of field names to `--primary_key_cols`.
Note this is done by a deduplication process at the end of the pipeline. This may be a bottleneck for large data volumes.
Also, using this parameter might cause you to fall short of `--num_records` output records due to the deduplicaiton.
To mitigate this you can set `--n_keys` to a number much larger than the number of records you are generating.

#### Date Parameters (optional)
To constrain the dates generated in date columns one can use the `--min_date` and `--max_date` parameters.

The minimum date will default to January 1, 2000 and the `max_date` will default to today.

If you are using these parameters be sure to use YYYY-MM-DD format.

```
--min_date=1970-01-01 \
--max_date=2010-01-01
```

#### Number Parameters (optional)
The range of integers and/or floats can be constrained with the `--max_int` and `--max_float` parameters.
These default to 100 Million.
The number of decimal places in a float can be controlled with the `--float_precision` parameter.
The default float precision is 2.
Both integers and floats can be constrained to strictly positive values using
the `--strictly_pos=True`.
True is the default.

#### Write Disposition (optional)
The BigQuery write disposition can be specified using the `--write_disp` parameter.

The default is `WRITE_APPEND`.

#### Dataflow Pipeline parameters
For basic usage we recommend the following parameters:
```
python data_generator_pipeline.py \
--project=<PROJECT ID> \
--setup_file=./setup.py \

--worker_machine_type=n1-highcpu-32 \ # This is a high cpu process so tuning the machine type will boost performance

--runner=DataflowRunner \ # run on Dataflow workers
--staging_location=gs://<BUCKET NAME>/test \
--temp_location=gs://<BUCKET NAME>/temp \
--save_main_session \ # serializes main session and sends to each worker
```

For isolating your Dataflow workers on a private network you can additionally specify:
```
...
--region=us-east1 \
--subnetwork=<FULL PATH TO SUBNET> \
--network=<NETWORK ID> \
--no_use_public_ips
```

### Modifying FakeRowGen
You may want to change the `FakeRowGen` DoFn class to more accurately spoof your data. You can use `special_map` to map
substrings in field names to [Faker Providers](https://faker.readthedocs.io/en/latest/providers.html). The only
requirement for this DoFn is for it to return a list containing a single python dictionary mapping field names to values.
So hack away if you need something more specific any python code is fair game. Keep in mind
that if you use a non-standard module (available in PyPI) you will need to make sure it gets installed on each of the workers or you will get

namespace issues. This can be done most simply by adding the module to `setup.py`.

### Generating Joinable tables Snowflake schema
To generate multiple tables that join based on certain keys, start by generating the central fact table with the above described
[`data_generator_pipeline.py`](data-generator-pipeline/data_generator_pipeline.py).
Then use [`data_generator_joinable_table.py`](data-generator-pipeline/data_generator_pipeline.py) with the above described parameters
for the new table plus three additional parameters described below.
 - `--fact_table` The existing fact table in BigQuery that will be queried to obtain list of distinct key values.
 - `--source_joining_key_col` The field name of the foreign key col in the existing table.
 - `--dest_joining_key_col` The field name in the table we are generating with the pipeline for joining to the existing table.

Note, this method selects distinct keys from the `--fact_table` as a side input which are passed as a list to the to each worker which randomly
selects a value to assign to this record. This means that this list must comfortably fit in memory. This makes this method only suitable for key
columns with relatively low cardinality (< 1 Billion distinct keys). If you have more rigorous needs for generating joinable schemas, you should
consider using the distribution matcher pipeline.

## Performance Testing Data Generator Usage
Steps:
 - Generate the posterior histogram table. For an example of how to do this on an existing BigQuery table look at the BigQuery Histogram Tool
described later in this doc.
 - Use the [`data_distribution_matcher.py`](data-generator-pipeline/data_distribution_matcher.py) pipeline.

You can specify `--schema_file` (or `--input_table`), `--gcs_output_prefix` and `--output_format` the same way as described above in the
Human Readable Data Generator section. Additionally, you must specify an `--histogram_table`. This table will have a field for each key column (which will store
a hash of each value) and a frequency with which these values occur.

### Generating Joinable Schemas
Joinable tables can be created by running the distribution matcher on a histogram for all relevant tables in the dataset. Because each histogram table
entry captures the hash of each key it refers to we can capture exact join scenarios without handing over any real data.

## BigQuery Scripts
Included are three BigQuery utility scripts to help you with your data generating needs. The first helps with loading many gcs files to BigQuery
while staying under the 15TB per load job limit, the next will help you profile the distribution of an existing dataset and the last will allow
you to resize BigQuery tables to be a desired size.

### BigQuery batch loads
This script is meant to orchestrate BigQuery load jobs of many
json files on Google Cloud Storage. It ensures that each load
stays under the 15 TB per load job limit. It operates on the

output of `gsutil -l`.

This script can be called with the following arguments:

`--project`: GCP project ID

`--dataset`: BigQuery dataset ID containing the table your wish
    to populate.

`--table`: BigQuery table ID of the table you wish to populate

`--source_file`: This is the output of `gsutil -l` with the URI of
    each file that you would like to load

`--create_table`: If passed this script will create
    the destination table.

`--schema_file`: Path to a json file defining the destination BigQuery
    table schema.

`--partitioning_column`: name of the field for date partitioning in
    the destination table.

`--max_bad_records`: Number of permissible bad records per load job.

#### Example Usage:
```
gsutil -l gs://<bucket>/path/to/json/<file prefix>-*.json >> ./files_to_load.txt

python bq_load_batches.py --project=<project> \
--dataset=<dataset_id> \
--table=<table_id> \
--partitioning_column date \
--source_file=files_to_load.txt
```
### BigQuery Histogram Tool
This script will create a BigQuery table containing the hashes of the key columns
specified as a comma separated list to the `--key_cols` parameter and the frequency
for which that group of key columns appears in the `--input_table`. This serves as
a histogram of the original table and will be used as the source for
[`data_distribution_matcher.py`](data_generator_pipeline/data_distribution_matcher.py)

#### Example Usage:
```
python bq_histogram_tool.py \
--input_table=<project>.<dataset>.<source_table> \
--output_table=<project>.<dataset>.<histogram_table> \
--key_cols=item_id,store_id
```

### BigQuery table resizer

This script is to help increase the size of a table based on a generated or sample.
If you are short on time and have a requirement to generate a 100TB table you can
use this script to generate a few GB and copy table into itself until it it is the
desired size or number of rows. While this would be inappropriate for accurate
performance benchmarking it can be used to get a query specific cost estimate.
This script can be used to copy a table in place or create a new table if you
want to maintain the record of the original records. You can specify the target
table size in either number of rows or GB.

#### Example Usage

```
python bq_table_resizer.py \
--project my-project-id \
--source_dataset my-dataset-id \
--source_table my-source-table-id \
--destination_dataset my-dataset-id \
--destination_table my-new-table-id \
--target_gb 15000 \
--location US
```

### Running the tests
Note, that the tests for the BigQuery table resizer require that you have
`GOOGLE_APPLICATION_DEFAULT` set to credentials with access to a BigQuery
environment where you can create and destroy tables.

```
cd data-generator-pipeline
python -m unittest discover
```

Data Generator This directory shows a series of pipelines used to generate data in GCS or BigQuery  The intention for these pipelines are to be a tool for partners  customers and SCEs who want to create a dummy dataset that looks like the schema of their actual data in order to run some queries in BigQuery  There are two different types of use cases for this kind of tool which we refer to throughout this documentation as Human Readable and Performance Testing Data Generators       Human Readable Data Generation These pipelines are a great place to get started when you only have a customer s schema and do not have a requirement for your generated dataset to have similar distribution to the source dataset  this is required for accurately capturing query performance      Human readable   queryable data  This includes smart populating columns with data formatted based on the field name     This can be used in scenarios where there are hurdles to get over in migrating actual data to BigQuery  to unblock integration tests and downstream development     Generate joinable schemas for   1 Billion distinct keys    Generates data from just a schema    Numeric columns trend upwards based on a  date  field if it exists     Alt text  img data generator png        Data Generator  data generator pipeline data generator pipeline py   This pipeline should     can be used to generate a central fact table in snowflake schema      Data Generator  Joinable Table   data generator pipeline data generator pipeline py       this pipeline should be used to generate data that joins to an existing BigQuery Table     on a certain key      Performance Testing Data Generation The final pipeline supports the later use case where matching the distribution of the source dataset for replicating query performance is the goal     Prioritizes speed and distribution matching over human readable data  ie  random strings rather than random sentences w  english words     Match the distribution of keys in a dataset to benchmark join performance    Generate joinable schemas on a larger scale     Generates data based on a schema and a histogram table containing the desired distribution of data across the key columns    Alt text  img distribution matcher png       Histogram Tool  bigquery scripts bq histogram tool py   This is an example script of what could     be run on a customer s table to extract the distribution information per key without collecting     meaningful data  This script would be run by the client and they would share the output table      If the customer is not already in BigQuery this histogram tool can serve as boilerplate for a     histogram tool that reads from their source database and writes to BigQuery      Distribution Matcher  data generator pipeline data distribution matcher py   This pipeline operates     on a BigQuery table containing key hashes and counts and will replicate this distribution in the     generated dataset       General Performance Recommendations A few recommendations when generating large datasets with any of these pipelines     Write to AVRO on GCS then load to BigQuery     Use machines with a lot of CPU  We recommend  n1 highcpu 32      Run on a private network to avoid using public ip addresses     Request higher quotas for your project to support scaling to 300  large workers     specifically  in the region you wish to run the pipeline       300  In use IP addresses      10 000  CPUs      Human Readable Data Generator Usage This tool has several parameters to specify what kind of data you would like to generate         Schema The schema may be specified using the    schema file  parameter  with a file containing a list of json objects with  name     type    mode  and optionally  description  fields  This form follows the output of bq show   format json   schema  table reference    This data generator now supports nested types like  RECORD   STRUCT   Note  that the approach taken was to generate a  REPEATED   RECORD   aka  ARRAY STRUCT    and each record generated will have between 0 and 3 elements in this array  ie        schema file gs   python dataflow example schemas lineorder schema json     lineorder schema json             fields                       name    lo order key                     type    STRING                     mode    REQUIRED                                       name    lo linenumber                     type    INTEGER                     mode    NULLABLE                                                                 Alternatively  the schema may be specified with a reference to an existing BigQuery table with the    input bq table  parameter  We suggest using the BigQuery UI to create an empty BigQuery table to avoid typos when writing your own schema json         input bq table BigQueryFaker lineorders      Note  if you are generating data that is also being loaded into an RDBMS you can specify the RDMS type in the  description  field of the schema  The data generator will parse this to extract datasize  ie  The below field will have strings truncated to be within 36 bytes              name    lo order key         type    STRING         mode    REQUIRED         description    VARCHAR 36                                Number of records To specify the number of records to generate use the    num records  parameter  Note we recommend only calling this pipeline for a maximum of 50 Million records at a time  For generating larger tables you can simply call the pipeline script several times         num records 1000000           Output Prefix The output is specified as a GCS prefix  Note that multiple files will be written with   prefix   this shard number  of  total shards   suffix    The suffix will be the appropriate suffix for the file type based on if you pass the    csv schema order  or    avro schema file  parameters described later     gcs output prefix gs    BUCKET NAME  path to myprefix  Will create files at   gs    BUCKET NAME  path to myprefix       of        suffix          Output format  Output format is specified by passing one of the    csv schema order      avro schema file   or    write to parquet  parameters      csv schema order  should be a comma separated list specifying the order of the fieldnames for writing  Note that  RECORD  are not supported when writing to CSV  because it is a flat file format         csv schema order lo order key lo linenumber             avro schema file  should be a file path to the avro schema to write         avro schema file  path to linorders avsc         write to parquet  is a flag that specifies the output should be parquet  In order for beam to write to parquet  a pyarrow schema is needed  Therefore  this tool translates the schema in the    schema file  to a pyarrow schema automatically if this flag is included  but pyarrow doesn t support all fields that are supported by BigQuery  STRING  NUMERIC  INTEGER  FLOAT  NUMERIC  BOOLEAN  TIMESTAMP  DATE  TIME  and DATETIME types are supported   There is limited support for writing RECORD types to parquet  Due to this  known pyarrow issue  https   jira apache org jira browse ARROW 2587 jql project 20 3D 20ARROW 20AND 20fixVersion 20 3D 200 14 0 20AND 20text 20  20 22struct 22  this tool does not support writing arrays nested within structs   However BYTE  and GEOGRAPHY fields are not supported and cannot be included in the    schema file  when writing to parquet         write to parquet      Alternatively  you can write directly to a BigQuery table by specifying an    output bq table   However  if you are generating more than 100K records  you may run into the limitation of the python SDK where WriteToBigQuery does not orchestrate multiple load jobs you hit one of the single load job limitations  BEAM 2801  https   issues apache org jira browse BEAM 2801   If you are not concerned with having many duplicates  you can generate an initial BigQuery table with    num records 10000000  and then use   bq table resizer py   bigquery scripts bq table resizer py  to copy the table into itself until it reaches the desired size         output bq table project dataset table            Sparsity  optional  Data is seldom full for every record so you can specify the probability of a NULLABLE column being null with the    p null  parameter         p null 0 2            Keys and IDs  optional   The data generator will parse your field names and generate keys ids for fields whose name contains    key   or    id    The cardinality of such key columns can be controlled with the    n keys  parameter   Additionally  you can parameterize the key skew by passing    key skew distribution   By default this is  None   meaning roughly equal distribution of rowcount across keys  This also supports   binomial   giving a maximum variance bell curve of keys over the range of the keyset or   zipf   giving a distribution across the keyset according to zipf s law          Primary Key  optional  The data generator can support a primary key columns by passing a comma separated list of field names to    primary key cols   Note this is done by a deduplication process at the end of the pipeline  This may be a bottleneck for large data volumes  Also  using this parameter might cause you to fall short of    num records  output records due to the deduplicaiton  To mitigate this you can set    n keys  to a number much larger than the number of records you are generating        Date Parameters  optional  To constrain the dates generated in date columns one can use the    min date  and    max date  parameters   The minimum date will default to January 1  2000 and the  max date  will default to today   If you are using these parameters be sure to use YYYY MM DD format         min date 1970 01 01     max date 2010 01 01           Number Parameters  optional  The range of integers and or floats can be constrained with the    max int  and    max float  parameters  These default to 100 Million  The number of decimal places in a float can be controlled with the    float precision  parameter  The default float precision is 2  Both integers and floats can be constrained to strictly positive values using the    strictly pos True   True is the default        Write Disposition  optional  The BigQuery write disposition can be specified using the    write disp  parameter   The default is  WRITE APPEND          Dataflow Pipeline parameters For basic usage we recommend the following parameters      python data generator pipeline py     project  PROJECT ID      setup file   setup py      worker machine type n1 highcpu 32     This is a high cpu process so tuning the machine type will boost performance    runner DataflowRunner     run on Dataflow workers   staging location gs    BUCKET NAME  test     temp location gs    BUCKET NAME  temp     save main session     serializes main session and sends to each worker      For isolating your Dataflow workers on a private network you can additionally specify            region us east1     subnetwork  FULL PATH TO SUBNET      network  NETWORK ID      no use public ips          Modifying FakeRowGen You may want to change the  FakeRowGen  DoFn class to more accurately spoof your data  You can use  special map  to map substrings in field names to  Faker Providers  https   faker readthedocs io en latest providers html   The only requirement for this DoFn is for it to return a list containing a single python dictionary mapping field names to values  So hack away if you need something more specific any python code is fair game  Keep in mind that if you use a non standard module  available in PyPI  you will need to make sure it gets installed on each of the workers or you will get  namespace issues  This can be done most simply by adding the module to  setup py        Generating Joinable tables Snowflake schema To generate multiple tables that join based on certain keys  start by generating the central fact table with the above described   data generator pipeline py   data generator pipeline data generator pipeline py   Then use   data generator joinable table py   data generator pipeline data generator pipeline py  with the above described parameters for the new table plus three additional parameters described below        fact table  The existing fact table in BigQuery that will be queried to obtain list of distinct key values        source joining key col  The field name of the foreign key col in the existing table        dest joining key col  The field name in the table we are generating with the pipeline for joining to the existing table   Note  this method selects distinct keys from the    fact table  as a side input which are passed as a list to the to each worker which randomly selects a value to assign to this record  This means that this list must comfortably fit in memory  This makes this method only suitable for key columns with relatively low cardinality    1 Billion distinct keys   If you have more rigorous needs for generating joinable schemas  you should consider using the distribution matcher pipeline      Performance Testing Data Generator Usage Steps     Generate the posterior histogram table  For an example of how to do this on an existing BigQuery table look at the BigQuery Histogram Tool described later in this doc     Use the   data distribution matcher py   data generator pipeline data distribution matcher py  pipeline    You can specify    schema file   or    input table       gcs output prefix  and    output format  the same way as described above in the Human Readable Data Generator section  Additionally  you must specify an    histogram table   This table will have a field for each key column  which will store a hash of each value  and a frequency with which these values occur       Generating Joinable Schemas Joinable tables can be created by running the distribution matcher on a histogram for all relevant tables in the dataset  Because each histogram table entry captures the hash of each key it refers to we can capture exact join scenarios without handing over any real data      BigQuery Scripts Included are three BigQuery utility scripts to help you with your data generating needs  The first helps with loading many gcs files to BigQuery while staying under the 15TB per load job limit  the next will help you profile the distribution of an existing dataset and the last will allow you to resize BigQuery tables to be a desired size        BigQuery batch loads This script is meant to orchestrate BigQuery load jobs of many json files on Google Cloud Storage  It ensures that each load stays under the 15 TB per load job limit  It operates on the  output of  gsutil  l     This script can be called with the following arguments      project   GCP project ID     dataset   BigQuery dataset ID containing the table your wish     to populate      table   BigQuery table ID of the table you wish to populate     source file   This is the output of  gsutil  l  with the URI of     each file that you would like to load     create table   If passed this script will create     the destination table      schema file   Path to a json file defining the destination BigQuery     table schema       partitioning column   name of the field for date partitioning in     the destination table       max bad records   Number of permissible bad records per load job         Example Usage      gsutil  l gs    bucket  path to json  file prefix    json      files to load txt  python bq load batches py   project  project      dataset  dataset id      table  table id      partitioning column date     source file files to load txt         BigQuery Histogram Tool This script will create a BigQuery table containing the hashes of the key columns specified as a comma separated list to the    key cols  parameter and the frequency for which that group of key columns appears in the    input table   This serves as a histogram of the original table and will be used as the source for   data distribution matcher py   data generator pipeline data distribution matcher py        Example Usage      python bq histogram tool py     input table  project   dataset   source table      output table  project   dataset   histogram table      key cols item id store id          BigQuery table resizer  This script is to help increase the size of a table based on a generated or sample  If you are short on time and have a requirement to generate a 100TB table you can use this script to generate a few GB and copy table into itself until it it is the desired size or number of rows  While this would be inappropriate for accurate performance benchmarking it can be used to get a query specific cost estimate  This script can be used to copy a table in place or create a new table if you want to maintain the record of the original records  You can specify the target table size in either number of rows or GB        Example Usage      python bq table resizer py     project my project id     source dataset my dataset id     source table my source table id     destination dataset my dataset id     destination table my new table id     target gb 15000     location US          Running the tests Note  that the tests for the BigQuery table resizer require that you have  GOOGLE APPLICATION DEFAULT  set to credentials with access to a BigQuery environment where you can create and destroy tables       cd data generator pipeline python  m unittest discover