GCP Unit Tests pytest bash pip install r requirements dev txt Run unit tests after installing development dependencis

Answer:
Unit Tests
===

Run unit tests after installing development dependencis:

```bash
pip install -r requirements-dev.txt
pytest
```

Save and Replay VM Deletion Events
===

It is useful to replay VM deletion events to test out changes to the Background
Function.  See the [Replay Quickstart][replay-qs] for more information.

 1. Deploy the function
 2. Get the subscription of the function with `gcloud pubsub subscriptions
    list`
 3. Create a pubsub snapshot with `gcloud pubsub snapshots create vm-deletions
    --subscription <subscription>`
 4. Delete a VM instance
 5. Deploy a new version of the same function
 6. Replay the deletion events to the new version `gcloud pubsub subscriptions
    seek <subscription> --snapshot=vm-deletions`
 7. Inspect the function output with `gcloud functions logs read --limit 50`

Interactive Testing
===

A run helper method is provided to run the function from a local workstation.
First, make sure you have the following environment variables set as if they
would be in the context of GCF.

```bash
# env | grep DNS
DNS_VM_GC_DNS_PROJECT=dnsregistration
DNS_VM_GC_DNS_ZONES=nonprod-private-zone
```

If necessary, use Application Default Credentials:

```bash
# env | grep GOOG
GOOGLE_APPLICATION_CREDENTIALS=/Users/jmccune/.credentials/dns-logging-83a9d261f444.json
```

Run the python REPL with debugging enabled via the DEBUG environment variable.

```ipython
Python 3.7.3 (default, Mar 27 2019, 09:23:39)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import conftest
>>> from main_test import run
>>> run()
{"message": "BEGIN Zone cleanup", "managed_zone": "nonprod-private-zone"}
{"message": "BEGIN search for deletion candidates", "instance": "test", "ip": "10.138.0.45"}
{"message": "Skipped, not an A record", "record": {"name": "gcp.example.com.", "type": "NS", "ttl": 21600, "rrdatas": ["ns-gcp-private.googledomains.com."], "signatureRrdatas": [], "kind": "dns#resourceRecordSet"}}
{"message": "Skipped, not an A record", "record": {"name": "gcp.example.com.", "type": "SOA", "ttl": 21600, "rrdatas": ["ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300"], "signatureRrdatas": [], "kind": "dns#resourceRecordSet"}}
{"message": "Skipped, shortname != instance", "record": {"name": "keep.gcp.example.com.", "type": "A", "ttl": 300, "rrdatas": ["10.138.0.45"], "signatureRrdatas": [], "kind": "dns#resourceRecordSet"}}
{"message": "Skipped, ip does not match", "record": {"name": "test.keep.gcp.example.com.", "type": "A", "ttl": 300, "rrdatas": ["10.138.0.43", "10.138.0.44", "10.138.0.45"], "signatureRrdatas": [], "kind": "dns#resourceRecordSet"}}
{"message": "Skipped, ip does not match", "record": {"name": "test.nonprod.gcp.example.com.", "type": "A", "ttl": 300, "rrdatas": ["10.138.0.250"], "signatureRrdatas": [], "kind": "dns#resourceRecordSet"}}
{"message": "END search for deletion candidates", "candidates": []}
{"message": "END Zone cleanup", "managed_zone": "nonprod-private-zone"}
>>>
```

Unit Tests
===

Run unit tests to interact with fixture data.  Fixture data is provided for the
main entry point of a pubsub message sent to Google Cloud Functions and for the
API response when collecting the IP address from the compute API.

Test dependencies
---

Install test dependencies:

```bash
pip install pytest
pip install google-api-python-client
```

Run the tests
---

```
pytest -vv tests
```

Sample events
===

This is the JSON log data sent from Stackdriver's compute.instances.delete
event to Pub/Sub and then to the Background Function implemented in Python.
There are two events, one for the GCE_API_CALL initiating the VM deletion, the
second for the GCE_OPERATION_DONE event concluding the VM deletion.

Obtained using `gcloud functions logs read --limit 100`.

```txt
D      dns_vm_gc  578383257362746  2019-06-12 01:30:09.128  Function execution started
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145  {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "insertId": "6o9bdnfxt05mn",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "jsonPayload": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "actor": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "user": "jmccune@google.com"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "event_subtype": "compute.instances.delete",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "event_timestamp_us": "1560300993146327",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "event_type": "GCE_API_CALL",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "ip_address": "",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "operation": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "id": "971500189857477422",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "name": "operation-1560300992590-58b15e267f2cc-e7529c4d-f1343924",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "type": "operation",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "zone": "us-west1-a"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "request": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "body": "null",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "url": "https://www.googleapis.com/compute/v1/projects/user-dev-242122/zones/us-west1-a/instances/test?key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "resource": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "id": "613579339353422259",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "name": "test",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "type": "instance",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "zone": "us-west1-a"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "trace_id": "operation-1560300992590-58b15e267f2cc-e7529c4d-f1343924",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "version": "1.2"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "labels": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "compute.googleapis.com/resource_id": "613579339353422259",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "compute.googleapis.com/resource_name": "test",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "compute.googleapis.com/resource_type": "instance",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "compute.googleapis.com/resource_zone": "us-west1-a"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "logName": "projects/user-dev-242122/logs/compute.googleapis.com%2Factivity_log",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "receiveTimestamp": "2019-06-12T00:56:33.188262452Z",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "resource": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "labels": {
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "instance_id": "613579339353422259",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "project_id": "user-dev-242122",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145              "zone": "us-west1-a"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145          "type": "gce_instance"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      },
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "severity": "INFO",
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145      "timestamp": "2019-06-12T00:56:33.146327Z"
I      dns_vm_gc  578383257362746  2019-06-12 01:30:09.145  }
D      dns_vm_gc  578383257362746  2019-06-12 01:30:09.146  Function execution took 20 ms, finished with status: 'ok'
D      dns_vm_gc  578389534045377  2019-06-12 01:30:20.433  Function execution started
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438  {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "insertId": "hrbv6bfh9qzr2",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "jsonPayload": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "actor": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "user": "jmccune@google.com"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "event_subtype": "compute.instances.delete",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "event_timestamp_us": "1560301035092322",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "event_type": "GCE_OPERATION_DONE",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "operation": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "id": "971500189857477422",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "name": "operation-1560300992590-58b15e267f2cc-e7529c4d-f1343924",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "type": "operation",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "zone": "us-west1-a"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "resource": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "id": "613579339353422259",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "name": "test",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "type": "instance",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "zone": "us-west1-a"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "trace_id": "operation-1560300992590-58b15e267f2cc-e7529c4d-f1343924",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "version": "1.2"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "labels": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "compute.googleapis.com/resource_id": "613579339353422259",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "compute.googleapis.com/resource_name": "test",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "compute.googleapis.com/resource_type": "instance",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "compute.googleapis.com/resource_zone": "us-west1-a"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "logName": "projects/user-dev-242122/logs/compute.googleapis.com%2Factivity_log",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "receiveTimestamp": "2019-06-12T00:57:15.146606691Z",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "resource": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "labels": {
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "instance_id": "613579339353422259",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "project_id": "user-dev-242122",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438              "zone": "us-west1-a"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438          "type": "gce_instance"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      },
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "severity": "INFO",
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438      "timestamp": "2019-06-12T00:57:15.092322Z"
I      dns_vm_gc  578389534045377  2019-06-12 01:30:20.438  }
D      dns_vm_gc  578389534045377  2019-06-12 01:30:20.439  Function execution took 7 ms, finished with status: 'ok'
```

Unit Tests      Run unit tests after installing development dependencis      bash pip install  r requirements dev txt pytest      Save and Replay VM Deletion Events      It is useful to replay VM deletion events to test out changes to the Background Function   See the  Replay Quickstart  replay qs  for more information    1  Deploy the function  2  Get the subscription of the function with  gcloud pubsub subscriptions     list   3  Create a pubsub snapshot with  gcloud pubsub snapshots create vm deletions       subscription  subscription    4  Delete a VM instance  5  Deploy a new version of the same function  6  Replay the deletion events to the new version  gcloud pubsub subscriptions     seek  subscription    snapshot vm deletions   7  Inspect the function output with  gcloud functions logs read   limit 50   Interactive Testing      A run helper method is provided to run the function from a local workstation  First  make sure you have the following environment variables set as if they would be in the context of GCF      bash   env   grep DNS DNS VM GC DNS PROJECT dnsregistration DNS VM GC DNS ZONES nonprod private zone      If necessary  use Application Default Credentials      bash   env   grep GOOG GOOGLE APPLICATION CREDENTIALS  Users jmccune  credentials dns logging 83a9d261f444 json      Run the python REPL with debugging enabled via the DEBUG environment variable      ipython Python 3 7 3  default  Mar 27 2019  09 23 39   Clang 10 0 0  clang 1000 11 45 5   on darwin Type  help    copyright    credits  or  license  for more information      import conftest     from main test import run     run     message    BEGIN Zone cleanup    managed zone    nonprod private zone     message    BEGIN search for deletion candidates    instance    test    ip    10 138 0 45     message    Skipped  not an A record    record     name    gcp example com     type    NS    ttl   21600   rrdatas     ns gcp private googledomains com      signatureRrdatas        kind    dns resourceRecordSet      message    Skipped  not an A record    record     name    gcp example com     type    SOA    ttl   21600   rrdatas     ns gcp private googledomains com  cloud dns hostmaster google com  1 21600 3600 259200 300     signatureRrdatas        kind    dns resourceRecordSet      message    Skipped  shortname    instance    record     name    keep gcp example com     type    A    ttl   300   rrdatas     10 138 0 45     signatureRrdatas        kind    dns resourceRecordSet      message    Skipped  ip does not match    record     name    test keep gcp example com     type    A    ttl   300   rrdatas     10 138 0 43    10 138 0 44    10 138 0 45     signatureRrdatas        kind    dns resourceRecordSet      message    Skipped  ip does not match    record     name    test nonprod gcp example com     type    A    ttl   300   rrdatas     10 138 0 250     signatureRrdatas        kind    dns resourceRecordSet      message    END search for deletion candidates    candidates         message    END Zone cleanup    managed zone    nonprod private zone            Unit Tests      Run unit tests to interact with fixture data   Fixture data is provided for the main entry point of a pubsub message sent to Google Cloud Functions and for the API response when collecting the IP address from the compute API   Test dependencies      Install test dependencies      bash pip install pytest pip install google api python client      Run the tests          pytest  vv tests      Sample events      This is the JSON log data sent from Stackdriver s compute instances delete event to Pub Sub and then to the Background Function implemented in Python  There are two events  one for the GCE API CALL initiating the VM deletion  the second for the GCE OPERATION DONE event concluding the VM deletion   Obtained using  gcloud functions logs read   limit 100       txt D      dns vm gc  578383257362746  2019 06 12 01 30 09 128  Function execution started I      dns vm gc  578383257362746  2019 06 12 01 30 09 145    I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       insertId    6o9bdnfxt05mn   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       jsonPayload     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           actor     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               user    jmccune google com  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145             I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           event subtype    compute instances delete   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           event timestamp us    1560300993146327   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           event type    GCE API CALL   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           ip address       I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           operation     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               id    971500189857477422   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               name    operation 1560300992590 58b15e267f2cc e7529c4d f1343924   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               type    operation   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               zone    us west1 a  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145             I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           request     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               body    null   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               url    https   www googleapis com compute v1 projects user dev 242122 zones us west1 a instances test key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145             I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           resource     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               id    613579339353422259   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               name    test   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               type    instance   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               zone    us west1 a  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145             I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           trace id    operation 1560300992590 58b15e267f2cc e7529c4d f1343924   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           user agent    Mozilla 5 0  Macintosh  Intel Mac OS X 10 14 5  AppleWebKit 537 36  KHTML  like Gecko  Chrome 74 0 3729 169 Safari 537 36   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           version    1 2  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145         I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       labels     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           compute googleapis com resource id    613579339353422259   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           compute googleapis com resource name    test   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           compute googleapis com resource type    instance   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           compute googleapis com resource zone    us west1 a  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145         I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       logName    projects user dev 242122 logs compute googleapis com 2Factivity log   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       receiveTimestamp    2019 06 12T00 56 33 188262452Z   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       resource     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           labels     I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               instance id    613579339353422259   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               project id    user dev 242122   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145               zone    us west1 a  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145             I      dns vm gc  578383257362746  2019 06 12 01 30 09 145           type    gce instance  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145         I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       severity    INFO   I      dns vm gc  578383257362746  2019 06 12 01 30 09 145       timestamp    2019 06 12T00 56 33 146327Z  I      dns vm gc  578383257362746  2019 06 12 01 30 09 145    D      dns vm gc  578383257362746  2019 06 12 01 30 09 146  Function execution took 20 ms  finished with status   ok  D      dns vm gc  578389534045377  2019 06 12 01 30 20 433  Function execution started I      dns vm gc  578389534045377  2019 06 12 01 30 20 438    I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       insertId    hrbv6bfh9qzr2   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       jsonPayload     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           actor     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               user    jmccune google com  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438             I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           event subtype    compute instances delete   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           event timestamp us    1560301035092322   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           event type    GCE OPERATION DONE   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           operation     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               id    971500189857477422   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               name    operation 1560300992590 58b15e267f2cc e7529c4d f1343924   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               type    operation   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               zone    us west1 a  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438             I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           resource     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               id    613579339353422259   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               name    test   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               type    instance   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               zone    us west1 a  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438             I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           trace id    operation 1560300992590 58b15e267f2cc e7529c4d f1343924   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           version    1 2  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438         I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       labels     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           compute googleapis com resource id    613579339353422259   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           compute googleapis com resource name    test   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           compute googleapis com resource type    instance   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           compute googleapis com resource zone    us west1 a  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438         I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       logName    projects user dev 242122 logs compute googleapis com 2Factivity log   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       receiveTimestamp    2019 06 12T00 57 15 146606691Z   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       resource     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           labels     I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               instance id    613579339353422259   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               project id    user dev 242122   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438               zone    us west1 a  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438             I      dns vm gc  578389534045377  2019 06 12 01 30 20 438           type    gce instance  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438         I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       severity    INFO   I      dns vm gc  578389534045377  2019 06 12 01 30 20 438       timestamp    2019 06 12T00 57 15 092322Z  I      dns vm gc  578389534045377  2019 06 12 01 30 20 438    D      dns vm gc  578389534045377  2019 06 12 01 30 20 439  Function execution took 7 ms  finished with status   ok


GCP guaranteed A race exists between the function obtaining the VM IP address and Please note DNS record deletion is implemented however cannot be is obtained the function will not delete the DNS record because it cannot the operation If the VM is deleted before the IP when a VM is deleted VM DNS Garbage Collection This folder contains a Background Function bg which deletes DNS A records

Answer:
VM DNS Garbage Collection
===

This folder contains a [Background Function][bg] which deletes DNS A records
when a VM is deleted.

**Please note** DNS record deletion is implemented, however, cannot be
guaranteed.  A race exists between the function obtaining the VM IP address and
the `compute.instances.delete` operation.  If the VM is deleted before the IP
is obtained, the function will not delete the DNS record because it cannot
check the IP address matches the VM being deleted.

In practice this background function collects the IP address well within the
~30 second window of the VM delete operation.

Structured logs enable VM deletions which were not processed because the race
was lost.  See [Lost Race](#lost-race) for log filters to identify VM's deleted
before cleanup could take place.

![Example Log Output](./img/example_logs.png)

Project Setup
===

This example has been developed for use with multiple service projects.  A
centralized logs project is used to host one pubsub topic for all VM deletion
events.  One deployment of the function implements the event handler.

 * The logs project contains the dns-vm-gc Pub/Sub topic and the
   dns_vm_gc function deployed as a Background Function.
 * One or more service projects contain VM resources to be deleted.
 * The host project contains a VPC shared with the user project and DNS
   resource record sets needing to be cleaned up automatically.

Identify the Logs Project
===

Identify a project to host the `vm-deletions` Pub/Sub topic and the DNS VM GC
Cloud Function.  Service projects are configured to export filtered logs into
this topic.

If a project does not already exist, create a new project.  A suggested name is
`logs`.  The rest of this document will use `logs-123456` as the project ID for
the centralized logs project.

Create the vm-deletions Pub/Sub topic
---

Service projects export `compute.instances.delete` events to the `vm-deletions`
topic.  The VM DNS GC background function subscribes to this topic and triggers
on each event.

Create a topic named `vm-deletions` in the logs project as per [Create a
topic][pubsub-quickstart].

Configure Log Exports
---

Configure Log Exports in one or more service projects.  Logs are exported to
the `vm-deletions` topic in the logs project.

[Stackdriver logs exports][logs-exports] are used to convey VM lifecycle events
to the DNS VM GC function via Cloud Pub/Sub.  A Stackdriver filter is used to
limit logs to VM deletion events, reducing data traveling through Pub/Sub.

Configure an export to the `vm-deletions` topic with the following filter, for
example `projects/logs-123456/topics/vm-deletions`.

```
resource.type="gce_instance"
jsonPayload.event_type="GCE_API_CALL"
jsonPayload.event_subtype="compute.instances.delete"
```

This filter results in one event published per VM deletion, a `GCE_API_CALL`
event when the VM deletion is requested.

If additional events are published to the topic, the function triggers, but
ignores events which do not match this filter.

Service Account
===

The Background Function runs with a service account identity.  Create a service
account named `dns-vm-gc` in the logs project for this purpose.  This example
assumes [GCP-managed][sa-gcp-managed] keys.

If you are modifying this example you may download the service account key and
run locally as the service account using the GOOGLE_APPLICATION_CREDENTIALS
environment file.  See [Providing credentials to your application][adc] for
details.

Service Account Roles
===

The Background Function service account requires the following roles.

DNS Admin
---

Grant the DNS Admin role to the dns-vm-gc service account in the host project.
DNS Admin allows the DNS VM GC function to delete DNS records in the host
project.

This role may be granted at the Shared VPC project level.

Compute Viewer
---

Grant the Compute Viewer role to the dns-vm-gc service account.  Compute Viewer
allows the DNS VM GC function to read the IP address of the VM, necessary to
ensure the correct A record is deleted.

This role may be granted at the project, folder or organization level as
appropriate.

Logs Writer
---

Grant the Logs Writer role to the dns-vm-gc service account.  Logs Writer is
required to write structured event logs to the [Reporting
Stream](#reporting-stream).

This role may be granted at the project, folder, or organization level as
appropriate.  It is recommended to grant the role at the same level the log
stream exists at, the logging project by default.  See [Custom Reporting
Destination](#custom-reporting-destination) for more information.

Deployment
===

Deploy this function into the logs project to simplify the subscription to the
`vm-deletions` topic.

Environment variables are used to configure the behavior of the function.
Update the env.yaml file to reflect the correct VPC Host project and Managed
Zone names for your environment.  A sample is provided in env.yaml.sample.

```yaml
# env.yaml
---
DNS_VM_GC_DNS_PROJECT: my-vpc-host-project
DNS_VM_GC_DNS_ZONES: my-nonprod-private-zone,my-prod-private-zone
```

```bash
gcloud functions deploy dns_vm_gc \
  --retry \
  --runtime=python37 \
  --service-account=dns-vm-gc@logs-123456.iam.gserviceaccount.com \
  --trigger-topic=vm-deletions \
  --env-vars-file=env.yaml
```

Logging and Reporting
===

The DNS VM GC function logs into two different locations.  Structured Events
intended for reporting are sent to a special purpose reporting stream.  Plain
text logs are sent to the standard Cloud Function logs accessible via `gcloud
functions logs read`.

Reporting Stream
---

The reporting stream is intended to answer two primary questions:

 1. Which VM deletion events, if any, were not processed?
 2. What records were deleted automatically?

When the function loses the race against the delete operation, the event is not
processed and the function reports a detail code of `LOST_RACE`.

When the function deletes a record automatically, the fully qualified domain
name is logged along with a detail code of `RR_DELETED` for resource record
deleted.

Custom Reporting Destination
---

By default the reporting stream is located at
`projects/<logs_project>/logs/<function_name>`.  The reporting stream is
configurable by setting the `DNS_VM_GC_REPORTING_LOG_STREAM` environment
variable when deploying the function.  For example, to send reporting events to
the organization level:

```yaml
# env.yaml
---
DNS_VM_GC_DNS_PROJECT: my-vpc-host-project
DNS_VM_GC_DNS_ZONES: my-nonprod-private-zone,my-prod-private-zone
DNS_VM_GC_REPORTING_LOG_STREAM: organizations/000000000000/logs/dns-vm-gc-report
```

See the `logName` field of the [LogEntry][logentry] resource for a list of
possible report stream destinations.

Reading the Report Logs
---

Download all structured logs to the report stream produced by the function
using:

```bash
gcloud functions logs read logName="projects/<logs_project>/logs/<function_name>"
```

Cloud Function Logs
---

The function also logs unstructured plain text logs using [Cloud Function
Logs][gcf-logs].  Becasue these logs are unstructured, they are less useful
than the Report Stream logs for reporting purposes, however, are present to
keep all activity associated together with each execution ID of the function.

Note the cloud function logs have an execution_id.  This execution ID is not
readily available at runtime and therefore absent from the structured report
log stream.  The function logs a message with the `event_id` being processed to
associate the execution_id with the event_id.  This behavior is intended to
correlate each execution in the Cloud Function Logs with each report in the
Report Stream.  The correlation of execution_id to event_id is not necessary
for day to day reporting.  The correlation is useful for the rare situation of
complete end-to-end tracing.

Reporting
===

Lost Race
---

Periodic reporting should be performed to monitor for `NOT_PROCESSED` results.
In the event of a lost race, automatic DNS record deletion is not guaranteed.

The following Stackdriver Advanced Filter identifies when a VM deletion event
was not processed automatically:

```
resource.type="cloud_function"
resource.labels.function_name="dns_vm_gc"
logName="projects/dns-logging/logs/dns-vm-gc-report"
jsonPayload.result="NOT_PROCESSED"
```

Deleted Resource Records
---

All records automatically deleted may be identified with the a filter on the
detail code.

```
resource.type="cloud_function"
resource.labels.function_name="dns_vm_gc"
logName="projects/dns-logging/logs/dns-vm-gc-report"
jsonPayload.detail="RR_DELETED"
```

Debug Logs
---

Debug logs are also available, but are not sent by default.  To enable, deploy
the function with the `DEBUG` environment variable set to a non-empty string.
Note, debug logs generates 2*N log events every time a VM is deleted where N is
the number of DNS records across all configured managed zones.  For example,
deleting 10 VM instances with 1,000 managed DNS records generates 20,000 debug
log entries at minimum.

Detail Codes
---

The following detail codes may be reported to the reporting stream:

| Detail Code   | Description                                     | Result        |
| -----------   | -----------                                     | ------        |
| NO_MATCHES    | No DNS records matched the VM deleted           | OK            |
| RR_DELETED    | A DNS record matched and has been deleted       | OK            |
| VM_NO_IP      | The function won the race, but the VM has no IP | OK            |
| IGNORED_EVENT | Trigger event is not a VM delete GCE_API_CALL   | OK            |
| LOST_RACE     | The VM was deleted before the IP was determined | NOT_PROCESSED |

In addition, there are detail codes when DEBUG is turned on indicating the
reason why DNS records were not automatically deleted.

| Detail Code      | Reason DNS record not deleted              | Result |
| -----------      | -----------------------------              | ------ |
| RR_NOT_A_RECORD  | Resource Record is not an A record         | OK     |
| RR_NAME_MISMATCH | Shortname doesn't match the VM name        | OK     |
| RR_IP_MISMATCH   | rrdatas is not one IP matching the VM's IP | OK     |

[bg]: https://cloud.google.com/functions/docs/writing/background
[sa-gcp-managed]: https://cloud.google.com/iam/docs/understanding-service-accounts#managing_service_account_keys
[pubsub-quickstart]: https://cloud.google.com/pubsub/docs/quickstart-console#create_a_topic
[logs-exports]: https://cloud.google.com/logging/docs/export/
[adc]: https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application
[logentry]: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
[gcf-logs]: https://cloud.google.com/functions/docs/monitoring/logging

VM DNS Garbage Collection      This folder contains a  Background Function  bg  which deletes DNS A records when a VM is deleted     Please note   DNS record deletion is implemented  however  cannot be guaranteed   A race exists between the function obtaining the VM IP address and the  compute instances delete  operation   If the VM is deleted before the IP is obtained  the function will not delete the DNS record because it cannot check the IP address matches the VM being deleted   In practice this background function collects the IP address well within the  30 second window of the VM delete operation   Structured logs enable VM deletions which were not processed because the race was lost   See  Lost Race   lost race  for log filters to identify VM s deleted before cleanup could take place     Example Log Output    img example logs png   Project Setup      This example has been developed for use with multiple service projects   A centralized logs project is used to host one pubsub topic for all VM deletion events   One deployment of the function implements the event handler      The logs project contains the dns vm gc Pub Sub topic and the    dns vm gc function deployed as a Background Function     One or more service projects contain VM resources to be deleted     The host project contains a VPC shared with the user project and DNS    resource record sets needing to be cleaned up automatically   Identify the Logs Project      Identify a project to host the  vm deletions  Pub Sub topic and the DNS VM GC Cloud Function   Service projects are configured to export filtered logs into this topic   If a project does not already exist  create a new project   A suggested name is  logs    The rest of this document will use  logs 123456  as the project ID for the centralized logs project   Create the vm deletions Pub Sub topic      Service projects export  compute instances delete  events to the  vm deletions  topic   The VM DNS GC background function subscribes to this topic and triggers on each event   Create a topic named  vm deletions  in the logs project as per  Create a topic  pubsub quickstart    Configure Log Exports      Configure Log Exports in one or more service projects   Logs are exported to the  vm deletions  topic in the logs project    Stackdriver logs exports  logs exports  are used to convey VM lifecycle events to the DNS VM GC function via Cloud Pub Sub   A Stackdriver filter is used to limit logs to VM deletion events  reducing data traveling through Pub Sub   Configure an export to the  vm deletions  topic with the following filter  for example  projects logs 123456 topics vm deletions        resource type  gce instance  jsonPayload event type  GCE API CALL  jsonPayload event subtype  compute instances delete       This filter results in one event published per VM deletion  a  GCE API CALL  event when the VM deletion is requested   If additional events are published to the topic  the function triggers  but ignores events which do not match this filter   Service Account      The Background Function runs with a service account identity   Create a service account named  dns vm gc  in the logs project for this purpose   This example assumes  GCP managed  sa gcp managed  keys   If you are modifying this example you may download the service account key and run locally as the service account using the GOOGLE APPLICATION CREDENTIALS environment file   See  Providing credentials to your application  adc  for details   Service Account Roles      The Background Function service account requires the following roles   DNS Admin      Grant the DNS Admin role to the dns vm gc service account in the host project  DNS Admin allows the DNS VM GC function to delete DNS records in the host project   This role may be granted at the Shared VPC project level   Compute Viewer      Grant the Compute Viewer role to the dns vm gc service account   Compute Viewer allows the DNS VM GC function to read the IP address of the VM  necessary to ensure the correct A record is deleted   This role may be granted at the project  folder or organization level as appropriate   Logs Writer      Grant the Logs Writer role to the dns vm gc service account   Logs Writer is required to write structured event logs to the  Reporting Stream   reporting stream    This role may be granted at the project  folder  or organization level as appropriate   It is recommended to grant the role at the same level the log stream exists at  the logging project by default   See  Custom Reporting Destination   custom reporting destination  for more information   Deployment      Deploy this function into the logs project to simplify the subscription to the  vm deletions  topic   Environment variables are used to configure the behavior of the function  Update the env yaml file to reflect the correct VPC Host project and Managed Zone names for your environment   A sample is provided in env yaml sample      yaml   env yaml     DNS VM GC DNS PROJECT  my vpc host project DNS VM GC DNS ZONES  my nonprod private zone my prod private zone         bash gcloud functions deploy dns vm gc       retry       runtime python37       service account dns vm gc logs 123456 iam gserviceaccount com       trigger topic vm deletions       env vars file env yaml      Logging and Reporting      The DNS VM GC function logs into two different locations   Structured Events intended for reporting are sent to a special purpose reporting stream   Plain text logs are sent to the standard Cloud Function logs accessible via  gcloud functions logs read    Reporting Stream      The reporting stream is intended to answer two primary questions    1  Which VM deletion events  if any  were not processed   2  What records were deleted automatically   When the function loses the race against the delete operation  the event is not processed and the function reports a detail code of  LOST RACE    When the function deletes a record automatically  the fully qualified domain name is logged along with a detail code of  RR DELETED  for resource record deleted   Custom Reporting Destination      By default the reporting stream is located at  projects  logs project  logs  function name     The reporting stream is configurable by setting the  DNS VM GC REPORTING LOG STREAM  environment variable when deploying the function   For example  to send reporting events to the organization level      yaml   env yaml     DNS VM GC DNS PROJECT  my vpc host project DNS VM GC DNS ZONES  my nonprod private zone my prod private zone DNS VM GC REPORTING LOG STREAM  organizations 000000000000 logs dns vm gc report      See the  logName  field of the  LogEntry  logentry  resource for a list of possible report stream destinations   Reading the Report Logs      Download all structured logs to the report stream produced by the function using      bash gcloud functions logs read logName  projects  logs project  logs  function name        Cloud Function Logs      The function also logs unstructured plain text logs using  Cloud Function Logs  gcf logs    Becasue these logs are unstructured  they are less useful than the Report Stream logs for reporting purposes  however  are present to keep all activity associated together with each execution ID of the function   Note the cloud function logs have an execution id   This execution ID is not readily available at runtime and therefore absent from the structured report log stream   The function logs a message with the  event id  being processed to associate the execution id with the event id   This behavior is intended to correlate each execution in the Cloud Function Logs with each report in the Report Stream   The correlation of execution id to event id is not necessary for day to day reporting   The correlation is useful for the rare situation of complete end to end tracing   Reporting      Lost Race      Periodic reporting should be performed to monitor for  NOT PROCESSED  results  In the event of a lost race  automatic DNS record deletion is not guaranteed   The following Stackdriver Advanced Filter identifies when a VM deletion event was not processed automatically       resource type  cloud function  resource labels function name  dns vm gc  logName  projects dns logging logs dns vm gc report  jsonPayload result  NOT PROCESSED       Deleted Resource Records      All records automatically deleted may be identified with the a filter on the detail code       resource type  cloud function  resource labels function name  dns vm gc  logName  projects dns logging logs dns vm gc report  jsonPayload detail  RR DELETED       Debug Logs      Debug logs are also available  but are not sent by default   To enable  deploy the function with the  DEBUG  environment variable set to a non empty string  Note  debug logs generates 2 N log events every time a VM is deleted where N is the number of DNS records across all configured managed zones   For example  deleting 10 VM instances with 1 000 managed DNS records generates 20 000 debug log entries at minimum   Detail Codes      The following detail codes may be reported to the reporting stream     Detail Code     Description                                       Result                                                                                NO MATCHES      No DNS records matched the VM deleted             OK                RR DELETED      A DNS record matched and has been deleted         OK                VM NO IP        The function won the race  but the VM has no IP   OK                IGNORED EVENT   Trigger event is not a VM delete GCE API CALL     OK                LOST RACE       The VM was deleted before the IP was determined   NOT PROCESSED    In addition  there are detail codes when DEBUG is turned on indicating the reason why DNS records were not automatically deleted     Detail Code        Reason DNS record not deleted                Result                                                                                RR NOT A RECORD    Resource Record is not an A record           OK         RR NAME MISMATCH   Shortname doesn t match the VM name          OK         RR IP MISMATCH     rrdatas is not one IP matching the VM s IP   OK          bg   https   cloud google com functions docs writing background  sa gcp managed   https   cloud google com iam docs understanding service accounts managing service account keys  pubsub quickstart   https   cloud google com pubsub docs quickstart console create a topic  logs exports   https   cloud google com logging docs export   adc   https   cloud google com docs authentication production providing credentials to your application  logentry   https   cloud google com logging docs reference v2 rest v2 LogEntry  gcf logs   https   cloud google com functions docs monitoring logging


GCP representation for any use or purpose Your use of it is subject to your agreement with Google Twilio Conversation Integration with a Virtual Agent using Dialogflow Project Structure This is an example how to integrate a Twilio Conversation Services with Virtual Copyright 2023 Google This software is provided as is without warranty or Agent using Dialogflow

Answer:
Copyright 2023 Google. This software is provided as-is, without warranty or
representation for any use or purpose. Your use of it is subject to your
agreement with Google.

# Twilio Conversation Integration with a Virtual Agent using Dialogflow

This is an example how to integrate a Twilio Conversation Services with Virtual
Agent using Dialogflow.

## Project Structure

```
.
└── twilio
   └── src
   └── main
           └── java
               └── com.middleware.controller
                   ├── cache # redis initialization and mapping of conversations
                   ├── dialogflow # handler of a conversation with dialogflow
                   ├── rest # entry point and request handler
                   ├── twilio # marketplace and twilio conversation services set up and initialization
                   ├── util # utility to inittialize a twilio conversation to test the integration
                   └── webhook # classes to process new message and new participant
          └── resources
              └── application.properties
          └── proto
              └── dialogflow.proto # dialogflow conversation information holder
   ├── pom.xml
   └── README.md
```

## Components

- Dialogflow
- Google Cloud MemoryStore (Redis)
- Twilio with Flex

## How mapping between Twilio Conversation and Dialogflow Conversation is implemented

We use a mapping between the Twilio Conversation SID and the Dialogflow Conversation ID
to maintain the conversation context. This mapping is stored in the [Redis
cache.](https://cloud.google.com/memorystore/docs/redis/redis-overview) and it works as followed:

1. If there is no active conversation available in
   the Redis cache, we create a new Dialogflow conversation and store the
   mapping in the Redis cache.
2. If there is an active conversation available in the Redis cache, we use
   the same Dialogflow conversation to send the message to the Dialogflow
   agent.
3. If the user is not already present in the conversation, we add them to the
   conversation.
4. On each message, we send the message to the Dialogflow agent and get the
   response
   using [AnalyzeContent API](https://developers.google.com/resources/api-libraries/documentation/dialogflow/v2beta1/java/latest/index.html?com/google/api/services/dialogflow/v2beta1/Dialogflow.Projects.Conversations.Participants.AnalyzeContent.html)
5. Reply message is sent to the user using the Twilio Conversations API.
6. At the time of handoff, we send the conversation context to the Flex UI
   using the [Interaction API](https://www.twilio.com/docs/flex/developer/conversations/interactions-api/interactions).
7. If the Agent Assist feature is enabled, we won't close the conversation in
   Dialogflow. Otherwise, we will close the conversation in Dialogflow if
   agent-handoff or end-of-conversation is detected.
9. If the conversation is closed in Dialogflow, we delete the mapping from
   the Redis cache.

## Setup Instructions

### GCP Project Setup

#### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
1. In the drop-down menu at the top, select **Create a project**.
1. Give your project a name.
1. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

#### Enabling billing for your project.

If you haven't already enabled billing for your project, [enable
billing][enable-billing] now. Enabling billing allows is required to use Cloud
Bigtable and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

#### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud SDK, [install the Google
Cloud SDK][cloud-sdk] now. The SDK contains tools and libraries that enable you
to create and manage resources on Google Cloud Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

#### Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
   gcloud init
```

Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
command:

```
    gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing

[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

### Local Development Set Up

This is a Spring Boot application designed to run on port 8080. Upon launch, the
application will initialize and bind to port 8080, making it accessible for
incoming connections.

#### Set of environment variables:

The following environmental variables need to be set up in the localhost:

```
REDIS_HOST :
   IP address of the Redis instance

REDIS_PORT :
   Port of the Redis instance (Default: 6379)
```

```
TWILIO_ADD_ON_CONVERSATION_PROFILE_ID :
   Dialogflow Conversation Profile ID (Provided by Twilio Integration)

TWILIO_ADD_ON_PROJECT_ID :
   GCP Project ID where the Dialogflow Agent is deployed (Provided
   by Twilio Integration)

TWILIO_ADD_ON_AGENT_LOCATION(Optional):
   Dialogflow Agent Location (Provided by Twilio Integration). Default: global
```

```
TWILIO_ACCOUNT_SID :
   Twilio Account SID

TWILIO_AUTH_TOKEN :
   OAuth Token for the Twilio Account

TWILIO_WORKSPACE_SID :
   Flex Workspace SID, (Used for interaction task creation)

TWILIO_WORKFLOW_SID :
   Flex Workflow SID, (Used for interaction task creation)
```

#### Redis Set Up

##### Install a Redis Emulator

Please refer to this [doc](https://redis.io/docs/getting-started/) to install a
redis emulator in your localhost

For Linux:
[Install Redis on Linux](https://redis.io/docs/getting-started/installation/install-redis-on-linux/)

##### Initialized the server

```
   $ redis-server
```

##### Basic commands to access the data

List all the keys

```
  $ redis-cli
   127.0.0.1:6379> KEYS *
   (empty array)
```

Delete a key

```
   127.0.0.1:6379> DEL "<key>"
```

Get a key

```
  127.0.0.1:6379> GET "<key>"
```

## Usage

### Endpoints

> POST /handleConversations

Events Handled

1. **onParticipantAdded** : Expected Variables from the request
    - ConversationSid
2. **onMessageAdded** : Expected Variables from the request
    - ConversationSid
    - Body
    - Source

### Initialize the application

Reference: [Building an Application with Spring Boot](https://spring.io/guides/gs/spring-boot/)

```
   ./mvnw spring-boot:run
```

### Send a request

```
 curl --location --request POST 'localhost:8080/handleConversations' \
 --header 'Content-Type: application/x-www-form-urlencoded' \
 --data-urlencode 'Body=Talk to agent' \
 --data-urlencode 'EventType=onMessageAdded' \
 --data-urlencode 'Source=whatsapp' \
 --data-urlencode 'ConversationSid=XXXXXX' \
```

## How to get the dialogflow conversation profile id

Dialogflow Conversation Profile ID also known as Integration ID is a unique
text which is used for the interaction of Dialogflow with 3rd party applications
like Twilio.

> [Create/Edit a Dialogflow Conversation Profile](https://cloud.google.com/agent-assist/docs/conversation-profile#create_and_edit_a_conversation_profile)

You can also create a conversation profile using Twilio One Click Integration
from the Dialogflow Console.

> Dialogflow Console > Corresponding Agent > Manage > Integrations >
> Twilio > One Click Integration > Connect

Once the conversation profile is created, you can find the conversation profile
id in the following ways:

> Open [Agent Assist](https://agentassist.cloud.google.com/), then Conversation
> Profiles on the left bottom

## How to create a Twilio conversation

1. Create a Conversation programmatically using the
   Twilio [Conversations API](https://www.twilio.com/docs/conversations/api/conversation-resource?code-sample=code-create-conversation&code-language=curl&code-sdk-version=json)
2. Add the customer chat participant to the conversation. Use the SID of the
   conversation that you created in the previous
   step. [Reference](https://www.twilio.com/docs/conversations/api/conversation-participant-resource?code-sample=code-create-conversation-participant-chat&code-language=curl&code-sdk-version=json)
3. Add a [scoped webhook](https://www.twilio.com/docs/conversations/api/conversation-scoped-webhook-resource?code-sample=code-create-attach-a-new-conversation-scoped-webhook&code-language=curl&code-sdk-version=json)
   with "webhook" as the target using the Conversations Webhook endpoint. With a [scoped webhook](https://www.twilio.com/docs/conversations/api/conversation-scoped-webhook-resource?code-sample=code-create-attach-a-new-conversation-scoped-webhook&code-language=curl&code-sdk-version=json)
   set to your endpoint and the configuration filter set to "onMessageAdded", any message that is added to the conversation will invoke the configured webhook URL.
4) Simulate a new customer message by using the Message endpoint.
   Remember to set Author to the identity you set in step 2 and to add the
   header “X-Twilio-Webhook-Enabled” to the request so our webhook gets invoked
5) Use the Twilio [Interaction API](https://www.twilio.com/docs/flex/developer/conversations/interactions-api)
   to invoke a handoff to the Flex UI

## How to run the initializer to programmatically create a Twilio conversation

> [ConversationInitializer](./src/main/java/com/middleware/controller/util/ConversationInitializer.java)

```
mvn -DskipTests package exec:java
        -Dexec.mainClass=com.middleware.controller.util.ConversationInitializer
        -Dexec.args="add"

mvn -DskipTests package exec:java
        -Dexec.mainClass=com.middleware.controller.util.ConversationInitializer
        -Dexec.args="delete <conversation sid>"
```

# Deployment

JIB is used to build the docker image and push it to the GCR.

```bash
 mvn compile jib:build
```

# References

1.[Twilio Conversations Fundamentals](https://www.twilio.com/docs/conversations/fundamentals)
2.[Dialogflow](https://cloud.google.com/dialogflow/es/docs)

Copyright 2023 Google  This software is provided as is  without warranty or representation for any use or purpose  Your use of it is subject to your agreement with Google     Twilio Conversation Integration with a Virtual Agent using Dialogflow  This is an example how to integrate a Twilio Conversation Services with Virtual  Agent using Dialogflow      Project Structure            twilio        src        main                java                    com middleware controller                        cache   redis initialization and mapping of conversations                        dialogflow   handler of a conversation with dialogflow                        rest   entry point and request handler                        twilio   marketplace and twilio conversation services set up and initialization                        util   utility to inittialize a twilio conversation to test the integration                          webhook   classes to process new message and new participant                   resources                   application properties               proto                   dialogflow proto   dialogflow conversation information holder        pom xml        README md         Components    Dialogflow   Google Cloud MemoryStore  Redis    Twilio with Flex     How mapping between Twilio Conversation and Dialogflow Conversation is implemented  We use a mapping between the Twilio Conversation SID and the Dialogflow Conversation ID to maintain the conversation context  This mapping is stored in the  Redis cache   https   cloud google com memorystore docs redis redis overview  and it works as followed   1  If there is no active conversation available in    the Redis cache  we create a new Dialogflow conversation and store the    mapping in the Redis cache  2  If there is an active conversation available in the Redis cache  we use    the same Dialogflow conversation to send the message to the Dialogflow    agent  3  If the user is not already present in the conversation  we add them to the    conversation  4  On each message  we send the message to the Dialogflow agent and get the    response    using  AnalyzeContent API  https   developers google com resources api libraries documentation dialogflow v2beta1 java latest index html com google api services dialogflow v2beta1 Dialogflow Projects Conversations Participants AnalyzeContent html  5  Reply message is sent to the user using the Twilio Conversations API  6  At the time of handoff  we send the conversation context to the Flex UI    using the  Interaction API  https   www twilio com docs flex developer conversations interactions api interactions   7  If the Agent Assist feature is enabled  we won t close the conversation in    Dialogflow  Otherwise  we will close the conversation in Dialogflow if    agent handoff or end of conversation is detected  9  If the conversation is closed in Dialogflow  we delete the mapping from    the Redis cache      Setup Instructions      GCP Project Setup       Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   1  In the drop down menu at the top  select   Create a project    1  Give your project a name  1  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com        Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now  Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings       Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk        Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command          gcloud init      Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command           gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing   application default credentials   https   developers google com identity protocols application default credentials      Local Development Set Up  This is a Spring Boot application designed to run on port 8080  Upon launch  the application will initialize and bind to port 8080  making it accessible for incoming connections        Set of environment variables   The following environmental variables need to be set up in the localhost       REDIS HOST      IP address of the Redis instance  REDIS PORT      Port of the Redis instance  Default  6379           TWILIO ADD ON CONVERSATION PROFILE ID       Dialogflow Conversation Profile ID  Provided by Twilio Integration   TWILIO ADD ON PROJECT ID      GCP Project ID where the Dialogflow Agent is deployed  Provided    by Twilio Integration   TWILIO ADD ON AGENT LOCATION Optional      Dialogflow Agent Location  Provided by Twilio Integration   Default  global             TWILIO ACCOUNT SID      Twilio Account SID  TWILIO AUTH TOKEN      OAuth Token for the Twilio Account  TWILIO WORKSPACE SID      Flex Workspace SID   Used for interaction task creation      TWILIO WORKFLOW SID      Flex Workflow SID   Used for interaction task creation               Redis Set Up        Install a Redis Emulator  Please refer to this  doc  https   redis io docs getting started   to install a redis emulator in your localhost  For Linux   Install Redis on Linux  https   redis io docs getting started installation install redis on linux          Initialized the server           redis server            Basic commands to access the data  List all the keys          redis cli    127 0 0 1 6379  KEYS       empty array       Delete a key         127 0 0 1 6379  DEL   key        Get a key        127 0 0 1 6379  GET   key           Usage      Endpoints    POST  handleConversations  Events Handled  1    onParticipantAdded     Expected Variables from the request       ConversationSid 2    onMessageAdded     Expected Variables from the request       ConversationSid       Body       Source       Initialize the application  Reference   Building an Application with Spring Boot  https   spring io guides gs spring boot             mvnw spring boot run          Send a request       curl   location   request POST  localhost 8080 handleConversations       header  Content Type  application x www form urlencoded       data urlencode  Body Talk to agent       data urlencode  EventType onMessageAdded       data urlencode  Source whatsapp       data urlencode  ConversationSid XXXXXX            How to get the dialogflow conversation profile id  Dialogflow Conversation Profile ID also known as Integration ID is a unique text which is used for the interaction of Dialogflow with 3rd party applications like Twilio      Create Edit a Dialogflow Conversation Profile  https   cloud google com agent assist docs conversation profile create and edit a conversation profile   You can also create a conversation profile using Twilio One Click Integration from the Dialogflow Console     Dialogflow Console   Corresponding Agent   Manage   Integrations     Twilio   One Click Integration   Connect  Once the conversation profile is created  you can find the conversation profile id in the following ways     Open  Agent Assist  https   agentassist cloud google com    then Conversation   Profiles on the left bottom      How to create a Twilio conversation  1  Create a Conversation programmatically using the    Twilio  Conversations API  https   www twilio com docs conversations api conversation resource code sample code create conversation code language curl code sdk version json  2  Add the customer chat participant to the conversation  Use the SID of the    conversation that you created in the previous    step   Reference  https   www twilio com docs conversations api conversation participant resource code sample code create conversation participant chat code language curl code sdk version json  3  Add a  scoped webhook  https   www twilio com docs conversations api conversation scoped webhook resource code sample code create attach a new conversation scoped webhook code language curl code sdk version json     with  webhook  as the target using the Conversations Webhook endpoint  With a  scoped webhook  https   www twilio com docs conversations api conversation scoped webhook resource code sample code create attach a new conversation scoped webhook code language curl code sdk version json     set to your endpoint and the configuration filter set to  onMessageAdded   any message that is added to the conversation will invoke the configured webhook URL  4  Simulate a new customer message by using the Message endpoint     Remember to set Author to the identity you set in step 2 and to add the    header  X Twilio Webhook Enabled  to the request so our webhook gets invoked 5  Use the Twilio  Interaction API  https   www twilio com docs flex developer conversations interactions api     to invoke a handoff to the Flex UI         How to run the initializer to programmatically create a Twilio conversation      ConversationInitializer    src main java com middleware controller util ConversationInitializer java       mvn  DskipTests package exec java           Dexec mainClass com middleware controller util ConversationInitializer           Dexec args  add           mvn  DskipTests package exec java           Dexec mainClass com middleware controller util ConversationInitializer           Dexec args  delete  conversation sid          Deployment  JIB is used to build the docker image and push it to the GCR      bash  mvn compile jib build        References  1  Twilio Conversations Fundamentals  https   www twilio com docs conversations fundamentals  2  Dialogflow  https   cloud google com dialogflow es docs


GCP Dataflow Streaming Schema Handler This package contains a set of components required to handle unanticipated incoming streaming data into BigQuery with schema mismatch The code will uses Schema enforcement and DLT Dead Letter Table approach to store schema incompability In case of schema incompability detected from the incoming message the incoming message will be evolved to match the targeted table schema by adding and removing fields which then will be store in the targetted table In addition the original message with schema incomability will be store in the DLT for debugging and record purposes Pipeline This is the main class deploying the pipeline The program waits for the Pub sub JSON push event If the predefined schema matches with the data ingests the streaming data into a successful BigQuery dataset

Answer:
# Dataflow Streaming Schema Handler

This package contains a set of components required to handle unanticipated incoming streaming data into BigQuery with schema mismatch.
The code will uses Schema enforcement and DLT (Dead Letter Table) approach to store schema incompability. In case of schema incompability detected from the incoming message, the incoming message will be evolved to match the targeted table schema by adding and removing fields which then will be store in the targetted table. In addition, the original message (with schema incomability) will be store in the DLT for debugging and record purposes.

## Pipeline

[Dataflow Streaming Schema Handler](src/main/java/com/google/cloud/pso/pipeline/PubSubToBigQueryJSON.java) -
This is the main class deploying the pipeline. The program waits for the Pub/sub JSON push event.
If the predefined schema matches with the data, ingests the streaming data into a successful BigQuery dataset.
Else if there is a schema mismatch, ingests data into an unsuccessful BigQuery dataset (DLT).

![Architecture Diagram](img/architecture.png "Architecture")

Happy path:
> Incoming message fields matches with the targetted schema
> Result are store in the targetted table

Schema mismatch:
> Incoming message fields did not match with the targetted schema
> The incoming message fields which matches with the targetted schema will be kept, any additional fields that does not match will be remove and in case of field are missing, the field will bet set to null value. The result will then be store in the targetted table
> The original message (with the mismatch schema) will be store in the DLT table

## Getting Started

### Requirements

* [gcloud sdk](https://cloud.google.com/sdk/docs/install-sdk)
* Java 11
* Maven 3

### Building the Project

Build the entire project using the maven compile command.
```sh
mvn clean compile
```

### Setting GCP Resources
```bash
# Set the pipeline vars
PROJECT_ID=<gcp-project-id>
BQ_DATASET=<bigquery-dataset-name>
BUCKET=<gcs-bucket>
PIPELINE_FOLDER=gs://${BUCKET}/dataflow/pipelines/streaming-benchmark
PUBSUB_TOPIC=<pubsub-topic>
PUBSUB_SUBS_ID=<subscriptions-id>
```

### Creating an example dataset
```sh
bq --location=US mk -d ${BQ_DATASET}
```

#### Create an example table
```sh
bq mk \
  --table \
  ${PROJECT_ID}:${BQ_DATASET}.tutorial_table \
  ./src/resources/person.json
```

#### Create an example PubSub topic and subscription

```sh
gcloud pubsub topics create ${PUBSUB_TOPIC}
gcloud pubsub subscriptions create ${PUBSUB_SUBS_ID} \
    --topic=${PUBSUB_TOPIC}
```

### Executing the Pipeline
The below instruction will deploy dataflow into the GCP project

```bash
# Set the runner
RUNNER=DataflowRunner

# Compute engine zone
REGION=us-central1

# Build the template
mvn compile exec:java \
-Dexec.mainClass=com.google.cloud.pso.pipeline.PubsubToBigQueryJSON \
-Dexec.cleanupDaemonThreads=false \
-Dexec.args=" \
    --project=${PROJECT_ID} \
    --dataset=${BQ_DATASET} \
    --stagingLocation=${PIPELINE_FOLDER}/staging \
    --tempLocation=${PIPELINE_FOLDER}/temp \
    --region=${REGION} \
    --maxNumWorkers=5 \
    --pubsubSubscription=projects/${PROJECT_ID}/subscriptions/${PUBSUB_SUBS_ID} \
    --runner=${RUNNER}"
```

### Sending sample data to pubsub

Once the dataflow job are deploy and running we can try to send few messages to see the output being store in the GCP project

Below are the happy path message
```sh
gcloud pubsub topics publish ${PUBSUB_TOPIC} \
  --message="{\"tutorial_table\":{\"person\":{\"name\":\"happyPath\",\"age\":22},\"date\": \"2022-08-03\",\"id\": \"xpze\"}}"
```

Below are the mismatch schema message, with "age" field being removed to the pubsub
```sh
gcloud pubsub topics publish ${PUBSUB_TOPIC} \
  --message="{\"tutorial_table\":{\"person\":{\"name\":\"mismatchSchema\"},\"date\": \"2022-08-03\",\"id\": \"avghy\"}}"
``

Dataflow Streaming Schema Handler  This package contains a set of components required to handle unanticipated incoming streaming data into BigQuery with schema mismatch  The code will uses Schema enforcement and DLT  Dead Letter Table  approach to store schema incompability  In case of schema incompability detected from the incoming message  the incoming message will be evolved to match the targeted table schema by adding and removing fields which then will be store in the targetted table  In addition  the original message  with schema incomability  will be store in the DLT for debugging and record purposes      Pipeline   Dataflow Streaming Schema Handler  src main java com google cloud pso pipeline PubSubToBigQueryJSON java    This is the main class deploying the pipeline  The program waits for the Pub sub JSON push event  If the predefined schema matches with the data  ingests the streaming data into a successful BigQuery dataset   Else if there is a schema mismatch  ingests data into an unsuccessful BigQuery dataset  DLT       Architecture Diagram  img architecture png  Architecture    Happy path    Incoming message fields matches with the targetted schema   Result are store in the targetted table  Schema mismatch    Incoming message fields did not match with the targetted schema   The incoming message fields which matches with the targetted schema will be kept  any additional fields that does not match will be remove and in case of field are missing  the field will bet set to null value  The result will then be store in the targetted table   The original message  with the mismatch schema  will be store in the DLT table       Getting Started      Requirements     gcloud sdk  https   cloud google com sdk docs install sdk    Java 11   Maven 3      Building the Project  Build the entire project using the maven compile command     sh mvn clean compile          Setting GCP Resources    bash   Set the pipeline vars PROJECT ID  gcp project id  BQ DATASET  bigquery dataset name  BUCKET  gcs bucket  PIPELINE FOLDER gs     BUCKET  dataflow pipelines streaming benchmark PUBSUB TOPIC  pubsub topic  PUBSUB SUBS ID  subscriptions id           Creating an example dataset    sh bq   location US mk  d   BQ DATASET            Create an example table    sh bq mk       table       PROJECT ID    BQ DATASET  tutorial table       src resources person json            Create an example PubSub topic and subscription     sh gcloud pubsub topics create   PUBSUB TOPIC  gcloud pubsub subscriptions create   PUBSUB SUBS ID          topic   PUBSUB TOPIC            Executing the Pipeline The below instruction will deploy dataflow into the GCP project     bash   Set the runner RUNNER DataflowRunner    Compute engine zone REGION us central1    Build the template mvn compile exec java    Dexec mainClass com google cloud pso pipeline PubsubToBigQueryJSON    Dexec cleanupDaemonThreads false    Dexec args           project   PROJECT ID          dataset   BQ DATASET          stagingLocation   PIPELINE FOLDER  staging         tempLocation   PIPELINE FOLDER  temp         region   REGION          maxNumWorkers 5         pubsubSubscription projects   PROJECT ID  subscriptions   PUBSUB SUBS ID          runner   RUNNER            Sending sample data to pubsub  Once the dataflow job are deploy and running we can try to send few messages to see the output being store in the GCP project  Below are the happy path message    sh gcloud pubsub topics publish   PUBSUB TOPIC        message     tutorial table      person      name     happyPath     age   22    date      2022 08 03     id      xpze           Below are the mismatch schema message  with  age  field being removed to the pubsub    sh gcloud pubsub topics publish   PUBSUB TOPIC        message     tutorial table      person      name     mismatchSchema      date      2022 08 03     id      avghy


GCP Everything in this repository has been developed as a parallel to CAI Config Validator The difference is that the Constraint Template schemas target instead of This ensures that the policies only target terraform resource changes instead of the entire CAI metadata library from a project folder or organization Use this when you intend to validate changes rather than declaratively manage a GCP cloud environment See for information on how to use this library Terraform Config Validator Policy Library This repo contains a library of constraint templates and sample constraints to be used for Terraform resource change requests If you re looking for the CAI variant please see User Guide

Answer:
# Terraform Config Validator Policy Library

This repo contains a library of constraint templates and sample constraints to be used for Terraform resource change requests. If you're looking for the CAI variant, please see [Config Validator](https://github.com/lykaasegura/w-secteam-repo).

Everything in this repository has been developed as a parallel to CAI Config Validator. The difference is that the Constraint/Template schemas target `validation.resourcechange.terraform.cloud.google.com` instead of `validation.gcp.forsetisecurity.org`. This ensures that the policies only target terraform resource changes, instead of the entire CAI metadata library from a project, folder, or organization. Use this when you intend to validate changes, rather than declaratively manage a GCP cloud environment.

## User Guide

See [docs/user_guide.md](docs/user_guide.md) for information on how to use this library.

See [docs/functional_principles.md](docs/functional_principles.md) for information on how to **develop** your own policies to use with `gcloud beta terraform vet`.

## Creating Policies in the Constraint Framework

This library is set up in the **Constraint Framework** style. This means that we utilize Gatekeeper Constraints and ConstraintTemplates to interpret and apply rego logic to incoming terraform change resources. This can be challenging to understand at first, so please refer to the [functional principles](docs/functional_principles.md) documentation found in the `docs` folder.

## General Differences

This library is intended to validate terraform plan resources. Therefore, as mentioned, the target has been swapped from `validation.gcp.forsetisecurity.org` to `validation.resourcechange.terraform.cloud.google.com`. This also means that the Constraint and ConstraintTemplate definitions have also had to be changed from Gatekeeper API version `v1alpha1` to `v1beta1`, as this functionality is currently under development. As a result, the Rego policy language has also had to change. If you user CAI Constraints and Templates (ie. v1apha1 Constraints/Templates), those inlined Rego policies **will not work.**

You can check out documentation on how to create terraform policies in the [`gcloud terraform beta vet` documentation](https://cloud.google.com/docs/terraform/policy-validation/create-terraform-constraints).

## Working with this policy library

The operation of this library is similar with the CAI library, as the development flow with Make and other tools has proven to be quite efficient and helpful. Therefore, you can check the [user guide](docs/user_guide.md) for relevant documentation required to get this library working for your needs.

### Initializing a policy library

You can easily set up a new (local) policy library by downloading a [bundle](./docs/index.md#policy-bundles) using [kpt](https://kpt.dev/).

Download the full policy library and install the [Forseti bundle](./docs/bundles/forseti-security.md):

```
export BUNDLE=forseti-security
kpt pkg get https://github.com/GoogleCloudPlatform/policy-library.git ./policy-library
kpt fn source policy-library/samples/ | \
  kpt fn eval - --image gcr.io/config-validator/get-policy-bundle:latest -- bundle=$BUNDLE | \
  kpt fn sink policy-library/policies/constraints/$BUNDLE
```

Once you have initialized a library, you might want to save it to [git](./docs/user_guide.md#https://github.com/GoogleCloudPlatform/policy-library/blob/master/docs/user_guide.md#get-started-with-the-policy-library-repository).

### Developing a Constraint

If this library doesn't contain a constraint that matches your use case, you can develop a new one
using the [Constraint Template Authoring Guide](docs/functional_principles.md).

#### Available Commands

```
make audit                          Run audit against real CAI dump data
make build                          Format and build
make build_templates                Inline Rego rules into constraint templates
make format                         Format Rego rules
make help                           Prints help for targets with comments
make test                           Test constraint templates via OPA
```

#### Inlining

You can run `make build` to automatically inline Rego rules into your constraint templates.

This is done by finding a `INLINE("filename")` and `#ENDINLINE` statements in your yaml,
and replacing everything in between with the contents of the file.

For example, running `make build` would replace the raw content with the replaced content below

Raw:

```
#INLINE("my_rule.rego")
# This text will be replaced
#ENDINLINE
```

Replaced:

```
#INLINE("my_rule.rego")
#contents of my_rule.rego
#ENDINLINE
```

#### Linting Policies

Config Validator provides a policy linter.  You can invoke it as:

```
go get github.com/GoogleCloudPlatform/config-validator/cmd/policy-tool
policy-tool --policies ./policies --policies ./samples --libs ./lib
```

#### Local CI

You can run the cloudbuild CI locally as follows:

```
gcloud components install cloud-build-local
cloud-build-local --config ./cloudbuild.yaml --dryrun=false .
```

#### Updating CI Images

You can update the CI images to add new versions of rego/opa as they are released.

```
# Rebuild all images.
make -j ci-images

# Rebuild a single image
make ci-image-v1.16.0
```

Terraform Config Validator Policy Library  This repo contains a library of constraint templates and sample constraints to be used for Terraform resource change requests  If you re looking for the CAI variant  please see  Config Validator  https   github com lykaasegura w secteam repo    Everything in this repository has been developed as a parallel to CAI Config Validator  The difference is that the Constraint Template schemas target  validation resourcechange terraform cloud google com  instead of  validation gcp forsetisecurity org   This ensures that the policies only target terraform resource changes  instead of the entire CAI metadata library from a project  folder  or organization  Use this when you intend to validate changes  rather than declaratively manage a GCP cloud environment      User Guide  See  docs user guide md  docs user guide md  for information on how to use this library   See  docs functional principles md  docs functional principles md  for information on how to   develop   your own policies to use with  gcloud beta terraform vet       Creating Policies in the Constraint Framework  This library is set up in the   Constraint Framework   style  This means that we utilize Gatekeeper Constraints and ConstraintTemplates to interpret and apply rego logic to incoming terraform change resources  This can be challenging to understand at first  so please refer to the  functional principles  docs functional principles md  documentation found in the  docs  folder      General Differences  This library is intended to validate terraform plan resources  Therefore  as mentioned  the target has been swapped from  validation gcp forsetisecurity org  to  validation resourcechange terraform cloud google com   This also means that the Constraint and ConstraintTemplate definitions have also had to be changed from Gatekeeper API version  v1alpha1  to  v1beta1   as this functionality is currently under development  As a result  the Rego policy language has also had to change  If you user CAI Constraints and Templates  ie  v1apha1 Constraints Templates   those inlined Rego policies   will not work     You can check out documentation on how to create terraform policies in the   gcloud terraform beta vet  documentation  https   cloud google com docs terraform policy validation create terraform constraints       Working with this policy library  The operation of this library is similar with the CAI library  as the development flow with Make and other tools has proven to be quite efficient and helpful  Therefore  you can check the  user guide  docs user guide md  for relevant documentation required to get this library working for your needs       Initializing a policy library  You can easily set up a new  local  policy library by downloading a  bundle    docs index md policy bundles  using  kpt  https   kpt dev     Download the full policy library and install the  Forseti bundle    docs bundles forseti security md        export BUNDLE forseti security kpt pkg get https   github com GoogleCloudPlatform policy library git   policy library kpt fn source policy library samples        kpt fn eval     image gcr io config validator get policy bundle latest    bundle  BUNDLE       kpt fn sink policy library policies constraints  BUNDLE      Once you have initialized a library  you might want to save it to  git    docs user guide md https   github com GoogleCloudPlatform policy library blob master docs user guide md get started with the policy library repository        Developing a Constraint  If this library doesn t contain a constraint that matches your use case  you can develop a new one using the  Constraint Template Authoring Guide  docs functional principles md         Available Commands      make audit                          Run audit against real CAI dump data make build                          Format and build make build templates                Inline Rego rules into constraint templates make format                         Format Rego rules make help                           Prints help for targets with comments make test                           Test constraint templates via OPA           Inlining  You can run  make build  to automatically inline Rego rules into your constraint templates   This is done by finding a  INLINE  filename    and   ENDINLINE  statements in your yaml  and replacing everything in between with the contents of the file   For example  running  make build  would replace the raw content with the replaced content below  Raw        INLINE  my rule rego     This text will be replaced  ENDINLINE      Replaced        INLINE  my rule rego    contents of my rule rego  ENDINLINE           Linting Policies  Config Validator provides a policy linter   You can invoke it as       go get github com GoogleCloudPlatform config validator cmd policy tool policy tool   policies   policies   policies   samples   libs   lib           Local CI  You can run the cloudbuild CI locally as follows       gcloud components install cloud build local cloud build local   config   cloudbuild yaml   dryrun false             Updating CI Images  You can update the CI images to add new versions of rego opa as they are released         Rebuild all images  make  j ci images    Rebuild a single image make ci image v1 16 0