GCP This example covers how distributed data preprocessing training and serving A simple machine learning system capable of recommending songs given a user as a query using collaborative filtering and TensorFlow user and item features to be included during training can be done on GCP CloudML Deep Collaborative Filtering Unlike classic matrix factorization approaches using a neural network allows

Answer:
# CloudML Deep Collaborative Filtering
A simple machine learning system capable of recommending songs given a user as a query
using collaborative filtering and TensorFlow.

Unlike classic matrix factorization approaches, using a neural network allows
user and item features to be included during training.

This example covers how distributed data preprocessing, training, and serving
can be done on [Google Cloud Platform](https://cloud.google.com/)(GCP).

Further reading:
  - [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031): A paper
    on using neural networks instead of matrix factorization to perform
    collaborative filtering.
  - [Deep Neural Networks for YouTube Recommendations](https://ai.google/research/pubs/pub45530):
    Youtube's approach to recommending millions of videos at low latencies by
    first generating candidates from multiple models and ranking the candidate
    pool.

For a fully managed service, check out [Recommendations
AI](https://cloud.google.com/recommendations/).

## Setup
Create a new project on GCP and set up GCP credentials:
```shell
gcloud auth login
gcloud auth application-default login
```

Enable the following APIS:
- [Dataflow](http://console.cloud.google.com/apis/api/dataflow.googleapis.com)
- [AI Platform](http://console.cloud.google.com/apis/api/ml.googleapis.com)

Using the `preprocessing/config.example.ini` template, create
`preprocessing/config.ini` with the GCP project id fields filled in.
Additionally, you will need to create a GCS bucket. This code assumes a bucket
exists by the name of `[project-id]-bucket`.

Set up your python environment:
```shell
python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
```

## Preprocessing
The data preprocessing pipeline uses the
[ListenBrainz](https://console.cloud.google.com/marketplace/details/metabrainz/listenbrainz)
dataset hosted on [Cloud
Marketplace](https://console.cloud.google.com/marketplace). Data is processed
and written to [Google Cloud Storage](https://cloud.google.com/storage/)(GCS) as
[TFRecords](https://www.tensorflow.org/tutorials/load_data/tf_records).

These files are read using [Cloud DataFlow](https://cloud.google.com/dataflow/).
The steps involved are as follows:
1. Read the data in using the
   [BigQuery](https://cloud.google.com/bigquery/) query found
   [here](trainer/query.py).
   This query cleans the features and creates a label for each unique user-item
   pair that exists. This label is 1 if a user has listened to a song more than
   twice and 0 otherwise. Samples are also given weights based on how many
   interactions there were between the user and item.
2. Using [TensorFlow
   Transform](https://www.tensorflow.org/tfx/transform/get_started), map each
   username and product id to an integer value and write the vocabularies to
   text files. Leave users and items under a set frequency threshold out of the
   vocabularies.
3. Filter away user-item pairs where either element is outside of its
   corresponding vocabulary.
4. Split the data into train, validation, and test sets.
5. Write each dataset as TFRecords to GCS.

### Execution
| Command | Description |
|---------|-------------|
| `bin/run.preprocess.local.sh` | Process a sample of the data locally and write outputs to a local directory. |
| `bin/run.preprocess.cloud.sh` | Process the data on GCP using DataFlow and write outputs to a GCS bucket. |
| `bin/run.test.sh`             | Run unit tests for the preprocessing pipeline. |

## Training
A [Custom Estimator](https://www.tensorflow.org/guide/custom_estimators) is
trained using TensorFlow and [Cloud AI Platform](https://cloud.google.com/ai-platform/)(CAIP).
The training steps are as follows:
1. Read TFRecords from GCS and create a `tf.data.Dataset` for each of them that
   yields data in batches.
2. Use the TensorFlow Transform output from preprocessing to transform usernames
   and product ids into int ids.
3. Use `user_id`s and `item_id`s to train embeddings.
4. Add item features and create two input layers: one with user embedding
   vectors, and another with the concatenation of item embedding vectors and
   item features.
5. Create a user neural net and item neural net from the input layers, ensuring
   that the final layers are the same size.
6. Compute the cosine similarity between the final layers of the user and item
   nets. Take the absolute value to get a value between 0 and 1.
7. Calculate error using log loss and train the model.
8. Evaluate the model performance by sampling 1000 random items and calculating
   the average recall@k when each positive sample's item is ranked against
   these random items for the sample's user.
9. Export a `SavedModel` for use in serving.

### Execution
Training job scripts expect the following argument:
- `MODEL_INPUTS_DIR`: The directory containing the TFRecords from preprocessing.

| Command | Description |
|---------|-------------|
| `bin/run.train.local.sh` | Train the model locally and save model checkpoints to a local model dir. |
| `bin/run.train.cloud.sh` | Train the model on CAIP and save model checkpoints to a GCS bucket. |
| `bin/run.train.tune.sh` | Train the model on CAIP as above, but using hyperparameter tuning. |

Note: `SCALE_TIER` is set to `STANDARD_1` to demonstrate distributed training.
However, it can be set to `BASIC` to reduce costs. See [scale
tiers](https://cloud.google.com/ml-engine/docs/tensorflow/machine-types).

### Tensorboard
Model training can be monitored on Tensorboard using the following command:
```shell
tensorboard --logdir <path to model dir>/<trial number>
```
Tensorboard's projector, in particular, is very useful for debugging
or analyzing embeddings. In the projector tab in Tensorboard, try setting the
label to `name`.

## Serving
Models can be hosted on CAIP, which can be used to make online and batch predictions via JSON requests.
1. Upload the `SavedModel` from training to CAIP.
2. Using a file containing a list of usernames, create inputs to pass to the
   model hosted on CAIP for predictions.
3. Make the predictions.

### Execution
The cloud serving job and prediction job scripts expect the same argument:
- `MODEL_OUTPUTS_DIR`: The model directory containing each model trial.
- `TRIAL` (optional): The trial number to use.
The local serving job expects no arguments, and the local prediction job expects
the model version number.

| Command | Description |
|---------|-------------|
| `bin/run.serve.local.sh` | Upload a new version of the recommender model to CAIP using a locally trained model. |
| `bin/run.serve.cloud.sh` | Upload a new version of the recommender model to CAIP using a model trained on CAIP. |
| `bin/run.predict.local.sh` | Using `serving/test.json`, create a prediction job on CAIP after using the local serving script. |
| `bin/run.predict.cloud.sh` | Using `serving/test.json`, create a prediction job on CAIP after using the cloud sering script. |

CloudML Deep Collaborative Filtering A simple machine learning system capable of recommending songs given a user as a query using collaborative filtering and TensorFlow   Unlike classic matrix factorization approaches  using a neural network allows user and item features to be included during training   This example covers how distributed data preprocessing  training  and serving can be done on  Google Cloud Platform  https   cloud google com   GCP    Further reading       Neural Collaborative Filtering  https   arxiv org abs 1708 05031   A paper     on using neural networks instead of matrix factorization to perform     collaborative filtering       Deep Neural Networks for YouTube Recommendations  https   ai google research pubs pub45530       Youtube s approach to recommending millions of videos at low latencies by     first generating candidates from multiple models and ranking the candidate     pool   For a fully managed service  check out  Recommendations AI  https   cloud google com recommendations        Setup Create a new project on GCP and set up GCP credentials     shell gcloud auth login gcloud auth application default login      Enable the following APIS     Dataflow  http   console cloud google com apis api dataflow googleapis com     AI Platform  http   console cloud google com apis api ml googleapis com   Using the  preprocessing config example ini  template  create  preprocessing config ini  with the GCP project id fields filled in  Additionally  you will need to create a GCS bucket  This code assumes a bucket exists by the name of   project id  bucket    Set up your python environment     shell python3  m venv venv source   venv bin activate pip install  r requirements txt         Preprocessing The data preprocessing pipeline uses the  ListenBrainz  https   console cloud google com marketplace details metabrainz listenbrainz  dataset hosted on  Cloud Marketplace  https   console cloud google com marketplace   Data is processed and written to  Google Cloud Storage  https   cloud google com storage   GCS  as  TFRecords  https   www tensorflow org tutorials load data tf records    These files are read using  Cloud DataFlow  https   cloud google com dataflow    The steps involved are as follows  1  Read the data in using the     BigQuery  https   cloud google com bigquery   query found     here  trainer query py      This query cleans the features and creates a label for each unique user item    pair that exists  This label is 1 if a user has listened to a song more than    twice and 0 otherwise  Samples are also given weights based on how many    interactions there were between the user and item  2  Using  TensorFlow    Transform  https   www tensorflow org tfx transform get started   map each    username and product id to an integer value and write the vocabularies to    text files  Leave users and items under a set frequency threshold out of the    vocabularies  3  Filter away user item pairs where either element is outside of its    corresponding vocabulary  4  Split the data into train  validation  and test sets  5  Write each dataset as TFRecords to GCS       Execution   Command   Description                                bin run preprocess local sh    Process a sample of the data locally and write outputs to a local directory       bin run preprocess cloud sh    Process the data on GCP using DataFlow and write outputs to a GCS bucket       bin run test sh                Run unit tests for the preprocessing pipeline         Training A  Custom Estimator  https   www tensorflow org guide custom estimators  is trained using TensorFlow and  Cloud AI Platform  https   cloud google com ai platform   CAIP   The training steps are as follows  1  Read TFRecords from GCS and create a  tf data Dataset  for each of them that    yields data in batches  2  Use the TensorFlow Transform output from preprocessing to transform usernames    and product ids into int ids  3  Use  user id s and  item id s to train embeddings  4  Add item features and create two input layers  one with user embedding    vectors  and another with the concatenation of item embedding vectors and    item features  5  Create a user neural net and item neural net from the input layers  ensuring    that the final layers are the same size  6  Compute the cosine similarity between the final layers of the user and item    nets  Take the absolute value to get a value between 0 and 1  7  Calculate error using log loss and train the model  8  Evaluate the model performance by sampling 1000 random items and calculating    the average recall k when each positive sample s item is ranked against    these random items for the sample s user  9  Export a  SavedModel  for use in serving       Execution Training job scripts expect the following argument     MODEL INPUTS DIR   The directory containing the TFRecords from preprocessing     Command   Description                                bin run train local sh    Train the model locally and save model checkpoints to a local model dir       bin run train cloud sh    Train the model on CAIP and save model checkpoints to a GCS bucket       bin run train tune sh    Train the model on CAIP as above  but using hyperparameter tuning     Note   SCALE TIER  is set to  STANDARD 1  to demonstrate distributed training  However  it can be set to  BASIC  to reduce costs  See  scale tiers  https   cloud google com ml engine docs tensorflow machine types        Tensorboard Model training can be monitored on Tensorboard using the following command     shell tensorboard   logdir  path to model dir   trial number      Tensorboard s projector  in particular  is very useful for debugging or analyzing embeddings  In the projector tab in Tensorboard  try setting the label to  name       Serving Models can be hosted on CAIP  which can be used to make online and batch predictions via JSON requests  1  Upload the  SavedModel  from training to CAIP  2  Using a file containing a list of usernames  create inputs to pass to the    model hosted on CAIP for predictions  3  Make the predictions       Execution The cloud serving job and prediction job scripts expect the same argument     MODEL OUTPUTS DIR   The model directory containing each model trial     TRIAL   optional   The trial number to use  The local serving job expects no arguments  and the local prediction job expects the model version number     Command   Description                                bin run serve local sh    Upload a new version of the recommender model to CAIP using a locally trained model       bin run serve cloud sh    Upload a new version of the recommender model to CAIP using a model trained on CAIP       bin run predict local sh    Using  serving test json   create a prediction job on CAIP after using the local serving script       bin run predict cloud sh    Using  serving test json   create a prediction job on CAIP after using the cloud sering script


GCP The processor service subscribes to the topic processes every message The DFDL definitions are stored in a Bigtable database The application send a request with the binary to process to a pubsub topic Project Structure Data Format Description Language Processor Example This module is a example how to process a binary using a DFDL definition applies the definition and publishes the json result to a topic in pubsub

Answer:
# Data Format Description Language ([DFDL](https://en.wikipedia.org/wiki/Data_Format_Description_Language)) Processor Example
This module is a example how to process a binary using a DFDL definition.
The DFDL definitions are stored in a Bigtable database.

The application send a request with the binary to process to a pubsub topic.

The processor service subscribes to the topic, processes every message,
applies the definition and publishes the json result to a topic in pubsub.

## Project Structure

```
.
└── dfdl_example
 ├── examples # Contain a binary and dfdl definition to be used to run this example
 └── src
       └── main
           └── java
               └── com.example.dfdl
                   ├── BigtableServer # Configures bigtable database
                   ├── BigtableService # Reads dfdl definitons from a bigtable database
                   ├── DfdlDef # Embedded entities
                   ├── DfdlService # Processes the binary using a dfdl definition and output a json
                   ├── MessageController # Publishes message to a topic with a binary to be processed.
                   ├── ProcessorService # Initializes components, configurations and services.
                   ├── PubSubServer # Publishes and subscribes to topics using channels adapters.
 └── resources
      └── application.properties
 └── pom.xml
 └── README.md
```

### Tools

Before you start is recommended that you install the following tools:

1. [Google Cloud SDK](https://cloud.google.com/sdk/docs/install)
2. [Cloud Bigtable Tool](https://cloud.google.com/bigtable/docs/cbt-overview)

## Technology Stack
1. Cloud Bigtable
2. Cloud Pubsub

## Frameworks
1. Spring Boot

## Libraries
1. [Apache Daffodil](https://daffodil.apache.org/)

## Setup Instructions
### Project Setup
#### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
1. In the drop-down menu at the top, select **Create a project**.
1. Give your project a name = my-dfdl-project
1. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

#### Enabling billing for your project.

If you haven't already enabled billing for your project, [enable
billing][enable-billing] now.  Enabling billing allows is required to use Cloud Bigtable
and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

#### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud SDK, [install the Google
Cloud SDK][cloud-sdk] now. The SDK contains tools and libraries that enable you
to create and manage resources on Google Cloud Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

#### Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
gcloud init
```
Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command:

```
    gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing
[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

### Bigtable Setup
How to create a Bigtable database instance can be found [here](https://cloud.google.com/bigtable/docs/creating-instance)

#### How to add data to bigtable
The following doc, [Writing to Bigtable](https://cloud.google.com/bigtable/docs/writing-data),
can be used to add data to bigtable to run the example.

This example connects to a Cloud Bigtable with a collection with the
following specification.
The configuration can be changed by changing the application.properties file.
```
    Table
     dfdl-schemas =>
         Column Family
            dfdl =>
               Column Family Qualifier =>
                   binary_example => {
                       'name': "dfdl-name"
                       'definiton':
                        "<?xml version"1.0" encoding="UTF-8"?>
                           <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
                            xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
                            targetNamespace="http://example.com/dfdl/helloworld/">
                              <xs:include
                            schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
                              <xs:annotation>
                                 <xs:appinfo source="http://www.ogf.org/dfdl/">
                                    <dfdl:format ref="GeneralFormat" />
                                 </xs:appinfo>
                              </xs:annotation>
                              <xs:element name="binary_example">
                                 <xs:complexType>
                                    <xs:sequence>
                                       <xs:element name="w" type="xs:int">
                                          <xs:annotation>
                                             <xs:appinfo source="http://www.ogf.org/dfdl/">
                                                <dfdl:element representation="binary"
                                                    binaryNumberRep="binary" byteOrder="bigEndian" lengthKind="implicit" />
                                             </xs:appinfo>
                                          </xs:annotation>
                                       </xs:element>
                                       <xs:element name="x" type="xs:int">
                                          <xs:annotation>
                                             <xs:appinfo source="http://www.ogf.org/dfdl/">
                                                <dfdl:element representation="binary"
                                                      binaryNumberRep="binary" byteOrder="bigEndian" lengthKind="implicit" />
                                             </xs:appinfo>
                                          </xs:annotation>
                                       </xs:element>
                                       <xs:element name="y" type="xs:double">
                                          <xs:annotation>
                                             <xs:appinfo source="http://www.ogf.org/dfdl/">
                                                <dfdl:element representation="binary"
                                                       binaryFloatRep="ieee" byteOrder="bigEndian" lengthKind="implicit" />
                                             </xs:appinfo>
                                          </xs:annotation>
                                       </xs:element>
                                       <xs:element name="z" type="xs:float">
                                          <xs:annotation>
                                             <xs:appinfo source="http://www.ogf.org/dfdl/">
                                                <dfdl:element representation="binary"
                                                      byteOrder="bigEndian" lengthKind="implicit" binaryFloatRep="ieee" />
                                             </xs:appinfo>
                                          </xs:annotation>
                                       </xs:element>
                                    </xs:sequence>
                                 </xs:complexType>
                              </xs:element>
                           </xs:schema>";
                        }
```
This dfdl definition example can be found in the binary_example.dfdl.xsd file.

### Pubsub Setup
The following [doc](https://cloud.google.com/pubsub/docs/quickstart-console)
can be used to set up the topics and subscriptions needed to run this example.

#### Topics
To run this example two topics need to be created:
1. A topic to publish the final json output: "data-output-json-topic"
2. A topic to publish the binary to be processed: "data-input-binary-topic"

#### Subscription
The following subscriptions need to be created:
1. A subscription to pull the binary data: data-input-binary-sub

## Usage
### Initialize the application
Reference: [Building an Application with Spring Boot](https://spring.io/guides/gs/spring-boot/)
```
      ./mvnw spring-boot:run
```
### Send a request
```
    curl --data "message=0000000500779e8c169a54dd0a1b4a3fce2946f6" localhost:8081/publish
``

Data Format Description Language   DFDL  https   en wikipedia org wiki Data Format Description Language   Processor Example This module is a example how to process a binary using a DFDL definition  The DFDL definitions are stored in a Bigtable database   The application send a request with the binary to process to a pubsub topic   The processor service subscribes to the topic  processes every message  applies the definition and publishes the json result to a topic in pubsub      Project Structure            dfdl example      examples   Contain a binary and dfdl definition to be used to run this example      src            main                java                    com example dfdl                        BigtableServer   Configures bigtable database                        BigtableService   Reads dfdl definitons from a bigtable database                        DfdlDef   Embedded entities                        DfdlService   Processes the binary using a dfdl definition and output a json                        MessageController   Publishes message to a topic with a binary to be processed                         ProcessorService   Initializes components  configurations and services                         PubSubServer   Publishes and subscribes to topics using channels adapters       resources           application properties      pom xml      README md          Tools  Before you start is recommended that you install the following tools   1   Google Cloud SDK  https   cloud google com sdk docs install  2   Cloud Bigtable Tool  https   cloud google com bigtable docs cbt overview      Technology Stack 1  Cloud Bigtable 2  Cloud Pubsub     Frameworks 1  Spring Boot     Libraries 1   Apache Daffodil  https   daffodil apache org       Setup Instructions     Project Setup      Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   1  In the drop down menu at the top  select   Create a project    1  Give your project a name   my dfdl project 1  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com        Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now   Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings       Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk        Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command       gcloud init     Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command           gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing  application default credentials   https   developers google com identity protocols application default credentials      Bigtable Setup How to create a Bigtable database instance can be found  here  https   cloud google com bigtable docs creating instance        How to add data to bigtable The following doc   Writing to Bigtable  https   cloud google com bigtable docs writing data   can be used to add data to bigtable to run the example   This example connects to a Cloud Bigtable with a collection with the following specification  The configuration can be changed by changing the application properties file          Table      dfdl schemas             Column Family             dfdl                    Column Family Qualifier                        binary example                              name    dfdl name                          definiton                              xml version 1 0  encoding  UTF 8                                xs schema xmlns xs  http   www w3 org 2001 XMLSchema                              xmlns dfdl  http   www ogf org dfdl dfdl 1 0                               targetNamespace  http   example com dfdl helloworld                                   xs include                             schemaLocation  org apache daffodil xsd DFDLGeneralFormat dfdl xsd                                    xs annotation                                    xs appinfo source  http   www ogf org dfdl                                         dfdl format ref  GeneralFormat                                        xs appinfo                                  xs annotation                                 xs element name  binary example                                     xs complexType                                       xs sequence                                          xs element name  w  type  xs int                                              xs annotation                                                xs appinfo source  http   www ogf org dfdl                                                     dfdl element representation  binary                                                      binaryNumberRep  binary  byteOrder  bigEndian  lengthKind  implicit                                                    xs appinfo                                              xs annotation                                           xs element                                          xs element name  x  type  xs int                                              xs annotation                                                xs appinfo source  http   www ogf org dfdl                                                     dfdl element representation  binary                                                        binaryNumberRep  binary  byteOrder  bigEndian  lengthKind  implicit                                                    xs appinfo                                              xs annotation                                           xs element                                          xs element name  y  type  xs double                                              xs annotation                                                xs appinfo source  http   www ogf org dfdl                                                     dfdl element representation  binary                                                         binaryFloatRep  ieee  byteOrder  bigEndian  lengthKind  implicit                                                    xs appinfo                                              xs annotation                                           xs element                                          xs element name  z  type  xs float                                              xs annotation                                                xs appinfo source  http   www ogf org dfdl                                                     dfdl element representation  binary                                                        byteOrder  bigEndian  lengthKind  implicit  binaryFloatRep  ieee                                                    xs appinfo                                              xs annotation                                           xs element                                        xs sequence                                     xs complexType                                  xs element                               xs schema                                  This dfdl definition example can be found in the binary example dfdl xsd file       Pubsub Setup The following  doc  https   cloud google com pubsub docs quickstart console  can be used to set up the topics and subscriptions needed to run this example        Topics To run this example two topics need to be created  1  A topic to publish the final json output   data output json topic  2  A topic to publish the binary to be processed   data input binary topic        Subscription The following subscriptions need to be created  1  A subscription to pull the binary data  data input binary sub     Usage     Initialize the application Reference   Building an Application with Spring Boot  https   spring io guides gs spring boot               mvnw spring boot run         Send a request         curl   data  message 0000000500779e8c169a54dd0a1b4a3fce2946f6  localhost 8081 publish


GCP 1 Drastically reduce time to perform or by supporting execution of resource plans A state scalable project factory pattern with Terragrunt 1 Providing a dynamic way to configure for categories of resources in directories Resolves the problem of state volume explotion with project factory Terragrunt helps with that by Overview 1 Providing configuration of source code by generating code in target directories using dynamic definitions

Answer:
# A 'state-scalable' project factory pattern with Terragrunt

## Overview

Resolves the problem of state volume explotion with project factory. Terragrunt helps with that by:

1. Providing a dynamic way to configure [remote_state](https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/#keep-your-remote-state-configuration-dry) for categories of resources in directories.
1. Providing [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) configuration of source code by generating code in target directories using dynamic [source](https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/#motivation) definitions.
1. Drastically reduce time to perform `terraform plan` or `terraform apply` by supporting [parallel](https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/) execution of resource plans.

This pattern scales the 'factory' oriented approach of IaC implementation, facilitating both scalability of the Terraform state file size and also developer productivity by minimizing time to run *plans*. By providing mechanisms to create resource group definitions using both local and common data configurations through `defaults`, and implementing `DRY` code in a central `source`, it encourages a mature `Infrastructure as Data` implementation practice.

## Expalanation

![Diagram](docs/images/image2.png)

Implementing a factory oriented pattern for deploying resource groups is a common practice in IaC (Infrastructure as Code). This is typically done by having a configurable blueprint of data to describe the infrastructure to avoid repetition of code. [Project-factory](https://registry.terraform.io/modules/terraform-google-modules/project-factory/google/latest) is a common manifestation of this requirement since in GCP, projects need to be created ubiquitously.

 This pattern can however result in intractable state file sizes resulting in slow pipeline steps everytime `terraform plan` or `apply` needs to run, because it reads all resources from the Cloud environment. The general guideline from [Google Cloud's documentation](https://cloud.google.com/docs/terraform/best-practices-for-terraform#minimize-resources) is that each state file should not have more than 100 resources -- which itself can be obfuscated by the use of modules. This example provides a solution to *state explosion* using Terragrunt.

In addition, it describes a pattern such that the *Terraform source code* for implementing resources can be defined in the sub-directory as a data configuraiton, instead of repeating the code.

Generally, this pattern splits IaC into `data` and `src` directories at the top level with their configuration connected by `terragrunt.hcl` files at different levels of the file hierarchy. In this example, the public `project-factory` module is used to create projects for `team1` and `team2` categories, while maintaining separate state files using the Terragrunt's configuration. As described in the diagram below, the state files for each category would be stored in the following GCS bucket URL paths:

```
Team1 -  gcs://<bucket>/data/team1/default.state
Team2 -  gcs://<bucket>/data/team2/default.state
```

This is enabled by the root `terragrunt.hcl` file located under the repository root, defining a dynamic `remote-back-end` configuration that is set at the subdirectory level.

```terraform
# Root -> terragrunt.hcl

remote_state {
  backend = "gcs"
  config = {
    bucket   = local.vars.bucket_prefix
    prefix   = path_relative_to_include()
    project  = local.vars.root_project
    location = local.vars.region
  }
}
```

Under the subdirectory for team1, the `include` block is defined and the `path` variable is set relative to the root configuration.

```
# Root -> data -> team1 -> terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}
```

### Dynamic source configuration

An additional configuration defined in the `terragrunt.hcl` file for team1 is the `terraform` block. This specifies that the source code for the data configuration describing the resources for this directory will be implemented by following `terraform` -> `source` code. Terragrunt manages a temporary instance of the source code inside a directory `.terragrunt-cache`, absolving the developer from maintaining several instances of the code base in different data subdirectories.

```terraform
# Root -> data -> team1 -> terragrunt.hcl

terraform {
  # Pull the terraform configuration from the local file system. Terragrunt will make a copy of the source folder in the
  # Terragrunt working directory (typically `.terragrunt-cache`).
  source = "../..//src"

  # Files to include from the Terraform src directory
  include_in_copy = [
    "main.tf",
    "variables.tf",
    "outputs.tf",
    "provider.tf"
  ]
}
```

In this example, the `terraform` configuration of a local source code module in `src/` is provided, which simply invokes the public `project-factory` module by flattening the specifications provided in `default` and `data` directories. In practice, instead of local ones this can be modules hosted in private or public registries implementing IaC blueprints for common use-cases -- eg. Projects and supporting resoources for Data Platform Teams and Application Teams.

## Requirements

A few resoures need to be created before running terragrunt. Either use the terraform scripts under setup folder or follow manual steps given below.
In either case make sure individual running the steps have folder creator, project creator and storage admin roles.

### Setup by terraform scripts

1. cd setup
2. Create a terraform.tfvars from the sample with the correct org_id, billing_account, default region and bucket name where state will be stored.
3. This creates resources that are needed to run the sample terragrunt project factory.

- "terragrunt_test" folder under org
- "terragrunt-seedproject" project under "terragrunt_test" folder
- "terragrunt-iac-core-bkt" GCS Bucket for storing state
- "Team1" and "Team2" folders
- Generate root.yaml and defaults.yaml files inside the teams' directoriees from template files.

```
terraform init
terraform plan
terraform apply
```

### Manual setup

- Create two Folders where Terragrunt will create projects. Add corresponding folders id's in data/team/defaults.yaml and data/team2/defaults.yaml
- Create a Project to store terraform state and a gcs bucket in that project.
- Add project_id and gcs_bucket name in root.yaml.

## How to run
*Steps 2 to 5 can be skipped if you ran the setup scripts*

1. [Install Terragrunt](https://terragrunt.gruntwork.io/docs/getting-started/install/)
1. Create [folders](https://cloud.google.com/resource-manager/docs/creating-managing-folders#creating-folders) in your organization; similar to `team1`, `team2` as shown in sample.
1. Create project files in data \<category\>  projects similar to `*.yaml.sample` files provided to project specific configurations.
1. Create defaults.yaml file for each category similar to `defaults.yaml.sample` file provided for common configurations.
1. Create root.yaml file similar to `root.yaml.sample` for remote backend configurations.

```
terragrunt run-all init
terragrunt run-all plan
terragrunt run-all apply
```

**Note: `terragrunt plan` or `apply` can be run directly in subdirectories (ie. data/team1 etc) with a `terragrunt.hcl` file, to create resources for each team. This is useful for separating pipelines. `terragrunt run-all` is useful for runnning all deployments at once and in parallel.**

## Variations

- A version of this pattern that integrates easy with [Fabric FAST](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/blueprints/factories/project-factory) or [Cloud Foundatiaon Toolkit](https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit)

## Resources

- [Terragrunt](https://terragrunt.gruntwork.io/docs/getting-started)

- [*Infrastructure as Data*](https://medium.com/dzerolabs/shifting-from-infrastructure-as-code-to-infrastructure-as-data-bdb1ae1840e3) Medium link
- [Splitting a monolithic Terraform state using Terragrunt](https://medium.com/cts-technologies/murdering-monoliths-using-terragrunt-to-split-monolithic-terraform-state-up-into-multiple-stacks-17ead2d8e0e9)

## TODO

1. Update example with different service accounts for team directories
1. Create branches for variations. Variations can be like, integrating with FAST Fabric project factory pattern eg.

## Caveats

- Terragrunt has [restrictions](https://docs.gruntwork.io/guides/working-with-code/tfc-integration) when it comes to integrating with Hashicorp's Terraform Cloud or Terraform Cloud Enterprise platform. TL;DR: You can still use TCE/TC for storing states, monitoring and auditing but cannot use the UI for Terraform runs. Initiating runs using the CLI is still possible.

A  state scalable  project factory pattern with Terragrunt     Overview  Resolves the problem of state volume explotion with project factory  Terragrunt helps with that by   1  Providing a dynamic way to configure  remote state  https   terragrunt gruntwork io docs features keep your remote state configuration dry  keep your remote state configuration dry  for categories of resources in directories  1  Providing  DRY  https   en wikipedia org wiki Don 27t repeat yourself  configuration of source code by generating code in target directories using dynamic  source  https   terragrunt gruntwork io docs features keep your terraform code dry  motivation  definitions  1  Drastically reduce time to perform  terraform plan  or  terraform apply  by supporting  parallel  https   terragrunt gruntwork io docs features execute terraform commands on multiple modules at once   execution of resource plans   This pattern scales the  factory  oriented approach of IaC implementation  facilitating both scalability of the Terraform state file size and also developer productivity by minimizing time to run  plans   By providing mechanisms to create resource group definitions using both local and common data configurations through  defaults   and implementing  DRY  code in a central  source   it encourages a mature  Infrastructure as Data  implementation practice      Expalanation    Diagram  docs images image2 png   Implementing a factory oriented pattern for deploying resource groups is a common practice in IaC  Infrastructure as Code   This is typically done by having a configurable blueprint of data to describe the infrastructure to avoid repetition of code   Project factory  https   registry terraform io modules terraform google modules project factory google latest  is a common manifestation of this requirement since in GCP  projects need to be created ubiquitously    This pattern can however result in intractable state file sizes resulting in slow pipeline steps everytime  terraform plan  or  apply  needs to run  because it reads all resources from the Cloud environment  The general guideline from  Google Cloud s documentation  https   cloud google com docs terraform best practices for terraform minimize resources  is that each state file should not have more than 100 resources    which itself can be obfuscated by the use of modules  This example provides a solution to  state explosion  using Terragrunt   In addition  it describes a pattern such that the  Terraform source code  for implementing resources can be defined in the sub directory as a data configuraiton  instead of repeating the code   Generally  this pattern splits IaC into  data  and  src  directories at the top level with their configuration connected by  terragrunt hcl  files at different levels of the file hierarchy  In this example  the public  project factory  module is used to create projects for  team1  and  team2  categories  while maintaining separate state files using the Terragrunt s configuration  As described in the diagram below  the state files for each category would be stored in the following GCS bucket URL paths       Team1    gcs    bucket  data team1 default state Team2    gcs    bucket  data team2 default state      This is enabled by the root  terragrunt hcl  file located under the repository root  defining a dynamic  remote back end  configuration that is set at the subdirectory level      terraform   Root    terragrunt hcl  remote state     backend    gcs    config         bucket     local vars bucket prefix     prefix     path relative to include       project    local vars root project     location   local vars region            Under the subdirectory for team1  the  include  block is defined and the  path  variable is set relative to the root configuration         Root    data    team1    terragrunt hcl  include  root      path   find in parent folders              Dynamic source configuration  An additional configuration defined in the  terragrunt hcl  file for team1 is the  terraform  block  This specifies that the source code for the data configuration describing the resources for this directory will be implemented by following  terraform      source  code  Terragrunt manages a temporary instance of the source code inside a directory   terragrunt cache   absolving the developer from maintaining several instances of the code base in different data subdirectories      terraform   Root    data    team1    terragrunt hcl  terraform       Pull the terraform configuration from the local file system  Terragrunt will make a copy of the source folder in the     Terragrunt working directory  typically   terragrunt cache      source           src       Files to include from the Terraform src directory   include in copy          main tf        variables tf        outputs tf        provider tf             In this example  the  terraform  configuration of a local source code module in  src   is provided  which simply invokes the public  project factory  module by flattening the specifications provided in  default  and  data  directories  In practice  instead of local ones this can be modules hosted in private or public registries implementing IaC blueprints for common use cases    eg  Projects and supporting resoources for Data Platform Teams and Application Teams      Requirements  A few resoures need to be created before running terragrunt  Either use the terraform scripts under setup folder or follow manual steps given below  In either case make sure individual running the steps have folder creator  project creator and storage admin roles       Setup by terraform scripts  1  cd setup 2  Create a terraform tfvars from the sample with the correct org id  billing account  default region and bucket name where state will be stored  3  This creates resources that are needed to run the sample terragrunt project factory      terragrunt test  folder under org    terragrunt seedproject  project under  terragrunt test  folder    terragrunt iac core bkt  GCS Bucket for storing state    Team1  and  Team2  folders   Generate root yaml and defaults yaml files inside the teams  directoriees from template files       terraform init terraform plan terraform apply          Manual setup    Create two Folders where Terragrunt will create projects  Add corresponding folders id s in data team defaults yaml and data team2 defaults yaml   Create a Project to store terraform state and a gcs bucket in that project    Add project id and gcs bucket name in root yaml       How to run  Steps 2 to 5 can be skipped if you ran the setup scripts   1   Install Terragrunt  https   terragrunt gruntwork io docs getting started install   1  Create  folders  https   cloud google com resource manager docs creating managing folders creating folders  in your organization  similar to  team1    team2  as shown in sample   1  Create project files in data   category    projects similar to    yaml sample  files provided to project specific configurations  1  Create defaults yaml file for each category similar to  defaults yaml sample  file provided for common configurations  1  Create root yaml file similar to  root yaml sample  for remote backend configurations       terragrunt run all init terragrunt run all plan terragrunt run all apply        Note   terragrunt plan  or  apply  can be run directly in subdirectories  ie  data team1 etc  with a  terragrunt hcl  file  to create resources for each team  This is useful for separating pipelines   terragrunt run all  is useful for runnning all deployments at once and in parallel         Variations    A version of this pattern that integrates easy with  Fabric FAST  https   github com GoogleCloudPlatform cloud foundation fabric tree master blueprints factories project factory  or  Cloud Foundatiaon Toolkit  https   github com GoogleCloudPlatform cloud foundation toolkit      Resources     Terragrunt  https   terragrunt gruntwork io docs getting started       Infrastructure as Data   https   medium com dzerolabs shifting from infrastructure as code to infrastructure as data bdb1ae1840e3  Medium link    Splitting a monolithic Terraform state using Terragrunt  https   medium com cts technologies murdering monoliths using terragrunt to split monolithic terraform state up into multiple stacks 17ead2d8e0e9      TODO  1  Update example with different service accounts for team directories 1  Create branches for variations  Variations can be like  integrating with FAST Fabric project factory pattern eg      Caveats    Terragrunt has  restrictions  https   docs gruntwork io guides working with code tfc integration  when it comes to integrating with Hashicorp s Terraform Cloud or Terraform Cloud Enterprise platform  TL DR  You can still use TCE TC for storing states  monitoring and auditing but cannot use the UI for Terraform runs  Initiating runs using the CLI is still possible


GCP This repo houses the example code for a blog post on using a persistent history Dataproc Persistent History Server server to view job history about your Spark MapReduce jobs and aggregated YARN logs from short lived clusters on GCS Directory structure

Answer:
# Dataproc Persistent History Server
This repo houses the example code for a blog post on using a persistent history
server to view job history about your Spark / MapReduce jobs and
aggregated YARN logs from short-lived clusters on GCS.

![Architecture Diagram](img/persistent-history-arch.png)

## Directory structure
- `cluster_templates/`
  - `history_server.yaml`
  - `ephemeral_cluster.yaml`
- `init_actions`
  - `disable_history_servers.sh`
- `workflow_templates`
  - `spark_mr_workflow_template.yaml`
- `terraform`
  - `variables.tf`
  - `network.tf`
  - `history-server.tf`
  - `history-bucket.tf`
  - `long-running-cluster.tf`
  - `firewall.tf`
  - `service-account.tf`

## Usage
The recommended way to run this example is to use terraform as it creates a vpc network
to run the example with the appropriate firewall rules.

### Enabling services
```
gcloud services enable \
compute.googleapis.com \
dataproc.googleapis.com
```

### Pre-requisites
- [Install Google Cloud SDK](https://cloud.google.com/sdk/)
- Enable the following APIs if not already enabled.
  - `gcloud services enable compute.googleapis.com dataproc.googleapis.com`
- \[Optional\] [Install Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html)

### Disclaimer
This is for example purposes only. You should take a much closer look at the firewall
rules that make sense for your organization's security requirements.

### Should I run with Terraform or `gcloud` SDK ?
This repo provides artifacts to spin up the infrastructure for persisting
job history and yarn logs with terraform or with gcloud. The recommended
way to use this is to modify the Terraform code to fit your needs for
long running resources.

However, the cluster templates are included as an example of
standardizing cluster creation for ephemeral clusters.
You might ask, "Why is there a cluster template for the history server?".
The history server is simply a cleaner interface for reading your logs
from GCS. For Spark, it is stateless and you may wish to only spin up
 a history server when you'll actually be using it. For MapReduce,
the history server will only be aware of the files on GCS when it was
created and those files which it moves from intermediate done directory
to the done directory. For this reason, MapReduce workflows should
use a persistent history server.

### Managing Log Retention
Often times, it makes sense to leave the history
server running because several teams may use it and you could configure
it to manage clean up of your logs by setting the following additional
properties:
 - `yarn:yarn.log-aggregation.retain-seconds`
 - `spark:spark.history.fs.cleaner.enabled`
 - `spark:spark.history.fs.cleaner.maxAge`
 - `mapred:mapreduce.jobhistory.cleaner.enable`
 - `mapred:mapreduce.jobhistory.max-age-ms`

### Terraform
To spin up the whole example you could simply edit the
`terraform.tfvars` file to set the variables to the
desired values and run the following commands.

Note, this assumes that you have an existing project and
the sufficient permissions to spin up the resources for this
example.
```
cd terraform
terraform init
terraform apply
```
This will create:
1. VPC Network and subnetwork for your dataproc clusters.
1. Various firewall rules for this network.
1. A single node dataproc history-server cluster.
1. A long running dataproc cluster.
1. A GCS Bucket for YARN log aggregation, and Spark MapReduce Job History
as well as initialization actions for your clusters.

### Alternatively, with Google Cloud SDK
These instructions detail how to run this entire example with `gcloud`.
1.  Replace `PROJECT` with your GCP project id in each file.
1.  Replace `HISTORY_BUCKET` with your GCS bucket for logs in each file.
1.  Replace `HISTORY_SERVER` with your dataproc history server.
1.  Replace `REGION` with your desired GCP Compute region.
1.  Replace `ZONE` with your desired GCP Compute zone.

```
cd workflow_templates
sed -i 's/PROJECT/your-gcp-project-id/g' *
sed -i 's/HISTORY_BUCKET/your-history-bucket/g' *
sed -i 's/HISTORY_SERVER/your-history-server/g' *
sed -i 's/REGION/us-central1/g' *
sed -i 's/ZONE/us-central1-f/g' *
sed -i 's/SUBNET/your-subnet-id/g' *

cd cluster_templates
sed -i 's/PROJECT/your-gcp-project-id/g' *
sed -i 's/HISTORY_BUCKET/your-history-bucket/g' *
sed -i 's/HISTORY_SERVER/your-history-server/g' *
sed -i 's/REGION/us-central1/g' *
sed -i 's/ZONE/us-central1-f/g' *
sed -i 's/SUBNET/your-subnet-id/g' *
```

Stage an empty file to create the spark-events path on GCS.

```
touch .keep
gsutil cp .keep gs://your-history-bucket/spark-events/.keep
rm .keep
```

Stage our initialization action for disabling history servers
on your ephemeral clusters.
```
gsutil cp init_actions/disable_history_servers.sh
```

Create the history server.

```sh
gcloud beta dataproc clusters import \
  history-server \
  --source=cluster_templates/history-server.yaml \
  --region=us-central1
```

Create a cluster which you can manually submit jobs to and tear down.

```sh
gcloud beta dataproc clusters import \
ephemeral-cluster \
--source=cluster_templates/ephemeral-cluster.yaml \
--region=us-central1
```

### Running the Workflow Template
Import the workflow template to run an example spark and hadoop job
to verify your setup is working.

```sh
gcloud dataproc workflow-templates import spark-mr-example \
--source=workflow_templates/spark_mr_workflow_template.yaml
```

Trigger the workflow template to spin up a cluster,
run the example jobs and tear it down.

```sh
gcloud dataproc workflow-templates instantiate spark-mr-example
```

### Viewing the History UI
Follow [these instructions](https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces)
 to look at the UI by ssh tunneling to the history server.
Ports to visit:
 - MapReduce Job History: 19888
 - Spark Job History: 18080

### Closing Note
If you're adapting this example for your own use consider the following:
1. Setting an appropriate retention for your logs.
1. Setting more appropriate firewall rules for your security requirements.

Dataproc Persistent History Server This repo houses the example code for a blog post on using a persistent history server to view job history about your Spark   MapReduce jobs and aggregated YARN logs from short lived clusters on GCS     Architecture Diagram  img persistent history arch png      Directory structure    cluster templates        history server yaml       ephemeral cluster yaml     init actions       disable history servers sh     workflow templates       spark mr workflow template yaml     terraform       variables tf       network tf       history server tf       history bucket tf       long running cluster tf       firewall tf       service account tf      Usage The recommended way to run this example is to use terraform as it creates a vpc network to run the example with the appropriate firewall rules       Enabling services     gcloud services enable   compute googleapis com   dataproc googleapis com          Pre requisites    Install Google Cloud SDK  https   cloud google com sdk     Enable the following APIs if not already enabled       gcloud services enable compute googleapis com dataproc googleapis com      Optional    Install Terraform  https   learn hashicorp com terraform getting started install html       Disclaimer This is for example purposes only  You should take a much closer look at the firewall rules that make sense for your organization s security requirements       Should I run with Terraform or  gcloud  SDK   This repo provides artifacts to spin up the infrastructure for persisting job history and yarn logs with terraform or with gcloud  The recommended way to use this is to modify the Terraform code to fit your needs for long running resources   However  the cluster templates are included as an example of standardizing cluster creation for ephemeral clusters  You might ask   Why is there a cluster template for the history server    The history server is simply a cleaner interface for reading your logs from GCS  For Spark  it is stateless and you may wish to only spin up  a history server when you ll actually be using it  For MapReduce  the history server will only be aware of the files on GCS when it was created and those files which it moves from intermediate done directory to the done directory  For this reason  MapReduce workflows should use a persistent history server       Managing Log Retention Often times  it makes sense to leave the history server running because several teams may use it and you could configure it to manage clean up of your logs by setting the following additional properties      yarn yarn log aggregation retain seconds      spark spark history fs cleaner enabled      spark spark history fs cleaner maxAge      mapred mapreduce jobhistory cleaner enable      mapred mapreduce jobhistory max age ms       Terraform To spin up the whole example you could simply edit the  terraform tfvars  file to set the variables to the desired values and run the following commands   Note  this assumes that you have an existing project and the sufficient permissions to spin up the resources for this example      cd terraform terraform init terraform apply     This will create  1  VPC Network and subnetwork for your dataproc clusters  1  Various firewall rules for this network  1  A single node dataproc history server cluster  1  A long running dataproc cluster  1  A GCS Bucket for YARN log aggregation  and Spark MapReduce Job History as well as initialization actions for your clusters       Alternatively  with Google Cloud SDK These instructions detail how to run this entire example with  gcloud   1   Replace  PROJECT  with your GCP project id in each file  1   Replace  HISTORY BUCKET  with your GCS bucket for logs in each file  1   Replace  HISTORY SERVER  with your dataproc history server  1   Replace  REGION  with your desired GCP Compute region  1   Replace  ZONE  with your desired GCP Compute zone       cd workflow templates sed  i  s PROJECT your gcp project id g    sed  i  s HISTORY BUCKET your history bucket g    sed  i  s HISTORY SERVER your history server g    sed  i  s REGION us central1 g    sed  i  s ZONE us central1 f g    sed  i  s SUBNET your subnet id g     cd cluster templates sed  i  s PROJECT your gcp project id g    sed  i  s HISTORY BUCKET your history bucket g    sed  i  s HISTORY SERVER your history server g    sed  i  s REGION us central1 g    sed  i  s ZONE us central1 f g    sed  i  s SUBNET your subnet id g         Stage an empty file to create the spark events path on GCS       touch  keep gsutil cp  keep gs   your history bucket spark events  keep rm  keep      Stage our initialization action for disabling history servers on your ephemeral clusters      gsutil cp init actions disable history servers sh      Create the history server      sh gcloud beta dataproc clusters import     history server       source cluster templates history server yaml       region us central1      Create a cluster which you can manually submit jobs to and tear down      sh gcloud beta dataproc clusters import   ephemeral cluster     source cluster templates ephemeral cluster yaml     region us central1          Running the Workflow Template Import the workflow template to run an example spark and hadoop job to verify your setup is working      sh gcloud dataproc workflow templates import spark mr example     source workflow templates spark mr workflow template yaml      Trigger the workflow template to spin up a cluster  run the example jobs and tear it down      sh gcloud dataproc workflow templates instantiate spark mr example          Viewing the History UI Follow  these instructions  https   cloud google com dataproc docs concepts accessing cluster web interfaces   to look at the UI by ssh tunneling to the history server  Ports to visit     MapReduce Job History  19888    Spark Job History  18080       Closing Note If you re adapting this example for your own use consider the following  1  Setting an appropriate retention for your logs  1  Setting more appropriate firewall rules for your security requirements