GCP The last year has been like a roller coaster for the cryptocurrency market At the end of 2017 the value of bitcoin BTC almost reached 20 000 USD only to fall below 4 000 USD a few months later What if there is a pattern in the high volatility of the cryptocurrencies market If so can we learn from it and get an edge on future trends Is there a way to observe all exchanges in real time and visualize it on a single chart A Google Cloud Dataflow Cloud Bigtable Websockets example CryptoRealTime In this tutorial we will graph the trades volume and time delta from trade execution until it reaches our system an indicator of how close to real time we can get the data

Answer:
# CryptoRealTime

## A Google Cloud Dataflow/Cloud Bigtable Websockets example

The last year has been like a roller coaster for the cryptocurrency market. At the end of 2017, the value of bitcoin (BTC) almost reached $20,000 USD, only to fall below $4,000 USD a few months later. What if there is a pattern in the high volatility of the cryptocurrencies market? If so, can we learn from it and get an edge on future trends? Is there a way to observe all exchanges in real time and visualize it on a single chart?

In this tutorial we will graph the trades, volume and time delta from trade execution until it reaches our system (an indicator of how close to real time we can get the data).

![realtime multi exchange BTC/USD observer](crypto.gif)

[Consider reading the Medium article](https://medium.com/@igalic/bigtable-beam-dataflow-cryptocurrencies-gcp-terraform-java-maven-4e7873811e86)

[Terraform - get this up and running in less than 5 minutes](https://github.com/galic1987/professional-services/blob/master/examples/cryptorealtime/TERRAFORM-README.md)

## Architecture
![Cryptorealtime Cloud Architecture overview](https://i.ibb.co/dMc9bMz/Screen-Shot-2019-02-11-at-4-56-29-PM.png)

## Frontend
![Cryptorealtime Cloud Fronted overview](https://i.ibb.co/2S28KYq/Screen-Shot-2019-02-12-at-2-53-41-PM.png)

## Costs
This tutorial uses billable components of GCP, including:
- Cloud Dataflow
- Compute Engine
- Cloud Storage
- Cloud Bigtable

We recommend to clean up the project after finishing this tutorial to avoid costs. Use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.

## Project setup
### Install the Google Cloud Platform SDK on a new VM
  * Log into the console, and activate a cloud console session
  * Create a new VM
```console
gcloud beta compute instances create crypto-driver \
  --zone=us-central1-a \
  --machine-type=n1-standard-1 \
  --service-account=$(gcloud iam service-accounts list --format='value(email)' --filter="compute") \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --image=debian-9-stretch-v20181210 \
  --image-project=debian-cloud \
  --boot-disk-size=20GB \
  --boot-disk-device-name=crypto-driver
```

  * SSH into that VM

```console
  gcloud compute ssh --zone=us-central1-a crypto-driver
```

  * Installing necessary tools like java, git, maven, pip, python and Cloud Bigtable command line tool cbt using the following command:
```console
  sudo -s
  apt-get update -y
  curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  sudo python3 get-pip.py
  sudo pip3 install virtualenv
  virtualenv -p python3 venv
  source venv/bin/activate
  sudo apt -y --allow-downgrades install openjdk-8-jdk git maven google-cloud-sdk=271.0.0-0 google-cloud-sdk-cbt=271.0.0-0

```

### Create a Google Cloud Bigtable instance
```console
export PROJECT=$(gcloud info --format='value(config.project)')
export ZONE=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/zone" -H "Metadata-Flavor: Google"|cut -d/ -f4)
gcloud services enable bigtable.googleapis.com \
bigtableadmin.googleapis.com \
dataflow.googleapis.com

gcloud bigtable instances create cryptorealtime \
  --cluster=cryptorealtime-c1 \
  --cluster-zone=${ZONE} \
  --display-name=cryptorealtime \
  --cluster-storage-type=HDD \
  --instance-type=DEVELOPMENT
cbt -instance=cryptorealtime createtable cryptorealtime families=market
```

### Create a Bucket
```console
gsutil mb -p ${PROJECT} gs://realtimecrypto-${PROJECT}
```

### Create firewall for visualization server on port 5000
```console
gcloud compute firewall-rules create crypto-dashboard --action=ALLOW --rules=tcp:5000 --source-ranges=0.0.0.0/0 --target-tags=crypto-console --description="Open port 5000 for crypto visualization tutorial"

gcloud compute instances add-tags crypto-driver --tags="crypto-console" --zone=${ZONE}
```

### Clone the repo
```console
git clone https://github.com/GoogleCloudPlatform/professional-services
```

### Build the pipeline
```console
cd professional-services/examples/cryptorealtime
mvn clean install
```

### Start the DataFlow pipeline
```console
./run.sh ${PROJECT} \
cryptorealtime gs://realtimecrypto-${PROJECT}/temp \
cryptorealtime market
```

### Start the Webserver and Visualization
```console
cd frontend/
pip install -r requirements.txt
python app.py ${PROJECT} cryptorealtime cryptorealtime market
```

Find your external IP in [Compute console instance](https://console.cloud.google.com/compute/instances) and open it in your browser with port 5000 at the end e.g.
http://external-ip:5000/stream

You should be able to see the visualization of aggregated BTC/USD pair on several exchanges (without predictor part)

# Cleanup
* To save on cost we can clean up the pipeline by running the following command
```console
gcloud dataflow jobs cancel \
  $(gcloud dataflow jobs list \
  --format='value(id)' \
  --filter="name:runthepipeline*")
```

* Empty and Delete the bucket:
```console
  gsutil -m rm -r gs://realtimecrypto-${PROJECT}/*
  gsutil rb gs://realtimecrypto-${PROJECT}
```

* Delete the Cloud Bigtable instance:
```console
  gcloud bigtable instances delete cryptorealtime
```

* Exit the VM and delete it.
```console
  gcloud compute instances delete crypto-driver --delete-disks
```

1. View the status of your Dataflow job in the Cloud Dataflow console

1. After a few minutes, from the shell,

```console
  cbt -instance=<bigtable-instance-name> read <bigtable-table-name>
```

Should return many rows of crypto trades data that the frontend project will read for it's dashboard.

## External libraries used to connect to exchanges
https://github.com/bitrich-info/xchange-stream

CryptoRealTime     A Google Cloud Dataflow Cloud Bigtable Websockets example  The last year has been like a roller coaster for the cryptocurrency market  At the end of 2017  the value of bitcoin  BTC  almost reached  20 000 USD  only to fall below  4 000 USD a few months later  What if there is a pattern in the high volatility of the cryptocurrencies market  If so  can we learn from it and get an edge on future trends  Is there a way to observe all exchanges in real time and visualize it on a single chart   In this tutorial we will graph the trades  volume and time delta from trade execution until it reaches our system  an indicator of how close to real time we can get the data       realtime multi exchange BTC USD observer  crypto gif    Consider reading the Medium article  https   medium com  igalic bigtable beam dataflow cryptocurrencies gcp terraform java maven 4e7873811e86    Terraform   get this up and running in less than 5 minutes  https   github com galic1987 professional services blob master examples cryptorealtime TERRAFORM README md      Architecture   Cryptorealtime Cloud Architecture overview  https   i ibb co dMc9bMz Screen Shot 2019 02 11 at 4 56 29 PM png      Frontend   Cryptorealtime Cloud Fronted overview  https   i ibb co 2S28KYq Screen Shot 2019 02 12 at 2 53 41 PM png      Costs This tutorial uses billable components of GCP  including    Cloud Dataflow   Compute Engine   Cloud Storage   Cloud Bigtable  We recommend to clean up the project after finishing this tutorial to avoid costs  Use the  Pricing Calculator  https   cloud google com products calculator   to generate a cost estimate based on your projected usage      Project setup     Install the Google Cloud Platform SDK on a new VM     Log into the console  and activate a cloud console session     Create a new VM    console gcloud beta compute instances create crypto driver       zone us central1 a       machine type n1 standard 1       service account   gcloud iam service accounts list   format  value email     filter  compute         scopes https   www googleapis com auth cloud platform       image debian 9 stretch v20181210       image project debian cloud       boot disk size 20GB       boot disk device name crypto driver           SSH into that VM     console   gcloud compute ssh   zone us central1 a crypto driver          Installing necessary tools like java  git  maven  pip  python and Cloud Bigtable command line tool cbt using the following command     console   sudo  s   apt get update  y   curl https   bootstrap pypa io get pip py  o get pip py   sudo python3 get pip py   sudo pip3 install virtualenv   virtualenv  p python3 venv   source venv bin activate   sudo apt  y   allow downgrades install openjdk 8 jdk git maven google cloud sdk 271 0 0 0 google cloud sdk cbt 271 0 0 0           Create a Google Cloud Bigtable instance    console export PROJECT   gcloud info   format  value config project    export ZONE   curl  http   metadata google internal computeMetadata v1 instance zone   H  Metadata Flavor  Google  cut  d   f4  gcloud services enable bigtable googleapis com   bigtableadmin googleapis com   dataflow googleapis com  gcloud bigtable instances create cryptorealtime       cluster cryptorealtime c1       cluster zone   ZONE        display name cryptorealtime       cluster storage type HDD       instance type DEVELOPMENT cbt  instance cryptorealtime createtable cryptorealtime families market          Create a Bucket    console gsutil mb  p   PROJECT  gs   realtimecrypto   PROJECT           Create firewall for visualization server on port 5000    console gcloud compute firewall rules create crypto dashboard   action ALLOW   rules tcp 5000   source ranges 0 0 0 0 0   target tags crypto console   description  Open port 5000 for crypto visualization tutorial   gcloud compute instances add tags crypto driver   tags  crypto console    zone   ZONE            Clone the repo    console git clone https   github com GoogleCloudPlatform professional services          Build the pipeline    console cd professional services examples cryptorealtime mvn clean install          Start the DataFlow pipeline    console   run sh   PROJECT    cryptorealtime gs   realtimecrypto   PROJECT  temp   cryptorealtime market          Start the Webserver and Visualization    console cd frontend  pip install  r requirements txt python app py   PROJECT  cryptorealtime cryptorealtime market      Find your external IP in  Compute console instance  https   console cloud google com compute instances  and open it in your browser with port 5000 at the end e g  http   external ip 5000 stream  You should be able to see the visualization of aggregated BTC USD pair on several exchanges  without predictor part      Cleanup   To save on cost we can clean up the pipeline by running the following command    console gcloud dataflow jobs cancel       gcloud dataflow jobs list       format  value id         filter  name runthepipeline           Empty and Delete the bucket     console   gsutil  m rm  r gs   realtimecrypto   PROJECT      gsutil rb gs   realtimecrypto   PROJECT         Delete the Cloud Bigtable instance     console   gcloud bigtable instances delete cryptorealtime        Exit the VM and delete it     console   gcloud compute instances delete crypto driver   delete disks      1  View the status of your Dataflow job in the Cloud Dataflow console  1  After a few minutes  from the shell      console   cbt  instance  bigtable instance name  read  bigtable table name       Should return many rows of crypto trades data that the frontend project will read for it s dashboard       External libraries used to connect to exchanges https   github com bitrich info xchange stream


GCP The word pipeline refers to a collection of all job instances of a query in the transformation process in a data warehouse in this case BigQuery Each pipeline involves source table s and destination table For example a query Table Access Pattern Analysis FROM project dataset flashsalepurchases SELECT purchaseId shop Pipeline Optimisation This module consists of deep dive analysis of a BigQuery environment in Google Cloud Platform according to audit logs data access data which can be used to optimise BigQuery usage and improve time space and cost of BigQuery b Definitions b

Answer:
# Table Access Pattern Analysis
This module consists of deep dive analysis of a BigQuery environment in Google Cloud Platform, according to audit logs - data access data, which can be used to optimise BigQuery usage, and improve time, space and cost of BigQuery.

## Pipeline Optimisation

#### Definitions
The word 'pipeline' refers to a collection of all job instances of a query in the transformation process in a data warehouse, in this case BigQuery. Each 'pipeline' involves source table(s) and destination table. For example, a query:
```
SELECT purchaseId, shop
FROM project.dataset.flash_sale_purchases
WHERE isReturn = TRUE
GROUP BY shop
```
with its destination table set to return_purchases table. The source table of this query is flash_sale_purchases and its destination table is return_purchases table.

![](assets/pipeline-definition.png)

In the illustration above, one of the pipeline involves T1 and T2 as its source tables and T5 as its destination table.

Given enough historical data from the audit logs, you can group queries which have the same source table(s) and destination table combination (each group becomes a single pipeline), and see the run history of each of the pipelines. Same source table(s) - destination table combination will almost always come from the same query, even if they are a different query, the semantics should be similar, so this assumption is still valid. After grouping into different pipelines, according to the source table(s) - destination table combination, you might be able to see a pattern in their execution history. You might see that this pipeline is executed hourly, or daily, or even monthly, and when it was last executed.

Note that each of the source table(s) can be a different table type, it can be either a view, materialized view, or a normal table. We only consider the normal tables because the notion of 'analysing the update pattern' does not really apply to view and materialized view. View does not gets updated, and materialized view gets updated automatically

We can categorise every pipeline to different pipeline type and scheduling patern. There are 3 different pipeline types, namely:

Live pipelines: Pipelines that run regularly on an obvious schedule, until now

Dead pipelines: Pipelines that used to run regularly on an obvious schedule, but it stopped some time ago

Ad hoc pipelines: Pipelines that does not have a regular scheduling pattern detected, or not enough repetition to conclude that it’s a scheduled live pipeline

The scheduling pattern can be identified as hourly, daily, weekly, or non-deterministically (no obvious pattern).

#### Purpose
This tool helps identify tables with a high difference between the write and read frequency across data warehouse queries. High discrepancy between the write and read frequency of a table might be a good starting point for identifying optimisation points. Using this tool we can visualise the pipelines that are involved with a table of interest. We can then further analyse the pipeline type, its scheduling pattern and the jobs in each pipeline, and pinpoint problems or optimisations from there.

![](assets/pipeline-example.gif)

This GIF shows the graph that will be visualised when running the tool. The graph is an HTML page that is rendered as an iFrame in the Jupyter Notebook. You can zoom in, zoom out, select (or deselect) a node to see more details about it. Each node represents a table, and each edge represents a pipeline from one table to another. The weight of the edges indicates the frequency of jobs of that pipeline compared to the rest of the pipelines in the current graph.

#### Analysing the Result
As can be seen from the GIF, the tool will visualise all the pipelines associated with a table. To be specific, this includes all query jobs that has this table of interest as its source or destination table. As mentioned above, for every query job, there are source table(s) and also a single destination table.

The tables that are involved in the pipelines associated with a table (all tables that are included when the pipeline graph of a table of interest is plotted) employs the below logic:

For every query jobs that has the table of interest as one of its source tables or destination table,
* For every source table(s) of every query job that has the table of interest as one of its source table(s), recursively find query jobs that has this source table as its destination table, and get its source table(s).
* For every destination table of every query job that has the table of interest as its destination table, recursively find query jobs that has destination table as its source table, and get its destination table.

As seen from the GIF, for every table that is involved in the pipeline of the table of interest, you can select it, and see the details of the job schedule of every query involving this particular table. It will list all ad-hoc, live or dead pipelines that have this table as its source or destination table.

For example, a pipeline information on the right side of the graph might look like this

![](assets/pipeline-information-example.png)

This means that the table `data-analytics-pocs.public.bigquery_audit_log` is a destination table in an ad-hoc pipeline, where `project4.finance.cloudaudit_googleapis_com_data_access_*` are the source table(s). The jobs of this pipeline has a non-deterministic schedule, and its pipeline ID is 208250. The pipeline ID information is useful if you want to further analyse this pipeline by querying the intermediate tables created. See [intermediate tables](#intermediate-tables-creation)

Given these insights, you can further deep dive into insights that are particularly interesting for you. For example, you might identify imbalance queries, the table `flash_sale_purchases` in your data warehouse might updated hourly but only queried daily. You might also identify queries that are already ‘dead’ and no longer scheduled, according to the last execution time, and identify if this is intended, or something might have happened inside the query that causes an error.

## Architecture
This tool is built on top of BigQuery and Python modules. The data source of the tool is audit log - data access which is located in BigQuery. The module will be responsible for the creation of intermediate tables (from the audit logs - data access source table), and the execution of all relevant queries towards those intermediate tables that will be used for analysis purposes. The analysis can be done through a Jupyter notebook, which can be run locally (if installed) or in AI Platform Notebooks. This guide will specifically be on running the tool on AI Platform Notebooks

![](assets/architecture.png)

### Directories and Files
```
data-dumpling-data-assessment/
├── table-access-pattern-analysis/
│   ├── assets/
│   ├── bq_routines/
│   ├── pipeline_graph/
│   ├── src/
│   ├── templates/
│   ├── README.md
│   ├── pipeline.ipynb
│   ├── pipeline-output_only.ipynb
│   ├── requirements.txt
│   └── var.env
```

There are several subdirectories under the `table-access-pattern-analysis` subdirectory.

 assets/

This directory contains images or other assets that are used in README.md

 bq_routines/

This directory contains all the [JS UDF](https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions#javascript-udf-structure) functions that will be created in BigQuery upon usage of the tool. These files are not to be run independently in a JS environment, these file contents will be loaded by the Python package, `src/` to be constructed as a function creation query to BigQuery.

For more information about each of the functions, look at this [section](#routines-creation)

 pipeline_graph/

This directory contains the HTML file, which is a webpage that is used to display the pipeline visualisation of the pipeline optimisation module.

 src/

This directory is the source Python package of the module, it drives the logic for table and routines creation, as well as query towards BigQuery tables.

 templates/

This directory consist of a template HTML file that will be filled using Jinja2 templating system, through the Python code.

 README.md

This is the README file which explains all the details fo this directory.

 pipeline.ipynb

This Notebook is used for the pipeline optimisation.

 pipeline-output_only.ipynb

This Notebook is used for demonstration purposes of the pipeline optimisation only, it shows the expected output and result of running the notebook.

 requirements.txt

This file consist of all the dependencies, you don't need to install it manually because it's part of the Jupyter Notebooks command.

 var.env

This is the file on which environment variables are to be defined and to be loaded by the different Jupyter Notebooks. For every 'analysis workflow', you should redefine some of the variables. For details, look at this [section](#environment-variables)

## Prerequisites
* Your account must have access to read the audit logs - data access table that will be used as a source table for the analysis. For more details regarding different kinds of audit logs, visit this [page](https://cloud.google.com/logging/docs/audit#data-access)
* The audit logs - data access table that will be used as a source table for the analysis should contain BigQuery logs version 1. For more details regarding audit logs version, visit this [page](https://cloud.google.com/bigquery/docs/reference/auditlogs)
* Your account must have access to write to the destination dataset.
* The source and destination dataset must be in the same location

## Set Up
This set up is for running JupyterLab Notebook in AI Platform Notebooks, you can also choose to run the Jupyter Notebook locally.
1. Go to a GCP project.
2. Navigate to AI Platform -> Notebooks. New Instance -> Choose Python3 Option -> Name the instance
3. Clone this repository
4. Go to the `table-access-pattern-analysis` directory of the project
5. Set the environment variables inside `var.env`.
6. Run the analysis, as described [below](#analysis).

### Environment Variables
The environment variables that you need to set includes:
* INPUT_PROJECT_ID
* INPUT_DATASET_ID
* INPUT_AUDIT_LOGS_TABLE_ID
* IS_AUDIT_LOGS_INPUT_TABLE_PARTITIONED
* OUTPUT_PROJECT_ID
* OUTPUT_DATASET_ID
* OUTPUT_TABLE_SUFFIX
* LOCATION
* IS_INTERACTIVE_TABLES_MODE

The details of each of the environment variables are as follows:

INPUT_PROJECT_ID, INPUT_DATASET_ID, INPUT_AUDIT_LOGS_TABLE_ID

 Definition

* These 3 environment variables should point to the audit logs - data access table that will be the source table of the analysis. The complete path to the audit logs table source will be `INPUT_PROJECT_ID.INPUT_DATASET_ID.INPUT_AUDIT_LOGS_TABLE_ID`. If you want to analyse on a table with a wildcard, include the wildcard in the INPUT_AUDIT_LOGS_TABLE_ID variable as well.

 Example values

* INPUT_PROJECT_ID = 'project-a'
* INPUT_DATASET_ID = 'dataset-b'
* INPUT_AUDIT_LOGS_TABLE_ID = 'cloudaudit_googleapis_com_data_access_*'

OUTPUT_PROJECT_ID, OUTPUT_DATASET_ID

 Definition

* These 2 environment variables should point to the dataset ID that will contain all the tables and routines that is going to be created during the analysis.

 Example values

* OUTPUT_PROJECT_ID = 'project-c'
* OUTPUT_DATASET_ID = 'dataset-d'

OUTPUT_TABLE_SUFFIX

 Definition

* The 'OUTPUT_TABLE_SUFFIX' variable is used to denote an 'analysis environment' that you intend to build. All tables that are produced by this run will have this variable as its suffix, thus it will not replace any existing tables that you have created for other analysis.

* If this variable is not set, the analysis cannot be run as you might unintentionally forgot to change the suffix and replace an existing set of tables with the same suffix set.

 Example value

* OUTPUT_TABLE_SUFFIX = 'first-analysis'.

IS_AUDIT_LOGS_INPUT_TABLE_PARTITIONED

 Definition

* The 'IS_AUDIT_LOGS_INPUT_TABLE_PARTITIONED' variable is a boolean value which denotes whether the input audit log table is a partitioned table.

 Value

* Its value should be either "TRUE" or "FALSE", with the exact casing.

LOCATION

 Definition

* The 'LOCATION' variable is used to specify the region on which the input dataset and output dataset is located, a common and most used location is 'US'.

 Example value

* LOCATION=US

IS_INTERACTIVE_TABLES_MODE

 Definition

* Boolean on whether you want the tables to be interactive, it is recommended to set this to "TRUE".
* If you want the tables output to be interactive (can filter, sort, search), you should set this value to "TRUE".
* If you want the tables output to not be interactive, you can set this value to "FALSE".

 Value

* Its value should be either "TRUE" or "FALSE", with the exact casing.

### Caveats
After resetting any environment variables, you need to restart the kernel because otherwise it will not be loaded by Jupyter. To restart, go to the menu 'Kernel' and choose 'Restart'

## Analysis
1. Open a notebook to run an analysis.
2. You can choose the interactivity mode of the output.
    * If you want the tables output to be interactive, you can choose to run the Classic Jupyter Notebook. The output of the tables produced by this notebook will be interactive (can filter, sort, search), but it is an older version of Jupyter notebook in AI Platform Notebooks. To do this,
        1. Navigate to `Help` menu in Jupyter
        2. Click on `Launch Classic Notebook`.
        3. Navigate the directory and open the Notebook that you want to do the analysis on.

    * If you prefer a newer version of the Jupyter notebook, you can choose to not run the Classic Jupyter Notebook. The output of the tables produced by this notebook is not interactive. You can double click on the intended Notebook from the list of files, without following the steps to launch a Classic Jupyter Notebook
2. Run the cells from top to bottom of the notebook.
3. In the first cell, there is a datetime picker, which is used to filter the audit logs data source to the start and end date range specified. If you select `05-05-2021` as a start date and `06-05-2021`, the analysis result of the notebook run will be based on audit logs data on 5th May 2021 to 6th May 2021.
4. Run pipeline optimisation analysis produced in Jupyter Notebook

Pipeline Optimisation, run `pipeline.ipynb`

    This tool helps identify pipeline optimisation points. At first, the tool will list down tables with high difference of writing and reading frequency throughout the data warehouse queries.

    After identifying the table that you would like to analyse further, you can select the table in the next part of the notebook and display the result in an iFrame inside the notebook.

## Appendix
### Intermediate Tables Creation
As mentioned in the [Architecture](#architecture) section, this module involves the creation of intermediate tables. These are important and relevant for users that are interested to analyse the insights generated from this tool even further. The schema and details of each intermediate tables created are explained below.

 job_info_with_tables_info<OUTPUT_TABLE_SUFFIX>

This table stores some of the details of job history that are relevant to pipeline optimisation. Each job history entry corresponds to a single entry in the audit logs. The audit logs are filtered to the ones that are relevant for pipeline optimisation.
```
[
    {
        "name": "jobId",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The job ID of this job run"
    },
    {
        "name": "timestamp",
        "type": "TIMESTAMP",
        "mode": "NULLABLE",
        "description": "Timestamp when the job was run"
    },
    {
        "name": "email",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The account that ran this job"
    },
    {
        "name": "projectId",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The project ID this job was ran on"
    },
    {
        "name": "totalSlotMs",
        "type": "INTEGER",
        "mode": "NULLABLE",
        "description": "The slot ms consumed by this job"
    },
    {
        "name": "totalProcessedBytes",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The total bytes processed when this job was ran"
    },
    {
        "name": "destinationTable",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The destination table of this job, in a concatenated 'project.dataset.table' string format"
    },
    {
        "name": "sourceTables",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The source tables of this job, in a JSON string format of the array of concatenated 'project.dataset.table' string format, for example it can be a string of '[tableA, tableB, tableC]'"
    }
]
```

 pipeline_info<OUTPUT_TABLE_SUFFIX>

This table stores the information of the different pipelines. Each unique pipeline is a collection of all job instances of a query (involving unique source table(s)-destination table combination) in the transformation process in BigQuery.
```
[
    {
        "name": "pipelineId",
        "type": "INTEGER",
        "mode": "NULLABLE",
        "description": "The pipeline ID of this job run"
    },
    {
        "name": "timestamps",
        "type": "ARRAY<TIMESTAMP>",
        "mode": "NULLABLE",
        "description": "Timestamps when this pipeline was run in the past"
    },
    {
        "name": "pipelineType",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The pipeline type of this pipeline, its value can be dead/live/ad hoc"
    },
    {
        "name": "schedule",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The schedule for this pipeline, its value can be non deterministic/hourly/daily/monthly"
    },
    {
        "name": "destinationTable",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The destination table of this pipeline, in a concatenated 'project.dataset.table' string format"
    },
    {
        "name": "sourceTables",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The source tables of this job, in a JSON string format of the array of concatenated 'project.dataset.table' string format, for example it can be a string of '[tableA, tableB, tableC]'"
    }
]
```

source_destination_table_pairs<OUTPUT_TABLE_SUFFIX>

This table stores all source-destination table pair. It also stores the pipeline ID, which is the pipeline ID that this pair was part of.

```
[
    {
        "name": "destinationTable",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The destination table"
    },
    {
        "name": "sourceTable",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The source table"
    },
    {
        "name": "pipelineId",
        "type": "INTEGER",
        "mode": "NULLABLE",
        "description": "The pipeline ID for the pipeline that this pair was part of"
    }
]
```

table_direct_pipelines<OUTPUT_TABLE_SUFFIX>

This table stores all table pipeline, as destination table and as source table
```
[
    {
        "name": "table",
        "type": "STRING",
        "mode": "NULLABLE",
        "description": "The table"
    },
    {
        "name": "directBackwardPipelines",
        "type": "ARRAY<STRUCT<INTEGER, STRING, STRING, INTEGER, STRING, STRING>>",
        "mode": "NULLABLE",
        "description": "An array of pipeline informations that have the current table as its destination table. Each of the struct has information about the pipelineId, sourceTables, destinationTable, frequency, pipelineType, and schedule"
    },
    {
        "name": "directForwardPipelines",
        "type": "ARRAY<STRUCT<INTEGER, STRING, STRING, INTEGER, STRING, STRING>>",
        "mode": "NULLABLE",
        "description": "An array of pipeline informations that have the current table as one of its source table. Each of the struct has information about the pipelineId, sourceTables, destinationTable, frequency, pipelineType, and schedule"
    }
]
```

### Routines Creation
There are several JavaScript UDFs created in BigQuery upon usage of the tool. These function files are not to be run independently in a JS environment, these file contents will be loaded by the Python package, `src/` to be constructed as a function creation query to BigQuery.

getPipelineTypeAndSchedule

This function takes an array of timestamps and returns a struct of the pipeline type and schedule according to the history. There are 3 possible values for pipeline type: live/dead/ad hoc, and there are 4 possible values for schedule: non deterministic/hourly/daily/monthly.

The routine file content is located in `bq_routines/getPipelineTypeAndSchedule.js`

getTablesInvolvedInPipelineOfTable

This function returns a list of tables that are involved in the pipeline of the table of input.

The routine file content is located in `bq_routines/getTablesInvolvedInPipelineOfTable.js`

</ul

Table Access Pattern Analysis This module consists of deep dive analysis of a BigQuery environment in Google Cloud Platform  according to audit logs   data access data  which can be used to optimise BigQuery usage  and improve time  space and cost of BigQuery      Pipeline Optimisation        b Definitions  b  The word  pipeline  refers to a collection of all job instances of a query in the transformation process in a data warehouse  in this case BigQuery  Each  pipeline  involves source table s  and destination table  For example  a query      SELECT purchaseId  shop FROM project dataset flash sale purchases WHERE isReturn   TRUE GROUP BY shop     with its destination table set to return purchases table  The source table of this query is flash sale purchases and its destination table is return purchases table        assets pipeline definition png   In the illustration above  one of the pipeline involves T1 and T2 as its source tables and T5 as its destination table   Given enough historical data from the audit logs  you can group queries which have the same source table s  and destination table combination  each group becomes a single pipeline   and see the run history of each of the pipelines  Same source table s    destination table combination will almost always come from the same query  even if they are a different query  the semantics should be similar  so this assumption is still valid  After grouping into different pipelines  according to the source table s    destination table combination  you might be able to see a pattern in their execution history  You might see that this pipeline is executed hourly  or daily  or even monthly  and when it was last executed     i Note that each of the source table s  can be a different table type  it can be either a view  materialized view  or a normal table  We only consider the normal tables because the notion of  analysing the update pattern  does not really apply to view and materialized view  View does not gets updated  and materialized view gets updated automatically  i   We can categorise every pipeline to different pipeline type and scheduling patern  There are 3 different pipeline types  namely   ul   li Live pipelines  Pipelines that run regularly on an obvious schedule  until now  li Dead pipelines  Pipelines that used to run regularly on an obvious schedule  but it stopped some time ago  li Ad hoc pipelines  Pipelines that does not have a regular scheduling pattern detected  or not enough repetition to conclude that it s a scheduled live pipeline   ul   The scheduling pattern can be identified as hourly  daily  weekly  or non deterministically  no obvious pattern          b Purpose  b  This tool helps identify tables with a high difference between the write and read frequency across data warehouse queries  High discrepancy between the write and read frequency of a table might be a good starting point for identifying optimisation points  Using this tool we can visualise the pipelines that are involved with a table of interest  We can then further analyse the pipeline type  its scheduling pattern and the jobs in each pipeline  and pinpoint problems or optimisations from there        assets pipeline example gif   This GIF shows the graph that will be visualised when running the tool  The graph is an HTML page that is rendered as an iFrame in the Jupyter Notebook  You can zoom in  zoom out  select  or deselect  a node to see more details about it  Each node represents a table  and each edge represents a pipeline from one table to another  The weight of the edges indicates the frequency of jobs of that pipeline compared to the rest of the pipelines in the current graph         b Analysing the Result  b  As can be seen from the GIF  the tool will visualise all the pipelines associated with a table  To be specific  this includes all query jobs that has this table of interest as its source or destination table  As mentioned above  for every query job  there are source table s  and also a single destination table    The tables that are involved in the pipelines associated with a table  all tables that are included when the pipeline graph of a table of interest is plotted  employs the below logic   For every query jobs that has the table of interest as one of its source tables or destination table     For every source table s  of every query job that has the table of interest as one of its source table s   recursively find query jobs that has this source table as its destination table  and get its source table s      For every destination table of every query job that has the table of interest as its destination table  recursively find query jobs that has destination table as its source table  and get its destination table   As seen from the GIF  for every table that is involved in the pipeline of the table of interest  you can select it  and see the details of the job schedule of every query involving this particular table  It will list all ad hoc  live or dead pipelines that have this table as its source or destination table    For example  a pipeline information on the right side of the graph might look like this      assets pipeline information example png   This means that the table  data analytics pocs public bigquery audit log  is a destination table in an ad hoc pipeline  where  project4 finance cloudaudit googleapis com data access    are the source table s   The jobs of this pipeline has a non deterministic schedule  and its pipeline ID is 208250  The pipeline ID information is useful if you want to further analyse this pipeline by querying the intermediate tables created  See  intermediate tables   intermediate tables creation   Given these insights  you can further deep dive into insights that are particularly interesting for you  For example  you might identify imbalance queries  the table  flash sale purchases  in your data warehouse might updated hourly but only queried daily  You might also identify queries that are already  dead  and no longer scheduled  according to the last execution time  and identify if this is intended  or something might have happened inside the query that causes an error      Architecture This tool is built on top of BigQuery and Python modules  The data source of the tool is audit log   data access which is located in BigQuery  The module will be responsible for the creation of intermediate tables  from the audit logs   data access source table   and the execution of all relevant queries towards those intermediate tables that will be used for analysis purposes  The analysis can be done through a Jupyter notebook  which can be run locally  if installed  or in AI Platform Notebooks  This guide will specifically be on running the tool on AI Platform Notebooks      assets architecture png       Directories and Files     data dumpling data assessment      table access pattern analysis          assets          bq routines          pipeline graph          src          templates          README md         pipeline ipynb         pipeline output only ipynb         requirements txt         var env      There are several subdirectories under the  table access pattern analysis  subdirectory   ul   li   b assets   b   This directory contains images or other assets that are used in README md   li   b bq routines   b   This directory contains all the  JS UDF  https   cloud google com bigquery docs reference standard sql user defined functions javascript udf structure  functions that will be created in BigQuery upon usage of the tool  These files are not to be run independently in a JS environment  these file contents will be loaded by the Python package   src   to be constructed as a function creation query to BigQuery   For more information about each of the functions  look at this  section   routines creation    li   b pipeline graph   b   This directory contains the HTML file  which is a webpage that is used to display the pipeline visualisation of the pipeline optimisation module    li   b src   b   This directory is the source Python package of the module  it drives the logic for table and routines creation  as well as query towards BigQuery tables    li   b templates   b   This directory consist of a template HTML file that will be filled using Jinja2 templating system  through the Python code    li   b README md  b   This is the README file which explains all the details fo this directory    li   b pipeline ipynb  b   This Notebook is used for the pipeline optimisation     li   b pipeline output only ipynb  b   This Notebook is used for demonstration purposes of the pipeline optimisation only  it shows the expected output and result of running the notebook     li   b requirements txt  b   This file consist of all the dependencies  you don t need to install it manually because it s part of the Jupyter Notebooks command    li   b var env  b   This is the file on which environment variables are to be defined and to be loaded by the different Jupyter Notebooks  For every  analysis workflow   you should redefine some of the variables  For details  look at this  section   environment variables     ul      Prerequisites   Your account must have access to read the audit logs   data access table that will be used as a source table for the analysis  For more details regarding different kinds of audit logs  visit this  page  https   cloud google com logging docs audit data access    The audit logs   data access table that will be used as a source table for the analysis should contain BigQuery logs version 1  For more details regarding audit logs version  visit this  page  https   cloud google com bigquery docs reference auditlogs    Your account must have access to write to the destination dataset    The source and destination dataset must be in the same location     Set Up This set up is for running JupyterLab Notebook in AI Platform Notebooks  you can also choose to run the Jupyter Notebook locally  1  Go to a GCP project  2  Navigate to  b AI Platform    Notebooks  b    b New Instance    Choose Python3 Option    Name the instance   b  3  Clone this repository  4  Go to the  table access pattern analysis  directory of the project 5  Set the environment variables inside  var env   6  Run the analysis  as described  below   analysis        Environment Variables The environment variables that you need to set includes    INPUT PROJECT ID   INPUT DATASET ID   INPUT AUDIT LOGS TABLE ID   IS AUDIT LOGS INPUT TABLE PARTITIONED   OUTPUT PROJECT ID   OUTPUT DATASET ID   OUTPUT TABLE SUFFIX   LOCATION   IS INTERACTIVE TABLES MODE  The details of each of the environment variables are as follows   ul   li  b INPUT PROJECT ID  INPUT DATASET ID  INPUT AUDIT LOGS TABLE ID  b    ul   li  Definition     These 3 environment variables should point to the audit logs   data access table that will be the source table of the analysis  The complete path to the audit logs table source will be  INPUT PROJECT ID INPUT DATASET ID INPUT AUDIT LOGS TABLE ID   If you want to analyse on a table with a wildcard  include the wildcard in the INPUT AUDIT LOGS TABLE ID variable as well     li  Example values    INPUT PROJECT ID    project a    INPUT DATASET ID    dataset b    INPUT AUDIT LOGS TABLE ID    cloudaudit googleapis com data access      ul    li  b OUTPUT PROJECT ID  OUTPUT DATASET ID  b   ul   li  Definition     These 2 environment variables should point to the dataset ID that will contain all the tables and routines that is going to be created during the analysis    li  Example values    OUTPUT PROJECT ID    project c    OUTPUT DATASET ID    dataset d     ul    li  b OUTPUT TABLE SUFFIX  b   ul   li  Definition     The  OUTPUT TABLE SUFFIX  variable is used to denote an  analysis environment  that you intend to build  All tables that are produced by this run will have this variable as its suffix  thus it will not replace any existing tables that you have created for other analysis      If this variable is not set  the analysis cannot be run as you might unintentionally forgot to change the suffix and replace an existing set of tables with the same suffix set    li  Example value    OUTPUT TABLE SUFFIX    first analysis      ul    li  b IS AUDIT LOGS INPUT TABLE PARTITIONED  b   ul   li  Definition     The  IS AUDIT LOGS INPUT TABLE PARTITIONED  variable is a boolean value which denotes whether the input audit log table is a partitioned table     li  Value    Its value should be either  TRUE  or  FALSE   with the exact casing     ul    li  b LOCATION  b    ul   li  Definition     The  LOCATION  variable is used to specify the region on which the input dataset and output dataset is located  a common and most used location is  US     li  Example value        LOCATION US   ul    li  b IS INTERACTIVE TABLES MODE  b   ul   li  Definition    Boolean on whether you want the tables to be interactive  it is recommended to set this to  TRUE      If you want the tables output to be interactive  can filter  sort  search   you should set this value to  TRUE      If you want the tables output to not be interactive  you can set this value to  FALSE         li  Value    Its value should be either  TRUE  or  FALSE   with the exact casing     ul    ul       Caveats After resetting any environment variables  you need to restart the kernel because otherwise it will not be loaded by Jupyter  To restart  go to the menu  Kernel  and choose  Restart      Analysis 1  Open a notebook to run an analysis  2  You can choose the interactivity mode of the output        If you want the tables output to be interactive  you can choose to run the Classic Jupyter Notebook  The output of the tables produced by this notebook will be interactive  can filter  sort  search   but it is an older version of Jupyter notebook in AI Platform Notebooks  To do this          1  Navigate to  Help  menu in Jupyter         2  Click on  Launch Classic Notebook            3  Navigate the directory and open the Notebook that you want to do the analysis on         If you prefer a newer version of the Jupyter notebook  you can choose to not run the Classic Jupyter Notebook  The output of the tables produced by this notebook is not interactive  You can double click on the intended Notebook from the list of files  without following the steps to launch a Classic Jupyter Notebook 2  Run the cells from top to bottom of the notebook   3  In the first cell  there is a datetime picker  which is used to filter the audit logs data source to the start and end date range specified  If you select  05 05 2021  as a start date and  06 05 2021   the analysis result of the notebook run will be based on audit logs data on 5th May 2021 to 6th May 2021  4  Run pipeline optimisation analysis produced in Jupyter Notebook      ul       li  b Pipeline Optimisation  run  pipeline ipynb   b       This tool helps identify pipeline optimisation points  At first  the tool will list down tables with high difference of writing and reading frequency throughout the data warehouse queries        After identifying the table that you would like to analyse further  you can select the table in the next part of the notebook and display the result in an iFrame inside the notebook         ul      Appendix     Intermediate Tables Creation As mentioned in the  Architecture   architecture  section  this module involves the creation of intermediate tables  These are important and relevant for users that are interested to analyse the insights generated from this tool even further  The schema and details of each intermediate tables created are explained below     ul   li  job info with tables info OUTPUT TABLE SUFFIX   This table stores some of the details of job history that are relevant to pipeline optimisation  Each job history entry corresponds to a single entry in the audit logs  The audit logs are filtered to the ones that are relevant for pipeline optimisation                       name    jobId            type    STRING            mode    NULLABLE            description    The job ID of this job run                        name    timestamp            type    TIMESTAMP            mode    NULLABLE            description    Timestamp when the job was run                        name    email            type    STRING            mode    NULLABLE            description    The account that ran this job                        name    projectId            type    STRING            mode    NULLABLE            description    The project ID this job was ran on                        name    totalSlotMs            type    INTEGER            mode    NULLABLE            description    The slot ms consumed by this job                        name    totalProcessedBytes            type    STRING            mode    NULLABLE            description    The total bytes processed when this job was ran                        name    destinationTable            type    STRING            mode    NULLABLE            description    The destination table of this job  in a concatenated  project dataset table  string format                        name    sourceTables            type    STRING            mode    NULLABLE            description    The source tables of this job  in a JSON string format of the array of concatenated  project dataset table  string format  for example it can be a string of   tableA  tableB  tableC                  li  pipeline info OUTPUT TABLE SUFFIX   This table stores the information of the different pipelines  Each unique pipeline is a collection of all job instances of a query  involving unique source table s  destination table combination  in the transformation process in BigQuery                       name    pipelineId            type    INTEGER            mode    NULLABLE            description    The pipeline ID of this job run                        name    timestamps            type    ARRAY TIMESTAMP             mode    NULLABLE            description    Timestamps when this pipeline was run in the past                        name    pipelineType            type    STRING            mode    NULLABLE            description    The pipeline type of this pipeline  its value can be dead live ad hoc                        name    schedule            type    STRING            mode    NULLABLE            description    The schedule for this pipeline  its value can be non deterministic hourly daily monthly                        name    destinationTable            type    STRING            mode    NULLABLE            description    The destination table of this pipeline  in a concatenated  project dataset table  string format                        name    sourceTables            type    STRING            mode    NULLABLE            description    The source tables of this job  in a JSON string format of the array of concatenated  project dataset table  string format  for example it can be a string of   tableA  tableB  tableC                  li source destination table pairs OUTPUT TABLE SUFFIX   This table stores all source destination table pair  It also stores the pipeline ID  which is the pipeline ID that this pair was part of                         name    destinationTable            type    STRING            mode    NULLABLE            description    The destination table                        name    sourceTable            type    STRING            mode    NULLABLE            description    The source table                        name    pipelineId            type    INTEGER            mode    NULLABLE            description    The pipeline ID for the pipeline that this pair was part of                li table direct pipelines OUTPUT TABLE SUFFIX   This table stores all table pipeline  as destination table and as source table                      name    table            type    STRING            mode    NULLABLE            description    The table                        name    directBackwardPipelines            type    ARRAY STRUCT INTEGER  STRING  STRING  INTEGER  STRING  STRING              mode    NULLABLE            description    An array of pipeline informations that have the current table as its destination table  Each of the struct has information about the pipelineId  sourceTables  destinationTable  frequency  pipelineType  and schedule                        name    directForwardPipelines            type    ARRAY STRUCT INTEGER  STRING  STRING  INTEGER  STRING  STRING              mode    NULLABLE            description    An array of pipeline informations that have the current table as one of its source table  Each of the struct has information about the pipelineId  sourceTables  destinationTable  frequency  pipelineType  and schedule                ul        Routines Creation There are several JavaScript UDFs created in BigQuery upon usage of the tool  These function files are not to be run independently in a JS environment  these file contents will be loaded by the Python package   src   to be constructed as a function creation query to BigQuery    ul     li getPipelineTypeAndSchedule  This function takes an array of timestamps and returns a struct of the pipeline type and schedule according to the history  There are 3 possible values for pipeline type  live dead ad hoc  and there are 4 possible values for schedule  non deterministic hourly daily monthly    The routine file content is located in  bq routines getPipelineTypeAndSchedule js    li getTablesInvolvedInPipelineOfTable  This function returns a list of tables that are involved in the pipeline of the table of input   The routine file content is located in  bq routines getTablesInvolvedInPipelineOfTable js     ul


GCP deployment neo4jbackuprestoreviagkegcsexample backup Backup Restore via and Example Project Structure

Answer:
# [Neo4j](https://neo4j.com/developer/graph-database/) Backup & Restore via [GKE Cronjob](https://cloud.google.com/kubernetes-engine/docs/how-to/cronjobs) and [GCS](https://cloud.google.com/storage) Example

## Project Structure
```
.
└── neo4j_backup_restore_via_gke_gcs_example
 └── backup
    └── deployment
          ├── backup-cronjob.yaml  #(Cronjob configuration)
          └── deploy-exec.sh  #(Executable for backup deployment)
    └── docker
          ├── Dockerfile  #(Backup pod docker image)
          ├── backup-via-admin.sh #(Helper used by docker image)
          └── pod-image-exec.sh #(Executable for build & push docker image)
    ├── neo4j-backup-architecture.png
    └──  backup.env  #(Update gcloud configuration)
 └── restore
    ├── restore.env  #(Update gcloud configuration)
    ├── download-backup.sh  #(Helper to copy backup from GCS)
    ├── restore-via-admin.sh #(Helper to run restore admin commands)
    ├── restore-exec.sh  #(Excutable for Restore)
    └── cleanup.sh  #(Helper to remove local backup copy on pod)
 └── README.md
```

## Backup

### Backup Architecture
![image info](./backup/neo4j-backup-architecture.png)

### Build and push backup pod image

* Make sure the Enviornment variables are set correctly in ```backup/backup.env``` file.
  - This file should point to the Google Cloud Storage `REMOTE_BACKUPSET` to backup the graphs, GCR bucket used to point to the `BACKUP_IMAGE` used by the backup pod, `GKE_NAMESPACE` to be used to backup from the correct Neo4j cluster, and the `NEO4J_ADMIN_SERVER` IPs to backup the servers from.
* Simply run ```pod-image-exec.sh``` file to build and push the backup pod image.

```bash
# Have execute-access to the script
$ chmod u+x backup/docker/pod-image-exec.sh

# Run the back-up pod image script
$ ./backup/docker/pod-image-exec.sh
```

### Deploy backup kubernetes cronjob
* Navigate to ```backup/deployment/backup-cronjob.yaml``` and edit the schedule or anything else you'd like to customize in the cronjob.
* Run the ```backup/deployment/deploy-exec.sh``` to deploy the cronjob on your neo4j cluster.
```bash
# Have execute-access to the script
$ chmod u+x backup/deployment/deploy-exec.sh

# Run the back-up cronjob deployment script
$ ./backup/deployment/deploy-exec.sh
```

### Update backup pod image
* Configuration for image used by the backup-pod can be found in the file `backup/docker/Dockerfile`.
* If any changes need to be made to the Backup configuration used by the back-up pod, please modify and save your changes on the following shell file `backup-via-admin.sh`
  - Once any of these files are changed, an updated container image needs to be built and pushed to container registry.

```bash
# Script to build and push the image
$ ./pod-image-exec.sh
```

### Delete backup cronjob
* Run the following command to delete the backup cronjob. Replace the <CRONJOB_NAME> with currently assigned cronjob name

```bash
# Delete cronjob
$ kubectl delete cronjob <CRONJOB_NAME>
```

### Re-deploy Backup Cronjob
* Run the following YAML file to de-deploy the Kubernetes Cronjob which schedules the backups

```bash
# Delete cronjob
$ kubectl apply -f backup-cronjob.yaml
```

## Restore

This procedure assumes that you either have sidecar container on your neo4j instance running the google cloud-sdk or your neo4j instance servers have the google cloud-sdk pre-installed

### Download and restore from Google Cloud Storage Bucket

Simply run the ```/restore/restore-exec.sh``` which will call the helper shell scripts and complete the restore process one server at a time.

```bash
# Have execute-access to the script
$ chmod u+x restore/restore-exec.sh

# Execute restore procedure script
$ ./restore/restore-exec.sh

```

## References

This software uses the following open source packages:

- [GKE Cronjobs](https://cloud.google.com/kubernetes-engine/docs/how-to/cronjobs)
- [Artifact Registry](https://cloud.google.com/artifact-registry)
- [Dockerfile](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)
- [Neo4j Enterprise](https://neo4j.com/licensing/)

--

Neo4j  https   neo4j com developer graph database   Backup   Restore via  GKE Cronjob  https   cloud google com kubernetes engine docs how to cronjobs  and  GCS  https   cloud google com storage  Example      Project Structure           neo4j backup restore via gke gcs example      backup         deployment                backup cronjob yaml    Cronjob configuration                deploy exec sh    Executable for backup deployment          docker               Dockerfile    Backup pod docker image                backup via admin sh   Helper used by docker image                pod image exec sh   Executable for build   push docker image          neo4j backup architecture png          backup env    Update gcloud configuration       restore         restore env    Update gcloud configuration          download backup sh    Helper to copy backup from GCS          restore via admin sh   Helper to run restore admin commands          restore exec sh    Excutable for Restore          cleanup sh    Helper to remove local backup copy on pod       README md         Backup      Backup Architecture   image info    backup neo4j backup architecture png       Build and push backup pod image    Make sure the Enviornment variables are set correctly in    backup backup env    file       This file should point to the Google Cloud Storage  REMOTE BACKUPSET  to backup the graphs  GCR bucket used to point to the  BACKUP IMAGE  used by the backup pod   GKE NAMESPACE  to be used to backup from the correct Neo4j cluster  and the  NEO4J ADMIN SERVER  IPs to backup the servers from    Simply run    pod image exec sh    file to build and push the backup pod image      bash   Have execute access to the script   chmod u x backup docker pod image exec sh    Run the back up pod image script     backup docker pod image exec sh          Deploy backup kubernetes cronjob   Navigate to    backup deployment backup cronjob yaml    and edit the schedule or anything else you d like to customize in the cronjob    Run the    backup deployment deploy exec sh    to deploy the cronjob on your neo4j cluster     bash   Have execute access to the script   chmod u x backup deployment deploy exec sh    Run the back up cronjob deployment script     backup deployment deploy exec sh           Update backup pod image   Configuration for image used by the backup pod can be found in the file  backup docker Dockerfile      If any changes need to be made to the Backup configuration used by the back up pod  please modify and save your changes on the following shell file  backup via admin sh      Once any of these files are changed  an updated container image needs to be built and pushed to container registry      bash   Script to build and push the image     pod image exec sh          Delete backup cronjob   Run the following command to delete the backup cronjob  Replace the  CRONJOB NAME  with currently assigned cronjob name     bash   Delete cronjob   kubectl delete cronjob  CRONJOB NAME           Re deploy Backup Cronjob   Run the following YAML file to de deploy the Kubernetes Cronjob which schedules the backups     bash   Delete cronjob   kubectl apply  f backup cronjob yaml         Restore  This procedure assumes that you either have sidecar container on your neo4j instance running the google cloud sdk or your neo4j instance servers have the google cloud sdk pre installed      Download and restore from Google Cloud Storage Bucket  Simply run the     restore restore exec sh    which will call the helper shell scripts and complete the restore process one server at a time      bash   Have execute access to the script   chmod u x restore restore exec sh    Execute restore procedure script     restore restore exec sh          References  This software uses the following open source packages      GKE Cronjobs  https   cloud google com kubernetes engine docs how to cronjobs     Artifact Registry  https   cloud google com artifact registry     Dockerfile  https   docs docker com develop develop images dockerfile best practices      Neo4j Enterprise  https   neo4j com licensing


GCP It explains how to create Flex template and run it in a restricted environment where there is no internet connectivity to dataflow launcher or worker nodes It also run dataflow template on shared VPC Example also contains a DAG which can be used to trigger Dataflow job from composer It also demonstrates how we can use cloudbuild to implement CI CD for this dataflow job Dataflow Python Flex Template This example contains a sample Dataflow job which reads a XML file and inserts the records to BQ table Resorces structure

Answer:
## Dataflow Python Flex Template

This example contains a sample Dataflow job which reads a XML file and inserts the records to BQ table.
It explains how to create Flex template and run it in a restricted environment where there is no
internet connectivity to dataflow launcher or worker nodes. It also run dataflow template on shared VPC.
Example also contains a DAG which can be used to trigger Dataflow job from composer. It also demonstrates
how we can use cloudbuild to implement CI/CD for this dataflow job.

### Resorces structure

Below Tree explains the purpose of each file in the folder.

```
dataflow-flex-python/
├── cloudbuild_base.yaml   --> Cloudbuild config to build SDK image
├── cloudbuild_df_job.yaml --> Cloudbuild config to build Launcher image and Flex template
├── composer_variables.template --> Definition of All Composer variables used by DAG
├── dag
│   └── xml-to-bq-dag.py --> Dag code to launch Dataflow Job
├── df-package ---> Dataflow template package
│   ├── corder
│   │   ├── bq_schema.py --> BQ Table Schemas
│   │   ├── models.py --> Data Model for input data, generated by xsdata and pydantic plugin
│   │   ├── customer_orders.py --> Dataflow pipeline Implementation
│   │   ├── customer_orders_test.py --> pytest for Dataflow pipeline code
│   │   └── __init__.py
│   ├── main.py --> Used by launcher to launch the pipeline
│   └── setup.py --> Used to Install the package
├── Dockerfile_Launcher --> Dockerfile to create Launcher Image
├── Dockerfile_SDK --> Dockerfile to create SDK image
├── metadata.json --> metadata file used during building flex teamplate
├── README.md
├── requirements-test.txt --> Python Requirements for running tests
├── requirements.txt --> Python Requirements for dataflow job
└── sample-data --> Directory holding some sample data for test
```

### Prerequisites

This example assumes Project, Network, DNS and Firewalls has already been setup.

#### Export Variables
```
export PROJECT_ID=<project_id>
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
export HOST_PROJECT_ID=<HOST_PROJECT_ID>
export INPUT_BUCKET_NAME=pw-df-input-bkt
export STAGING_BUCKET_NAME=pw-df-temp-bkt
export LOCATION=us-central1
export BQ_DATASET=bqtoxmldataset
export NETWORK=shared-vpc
export SUBNET=bh-subnet-usc1
export REPO=dataflowf-image-repo
export DF_WORKER_SA=dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com
```

#### Setup IAM
```
# Create service account for dataflow workers and launchers
gcloud iam service-accounts create dataflow-worker-sa --project=$PROJECT_ID

# Assign dataflow worker permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/dataflow.worker

# Assign Object viewer permissions in order to read the data from cloud storage
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/storage.objectViewer

# Assign Object viewer permissions in order to Create temp files cloud storage
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/storage.objectCreator

# Assign Service Account User permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/iam.serviceAccountUser

# Assign BigQuery job user permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/bigquery.jobUser

# Assign bigquery data editor permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/bigquery.dataEditor

# Assign Artifactory Reader Permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com" --role roles/artifactregistry.reader

# Assign network user permissions on Host project, this is needed only if dataflow workers will be using shared VPC
gcloud projects add-iam-policy-binding $HOST_PROJECT_ID --member "serviceAccount:service-$PROJECT_NUMBER@dataflow-service-producer-prod.iam.gserviceaccount.com" --role roles/compute.networkUser
```

#### Setup Cloud Storage
```
# Create Cloud Storage bucket for input data
gcloud storage buckets create gs://$INPUT_BUCKET_NAME --location $LOCATION --project $PROJECT_ID

# Create a bucket for dataflow staging and temp locations
gcloud storage buckets create gs://$STAGING_BUCKET_NAME --location $LOCATION --project $PROJECT_ID

gsutil iam ch serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com:roles/storage.legacyBucketWriter gs://$STAGING_BUCKET_NAME

# Assign Legacy Bucket Writer Role on Input bucket in order to move the object
gsutil iam ch serviceAccount:dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com:roles/storage.legacyBucketWriter gs://$INPUT_BUCKET_NAME
```

#### Create BQ Dataset
```
bq --location=$LOCATION mk --dataset $PROJECT_ID:$BQ_DATASET
```

#### Create ARtifactory registry
```
gcloud  artifacts  repositories create $REPO --location $LOCATION --repository-format docker --project $PROJECT_ID
```

### Build Templates

#### Build and Push Docker Images for template
```
# Build Base Image, all packages will be used from this image when dataflow job runs
docker build -t $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/dataflow-2.40-base:dev -f Dockerfile_SDK .

# Build Image used by launcher to launch the Dataflow job
docker build -t $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/df-xml-to-bq:dev -f Dockerfile_Launcher .

# Push both the images to repo
docker push $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/dataflow-2.40-base:dev
docker push $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/df-xml-to-bq:dev
```

#### Build Dataflow flex template
```
gcloud dataflow flex-template build gs://$INPUT_BUCKET_NAME/dataflow-templates/xml-to-bq.json \
--image "$LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/df-xml-to-bq:dev" \
--sdk-language "PYTHON" \
--metadata-file metadata.json \
--project $PROJECT_ID
```

### Demo

#### Upload Sample Data
```
gcloud storage cp ./sample-data/*.xml gs://$INPUT_BUCKET_NAME/data/
```

#### Run job using DirectRunner locally
```
# Install Reuirements
pip3 install -r requirements.txt requirements-test.txt

cd df-package

# Run tests
python3 -m pytest

# Run Job
python3 main.py --input=../sample-data/customer-orders.xml \
--temp_location=gs://$STAGING_BUCKET_NAME/tmp \
--staging_location=gs://$STAGING_BUCKET_NAME/staging \
--output=$PROJECT_ID:$BQ_DATASET \
--dead_letter_dir=../dead/ \
--runner=DirectRunner

cd ../
```

#### Run Job using Gcloud Command
```
gcloud dataflow flex-template run xml-to-bq-sample-pipeline-$(date '+%Y-%m-%d-%H-%M-%S') \
--template-file-gcs-location gs://$INPUT_BUCKET_NAME/dataflow-templates/xml-to-bq.json \
--additional-experiments use_runner_v2 \
--additional-experiments=use_network_tags_for_flex_templates="dataflow-worker;allow-iap-ssh" \
--additional-experiments=use_network_tags="dataflow-worker;allow-iap-ssh" \
--additional-experiments=use_unsupported_python_version \
--disable-public-ips \
--network projects/$HOST_PROJECT_ID/global/networks/$NETWORK \
--subnetwork https://www.googleapis.com/compute/v1/projects/$HOST_PROJECT_ID/regions/$LOCATION/subnetworks/$SUBNET \
--service-account-email dataflow-worker-sa@$PROJECT_ID.iam.gserviceaccount.com \
--staging-location gs://$STAGING_BUCKET_NAME/staging \
--temp-location gs://$STAGING_BUCKET_NAME/tmp \
--region $LOCATION --worker-region=$LOCATION \
--parameters output=$PROJECT_ID:$BQ_DATASET \
--parameters input=gs://$INPUT_BUCKET_NAME/data/* \
--parameters dead_letter_dir=gs://$INPUT_BUCKET_NAME/invalid_files \
--parameters sdk_location=container \
--parameters sdk_container_image=$LOCATION-docker.pkg.dev/$PROJECT_ID/$REPO/dataflow-2.40-base:dev \
--project $PROJECT_ID
```

### CI/CD using Cloudbuild

#### Build Docker Image and Template with Cloud build
The below section uses gcloud command. In Real World scenario Cloud build triggers
can be created ahich can run this build job whenever their is change in the code.
```

# Build and push Base Image
gcloud builds submit --config cloudbuild_base.yaml . --project $PROJECT_ID --substitutions _LOCATION=$LOCATION,_PROJECT_ID=$PROJECT_ID,_REPOSITORY=$REPO

# Build and push launcher image and create flex template
gcloud builds submit --config cloudbuild_df_job.yaml . --project $PROJECT_ID --substitutions _LOCATION=$LOCATION,_PROJECT_ID=$PROJECT_ID,_REPOSITORY=$REPO,_TEMPLATE_PATH=gs://$INPUT_BUCKET_NAME/dataflow-templates
```

### Run Dataflow Flex template job from Composer Dags

#### Set Environment Variables for composer
```
export COMPOSER_ENV_NAME=<composer-env-name>
export COMPOSER_REGION=$LOCATION

COMPOSER_VAR_FILE=composer_variables.json
if [ ! -f "${COMPOSER_VAR_FILE}" ]; then
    envsubst < composer_variables.template > ${COMPOSER_VAR_FILE}
fi

gcloud composer environments storage data import \
--environment ${COMPOSER_ENV_NAME} \
--location ${COMPOSER_REGION} \
--source ${COMPOSER_VAR_FILE}

gcloud composer environments run \
${COMPOSER_ENV_NAME} \
--location ${COMPOSER_REGION} \
variables import -- /home/airflow/gcs/data/${COMPOSER_VAR_FILE}
```

#### Assign permissions to Composer Worker SA
```
COMPOSER_SA=$(gcloud composer environments describe  $COMPOSER_ENV_NAME --location $COMPOSER_REGION --project $PROJECT_ID --format json | jq '.config.nodeConfig.serviceAccount')

# Assign Service Account User permissions
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:$COMPOSER_SA" --role roles/iam.serviceAccountUser
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:$COMPOSER_SA" --role roles/dataflow.admin
```

#### Upload the dag to composer's dag bucket
```
DAG_PATH=$(gcloud composer environments describe $COMPOSER_ENV_NAME --location $COMPOSER_REGION --project $PROJECT_ID --format json | jq '.config.dagGcsPrefix')
gcloud storage cp dag/xml-to-bq-dag.py $DAG_PATH
```

### Limitations

Currently this pipleine loads the whole XML file into memory
for the conversion to dict via xmltodict. This approach works for small
files but is not parallelizable on super large XML files as they are not
read in chunks but in one go. This risks having a single worker dealing
with very large file instances (slow) and running potentially out of memmory.
In our experience any XML file above ~ 300mb would start slowing down the
pipeline considerably and potentially memory failures can start showing up at ~ 500mb.
This is if you go with the default worker.

**Contributors:** @singhpradeepk, @kkulczak, @akolkiewicz

**Credit:**

Sample data has been borrowed from https://learn.microsoft.com/en-in/dotnet/standard/linq/sample-xml-file-customers-orders-namespace#customersordersinnamespacexml

Data Model has been borrowed from https://learn.microsoft.com/en-in/dotnet/standard/linq/sample-xsd-file-customers-orders#customersordersxsd

Dataflow Python Flex Template  This example contains a sample Dataflow job which reads a XML file and inserts the records to BQ table  It explains how to create Flex template and run it in a restricted environment where there is no  internet connectivity to dataflow launcher or worker nodes  It also run dataflow template on shared VPC  Example also contains a DAG which can be used to trigger Dataflow job from composer  It also demonstrates how we can use cloudbuild to implement CI CD for this dataflow job        Resorces structure  Below Tree explains the purpose of each file in the folder       dataflow flex python      cloudbuild base yaml       Cloudbuild config to build SDK image     cloudbuild df job yaml     Cloudbuild config to build Launcher image and Flex template     composer variables template     Definition of All Composer variables used by DAG     dag         xml to bq dag py     Dag code to launch Dataflow Job     df package      Dataflow template package         corder             bq schema py     BQ Table Schemas             models py     Data Model for input data  generated by xsdata and pydantic plugin             customer orders py     Dataflow pipeline Implementation             customer orders test py     pytest for Dataflow pipeline code               init   py         main py     Used by launcher to launch the pipeline         setup py     Used to Install the package     Dockerfile Launcher     Dockerfile to create Launcher Image     Dockerfile SDK     Dockerfile to create SDK image     metadata json     metadata file used during building flex teamplate     README md     requirements test txt     Python Requirements for running tests     requirements txt     Python Requirements for dataflow job     sample data     Directory holding some sample data for test          Prerequisites  This example assumes Project  Network  DNS and Firewalls has already been setup        Export Variables     export PROJECT ID  project id  export PROJECT NUMBER   gcloud projects describe  PROJECT ID   format  value projectNumber    export HOST PROJECT ID  HOST PROJECT ID  export INPUT BUCKET NAME pw df input bkt export STAGING BUCKET NAME pw df temp bkt export LOCATION us central1 export BQ DATASET bqtoxmldataset export NETWORK shared vpc export SUBNET bh subnet usc1 export REPO dataflowf image repo export DF WORKER SA dataflow worker sa  PROJECT ID iam gserviceaccount com           Setup IAM       Create service account for dataflow workers and launchers gcloud iam service accounts create dataflow worker sa   project  PROJECT ID    Assign dataflow worker permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles dataflow worker    Assign Object viewer permissions in order to read the data from cloud storage gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles storage objectViewer    Assign Object viewer permissions in order to Create temp files cloud storage gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles storage objectCreator    Assign Service Account User permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles iam serviceAccountUser    Assign BigQuery job user permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles bigquery jobUser    Assign bigquery data editor permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles bigquery dataEditor    Assign Artifactory Reader Permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com    role roles artifactregistry reader    Assign network user permissions on Host project  this is needed only if dataflow workers will be using shared VPC gcloud projects add iam policy binding  HOST PROJECT ID   member  serviceAccount service  PROJECT NUMBER dataflow service producer prod iam gserviceaccount com    role roles compute networkUser           Setup Cloud Storage       Create Cloud Storage bucket for input data gcloud storage buckets create gs    INPUT BUCKET NAME   location  LOCATION   project  PROJECT ID    Create a bucket for dataflow staging and temp locations gcloud storage buckets create gs    STAGING BUCKET NAME   location  LOCATION   project  PROJECT ID  gsutil iam ch serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com roles storage legacyBucketWriter gs    STAGING BUCKET NAME    Assign Legacy Bucket Writer Role on Input bucket in order to move the object gsutil iam ch serviceAccount dataflow worker sa  PROJECT ID iam gserviceaccount com roles storage legacyBucketWriter gs    INPUT BUCKET NAME           Create BQ Dataset     bq   location  LOCATION mk   dataset  PROJECT ID  BQ DATASET           Create ARtifactory registry     gcloud  artifacts  repositories create  REPO   location  LOCATION   repository format docker   project  PROJECT ID          Build Templates       Build and Push Docker Images for template       Build Base Image  all packages will be used from this image when dataflow job runs docker build  t  LOCATION docker pkg dev  PROJECT ID  REPO dataflow 2 40 base dev  f Dockerfile SDK      Build Image used by launcher to launch the Dataflow job docker build  t  LOCATION docker pkg dev  PROJECT ID  REPO df xml to bq dev  f Dockerfile Launcher      Push both the images to repo docker push  LOCATION docker pkg dev  PROJECT ID  REPO dataflow 2 40 base dev docker push  LOCATION docker pkg dev  PROJECT ID  REPO df xml to bq dev           Build Dataflow flex template     gcloud dataflow flex template build gs    INPUT BUCKET NAME dataflow templates xml to bq json     image   LOCATION docker pkg dev  PROJECT ID  REPO df xml to bq dev      sdk language  PYTHON      metadata file metadata json     project  PROJECT ID          Demo       Upload Sample Data     gcloud storage cp   sample data   xml gs    INPUT BUCKET NAME data            Run job using DirectRunner locally       Install Reuirements pip3 install  r requirements txt requirements test txt  cd df package    Run tests python3  m pytest    Run Job python3 main py   input    sample data customer orders xml     temp location gs    STAGING BUCKET NAME tmp     staging location gs    STAGING BUCKET NAME staging     output  PROJECT ID  BQ DATASET     dead letter dir    dead      runner DirectRunner  cd               Run Job using Gcloud Command     gcloud dataflow flex template run xml to bq sample pipeline   date    Y  m  d  H  M  S       template file gcs location gs    INPUT BUCKET NAME dataflow templates xml to bq json     additional experiments use runner v2     additional experiments use network tags for flex templates  dataflow worker allow iap ssh      additional experiments use network tags  dataflow worker allow iap ssh      additional experiments use unsupported python version     disable public ips     network projects  HOST PROJECT ID global networks  NETWORK     subnetwork https   www googleapis com compute v1 projects  HOST PROJECT ID regions  LOCATION subnetworks  SUBNET     service account email dataflow worker sa  PROJECT ID iam gserviceaccount com     staging location gs    STAGING BUCKET NAME staging     temp location gs    STAGING BUCKET NAME tmp     region  LOCATION   worker region  LOCATION     parameters output  PROJECT ID  BQ DATASET     parameters input gs    INPUT BUCKET NAME data       parameters dead letter dir gs    INPUT BUCKET NAME invalid files     parameters sdk location container     parameters sdk container image  LOCATION docker pkg dev  PROJECT ID  REPO dataflow 2 40 base dev     project  PROJECT ID          CI CD using Cloudbuild       Build Docker Image and Template with Cloud build The below section uses gcloud command  In Real World scenario Cloud build triggers can be created ahich can run this build job whenever their is change in the code         Build and push Base Image gcloud builds submit   config cloudbuild base yaml     project  PROJECT ID   substitutions  LOCATION  LOCATION  PROJECT ID  PROJECT ID  REPOSITORY  REPO    Build and push launcher image and create flex template gcloud builds submit   config cloudbuild df job yaml     project  PROJECT ID   substitutions  LOCATION  LOCATION  PROJECT ID  PROJECT ID  REPOSITORY  REPO  TEMPLATE PATH gs    INPUT BUCKET NAME dataflow templates          Run Dataflow Flex template job from Composer Dags       Set Environment Variables for composer     export COMPOSER ENV NAME  composer env name  export COMPOSER REGION  LOCATION   COMPOSER VAR FILE composer variables json if      f    COMPOSER VAR FILE      then     envsubst   composer variables template     COMPOSER VAR FILE  fi  gcloud composer environments storage data import     environment   COMPOSER ENV NAME      location   COMPOSER REGION      source   COMPOSER VAR FILE   gcloud composer environments run     COMPOSER ENV NAME      location   COMPOSER REGION    variables import     home airflow gcs data   COMPOSER VAR FILE            Assign permissions to Composer Worker SA     COMPOSER SA   gcloud composer environments describe   COMPOSER ENV NAME   location  COMPOSER REGION   project  PROJECT ID   format json   jq   config nodeConfig serviceAccount      Assign Service Account User permissions gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount  COMPOSER SA    role roles iam serviceAccountUser gcloud projects add iam policy binding  PROJECT ID   member  serviceAccount  COMPOSER SA    role roles dataflow admin           Upload the dag to composer s dag bucket     DAG PATH   gcloud composer environments describe  COMPOSER ENV NAME   location  COMPOSER REGION   project  PROJECT ID   format json   jq   config dagGcsPrefix   gcloud storage cp dag xml to bq dag py  DAG PATH          Limitations  Currently this pipleine loads the whole XML file into memory for the conversion to dict via xmltodict  This approach works for small files but is not parallelizable on super large XML files as they are not read in chunks but in one go  This risks having a single worker dealing with very large file instances  slow  and running potentially out of memmory  In our experience any XML file above   300mb would start slowing down the pipeline considerably and potentially memory failures can start showing up at   500mb  This is if you go with the default worker     Contributors     singhpradeepk   kkulczak   akolkiewicz    Credit     Sample data has been borrowed from https   learn microsoft com en in dotnet standard linq sample xml file customers orders namespace customersordersinnamespacexml  Data Model has been borrowed from https   learn microsoft com en in dotnet standard linq sample xsd file customers orders customersordersxsd


GCP Google Cloud Storage Google Speech to Text representation for any use or purpose Your use of it is subject to your agreement with Google Technology Stack Copyright 2023 Google This software is provided as is without warranty or Google Cloud Run Google Artifact Registry

Answer:
```
Copyright 2023 Google. This software is provided as-is, without warranty or
representation for any use or purpose. Your use of it is subject to your
agreement with Google.
```
## Technology Stack
- Google Cloud Run
- Google Artifact Registry
- Google Cloud Storage
- Google Speech to Text
- Vertex AI Conversation
- Dialogflow CX
- Dialogflow CX Agent
- Google Data Store
- Google Secret Manager
- Gradio

## GCP Project Setup

### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
2. In the drop-down menu at the top, select **NEW PROJECT**.
3. Give your project a name.
4. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

### Enabling billing for your project.

If you haven't already enabled billing for your project, [enable
billing][enable-billing] now. Enabling billing allows is required to use Cloud
Bigtable and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud SDK, [install the Google
Cloud SDK][cloud-sdk] now. The SDK contains tools and libraries that enable you
to create and manage resources on Google Cloud Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

### Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
   gcloud init
```

Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
command:

```
    gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing

[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

## Upload your data to a Cloud Storage bucket
Follow these [instructions][instructions] to upload your pdf documents
or pdf manuals to be used in this example

[instructions]:https://cloud.google.com/storage/docs/uploading-objects

## Create a Generative AI Agent
Follow the instructions at this [link][link] and perform the following:
1. Create Data Stores: Select information that you would like the Vertex AI Search and Conversation to query
2. Create an Agent: Create the Dialogflow CX agent that queries the Data Store
3. Test the agent in the simulator
4. Take note of you agent link by going to [Dialogflow CX Console][Dialogflow CX Console] and see the information about the agent you created

[link]: https://cloud.google.com/generative-ai-app-builder/docs/a
[Dialogflow CX Console]:https://cloud.google.com/dialogflow/cx/docs/concept/console#agent

### Dialogflow CX Agent Data Stores
Data Stores are used to find answers for end-user's questions.
Data Stores are a collection documents, each of which reference your data.

For this particular example data store will consist of the following characteristics:
1. Your organizational documents or manuals.
2. The data store type will be unstructured in a pdf format
3. The data is uploaded without metadata for simplicity.
Only need to point the import to the gcp bucket folder where the pdf files are.
Their extension will decide their type.

When an end-user asks the agent a question, the agent searches for an answer from the
given source content and summarizes the findings into a coherent agent response.
It also provides supporting links to the sources of the response for the end-user to learn more.

Copyright 2023 Google  This software is provided as is  without warranty or representation for any use or purpose  Your use of it is subject to your agreement with Google         Technology Stack   Google Cloud Run   Google Artifact Registry   Google Cloud Storage   Google Speech to Text   Vertex AI Conversation   Dialogflow CX   Dialogflow CX Agent   Google Data Store   Google Secret Manager   Gradio     GCP Project Setup      Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   2  In the drop down menu at the top  select   NEW PROJECT    3  Give your project a name  4  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com       Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now  Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings      Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk       Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command          gcloud init      Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command           gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing   application default credentials   https   developers google com identity protocols application default credentials     Upload your data to a Cloud Storage bucket Follow these  instructions  instructions  to upload your pdf documents  or pdf manuals to be used in this example   instructions  https   cloud google com storage docs uploading objects      Create a Generative AI Agent Follow the instructions at this  link  link  and perform the following  1  Create Data Stores  Select information that you would like the Vertex AI Search and Conversation to query 2  Create an Agent  Create the Dialogflow CX agent that queries the Data Store 3  Test the agent in the simulator 4  Take note of you agent link by going to  Dialogflow CX Console  Dialogflow CX Console  and see the information about the agent you created   link   https   cloud google com generative ai app builder docs a  Dialogflow CX Console  https   cloud google com dialogflow cx docs concept console agent      Dialogflow CX Agent Data Stores Data Stores are used to find answers for end user s questions   Data Stores are a collection documents  each of which reference your data   For this particular example data store will consist of the following characteristics  1  Your organizational documents or manuals    2  The data store type will be unstructured in a pdf format 3  The data is uploaded without metadata for simplicity  Only need to point the import to the gcp bucket folder where the pdf files are   Their extension will decide their type   When an end user asks the agent a question  the agent searches for an answer from the  given source content and summarizes the findings into a coherent agent response   It also provides supporting links to the sources of the response for the end user to learn more