GCP BigQuery Benchmark Repos While Google provides some high level guidelines on BigQuery performance in these scenarios we don t provide consistent metrics on how the above factors can impact Customers new to BigQuery often have questions on how to best utilize the platform with regards to performance For example a common question which has routinely resurfaced in this area is the performance of file loads for efficient load times As a second example when informing customers that queries run on external data sources are less efficient than those run on BigQuery managed tables customers have followed up asking exactly how much less efficient queries on external sources are into BigQuery specifically the optimal file parameters file type columns column types file size etc

Answer:
# BigQuery Benchmark Repos
Customers new to BigQuery often have questions on how to best utilize the platform with regards to performance.
For example, a common question which has routinely resurfaced in this area is the performance of file loads
into BigQuery, specifically the optimal file parameters (file type, # columns, column types, file size, etc)
for efficient load times.  As a second example, when informing customers that queries run on external data
sources are less efficient than those run on BigQuery managed tables, customers have followed up, asking
exactly how much less efficient queries on external sources are.

While Google provides some high-level guidelines on BigQuery performance in these
scenarios, we don’t provide consistent metrics on how the above factors can impact
performance. This repository seeks to create benchmarks to address questions with
quantitative data and a higher level of confidence, allowing more definitive answers
when interacting with customers.

While this repository is intended to continue growing, it currently includes the following
benchmarks:
### File Loader Benchmark
The File Loader benchmark measures the affect of file properties on performance when loading
files into BigQuery tables. Files are created using a combination of properties such as file type, compression type,
number of columns, column types (such as 100% STRING vs 50% STRING/ 50% NUMERIC), number of files,
and the size of files. Once the files are created, they are loaded into BigQuery tables.

#### Benchmark Parameters
Specific file parameters are used in this project for performance testing. While
the list of parameters is growing, the current list of parameters and values
is as follows:

**File Type**:

* Avro
* CSV
* JSON
* Parquet

**Compression**:

* gzip (for CSV and JSON)
* snappy (for AVRO)

**Number of columns**:
* 10
* 100
* 1000

**Column Types**:
* String-only
* 50% String / 50% NUMERIC
* 10% String / 90% NUMERIC

**Number of files**:
* 1
* 100
* 1000
* 10000

**Target Data Size (Size of the BigQuery staging table used to generate each file)**:
* 10MB
* 100MB
* 1GB
* 2GB

These parameters are used to create combinations of file types stored on in a
bucket on GCS. An example of a file prefix generated from the above list of file parameters is:
`fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* `.
This prefix holds 100 uncompressed CSV files, each generated from a 10 MB BigQuery
staging table with 10 string columns. The tool loads the 100 CSV files with this prefix
to a BigQuery table and records the performance to create a benchmark.

In the future, a parameter for slot type will be added with values for communal
and reserved. In addition, ORC will be added as a value for file type, and struct/
array types will be added to the values for column types.

### Federated Query Benchmark
The federated query benchmark quantifies the difference in performance between
queries on federated (external) and managed BigQuery tables. A variety of queries
ranging in complexity will be created. These queries will be run on managed BigQuery
tables and federated Google Cloud Storage files (including AVRO, CSV, JSON, and PARQUET) of
identical schemas and sizes. The files created for the File Loader Benchmark will be reused
here to run external queries on and to create BQ Managed tables with.

#### Benchmark Parameters
Parameters for this benchmark will include the type of table, type of query,
and the table properties.

**Table Type**:

* `BQ_MANAGED`: Tables located within and managed by BigQuery.
* `EXTERNAL`: Data located in GCS files, which are used to create a temporary
external table for querying.

**Query Type**:
* `SIMPLE_SELECT_*`: Select all columns and all rows.
* `SELECT_ONE_STRING`: Select the first string field in the schema. All schemas used in the
benchmark contain at least one string field.
* `SELECT_50_PERCENT`: Select the first 50% of the table's fields.

Future iterations of this benchmark will include more complex queries, such as those
that utilize joins, subqueries, window functions, etc.

Since the files created for the File Loader Benchmark will be reused for this
benchmark, both the BQ Managed tables and GCS files will share the File Loader Benchmark
parameters, with the only difference being that the snappy compression type is not
supported for federated queries and therefore will not be included for comparison.

**File Type**:

* Avro
* CSV
* JSON
* Parquet

**Compression**:

* gzip (for CSV and JSON)

**Number of columns**:
* 10
* 100
* 1000

**Column Types**:
* String-only
* 50% String / 50% NUMERIC
* 10% String / 90% NUMERIC

**Number of files**:
* 1
* 100
* 1000
* 10000

**Target Data Size (Size of the BigQuery staging table used to generate each file)**:
* 10MB
* 100MB
* 1GB
* 2GB

## Benchmark Results

#### BigQuery

The results of the benchmarks will be saved in a separate BigQuery table for
ad hoc analysis. The results table will use the following schema: [results_table_schema.json](json_schemas/results_table_schema.json)

#### DataStudio
Once the results table is populated with data, DataStudio can be used to visualize results.
See [this article](https://support.google.com/datastudio/answer/6283323?hl=en) to get started
with DataStudio.

## Usage
This project contains the tools to create the resources needed to run the benchmarks.
The main method for the project is located in
[`bq_benchmark.py`](bq_benchmark.py).

### Prepping the Benchmarks Resources from Scratch
The following steps are needed to create the resources needed for the benchmarks.
Some steps will only be needed for certain benchmarks, so feel free to skip them if you
are only focused on a certain set of benchmarks.

#### 1. Create the Results Table (Needed for all benchmarks)

If running the whole project from scratch, the first step is to create a table
in BigQuery to store the results of the benchmark loads. A json file has been
provided in the json_schemas directory ([results_table_schema.json](json_schemas/results_table_schema.json))
with the  above schema. The schema can be used to create the results
table by running the using the following command:
```
python bq_benchmark.py \
--create_results_table \
--results_table_schema_path=<optional path to json schema for results table> \
--results_table_name=<results table name> \
--results_dataset_id=<dataset ID>
```
Parameters:

`--create_results_table`: Flag to indicate that a results table should be created. It has a value of
`store_true`, so this flag will be set to False, unless it is provided in the
command.

`--results_table_schema_path`: Optional argument. It defaults to `json_schemas/results_table_schema.json`.
If using a json schema in a different location, provide the path to that
schema.

`--results_table_name`: String representing the name of the results table. Note that just the table name
is needed, not the full project_id.dataset_id.table_name indicator.

`--dataset_id`: ID of the dataset to hold the results table.

#### 2. Select File Parameters (Needed for File Loader and Federated Query Benchmarks)
File parameters are used to help create the files needed for both the File Loader Benchmark and
 the Federated Query Benchmark. They can be configured in the `FILE_PARAMETERS` dictionary in
[`generic_benchmark_tools/file_parameters.py`](generic_benchmark_tools/file_parameters.py). Currently,
no file parameters can be added to the dictionary, as this will cause errors.
However, parameters can be removed
from the dictionary if you are looking for a smaller set of file combinations.
Note that the parameter `numFiles` has to include at least the number 1 to
ensure that the subsequent number of files are properly created. This is
because the program uses this first file to make copies to create subsequent
files. This is a much faster alternative than recreating identical files.
For example, if you don't want the 1000 or 10000 as `numFile` parameters,
you can take them out, but you must leave 1 (e.g. [1, 100]). That way the first
file can be copied to create the 100 files.

#### 3. Create Schemas for the Benchmark Staging Tables (Needed for File Loader and Federated Query Benchmarks)
In order to create the files with the above parameters, the [Dataflow Data Generator
tool](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-data-generator)
from the Professional Services Examples library needs to be leveraged to create
staging tables containing combinations of `columnTypes` and `numColumns` from the
list of file parameters in [`generic_benchmark_tools/file_parameters.py`](generic_benchmark_tools/file_parameters.py). The staging tables
will later be resized to match the sizes in `targetDataSize` file parameter, and then
they will be extracted to files in GCS. However, before any of this can be done, JSON schemas for
the staging tables must be created. To do this run the following command:

```
python bq_benchmark.py \
--create_benchmark_schemas \
--benchmark_table_schemas_directory=<optional directory where schemas should be stored>
```

Parameters:

`--create_benchmark_schemas`: Flag to indicate that benchmark schemas should be created. It has a value of
`store_true`, so this flag will be set to False, unless it is provided in the
command.

`--benchmark_table_schemas_directory`: Optional argument for the directory where
the schemas for the staging tables are to be stored. It defaults to `json_schemas/benchmark_table_schemas`.
If you would prefer that the schemas are written to a different directory, provide that directory.

#### 4. Create Staging Tables (Needed for File Loader and Federated Query Benchmarks)
Once the schemas are created for the staging tables, the staging tables themselves can be
created. This is a two step process.

First, a set of staging tables are created using the data_generator_pipeline
module in the [Dataflow Data Generator
tool](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-data-generator) using
the schemas created in step 3. One staging table is created for each combination of columnTypes and
numColumns file parameters. A small number of rows are created in each staging table (500 rows) to
get the process started. Once the tables are created, they are saved in a staging dataset. The names of
staging tables are generated using their respective columnTypes and numColumms parameters.
For example, a staging table created using the 100_STRING `columnTypes` param and 10
`numColumns` would be named `100_STRING_10`.

Second, each staging table is used to create resized staging tables to match the sizes in the `targetDataSizes` parameter.
This is accomplished using the [bq_table_resizer module](https://github.com/GoogleCloudPlatform/professional-services/blob/master/examples/dataflow-data-generator/bigquery-scripts/bq_table_resizer.py)
 of the [Dataflow Data Generator
tool](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-data-generator).
The resized staging tables are saved in a second staging dataset ws). The names of
resized staging tables are generated using the name of the staging table they were
resized from, plus the `targetDataSizes` param. For example, the `100_STRING_10` staging table
from above will be used to create the following four tables in the
resized staging dataset: `100_STRING_10_10MB`, `100_STRING_10_100MB`, `100_STRING_10_1GB`,
`100_STRING_10_2GB`.

To run the process of creating staging and resized staging tables, run the following
command:
```
python bq_benchmark.py \
--create_staging_tables \
--bq_project_id=<ID of project holding BigQuery resources> \
--staging_dataset_id=<ID of dataset holding staging tables> \
--resized_staging_dataset_id=<ID of dataset holding resized staging tables> \
--benchmark_table_schemas_directory=<optional directory where staging table schemas are stored> \
--dataflow_staging_location=<path on GCS to serve as staging location for Dataflow> \
--dataflow_temp_location=<path on GCS to serve as temp location for Dataflow>
```

Parameters:

`--create_staging_tables`: Flag to indicate that staging and resized staging
tables should be created. It has a value of `store_true`, so this flag will be
set to False, unless it is provided in the
command.

`--bq_project_id`: The ID of the project that will hold the BigQuery resources for
the benchmark, including all datasets, results tables, staging tables, and
benchmark tables.

`--staging_dataset_id`: The ID of the dataset that will hold the first set of staging
tables. For the tool to work correctly, the `staging_dataset_id` must only contain
staging tables, and it must be different than the `--resized_staging_dataset_id`.
Do not store tables for any other purposes in this dataset.

`--resized_staging_dataset_id`: The ID of the dataset that will hold the resized staging
tables. For the tool to work correctly, the `resized_staging_dataset_id` must only contain
resized staging tables, and it must be different than the `--staging_dataset_id`.
Do not store tables for any other purposes in this dataset.

`--benchmark_table_schemas_directory`: Optional argument for the directory where
the schemas for the staging tables are stored. It defaults to
`json_schemas/benchmark_table_schemas`. If your schemas are elsewhere, provide
that directory.

`--dataflow_staging_location`: Staging location for Dataflow on GCS. Include
the 'gs://' prefix, the name of the bucket you want to use, and any prefix. For example
'`gs://<bucket_name>/staging`. Note: be sure to use a different bucket than the one
provided in the --bucket_name parameter used below with the `--create_files` and
-`-create_benchmark` tables flags.

`--dataflow_temp_location`: Temp location for Dataflow on GCS. Include
the 'gs://' prefix, the name of the bucket you want to use, and any prefix. For example
'`gs://<bucket_name>/temp`. Note: be sure to use a different bucket than the one
provided in the --bucket_name parameter used below with the `--create_files` and
-`-create_benchmark tables flags`.

#### 5. Create Files (Needed for File Loader and Federated Query Benchmarks)
Once the resized staging tables are created, the next step is to use the resized
staging tables to create the files on GCS. The resized staging tables already contain
combinations of the `columnTypes`, `numColumns`, and `targetDataSize` parameters. Now
each of the resized staging tables must be extracted to combinations of files
generated from the fileType and compression parameters. In each combination,
the extraction is only done for the first file (`numFiles`=1). For example,
the resized staging table `100_STRING_10_10MB` must be use to create the following
files on GCS:

* fileType=avro/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.avro
* fileType=avro/compression=snappy/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.snappy
* fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.csv
* fileType=csv/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.gzip
* fileType=json/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.json
* fileType=json/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.gzip
* fileType=parquet/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1/tableSize=10MB/file1.parquet

The method of extracting the resized staging table depends on the combination of parameters.
BigQuery extract jobs are used if the `fileType` is csv or json, or if the `fileType` is avro and
the resized staging table size is <= 1 GB. If the `fileType` is avro and the `targetDataSize`
is > 1 GB, DataFlow is used to generate the file, since attempting to extract a staging table
of this size to avro causes errors. If the `fileType` is parquet, DataFlow is used as well,
since BigQuery extract jobs don't support the parquet file type.

Once the first file for each combination is generated (`numFiles`=1), it is copied
to create the same combination of files, but where numFiles > 1. More specifically,
it is copied 100 times for `numFiles`=100, 1000 times for `numFiles`=1000, and
10000 times for `numFiles`=10000. Copying is much faster than extracting each
table tens of thousands of times. As an example, the files listed above are
copied to create the following 77,700 files:

* fileType=avro/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.avro- file100.avro)
* fileType=avro/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.avro - file1000.avro)
* fileType=avro/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.avro - file10000.avro)
* fileType=avro/compression=snappy/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.snappy- file100.snappy)
* fileType=avro/compression=snappy/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.snappy - file1000.snappy)
* fileType=avro/compression=snappy/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.snappy - file10000.snappy)
* fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.csv - file100.csv)
* fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.csv - file1000.csv)
* fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.csv - file10000.csv)
* fileType=csv/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.gzip - file100.gzip)
* fileType=csv/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.gzip - file1000.gzip)
* fileType=csv/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.gzip - file10000.gzip)
* fileType=json/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.json - file100.json)
* fileType=json/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.json - file1000.json)
* fileType=json/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.json - file10000.json)
* fileType=json/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.gzip - file100.gzip)
* fileType=json/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.gzip - file1000.gzip)
* fileType=json/compression=gzip/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.gzip - file10000.gzip)
* fileType=parquet/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=100/tableSize=10MB/* (contains file1.parquet- file100.parquet)
* fileType=parquet/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/* (contains file1.parquet - file1000.parquet)
* fileType=parquet/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=10000/tableSize=10MB/* (contains file1.parquet - file10000.parquet)

To complete the process of creating files, run the following command:
```
python bq_benchmark.py \
--create_files \
--gcs_project_id=<ID of project holding GCS resources> \
--resized_staging_dataset_id=<ID of dataset holding resized staging tables> \
--bucket_name=<name of bucket to hold files> \
--dataflow_staging_location=<path on GCS to serve as staging location for Dataflow> \
--dataflow_temp_location=<path on GCS to serve as temp location for Dataflow> \
--restart_file=<optional file name to restart with if program is stopped> \

```

Parameters:

`--create_files`: Flag to indicate that files should be created and stored on GCS.
It has a value of `store_true`, so this flag will be
set to False, unless it is provided in the
command.

`--gcs_project_id`: The ID of the project that will hold the GCS resources for
the benchmark, including all files and the bucket that holds them.

`--resized_staging_dataset_id`: The ID of the dataset that holds the resized
staging tables generated using the `--create_staging_tables` command.

`--bucket_name`: Name of the bucket that will hold the created files. Note that
the only purpose of this bucket should be to hold the created files, and that files
used for any other reason should be stored in a different bucket.

`--dataflow_staging_location`: Staging location for Dataflow on GCS. Include
the 'gs://' prefix, the name of the bucket you want to use, and any prefix. For example
`gs://<bucket_name>/staging`. Note: be sure to use a different bucket than the one
provided in the `--bucket_name parameter`.

`--dataflow_temp_location`: Temp location for Dataflow on GCS. Include
the 'gs://' prefix, the name of the bucket you want to use, and any prefix. For example
`gs://<bucket_name>/temp`. Note: be sure to use a different bucket than the one
provided in the `--bucket_name parameter`.

`--restart_file`: Optional file name to start the file creation process with. Creating
each file combination can take hours, and often a backend error or a timeout will
occur, preventing all the files from being created. If this happens, copy the last file
that was successfully created from the logs and use it here. It should start with `fileType=`
and end with the file extension. For example,
`fileType=csv/compression=none/numColumns=10/columnTypes=100_STRING/numFiles=1000/tableSize=10MB/file324.csv`

### Running the benchmarks

#### File Loader Benchmark
Once the files are created, the File Loader Benchmark can be run. As a prerequisite for this step, a log sink in BigQuery that captures logs
about BigQuery must be set up in the same project that holds the benchmark
tables. If a BigQuery log sink is not already set up, follow [these steps](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/bigquery-audit-log#1-getting-the-bigquery-log-data).

Note that this benchmark will delete tables after recording information on load time. Before the
tables are deleted, the tables and their respective files can be used to run the Federated Query Benchmark. If
running the two benchmarks independently, each file will be used to create a BigQuery table two different times. Running the two benchmarks
at the same time can save time if results for both benchmarks are desired. In this case, the `--include_federated_query_benchmark` flag can
be added to the below command. Be aware that running the queries will add significant time to the benchmark run, so leave the
flag out of the command if the primary goal is to obtain results for the File Loader Benchmark.

To run the benchmark, use the following command:

```
python bq_benchmark.py \
--run_file_loader_benchmark \
--bq_project_id=<ID of the project holding the BigQuery resources> \
--gcs_project_id=<ID of project holding GCS resources> \
--staging_project_id=<ID of project holding staging tables> \
--staging_dataset_id=<ID of dataset holding staging tables> \
--benchmark_dataset_id=<ID of the dataset holding the benchmark tables> \
--bucket_name=<name of bucket to hold files> \
--results_table_name=<Name of results table> \
--results_dataset_id=<Name dataset holding results table> \
--duplicate_benchmark_tables \
--bq_logs_dataset=<Name of dataset hold BQ logs table>
--include_federated_query_benchmark

```

Parameters:
`--run_file_loader_benchmark`: Flag to initiate process of running the File Loader Benchmark by creating tables from files and storing results for comparison.
It has a value of `store_true`, so this flag will be
set to False, unless it is provided in the
command.

`--gcs_project_id`: The ID of the project that will hold the GCS resources for
the benchmark, including all files and the bucket that holds them.

`--bq_project_id`: The ID of the project that will hold the BigQuery resources for
the benchmark, including all datasets, results tables, staging tables, and
benchmark tables.

`--staging_project_id`: The ID of the project that holds the first set of staging
tables. While this will be the same as the `--bq_project_id` if running the project
from scratch, it will differ from `--bq_project_id` if you are using file combinations
that have already been created and running benchmarks/saving results in your own project.

`--staging_dataset_id`: The ID of the dataset that will hold the first set of staging
tables. For the tool to work correctly, the `staging_dataset_id` must only contain
staging tables, and it must be different than the `--resized_staging_dataset_id`.
Do not store tables for any other purposes in this dataset.

`--dataset_id`: The ID of the dataset that will hold the benchmark tables.

`--bucket_name`: Name of the bucket that will hold the file combinations to be
loaded into benchmark tables. Note that the only purpose of this bucket should
be to hold the file combinations, and that files used for any other reason
should be stored in a different bucket.

`--results_table_name`: Name of the results table to hold relevant information
about the benchmark loads.

`--results_dataset_id`: Name of the dataset that holds the results table.

`--duplicate_benchmark_tables`: Flag to indicate that a benchmark table should be
created for a given file combination, even if that file combination has a benchmark
table already. Creating multiple benchmark tables for each file combination can
increase the accuracy of the average runtimes calculated from the results. If
this behavior is desired, include the flag. However, if you want to ensure that you
first have at least one benchmark table for each file combination, then leave the
flag off. In that case, the benchmark creation process will skip a file combination
if it already has a benchmark table.

`--bq_logs_dataset`: Name of dataset hold BQ logs table. This dataset must be
in project used for `--bq_project_id`.

`--include_federated_query_benchmark`: Flag to indicate that the Federated Query Benchmark should
be run on the created tables and the files the tables were loaded from before
the tables are deleted. If results for both benchmarks are desired, this will save time
when compared to running each benchmark independently, since the same tables needed for the
File Loader Benchmark are needed for the Federated Query Benchmark. It has a value of `store_true`,
so this flag will be set to False, unless it is provided in the
command.

#### Federated Query Benchmark
Once the files are created, the Federated Query Benchmark can be run. As a prerequisite for this step, a log sink in BigQuery that captures logs
about BigQuery must be set up in the same project that holds the benchmark
tables. If a BigQuery log sink is not already set up, follow [these steps](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/bigquery-audit-log#1-getting-the-bigquery-log-data).

As mentioned above, the Federated Query Benchmark can be run while running the File Loader Benchmark in addition to using
the command below. Note, though , that running federated queries on
snappy compressed files is not supported. When the File Loader Benchmark encounters a snappy compressed file, it still
loads the file into a BigQuery table to capture load results, but it will skip the Federated Query portion. When the Federated
Query Benchmark encounters a snappy compressed file, it will skip the load all together. Therefore, if obtaining Federated
Query Benchmark results is the primary goal, use the command below.

It should also be noted that since the Federated Query Benchmark loads files into tables, the load results for the File
Loader Benchmark will also be captured. This will not add significant time to the benchmark run since the tables have to
be loaded regardless.

To run the benchmark, use the following command:

```
python bq_benchmark.py \
--run_federated_query_benchmark \
--bq_project_id=<ID of the project holding the BigQuery resources> \
--gcs_project_id=<ID of project holding GCS resources> \
--staging_project_id=<ID of project holding staging tables> \
--staging_dataset_id=<ID of dataset holding staging tables> \
--benchmark_dataset_id=<ID of the dataset holding the benchmark tables> \
--bucket_name=<name of bucket to hold files> \
--results_table_name=<Name of results table> \
--results_dataset_id=<Name dataset holding results table> \
--bq_logs_dataset=<Name of dataset hold BQ logs table>

```

Parameters:
`--run_federated_query_benchmark`: Flag to initiate the process running the Federated Query
Benchmark by creating tables from files, running queries on both
the table and the files, and storing performance results.
It has a value of `store_true`, so this flag will be
set to False, unless it is provided in the
command.

`--gcs_project_id`: The ID of the project that will hold the GCS resources for
the benchmark, including all files and the bucket that holds them.

`--bq_project_id`: The ID of the project that will hold the BigQuery resources for
the benchmark, including all datasets, results tables, staging tables, and
benchmark tables.

`--staging_project_id`: The ID of the project that holds the first set of staging
tables. While this will be the same as the `--bq_project_id` if running the project
from scratch, it will differ from `--bq_project_id` if you are using file combinations
that have already been created and running benchmarks/saving results in your own project.

`--staging_dataset_id`: The ID of the dataset that will hold the first set of staging
tables. For the tool to work correctly, the `staging_dataset_id` must only contain
staging tables, and it must be different than the `--resized_staging_dataset_id`.
Do not store tables for any other purposes in this dataset.

`--dataset_id`: The ID of the dataset that will hold the benchmark tables.

`--bucket_name`: Name of the bucket that will hold the file combinations to be
loaded into benchmark tables. Note that the only purpose of this bucket should
be to hold the file combinations, and that files used for any other reason
should be stored in a different bucket.

`--results_table_name`: Name of the results table to hold relevant information
about the benchmark loads.

`--results_dataset_id`: Name of the dataset that holds the results table.

`--bq_logs_dataset`: Name of dataset hold BQ logs table. This dataset must be
in project used for `--bq_project_id`.

## Testing

Tests can be run by running the following command in the bq_file_load_benchmark
directory:

```
python -m pytest --project_id=<ID of project that will hold test resources>

```

Note that the tests will create and destroy resources in the project denoted
by `--project_id`.

BigQuery Benchmark Repos Customers new to BigQuery often have questions on how to best utilize the platform with regards to performance  For example  a common question which has routinely resurfaced in this area is the performance of file loads into BigQuery  specifically the optimal file parameters  file type    columns  column types  file size  etc  for efficient load times   As a second example  when informing customers that queries run on external data sources are less efficient than those run on BigQuery managed tables  customers have followed up  asking exactly how much less efficient queries on external sources are   While Google provides some high level guidelines on BigQuery performance in these scenarios  we don t provide consistent metrics on how the above factors can impact performance  This repository seeks to create benchmarks to address questions with quantitative data and a higher level of confidence  allowing more definitive answers when interacting with customers   While this repository is intended to continue growing  it currently includes the following benchmarks      File Loader Benchmark The File Loader benchmark measures the affect of file properties on performance when loading files into BigQuery tables  Files are created using a combination of properties such as file type  compression type  number of columns  column types  such as 100  STRING vs 50  STRING  50  NUMERIC   number of files  and the size of files  Once the files are created  they are loaded into BigQuery tables        Benchmark Parameters Specific file parameters are used in this project for performance testing  While the list of parameters is growing  the current list of parameters and values is as follows     File Type       Avro   CSV   JSON   Parquet    Compression       gzip  for CSV and JSON    snappy  for AVRO     Number of columns      10   100   1000    Column Types      String only   50  String   50  NUMERIC   10  String   90  NUMERIC    Number of files      1   100   1000   10000    Target Data Size  Size of the BigQuery staging table used to generate each file       10MB   100MB   1GB   2GB  These parameters are used to create combinations of file types stored on in a bucket on GCS  An example of a file prefix generated from the above list of file parameters is   fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB      This prefix holds 100 uncompressed CSV files  each generated from a 10 MB BigQuery staging table with 10 string columns  The tool loads the 100 CSV files with this prefix to a BigQuery table and records the performance to create a benchmark   In the future  a parameter for slot type will be added with values for communal and reserved  In addition  ORC will be added as a value for file type  and struct  array types will be added to the values for column types       Federated Query Benchmark The federated query benchmark quantifies the difference in performance between queries on federated  external  and managed BigQuery tables  A variety of queries ranging in complexity will be created  These queries will be run on managed BigQuery tables and federated Google Cloud Storage files  including AVRO  CSV  JSON  and PARQUET  of identical schemas and sizes  The files created for the File Loader Benchmark will be reused here to run external queries on and to create BQ Managed tables with        Benchmark Parameters Parameters for this benchmark will include the type of table  type of query  and the table properties     Table Type        BQ MANAGED   Tables located within and managed by BigQuery     EXTERNAL   Data located in GCS files  which are used to create a temporary external table for querying      Query Type       SIMPLE SELECT     Select all columns and all rows     SELECT ONE STRING   Select the first string field in the schema  All schemas used in the benchmark contain at least one string field     SELECT 50 PERCENT   Select the first 50  of the table s fields   Future iterations of this benchmark will include more complex queries  such as those that utilize joins  subqueries  window functions  etc   Since the files created for the File Loader Benchmark will be reused for this benchmark  both the BQ Managed tables and GCS files will share the File Loader Benchmark parameters  with the only difference being that the snappy compression type is not supported for federated queries and therefore will not be included for comparison     File Type       Avro   CSV   JSON   Parquet    Compression       gzip  for CSV and JSON     Number of columns      10   100   1000    Column Types      String only   50  String   50  NUMERIC   10  String   90  NUMERIC    Number of files      1   100   1000   10000    Target Data Size  Size of the BigQuery staging table used to generate each file       10MB   100MB   1GB   2GB      Benchmark Results       BigQuery  The results of the benchmarks will be saved in a separate BigQuery table for ad hoc analysis  The results table will use the following schema   results table schema json  json schemas results table schema json        DataStudio Once the results table is populated with data  DataStudio can be used to visualize results  See  this article  https   support google com datastudio answer 6283323 hl en  to get started with DataStudio      Usage This project contains the tools to create the resources needed to run the benchmarks  The main method for the project is located in   bq benchmark py   bq benchmark py        Prepping the Benchmarks Resources from Scratch The following steps are needed to create the resources needed for the benchmarks  Some steps will only be needed for certain benchmarks  so feel free to skip them if you are only focused on a certain set of benchmarks        1  Create the Results Table  Needed for all benchmarks   If running the whole project from scratch  the first step is to create a table in BigQuery to store the results of the benchmark loads  A json file has been provided in the json schemas directory   results table schema json  json schemas results table schema json   with the  above schema  The schema can be used to create the results table by running the using the following command      python bq benchmark py     create results table     results table schema path  optional path to json schema for results table      results table name  results table name      results dataset id  dataset ID      Parameters      create results table   Flag to indicate that a results table should be created  It has a value of  store true   so this flag will be set to False  unless it is provided in the command       results table schema path   Optional argument  It defaults to  json schemas results table schema json   If using a json schema in a different location  provide the path to that schema       results table name   String representing the name of the results table  Note that just the table name is needed  not the full project id dataset id table name indicator      dataset id   ID of the dataset to hold the results table        2  Select File Parameters  Needed for File Loader and Federated Query Benchmarks  File parameters are used to help create the files needed for both the File Loader Benchmark and  the Federated Query Benchmark  They can be configured in the  FILE PARAMETERS  dictionary in   generic benchmark tools file parameters py   generic benchmark tools file parameters py   Currently  no file parameters can be added to the dictionary  as this will cause errors  However  parameters can be removed from the dictionary if you are looking for a smaller set of file combinations  Note that the parameter  numFiles  has to include at least the number 1 to ensure that the subsequent number of files are properly created  This is because the program uses this first file to make copies to create subsequent files  This is a much faster alternative than recreating identical files  For example  if you don t want the 1000 or 10000 as  numFile  parameters  you can take them out  but you must leave 1  e g   1  100    That way the first file can be copied to create the 100 files          3  Create Schemas for the Benchmark Staging Tables  Needed for File Loader and Federated Query Benchmarks  In order to create the files with the above parameters  the  Dataflow Data Generator tool  https   github com GoogleCloudPlatform professional services tree master examples dataflow data generator  from the Professional Services Examples library needs to be leveraged to create staging tables containing combinations of  columnTypes  and  numColumns  from the list of file parameters in   generic benchmark tools file parameters py   generic benchmark tools file parameters py   The staging tables will later be resized to match the sizes in  targetDataSize  file parameter  and then they will be extracted to files in GCS  However  before any of this can be done  JSON schemas for the staging tables must be created  To do this run the following command       python bq benchmark py     create benchmark schemas     benchmark table schemas directory  optional directory where schemas should be stored       Parameters      create benchmark schemas   Flag to indicate that benchmark schemas should be created  It has a value of  store true   so this flag will be set to False  unless it is provided in the command      benchmark table schemas directory   Optional argument for the directory where the schemas for the staging tables are to be stored  It defaults to  json schemas benchmark table schemas   If you would prefer that the schemas are written to a different directory  provide that directory        4  Create Staging Tables  Needed for File Loader and Federated Query Benchmarks  Once the schemas are created for the staging tables  the staging tables themselves can be created  This is a two step process   First  a set of staging tables are created using the data generator pipeline module in the  Dataflow Data Generator tool  https   github com GoogleCloudPlatform professional services tree master examples dataflow data generator  using the schemas created in step 3  One staging table is created for each combination of columnTypes and numColumns file parameters  A small number of rows are created in each staging table  500 rows  to get the process started  Once the tables are created  they are saved in a staging dataset  The names of staging tables are generated using their respective columnTypes and numColumms parameters  For example  a staging table created using the 100 STRING  columnTypes  param and 10  numColumns  would be named  100 STRING 10    Second  each staging table is used to create resized staging tables to match the sizes in the  targetDataSizes  parameter  This is accomplished using the  bq table resizer module  https   github com GoogleCloudPlatform professional services blob master examples dataflow data generator bigquery scripts bq table resizer py   of the  Dataflow Data Generator tool  https   github com GoogleCloudPlatform professional services tree master examples dataflow data generator   The resized staging tables are saved in a second staging dataset ws   The names of resized staging tables are generated using the name of the staging table they were resized from  plus the  targetDataSizes  param  For example  the  100 STRING 10  staging table from above will be used to create the following four tables in the resized staging dataset   100 STRING 10 10MB    100 STRING 10 100MB    100 STRING 10 1GB    100 STRING 10 2GB    To run the process of creating staging and resized staging tables  run the following command      python bq benchmark py     create staging tables     bq project id  ID of project holding BigQuery resources      staging dataset id  ID of dataset holding staging tables      resized staging dataset id  ID of dataset holding resized staging tables      benchmark table schemas directory  optional directory where staging table schemas are stored      dataflow staging location  path on GCS to serve as staging location for Dataflow      dataflow temp location  path on GCS to serve as temp location for Dataflow       Parameters      create staging tables   Flag to indicate that staging and resized staging tables should be created  It has a value of  store true   so this flag will be set to False  unless it is provided in the command      bq project id   The ID of the project that will hold the BigQuery resources for the benchmark  including all datasets  results tables  staging tables  and benchmark tables      staging dataset id   The ID of the dataset that will hold the first set of staging tables  For the tool to work correctly  the  staging dataset id  must only contain staging tables  and it must be different than the    resized staging dataset id   Do not store tables for any other purposes in this dataset      resized staging dataset id   The ID of the dataset that will hold the resized staging tables  For the tool to work correctly  the  resized staging dataset id  must only contain resized staging tables  and it must be different than the    staging dataset id   Do not store tables for any other purposes in this dataset      benchmark table schemas directory   Optional argument for the directory where the schemas for the staging tables are stored  It defaults to  json schemas benchmark table schemas   If your schemas are elsewhere  provide that directory      dataflow staging location   Staging location for Dataflow on GCS  Include the  gs     prefix  the name of the bucket you want to use  and any prefix  For example   gs    bucket name  staging   Note  be sure to use a different bucket than the one provided in the   bucket name parameter used below with the    create files  and    create benchmark  tables flags      dataflow temp location   Temp location for Dataflow on GCS  Include the  gs     prefix  the name of the bucket you want to use  and any prefix  For example   gs    bucket name  temp   Note  be sure to use a different bucket than the one provided in the   bucket name parameter used below with the    create files  and    create benchmark tables flags         5  Create Files  Needed for File Loader and Federated Query Benchmarks  Once the resized staging tables are created  the next step is to use the resized staging tables to create the files on GCS  The resized staging tables already contain combinations of the  columnTypes    numColumns   and  targetDataSize  parameters  Now each of the resized staging tables must be extracted to combinations of files generated from the fileType and compression parameters  In each combination  the extraction is only done for the first file   numFiles  1   For example  the resized staging table  100 STRING 10 10MB  must be use to create the following files on GCS     fileType avro compression none numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 avro   fileType avro compression snappy numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 snappy   fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 csv   fileType csv compression gzip numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 gzip   fileType json compression none numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 json   fileType json compression gzip numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 gzip   fileType parquet compression none numColumns 10 columnTypes 100 STRING numFiles 1 tableSize 10MB file1 parquet   The method of extracting the resized staging table depends on the combination of parameters  BigQuery extract jobs are used if the  fileType  is csv or json  or if the  fileType  is avro and the resized staging table size is    1 GB  If the  fileType  is avro and the  targetDataSize  is   1 GB  DataFlow is used to generate the file  since attempting to extract a staging table of this size to avro causes errors  If the  fileType  is parquet  DataFlow is used as well  since BigQuery extract jobs don t support the parquet file type   Once the first file for each combination is generated   numFiles  1   it is copied to create the same combination of files  but where numFiles   1  More specifically  it is copied 100 times for  numFiles  100  1000 times for  numFiles  1000  and 10000 times for  numFiles  10000  Copying is much faster than extracting each table tens of thousands of times  As an example  the files listed above are copied to create the following 77 700 files     fileType avro compression none numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 avro  file100 avro    fileType avro compression none numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 avro   file1000 avro    fileType avro compression none numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 avro   file10000 avro    fileType avro compression snappy numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 snappy  file100 snappy    fileType avro compression snappy numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 snappy   file1000 snappy    fileType avro compression snappy numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 snappy   file10000 snappy    fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 csv   file100 csv    fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 csv   file1000 csv    fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 csv   file10000 csv    fileType csv compression gzip numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 gzip   file100 gzip    fileType csv compression gzip numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 gzip   file1000 gzip    fileType csv compression gzip numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 gzip   file10000 gzip    fileType json compression none numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 json   file100 json    fileType json compression none numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 json   file1000 json    fileType json compression none numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 json   file10000 json    fileType json compression gzip numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 gzip   file100 gzip    fileType json compression gzip numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 gzip   file1000 gzip    fileType json compression gzip numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 gzip   file10000 gzip    fileType parquet compression none numColumns 10 columnTypes 100 STRING numFiles 100 tableSize 10MB    contains file1 parquet  file100 parquet    fileType parquet compression none numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB    contains file1 parquet   file1000 parquet    fileType parquet compression none numColumns 10 columnTypes 100 STRING numFiles 10000 tableSize 10MB    contains file1 parquet   file10000 parquet   To complete the process of creating files  run the following command      python bq benchmark py     create files     gcs project id  ID of project holding GCS resources      resized staging dataset id  ID of dataset holding resized staging tables      bucket name  name of bucket to hold files      dataflow staging location  path on GCS to serve as staging location for Dataflow      dataflow temp location  path on GCS to serve as temp location for Dataflow      restart file  optional file name to restart with if program is stopped          Parameters      create files   Flag to indicate that files should be created and stored on GCS  It has a value of  store true   so this flag will be set to False  unless it is provided in the command      gcs project id   The ID of the project that will hold the GCS resources for the benchmark  including all files and the bucket that holds them      resized staging dataset id   The ID of the dataset that holds the resized staging tables generated using the    create staging tables  command      bucket name   Name of the bucket that will hold the created files  Note that the only purpose of this bucket should be to hold the created files  and that files used for any other reason should be stored in a different bucket      dataflow staging location   Staging location for Dataflow on GCS  Include the  gs     prefix  the name of the bucket you want to use  and any prefix  For example  gs    bucket name  staging   Note  be sure to use a different bucket than the one provided in the    bucket name parameter       dataflow temp location   Temp location for Dataflow on GCS  Include the  gs     prefix  the name of the bucket you want to use  and any prefix  For example  gs    bucket name  temp   Note  be sure to use a different bucket than the one provided in the    bucket name parameter       restart file   Optional file name to start the file creation process with  Creating each file combination can take hours  and often a backend error or a timeout will occur  preventing all the files from being created  If this happens  copy the last file that was successfully created from the logs and use it here  It should start with  fileType   and end with the file extension  For example   fileType csv compression none numColumns 10 columnTypes 100 STRING numFiles 1000 tableSize 10MB file324 csv       Running the benchmarks       File Loader Benchmark Once the files are created  the File Loader Benchmark can be run  As a prerequisite for this step  a log sink in BigQuery that captures logs about BigQuery must be set up in the same project that holds the benchmark tables  If a BigQuery log sink is not already set up  follow  these steps  https   github com GoogleCloudPlatform professional services tree master examples bigquery audit log 1 getting the bigquery log data    Note that this benchmark will delete tables after recording information on load time  Before the tables are deleted  the tables and their respective files can be used to run the Federated Query Benchmark  If running the two benchmarks independently  each file will be used to create a BigQuery table two different times  Running the two benchmarks at the same time can save time if results for both benchmarks are desired  In this case  the    include federated query benchmark  flag can be added to the below command  Be aware that running the queries will add significant time to the benchmark run  so leave the flag out of the command if the primary goal is to obtain results for the File Loader Benchmark    To run the benchmark  use the following command        python bq benchmark py     run file loader benchmark     bq project id  ID of the project holding the BigQuery resources      gcs project id  ID of project holding GCS resources      staging project id  ID of project holding staging tables      staging dataset id  ID of dataset holding staging tables      benchmark dataset id  ID of the dataset holding the benchmark tables      bucket name  name of bucket to hold files      results table name  Name of results table      results dataset id  Name dataset holding results table      duplicate benchmark tables     bq logs dataset  Name of dataset hold BQ logs table    include federated query benchmark       Parameters     run file loader benchmark   Flag to initiate process of running the File Loader Benchmark by creating tables from files and storing results for comparison  It has a value of  store true   so this flag will be set to False  unless it is provided in the command      gcs project id   The ID of the project that will hold the GCS resources for the benchmark  including all files and the bucket that holds them      bq project id   The ID of the project that will hold the BigQuery resources for the benchmark  including all datasets  results tables  staging tables  and benchmark tables      staging project id   The ID of the project that holds the first set of staging tables  While this will be the same as the    bq project id  if running the project from scratch  it will differ from    bq project id  if you are using file combinations that have already been created and running benchmarks saving results in your own project      staging dataset id   The ID of the dataset that will hold the first set of staging tables  For the tool to work correctly  the  staging dataset id  must only contain staging tables  and it must be different than the    resized staging dataset id   Do not store tables for any other purposes in this dataset      dataset id   The ID of the dataset that will hold the benchmark tables      bucket name   Name of the bucket that will hold the file combinations to be loaded into benchmark tables  Note that the only purpose of this bucket should be to hold the file combinations  and that files used for any other reason should be stored in a different bucket      results table name   Name of the results table to hold relevant information about the benchmark loads      results dataset id   Name of the dataset that holds the results table      duplicate benchmark tables   Flag to indicate that a benchmark table should be created for a given file combination  even if that file combination has a benchmark table already  Creating multiple benchmark tables for each file combination can increase the accuracy of the average runtimes calculated from the results  If this behavior is desired  include the flag  However  if you want to ensure that you first have at least one benchmark table for each file combination  then leave the flag off  In that case  the benchmark creation process will skip a file combination if it already has a benchmark table      bq logs dataset   Name of dataset hold BQ logs table  This dataset must be in project used for    bq project id       include federated query benchmark   Flag to indicate that the Federated Query Benchmark should be run on the created tables and the files the tables were loaded from before the tables are deleted  If results for both benchmarks are desired  this will save time when compared to running each benchmark independently  since the same tables needed for the File Loader Benchmark are needed for the Federated Query Benchmark  It has a value of  store true   so this flag will be set to False  unless it is provided in the command         Federated Query Benchmark Once the files are created  the Federated Query Benchmark can be run  As a prerequisite for this step  a log sink in BigQuery that captures logs about BigQuery must be set up in the same project that holds the benchmark tables  If a BigQuery log sink is not already set up  follow  these steps  https   github com GoogleCloudPlatform professional services tree master examples bigquery audit log 1 getting the bigquery log data    As mentioned above  the Federated Query Benchmark can be run while running the File Loader Benchmark in addition to using the command below  Note  though   that running federated queries on snappy compressed files is not supported  When the File Loader Benchmark encounters a snappy compressed file  it still loads the file into a BigQuery table to capture load results  but it will skip the Federated Query portion  When the Federated Query Benchmark encounters a snappy compressed file  it will skip the load all together  Therefore  if obtaining Federated Query Benchmark results is the primary goal  use the command below   It should also be noted that since the Federated Query Benchmark loads files into tables  the load results for the File Loader Benchmark will also be captured  This will not add significant time to the benchmark run since the tables have to be loaded regardless   To run the benchmark  use the following command       python bq benchmark py     run federated query benchmark     bq project id  ID of the project holding the BigQuery resources      gcs project id  ID of project holding GCS resources      staging project id  ID of project holding staging tables      staging dataset id  ID of dataset holding staging tables      benchmark dataset id  ID of the dataset holding the benchmark tables      bucket name  name of bucket to hold files      results table name  Name of results table      results dataset id  Name dataset holding results table      bq logs dataset  Name of dataset hold BQ logs table        Parameters     run federated query benchmark   Flag to initiate the process running the Federated Query Benchmark by creating tables from files  running queries on both the table and the files  and storing performance results  It has a value of  store true   so this flag will be set to False  unless it is provided in the command      gcs project id   The ID of the project that will hold the GCS resources for the benchmark  including all files and the bucket that holds them      bq project id   The ID of the project that will hold the BigQuery resources for the benchmark  including all datasets  results tables  staging tables  and benchmark tables      staging project id   The ID of the project that holds the first set of staging tables  While this will be the same as the    bq project id  if running the project from scratch  it will differ from    bq project id  if you are using file combinations that have already been created and running benchmarks saving results in your own project      staging dataset id   The ID of the dataset that will hold the first set of staging tables  For the tool to work correctly  the  staging dataset id  must only contain staging tables  and it must be different than the    resized staging dataset id   Do not store tables for any other purposes in this dataset      dataset id   The ID of the dataset that will hold the benchmark tables      bucket name   Name of the bucket that will hold the file combinations to be loaded into benchmark tables  Note that the only purpose of this bucket should be to hold the file combinations  and that files used for any other reason should be stored in a different bucket      results table name   Name of the results table to hold relevant information about the benchmark loads      results dataset id   Name of the dataset that holds the results table      bq logs dataset   Name of dataset hold BQ logs table  This dataset must be in project used for    bq project id        Testing  Tests can be run by running the following command in the bq file load benchmark directory       python  m pytest   project id  ID of project that will hold test resources        Note that the tests will create and destroy resources in the project denoted by    project id


GCP grpcexample gRPC Example user for a given user id This example creates a gRCP server that connect to spanner to find the name of Application Project Structure

Answer:
# gRPC Example

This example creates a gRCP server that connect to spanner to find the name of
user for a given user id.

## Application Project Structure

 ```
    .
    └── grpc_example
        └── src
            └── main
                ├── java
                        └── com.example.grpc
                            ├── client
                                └── ConnectClient # Example Client
                            ├── server
                                └── ConnectServer # Initializes gRPC Server
                            └── service
                                └── ConnectService # Implementation of rpc services in the proto
                └── proto
                    └── connect_service.proto # Proto definition of the server
       ├── pom.xml
       └── README.md
```

## Technology Stack

1. gRPC
2. Spanner

## Setup Instructions

### Project Setup

#### Creating a Project in the Google Cloud Platform Console

If you haven't already created a project, create one now. Projects enable you to
manage all Google Cloud Platform resources for your app, including deployment,
access control, billing, and services.

1. Open the [Cloud Platform Console][cloud-console].
1. In the drop-down menu at the top, select **Create a project**.
1. Give your project a name = my-dfdl-project
1. Make a note of the project ID, which might be different from the project
   name. The project ID is used in commands and in configurations.

[cloud-console]: https://console.cloud.google.com/

#### Enabling billing for your project.

If you haven't already enabled billing for your
project, [enable billing][enable-billing] now. Enabling billing allows is
required to use Cloud Bigtable and to create VM instances.

[enable-billing]: https://console.cloud.google.com/project/_/settings

#### Install the Google Cloud SDK.

If you haven't already installed the Google Cloud
SDK, [install the Google Cloud SDK][cloud-sdk] now. The SDK contains tools and
libraries that enable you to create and manage resources on Google Cloud
Platform.

[cloud-sdk]: https://cloud.google.com/sdk/

#### Setting Google Application Default Credentials

Set
your [Google Application Default Credentials][application-default-credentials]
by [initializing the Google Cloud SDK][cloud-sdk-init] with the command:

```
gcloud init
```

Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
command:

```
gcloud auth application-default login
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing

[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

### Spanner Setup from the Console

1. Create an instance called grpc-example
2. Create a database called grpc_example_db
3. Create a table named Users using the following Database Definition Language (
   DDL) statement for the database

```
  CREATE TABLE Users (
  user_id INT64 NOT NULL,
  name STRING(MAX),
) PRIMARY KEY(user_id);
```

3. Insert the following record into the table Users

```
INSERT INTO
  Users (user_id,
    name)
VALUES
  (1234, -- type: INT64
    "MyName" -- type: STRING(MAX)
    );
```

## Usage

### Initialize the server

```
$ mvn -DskipTests package exec:java
    -Dexec.mainClass=com.example.grpc.server.ConnectServer

```

### Run the Client

```
$ mvn -DskipTests package exec:java
      -Dexec.mainClass=com.example.grpc.client.ConnectClient
``

gRPC Example  This example creates a gRCP server that connect to spanner to find the name of user for a given user id      Application Project Structure                     grpc example             src                 main                     java                             com example grpc                                 client                                     ConnectClient   Example Client                                 server                                     ConnectServer   Initializes gRPC Server                                 service                                     ConnectService   Implementation of rpc services in the proto                     proto                         connect service proto   Proto definition of the server            pom xml            README md         Technology Stack  1  gRPC 2  Spanner     Setup Instructions      Project Setup       Creating a Project in the Google Cloud Platform Console  If you haven t already created a project  create one now  Projects enable you to manage all Google Cloud Platform resources for your app  including deployment  access control  billing  and services   1  Open the  Cloud Platform Console  cloud console   1  In the drop down menu at the top  select   Create a project    1  Give your project a name   my dfdl project 1  Make a note of the project ID  which might be different from the project    name  The project ID is used in commands and in configurations    cloud console   https   console cloud google com        Enabling billing for your project   If you haven t already enabled billing for your project   enable billing  enable billing  now  Enabling billing allows is required to use Cloud Bigtable and to create VM instances    enable billing   https   console cloud google com project   settings       Install the Google Cloud SDK   If you haven t already installed the Google Cloud SDK   install the Google Cloud SDK  cloud sdk  now  The SDK contains tools and libraries that enable you to create and manage resources on Google Cloud Platform    cloud sdk   https   cloud google com sdk        Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command       gcloud init      Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command       gcloud auth application default login       cloud sdk init   https   cloud google com sdk docs initializing   application default credentials   https   developers google com identity protocols application default credentials      Spanner Setup from the Console  1  Create an instance called grpc example 2  Create a database called grpc example db 3  Create a table named Users using the following Database Definition Language      DDL  statement for the database        CREATE TABLE Users     user id INT64 NOT NULL    name STRING MAX     PRIMARY KEY user id        3  Insert the following record into the table Users      INSERT INTO   Users  user id      name  VALUES    1234     type  INT64      MyName     type  STRING MAX                 Usage      Initialize the server        mvn  DskipTests package exec java       Dexec mainClass com example grpc server ConnectServer           Run the Client        mvn  DskipTests package exec java         Dexec mainClass com example grpc client ConnectClient


GCP The popularization of IoT devices and the evolvement of machine learning technologies have brought tremendous opportunities for new businesses We demonstrate how home appliances e g kettle and washing machine working status on off can be inferred from gross power readings collected by a smart meter together with state of art machine learning techniques An end to end demo system is developed entirely on Google Cloud Platform as shown in the following figure It includes Data collection and ingesting through Cloud IoT Core and Cloud Pub Sub Home Appliances Working Status Monitoring Using Gross Power Readings Machine Learning model serving using CMLE together with App Engine as frontend Steps to deploy the demo system Data visualization and exploration using Colab Machine learning model development using Tensorflow and training using Cloud Machine Learning Engine CMLE

Answer:
# Home Appliances’ Working Status Monitoring Using Gross Power Readings
The popularization of IoT devices and the evolvement of machine learning technologies have brought tremendous opportunities for new businesses. We demonstrate how home appliances’ (e.g. kettle and washing machine) working status (on/off) can be inferred from gross power readings collected by a smart meter together with state-of-art machine learning techniques. An end-to-end demo system is developed entirely on Google Cloud Platform as shown in the following figure. It includes
* Data collection and ingesting through Cloud IoT Core and Cloud Pub/Sub
* Machine learning model development using Tensorflow and training using Cloud Machine Learning Engine (CMLE)
* Machine Learning model serving using CMLE together with App Engine as frontend
* Data visualization and exploration using Colab
![system architecture](./img/arch.jpg)

## Steps to deploy the demo system

### Step 0. Prerequisite
Before you follow the instructions below to deploy our demo system, you need a Google cloud project if you don't have one. You can find detailed instructions [here](https://cloud.google.com/dataproc/docs/guides/setup-project).

After you have created a google cloud project, follow the instructions below:
```shell
# clone the repository storing all the necessary codes
git clone [REPO_URL]

cd professional-services/examples/e2e-home-appliance-status-monitoring

# remember your project's id in an environment variable
GOOGLE_PROJECT_ID=[your-google-project-id]

# create and download a service account
gcloud --project ${GOOGLE_PROJECT_ID} iam service-accounts create e2e-demo-sc
gcloud --project ${GOOGLE_PROJECT_ID} projects add-iam-policy-binding ${GOOGLE_PROJECT_ID} \
       --member "serviceAccount:e2e-demo-sc@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com" \
       --role "roles/owner"
gcloud --project ${GOOGLE_PROJECT_ID} iam service-accounts keys create e2e_demo_credential.json \
       --iam-account e2e-demo-sc@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com
GOOGLE_APPLICATION_CREDENTIALS=${PWD}"/e2e_demo_credential.json"

# create a new GCS bucket if you don't have one
BUCKET_NAME=[your-bucket-name]
gsutil mb -p ${GOOGLE_PROJECT_ID} gs://${BUCKET_NAME}/
```

You also need to enable the following APIs in the APIs & Services menu.
* Cloud ML Engine API
* Cloud IoT API
* Cloud PubSub API

### Step 1. Deploy a trained ML model in Cloud ML Engine.
You can download our trained model [from the `data` directory](./data/model.tar.bz2) or you can train your own model using the `ml/start.sh`.
Notice: you need to enable CLoud ML Engine API first.

If you are using our trained model:
```shell
# download our trained model
tar jxvf data/model.tar.bz2

# upload the model to your bucket
gsutil cp -r model gs://${BUCKET_NAME}
```

If you want to train your own model:
```shell
pip install ml/
cd ml

# use one of the following commands and your model should be saved in your cloud storage bucket

# train locally with default parameter
bash start.sh -l
# train locally with specified parameters
bash start.sh -l learning-rate=0.00001 lstm-size=128
# train on Cloud ML Engine with default parameter
bash start.sh -p ${GOOGLE_PROJECT_ID} -b ${BUCKET_NAME}
# train on Cloud ML Engine with specified parameters
bash start.sh -p ${GOOGLE_PROJECT_ID} -b ${BUCKET_NAME} learning-rate=0.00001 lstm-size=128
# run hyper-parameter tuning on Cloud ML Engine
bash start.sh -p ${GOOGLE_PROJECT_ID} -b ${BUCKET_NAME} -t
```

Finally let's deploy our model to ML engine:
```shell
# Set up an appropriate region
# Available regions: https://cloud.google.com/ml-engine/docs/tensorflow/regions
REGION="your-application-region"

# create a model
gcloud ml-engine models create EnergyDisaggregationModel \
      --regions ${REGION} \
      --project ${GOOGLE_PROJECT_ID}

# create a model version
gcloud ml-engine versions create v01 \
      --model EnergyDisaggregationModel \
      --origin gs://${BUCKET_NAME}/model \
      --runtime-version 1.12 \
      --framework TensorFlow \
      --python-version 3.5 \
      --project ${GOOGLE_PROJECT_ID}
```

### Step 2. Deploy server.
Type in the following commands to start server in app engine.
```shell
cd e2e_demo/server
cp ${GOOGLE_APPLICATION_CREDENTIALS} .
echo "  GOOGLE_APPLICATION_CREDENTIALS: '${GOOGLE_APPLICATION_CREDENTIALS##*/}'" >> app.yaml
echo "  GOOGLE_CLOUD_PROJECT: '${GOOGLE_PROJECT_ID}'" >> app.yaml

# deploy application engine, choose any region that suits and answer yes at the end.
gcloud --project ${GOOGLE_PROJECT_ID} app deploy

# create a pubsub topic "data" and a subscription in the topic.
# this is the pubsub between IoT devices and the server.
gcloud --project ${GOOGLE_PROJECT_ID} pubsub topics create data
gcloud --project ${GOOGLE_PROJECT_ID} pubsub subscriptions create sub0 \
       --topic=data --push-endpoint=https://${GOOGLE_PROJECT_ID}.appspot.com/upload

# create a pubsub topic "pred" and a subscription in the topic.
# this is the pubsub between the server and a result utilizing client.
gcloud --project ${GOOGLE_PROJECT_ID} pubsub topics create pred
gcloud --project ${GOOGLE_PROJECT_ID} pubsub subscriptions create sub1 --topic=pred

# uncompress the data
bunzip data/*csv.bz2

# create BigQuery dataset and tables.
bq --project_id ${GOOGLE_PROJECT_ID} mk \
   --dataset ${GOOGLE_PROJECT_ID}:EnergyDisaggregation
bq --project_id ${GOOGLE_PROJECT_ID} load --autodetect \
   --source_format=CSV EnergyDisaggregation.ApplianceInfo \
   ./data/appliance_info.csv
bq --project_id ${GOOGLE_PROJECT_ID} load --autodetect \
   --source_format=CSV EnergyDisaggregation.ApplianceStatusGroundTruth \
   ./data/appliance_status_ground_truth.csv
bq --project_id ${GOOGLE_PROJECT_ID} mk \
   --table ${GOOGLE_PROJECT_ID}:EnergyDisaggregation.ActivePower \
   time:TIMESTAMP,device_id:STRING,power:INTEGER
bq --project_id ${GOOGLE_PROJECT_ID} mk \
   --table ${GOOGLE_PROJECT_ID}:EnergyDisaggregation.Predictions \
   time:TIMESTAMP,device_id:STRING,appliance_id:INTEGER,pred_status:INTEGER,pred_prob:FLOAT
```

### Step 3. Setup your Cloud IoT Client(s)
Follow the instructions below to set up your client(s).
Note: you need to enable the Cloud IoT API first.
```shell
# You need to specify the IDs for cloud iot registry and the devices you want.
# See more details for permitted characters and sizes for each resource:
# https://cloud.google.com/iot/docs/requirements#permitted_characters_and_size_requirements
REGISTRY_ID="your-registry-id"
DEVICE_IDS=("your-device-id1" "your-device-id2" ...)

# create an iot registry with created pubsub topic above
gcloud --project ${GOOGLE_PROJECT_ID} iot registries create ${REGISTRY_ID} \
       --region ${REGION} --event-notification-config topic=data

# generates key pair for setting in Cloud IoT device.
# The rs256.key generated by the following command would be used to create JWT.
ssh-keygen -t rsa -b 4096 -f ./rs256.key
# Press "Enter" twice
openssl rsa -in ./rs256.key -pubout -outform PEM -out ./rs256.pub

# create multiple devices
for device in ${DEVICE_IDS[@]}; do
  # create an iot device with generated public key and the registry.
  gcloud --project ${GOOGLE_PROJECT_ID} iot devices create ${device} \
         --region ${REGION} --public-key path=./rs256.pub,type=rsa-pem \
         --registry ${REGISTRY_ID} --log-level=debug
done

# download root ca certificates for use in mqtt client communicated with iot server.
# download using curl: curl -o roots.pem https://pki.goog/roots.pem
wget https://pki.goog/roots.pem --no-check-certificate
```

### Complete! Try the demo system in colab or locally.
If you want to use colab, visit https://colab.research.google.com/ and you can import our notebooks either directly from Github or upload from your cloned repository. Follow the instructions in the notebooks and you should be able to reproduce our results.

If you want to run the demo locally:
```
cd notebook
virtualenv env
source env/bin/activate
pip install jupyter
jupyter notebook
```

*notebook/EnergyDisaggregationDemo_View.ipynb* allows you to view raw power consumption data and our model's prediction results in almost real time. It pulls data from our server's pubsub topic for visualization. Fill in the necessary information in the *Configuration* block and run all the cells.

*notebook/EnergyDisaggregationDemo_Client.ipynb* simulates multiple smart meters by reading in power consumption data from a real world dataset and sends the readings to our server. All Cloud IoT Core related code resides in this notebook. Fill in the necessary information in the *Configuration* block and run all the cells, once you see messages being sent you should be able to see plots like the one shown below in *notebook/EnergyDisaggregationDemo_View.ipynb*.

![Demo system sample output](./img/demo03.gif)

Home Appliances  Working Status Monitoring Using Gross Power Readings The popularization of IoT devices and the evolvement of machine learning technologies have brought tremendous opportunities for new businesses  We demonstrate how home appliances   e g  kettle and washing machine  working status  on off  can be inferred from gross power readings collected by a smart meter together with state of art machine learning techniques  An end to end demo system is developed entirely on Google Cloud Platform as shown in the following figure  It includes   Data collection and ingesting through Cloud IoT Core and Cloud Pub Sub   Machine learning model development using Tensorflow and training using Cloud Machine Learning Engine  CMLE    Machine Learning model serving using CMLE together with App Engine as frontend   Data visualization and exploration using Colab   system architecture    img arch jpg      Steps to deploy the demo system      Step 0  Prerequisite Before you follow the instructions below to deploy our demo system  you need a Google cloud project if you don t have one  You can find detailed instructions  here  https   cloud google com dataproc docs guides setup project    After you have created a google cloud project  follow the instructions below     shell   clone the repository storing all the necessary codes git clone  REPO URL   cd professional services examples e2e home appliance status monitoring    remember your project s id in an environment variable GOOGLE PROJECT ID  your google project id     create and download a service account gcloud   project   GOOGLE PROJECT ID  iam service accounts create e2e demo sc gcloud   project   GOOGLE PROJECT ID  projects add iam policy binding   GOOGLE PROJECT ID             member  serviceAccount e2e demo sc   GOOGLE PROJECT ID  iam gserviceaccount com             role  roles owner  gcloud   project   GOOGLE PROJECT ID  iam service accounts keys create e2e demo credential json            iam account e2e demo sc   GOOGLE PROJECT ID  iam gserviceaccount com GOOGLE APPLICATION CREDENTIALS   PWD   e2e demo credential json     create a new GCS bucket if you don t have one BUCKET NAME  your bucket name  gsutil mb  p   GOOGLE PROJECT ID  gs     BUCKET NAME        You also need to enable the following APIs in the APIs   Services menu    Cloud ML Engine API   Cloud IoT API   Cloud PubSub API      Step 1  Deploy a trained ML model in Cloud ML Engine  You can download our trained model  from the  data  directory    data model tar bz2  or you can train your own model using the  ml start sh   Notice  you need to enable CLoud ML Engine API first   If you are using our trained model     shell   download our trained model tar jxvf data model tar bz2    upload the model to your bucket gsutil cp  r model gs     BUCKET NAME       If you want to train your own model     shell pip install ml  cd ml    use one of the following commands and your model should be saved in your cloud storage bucket    train locally with default parameter bash start sh  l   train locally with specified parameters bash start sh  l learning rate 0 00001 lstm size 128   train on Cloud ML Engine with default parameter bash start sh  p   GOOGLE PROJECT ID   b   BUCKET NAME    train on Cloud ML Engine with specified parameters bash start sh  p   GOOGLE PROJECT ID   b   BUCKET NAME  learning rate 0 00001 lstm size 128   run hyper parameter tuning on Cloud ML Engine bash start sh  p   GOOGLE PROJECT ID   b   BUCKET NAME   t      Finally let s deploy our model to ML engine     shell   Set up an appropriate region   Available regions  https   cloud google com ml engine docs tensorflow regions REGION  your application region     create a model gcloud ml engine models create EnergyDisaggregationModel           regions   REGION            project   GOOGLE PROJECT ID     create a model version gcloud ml engine versions create v01           model EnergyDisaggregationModel           origin gs     BUCKET NAME  model           runtime version 1 12           framework TensorFlow           python version 3 5           project   GOOGLE PROJECT ID           Step 2  Deploy server  Type in the following commands to start server in app engine     shell cd e2e demo server cp   GOOGLE APPLICATION CREDENTIALS    echo    GOOGLE APPLICATION CREDENTIALS     GOOGLE APPLICATION CREDENTIALS           app yaml echo    GOOGLE CLOUD PROJECT     GOOGLE PROJECT ID       app yaml    deploy application engine  choose any region that suits and answer yes at the end  gcloud   project   GOOGLE PROJECT ID  app deploy    create a pubsub topic  data  and a subscription in the topic    this is the pubsub between IoT devices and the server  gcloud   project   GOOGLE PROJECT ID  pubsub topics create data gcloud   project   GOOGLE PROJECT ID  pubsub subscriptions create sub0            topic data   push endpoint https     GOOGLE PROJECT ID  appspot com upload    create a pubsub topic  pred  and a subscription in the topic    this is the pubsub between the server and a result utilizing client  gcloud   project   GOOGLE PROJECT ID  pubsub topics create pred gcloud   project   GOOGLE PROJECT ID  pubsub subscriptions create sub1   topic pred    uncompress the data  bunzip data  csv bz2    create BigQuery dataset and tables  bq   project id   GOOGLE PROJECT ID  mk        dataset   GOOGLE PROJECT ID  EnergyDisaggregation bq   project id   GOOGLE PROJECT ID  load   autodetect        source format CSV EnergyDisaggregation ApplianceInfo        data appliance info csv bq   project id   GOOGLE PROJECT ID  load   autodetect        source format CSV EnergyDisaggregation ApplianceStatusGroundTruth        data appliance status ground truth csv bq   project id   GOOGLE PROJECT ID  mk        table   GOOGLE PROJECT ID  EnergyDisaggregation ActivePower      time TIMESTAMP device id STRING power INTEGER bq   project id   GOOGLE PROJECT ID  mk        table   GOOGLE PROJECT ID  EnergyDisaggregation Predictions      time TIMESTAMP device id STRING appliance id INTEGER pred status INTEGER pred prob FLOAT          Step 3  Setup your Cloud IoT Client s  Follow the instructions below to set up your client s   Note  you need to enable the Cloud IoT API first     shell   You need to specify the IDs for cloud iot registry and the devices you want    See more details for permitted characters and sizes for each resource    https   cloud google com iot docs requirements permitted characters and size requirements REGISTRY ID  your registry id  DEVICE IDS   your device id1   your device id2          create an iot registry with created pubsub topic above gcloud   project   GOOGLE PROJECT ID  iot registries create   REGISTRY ID             region   REGION    event notification config topic data    generates key pair for setting in Cloud IoT device    The rs256 key generated by the following command would be used to create JWT  ssh keygen  t rsa  b 4096  f   rs256 key   Press  Enter  twice openssl rsa  in   rs256 key  pubout  outform PEM  out   rs256 pub    create multiple devices for device in   DEVICE IDS      do     create an iot device with generated public key and the registry    gcloud   project   GOOGLE PROJECT ID  iot devices create   device               region   REGION    public key path   rs256 pub type rsa pem              registry   REGISTRY ID    log level debug done    download root ca certificates for use in mqtt client communicated with iot server    download using curl  curl  o roots pem https   pki goog roots pem wget https   pki goog roots pem   no check certificate          Complete  Try the demo system in colab or locally  If you want to use colab  visit https   colab research google com  and you can import our notebooks either directly from Github or upload from your cloned repository  Follow the instructions in the notebooks and you should be able to reproduce our results   If you want to run the demo locally      cd notebook virtualenv env source env bin activate pip install jupyter jupyter notebook       notebook EnergyDisaggregationDemo View ipynb  allows you to view raw power consumption data and our model s prediction results in almost real time  It pulls data from our server s pubsub topic for visualization  Fill in the necessary information in the  Configuration  block and run all the cells    notebook EnergyDisaggregationDemo Client ipynb  simulates multiple smart meters by reading in power consumption data from a real world dataset and sends the readings to our server  All Cloud IoT Core related code resides in this notebook  Fill in the necessary information in the  Configuration  block and run all the cells  once you see messages being sent you should be able to see plots like the one shown below in  notebook EnergyDisaggregationDemo View ipynb      Demo system sample output    img demo03 gif