gcp gke docs Implement GCP Google Kubernetes Engine GKE Continuous Integration Step 00 Pre requisites 2 Verify if kubeconfig for kubectl is configured in your local terminal Configure kubeconfig for kubectl title GCP Google Kubernetes Engine GKE CI 1 Verify if GKE Cluster is created t

Answer:
---
title: GCP Google Kubernetes Engine GKE CI
description: Implement GCP Google Kubernetes Engine GKE Continuous Integration
---

## Step-00: Pre-requisites
1. Verify if GKE Cluster is created
2. Verify if kubeconfig for kubectl is configured in your local terminal
```t
# Configure kubeconfig for kubectl
gcloud container clusters get-credentials <CLUSTER-NAME> --region <REGION> --project <PROJECT>

# Replace Values CLUSTER-NAME, REGION, PROJECT
gcloud container clusters get-credentials standard-cluster-private-1 --region us-central1 --project kdaida123

# List Kubernetes Nodes
kubectl get nodes
```

## Step-01: Introduction
- Implement Continuous Integration for GKE Workloads using
- Google Cloud Source
- Google Cloud Build
- Google Artifact Repository

## Step-02: Enable APIs in Google Cloud
```t
# Enable APIs in Google Cloud
gcloud services enable container.googleapis.com \
    cloudbuild.googleapis.com \
    sourcerepo.googleapis.com \
    artifactregistry.googleapis.com

# Google Cloud Services
GKE: container.googleapis.com
Cloud Build: cloudbuild.googleapis.com
Cloud Source: sourcerepo.googleapis.com
Artifact Registry: artifactregistry.googleapis.com
```

## Step-03: Create Artifact Repository
```t
# List Artifact Repositories
gcloud artifacts repositories list

# Create Artifact Repository
gcloud artifacts repositories create myapps-repository \
  --repository-format=docker \
  --location=us-central1

# List Artifact Repositories
gcloud artifacts repositories list

# Describe Artifact Repository
gcloud artifacts repositories describe myapps-repository --location=us-central1
```

## Step-04: Install Git client on local desktop (if not present)
```t
# Download and Install Git Client and Installed
https://git-scm.com/downloads
```

## Step-05: Create SSH Keys for Git Repo Access
- [Generating SSH Key Pair](https://cloud.google.com/source-repositories/docs/authentication#generate_a_key_pair)
```t
# Change Directory
cd 01-SSH-Keys

# Create SSH Keys
ssh-keygen -t [KEY_TYPE] -C "[USER_EMAIL]"
KEY_TYPE: rsa, ecdsa, ed25519
USER_EMAIL: dkalyanreddy@gmail.com

# Replace Values KEY_TYPE, USER_EMAIL
ssh-keygen -t ed25519 -C "dkalyanreddy@gmail.com"
Provide the File Name as "id_gcp_cloud_source"

## Sample Output
Kalyans-Mac-mini:01-SSH-Keys kalyanreddy$ ssh-keygen -t ed25519 -C "dkalyanreddy@gmail.com"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/Users/kalyanreddy/.ssh/id_ed25519): id_gcp_cloud_source
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_gcp_cloud_source
Your public key has been saved in id_gcp_cloud_source.pub
The key fingerprint is:
SHA256:YialyCj3XaSa4b8ewk4bcK1hXxO7DDM5uiCP1J2TOZ0 dkalyanreddy@gmail.com
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|                 |
|      . o        |
| o . + + o       |
|o = B % S        |
|...B.&=X.o       |
|....%B+Eo        |
|.+ + *o.         |
|. . +.+.         |
+----[SHA256]-----+
Kalyans-Mac-mini:01-SSH-Keys kalyanreddy$ ls -lrta
total 16
drwxr-xr-x  6 kalyanreddy  staff  192 Jun 29 09:45 ..
-rw-------  1 kalyanreddy  staff  419 Jun 29 09:46 id_gcp_cloud_source
drwxr-xr-x  4 kalyanreddy  staff  128 Jun 29 09:46 .
-rw-r--r--  1 kalyanreddy  staff  104 Jun 29 09:46 id_gcp_cloud_source.pub
Kalyans-Mac-mini:01-SSH-Keys kalyanreddy$
```

## Step-06: Review SSH Keys (Public and Private Keys)
```t
# Change Directroy
cd 01-SSH-Keys

# Review Private Key: id_gcp_cloud_source
cat id_gcp_cloud_source

# Review Public Key: id_gcp_cloud_source.pub
cat id_gcp_cloud_source.pub
```

## Step-07: Update SSH Public Key in Google Cloud Source
- Go to -> Source Repositories -> 3 Dots -> Manage SSH Keys -> Register SSH Key
- [Google Cloud Source URL](https://source.cloud.google.com/)
```t
# Key Name
Name: gke-course
Key: Output from command "cat id_gcp_cloud_source.pub" in previous step. Put content from Public Key
```
- Click on **Register**

## Step-08: Update SSH Private Key in Git Config
- Update SSH Private Key in your local desktop Git Config
```t
# Copy SSH Private Key to your ".ssh" folder in your Home Directory from your course
cd 01-SSH-Keys
cp id_gcp_cloud_source $HOME/.ssh

# Change Directory (Your local desktop home directory)
cd $HOME/.ssh

# Verify File in "$HOME/.ssh"
ls -lrta id_gcp_cloud_source

# Verify existing git "config" file
cat config

# Backup any existing "config" file
cp config config_bkup_before_cloud_source

# Update "config" file to point to "id_gcp_cloud_source" private key
vi config

## Sample Output after changes
Kalyans-Mac-mini:.ssh kalyanreddy$ cat config
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_gcp_cloud_source
Kalyans-Mac-mini:.ssh kalyanreddy$

# Backup config with cloudsource
cp config config_with_cloud_source_key
```

## Step-09: Update Git Global Config in your local deskopt
```t
# List Global Git Config
git config --list

# Update Global Git Config
git config --global user.email "YOUR_EMAIL_ADDRESS"
git config --global user.name "YOUR_NAME"

# Replace YOUR_EMAIL_ADDRESS, YOUR_NAME
git config --global user.name "Kalyan Reddy Daida"
git config --global user.email "dkalyanreddy@gmail.com"

# List Global Git Config
git config --list
```

## Step-10: Create Git repositories in Cloud Source
```t
# List Cloud Source Repository
gcloud source repos list

# Create Git repositories in Cloud Source
gcloud source repos create myapp1-app-repo

# List Cloud Source Repository
gcloud source repos list

# Verify using Cloud Console
Search for -> Source Repositories
https://source.cloud.google.com/repos
```

## Step-11: Clone Cloud Source Git Repository, Commit a Change, Push to Remote Repo and Verify
```t
# Change Directory
cd course-repos

# Verify using Cloud Console
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo -> SSH Authentication

# Copy the git clone command and run
git clone ssh://dkalyanreddy@gmail.com@source.developers.google.com:2022/p/kdaida123/r/myapp1-app-repo

# Change Directory
cd myapp1-app-repo

# Create a simple readme file
touch README.md
echo "# GKE CI Demo" > README.md
ls -lrta

# Add Files and do local commit
git add .
git commit -am "First Commit"

# Push file to Cloud Source Git Repo (Remote Repo)
git push

# Verify in Git Remote Repo
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo
```

## Step-12: Review Files in 02-Docker-Image folder
1. Dockerfile
2. index.html

## Step-13: Copy files from 02-Docker-Image folder to Git Repo
```t
# Change Directroy
cd 57-GKE-Continuous-Integration/02-Docker-Image

# Copy Files to Git repo "myapp1-app-repo"
1. Dockerfile
2. index.html

# Local Git Commit and Push to Remote Repo
git add .
git commit -am "Second Commit"
git push

# Verify in Git Remote Repo
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo
```

## Step-14: Create a container image with Cloud Build and store it in Artifact Registry using glcoud builds command
```t
# Change Directory (Git App Repo: myapp1-app-repo)
cd myapp1-app-repo

# Get latest git commit id (current branch)
git rev-parse HEAD

# Get latest git commit id first 7 chars (current branch)
git rev-parse --short=7 HEAD

# Ensure you are in local git repo folder where "Dockerfile, index.html" present
cd myapp1-app-repo

# Create a Cloud Build build based on the latest commit
gcloud builds submit --tag="us-central1-docker.pkg.dev/${PROJECT_ID}/${$APP_ARTIFACT_REPO}/myapp1:${COMMIT_ID}" .

# Replace Values ${PROJECT_ID}, ${$APP_ARTIFACT_REPO}, ${COMMIT_ID}
gcloud builds submit --tag="us-central1-docker.pkg.dev/kdaida123/myapps-repository/myapp1:6f7d338" .
```

## Step-15: Review Cloud Build YAML file
```yaml
steps:
# This step builds the container image.
- name: 'gcr.io/cloud-builders/docker'
  id: Build
  args:
  - 'build'
  - '-t'
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/myapps-repository/myapp1:$SHORT_SHA'
  - '.'

# This step pushes the image to Artifact Registry
# The PROJECT_ID and SHORT_SHA variables are automatically
# replaced by Cloud Build.
- name: 'gcr.io/cloud-builders/docker'
  id: Push
  args:
  - 'push'
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/myapps-repository/myapp1:$SHORT_SHA'
```

## Step-16: Copy cloudbuild.yaml to Git Repo
```t
# Change Directroy
cd 57-GKE-Continuous-Integration/03-cloudbuild-yaml

# Copy Files to Git repo
1. cloudbuild.yaml

# Local Git Commit and Push to Remote Repo
git add .
git commit -am "Third Commit"
git push

# Verify in Git Remote Repo
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo
```

## Step-17: Create Continuous Integration Pipeline in Cloud Build
- Go to Cloud Build -> Dashboard -> Region: us-central-1 -> Click on **SET UP BUILD TRIGGERS** [OR]
- Go to Cloud Build -> TRIGGERS -> Click on **CREATE TRIGGER**
- **Name:** myapp1-ci
- **Region:** us-central1
- **Description:** myapp1 Continuous Integration Pipeline
- **Tags:** environment=dev
- **Event:** Push to a branch
- **Source:** myapp1-app-repo
- **Branch:** main (Auto-populated)
- **Configuration:** Cloud Build configuration file (yaml or json)
- **Location:** Repository
- **Cloud Build Configuration file location:** /cloudbuild.yaml
- **Approval:** leave unchecked
- **Service account:** leave to default
- Click on **CREATE**

## Step-18: Make a simple change in "index.html" and push the changes to Git Repo
```t
# Change Directroy
cd myapp1-app-repo

# Update file index.html (change V1 to V2)
<p>Application Version: V2</p>

# Local Git Commit and Push to Remote Repo
git status
git add .
git commit -am "V2 Commit"
git push

# Verify in Git Remote Repo
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo
```

## Step-19: Verify Code Build CI Pipeline
```t
# Verify Code Build
1. Go to Code Build -> Dashboard or go directly to Code Build -> History
2. Click on Build History -> View All
3. Verify "BUILD LOG"
4. Verify "EXECUTION DETAILS"
5. Verify "VIEW RAW"

# Verify Artifact Repository
1. Go to Artifact Registry -> myapps-repository -> myapp1
2. You should find the docker image pushed to Artifact Registry
```

## Step-20: Review Kubernetes Manifests
- **Project Folder:** 04-kube-manifests
- 01-kubernetes-deployment.yaml
- 02-kubernetes-loadBalancer-service.yaml

## Step-21: Update Container Image to V1 Docker Image we built
```yaml
# 01-kubernetes-deployment.yaml: Update "image"
    spec:
      containers: # List
        - name: myapp1-container
          image: us-central1-docker.pkg.dev/kdaida123/myapps-repository/myapp1:d1c3b88
          ports:
            - containerPort: 80
```

## Step-22: Deploy Kubernetes Manifests and Verify
```t
# Change Directory
You should in Course Content folder
google-kubernetes-engine/<RESPECTIVE-SECTION>

# Deploy Kubernetes Manifests
kubectl apply -f 04-kube-manifests

# List Deployments
kubectl get deploy

# List Pods
kubectl get pods

# Describe Pod (Review Events to understand from where Docker Image downloaded)
kubectl describe pod <POD-NAME>

# List Services
kubectl get svc

# Access Application
http://<EXTERNAL-IP-GET-SERVICE-OUTPUT>
Observation:
1. You should see "Application Version: V1"
```

## Step-23: Update Container Image to V2 Docker Image we built
```yaml
# 01-kubernetes-deployment.yaml: Update "image"
    spec:
      containers: # List
        - name: myapp1-container
          image: us-central1-docker.pkg.dev/kdaida123/myapps-repository/myapp1:3af592c
          ports:
            - containerPort: 80
```

## Step-24: Update Kubernetes Deployment and Verify
```t
# Deply Kubernetes Manifests (Updated Image Tag)
kubectl apply -f 04-kube-manifests

# Restart Kubernetes Deployment (Optional - if it is not updated)
kubectl rollout restart deployment myapp1-deployment

# List Deployments
kubectl get deploy

# List Pods
kubectl get pods

# Describe Pod (Review Events to understand from where Docker Image downloaded)
kubectl describe pod <POD-NAME>

# List Services
kubectl get svc

# Access Application
http://<EXTERNAL-IP-GET-SERVICE-OUTPUT>
Observation:
1. You should see "Application Version: V2"
```

## Step-25: Clean-Up
```t
# Delete Kubernetes Resources
kubectl delete -f 04-kube-manifests
```

## Step-26: How to add Approvals before starting the Build Process ?
### Step-26-01: Enable Approval in Cloud Build
- Go to Cloud Build -> Triggers -> myapp1-ci
- Check the box in **Approval: Require approval before build executes**

### Step-26-02: Add Users to Cloud Build Approver IAM Role
- Go to IAM & Admin -> GRANT ACCESS
- **Add Principal:** dkalyanreddy@gmail.com
- **Assign Roles:** Cloud Build Approver
- Click on **SAVE**

## Step-27: Update the Git Repo to test Build Approval Process
```t
# Change Directroy
cd myapp1-app-repo

# Update file index.html (change V2 to V3)
<p>Application Version: V3</p>

# Local Git Commit and Push to Remote Repo
git status
git add .
git commit -am "V3 Commit"
git push

# Verify in Git Remote Repo
Search for -> Source Repositories
https://source.cloud.google.com/repos
Go to Repo -> myapp1-app-repo
```

## Step-28: Verify and Approve the Build
- Go to Cloud Build -> Triggers -> myapp1-ci -> Select and Approve
- Verify if build is successful.

## References
- [Cloud Build for Docker Images](https://cloud.google.com/kubernetes-engine/docs/tutorials/gitops-cloud-build

title  GCP Google Kubernetes Engine GKE CI description  Implement GCP Google Kubernetes Engine GKE Continuous Integration         Step 00  Pre requisites 1  Verify if GKE Cluster is created 2  Verify if kubeconfig for kubectl is configured in your local terminal    t   Configure kubeconfig for kubectl gcloud container clusters get credentials  CLUSTER NAME    region  REGION    project  PROJECT     Replace Values CLUSTER NAME  REGION  PROJECT gcloud container clusters get credentials standard cluster private 1   region us central1   project kdaida123    List Kubernetes Nodes kubectl get nodes          Step 01  Introduction   Implement Continuous Integration for GKE Workloads using   Google Cloud Source   Google Cloud Build   Google Artifact Repository      Step 02  Enable APIs in Google Cloud    t   Enable APIs in Google Cloud gcloud services enable container googleapis com       cloudbuild googleapis com       sourcerepo googleapis com       artifactregistry googleapis com    Google Cloud Services  GKE  container googleapis com      Cloud Build  cloudbuild googleapis com Cloud Source  sourcerepo googleapis com Artifact Registry  artifactregistry googleapis com         Step 03  Create Artifact Repository    t   List Artifact Repositories gcloud artifacts repositories list    Create Artifact Repository gcloud artifacts repositories create myapps repository       repository format docker       location us central1     List Artifact Repositories gcloud artifacts repositories list    Describe Artifact Repository  gcloud artifacts repositories describe myapps repository   location us central1         Step 04  Install Git client on local desktop  if not present     t   Download and Install Git Client and Installed https   git scm com downloads         Step 05  Create SSH Keys for Git Repo Access    Generating SSH Key Pair  https   cloud google com source repositories docs authentication generate a key pair     t   Change Directory cd 01 SSH Keys    Create SSH Keys ssh keygen  t  KEY TYPE   C   USER EMAIL   KEY TYPE  rsa  ecdsa  ed25519 USER EMAIL  dkalyanreddy gmail com     Replace Values KEY TYPE  USER EMAIL ssh keygen  t ed25519  C  dkalyanreddy gmail com  Provide the File Name as  id gcp cloud source      Sample Output Kalyans Mac mini 01 SSH Keys kalyanreddy  ssh keygen  t ed25519  C  dkalyanreddy gmail com  Generating public private ed25519 key pair  Enter file in which to save the key   Users kalyanreddy  ssh id ed25519   id gcp cloud source Enter passphrase  empty for no passphrase    Enter same passphrase again   Your identification has been saved in id gcp cloud source Your public key has been saved in id gcp cloud source pub The key fingerprint is  SHA256 YialyCj3XaSa4b8ewk4bcK1hXxO7DDM5uiCP1J2TOZ0 dkalyanreddy gmail com The key s randomart image is      ED25519 256                                                      o            o       o          o   B   S              B   X o               B Eo                 o                                      SHA256        Kalyans Mac mini 01 SSH Keys kalyanreddy  ls  lrta total 16 drwxr xr x  6 kalyanreddy  staff  192 Jun 29 09 45     rw         1 kalyanreddy  staff  419 Jun 29 09 46 id gcp cloud source drwxr xr x  4 kalyanreddy  staff  128 Jun 29 09 46    rw r  r    1 kalyanreddy  staff  104 Jun 29 09 46 id gcp cloud source pub Kalyans Mac mini 01 SSH Keys kalyanreddy           Step 06  Review SSH Keys  Public and Private Keys     t   Change Directroy  cd 01 SSH Keys    Review Private Key  id gcp cloud source cat id gcp cloud source    Review Public Key  id gcp cloud source pub  cat id gcp cloud source pub          Step 07  Update SSH Public Key in Google Cloud Source   Go to    Source Repositories    3 Dots    Manage SSH Keys    Register SSH Key    Google Cloud Source URL  https   source cloud google com      t   Key Name Name  gke course Key  Output from command  cat id gcp cloud source pub  in previous step  Put content from Public Key       Click on   Register        Step 08  Update SSH Private Key in Git Config   Update SSH Private Key in your local desktop Git Config    t   Copy SSH Private Key to your   ssh  folder in your Home Directory from your course cd 01 SSH Keys cp id gcp cloud source  HOME  ssh      Change Directory  Your local desktop home directory  cd  HOME  ssh      Verify File in   HOME  ssh  ls  lrta id gcp cloud source    Verify existing git  config  file cat config    Backup any existing  config  file cp config config bkup before cloud source    Update  config  file to point to  id gcp cloud source  private key vi config     Sample Output after changes Kalyans Mac mini  ssh kalyanreddy  cat config Host     AddKeysToAgent yes   UseKeychain yes   IdentityFile    ssh id gcp cloud source Kalyans Mac mini  ssh kalyanreddy      Backup config with cloudsource cp config config with cloud source key         Step 09  Update Git Global Config in your local deskopt    t   List Global Git Config git config   list    Update Global Git Config git config   global user email  YOUR EMAIL ADDRESS  git config   global user name  YOUR NAME     Replace YOUR EMAIL ADDRESS  YOUR NAME git config   global user name  Kalyan Reddy Daida  git config   global user email  dkalyanreddy gmail com     List Global Git Config git config   list         Step 10  Create Git repositories in Cloud Source    t   List Cloud Source Repository gcloud source repos list    Create Git repositories in Cloud Source gcloud source repos create myapp1 app repo    List Cloud Source Repository gcloud source repos list    Verify using Cloud Console Search for    Source Repositories  https   source cloud google com repos         Step 11  Clone Cloud Source Git Repository  Commit a Change  Push to Remote Repo and Verify    t   Change Directory  cd course repos    Verify using Cloud Console Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo    SSH Authentication    Copy the git clone command and run  git clone ssh   dkalyanreddy gmail com source developers google com 2022 p kdaida123 r myapp1 app repo    Change Directory cd myapp1 app repo    Create a simple readme file touch README md echo    GKE CI Demo    README md ls  lrta    Add Files and do local commit git add   git commit  am  First Commit     Push file to Cloud Source Git Repo  Remote Repo  git push    Verify in Git Remote Repo Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo          Step 12  Review Files in 02 Docker Image folder 1  Dockerfile 2  index html     Step 13  Copy files from 02 Docker Image folder to Git Repo    t   Change Directroy  cd 57 GKE Continuous Integration 02 Docker Image    Copy Files to Git repo  myapp1 app repo  1  Dockerfile 2  index html    Local Git Commit and Push to Remote Repo git add   git commit  am  Second Commit  git push    Verify in Git Remote Repo Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo          Step 14  Create a container image with Cloud Build and store it in Artifact Registry using glcoud builds command    t   Change Directory  Git App Repo  myapp1 app repo  cd myapp1 app repo    Get latest git commit id  current branch  git rev parse HEAD    Get latest git commit id first 7 chars  current branch  git rev parse   short 7 HEAD    Ensure you are in local git repo folder where  Dockerfile  index html  present cd myapp1 app repo     Create a Cloud Build build based on the latest commit  gcloud builds submit   tag  us central1 docker pkg dev   PROJECT ID     APP ARTIFACT REPO  myapp1   COMMIT ID        Replace Values   PROJECT ID      APP ARTIFACT REPO     COMMIT ID  gcloud builds submit   tag  us central1 docker pkg dev kdaida123 myapps repository myapp1 6f7d338            Step 15  Review Cloud Build YAML file    yaml steps    This step builds the container image    name   gcr io cloud builders docker    id  Build   args       build        t       us central1 docker pkg dev  PROJECT ID myapps repository myapp1  SHORT SHA             This step pushes the image to Artifact Registry   The PROJECT ID and SHORT SHA variables are automatically   replaced by Cloud Build    name   gcr io cloud builders docker    id  Push   args       push       us central1 docker pkg dev  PROJECT ID myapps repository myapp1  SHORT SHA          Step 16  Copy cloudbuild yaml to Git Repo    t   Change Directroy  cd 57 GKE Continuous Integration 03 cloudbuild yaml    Copy Files to Git repo 1  cloudbuild yaml    Local Git Commit and Push to Remote Repo git add   git commit  am  Third Commit  git push    Verify in Git Remote Repo Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo          Step 17  Create Continuous Integration Pipeline in Cloud Build   Go to Cloud Build    Dashboard    Region  us central 1    Click on   SET UP BUILD TRIGGERS    OR    Go to Cloud Build    TRIGGERS    Click on   CREATE TRIGGER        Name    myapp1 ci     Region    us central1     Description    myapp1 Continuous Integration Pipeline     Tags    environment dev     Event    Push to a branch     Source    myapp1 app repo     Branch    main  Auto populated      Configuration    Cloud Build configuration file  yaml or json      Location    Repository     Cloud Build Configuration file location     cloudbuild yaml     Approval    leave unchecked     Service account    leave to default   Click on   CREATE        Step 18  Make a simple change in  index html  and push the changes to Git Repo    t   Change Directroy  cd myapp1 app repo    Update file index html  change V1 to V2   p Application Version  V2  p     Local Git Commit and Push to Remote Repo git status git add   git commit  am  V2 Commit  git push    Verify in Git Remote Repo Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo          Step 19  Verify Code Build CI Pipeline    t   Verify Code Build 1  Go to Code Build    Dashboard or go directly to Code Build    History 2  Click on Build History    View All 3  Verify  BUILD LOG  4  Verify  EXECUTION DETAILS  5  Verify  VIEW RAW     Verify Artifact Repository 1  Go to Artifact Registry    myapps repository    myapp1 2  You should find the docker image pushed to Artifact Registry         Step 20  Review Kubernetes Manifests     Project Folder    04 kube manifests   01 kubernetes deployment yaml   02 kubernetes loadBalancer service yaml     Step 21  Update Container Image to V1 Docker Image we built    yaml   01 kubernetes deployment yaml  Update  image       spec        containers    List           name  myapp1 container           image  us central1 docker pkg dev kdaida123 myapps repository myapp1 d1c3b88           ports                 containerPort  80           Step 22  Deploy Kubernetes Manifests and Verify    t   Change Directory You should in Course Content folder  google kubernetes engine  RESPECTIVE SECTION     Deploy Kubernetes Manifests kubectl apply  f 04 kube manifests    List Deployments kubectl get deploy    List Pods kubectl get pods    Describe Pod  Review Events to understand from where Docker Image downloaded  kubectl describe pod  POD NAME     List Services kubectl get svc    Access Application http    EXTERNAL IP GET SERVICE OUTPUT  Observation  1  You should see  Application Version  V1          Step 23  Update Container Image to V2 Docker Image we built    yaml   01 kubernetes deployment yaml  Update  image       spec        containers    List           name  myapp1 container           image  us central1 docker pkg dev kdaida123 myapps repository myapp1 3af592c           ports                 containerPort  80           Step 24  Update Kubernetes Deployment and Verify    t   Deply Kubernetes Manifests  Updated Image Tag  kubectl apply  f 04 kube manifests    Restart Kubernetes Deployment  Optional   if it is not updated  kubectl rollout restart deployment myapp1 deployment    List Deployments kubectl get deploy    List Pods kubectl get pods    Describe Pod  Review Events to understand from where Docker Image downloaded  kubectl describe pod  POD NAME     List Services kubectl get svc    Access Application http    EXTERNAL IP GET SERVICE OUTPUT  Observation  1  You should see  Application Version  V2          Step 25  Clean Up    t   Delete Kubernetes Resources kubectl delete  f 04 kube manifests         Step 26  How to add Approvals before starting the Build Process       Step 26 01  Enable Approval in Cloud Build   Go to Cloud Build    Triggers    myapp1 ci   Check the box in   Approval  Require approval before build executes        Step 26 02  Add Users to Cloud Build Approver IAM Role   Go to IAM   Admin    GRANT ACCESS      Add Principal    dkalyanreddy gmail com     Assign Roles    Cloud Build Approver   Click on   SAVE       Step 27  Update the Git Repo to test Build Approval Process    t   Change Directroy  cd myapp1 app repo    Update file index html  change V2 to V3   p Application Version  V3  p     Local Git Commit and Push to Remote Repo git status git add   git commit  am  V3 Commit  git push    Verify in Git Remote Repo Search for    Source Repositories  https   source cloud google com repos Go to Repo    myapp1 app repo          Step 28  Verify and Approve the Build   Go to Cloud Build    Triggers    myapp1 ci    Select and Approve   Verify if build is successful         References    Cloud Build for Docker Images  https   cloud google com kubernetes engine docs tutorials gitops cloud build


gcp gke docs title Kubernetes ReplicaSets Step 02 Create ReplicaSet Learn about Kubernetes ReplicaSets What are ReplicaSets What is the advantage of using ReplicaSets Step 01 Introduction to ReplicaSets

Answer:
---
title: Kubernetes ReplicaSets
description: Learn about Kubernetes ReplicaSets
---

## Step-01: Introduction to ReplicaSets
- What are ReplicaSets?
- What is the advantage of using ReplicaSets?

## Step-02: Create ReplicaSet

### Step-02-01: Create ReplicaSet
- Create ReplicaSet
```t
# Kubernetes ReplicaSet
kubectl create -f replicaset-demo.yml
```
- **replicaset-demo.yml**
```yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-helloworld-rs
  labels:
    app: my-helloworld
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-helloworld
  template:
    metadata:
      labels:
        app: my-helloworld
    spec:
      containers:
      - name: my-helloworld-app
        image: stacksimplify/kube-helloworld:1.0.0
```

### Step-02-02: List ReplicaSets
- Get list of ReplicaSets
```t
# List ReplicaSets
kubectl get replicaset
kubectl get rs
```

### Step-02-03: Describe ReplicaSet
- Describe the newly created ReplicaSet
```t
# Describe ReplicaSet
kubectl describe rs/<replicaset-name>

kubectl describe rs/my-helloworld-rs
[or]
kubectl describe rs my-helloworld-rs
```

### Step-02-04: List of Pods
- Get list of Pods
```t
# Get list of Pods
kubectl get pods
kubectl describe pod <pod-name>

# Get list of Pods with Pod IP and Node in which it is running
kubectl get pods -o wide
```

### Step-02-05: Verify the Owner of the Pod
- Verify the owner reference of the pod.
- Verify under **"name"** tag under **"ownerReferences"**. We will find the replicaset name to which this pod belongs to.
```t
# List Pod with Output as YAML
kubectl get pods <pod-name> -o yaml
kubectl get pods my-helloworld-rs-c8rrj -o yaml
```

## Step-03: Expose ReplicaSet as a Service
- Expose ReplicaSet with a service (Load Balancer Service) to access the application externally (from internet)
```t
# Expose ReplicaSet as a Service
kubectl expose rs <ReplicaSet-Name>  --type=LoadBalancer --port=80 --target-port=8080 --name=<Service-Name-To-Be-Created>
kubectl expose rs my-helloworld-rs  --type=LoadBalancer --port=80 --target-port=8080 --name=my-helloworld-rs-service

# List Services
kubectl get service
kubectl get svc
```
- **Access the Application using External or Public IP**
```t
# Access Application
http://<External-IP-from-get-service-output>/hello
curl http://<External-IP-from-get-service-output>/hello

# Observation
1. Each time we access the application, request will be sent to different pod and pods id will be displayed for us.
```

## Step-04: Test Replicaset Reliability or High Availability
- Test how the high availability or reliability concept is achieved automatically in Kubernetes
- Whenever a POD is accidentally terminated due to some application issue, ReplicaSet should auto-create that Pod to maintain desired number of Replicas configured to achive High Availability.
```t
# To get Pod Name
kubectl get pods

# Delete the Pod
kubectl delete pod <Pod-Name>

# Verify the new pod got created automatically
kubectl get pods   (Verify Age and name of new pod)
```

## Step-05: Test ReplicaSet Scalability feature
- Test how scalability is going to seamless & quick
- Update the **replicas** field in **replicaset-demo.yml** from 3 to 6.
```yaml
# Before change
spec:
  replicas: 3

# After change
spec:
  replicas: 6
```
- Update the ReplicaSet
```t
# Apply latest changes to ReplicaSet
kubectl replace -f replicaset-demo.yml

# Verify if new pods got created
kubectl get pods -o wide
```

## Step-06: Delete ReplicaSet & Service
### Step-06-01: Delete ReplicaSet
```t
# Delete ReplicaSet
kubectl delete rs <ReplicaSet-Name>

# Sample Commands
kubectl delete rs/my-helloworld-rs
[or]
kubectl delete rs my-helloworld-rs

# Verify if ReplicaSet got deleted
kubectl get rs
```

### Step-06-02: Delete Service created for ReplicaSet
```t
# Delete Service
kubectl delete svc <service-name>

# Sample Commands
kubectl delete svc my-helloworld-rs-service
[or]
kubectl delete svc/my-helloworld-rs-service

# Verify if Service got deleted
kubectl get svc
```

title  Kubernetes ReplicaSets description  Learn about Kubernetes ReplicaSets         Step 01  Introduction to ReplicaSets   What are ReplicaSets    What is the advantage of using ReplicaSets      Step 02  Create ReplicaSet      Step 02 01  Create ReplicaSet   Create ReplicaSet    t   Kubernetes ReplicaSet kubectl create  f replicaset demo yml         replicaset demo yml      yaml apiVersion  apps v1 kind  ReplicaSet metadata    name  my helloworld rs   labels      app  my helloworld spec    replicas  3   selector      matchLabels        app  my helloworld   template      metadata        labels          app  my helloworld     spec        containers          name  my helloworld app         image  stacksimplify kube helloworld 1 0 0          Step 02 02  List ReplicaSets   Get list of ReplicaSets    t   List ReplicaSets kubectl get replicaset kubectl get rs          Step 02 03  Describe ReplicaSet   Describe the newly created ReplicaSet    t   Describe ReplicaSet kubectl describe rs  replicaset name   kubectl describe rs my helloworld rs  or  kubectl describe rs my helloworld rs          Step 02 04  List of Pods   Get list of Pods    t   Get list of Pods kubectl get pods kubectl describe pod  pod name     Get list of Pods with Pod IP and Node in which it is running kubectl get pods  o wide          Step 02 05  Verify the Owner of the Pod   Verify the owner reference of the pod    Verify under    name    tag under    ownerReferences     We will find the replicaset name to which this pod belongs to      t   List Pod with Output as YAML kubectl get pods  pod name   o yaml kubectl get pods my helloworld rs c8rrj  o yaml          Step 03  Expose ReplicaSet as a Service   Expose ReplicaSet with a service  Load Balancer Service  to access the application externally  from internet     t   Expose ReplicaSet as a Service kubectl expose rs  ReplicaSet Name     type LoadBalancer   port 80   target port 8080   name  Service Name To Be Created  kubectl expose rs my helloworld rs    type LoadBalancer   port 80   target port 8080   name my helloworld rs service    List Services kubectl get service kubectl get svc         Access the Application using External or Public IP      t   Access Application http    External IP from get service output  hello curl http    External IP from get service output  hello    Observation 1  Each time we access the application  request will be sent to different pod and pods id will be displayed for us           Step 04  Test Replicaset Reliability or High Availability    Test how the high availability or reliability concept is achieved automatically in Kubernetes   Whenever a POD is accidentally terminated due to some application issue  ReplicaSet should auto create that Pod to maintain desired number of Replicas configured to achive High Availability     t   To get Pod Name kubectl get pods    Delete the Pod kubectl delete pod  Pod Name     Verify the new pod got created automatically kubectl get pods    Verify Age and name of new pod           Step 05  Test ReplicaSet Scalability feature    Test how scalability is going to seamless   quick   Update the   replicas   field in   replicaset demo yml   from 3 to 6     yaml   Before change spec    replicas  3    After change spec    replicas  6       Update the ReplicaSet    t   Apply latest changes to ReplicaSet kubectl replace  f replicaset demo yml    Verify if new pods got created kubectl get pods  o wide         Step 06  Delete ReplicaSet   Service     Step 06 01  Delete ReplicaSet    t   Delete ReplicaSet kubectl delete rs  ReplicaSet Name     Sample Commands kubectl delete rs my helloworld rs  or  kubectl delete rs my helloworld rs    Verify if ReplicaSet got deleted kubectl get rs          Step 06 02  Delete Service created for ReplicaSet    t   Delete Service kubectl delete svc  service name     Sample Commands kubectl delete svc my helloworld rs service  or  kubectl delete svc my helloworld rs service    Verify if Service got deleted kubectl get svc


gcp gke docs Step 00 Pre requisites title GCP Google Kubernetes Engine GKE Ingress SSL Redirect 2 Verify if kubeconfig for kubectl is configured in your local terminal Configure kubeconfig for kubectl Implement GCP Google Kubernetes Engine GKE Ingress SSL Redirect 1 Verify if GKE Cluster is created gcloud container clusters get credentials CLUSTER NAME region REGION project PROJECT t

Answer:
---
title: GCP Google Kubernetes Engine GKE Ingress SSL Redirect
description: Implement GCP Google Kubernetes Engine GKE Ingress SSL Redirect
---
## Step-00: Pre-requisites
1. Verify if GKE Cluster is created
2. Verify if kubeconfig for kubectl is configured in your local terminal
```t
# Configure kubeconfig for kubectl
gcloud container clusters get-credentials <CLUSTER-NAME> --region <REGION> --project <PROJECT>

# Replace Values CLUSTER-NAME, ZONE, PROJECT
gcloud container clusters get-credentials standard-cluster-private-1 --region us-central1 --project kdaida123
```
3. Registered Domain using Google Cloud Domains
4. DNS name for which SSL Certificate should be created should already be added as DNS in Google Cloud DNS (Example: demo1.kalyanreddydaida.com)

## Step-01: Introduction
- Google Managed Certificates for GKE Ingress
- Ingress SSL
- Ingress SSL Redirect (HTTP to HTTPS)

## Step-02: 06-frontendconfig.yaml
```yaml
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: my-frontend-config
spec:
  redirectToHttps:
    enabled: true
    #responseCodeName: RESPONSE_CODE
```

## Step-03: 04-Ingress-SSL.yaml
- Add the Annotation `networking.gke.io/v1beta1.FrontendConfig: "my-frontend-config"`
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-ssl
  annotations:
    # External Load Balancer
    kubernetes.io/ingress.class: "gce"
    # Static IP for Ingress Service
    kubernetes.io/ingress.global-static-ip-name: "gke-ingress-extip1"
    # Google Managed SSL Certificates
    networking.gke.io/managed-certificates: managed-cert-for-ingress
    # SSL Redirect HTTP to HTTPS
    networking.gke.io/v1beta1.FrontendConfig: "my-frontend-config"
spec:
  defaultBackend:
    service:
      name: app3-nginx-nodeport-service
      port:
        number: 80
  rules:
    - http:
        paths:
          - path: /app1
            pathType: Prefix
            backend:
              service:
                name: app1-nginx-nodeport-service
                port:
                  number: 80
          - path: /app2
            pathType: Prefix
            backend:
              service:
                name: app2-nginx-nodeport-service
                port:
                  number: 80
```

## Step-04: Deploy kube-manifests and Verify
- From previous `Ingress SSL` demo we didn't clean-up those Kubernetes resources.
- We are going use them here, in addition to previous demo in this demo we are just adding `06-frontendconfig.yaml`
```t
# Deploy Kubernetes manifests
kubectl apply -f kube-manifests
Observation:
1. Only "my-frontend-config" will be created, rest all unchanged

### Sample Output
Kalyans-Mac-mini:38-GKE-Ingress-Google-Managed-SSL-Redirect kalyanreddy$ kubectl apply -f kube-manifests/
deployment.apps/app1-nginx-deployment unchanged
service/app1-nginx-nodeport-service unchanged
deployment.apps/app2-nginx-deployment unchanged
service/app2-nginx-nodeport-service unchanged
deployment.apps/app3-nginx-deployment unchanged
service/app3-nginx-nodeport-service unchanged
ingress.networking.k8s.io/ingress-ssl configured
managedcertificate.networking.gke.io/managed-cert-for-ingress unchanged
frontendconfig.networking.gke.io/my-frontend-config created
Kalyans-Mac-mini:38-GKE-Ingress-Google-Managed-SSL-Redirect kalyanreddy$

# List FrontendConfig
kubectl get frontendconfig

# Describe FrontendConfig
kubectl describe frontendconfig my-frontend-config

# List Ingress Load Balancers
kubectl get ingress

# Describe Ingress and view Rules
kubectl describe ingress ingress-ssl
```

## Step-05: Access Application
```t
# Important Note
Wait for 2 to 3 minutes for the Load Balancer to completely create and ready for use else we will get HTTP 502 errors

# Access Application
http://<DNS-DOMAIN-NAME>/app1/index.html
http://<DNS-DOMAIN-NAME>/app2/index.html
http://<DNS-DOMAIN-NAME>/

# Note: Replace Domain Name registered in Cloud DNS
# HTTP URLs: Should redirect to HTTPS URLs
http://demo1.kalyanreddydaida.com/app1/index.html
http://demo1.kalyanreddydaida.com/app2/index.html
http://demo1.kalyanreddydaida.com/

# HTTPS URLs
https://demo1.kalyanreddydaida.com/app1/index.html
https://demo1.kalyanreddydaida.com/app2/index.html
https://demo1.kalyanreddydaida.com/
```

## Step-06: Clean Up
```t
# Delete Kubernetes Resources
kubectl delete -f kube-manifests

# Verify Load Balancer Deleted
Go to Network Services -> Load Balancing -> No Load balancers should be present
```

title  GCP Google Kubernetes Engine GKE Ingress SSL Redirect description  Implement GCP Google Kubernetes Engine GKE Ingress SSL Redirect        Step 00  Pre requisites 1  Verify if GKE Cluster is created 2  Verify if kubeconfig for kubectl is configured in your local terminal    t   Configure kubeconfig for kubectl gcloud container clusters get credentials  CLUSTER NAME    region  REGION    project  PROJECT     Replace Values CLUSTER NAME  ZONE  PROJECT gcloud container clusters get credentials standard cluster private 1   region us central1   project kdaida123     3  Registered Domain using Google Cloud Domains 4  DNS name for which SSL Certificate should be created should already be added as DNS in Google Cloud DNS  Example  demo1 kalyanreddydaida com       Step 01  Introduction   Google Managed Certificates for GKE Ingress   Ingress SSL   Ingress SSL Redirect  HTTP to HTTPS      Step 02  06 frontendconfig yaml    yaml apiVersion  networking gke io v1beta1 kind  FrontendConfig metadata    name  my frontend config spec    redirectToHttps      enabled  true      responseCodeName  RESPONSE CODE         Step 03  04 Ingress SSL yaml   Add the Annotation  networking gke io v1beta1 FrontendConfig   my frontend config      yaml apiVersion  networking k8s io v1 kind  Ingress metadata    name  ingress ssl   annotations        External Load Balancer     kubernetes io ingress class   gce          Static IP for Ingress Service     kubernetes io ingress global static ip name   gke ingress extip1           Google Managed SSL Certificates     networking gke io managed certificates  managed cert for ingress       SSL Redirect HTTP to HTTPS     networking gke io v1beta1 FrontendConfig   my frontend config      spec     defaultBackend      service        name  app3 nginx nodeport service       port          number  80                               rules        http          paths                         path   app1             pathType  Prefix             backend                service                  name  app1 nginx nodeport service                 port                     number  80             path   app2             pathType  Prefix             backend                service                  name  app2 nginx nodeport service                 port                     number  80                            Step 04  Deploy kube manifests and Verify   From previous  Ingress SSL  demo we didn t clean up those Kubernetes resources    We are going use them here  in addition to previous demo in this demo we are just adding  06 frontendconfig yaml     t   Deploy Kubernetes manifests kubectl apply  f kube manifests Observation  1  Only  my frontend config  will be created  rest all unchanged      Sample Output Kalyans Mac mini 38 GKE Ingress Google Managed SSL Redirect kalyanreddy  kubectl apply  f kube manifests  deployment apps app1 nginx deployment unchanged service app1 nginx nodeport service unchanged deployment apps app2 nginx deployment unchanged service app2 nginx nodeport service unchanged deployment apps app3 nginx deployment unchanged service app3 nginx nodeport service unchanged ingress networking k8s io ingress ssl configured managedcertificate networking gke io managed cert for ingress unchanged frontendconfig networking gke io my frontend config created   Kalyans Mac mini 38 GKE Ingress Google Managed SSL Redirect kalyanreddy       List FrontendConfig kubectl get frontendconfig    Describe FrontendConfig kubectl describe frontendconfig my frontend config    List Ingress Load Balancers kubectl get ingress    Describe Ingress and view Rules kubectl describe ingress ingress ssl          Step 05  Access Application    t   Important Note Wait for 2 to 3 minutes for the Load Balancer to completely create and ready for use else we will get HTTP 502 errors    Access Application http    DNS DOMAIN NAME  app1 index html http    DNS DOMAIN NAME  app2 index html http    DNS DOMAIN NAME      Note  Replace Domain Name registered in Cloud DNS   HTTP URLs  Should redirect to HTTPS URLs  http   demo1 kalyanreddydaida com app1 index html http   demo1 kalyanreddydaida com app2 index html http   demo1 kalyanreddydaida com     HTTPS URLs https   demo1 kalyanreddydaida com app1 index html https   demo1 kalyanreddydaida com app2 index html https   demo1 kalyanreddydaida com          Step 06  Clean Up    t   Delete Kubernetes Resources kubectl delete  f kube manifests    Verify Load Balancer Deleted Go to Network Services    Load Balancing    No Load balancers should be present


gcp gke docs Step 00 Pre requisites title GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity 2 Verify if kubeconfig for kubectl is configured in your local terminal Implement GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity Configure kubeconfig for kubectl 1 Verify if GKE Cluster is created t

Answer:
---
title: GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity
description: Implement GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity
---

## Step-00: Pre-requisites
1. Verify if GKE Cluster is created
2. Verify if kubeconfig for kubectl is configured in your local terminal
```t
# Configure kubeconfig for kubectl
gcloud container clusters get-credentials <CLUSTER-NAME> --region <REGION> --project <PROJECT>

# Replace Values CLUSTER-NAME, REGION, PROJECT
gcloud container clusters get-credentials standard-cluster-private-1 --region us-central1 --project kdaida123

# List Kubernetes Nodes
kubectl get nodes
```

3. ExternalDNS Controller should be installed and ready to use
```t
# List Namespaces (external-dns-ns namespace should be present)
kubectl get ns

# List External DNS Pods
kubectl -n external-dns-ns get pods
```

## Step-01: Introduction
- Implement following Features for Ingress Service
- BackendConfig - CLIENT_IP Affinity for Ingress Service
- We are going to create two projects
  - **Project-01:** CLIENT_IP Affinity enabled
  - **Project-02:** CLIENT_IP Affinity disabled

## Step-02: Create External IP Address using gcloud
```t
# Create External IP Address 1 (IF NOT CREATED - ALREADY CREATED IN PREVIOUS SECTIONS)
gcloud compute addresses create ADDRESS_NAME --global
gcloud compute addresses create gke-ingress-extip1 --global

# Create External IP Address 2
gcloud compute addresses create ADDRESS_NAME --global
gcloud compute addresses create gke-ingress-extip2 --global

# Describe External IP Address to get
gcloud compute addresses describe ADDRESS_NAME --global
gcloud compute addresses describe gke-ingress-extip2 --global

# Verify
Go to VPC Network -> IP Addresses -> External IP Address
```

## Step-03: Project-01: Review YAML Manifests
- **Project Folder:** 01-kube-manifests-with-clientip-affinity
- 01-kubernetes-deployment.yaml
- 02-kubernetes-NodePort-service.yaml
- 03-ingress.yaml
- 04-backendconfig.yaml
```yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 42 # Backend service timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#timeout
  connectionDraining: # Connection draining timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#draining_timeout
    drainingTimeoutSec: 62
  logging: # HTTP access logging: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#http_logging
    enable: true
    sampleRate: 1.0
  sessionAffinity:
    affinityType: "CLIENT_IP"
```

## Step-04: Project-02: Review YAML Manifests
- **Project Folder:** 02-kube-manifests-without-clientip-affinity
- 01-kubernetes-deployment.yaml
- 02-kubernetes-NodePort-service.yaml
- 03-ingress.yaml
- 04-backendconfig.yaml
```yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig2
spec:
  timeoutSec: 42 # Backend service timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#timeout
  connectionDraining: # Connection draining timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#draining_timeout
    drainingTimeoutSec: 62
  logging: # HTTP access logging: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#http_logging
    enable: true
    sampleRate: 1.0
```
## Step-05: Deploy Kubernetes Manifests
```t
# Project-01: Deploy Kubernetes Manifests
kubectl apply -f 01-kube-manifests-with-clientip-affinity

# Project-02: Deploy Kubernetes Manifests
kubectl apply -f 02-kube-manifests-without-clientip-affinity

# Verify Deployments
kubectl get deploy

# Verify Pods
kubectl get pods

# Verify Node Port Services
kubectl get svc

# Verify Ingress Services
kubectl get ingress

# Verify Backend Config
kubectl get backendconfig

# Project-01: Verify Load Balancer Settings
Go to Network Services -> Load Balancing -> Load Balancer -> Backends -> Verify Client IP Affinity Setting
Observation:
Client IP Affinity setting should be in enabled state

# Project-02: Verify Load Balancer Settings
Go to Network Services -> Load Balancing -> Load Balancer -> Backends -> Verify Client IP Affinity Setting
Client IP Affinity setting should be in disabled state
```

## Step-06: Access Application
```t
# Project-01: Access Application using DNS or ExtIP
http://ingress-with-clientip-affinity.kalyanreddydaida.com
http://<EXT-IP-1>
curl ingress-with-clientip-affinity.kalyanreddydaida.com
Observation:
1. Request will keep going always to only one POD due to CLIENT_IP Affinity we configured

# Project-02: Access Application using DNS or ExtIP
http://ingress-without-clientip-affinity.kalyanreddydaida.com
http://<EXT-IP-2>
curl ingress-without-clientip-affinity.kalyanreddydaida.com
Observation:
1. Requests will be load balanced to 4 pods created as part of "cdn-demo2" deployment.
```

## Step-07: How to remove a setting from FrontendConfig or BackendConfig
- To revoke an Ingress feature, you must explicitly disable the feature configuration in the FrontendConfig or BackendConfig CRD
- **Important Note:** To clear or disable a previously enabled configuration, set the field's value to an empty string ("") or to a Boolean value of false, depending on the field type.
```yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 42 # Backend service timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#timeout
  connectionDraining: # Connection draining timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#draining_timeout
    drainingTimeoutSec: 62
  logging: # HTTP access logging: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#http_logging
    enable: true
    sampleRate: 1.0
  sessionAffinity:
    #affinityType: "CLIENT_IP"  # Disable at Step-07
    affinityType: ""          # Enable at Step-07
```

## Step-08: Apply Changes and Verify
```t
# Apply Changes
kubectl apply -f 01-kube-manifests-with-clientip-affinity

# Verify Load Balancer
Go to Network Services -> Load Balancing -> Load Balancer -> Backends -> Verify Client IP Affinity Setting
Observation:
Should be disabled
```

## Step-09: Deleting a FrontendConfig or BackendConfig
- [Deleting a FrontendConfig or BackendConfig](https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#deleting_a_frontendconfig_or_backendconfig)

## Step-10: Clean-Up
```t
# Project-01: Delete Kubernetes Resources
kubectl delete -f 01-kube-manifests-with-clientip-affinity

# Project-02: Delete Kubernetes Resources
kubectl delete -f 02-kube-manifests-without-clientip-affinity
```

## Step-11: Rollback 04-backendconfig.yaml
- Put back `affinityType: "CLIENT_IP"` it will be ready for Students Demo.
```yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 42 # Backend service timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#timeout
  connectionDraining: # Connection draining timeout: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#draining_timeout
    drainingTimeoutSec: 62
  logging: # HTTP access logging: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#http_logging
    enable: true
    sampleRate: 1.0
  sessionAffinity:
    affinityType: "CLIENT_IP"  # Disable at Step-07
    #affinityType: ""          # Enable at Step-07
```

## References
- [Ingress Features](https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features)

title  GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity description  Implement GCP Google Kubernetes Engine GKE Ingress ClientIP Affinity         Step 00  Pre requisites 1  Verify if GKE Cluster is created 2  Verify if kubeconfig for kubectl is configured in your local terminal    t   Configure kubeconfig for kubectl gcloud container clusters get credentials  CLUSTER NAME    region  REGION    project  PROJECT     Replace Values CLUSTER NAME  REGION  PROJECT gcloud container clusters get credentials standard cluster private 1   region us central1   project kdaida123    List Kubernetes Nodes kubectl get nodes      3  ExternalDNS Controller should be installed and ready to use    t   List Namespaces  external dns ns namespace should be present  kubectl get ns    List External DNS Pods kubectl  n external dns ns get pods         Step 01  Introduction   Implement following Features for Ingress Service   BackendConfig   CLIENT IP Affinity for Ingress Service   We are going to create two projects       Project 01    CLIENT IP Affinity enabled       Project 02    CLIENT IP Affinity disabled     Step 02  Create External IP Address using gcloud    t   Create External IP Address 1  IF NOT CREATED   ALREADY CREATED IN PREVIOUS SECTIONS  gcloud compute addresses create ADDRESS NAME   global gcloud compute addresses create gke ingress extip1   global    Create External IP Address 2 gcloud compute addresses create ADDRESS NAME   global gcloud compute addresses create gke ingress extip2   global    Describe External IP Address to get gcloud compute addresses describe ADDRESS NAME   global gcloud compute addresses describe gke ingress extip2   global    Verify Go to VPC Network    IP Addresses    External IP Address         Step 03  Project 01  Review YAML Manifests     Project Folder    01 kube manifests with clientip affinity   01 kubernetes deployment yaml   02 kubernetes NodePort service yaml   03 ingress yaml   04 backendconfig yaml    yaml apiVersion  cloud google com v1 kind  BackendConfig metadata    name  my backendconfig spec    timeoutSec  42   Backend service timeout  https   cloud google com kubernetes engine docs how to ingress features timeout   connectionDraining    Connection draining timeout  https   cloud google com kubernetes engine docs how to ingress features draining timeout     drainingTimeoutSec  62   logging    HTTP access logging  https   cloud google com kubernetes engine docs how to ingress features http logging     enable  true     sampleRate  1 0   sessionAffinity      affinityType   CLIENT IP          Step 04  Project 02  Review YAML Manifests     Project Folder    02 kube manifests without clientip affinity   01 kubernetes deployment yaml   02 kubernetes NodePort service yaml   03 ingress yaml   04 backendconfig yaml    yaml apiVersion  cloud google com v1 kind  BackendConfig metadata    name  my backendconfig2 spec    timeoutSec  42   Backend service timeout  https   cloud google com kubernetes engine docs how to ingress features timeout   connectionDraining    Connection draining timeout  https   cloud google com kubernetes engine docs how to ingress features draining timeout     drainingTimeoutSec  62   logging    HTTP access logging  https   cloud google com kubernetes engine docs how to ingress features http logging     enable  true     sampleRate  1 0        Step 05  Deploy Kubernetes Manifests    t   Project 01  Deploy Kubernetes Manifests  kubectl apply  f 01 kube manifests with clientip affinity    Project 02  Deploy Kubernetes Manifests  kubectl apply  f 02 kube manifests without clientip affinity    Verify Deployments kubectl get deploy     Verify Pods kubectl get pods    Verify Node Port Services kubectl get svc    Verify Ingress Services kubectl get ingress    Verify Backend Config kubectl get backendconfig    Project 01  Verify Load Balancer Settings Go to Network Services    Load Balancing    Load Balancer    Backends    Verify Client IP Affinity Setting Observation  Client IP Affinity setting should be in enabled state    Project 02  Verify Load Balancer Settings Go to Network Services    Load Balancing    Load Balancer    Backends    Verify Client IP Affinity Setting Client IP Affinity setting should be in disabled state         Step 06  Access Application    t   Project 01  Access Application using DNS or ExtIP http   ingress with clientip affinity kalyanreddydaida com http    EXT IP 1  curl ingress with clientip affinity kalyanreddydaida com Observation  1  Request will keep going always to only one POD due to CLIENT IP Affinity we configured    Project 02  Access Application using DNS or ExtIP http   ingress without clientip affinity kalyanreddydaida com http    EXT IP 2  curl ingress without clientip affinity kalyanreddydaida com Observation  1  Requests will be load balanced to 4 pods created as part of  cdn demo2  deployment          Step 07  How to remove a setting from FrontendConfig or BackendConfig   To revoke an Ingress feature  you must explicitly disable the feature configuration in the FrontendConfig or BackendConfig CRD     Important Note    To clear or disable a previously enabled configuration  set the field s value to an empty string      or to a Boolean value of false  depending on the field type     yaml apiVersion  cloud google com v1 kind  BackendConfig metadata    name  my backendconfig spec    timeoutSec  42   Backend service timeout  https   cloud google com kubernetes engine docs how to ingress features timeout   connectionDraining    Connection draining timeout  https   cloud google com kubernetes engine docs how to ingress features draining timeout     drainingTimeoutSec  62   logging    HTTP access logging  https   cloud google com kubernetes engine docs how to ingress features http logging     enable  true     sampleRate  1 0   sessionAffinity       affinityType   CLIENT IP     Disable at Step 07     affinityType                Enable at Step 07         Step 08  Apply Changes and Verify    t   Apply Changes kubectl apply  f 01 kube manifests with clientip affinity    Verify Load Balancer  Go to Network Services    Load Balancing    Load Balancer    Backends    Verify Client IP Affinity Setting Observation  Should be disabled         Step 09  Deleting a FrontendConfig or BackendConfig    Deleting a FrontendConfig or BackendConfig  https   cloud google com kubernetes engine docs how to ingress features deleting a frontendconfig or backendconfig      Step 10  Clean Up    t   Project 01  Delete Kubernetes Resources  kubectl delete  f 01 kube manifests with clientip affinity    Project 02  Delete Kubernetes Resources  kubectl delete  f 02 kube manifests without clientip affinity         Step 11  Rollback 04 backendconfig yaml   Put back  affinityType   CLIENT IP   it will be ready for Students Demo     yaml apiVersion  cloud google com v1 kind  BackendConfig metadata    name  my backendconfig spec    timeoutSec  42   Backend service timeout  https   cloud google com kubernetes engine docs how to ingress features timeout   connectionDraining    Connection draining timeout  https   cloud google com kubernetes engine docs how to ingress features draining timeout     drainingTimeoutSec  62   logging    HTTP access logging  https   cloud google com kubernetes engine docs how to ingress features http logging     enable  true     sampleRate  1 0   sessionAffinity      affinityType   CLIENT IP     Disable at Step 07      affinityType                Enable at Step 07           References    Ingress Features  https   cloud google com kubernetes engine docs how to ingress features


gcp gke docs Learn to write and test Kubernetes Pods with YAML yml Step 01 Kubernetes YAML Top level Objects title Kubernetes Pods with YAML apiVersion Discuss about the k8s YAML top level objects kube base definition yml

Answer:
---
title: Kubernetes Pods with YAML
description: Learn to write and test Kubernetes Pods with YAML
---

## Step-01: Kubernetes YAML Top level Objects
- Discuss about the k8s YAML top level objects
- **kube-base-definition.yml**
```yml
apiVersion:
kind:
metadata:

spec:
```
- [Kubernetes Reference](https://kubernetes.io/docs/reference/)
- [Kubernetes API Reference](https://kubernetes.io/docs/reference/kubernetes-api/)
-  [Pod API Objects Reference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core)

## Step-02: Create Simple Pod Definition using YAML
- We are going to create a very basic pod definition
- **01-pod-definition.yml**
```yaml
apiVersion: v1 # String
kind: Pod  # String
metadata: # Dictionary
  name: myapp-pod
  labels: # Dictionary
    app: myapp
spec:
  containers: # List
    - name: myapp
      image: stacksimplify/kubenginx:1.0.0
      ports:
        - containerPort: 80
```
- **Create Pod**
```t
# Change Directory
cd kube-manifests

# Create Pod
kubectl create -f 01-pod-definition.yml
[or]
kubectl apply -f 01-pod-definition.yml

# List Pods
kubectl get pods
```

## Step-03: Create a LoadBalancer Service
- **02-pod-LoadBalancer-service.yml**
```yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp-pod-loadbalancer-service  # Name of the Service
spec:
  type: LoadBalancer
  selector:
  # Loadbalance traffic across Pods matching this label selector
    app: myapp
  # Accept traffic sent to port 80
  ports:
    - name: http
      port: 80    # Service Port
      targetPort: 80 # Container Port
```
- **Create LoadBalancer Service for Pod**
```t
# Create Service
kubectl apply -f 02-pod-LoadBalancer-service.yml

# List Service
kubectl get svc

# Access Application
http://<Load-Balancer-Service-IP>
curl http://<Load-Balancer-Service-IP>
```

## Step-04: Clean-Up Kubernetes Pod and Service
```t
# Change Directory
cd kube-manifests

# Delete Pod
kubectl delete -f 01-pod-definition.yml

# Delete Service
kubectl delete -f  02-pod-LoadBalancer-service.yml
```

## API Object References
- [Kubernetes API Spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/)
- [Pod Spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core)
- [Service Spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core)
- [Kubernetes API Reference](https://kubernetes.io/docs/reference/kubernetes-api/)

title  Kubernetes Pods with YAML description  Learn to write and test Kubernetes Pods with YAML         Step 01  Kubernetes YAML Top level Objects   Discuss about the k8s YAML top level objects     kube base definition yml      yml apiVersion  kind  metadata     spec         Kubernetes Reference  https   kubernetes io docs reference      Kubernetes API Reference  https   kubernetes io docs reference kubernetes api       Pod API Objects Reference  https   kubernetes io docs reference generated kubernetes api v1 24  pod v1 core      Step 02  Create Simple Pod Definition using YAML    We are going to create a very basic pod definition     01 pod definition yml      yaml apiVersion  v1   String kind  Pod    String metadata    Dictionary   name  myapp pod   labels    Dictionary      app  myapp          spec    containers    List       name  myapp       image  stacksimplify kubenginx 1 0 0       ports            containerPort  80         Create Pod      t   Change Directory cd kube manifests    Create Pod kubectl create  f 01 pod definition yml  or  kubectl apply  f 01 pod definition yml    List Pods kubectl get pods         Step 03  Create a LoadBalancer Service     02 pod LoadBalancer service yml      yaml apiVersion  v1 kind  Service metadata    name  myapp pod loadbalancer service    Name of the Service spec    type  LoadBalancer   selector      Loadbalance traffic across Pods matching this label selector     app  myapp     Accept traffic sent to port 80       ports         name  http       port  80      Service Port       targetPort  80   Container Port         Create LoadBalancer Service for Pod      t   Create Service kubectl apply  f 02 pod LoadBalancer service yml    List Service kubectl get svc    Access Application http    Load Balancer Service IP  curl http    Load Balancer Service IP          Step 04  Clean Up Kubernetes Pod and Service    t   Change Directory cd kube manifests    Delete Pod kubectl delete  f 01 pod definition yml    Delete Service kubectl delete  f  02 pod LoadBalancer service yml          API Object References    Kubernetes API Spec  https   kubernetes io docs reference generated kubernetes api v1 24      Pod Spec  https   kubernetes io docs reference generated kubernetes api v1 24  pod v1 core     Service Spec  https   kubernetes io docs reference generated kubernetes api v1 24  service v1 core     Kubernetes API Reference  https   kubernetes io docs reference kubernetes api


GCP and reproducibility and many others What s important you should expect that other engineers would be working on your code and occasionally changing it Your code should be readable and maintainable are covered by unit tests This helps you to find errors earlier before submitting a training job or even worse deploying a model to production to find out serving is broken because of a typo One way to achieve this is splitting it into smaller pieces separate functions and classes that After you ve built a successful prototype of a machine learning model there s still plenty of Get to know recommended for testing Tensorflow code It s also typically a good idea to care about plenty of things such as operationalization of your model monitoring CI CD reliability things to do To some extent your journey as an ML engineer only begins here You d need to take

Answer:
After you’ve built a successful prototype of a machine learning model, there’s still plenty of
things to do. To some extent, your journey as an ML engineer only begins here. You’d need to take
care about plenty of things such as operationalization of your model: monitoring, CI/CD, reliability
and reproducibility, and many others. What’s important, you should expect that other engineers would
be working on your code and occasionally changing it. Your code should be readable and maintainable.
One way to achieve this  is splitting it into smaller pieces (separate functions and classes) that
are covered by unit tests. This helps you to find errors earlier (before submitting a training job
or - even worse - deploying a model to production to find out serving is broken because of a typo)

Get to know recommended [practices](https://www.tensorflow.org/community/contribute/tests) for testing Tensorflow code. It’s also typically a good idea to
look at test cases examples in TensorFlow source code. Use [tf.test.TestCase](https://www.tensorflow.org/api_docs/python/tf/test/TestCase) class to implement your
unit tests. Get to know this class and how it makes your life easier - e.g., it takes care to use
the same random seed (to make your tests more stable), has a lot of useful assertions as well as
takes care about creating temp dir or managing TensorFlow sessions.

Start simple and continue extending your test coverage as long as your model gets more complex and
needs more debugging. It’s typically a good idea to add a unit test each time you fix any specific
error to make sure this error wouldn’t occur again.

## Testing custom layers
When you implement custom training routines, the recommended
practice is to create
[custom layers](https://www.tensorflow.org/guide/keras/custom_layers_and_models)
by subclassing [tf.keras.layers.Layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer).
This gives you a possibility to isolate (and test) logical pieces of your code, which makes it more
readable and easier to maintain.

Let's play with `ExampleBlock` - a simple custom layer. First, we’d like to test whether the shape of the output tensor is the one we expect.
```python
class LinearBlockTest(tf.test.TestCase):
   def test_shape_default(self):
       x = np.ones((4, 32))
       layer = example.LinearBlock()
       output = layer(x)
       self.assertAllEqual(output.shape, (4, 32))
```

You can find more examples testing output shape by exploring the `LinearBlockTest` class.

The next thing we can also check is the actual output. It’s not always needed but it might be a good
idea when you have a layer with a custom logic that needs to be double-checked.
Please note that despite using initializers for weights, our test is not flaky (tf.test.TestCase
takes care about it).
We can also patch various pieces of the layer we’re concerned about (other layers used, loss
functions, stdout, etc.) to check the desired output. Let’s have a look at the example where we’ve
patched initializer:
```python
@patch.object(initializers, 'get', lambda _: tf.compat.v1.keras.initializers.Ones)
def test_output_ones(self):
   dim = 4
   batch_size = 3
   output_dim = 2
   x = np.ones((batch_size, dim))
   layer = example.LinearBlock(output_dim)
   output = layer(x)
   expected_output = np.ones((batch_size, output_dim)) * (dim + 1)
   self.assertAllClose(output, expected_output, atol=1e-4)
```

## Testing custom keras models
The easiest way to test your model is to prepare a small fake dataset and run a few training steps on
this dataset to check whether the model can be successfully trained, and validation and prediction
also works. Please keep in mind, that successfully means all steps can be run without generating
errors, but in the basic case we don’t check whether the training itself makes sense - i.e., whether
a loss decreases to any meaningful value. But more about it later.
Let’s have a look at a simple example of how to test a model from this [tutorial](https://www.tensorflow.org/tutorials/keras/regression).
```python
class ExampleModelTest(tf.test.TestCase):
   def _get_data(self):
       dataset_path = tf.keras.utils.get_file(
           'auto-mpg.data',
           'http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data')
       column_names = ['MPG','Cylinders','Displacement','Horsepower','Weight',
                       'Acceleration', 'Model Year', 'Origin']
       dataset = pd.read_csv(dataset_path, names=column_names, na_values='?', comment='\t',
                             sep=' ', skipinitialspace=True)
       dataset = dataset.dropna()
       dataset['Origin'] = dataset['Origin'].map({1: 'USA', 2: 'Europe', 3: 'Japan'})
       dataset = pd.get_dummies(dataset, prefix='', prefix_sep='')
       dataset = dataset[dataset.columns].astype('float64')
       labels = dataset.pop('MPG')
       return dataset, labels

   def test_basic(self):
       train_dataset, train_labels = self._get_data()
       dim = len(train_dataset.keys())
       example_model = example.get_model(dim)
       test_ind = train_dataset.sample(10).index
       test_dataset, test_labels = train_dataset.iloc[test_ind], train_labels.iloc[test_ind]
       history = example_model.fit(
           train_dataset, train_labels, steps_per_epoch=2, epochs=2,
           batch_size=10,
           validation_split=0.1,
           validation_steps=1)
       self.assertAlmostEqual(
           history.history['mse'][-1],
           history.history['loss'][-1],
           places=2)
       _ = example_model.evaluate(test_dataset, test_labels)
       _ = example_model.predict(test_dataset)
```
You can find additional examples by looking at `ExampleModelTest` class.

## Testing custom estimators
If we’re using tf.estimator API, we can use the same approach. There are a few moments to keep in
mind though. First, if you might want to convert your pandas DataFrame (you’ve read from csv or other
source) into a [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) to make a dataset for testing purposes:
```
faked_dataset = tf.data.Dataset.from_tensor_slices((dict(df.pop(LABEL)), df[LABEL].values))
return faked_dataset.repeat().batch(batch_size)
```

Tensorflow training routine consists of two main pieces - first, the graph is created and compiled,
and then the same computational graph is run over input data step by step. You can test whether the
graph can be compiled in train/eval/... mode with just invoking the `model_fn`:
```python
...
features = make_faked_batch(...)
model_fn(features, {}, tf.estimator.ModeKeys.TRAIN))
```

Or you actually can create an estimator run a few training steps on a faked dataset you’ve prepared:
```python
...
e = tf.estimator.Estimator(model_fn, config=tf.estimator.RunConfig(model_dir=self.get_temp_dir())))
e.train(input_fn=lambda: make_faked_batch(...), steps=3)
```

## Testing model logic
All above guarantees only that our model is formally correct (i.e., tensor input and output shapes
match one another, there’s no typos and other formal errors in the code). Having these unit tests
typically is a huge step forward since it speeds up the model development process. We still might
want to extend the test coverage, but it’s also worth looking into other possibilities, for example:
* use [TensorFlow Data Validation](https://www.tensorflow.org/tfx/tutorials/data_validation/tfdv_basic) to inspect your data for anomalies and skewness
* use [TensorFlow Model Analysis](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic) to check for performance of your trained model
* have a look how [PerfZero](https://github.com/tensorflow/benchmarks/tree/master/perfzero) helps
debug and track TensorFlow models performance with help of [tf.test.Benchmark](https://www.tensorflow.org/api_docs/python/tf/test/Benchmark)
* implement integration tests if needed
* consider adding unit tests for your tfx/kubeflow pipelines
* implement proper monitoring and alerting for your ML models
* check additional materials, e.g. [What is your ML test score](https://research.google/pubs/pub45742/)
and [Testing & Debugging ML systems](https://developers.google.com/machine-learning/testing-debugging) course

After you ve built a successful prototype of a machine learning model  there s still plenty of things to do  To some extent  your journey as an ML engineer only begins here  You d need to take care about plenty of things such as operationalization of your model  monitoring  CI CD  reliability and reproducibility  and many others  What s important  you should expect that other engineers would be working on your code and occasionally changing it  Your code should be readable and maintainable  One way to achieve this  is splitting it into smaller pieces  separate functions and classes  that are covered by unit tests  This helps you to find errors earlier  before submitting a training job or   even worse   deploying a model to production to find out serving is broken because of a typo   Get to know recommended  practices  https   www tensorflow org community contribute tests  for testing Tensorflow code  It s also typically a good idea to look at test cases examples in TensorFlow source code  Use  tf test TestCase  https   www tensorflow org api docs python tf test TestCase  class to implement your unit tests  Get to know this class and how it makes your life easier   e g   it takes care to use the same random seed  to make your tests more stable   has a lot of useful assertions as well as takes care about creating temp dir or managing TensorFlow sessions   Start simple and continue extending your test coverage as long as your model gets more complex and needs more debugging  It s typically a good idea to add a unit test each time you fix any specific error to make sure this error wouldn t occur again      Testing custom layers When you implement custom training routines  the recommended practice is to create  custom layers  https   www tensorflow org guide keras custom layers and models  by subclassing  tf keras layers Layer  https   www tensorflow org api docs python tf keras layers Layer   This gives you a possibility to isolate  and test  logical pieces of your code  which makes it more readable and easier to maintain   Let s play with  ExampleBlock    a simple custom layer  First  we d like to test whether the shape of the output tensor is the one we expect     python class LinearBlockTest tf test TestCase      def test shape default self          x   np ones  4  32          layer   example LinearBlock          output   layer x         self assertAllEqual output shape   4  32        You can find more examples testing output shape by exploring the  LinearBlockTest  class   The next thing we can also check is the actual output  It s not always needed but it might be a good idea when you have a layer with a custom logic that needs to be double checked  Please note that despite using initializers for weights  our test is not flaky  tf test TestCase takes care about it   We can also patch various pieces of the layer we re concerned about  other layers used  loss functions  stdout  etc   to check the desired output  Let s have a look at the example where we ve patched initializer     python  patch object initializers   get   lambda    tf compat v1 keras initializers Ones  def test output ones self      dim   4    batch size   3    output dim   2    x   np ones  batch size  dim      layer   example LinearBlock output dim     output   layer x     expected output   np ones  batch size  output dim      dim   1     self assertAllClose output  expected output  atol 1e 4          Testing custom keras models The easiest way to test your model is to prepare a small fake dataset and run a few training steps on this dataset to check whether the model can be successfully trained  and validation and prediction also works  Please keep in mind  that successfully means all steps can be run without generating errors  but in the basic case we don t check whether the training itself makes sense   i e   whether a loss decreases to any meaningful value  But more about it later  Let s have a look at a simple example of how to test a model from this  tutorial  https   www tensorflow org tutorials keras regression      python class ExampleModelTest tf test TestCase      def  get data self          dataset path   tf keras utils get file              auto mpg data               http   archive ics uci edu ml machine learning databases auto mpg auto mpg data          column names     MPG   Cylinders   Displacement   Horsepower   Weight                           Acceleration    Model Year    Origin          dataset   pd read csv dataset path  names column names  na values      comment   t                                sep      skipinitialspace True         dataset   dataset dropna          dataset  Origin     dataset  Origin   map  1   USA   2   Europe   3   Japan           dataset   pd get dummies dataset  prefix     prefix sep            dataset   dataset dataset columns  astype  float64          labels   dataset pop  MPG          return dataset  labels     def test basic self          train dataset  train labels   self  get data          dim   len train dataset keys           example model   example get model dim         test ind   train dataset sample 10  index        test dataset  test labels   train dataset iloc test ind   train labels iloc test ind         history   example model fit             train dataset  train labels  steps per epoch 2  epochs 2             batch size 10             validation split 0 1             validation steps 1         self assertAlmostEqual             history history  mse    1              history history  loss    1              places 2             example model evaluate test dataset  test labels             example model predict test dataset      You can find additional examples by looking at  ExampleModelTest  class      Testing custom estimators If we re using tf estimator API  we can use the same approach  There are a few moments to keep in mind though  First  if you might want to convert your pandas DataFrame  you ve read from csv or other source  into a  tf data Dataset  https   www tensorflow org api docs python tf data Dataset  to make a dataset for testing purposes      faked dataset   tf data Dataset from tensor slices  dict df pop LABEL    df LABEL  values   return faked dataset repeat   batch batch size       Tensorflow training routine consists of two main pieces   first  the graph is created and compiled  and then the same computational graph is run over input data step by step  You can test whether the  graph can be compiled in train eval     mode with just invoking the  model fn      python     features   make faked batch      model fn features      tf estimator ModeKeys TRAIN        Or you actually can create an estimator run a few training steps on a faked dataset you ve prepared     python     e   tf estimator Estimator model fn  config tf estimator RunConfig model dir self get temp dir      e train input fn lambda  make faked batch       steps 3          Testing model logic All above guarantees only that our model is formally correct  i e   tensor input and output shapes match one another  there s no typos and other formal errors in the code   Having these unit tests typically is a huge step forward since it speeds up the model development process  We still might want to extend the test coverage  but it s also worth looking into other possibilities  for example    use  TensorFlow Data Validation  https   www tensorflow org tfx tutorials data validation tfdv basic  to inspect your data for anomalies and skewness   use  TensorFlow Model Analysis  https   www tensorflow org tfx tutorials model analysis tfma basic  to check for performance of your trained model   have a look how  PerfZero  https   github com tensorflow benchmarks tree master perfzero  helps debug and track TensorFlow models performance with help of  tf test Benchmark  https   www tensorflow org api docs python tf test Benchmark    implement integration tests if needed   consider adding unit tests for your tfx kubeflow pipelines   implement proper monitoring and alerting for your ML models   check additional materials  e g   What is your ML test score  https   research google pubs pub45742   and  Testing   Debugging ML systems  https   developers google com machine learning testing debugging  course


GCP This example creates a gRCP server that connect to redis to find the name of a gRPC Example user for a given user id Application Project Structure grpcexampleredis

Answer:
# gRPC Example

This example creates a gRCP server that connect to redis to find the name of a
user for a given user id.

## Application Project Structure

 ```
    .
    └── grpc_example_redis
        └── src
            └── main
                ├── java
                        └── com.example.grpc
                            ├── client
                                └── ConnectClient # Example Client
                            ├── server
                                └── ConnectServer # Initializes gRPC Server
                            └── service
                                ├── ConnectServiceImpl # Implementation of rpc services in the proto
                                └── RedisUtil # Tool to initialize redis
                └── proto
                    └── connect_service.proto # Proto definition of the server
       ├── pom.xml
       └── README.md
```

## Technology Stack

1. gRPC
2. Redis

## Setup Instructions

### Redis Emulator Setup

#### Quick Start

Reference guide can be found [here](https://redis.io/topics/quickstart)

#### Installation

```
$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make
$ make test
$ make install
```

#### Start the Server

```
$ redis-server
[28550] 01 Aug 19:29:28 # Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'
[28550] 01 Aug 19:29:28 * Server started, Redis version 2.2.12
[28550] 01 Aug 19:29:28 * The server is now ready to accept connections on port 6379
... more logs ...

```

#### Test the Server

```
$ redis-cli
redis 127.0.0.1:6379> ping
PONG
```

#### Set Data

```
redis 127.0.0.1:6379> set 1234 MyName
OK
redis 127.0.0.1:6379> get mykey
"MyName"
```

#### Set environmental variables

```
$ export REDIS_HOST=127.0.0.1
$ export REDIS_PORT=6379
$ export REDIS_MAX_TOTAL_CONNECTIONS=128
```

## Usage

### Initialize the server

```
$ mvn -DskipTests package exec:java
    -Dexec.mainClass=com.example.grpc.server.ConnectServer

```

### Run the Client

```
$ mvn -DskipTests package exec:java
      -Dexec.mainClass=com.example.grpc.client.ConnectClient
``

gRPC Example  This example creates a gRCP server that connect to redis to find the name of a user for a given user id      Application Project Structure                     grpc example redis             src                 main                     java                             com example grpc                                 client                                     ConnectClient   Example Client                                 server                                     ConnectServer   Initializes gRPC Server                                 service                                     ConnectServiceImpl   Implementation of rpc services in the proto                                     RedisUtil   Tool to initialize redis                     proto                         connect service proto   Proto definition of the server            pom xml            README md         Technology Stack  1  gRPC 2  Redis     Setup Instructions      Redis Emulator Setup       Quick Start  Reference guide can be found  here  https   redis io topics quickstart        Installation        wget http   download redis io redis stable tar gz   tar xvzf redis stable tar gz   cd redis stable   make   make test   make install           Start the Server        redis server  28550  01 Aug 19 29 28   Warning  no config file specified  using the default config  In order to specify a config file use  redis server  path to redis conf   28550  01 Aug 19 29 28   Server started  Redis version 2 2 12  28550  01 Aug 19 29 28   The server is now ready to accept connections on port 6379     more logs                Test the Server        redis cli redis 127 0 0 1 6379  ping PONG           Set Data      redis 127 0 0 1 6379  set 1234 MyName OK redis 127 0 0 1 6379  get mykey  MyName            Set environmental variables        export REDIS HOST 127 0 0 1   export REDIS PORT 6379   export REDIS MAX TOTAL CONNECTIONS 128         Usage      Initialize the server        mvn  DskipTests package exec java      Dexec mainClass com example grpc server ConnectServer           Run the Client        mvn  DskipTests package exec java        Dexec mainClass com example grpc client ConnectClient


GCP Dataflow PubSub XML to Google cloud storage sample pipeline you may not use this file except in compliance with the License License https www apache org licenses LICENSE 2 0 You may obtain a copy of the License at Licensed under the Apache License Version 2 0 the License Copyright 2023 Google LLC

Answer:
# Dataflow PubSub XML to Google cloud storage sample pipeline

## License
Copyright 2023 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

# 1. Getting started

## Create a new Google Cloud project

**It is recommended to go through this walkthrough using a new temporary Google
Cloud project, unrelated to any of your existing Google Cloud projects.**

See https://cloud.google.com/resource-manager/docs/creating-managing-projects
for more details. For a quick reference, please follow these steps:

1. Open the [Cloud Platform Console][cloud-console].
2. In the drop-down menu at the top, select **Create a project**.
3. Give your project a name = <CHANGE_ME>
4. Save your project's name to an environment variable for ease of use:
```
export PROJECT=<CHANGE_ME>
```

# 2. Configure a local environment

## Setup the test environment
```
python -m venv dataflow_pub_sub_xml_to_gcs
source ./dataflow_pub_sub_xml_to_gcs/bin/activate
pip install -q --upgrade pip setuptools wheel
pip install 'apache-beam[gcp]' # Linux, Mac
\path\to\env\Scripts\activate # Windows
```
If you're running this on an Apple Silicon Mac and face issues when running, please run the following commands to build the _grpcio_ library from source:
```
pip uninstall grpcio
export GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation"
pip install grpcio --no-binary :all:
```

# 3. Configure the cloud environment

## Setting Google Application Default Credentials

Set your [Google Application Default
Credentials][application-default-credentials] by [initializing the Google Cloud
SDK][cloud-sdk-init] with the command:

```
gcloud init
```
Generate a credentials file by running the
[application-default login](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command:

```
gcloud auth application-default login
```
Make sure to enable necessary APIs:
```
gcloud services enable dataflow.googleapis.com  compute.googleapis.com  logging.googleapis.com  storage-component.googleapis.com  storage-api.googleapis.com  pubsub.googleapis.com  cloudresourcemanager.googleapis.com  cloudscheduler.googleapis.com
```

[cloud-sdk-init]: https://cloud.google.com/sdk/docs/initializing
[application-default-credentials]: https://developers.google.com/identity/protocols/application-default-credentials

## Configure a PubSub topic
### Pubsub Setup
The following [doc](https://cloud.google.com/pubsub/docs/quickstart-console) can be used to set up the topic and optional subscription needed to run this example.

#### Topics
To run this example one topic needs to be created:
1. A topic to publish the XML formatted data
```
export TOPIC_ID=<CHANGE_ME>
gcloud pubsub topics create $TOPIC_ID
```

#### Subscription
**Optionally** You can set up a custom subscription. However, this is not mandatory since the Dataflow PubSub source automatically creates one if a topic is provided.

## Create a GCS bucket

The output will write to a GCS bucket:
```
export BUCKET_NAME=<CHANGE_ME>
gsutil mb gs://$BUCKET_NAME
```

# 4. Run the test
## Start sending messages to PubSub
Execute the message sending script as follows:
```
python publish2PubSub.py \
--project_id $PROJECT \
--pub_sub_topic_id $TOPIC_ID \
--xml_string XML_STRING \
--message_send_interval MESSAGE_SEND_INTERVAL
```
For example:
```
python publish2PubSub.py \
--project_id $PROJECT \
--pub_sub_topic_id $TOPIC_ID \
--xml_string "<note><to>PubSub</to><from>Test</from><heading>Test</heading><body>Sample body</body></note>" \
--message_send_interval 1
```

## Start the Pipeline
Open up a new terminal and execute the following command:
```
python beamPubSubXml2Gcs.py \
--project_id $PROJECT \
--input_topic_id $TOPIC_ID \
--runner RUNNER \
--window_size WINDOW_SIZE \
--output_path "gs://$BUCKET_NAME/" \
--num_shards NUM_SHARDS
```
For example:
```
python beamPubSubXml2Gcs.py \
--project_id $PROJECT \
--input_topic_id $TOPIC_ID \
--runner DataflowRunner \
--window_size 1.0 \
--gcs_path gs://$BUCKET_NAME/ \
--num_shards 2
```

## Monitor the Dataflow Job
Navigate to https://console.cloud.google.com/dataflow/jobs to locate the job
you just created.  Clicking on the job will let you navigate to the job
monitoring screen.

## Debug the Pipeline
**Optionally** This sample contains the necessary bindings to debug step by step and/or breakpoint this code in Vs Code. To do so, please install the VsCode Google Cloud [extension](https://cloud.google.com/code/docs/vscode/install)

## View the output in CGS

List the generated files in the GCS bucket and inspect their contents
```
gsutil ls gs://${BUCKET_NAME}/output_location/
gsutil cat gs://${BUCKET_NAME}/output_location/*
```

# 5. Clean up

## Remove cloud resources
1. Delete the PubSub topic
```
gcloud pubsub topics delete $TOPIC_ID
```
2. Delete the GCS files
```
gsutil -m rm -rf "gs://${BUCKET_NAME}/output_location/*"
```
3. Remove the GCS bucket
```
gsutil rb gs://${BUCKET_NAME}
```
4. **Optionally** Revoke the authentication credentials that you created, and delete the local credential file.
```
gcloud auth application-default revoke
```
5. **Optionally** Revoke credentials from the gcloud CLI.
```
gcloud auth revoke
```
## Terminate the PubSub streaming
On the terminal where you ran the _publish2PubSub_ script, press _Ctrl+C_ and _Y_ to confirm

Dataflow PubSub XML to Google cloud storage sample pipeline     License Copyright 2023 Google LLC  Licensed under the Apache License  Version 2 0  the  License    you may not use this file except in compliance with the License  You may obtain a copy of the License at      https   www apache org licenses LICENSE 2 0  Unless required by applicable law or agreed to in writing  software distributed under the License is distributed on an  AS IS  BASIS  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND  either express or implied  See the License for the specific language governing permissions and limitations under the License     1  Getting started     Create a new Google Cloud project    It is recommended to go through this walkthrough using a new temporary Google Cloud project  unrelated to any of your existing Google Cloud projects     See https   cloud google com resource manager docs creating managing projects for more details  For a quick reference  please follow these steps   1  Open the  Cloud Platform Console  cloud console   2  In the drop down menu at the top  select   Create a project    3  Give your project a name    CHANGE ME  4  Save your project s name to an environment variable for ease of use      export PROJECT  CHANGE ME         2  Configure a local environment     Setup the test environment     python  m venv dataflow pub sub xml to gcs source   dataflow pub sub xml to gcs bin activate pip install  q   upgrade pip setuptools wheel pip install  apache beam gcp     Linux  Mac  path to env Scripts activate   Windows     If you re running this on an Apple Silicon Mac and face issues when running  please run the following commands to build the  grpcio  library from source      pip uninstall grpcio export GRPC PYTHON LDFLAGS    framework CoreFoundation  pip install grpcio   no binary  all         3  Configure the cloud environment     Setting Google Application Default Credentials  Set your  Google Application Default Credentials  application default credentials  by  initializing the Google Cloud SDK  cloud sdk init  with the command       gcloud init     Generate a credentials file by running the  application default login  https   cloud google com sdk gcloud reference auth application default login  command       gcloud auth application default login     Make sure to enable necessary APIs      gcloud services enable dataflow googleapis com  compute googleapis com  logging googleapis com  storage component googleapis com  storage api googleapis com  pubsub googleapis com  cloudresourcemanager googleapis com  cloudscheduler googleapis com       cloud sdk init   https   cloud google com sdk docs initializing  application default credentials   https   developers google com identity protocols application default credentials     Configure a PubSub topic     Pubsub Setup The following  doc  https   cloud google com pubsub docs quickstart console  can be used to set up the topic and optional subscription needed to run this example        Topics To run this example one topic needs to be created  1  A topic to publish the XML formatted data     export TOPIC ID  CHANGE ME  gcloud pubsub topics create  TOPIC ID           Subscription   Optionally   You can set up a custom subscription  However  this is not mandatory since the Dataflow PubSub source automatically creates one if a topic is provided      Create a GCS bucket  The output will write to a GCS bucket      export BUCKET NAME  CHANGE ME  gsutil mb gs    BUCKET NAME        4  Run the test    Start sending messages to PubSub Execute the message sending script as follows      python publish2PubSub py     project id  PROJECT     pub sub topic id  TOPIC ID     xml string XML STRING     message send interval MESSAGE SEND INTERVAL     For example      python publish2PubSub py     project id  PROJECT     pub sub topic id  TOPIC ID     xml string   note  to PubSub  to  from Test  from  heading Test  heading  body Sample body  body   note       message send interval 1         Start the Pipeline Open up a new terminal and execute the following command      python beamPubSubXml2Gcs py     project id  PROJECT     input topic id  TOPIC ID     runner RUNNER     window size WINDOW SIZE     output path  gs    BUCKET NAME       num shards NUM SHARDS     For example      python beamPubSubXml2Gcs py     project id  PROJECT     input topic id  TOPIC ID     runner DataflowRunner     window size 1 0     gcs path gs    BUCKET NAME      num shards 2         Monitor the Dataflow Job Navigate to https   console cloud google com dataflow jobs to locate the job you just created   Clicking on the job will let you navigate to the job monitoring screen      Debug the Pipeline   Optionally   This sample contains the necessary bindings to debug step by step and or breakpoint this code in Vs Code  To do so  please install the VsCode Google Cloud  extension  https   cloud google com code docs vscode install      View the output in CGS  List the generated files in the GCS bucket and inspect their contents     gsutil ls gs     BUCKET NAME  output location  gsutil cat gs     BUCKET NAME  output location          5  Clean up     Remove cloud resources 1  Delete the PubSub topic     gcloud pubsub topics delete  TOPIC ID     2  Delete the GCS files     gsutil  m rm  rf  gs     BUCKET NAME  output location        3  Remove the GCS bucket     gsutil rb gs     BUCKET NAME      4    Optionally   Revoke the authentication credentials that you created  and delete the local credential file      gcloud auth application default revoke     5    Optionally   Revoke credentials from the gcloud CLI      gcloud auth revoke        Terminate the PubSub streaming On the terminal where you ran the  publish2PubSub  script  press  Ctrl C  and  Y  to confirm