eks In tree Out of tree exclude true search

Answer:
---
search:
  exclude: true
---

# 영구 스토리지 옵션

## In-tree와 Out-of-tree 볼륨 플러그인

컨테이너 스토리지 인터페이스(CSI)가 도입되기 전에는 모든 볼륨 플러그인이 in-tree 였습니다. 즉, 코어 쿠버네티스 바이너리와 함께 빌드, 연결, 컴파일 및 제공되고 핵심 쿠버네티스 API를 확장했습니다. 이는 쿠버네티스에 새로운 스토리지 시스템(볼륨 플러그인)을 추가하려면 핵심 코어 쿠버네티스 코드 저장소에 대한 코드를 확인해야 하는 것을 의미했습니다.

Out-of-tree 볼륨 플러그인은 쿠버네티스 코드 베이스와 독립적으로 개발되며 쿠버네티스 클러스터에 확장으로 배포(및 설치) 됩니다. 이를 통해 벤더는 쿠버네티스 릴리스 주기와 별도로 드라이버를 업데이트 할 수 있습니다. 이는 쿠버네티스가 벤더에 k8s와 인터페이스하는 표준 방법을 제공하는 스토리지 인터페이스 혹은 CSI를 만들었기 때문에 가능합니다.

Amazon Elastic Kubernetes Services (EKS) 스토리지 클래스 및 CSI 드라이버에 대한 자세한 내용을 [AWS 문서](https://docs.aws.amazon.com/eks/latest/userguide/storage.html)에서 확인할 수 있습니다.

## 윈도우용 In-tree 볼륨 플러그인
쿠버네티스 볼륨을 사용하면 데이터 지속성 요구 사항이 있는 애플리케이션을 쿠버네티스에 배포할 수 있습니다. 퍼시스턴트 볼륨 관리는 볼륨 프로비저닝/프로비저닝 해제/크기 조정, 쿠버네티스 노드에 볼륨 연결/분리, 파드의 개별 컨테이너에 볼륨 마운트/마운트 해제로 구성됩니다. 특정 스토리지 백엔드 또는 프로토콜에 대해 이런 볼륨 관리 작업을 구현하기 위한 코드는 쿠버네티스 볼륨 플러그인 **(In-tree 볼륨 플러그인)** 형식으로 제공됩니다. Amazon EKS에서는 윈도우에서 다음과 같은 클래스의 쿠버네티스 볼륨 플러그인이 지원됩니다:

*In-tree 볼륨 플러그인:* [awsElasticBlockStore](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore)

윈도우 노드에서 In-tree 볼륨 플러그인을 사용하기 위해서는 NTFS를 fsType 으로 사용하기 위한 추가 StorageClass를 생성해야 합니다. EKS에서 기본 StorageClass는 ext4를 기본 fsType으로 사용합니다.

StorageClass는 관리자가 제공하는 스토리지의 "클래스"를 설명하는 방법을 제공합니다. 다양한 클래스는 QoS 수준, 백업 정책 또는 클러스터 관리자가 결정한 임의 정책에 매핑될 수 있습니다. 쿠버네티스는 클래스가 무엇을 나타내는지에 대해 의견이 없습니다. 이 개념은 다른 스토리지 시스템에서는 "프로파일" 이라고 부르기도 합니다.

다음 명령을 실행하여 확인할 수 있습니다:

```bash
kubectl describe storageclass gp2
```

출력:

```bash
Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClas
","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType"
"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>
```

**NTFS**를 지원하는 새 StorageClass를 생성하려면 다음 매니페스트를 사용하십시오:

```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2-windows
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ntfs
volumeBindingMode: WaitForFirstConsumer
```

다음 명령을 실행하여 StorageClass를 생성합니다:

```bash
kubectl apply -f NTFSStorageClass.yaml
```

다음 단계는 퍼시스턴트 볼륨 클레임(PVC)을 생성하는 것입니다.

퍼시스턴트 볼륨(PV)은 관리자가 프로비저닝했거나 PVC를 사용하여 동적으로 프로비저닝된 클러스터의 스토리지입니다. 노드가 클러스터 리소스인 것처럼 클러스터의 리소스입니다. 이 API 객체는 NFS, iSCSI 또는 클라우드 공급자별 스토리지 시스템 등 스토리지 구현의 세부 정보를 캡처합니다.

PVC는 사용자의 스토리지 요청입니다. 클레임은 특정 크기 및 액세스 모드를 요청할 수 있습니다(예: ReadWriteOnce, ReadOnlyMany 또는 ReadWriteMan로 마운트될 수 있음).

사용자에게는 다양한 유즈케이스를 위해 성능과 같은 다양한 속성을 가진 PV가 필요합니다. 클러스터 관리자는 사용자에게 해당 볼륨이 구현되는 방법에 대한 세부 정보를 노출시키지 않고도 크기 및 액세스 모드보다 더 다양한 방식으로 PV를 제공할 수 있어야 합니다. 이런 요구 사항을 충족하기 위해 StorageClass 리소스가 있습니다.

아래 예제에서 윈도우 네임스페이스 내에 PVC가 생성되었습니다.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-windows-pv-claim
  namespace: windows
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp2-windows
  resources:
    requests:
      storage: 1Gi
```

다음 명령을 실행하여 PVC를 만듭니다:

```bash
kubectl apply -f persistent-volume-claim.yaml
```

다음 매니페스트에서는 윈도우 파드를 생성하고 VolumeMount를 `C:\Data`로 설정하고 PVC를 `C:\Data`에 연결된 스토리지로 사용합니다.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-server-ltsc2019
  namespace: windows
spec:
  selector:
    matchLabels:
      app: windows-server-ltsc2019
      tier: backend
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-server-ltsc2019
        tier: backend
        track: stable
    spec:
      containers:
      - name: windows-server-ltsc2019
        image: mcr.microsoft.com/windows/servercore:ltsc2019
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: "C:\\data"
          name: test-volume
      volumes:
        - name: test-volume
          persistentVolumeClaim:
            claimName: ebs-windows-pv-claim
      nodeSelector:
        kubernetes.io/os: windows
        node.kubernetes.io/windows-build: '10.0.17763'
```

PowerShell을 통해 윈도우 파드에 액세스하여 결과를 테스트합니다:

```bash
kubectl exec -it podname powershell -n windows
```

윈도우 파드 내에서 `ls`를 수행합니다:

출력:

```bash
PS C:\> ls

    Directory: C:\

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          3/8/2021   1:54 PM                data
d-----          3/8/2021   3:37 PM                inetpub
d-r---          1/9/2021   7:26 AM                Program Files
d-----          1/9/2021   7:18 AM                Program Files (x86)
d-r---          1/9/2021   7:28 AM                Users
d-----          3/8/2021   3:36 PM                var
d-----          3/8/2021   3:36 PM                Windows
-a----         12/7/2019   4:20 AM           5510 License.txt
```

**data directory** 는 EBS 볼륨에 의해 제공됩니다.

## 윈도우 Out-of-tree
CSI 플러그인과 연결된 코드는 일반적으로 컨테이너 이미지로 배포되고 데몬셋(DaemonSet) 및 스테이트풀셋(StatefulSet)과 같은 표준 쿠버네티스 구성을 사용하여 배포되는 out-of-tree 스크립트 및 바이너리로 제공됩니다. CSI 플러그인은 쿠버네티스에서 광범위한 볼륨 관리 작업을 처리합니다. CSI 플러그인은 일반적으로 노드 플러그인(각 노드에서 데몬셋으로 실행됨)과 컨트롤러 플러그인으로 구성됩니다.

CSI 노드 플러그인 (특히 블록 장치 또는 공유 파일 시스템을 통해 노출되는 퍼시스턴트 볼륨과 관련된 플러그인)은 디스크 장치 스캔, 파일 시스템 탑재 등과 같은 다양한 권한 작업을 수행해야 합니다. 이런 작업은 호스트 운영 체제마다 다릅니다. 리눅스 워커 노드의 경우 컨테이너화된 CSI 노드 플러그인은 일반적으로 권한 있는 컨테이너로 배포됩니다. 윈도우 워커 노드의 경우 각 윈도우 노드에 사전 설치해야 하는 커뮤니티 관리 독립 실행형 바이너리인 [csi-proxy](https://github.com/kubernetes-csi/csi-proxy) 를 사용하여 컨테이너화된 CSI 노드 플러그인에 대한 권한 있는 작업이 지원됩니다.

[Amazon EKS 최적화 윈도우 AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html)에서는 2022년 4월부터 CSI-proxy가 포함됩니다. 고객은 윈도우 노드의 [SMB CSI 드라이버](https://github.com/kubernetes-csi/csi-driver-smb)를 사용하여 [Amazon FSx for Windows File Server](https://aws.amazon.com/fsx/windows/), [Amazon FSx for NetApp ONTAP SMB Shares](https://aws.amazon.com/fsx/netapp-ontap/), 및/또는 [AWS Storage Gateway – File Gateway](https://aws.amazon.com/storagegateway/file/)에 액세스 할 수 있습니다.

다음 [블로그](https://aws.amazon.com/blogs/modernizing-with-aws/using-smb-csi-driver-on-amazon-eks-windows-nodes/)에서는 Amazon FSx for Windows File Server를 위도우 파드용 영구 스토리지로 사용하도록 SMB CSI 드라이버를 설정하는 방법에 대한 세부 정보가 나와 있습니다.

## Amazon FSx for Windows File Server
한 가지 옵션은 [SMB Global Mapping](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/persistent-storage)이라는 SMB 기능을 통해 Windows 파일 서버용 Amazon FSx를 사용하는 것입니다. 이 기능을 사용하면 호스트에 SMB 공유를 마운트한 다음 해당 공유의 디렉터리를 컨테이너로 전달할 수 있습니다. 컨테이너를 특정 서버, 공유, 사용자 이름 또는 암호로 구성할 필요가 없습니다. 대신 호스트에서 모두 처리됩니다. 컨테이너는 로컬 스토리지가 있는 것처럼 작동합니다.

> SMB Global Mapping은 오케스트레이터에게 투명하게 전달되며 HostPath를 통해 마운트되므로 **보안 문제가 발생할 수 있습니다**.

아래 예제에서, `G:\Directory\app-state` 경로는 윈도우 노드의 SMB 공유입니다.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-fsx
spec:
  containers:
  - name: test-fsx
    image: mcr.microsoft.com/windows/servercore:ltsc2019
    command:
      - powershell.exe
      - -command
      - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
    volumeMounts:
      - mountPath: C:\dotnetapp\app-state
        name: test-mount
  volumes:
    - name: test-mount
      hostPath:
        path: G:\Directory\app-state
        type: Directory
  nodeSelector:
      beta.kubernetes.io/os: windows
      beta.kubernetes.io/arch: amd64
```

다음 [블로그](https://aws.amazon.com/blogs/containers/using-amazon-fsx-for-windows-file-server-on-eks-windows-containers/) 에서는 Amazon FSx for Windows File Server를 윈도우 파드용 영구 스토리지로 설정하는 방법에 대한 세부 정보가 나와 있습니다.

search    exclude  true                        In tree  Out of tree                          CSI                        in tree                                                        API                                                                                Out of tree                                                                                k8s                                    CSI                   Amazon Elastic Kubernetes Services  EKS             CSI                   AWS     https   docs aws amazon com eks latest userguide storage html                         In tree                                                                                In tree                        Amazon EKS                                              In tree            awsElasticBlockStore  https   kubernetes io docs concepts storage volumes  awselasticblockstore            In tree                    NTFS  fsType               StorageClass            EKS      StorageClass  ext4     fsType           StorageClass                                                  QoS                                                                                bash kubectl describe storageclass gp2              bash Name             gp2 IsDefaultClass   Yes Annotations      kubectl kubernetes io last applied configuration   apiVersion   storage k8s io v1   kind   StorageClas    metadata    annotations    storageclass kubernetes io is default class   true    name   gp2    parameters    fsType   ext4   type   gp2    provisioner   kubernetes io aws ebs   volumeBindingMode   WaitForFirstConsumer    storageclass kubernetes io is default class true Provisioner            kubernetes io aws ebs Parameters             fsType ext4 type gp2 AllowVolumeExpansion    unset  MountOptions            none  ReclaimPolicy          Delete VolumeBindingMode      WaitForFirstConsumer Events                  none         NTFS           StorageClass                              yaml kind  StorageClass apiVersion  storage k8s io v1 metadata    name  gp2 windows provisioner  kubernetes io aws ebs parameters    type  gp2   fsType  ntfs volumeBindingMode  WaitForFirstConsumer                  StorageClass             bash  kubectl apply  f NTFSStorageClass yaml                          PVC                        PV                 PVC                                                                    API     NFS  iSCSI                                                 PVC                                                      ReadWriteOnce  ReadOnlyMany    ReadWriteMan                                                      PV                                                                                PV                                   StorageClass                                    PVC               yaml  apiVersion  v1 kind  PersistentVolumeClaim metadata    name  ebs windows pv claim   namespace  windows spec     accessModes        ReadWriteOnce   storageClassName  gp2 windows   resources       requests        storage  1Gi                  PVC            bash  kubectl apply  f persistent volume claim yaml                               VolumeMount   C  Data        PVC   C  Data                        yaml  apiVersion  apps v1 kind  Deployment metadata    name  windows server ltsc2019   namespace  windows spec    selector      matchLabels        app  windows server ltsc2019       tier  backend       track  stable   replicas  1   template      metadata        labels          app  windows server ltsc2019         tier  backend         track  stable     spec        containers          name  windows server ltsc2019         image  mcr microsoft com windows servercore ltsc2019         ports            name  http           containerPort  80         imagePullPolicy  IfNotPresent         volumeMounts            mountPath   C   data            name  test volume       volumes            name  test volume           persistentVolumeClaim              claimName  ebs windows pv claim       nodeSelector          kubernetes io os  windows         node kubernetes io windows build   10 0 17763       PowerShell                                   bash  kubectl exec  it podname powershell  n windows                  ls                   bash  PS C    ls       Directory  C     Mode                 LastWriteTime         Length Name                                                        d               3 8 2021   1 54 PM                data d               3 8 2021   3 37 PM                inetpub d r             1 9 2021   7 26 AM                Program Files d               1 9 2021   7 18 AM                Program Files  x86  d r             1 9 2021   7 28 AM                Users d               3 8 2021   3 36 PM                var d               3 8 2021   3 36 PM                Windows  a             12 7 2019   4 20 AM           5510 License txt        data directory     EBS                       Out of tree CSI                                        DaemonSet           StatefulSet                             out of tree                     CSI                                     CSI                                                            CSI                                                                                CSI                                                                                csi proxy  https   github com kubernetes csi csi proxy                CSI                                 Amazon EKS         AMI  https   docs aws amazon com eks latest userguide eks optimized windows ami html     2022  4    CSI proxy                      SMB CSI       https   github com kubernetes csi csi driver smb         Amazon FSx for Windows File Server  https   aws amazon com fsx windows     Amazon FSx for NetApp ONTAP SMB Shares  https   aws amazon com fsx netapp ontap          AWS Storage Gateway   File Gateway  https   aws amazon com storagegateway file                            https   aws amazon com blogs modernizing with aws using smb csi driver on amazon eks windows nodes      Amazon FSx for Windows File Server                         SMB CSI                                       Amazon FSx for Windows File Server           SMB Global Mapping  https   docs microsoft com en us virtualization windowscontainers manage containers persistent storage     SMB        Windows        Amazon FSx                             SMB                                                                                SMB Global Mapping                      HostPath                                              G  Directory app state              SMB            yaml apiVersion  v1 kind  Pod metadata    name  test fsx spec    containers      name  test fsx     image  mcr microsoft com windows servercore ltsc2019     command          powershell exe          command          Add WindowsFeature Web Server  Invoke WebRequest  UseBasicParsing  Uri  https   dotnetbinaries blob core windows net servicemonitor 2 0 1 6 ServiceMonitor exe   OutFile  C   ServiceMonitor exe   echo   html  body  br   br   marquee  H1 Hello EKS    H1  marquee   body  html     C   inetpub  wwwroot  default html  C   ServiceMonitor exe  w3svc         volumeMounts          mountPath  C  dotnetapp app state         name  test mount   volumes        name  test mount       hostPath           path  G  Directory app state         type  Directory   nodeSelector        beta kubernetes io os  windows       beta kubernetes io arch  amd64               https   aws amazon com blogs containers using amazon fsx for windows file server on eks windows containers       Amazon FSx for Windows File Server


Answer:
---
search:
  exclude: true
---

# 이기종 워크로드 실행¶

쿠버네티스는 동일한 클러스터에 리눅스 및 윈도우 노드를 혼합하여 사용할 수 있는 이기종(Heterogeneous) 클러스터를 지원합니다. 해당 클러스터 내에는 리눅스에서 실행되는 파드와 윈도우에서 실행되는 파드를 혼합하여 사용할 수 있습니다. 동일한 클러스터에서 여러 버전의 윈도우를 실행할 수도 있습니다. 하지만 이 결정을 내릴 때 고려해야 할 몇 가지 요소(아래 설명 참조)가 있습니다.

# 노드 내 파드 할당 모범 사례

리눅스 및 윈도우 워크로드를 각각의 OS별 노드에 유지하려면 노드 셀렉터(NodeSelector)와 테인트(Taint)/톨러레이션(Toleration)을 조합하여 사용해야 합니다. 이기종 환경에서 워크로드를 스케줄링하는 주된 목적은 기존 리눅스 워크로드와의 호환성이 깨지지 않도록 하는 것입니다.

## 특정 OS 워크로드가 적절한 컨테이너 노드에서 실행 보장

사용자는 노드셀렉터를 사용하여 윈도우 컨테이너를 적절한 호스트에서 스케줄링 할 수 있습니다. 현재 모든 쿠버네티스 노드에는 다음과 같은 default labels 이 있습니다:

    kubernetes.io/os = [windows|linux]
    kubernetes.io/arch = [amd64|arm64|...]

파드 스펙에 ``"kubernetes.io/os": windows`` 와 같은 노드셀렉터가 포함되지 않는 경우, 파드는 윈도우 또는 리눅스 어느 호스트에서나 스케줄링 될 수 있습니다. 윈도우 컨테이너는 윈도우에서만 실행할 수 있고 리눅스 컨테이너는 리눅스에서만 실행할 수 있기 때문에 문제가 될 수 있습니다.

엔터프라이즈 환경에서는 리눅스 컨테이너에 대한 기존 배포가 많을 뿐만 아니라 헬름 차트와 같은 기성 구성 에코시스템(off-the-shelf configurations)을 갖는 것이 일반적입니다. 이런 상황에서는 디플로이먼트의 노드셀렉터를 변경하는 것을 주저할 수 있습니다. **대안은 테인트를 사용하는 것**입니다.

예를 들어: `--register-with-taints='os=windows:NoSchedule'`

EKS를 사용하는 경우, eksctl은 clusterConfig를 통해 테인트를 적용하는 방법을 제공합니다:

```yaml
NodeGroups:
  - name: windows-ng
    amiFamily: WindowsServer2022FullContainer
    ...
    labels:
      nodeclass: windows2022
    taints:
      os: "windows:NoSchedule"
```

모든 윈도우 노드에 테인트를 추가하는 경우, 스케줄러는 테인트를 허용하지 않는 한 해당 노드에서 파드를 스케줄링하지 않습니다. 다음은 파드 매니페스트의 예시입니다:

```yaml
nodeSelector:
    kubernetes.io/os: windows
tolerations:
    - key: "os"
      operator: "Equal"
      value: "windows"
      effect: "NoSchedule"
```

## 동일한 클러스터에서 여러 윈도우 빌드 처리

각 파드에서 사용하는 윈도우 컨테이너 베이스 이미지는 노드와 동일한 커널 빌드 버전과 일치해야 합니다. 동일한 클러스터에서 여러 윈도우 Server 빌드를 사용하려면 추가 노드 레이블인 노드셀렉터를 설정하거나 **windows-build** 레이블을 활용해야 합니다.

쿠버네티스 1.17 버전에서는 **node.kubernetes.io/windows-build** 라는 새로운 레이블을 자동으로 추가하여 동일한 클러스터에서 여러 윈도우 빌드의 관리를 단순화 합니다. 이전 버전을 실행 중인 경우 이 레이블을 윈도우 노드에 수동으로 추가하는 것이 좋습니다.

이 레이블에는 호환성을 위해 일치해야 하는 윈도우 메이저, 마이너, 그리고 빌드 번호가 반영되어 있습니다. 다음은 현재 각 윈도우 서버 버전에 사용되는 값입니다.

중요한 점은 윈도우 서버가 장기 서비스 채널(LTSC)를 기본 릴리스 채널로 이동하고 있다는 것입니다. 윈도우 서버 반기 채널(SAC)은 2022년 8월 9일에 사용 중지되었습니다. 윈도우 서버의 향후 SAC 릴리스는 없습니다.

| Product Name | Build Number(s) |
| -------- | -------- |
| Server full 2022 LTSC    | 10.0.20348    |
| Server core 2019 LTSC    | 10.0.17763    |

다음 명령을 통해 OS 빌드 버전을 확인할 수 있습니다:

```bash
kubectl get nodes -o wide
```

KERNEL-VERSION 출력은 윈도우 OS 빌드 버전을 나타냅니다.

```bash
NAME                          STATUS   ROLES    AGE   VERSION                INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-10-10-2-235.ec2.internal   Ready    <none>   23m   v1.24.7-eks-fb459a0    10.10.2.235   3.236.30.157    Windows Server 2022 Datacenter   10.0.20348.1607                 containerd://1.6.6
ip-10-10-31-27.ec2.internal   Ready    <none>   23m   v1.24.7-eks-fb459a0    10.10.31.27   44.204.218.24   Windows Server 2019 Datacenter   10.0.17763.4131                 containerd://1.6.6
ip-10-10-7-54.ec2.internal    Ready    <none>   31m   v1.24.11-eks-a59e1f0   10.10.7.54    3.227.8.172     Amazon Linux 2                   5.10.173-154.642.amzn2.x86_64   containerd://1.6.19
```

아래 예제에서는 다양한 윈도우 노드 그룹 OS 버전을 실행할 때 올바른 윈도우 빌드 버전을 일치시키기 위해 추가 노드셀렉터를 파드 스펙에 적용합니다.

```yaml
nodeSelector:
    kubernetes.io/os: windows
    node.kubernetes.io/windows-build: '10.0.20348'
tolerations:
    - key: "os"
    operator: "Equal"
    value: "windows"
    effect: "NoSchedule"
```

## RuntimeClass를 사용하여 파드 매니페스트의 노드셀렉터와 톨러레이션 단순화

RuntimeClass를 사용하여 테인트와 톨러레이션을 사용하는 프로세스를 간소화할 수 있습니다. 이런 테인트와 톨러레이션을 캡슐화하는 RuntimeClass 오브젝트를 만들어 이 작업을 수행할 수 있습니다.

다음 매니페스트를 통해 RuntimeClass를 생성합니다:

```yaml
apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: windows-2022
handler: 'docker'
scheduling:
  nodeSelector:
    kubernetes.io/os: 'windows'
    kubernetes.io/arch: 'amd64'
    node.kubernetes.io/windows-build: '10.0.20348'
  tolerations:
  - effect: NoSchedule
    key: os
    operator: Equal
    value: "windows"
```

RuntimeClass가 생성되면, 파드 매니페스트의 스펙을 통해 사용합니다.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iis-2022
  labels:
    app: iis-2022
spec:
  replicas: 1
  template:
    metadata:
      name: iis-2022
      labels:
        app: iis-2022
    spec:
      runtimeClassName: windows-2022
      containers:
      - name: iis
```

## 관리형 노드 그룹 지원
고객이 윈도우 애플리케이션을 보다 간소화된 방식으로 실행할 수 있도록 AWS에서는 2022년 12월 15일에 [윈도우 컨테이너에 대한 EKS 관리형 노드 그룹 (MNG) 지원](https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-eks-automated-provisioning-lifecycle-management-windows-containers/)을 시작 했습니다. [윈도우 관리형 노드 그룹](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)은 [리눅스 관리형 노드 그룹](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)과 동일한 워크플로우와 도구를 사용하여 활성화됩니다. 윈도우 서버 2019 및 2022 패밀리의 Full, Core AMI(Amazon Machine Image)가 지원 됩니다.

관리형 노드 그룹(MNG)에 지원되는 AMI 패밀리는 다음과 같습니다:

| AMI Family |
| ---------   |
| WINDOWS_CORE_2019_x86_64    |
| WINDOWS_FULL_2019_x86_64    |
| WINDOWS_CORE_2022_x86_64    |
| WINDOWS_FULL_2022_x86_64    |

## 추가 문서

AWS 공식 문서:
https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html

파드 네트워킹(CNI)의 작동 방식을 더 잘 이해하려면 다음 링크를 확인하십시오: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html

EKS 기반 윈도우용 관리형 노드 그룹 배포에 관한 AWS 블로그:
https://aws.amazon.com/blogs/containers/deploying-amazon-eks-windows-managed-node-groups

search    exclude  true                                                                        Heterogeneous                                                                                OS                   NodeSelector       Taint        Toleration                                                                                OS                                                                                default labels              kubernetes io os    windows linux      kubernetes io arch    amd64 arm64                 kubernetes io os   windows                                                                                off the shelf configurations                                                                                register with taints  os windows NoSchedule    EKS           eksctl  clusterConfig                              yaml NodeGroups      name  windows ng     amiFamily  WindowsServer2022FullContainer             labels        nodeclass  windows2022     taints        os   windows NoSchedule                                                                                yaml nodeSelector      kubernetes io os  windows tolerations        key   os        operator   Equal        value   windows        effect   NoSchedule                                                                                Server                                     windows build                         1 17         node kubernetes io windows build                                                                                LTSC                                          SAC   2022  8  9                          SAC                Product Name   Build Number s                              Server full 2022 LTSC      10 0 20348        Server core 2019 LTSC      10 0 17763                 OS                        bash     kubectl get nodes  o wide      KERNEL VERSION         OS                   bash  NAME                          STATUS   ROLES    AGE   VERSION                INTERNAL IP   EXTERNAL IP     OS IMAGE                         KERNEL VERSION                  CONTAINER RUNTIME ip 10 10 2 235 ec2 internal   Ready     none    23m   v1 24 7 eks fb459a0    10 10 2 235   3 236 30 157    Windows Server 2022 Datacenter   10 0 20348 1607                 containerd   1 6 6 ip 10 10 31 27 ec2 internal   Ready     none    23m   v1 24 7 eks fb459a0    10 10 31 27   44 204 218 24   Windows Server 2019 Datacenter   10 0 17763 4131                 containerd   1 6 6 ip 10 10 7 54 ec2 internal    Ready     none    31m   v1 24 11 eks a59e1f0   10 10 7 54    3 227 8 172     Amazon Linux 2                   5 10 173 154 642 amzn2 x86 64   containerd   1 6 19                             OS                                                               yaml nodeSelector      kubernetes io os  windows     node kubernetes io windows build   10 0 20348  tolerations        key   os      operator   Equal      value   windows      effect   NoSchedule          RuntimeClass                                   RuntimeClass                                                                RuntimeClass                                           RuntimeClass             yaml apiVersion  node k8s io v1beta1 kind  RuntimeClass metadata    name  windows 2022 handler   docker  scheduling    nodeSelector      kubernetes io os   windows      kubernetes io arch   amd64      node kubernetes io windows build   10 0 20348    tolerations      effect  NoSchedule     key  os     operator  Equal     value   windows       RuntimeClass                                    yaml apiVersion  apps v1 kind  Deployment metadata    name  iis 2022   labels      app  iis 2022 spec    replicas  1   template      metadata        name  iis 2022       labels          app  iis 2022     spec        runtimeClassName  windows 2022       containers          name  iis                                                             AWS    2022  12  15                 EKS            MNG      https   aws amazon com about aws whats new 2022 12 amazon eks automated provisioning lifecycle management windows containers                             https   docs aws amazon com eks latest userguide managed node groups html                   https   docs aws amazon com eks latest userguide managed node groups html                                      2019   2022      Full  Core AMI Amazon Machine Image                      MNG        AMI                   AMI Family                      WINDOWS CORE 2019 x86 64         WINDOWS FULL 2019 x86 64         WINDOWS CORE 2022 x86 64         WINDOWS FULL 2022 x86 64                   AWS        https   docs aws amazon com eks latest userguide windows support html          CNI                                   https   docs aws amazon com eks latest userguide pod networking html  EKS                          AWS      https   aws amazon com blogs containers deploying amazon eks windows managed node groups


eks info We ve Moved to the AWS Docs Bookmarks and links will continue to work but we recommend updating them for faster access in the future on the AWS Docs Please visit our new site for the latest version This content has been updated and relocated to improve your experience

Answer:
!!! info "We've Moved to the AWS Docs! 🚀"
    This content has been updated and relocated to improve your experience.
    Please visit our new site for the latest version:
    [AWS EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/windows-monitoring.html) on the AWS Docs

    Bookmarks and links will continue to work, but we recommend updating them for faster access in the future.

---

# Monitoring

Prometheus, a [graduated CNCF project](https://www.cncf.io/projects/) is by far the most popular monitoring system with native integration into Kubernetes. Prometheus collects metrics around containers, pods, nodes, and clusters. Additionally, Prometheus leverages AlertsManager which lets you program alerts to warn you if something in your cluster is going wrong. Prometheus stores the metric data as a time series data identified by metric name and key/value pairs. Prometheus includes away to query using a language called PromQL, which is short for Prometheus Query Language.

The high level architecture of Prometheus metrics collection is shown below:

![Prometheus Metrics collection](./images/prom.png)

Prometheus uses a pull mechanism and scrapes metrics from targets using exporters and from the Kubernetes API using the [kube state metrics](https://github.com/kubernetes/kube-state-metrics). This means applications and services must expose a HTTP(S) endpoint containing Prometheus formatted metrics. Prometheus will then, as per its configuration, periodically pull metrics from these HTTP(S) endpoints.

An exporter lets you consume third party metrics as Prometheus formatted metrics. A Prometheus exporter is typically deployed on each node. For a complete list of exporters please refer to the Prometheus [exporters](https://prometheus.io/docs/instrumenting/exporters/). While [node exporter](https://github.com/prometheus/node_exporter) is suited for exporting host hardware and OS metrics for linux nodes, it wont work for Windows nodes.

In a **mixed node EKS cluster with Windows nodes** when you use the stable [Prometheus helm chart](https://github.com/prometheus-community/helm-charts), you will see failed pods on the Windows nodes, as this exporter is not intended for Windows. You will need to treat the Windows worker pool separate and instead install the [Windows exporter](https://github.com/prometheus-community/windows_exporter) on the Windows worker node group.

In order to setup Prometheus monitoring for Windows nodes, you need to download and install the WMI exporter on the Windows server itself and then setup the targets inside the scrape configuration of the Prometheus configuration file.
The [releases page](https://github.com/prometheus-community/windows_exporter/releases) provides all available .msi installers, with respective feature sets and bug fixes. The installer will setup the windows_exporter as a Windows service, as well as create an exception in the Windows firewall. If the installer is run without any parameters, the exporter will run with default settings for enabled collectors, ports, etc.

You can check out the **scheduling best practices** section of this guide which suggests the use of taints/tolerations or RuntimeClass to selectively deploy node exporter only to linux nodes, while the Windows exporter is installed on Windows nodes as you bootstrap the node or using a configuration management tool of your choice (example chef, Ansible, SSM etc).

Note that, unlike the linux nodes where the node exporter is installed as a daemonset , on Windows nodes the WMI exporter is installed on the host itself. The exporter will export metrics such as the CPU usage, the memory and the disk I/O usage and can also be used to monitor IIS sites and applications, the network interfaces and services.

The windows_exporter will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors. However, for advanced use the windows_exporter can be passed an optional list of collectors to filter metrics. The collect[] parameter, in the Prometheus configuration lets you do that.

The default install steps for Windows include downloading and starting the exporter as a service during the bootstrapping process with arguments, such as the collectors you want to filter.

```powershell
> Powershell Invoke-WebRequest https://github.com/prometheus-community/windows_exporter/releases/download/v0.13.0/windows_exporter-0.13.0-amd64.msi -OutFile <DOWNLOADPATH>

> msiexec /i <DOWNLOADPATH> ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,system,container,memory"
```

By default, the metrics can be scraped at the /metrics endpoint on port 9182.
At this point, Prometheus can consume the metrics by adding the following scrape_config to the Prometheus configuration

```yaml
scrape_configs:
    - job_name: "prometheus"
      static_configs:
        - targets: ['localhost:9090']
    ...
    - job_name: "wmi_exporter"
      scrape_interval: 10s
      static_configs:
        - targets: ['<windows-node1-ip>:9182', '<windows-node2-ip>:9182', ...]
```

Prometheus configuration is reloaded using

```bash

> ps aux | grep prometheus
> kill HUP <PID>

```

A better and recommended way to add targets is to use a  Custom Resource Definition called ServiceMonitor, which comes as part of the [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus/releases)] that provides the definition for a ServiceMonitor Object and a controller that will activate the ServiceMonitors we define and automatically build the required Prometheus configuration.

The ServiceMonitor, which declaratively specifies how groups of Kubernetes services should be monitored, is used to define an application you wish to scrape metrics from within Kubernetes. Within the ServiceMonitor we specify the Kubernetes labels that the operator can use to identify the Kubernetes Service which in turn identifies the Pods, that we wish to monitor.

In order to leverage the ServiceMonitor, create an Endpoint object pointing to specific Windows targets, a headless service and a ServiceMontor for the Windows nodes.

```yaml
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: kube-system
subsets:
- addresses:
  - ip: NODE-ONE-IP
    targetRef:
      kind: Node
      name: NODE-ONE-NAME
  - ip: NODE-TWO-IP
    targetRef:
      kind: Node
      name: NODE-TWO-NAME
  - ip: NODE-THREE-IP
    targetRef:
      kind: Node
      name: NODE-THREE-NAME
  ports:
  - name: http-metrics
    port: 9182
    protocol: TCP

---
apiVersion: v1
kind: Service ##Headless Service
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 9182
    protocol: TCP
    targetPort: 9182
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor ##Custom ServiceMonitor Object
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: http-metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: wmiexporter
```

For more details on the operator and the usage of ServiceMonitor, checkout the official [operator](https://github.com/prometheus-operator/kube-prometheus) documentation. Note that Prometheus does support dynamic target discovery using many [service discovery](https://prometheus.io/blog/2015/06/01/advanced-service-discovery/) options.

info  We ve Moved to the AWS Docs         This content has been updated and relocated to improve your experience       Please visit our new site for the latest version       AWS EKS Best Practices Guide  https   docs aws amazon com eks latest best practices windows monitoring html  on the AWS Docs      Bookmarks and links will continue to work  but we recommend updating them for faster access in the future          Monitoring  Prometheus  a  graduated CNCF project  https   www cncf io projects   is by far the most popular monitoring system with native integration into Kubernetes  Prometheus collects metrics around containers  pods  nodes  and clusters  Additionally  Prometheus leverages AlertsManager which lets you program alerts to warn you if something in your cluster is going wrong  Prometheus stores the metric data as a time series data identified by metric name and key value pairs  Prometheus includes away to query using a language called PromQL  which is short for Prometheus Query Language    The high level architecture of Prometheus metrics collection is shown below      Prometheus Metrics collection    images prom png    Prometheus uses a pull mechanism and scrapes metrics from targets using exporters and from the Kubernetes API using the  kube state metrics  https   github com kubernetes kube state metrics   This means applications and services must expose a HTTP S  endpoint containing Prometheus formatted metrics  Prometheus will then  as per its configuration  periodically pull metrics from these HTTP S  endpoints   An exporter lets you consume third party metrics as Prometheus formatted metrics  A Prometheus exporter is typically deployed on each node  For a complete list of exporters please refer to the Prometheus  exporters  https   prometheus io docs instrumenting exporters    While  node exporter  https   github com prometheus node exporter  is suited for exporting host hardware and OS metrics for linux nodes  it wont work for Windows nodes    In a   mixed node EKS cluster with Windows nodes   when you use the stable  Prometheus helm chart  https   github com prometheus community helm charts   you will see failed pods on the Windows nodes  as this exporter is not intended for Windows  You will need to treat the Windows worker pool separate and instead install the  Windows exporter  https   github com prometheus community windows exporter  on the Windows worker node group    In order to setup Prometheus monitoring for Windows nodes  you need to download and install the WMI exporter on the Windows server itself and then setup the targets inside the scrape configuration of the Prometheus configuration file  The  releases page  https   github com prometheus community windows exporter releases  provides all available  msi installers  with respective feature sets and bug fixes  The installer will setup the windows exporter as a Windows service  as well as create an exception in the Windows firewall  If the installer is run without any parameters  the exporter will run with default settings for enabled collectors  ports  etc   You can check out the   scheduling best practices   section of this guide which suggests the use of taints tolerations or RuntimeClass to selectively deploy node exporter only to linux nodes  while the Windows exporter is installed on Windows nodes as you bootstrap the node or using a configuration management tool of your choice  example chef  Ansible  SSM etc    Note that  unlike the linux nodes where the node exporter is installed as a daemonset   on Windows nodes the WMI exporter is installed on the host itself  The exporter will export metrics such as the CPU usage  the memory and the disk I O usage and can also be used to monitor IIS sites and applications  the network interfaces and services    The windows exporter will expose all metrics from enabled collectors by default  This is the recommended way to collect metrics to avoid errors  However  for advanced use the windows exporter can be passed an optional list of collectors to filter metrics  The collect   parameter  in the Prometheus configuration lets you do that   The default install steps for Windows include downloading and starting the exporter as a service during the bootstrapping process with arguments  such as the collectors you want to filter      powershell    Powershell Invoke WebRequest https   github com prometheus community windows exporter releases download v0 13 0 windows exporter 0 13 0 amd64 msi  OutFile  DOWNLOADPATH      msiexec  i  DOWNLOADPATH  ENABLED COLLECTORS  cpu cs logical disk net os system container memory        By default  the metrics can be scraped at the  metrics endpoint on port 9182  At this point  Prometheus can consume the metrics by adding the following scrape config to the Prometheus configuration      yaml  scrape configs        job name   prometheus        static configs             targets    localhost 9090                 job name   wmi exporter        scrape interval  10s       static configs             targets     windows node1 ip  9182     windows node2 ip  9182             Prometheus configuration is reloaded using      bash     ps aux   grep prometheus   kill HUP  PID         A better and recommended way to add targets is to use a  Custom Resource Definition called ServiceMonitor  which comes as part of the  Prometheus operator  https   github com prometheus operator kube prometheus releases   that provides the definition for a ServiceMonitor Object and a controller that will activate the ServiceMonitors we define and automatically build the required Prometheus configuration    The ServiceMonitor  which declaratively specifies how groups of Kubernetes services should be monitored  is used to define an application you wish to scrape metrics from within Kubernetes  Within the ServiceMonitor we specify the Kubernetes labels that the operator can use to identify the Kubernetes Service which in turn identifies the Pods  that we wish to monitor    In order to leverage the ServiceMonitor  create an Endpoint object pointing to specific Windows targets  a headless service and a ServiceMontor for the Windows nodes      yaml apiVersion  v1 kind  Endpoints metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  kube system subsets    addresses      ip  NODE ONE IP     targetRef        kind  Node       name  NODE ONE NAME     ip  NODE TWO IP     targetRef        kind  Node       name  NODE TWO NAME     ip  NODE THREE IP     targetRef        kind  Node       name  NODE THREE NAME   ports      name  http metrics     port  9182     protocol  TCP      apiVersion  v1 kind  Service   Headless Service metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  kube system spec    clusterIP  None   ports      name  http metrics     port  9182     protocol  TCP     targetPort  9182   sessionAffinity  None   type  ClusterIP        apiVersion  monitoring coreos com v1 kind  ServiceMonitor   Custom ServiceMonitor Object metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  monitoring spec    endpoints      interval  30s     port  http metrics   jobLabel  k8s app   namespaceSelector      matchNames        kube system   selector      matchLabels        k8s app  wmiexporter      For more details on the operator and the usage of ServiceMonitor  checkout the official  operator  https   github com prometheus operator kube prometheus  documentation  Note that Prometheus does support dynamic target discovery using many  service discovery  https   prometheus io blog 2015 06 01 advanced service discovery   options


eks Prometheus AlertsManager AlertsManager PromQL exclude true search

Answer:
---
search:
  exclude: true
---

# 모니터링

프로메테우스(Prometheus)는, [CNCF 졸업 프로젝트](https://www.cncf.io/projects/)로서 쿠버네티스에 기본적으로 통합되는 가장 인기 있는 모니터링 시스템입니다. 프로메테우스는 컨테이너, 파드, 노드 및 클러스터와 관련된 메트릭을 수집합니다. 또한 프로메테우스는 AlertsManager를 활용합니다. AlertsManager를 사용하면 클러스터에서 문제가 발생할 경우 경고를 프로그래밍하여 경고할 수 있습니다. 프로메테우스는 지표 데이터를 지표 이름 및 키/값 쌍으로 식별되는 시계열 데이터로 저장합니다. 프로메테우스에는 프로메테우스 쿼리 언어의 줄임말인 PromQL이라는 언어를 사용하여 쿼리하는 방법이 포함되어 있습니다.

프로메테우스 메트릭 수집의 상위 수준 아키텍처는 다음과 같습니다.

![프로메테우스 메트릭 컬렉션](./images/prom.png)

프로메테우스는 풀 메커니즘을 사용하고 익스포터(Exporter)를 사용하여 타겟에서 메트릭을 스크랩하고 [kube state metrics](https://github.com/kubernetes/kube-state-metrics)를 사용하여 쿠버네티스 API에서 메트릭을 스크랩합니다. 즉, 애플리케이션과 서비스는 프로메테우스 형식의 메트릭이 포함된 HTTP(S) 엔드포인트를 노출해야 합니다. 그러면 프로메테우스는 구성에 따라 주기적으로 이런 HTTP(S) 엔드포인트에서 메트릭을 가져옵니다.

익스포터를 사용하면 타사 지표를 프로메테우스 형식의 지표로 사용할 수 있습니다. 프로메테우스 익스포터는 일반적으로 각 노드에 배포됩니다. 익스포터 전체 목록은 프로메테우스 [익스포터 문서](https://prometheus.io/docs/instrumenting/exporters/)를 참조하십시오. [node exporter](https://github.com/prometheus/node_exporter)는 Linux 노드 용 호스트 하드웨어 및 OS 메트릭을 내보내는 데 적합하지만 윈도우 노드에서는 작동하지 않습니다.

**윈도우 노드가 있는 혼합 노드 EKS 클러스터**에서 안정적인 [프로메테우스 헬름 차트](https://github.com/prometheus-community/helm-charts)를 사용하면 윈도우 노드에 장애가 발생한 파드가 표시됩니다. 이 익스포터는 윈도우용이 아니기 때문입니다. 윈도우 작업자 풀을 별도로 처리하고 대신 윈도우 워커 노드 그룹에 [윈도우 익스포터](https://github.com/prometheus-community/windows_exporter)를 설치해야 합니다.

윈도우 노드에 대해 프로메테우스 모니터링을 설정하려면 윈도우 서버 자체에 WMI 익스포터를 다운로드하여 설치한 다음 프로메테우스 구성 파일의 스크랩 구성 내에서 대상을 설정해야 합니다.
[릴리스 페이지](https://github.com/prometheus-community/windows_exporter/releases)는 사용 가능한 모든.msi 설치 프로그램을 각 기능 세트 및 버그 수정과 함께 제공합니다. 설치 프로그램은 windows_exporter를 윈도우 서비스로 설정하고 윈도우 방화벽에서 예외를 생성합니다. 파라미터 없이 설치 프로그램을 실행하는 경우 익스포터는 활성화된 컬렉터, 포트 등에 대한 기본 설정으로 실행됩니다.

이 가이드의 **스케줄링 모범 사례** 섹션은 테인트/톨러레이션 또는 RuntimeClass를 사용하여 노드 익스포터를 Linux 노드에만 선택적으로 배포하는 방법을 제안하는 반면, 윈도우 익스포터는 노드를 부트스트랩하거나 원하는 구성 관리 도구(예: chef, Ansible, SSM 등)를 사용하여 노드 익스포터를 설치하도록 제안합니다.

참고로, 노드 익스포터가 데몬셋으로 설치되는 Linux 노드와 달리 윈도우 노드에서는 WMI 익스포터가 호스트 자체에 설치됩니다. 익스포터는 CPU 사용량, 메모리 및 디스크 I/O 사용량과 같은 메트릭을 내보내고 IIS 사이트 및 응용 프로그램, 네트워크 인터페이스 및 서비스를 모니터링하는 데에도 사용할 수 있습니다.

windows_exporter는 기본적으로 활성화된 컬렉터의 모든 메트릭을 노출합니다. 오류를 방지하기 위해 지표를 수집하는 데 권장되는 방법입니다. 하지만 고급 사용을 위해 windows_exporter에 선택적 수집기 목록을 전달하여 메트릭을 필터링할 수 있습니다.프로메테우스 구성의 collect[] 파라미터를 사용하면 이 작업을 수행할 수 있습니다.

윈도우의 기본 설치 단계에는 부트스트랩 프로세스 중에 필터링하려는 컬렉터와 같은 인수를 사용하여 익스포터를 서비스로 다운로드하고 시작하는 작업이 포함됩니다.

```powershell
> Powershell Invoke-WebRequest https://github.com/prometheus-community/windows_exporter/releases/download/v0.13.0/windows_exporter-0.13.0-amd64.msi -OutFile <DOWNLOADPATH>

> msiexec /i <DOWNLOADPATH> ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,system,container,memory"
```

기본적으로 메트릭은 포트 9182의 /metrics 엔드포인트에서 스크랩할 수 있습니다.
이제 프로메테우스는 다음 scrape_config를 프로메테우스 구성에 추가하여 메트릭을 사용할 수 있습니다.

```yaml
scrape_configs:
    - job_name: "prometheus"
      static_configs:
        - targets: ['localhost:9090']
    ...
    - job_name: "wmi_exporter"
      scrape_interval: 10s
      static_configs:
        - targets: ['<windows-node1-ip>:9182', '<windows-node2-ip>:9182', ...]
```

프로메테우스 구성을 다음과 같이 다시 로드합니다.

```bash

> ps aux | grep prometheus
> kill HUP <PID>

```

대상을 추가하는 더 좋고 권장되는 방법은 ServiceMonitor라는 사용자 지정 리소스 정의를 사용하는 것입니다. 이 정의는 ServiceMonitor 개체에 대한 정의와 우리가 정의한 ServiceMonitor를 활성화하고 필요한 프로메테우스 구성을 자동으로 빌드하는 컨트롤러를 제공하는 [Prometheus 운영자](https://github.com/prometheus-operator/kube-prometheus/releases)의 일부로 제공됩니다.

쿠버네티스 서비스 그룹을 모니터링하는 방법을 선언적으로 지정하는 ServiceMonitor는 쿠버네티스 내에서 메트릭을 스크랩하려는 애플리케이션을 정의하는 데 사용됩니다.ServiceMonitor 내에서 운영자가 쿠버네티스 서비스를 식별하는 데 사용할 수 있는 쿠버네티스 레이블을 지정합니다. 쿠버네티스 서비스는 쿠버네티스 서비스를 식별하고, 쿠버네티스 서비스는 다시 우리가 모니터링하고자 하는 파드를 식별합니다.

ServiceMonitor를 활용하려면 특정 윈도우 대상을 가리키는 엔드포인트 객체, 윈도우 노드용 헤드리스 서비스 및 ServiceMontor를 생성해야 합니다.

```yaml
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: kube-system
subsets:
- addresses:
  - ip: NODE-ONE-IP
    targetRef:
      kind: Node
      name: NODE-ONE-NAME
  - ip: NODE-TWO-IP
    targetRef:
      kind: Node
      name: NODE-TWO-NAME
  - ip: NODE-THREE-IP
    targetRef:
      kind: Node
      name: NODE-THREE-NAME
  ports:
  - name: http-metrics
    port: 9182
    protocol: TCP

---
apiVersion: v1
kind: Service ##Headless Service
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 9182
    protocol: TCP
    targetPort: 9182
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor ##Custom ServiceMonitor Object
metadata:
  labels:
    k8s-app: wmiexporter
  name: wmiexporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: http-metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: wmiexporter
```

운영자 및 ServiceMonitor 사용에 대한 자세한 내용은 공식 [오퍼레이터](https://github.com/prometheus-operator/kube-prometheus) 설명서를 참조하십시오. 참고로 프로메테우스는 다양한 [서비스 디스커버리](https://prometheus.io/blog/2015/06/01/advanced-service-discovery/) 옵션을 사용한 동적 타겟 디스커버리를 지원합니다.

search    exclude  true                      Prometheus     CNCF          https   www cncf io projects                                                                                AlertsManager         AlertsManager                                                                                PromQL                                                                                images prom png                              Exporter                         kube state metrics  https   github com kubernetes kube state metrics              API                                                    HTTP S                                               HTTP S                                                                                https   prometheus io docs instrumenting exporters             node exporter  https   github com prometheus node exporter   Linux                 OS                                                            EKS                              https   github com prometheus community helm charts                                                                                https   github com prometheus community windows exporter                                                       WMI                                                                      https   github com prometheus community windows exporter releases             msi                                              windows exporter                                                                                RuntimeClass                Linux                                                                     chef  Ansible  SSM                                                          Linux                  WMI                            CPU                I O                   IIS                                                           windows exporter                                                                                windows exporter                                               collect                                                                                powershell    Powershell Invoke WebRequest https   github com prometheus community windows exporter releases download v0 13 0 windows exporter 0 13 0 amd64 msi  OutFile  DOWNLOADPATH      msiexec  i  DOWNLOADPATH  ENABLED COLLECTORS  cpu cs logical disk net os system container memory                      9182   metrics                                    scrape config                                        yaml  scrape configs        job name   prometheus        static configs             targets    localhost 9090                 job name   wmi exporter        scrape interval  10s       static configs             targets     windows node1 ip  9182     windows node2 ip  9182                                              bash     ps aux   grep prometheus   kill HUP  PID                                ServiceMonitor                                   ServiceMonitor                    ServiceMonitor                                             Prometheus      https   github com prometheus operator kube prometheus releases                                                    ServiceMonitor                                             ServiceMonitor                                                                                ServiceMonitor                                                     ServiceMontor                yaml apiVersion  v1 kind  Endpoints metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  kube system subsets    addresses      ip  NODE ONE IP     targetRef        kind  Node       name  NODE ONE NAME     ip  NODE TWO IP     targetRef        kind  Node       name  NODE TWO NAME     ip  NODE THREE IP     targetRef        kind  Node       name  NODE THREE NAME   ports      name  http metrics     port  9182     protocol  TCP      apiVersion  v1 kind  Service   Headless Service metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  kube system spec    clusterIP  None   ports      name  http metrics     port  9182     protocol  TCP     targetPort  9182   sessionAffinity  None   type  ClusterIP        apiVersion  monitoring coreos com v1 kind  ServiceMonitor   Custom ServiceMonitor Object metadata    labels      k8s app  wmiexporter   name  wmiexporter   namespace  monitoring spec    endpoints      interval  30s     port  http metrics   jobLabel  k8s app   namespaceSelector      matchNames        kube system   selector      matchLabels        k8s app  wmiexporter            ServiceMonitor                           https   github com prometheus operator kube prometheus                                           https   prometheus io blog 2015 06 01 advanced service discovery


eks Definition Performance Efficiency Pillar The performance efficiency pillar focuses on the efficient use of computing resources to meet requirements and how to maintain that efficiency as demand changes and technologies evolve This section provides in depth best practices guidance for architecting for performance efficiency on AWS Performance efficiency for EKS containers is composed of three areas To ensure the efficient use of EKS container services you should gather data on all aspects of the architecture from the high level design to the selection of EKS resource types By reviewing your choices on a regular basis you ensure that you are taking advantage of the continually evolving Amazon EKS and Container services Monitoring will ensure that you are aware of any deviance from expected performance so you can take action on it

Answer:
# Performance Efficiency Pillar

The performance efficiency pillar focuses on the efficient use of computing resources to meet requirements and how to maintain that efficiency as demand changes and technologies evolve. This section provides in-depth, best practices guidance for architecting for performance efficiency on AWS.

## Definition

To ensure the efficient use of EKS container services, you should gather data on all aspects of the architecture, from the high-level design to the selection of EKS resource types. By reviewing your choices on a regular basis, you ensure that you are taking advantage of the continually evolving Amazon EKS and Container services. Monitoring will ensure that you are aware of any deviance from expected performance so you can take action on it.

Performance efficiency for EKS containers is composed of three areas:

- Optimize your container

- Resource Management

- Scalability Management

## Best Practices

### Optimize your container

You can run most applications in a Docker container without too much hassle. There are a number of things that you need to do to ensure it's running effectively in a production environment, including streamlining the build process. The following best practices will help you to achieve that.

#### Recommendations

- **Make your container images stateless:** A container created with a Docker image should be ephemeral and immutable. In other words, the container should be disposable and independent, i.e. a new one can be built and put in place with absolutely no configuration changes. Design your containers to be stateless. If you would like to use persistent data, use [volumes](https://docs.docker.com/engine/admin/volumes/volumes/) instead. If you would like to store secrets or sensitive application data used by services, you can use solutions like AWS [Systems Manager](https://aws.amazon.com/systems-manager/)[Parameter Store](https://aws.amazon.com/ec2/systems-manager/parameter-store/) or third-party offerings or open source solutions, such as [HashiCorp Valut](https://www.vaultproject.io/) and [Consul](https://www.consul.io/), for runtime configurations.
- [**Minimal base image**](https://docs.docker.com/develop/develop-images/baseimages/) **:** Start with a small base image. Every other instruction in the Dockerfile builds on top of this image. The smaller the base image, the smaller the resulting image is, and the more quickly it can be downloaded. For example, the [alpine:3.7](https://hub.docker.com/r/library/alpine/tags/) image is 71 MB smaller than the [centos:7](https://hub.docker.com/r/library/centos/tags/) image. You can even use the [scratch](https://hub.docker.com/r/library/scratch/) base image, which is an empty image on which you can build your own runtime environment.
- **Avoid unnecessary packages:** When building a container image, include only the dependencies what your application needs and avoid installing unnecessary packages. For example if your application does not need an SSH server, don't include one.  This will reduce complexity, dependencies, file sizes, and build times.  To exclude files not relevant to the build use a .dockerignore file.
- [**Use multi-stage build**](https://docs.docker.com/v17.09/engine/userguide/eng-image/multistage-build/#use-multi-stage-builds):Multi-stage builds allow you to build your application in a first "build" container and use the result in another container, while using the same Dockerfile.  To expand a bit on that, in multi-stage builds, you use multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don't want in the final image.  This method drastically reduces the size of your final image, without struggling to reduce the number of intermediate layers and files.
- **Minimize number of layers:** Each instruction in the Dockerfile adds an extra layer to the Docker image. The number of instructions and layers should be kept to a minimum as this affects build performance and time. For example, the first  instruction below will create multiple layers, whereas the second instruction by using &&(chaining) we reduced the number of layers, which will help provide better performance. The is the best way to reduce the number of layers that will be created in your Dockerfile.
-
    ```
            RUN apt-get -y update
            RUN apt-get install -y python
            RUN apt-get -y update && apt-get install -y python
    ```

- **Properly tag your images:** When building images, always tag them with useful and meaningful tags. This is a good way to organize and document metadata describing an image, for example, by including a unique counter like build id from a CI server (e.g. CodeBuild or Jenkins) to help with identifying the correct image. The tag latest is used by default if you do not provide one in your Docker commands.  We recommend not to use the automatically created latest tag, because by using this tag you'll automatically be running future major releases, which could include breaking changes for your application. The best practice is to avoid the latest tag and instead use the unique digest created by your CI server.
- **Use** [**Build Cache**](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) **to improve build speed** : The cache allows you to take advantage of existing cached images, rather than building each image from scratch. For example, you should add the source code of your application as late as possible in your Dockerfile so that the base image and your application's dependencies get cached and aren't rebuilt on every build. To reuse already cached images, By default in Amazon EKS, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively).
- **Image Security :** Using public images  may be  a great way to start working on containers and deploying it to Kubernetes. However, using them in production can come with a set of challenges. Especially when it comes to security. Ensure to follow the best practices for packaging and distributing the containers/applications. For example, don't build your containers with passwords baked in also you might need to control what's inside them.  Recommend to use private repository such as [Amazon ECR](https://aws.amazon.com/ecr/) and leverage the in-built [image scanning](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html) feature to identify software vulnerabilities in your container images.

- **Right size your containers:**  As you develop and run applications in containers, there are a few key areas to consider. How you size containers and manage your application deployments can negatively impact the end-user experience of services that you provide. To help you succeed, the following best practices will help you right size your containers. After you determine the number of resources required for your application, you should set requests and limits Kubernetes to ensure that your applications are running correctly.

                *(a) Perform testing of the application*:  to gather vital statistics and other performance                 Based upon this data you can work out the optimal configuration, in terms of memory and                 CPU, for your container. Vital statistics such as : __*CPU, Latency, I/O, Memory usage,                Network*__ . Determine expected, mean, and peak container memory and CPU usage by                 doing a separate load test if necessary. Also consider all the processes that might                 potentially run in parallel in the container.

                    Recommend to use  [CloudWatch Container insights](https://aws.amazon.com/blogs/mt/introducing-container-insights-for-amazon-ecs/) or partner products, which will give                 you the right information to size containers and the Worker nodes.

                *(b)Test services independently:*  As many applications depend on each other in a true                 microservice architecture, you need to test them with a high degree of independence                 meaning that the services are both able to properly function by themselves, as well as                 function as part of a cohesive system.

### Resource Management

One of the most common questions that asked in the adoption of Kubernetes is "*What should I put in a Pod?*". For example, a three tier LAMP application container. Should we keep this application in the same pod? Well, this works effectively as a single pod but this is an example of an anti-pattern for Pod creation. There are two reasons for that

***(a)*** If you have both the containers in the same Pod, you are forced to use the same scaling strategy which is not ideal for production environment also you can't effectively manage or constraint resources based on the usage. *E.g:* you might need to scale just the frontend not frontend and backend (MySQL) as a unit also if you would like to increase the resources dedicated just to the backend, you cant just do that.

***(b)*** If you have two separate pods, one for frontend and other for backend. Scaling would be very easy and you get a better reliability.

The above might not work in all the use-cases. In the above example frontend and backend may land in different machines and they will communicate with each other via network, So you need to ask the question "***Will my application work correctly, If they are placed and run on different machines?***" If the answer is a "***no***" may be because of the application design or for some other technical reasons, then grouping of containers in a single pod makes sense. If the answer is "***Yes***" then multiple Pods is the correct approach.

#### Recommendations

+ **Package a single application per container:**
A container works best when a single application runs inside it. This application should have a single parent process. For example, do not run PHP and MySQL in the same container: it's harder to debug, and you can't horizontally scale the PHP container alone. This separation allows you to better tie the lifecycle of the application to that of the container.  Your containers should be both stateless and immutable. Stateless means that any state (persistent data of any kind) is stored outside of the container, for example, you can use different kinds of external storage like Persistent disk, Amazon EBS, and Amazon EFS if needed, or managed database like Amazon RDS.  Immutable means that a container will not be modified during its life: no updates, no patches, and no configuration changes. To update the application code or apply a patch, you build a new image and deploy it.

+ **Use Labels to Kubernetes Objects:**
[Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels) allow Kubernetes objects to be queried and operated upon in bulk. They can also be used to identify and organize Kubernetes objects into groups. As such defining labels should figure right at the top of any Kubernetes best practices list.

+ **Setting resource request limits:**
Setting request limits is the mechanism used to control the amount of system resources that a container can consume such as CPU and memory. These settings are what the container is guaranteed to get when the container initially starts. If a container requests a resource, container orchestrators such as Kubernetes will only schedule it on a node that can provide that resource. Limits, on the other hand, make sure that a container never goes above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

                In the below example Pod manifest, we add a limit of 1.0 CPU and 256 MB of memory

```
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx-pod-webserver
          labels:
            name: nginx-pod
        spec:
          containers:
          - name: nginx
            image: nginx:latest
            resources:
              limits:
                memory: "256Mi"
                cpu: "1000m"
              requests:
                memory: "128Mi"
                cpu: "500m"
            ports:
            - containerPort: 80

```

               It's a best practice to define these requests and limits in your pod definitions. If you don't                include these values, the scheduler doesn't understand what resources are needed.                Without this information, the scheduler might schedule the pod on a node without                sufficient resources to provide acceptable application performance.

+ **Limit the number of concurrent disruptions:**
Use  _PodDisruptionBudget_, This settings allows you to set a policy on the minimum available and maximum unavailable pods during voluntary eviction events. An example of an eviction would be when perform maintenance on the node or draining a node.

                 _Example:_ A web frontend might want to ensure that 8 Pods to be available at any                  given time. In this scenario, an eviction can evict as many pods as it wants, as long as                  eight are available.
```
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: frontend-demo
spec:
  minAvailable: 8
  selector:
    matchLabels:
      app: frontend
```

                 **N.B:** You can also specify pod disruption budget as a percentage by using maxAvailable                  or maxUnavailable parameter.

+ **Use Namespaces:**
Namespaces allows a physical cluster to be shared by multiple teams. A namespace allows to partition created resources into a logically named group. This allows you to set resource quotas per namespace, Role-Based Access Control (RBAC) per namespace, and also network policies per namespace. It gives you soft multitenancy features.

                 For example, If you have three applications running on a single Amazon EKS cluster                  accessed by three different teams which requires multiple resource constraints and                  different levels of QoS each group  you could create a namespace per team and give each                  team a quota on the number of resources that it can utilize, such as CPU and memory.

                 You can also specify default limits in Kubernetes namespaces level by enabling                  [LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/) admission controller. These default limits will constrain the amount of CPU                  or memory a given Pod can use unless the defaults are explicitly overridden by the Pod's                  configuration.

+ **Manage Resource Quota:**
Each namespace can be assigned resource quota. Specifying quota allows to restrict how much of cluster resources can be consumed across all resources in a namespace. Resource quota can be defined by a [ResourceQuota](https://kubernetes.io/docs/concepts/policy/resource-quotas/) object. A presence of ResourceQuota object in a namespace ensures that resource quotas are enforced.

+ **Configure Health Checks for Pods:**
Health checks are a simple way to let the system know if an instance of your app is working or not. If an instance of your app is not working, then other services should not access it or send requests to it. Instead, requests should be sent to another instance of the app that is working. The system also should bring your app back to a healthy state.  By default, all the running pods have the restart policy set to always which means the kubelet running within a node will automatically restart a pod when the container encounters an error. Health checks extend this capability of kubelet through the concept of [container probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes).

  Kubernetes provides two types of [health checks](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/): readiness and liveness probes.  For example, consider if one of your applications, which typically runs for long periods of time, transitions to a non-running state and can only recover by being restarted. You can use liveness probes to detect and remedy such situations. Using health checks gives your applications better reliability, and higher uptime.

+ **Advanced Scheduling Techniques:**
Generally, schedulers ensure that pods are placed only on nodes that have sufficient free resources, and across nodes, they try to balance out the resource utilization across nodes, deployments, replicas, and so on. But sometimes you want to control how your pods are scheduled. For example, perhaps you want to ensure that certain pods are only scheduled on nodes with specialized hardware, such as requiring a GPU machine for an ML workload. Or you want to collocate services that communicate frequently.

  Kubernetes offers many[advanced scheduling features](https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/)and multiple filters/constraints to schedule the pods on the right node.  For example, when using Amazon EKS, you can use[taints and tolerations](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-toleations-beta-feature)to restrict what workloads can run on specific nodes. You can also control pod scheduling using [node selectors](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector)and[affinity and anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity)constructs and even have your own custom scheduler built for this purpose.

#### Scalability Management
  Containers are stateless. They are born and when they die, they are not resurrected.  There are many techniques that you can leverage on Amazon EKS, not only to scale out your containerized applications but also the Kubernetes worker node.

#### Recommendations

  + On Amazon EKS, you can configure [Horizontal pod autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),which automatically scales the number of pods in a replication controller, deployment, or replica set based on observed CPU utilization (or use[custom metrics](https://git.k8s.io/community/contributors/design-proposals/instrumentation/custom-metrics-api.md)based on application-provided metrics).

  + You can use  [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) which automatically adjusts the CPU and memory reservations for your pods to help "right size" your applications. This adjustment can improve cluster resource utilization and free up CPU and memory for other pods.  This is useful in scenarios like your production database "MongoDB" does not scale the same way as a stateless application frontend, In this scenario you could use VPA to scale up the MongoDB Pod.

  + To enable VPA you need to use  Kubernetes metrics server, which is an aggregator of resource usage data in your cluster. It is not deployed by default in Amazon EKS clusters.  You need to configure it before [configure VPA](https://docs.aws.amazon.com/eks/latest/userguide/vertical-pod-autoscaler.html) alternatively you can also use Prometheus to provide metrics for the Vertical Pod Autoscaler.

  + While HPA and VPA scale the deployments and pods, [Cluster Autoscaler](https://github.com/kubernetes/autoscaler) will scale-out  and scale-in the size of the pool of worker nodes. It adjusts the size of a Kubernetes cluster based on the current utilization. Cluster Autoscaler increases the size of the cluster when there are pods that failed to schedule on any of the current nodes due to insufficient resources or when adding a new node would increase the overall availability of cluster resources. Please follow this [step by step](https://eksworkshop.com/scaling/deploy_ca/) guide to setup Cluster Autoscaler.  If you are using Amazon EKS on AWS Fargate, AWS Manages the control plane for you.

     Please have a look at the reliability pillar for detailed information.

#### Monitoring
#### Deployment Best Practices
#### Trade-Offs

Performance Efficiency Pillar  The performance efficiency pillar focuses on the efficient use of computing resources to meet requirements and how to maintain that efficiency as demand changes and technologies evolve  This section provides in depth  best practices guidance for architecting for performance efficiency on AWS      Definition  To ensure the efficient use of EKS container services  you should gather data on all aspects of the architecture  from the high level design to the selection of EKS resource types  By reviewing your choices on a regular basis  you ensure that you are taking advantage of the continually evolving Amazon EKS and Container services  Monitoring will ensure that you are aware of any deviance from expected performance so you can take action on it   Performance efficiency for EKS containers is composed of three areas     Optimize your container    Resource Management    Scalability Management     Best Practices      Optimize your container  You can run most applications in a Docker container without too much hassle  There are a number of things that you need to do to ensure it  39 s running effectively in a production environment  including streamlining the build process  The following best practices will help you to achieve that        Recommendations      Make your container images stateless    A container created with a Docker image should be ephemeral and immutable  In other words  the container should be disposable and independent  i e  a new one can be built and put in place with absolutely no configuration changes  Design your containers to be stateless  If you would like to use persistent data  use  volumes  https   docs docker com engine admin volumes volumes   instead  If you would like to store secrets or sensitive application data used by services  you can use solutions like AWS  Systems Manager  https   aws amazon com systems manager   Parameter Store  https   aws amazon com ec2 systems manager parameter store   or third party offerings or open source solutions  such as  HashiCorp Valut  https   www vaultproject io   and  Consul  https   www consul io    for runtime configurations       Minimal base image    https   docs docker com develop develop images baseimages         Start with a small base image  Every other instruction in the Dockerfile builds on top of this image  The smaller the base image  the smaller the resulting image is  and the more quickly it can be downloaded  For example  the  alpine 3 7  https   hub docker com r library alpine tags   image is 71 MB smaller than the  centos 7  https   hub docker com r library centos tags   image  You can even use the  scratch  https   hub docker com r library scratch   base image  which is an empty image on which you can build your own runtime environment      Avoid unnecessary packages    When building a container image  include only the dependencies what your application needs and avoid installing unnecessary packages  For example if your application does not need an SSH server  don  39 t include one   This will reduce complexity  dependencies  file sizes  and build times   To exclude files not relevant to the build use a  dockerignore file       Use multi stage build    https   docs docker com v17 09 engine userguide eng image multistage build  use multi stage builds  Multi stage builds allow you to build your application in a first  quot build quot  container and use the result in another container  while using the same Dockerfile   To expand a bit on that  in multi stage builds  you use multiple FROM statements in your Dockerfile  Each FROM instruction can use a different base  and each of them begins a new stage of the build  You can selectively copy artifacts from one stage to another  leaving behind everything you don  39 t want in the final image   This method drastically reduces the size of your final image  without struggling to reduce the number of intermediate layers and files      Minimize number of layers    Each instruction in the Dockerfile adds an extra layer to the Docker image  The number of instructions and layers should be kept to a minimum as this affects build performance and time  For example  the first  instruction below will create multiple layers  whereas the second instruction by using  amp  amp  chaining  we reduced the number of layers  which will help provide better performance  The is the best way to reduce the number of layers that will be created in your Dockerfile                         RUN apt get  y update             RUN apt get install  y python             RUN apt get  y update    apt get install  y python                          Properly tag your images    When building images  always tag them with useful and meaningful tags  This is a good way to organize and document metadata describing an image  for example  by including a unique counter like build id from a CI server  e g  CodeBuild or Jenkins  to help with identifying the correct image  The tag latest is used by default if you do not provide one in your Docker commands   We recommend not to use the automatically created latest tag  because by using this tag you  39 ll automatically be running future major releases  which could include breaking changes for your application  The best practice is to avoid the latest tag and instead use the unique digest created by your CI server      Use      Build Cache    https   docs docker com develop develop images dockerfile best practices     to improve build speed     The cache allows you to take advantage of existing cached images  rather than building each image from scratch  For example  you should add the source code of your application as late as possible in your Dockerfile so that the base image and your application  39 s dependencies get cached and aren  39 t rebuilt on every build  To reuse already cached images  By default in Amazon EKS  the kubelet will try to pull each image from the specified registry  However  if the imagePullPolicy property of the container is set to IfNotPresent or Never  then a local image is used  preferentially or exclusively  respectively       Image Security     Using public images  may be  a great way to start working on containers and deploying it to Kubernetes  However  using them in production can come with a set of challenges  Especially when it comes to security  Ensure to follow the best practices for packaging and distributing the containers applications  For example  don  39 t build your containers with passwords baked in also you might need to control what  39 s inside them   Recommend to use private repository such as  Amazon ECR  https   aws amazon com ecr   and leverage the in built  image scanning  https   docs aws amazon com AmazonECR latest userguide image scanning html  feature to identify software vulnerabilities in your container images         Right size your containers     As you develop and run applications in containers  there are a few key areas to consider  How you size containers and manage your application deployments can negatively impact the end user experience of services that you provide  To help you succeed  the following best practices will help you right size your containers  After you determine the number of resources required for your application  you should set requests and limits Kubernetes to ensure that your applications are running correctly     nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp   a  Perform testing of the application    to gather vital statistics and other performance  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp Based upon this data you can work out the optimal configuration  in terms of memory and  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp CPU  for your container  Vital statistics such as      CPU  Latency  I O  Memory usage   nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp Network      Determine expected  mean  and peak container memory and CPU usage by  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp doing a separate load test if necessary  Also consider all the processes that might  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp potentially run in parallel in the container     nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp      Recommend to use   CloudWatch Container insights  https   aws amazon com blogs mt introducing container insights for amazon ecs   or partner products  which will give  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp you the right information to size containers and the Worker nodes     nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp   b Test services independently    As many applications depend on each other in a true  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp microservice architecture  you need to test them with a high degree of independence  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp meaning that the services are both able to properly function by themselves  as well as  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp function as part of a cohesive system       Resource Management   One of the most common questions that asked in the adoption of Kubernetes is  quot  What should I put in a Pod   quot   For example  a three tier LAMP application container  Should we keep this application in the same pod  Well  this works effectively as a single pod but this is an example of an anti pattern for Pod creation  There are two reasons for that       a     If you have both the containers in the same Pod  you are forced to use the same scaling strategy which is not ideal for production environment also you can  39 t effectively manage or constraint resources based on the usage   E g   you might need to scale just the frontend not frontend and backend  MySQL  as a unit also if you would like to increase the resources dedicated just to the backend  you cant just do that       b     If you have two separate pods  one for frontend and other for backend  Scaling would be very easy and you get a better reliability   The above might not work in all the use cases  In the above example frontend and backend may land in different machines and they will communicate with each other via network  So you need to ask the question  quot    Will my application work correctly  If they are placed and run on different machines     quot  If the answer is a  quot    no    quot  may be because of the application design or for some other technical reasons  then grouping of containers in a single pod makes sense  If the answer is  quot    Yes    quot  then multiple Pods is the correct approach        Recommendations      Package a single application per container    A container works best when a single application runs inside it  This application should have a single parent process  For example  do not run PHP and MySQL in the same container  it  39 s harder to debug  and you can  39 t horizontally scale the PHP container alone  This separation allows you to better tie the lifecycle of the application to that of the container   Your containers should be both stateless and immutable  Stateless means that any state  persistent data of any kind  is stored outside of the container  for example  you can use different kinds of external storage like Persistent disk  Amazon EBS  and Amazon EFS if needed  or managed database like Amazon RDS   Immutable means that a container will not be modified during its life  no updates  no patches  and no configuration changes  To update the application code or apply a patch  you build a new image and deploy it       Use Labels to Kubernetes Objects     Labels  https   kubernetes io docs concepts overview working with objects common labels  labels  allow Kubernetes objects to be queried and operated upon in bulk  They can also be used to identify and organize Kubernetes objects into groups  As such defining labels should figure right at the top of any Kubernetes best practices list       Setting resource request limits    Setting request limits is the mechanism used to control the amount of system resources that a container can consume such as CPU and memory  These settings are what the container is guaranteed to get when the container initially starts  If a container requests a resource  container orchestrators such as Kubernetes will only schedule it on a node that can provide that resource  Limits  on the other hand  make sure that a container never goes above a certain value  The container is only allowed to go up to the limit  and then it is restricted    nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  In the below example Pod manifest  we add a limit of 1 0 CPU and 256 MB of memory              apiVersion  v1         kind  Pod         metadata            name  nginx pod webserver           labels              name  nginx pod         spec            containers              name  nginx             image  nginx latest             resources                limits                  memory   256Mi                  cpu   1000m                requests                  memory   128Mi                  cpu   500m              ports                containerPort  80                   nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp It  39 s a best practice to define these requests and limits in your pod definitions  If you don  39 t  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp include these values  the scheduler doesn  39 t understand what resources are needed   nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp Without this information  the scheduler might schedule the pod on a node without  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp sufficient resources to provide acceptable application performance       Limit the number of concurrent disruptions    Use   PodDisruptionBudget   This settings allows you to set a policy on the minimum available and maximum unavailable pods during voluntary eviction events  An example of an eviction would be when perform maintenance on the node or draining a node    nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp   Example   A web frontend might want to ensure that 8 Pods to be available at any  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp given time  In this scenario  an eviction can evict as many pods as it wants  as long as  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp eight are available      apiVersion  policy v1beta1 kind  PodDisruptionBudget metadata    name  frontend demo spec    minAvailable  8   selector      matchLabels        app  frontend       nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp   N B    You can also specify pod disruption budget as a percentage by using maxAvailable  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp or maxUnavailable parameter       Use Namespaces    Namespaces allows a physical cluster to be shared by multiple teams  A namespace allows to partition created resources into a logically named group  This allows you to set resource quotas per namespace  Role Based Access Control  RBAC  per namespace  and also network policies per namespace  It gives you soft multitenancy features    nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp For example  If you have three applications running on a single Amazon EKS cluster  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp accessed by three different teams which requires multiple resource constraints and  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp different levels of QoS each group  you could create a namespace per team and give each  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp team a quota on the number of resources that it can utilize  such as CPU and memory    nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp You can also specify default limits in Kubernetes namespaces level by enabling  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  LimitRange  https   kubernetes io docs concepts policy limit range   admission controller  These default limits will constrain the amount of CPU  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp or memory a given Pod can use unless the defaults are explicitly overridden by the Pod  39 s  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp  nbsp configuration       Manage Resource Quota     Each namespace can be assigned resource quota  Specifying quota allows to restrict how much of cluster resources can be consumed across all resources in a namespace  Resource quota can be defined by a  ResourceQuota  https   kubernetes io docs concepts policy resource quotas   object  A presence of ResourceQuota object in a namespace ensures that resource quotas are enforced       Configure Health Checks for Pods    Health checks are a simple way to let the system know if an instance of your app is working or not  If an instance of your app is not working  then other services should not access it or send requests to it  Instead  requests should be sent to another instance of the app that is working  The system also should bring your app back to a healthy state   By default  all the running pods have the restart policy set to always which means the kubelet running within a node will automatically restart a pod when the container encounters an error  Health checks extend this capability of kubelet through the concept of  container probes  https   kubernetes io docs concepts workloads pods pod lifecycle  container probes      Kubernetes provides two types of  health checks  https   kubernetes io docs tasks configure pod container configure liveness readiness startup probes    readiness and liveness probes   For example  consider if one of your applications  which typically runs for long periods of time  transitions to a non running state and can only recover by being restarted  You can use liveness probes to detect and remedy such situations  Using health checks gives your applications better reliability  and higher uptime        Advanced Scheduling Techniques    Generally  schedulers ensure that pods are placed only on nodes that have sufficient free resources  and across nodes  they try to balance out the resource utilization across nodes  deployments  replicas  and so on  But sometimes you want to control how your pods are scheduled  For example  perhaps you want to ensure that certain pods are only scheduled on nodes with specialized hardware  such as requiring a GPU machine for an ML workload  Or you want to collocate services that communicate frequently     Kubernetes offers many advanced scheduling features  https   kubernetes io blog 2017 03 advanced scheduling in kubernetes  and multiple filters constraints to schedule the pods on the right node   For example  when using Amazon EKS  you can use taints and tolerations  https   kubernetes io docs concepts configuration assign pod node  taints and toleations beta feature to restrict what workloads can run on specific nodes  You can also control pod scheduling using  node selectors  https   kubernetes io docs concepts configuration assign pod node  nodeselector and affinity and anti affinity  https   kubernetes io docs concepts configuration assign pod node  affinity and anti affinity constructs and even have your own custom scheduler built for this purpose        Scalability Management    Containers are stateless  They are born and when they die  they are not resurrected   There are many techniques that you can leverage on Amazon EKS  not only to scale out your containerized applications but also the Kubernetes worker node          Recommendations      On Amazon EKS  you can configure  Horizontal pod autoscaler  https   kubernetes io docs tasks run application horizontal pod autoscale   which automatically scales the number of pods in a replication controller  deployment  or replica set based on observed CPU utilization  or use custom metrics  https   git k8s io community contributors design proposals instrumentation custom metrics api md based on application provided metrics        You can use   Vertical Pod Autoscaler  https   github com kubernetes autoscaler tree master vertical pod autoscaler  which automatically adjusts the CPU and memory reservations for your pods to help  quot right size quot  your applications  This adjustment can improve cluster resource utilization and free up CPU and memory for other pods   This is useful in scenarios like your production database  quot MongoDB quot  does not scale the same way as a stateless application frontend  In this scenario you could use VPA to scale up the MongoDB Pod       To enable VPA you need to use  Kubernetes metrics server  which is an aggregator of resource usage data in your cluster  It is not deployed by default in Amazon EKS clusters   You need to configure it before  configure VPA  https   docs aws amazon com eks latest userguide vertical pod autoscaler html  alternatively you can also use Prometheus to provide metrics for the Vertical Pod Autoscaler       While HPA and VPA scale the deployments and pods   Cluster Autoscaler  https   github com kubernetes autoscaler  will scale out  and scale in the size of the pool of worker nodes  It adjusts the size of a Kubernetes cluster based on the current utilization  Cluster Autoscaler increases the size of the cluster when there are pods that failed to schedule on any of the current nodes due to insufficient resources or when adding a new node would increase the overall availability of cluster resources  Please follow this  step by step  https   eksworkshop com scaling deploy ca   guide to setup Cluster Autoscaler   If you are using Amazon EKS on AWS Fargate  AWS Manages the control plane for you         Please have a look at the reliability pillar for detailed information             Monitoring       Deployment Best Practices       Trade Offs