kubernetes reference cici37 contenttype concept min kubernetes server version 1 25 reviewers weight 35 jpbetz title Common Expression Language in Kubernetes

Answer:
---
title: Common Expression Language in Kubernetes
reviewers:
- jpbetz
- cici37
content_type: concept
weight: 35
min-kubernetes-server-version: 1.25
---

<!-- overview -->

The [Common Expression Language (CEL)](https://github.com/google/cel-go) is used
in the Kubernetes API to declare validation rules, policy rules, and other
constraints or conditions.

CEL expressions are evaluated directly in the
, making CEL a
convenient alternative to out-of-process mechanisms, such as webhooks, for many
extensibility use cases. Your CEL expressions continue to execute so long as the
control plane's API server component remains available.

<!-- body -->

## Language overview

The [CEL language](https://github.com/google/cel-spec/blob/master/doc/langdef.md)
has a straightforward syntax that is similar to the expressions in C, C++, Java,
JavaScript and Go.

CEL was designed to be embedded into applications. Each CEL "program" is a
single expression that evaluates to a single value. CEL expressions are
typically short "one-liners" that inline well into the string fields of Kubernetes
API resources.

Inputs to a CEL program are "variables". Each Kubernetes API field that contains
CEL declares in the API documentation which variables are available to use for
that field. For example, in the `x-kubernetes-validations[i].rules` field of
CustomResourceDefinitions, the `self` and `oldSelf` variables are available and
refer to the previous and current state of the custom resource data to be
validated by the CEL expression. Other Kubernetes API fields may declare
different variables. See the API documentation of the API fields to learn which
variables are available for that field.

Example CEL expressions:

| Rule                                                                               | Purpose                                                                           |
|------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| `self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas`           | Validate that the three fields defining replicas are ordered appropriately        |
| `'Available' in self.stateCounts`                                                  | Validate that an entry with the 'Available' key exists in a map                   |
| `(self.list1.size() == 0) != (self.list2.size() == 0)`                             | Validate that one of two lists is non-empty, but not both                         |
| `self.envars.filter(e, e.name = 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$'))`  | Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV'  |
| `has(self.expired) && self.created + self.ttl < self.expired`                      | Validate that 'expired' date is after a 'create' date plus a 'ttl' duration       |
| `self.health.startsWith('ok')`                                                     | Validate a 'health' string field has the prefix 'ok'                              |
| `self.widgets.exists(w, w.key == 'x' && w.foo < 10)`                               | Validate that the 'foo' property of a listMap item with a key 'x' is less than 10 |
| `type(self) == string ? self == '99%' : self == 42`                                | Validate an int-or-string field for both the int and string cases                 |
| `self.metadata.name == 'singleton'`                                                | Validate that an object's name matches a specific value (making it a singleton)   |
| `self.set1.all(e, !(e in self.set2))`                                              | Validate that two listSets are disjoint                                           |
| `self.names.size() == self.details.size() && self.names.all(n, n in self.details)` | Validate the 'details' map is keyed by the items in the 'names' listSet           |
| `self.details.all(key, key.matches('^[a-zA-Z]*$'))`                                | Validate the keys of the 'details' map                                            |
| `self.details.all(key, self.details[key].matches('^[a-zA-Z]*$'))`                  | Validate the values of the 'details' map                                          |

## CEL options, language features, and libraries

CEL is configured with the following options, libraries and language features, introduced at the specified Kubernetes versions:

| CEL option, library or language feature | Included | Availablity |
|-----------------------------------------|----------|-------------|
| [Standard macros](https://github.com/google/cel-spec/blob/v0.7.0/doc/langdef.md#macros) | `has`, `all`, `exists`, `exists_one`, `map`, `filter` | All Kubernetes versions |
| [Standard functions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#list-of-standard-definitions) | See [official list of standard definitions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#list-of-standard-definitions) | All Kubernetes versions |
| [Homogeneous Aggregate Literals](https://pkg.go.dev/github.com/google/cel-go@v0.17.4/cel#HomogeneousAggregateLiterals) | | All Kubernetes versions |
| [Default UTC Time Zone](https://pkg.go.dev/github.com/google/cel-go@v0.17.4/cel#DefaultUTCTimeZone) | | All Kubernetes versions |
| [Eagerly Validate Declarations](https://pkg.go.dev/github.com/google/cel-go@v0.17.4/cel#EagerlyValidateDeclarations) | | All Kubernetes versions |
| [Extended strings library](https://pkg.go.dev/github.com/google/cel-go/ext#Strings), Version 1 | `charAt`, `indexOf`, `lastIndexOf`, `lowerAscii`, `upperAscii`, `replace`, `split`, `join`, `substring`, `trim` | All Kubernetes versions |
| Kubernetes list library | See [Kubernetes list library](#kubernetes-list-library) | All Kubernetes versions |
| Kubernetes regex library | See [Kubernetes regex library](#kubernetes-regex-library) | All Kubernetes versions |
| Kubernetes URL library | See [Kubernetes URL library](#kubernetes-url-library) | All Kubernetes versions |
| Kubernetes authorizer library | See [Kubernetes authorizer library](#kubernetes-authorizer-library) | All Kubernetes versions |
| Kubernetes quantity library | See [Kubernetes quantity library](#kubernetes-quantity-library) | Kubernetes versions 1.29+ |
| CEL optional types | See [CEL optional types](https://pkg.go.dev/github.com/google/cel-go@v0.17.4/cel#OptionalTypes) | Kubernetes versions 1.29+ |
| CEL CrossTypeNumericComparisons | See [CEL CrossTypeNumericComparisons](https://pkg.go.dev/github.com/google/cel-go@v0.17.4/cel#CrossTypeNumericComparisons) | Kubernetes versions 1.29+ |

CEL functions, features and language settings support Kubernetes control plane
rollbacks. For example, _CEL Optional Values_ was introduced at Kubernetes 1.29
and so only API servers at that version or newer will accept write requests to
CEL expressions that use _CEL Optional Values_. However, when a cluster is
rolled back to Kubernetes 1.28 CEL expressions using "CEL Optional Values" that
are already stored in API resources will continue to evaluate correctly.

## Kubernetes CEL libraries

In additional to the CEL community libraries, Kubernetes includes CEL libraries
that are available everywhere CEL is used in Kubernetes.

### Kubernetes list library

The list library includes `indexOf` and `lastIndexOf`, which work similar to the
strings functions of the same names. These functions either the first or last
positional index of the provided element in the list.

The list library also includes `min`, `max` and `sum`. Sum is supported on all
number types as well as the duration type. Min and max are supported on all
comparable types.

`isSorted` is also provided as a convenience function and is supported on all
comparable types.

Examples:

| CEL Expression                                                                     | Purpose                                                   |
|------------------------------------------------------------------------------------|-----------------------------------------------------------|
| `names.isSorted()`                                                                 | Verify that a list of names is kept in alphabetical order |
| `items.map(x, x.weight).sum() == 1.0`                                              | Verify that the "weights" of a list of objects sum to 1.0 |
| `lowPriorities.map(x, x.priority).max() < highPriorities.map(x, x.priority).min()` | Verify that two sets of priorities do not overlap         |
| `names.indexOf('should-be-first') == 1`                                            | Require that the first name in a list if a specific value |

See the [Kubernetes List Library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#Lists)
godoc for more information.

### Kubernetes regex library

In addition to the `matches` function provided by the CEL standard library, the
regex library provides `find` and `findAll`, enabling a much wider range of
regex operations.

Examples:

| CEL Expression                                              | Purpose                                                  |
|-------------------------------------------------------------|----------------------------------------------------------|
| `"abc 123".find('[0-9]+')`                                  | Find the first number in a string                        |
| `"1, 2, 3, 4".findAll('[0-9]+').map(x, int(x)).sum() < 100` | Verify that the numbers in a string sum to less than 100 |

See the [Kubernetes regex library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#Regex)
godoc for more information.

### Kubernetes URL library

To make it easier and safer to process URLs, the following functions have been added:

- `isURL(string)` checks if a string is a valid URL according to the
  [Go's net/url](https://pkg.go.dev/net/url#URL) package. The string must be an
  absolute URL.
- `url(string) URL` converts a string to a URL or results in an error if the
  string is not a valid URL.

Once parsed via the `url` function, the resulting URL object has `getScheme`,
`getHost`, `getHostname`, `getPort`, `getEscapedPath` and `getQuery` accessors.

Examples:

| CEL Expression                                                  | Purpose                                        |
|-----------------------------------------------------------------|------------------------------------------------|
| `url('https://example.com:80/').getHost()`                      | Gets the 'example.com:80' host part of the URL |
| `url('https://example.com/path with spaces/').getEscapedPath()` | Returns '/path%20with%20spaces/'               |

See the [Kubernetes URL library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#URLs)
godoc for more information.

### Kubernetes authorizer library

For CEL expressions in the API where a variable of type `Authorizer` is available,
the authorizer may be used to perform authorization checks for the principal
(authenticated user) of the request.

API resource checks are performed as follows:

1. Specify the group and resource to check: `Authorizer.group(string).resource(string) ResourceCheck`
1. Optionally call any combination of the following builder functions to further narrow the authorization check.
   Note that these functions return the receiver type and can be chained:
   - `ResourceCheck.subresource(string) ResourceCheck`
   - `ResourceCheck.namespace(string) ResourceCheck`
   - `ResourceCheck.name(string) ResourceCheck`
1. Call `ResourceCheck.check(verb string) Decision` to perform the authorization check.
1. Call `allowed() bool` or `reason() string` to inspect the result of the authorization check.

Non-resource authorization performed are used as follows:

1. Specify only a path: `Authorizer.path(string) PathCheck`
1. Call `PathCheck.check(httpVerb string) Decision` to perform the authorization check.
1. Call `allowed() bool` or `reason() string` to inspect the result of the authorization check.

To perform an authorization check for a service account:

- `Authorizer.serviceAccount(namespace string, name string) Authorizer`

| CEL Expression | Purpose |
|----------------|---------|
| `authorizer.group('').resource('pods').namespace('default').check('create').allowed()` | Returns true if the principal (user or service account) is allowed create pods in the 'default' namespace. |
| `authorizer.path('/healthz').check('get').allowed()` | Checks if the principal (user or service account) is authorized to make HTTP GET requests to the /healthz API path. |
| `authorizer.serviceAccount('default', 'myserviceaccount').resource('deployments').check('delete').allowed()` | Checks if the service account is authorized to delete deployments. |

With the alpha `AuthorizeWithSelectors` feature enabled, field and label selectors can be added to authorization checks.

| CEL Expression | Purpose |
|----------------|---------|
| `authorizer.group('').resource('pods').fieldSelector('spec.nodeName=mynode').check('list').allowed()` | Returns true if the principal (user or service account) is allowed to list pods with the field selector `spec.nodeName=mynode`. |
| `authorizer.group('').resource('pods').labelSelector('example.com/mylabel=myvalue').check('list').allowed()` | Returns true if the principal (user or service account) is allowed to list pods with the label selector `example.com/mylabel=myvalue`. |

See the [Kubernetes Authz library](https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz)
and [Kubernetes AuthzSelectors library](https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#AuthzSelectors)
godoc for more information.

### Kubernetes quantity library

Kubernetes 1.28 adds support for manipulating quantity strings (ex 1.5G, 512k, 20Mi)

- `isQuantity(string)` checks if a string is a valid Quantity according to
  [Kubernetes' resource.Quantity](https://pkg.go.dev/k8s.io/apimachinery/pkg/api/resource#Quantity).
- `quantity(string) Quantity` converts a string to a Quantity or results in an error if the
  string is not a valid quantity.

Once parsed via the `quantity` function, the resulting Quantity object has the
following library of member functions:

| Member Function             | CEL Return Value | Description |
|-----------------------------|------------------|-------------|
| `isInteger()`               | bool             | Returns true if and only if asInteger is safe to call without an error |
| `asInteger()`               | int              | Returns a representation of the current value as an int64 if possible or results in an error if conversion would result in overflow or loss of precision. |
| `asApproximateFloat()`      | float            | Returns a float64 representation of the quantity which may lose precision. If the value of the quantity is outside the range of a float64 +Inf/-Inf will be returned. |
| `sign()`                    | int              | Returns `1` if the quantity is positive, `-1` if it is negative. `0` if it is zero |
| `add(<Quantity>)`           | Quantity         | Returns sum of two quantities |
| `add(<int>)`                | Quantity         | Returns sum of quantity and an integer |
| `sub(<Quantity>)`           | Quantity         | Returns difference between two quantities |
| `sub(<int>)`                | Quantity         | Returns difference between a quantity and an integer |
| `isLessThan(<Quantity>)`    | bool             | Returns true if and only if the receiver is less than the operand |
| `isGreaterThan(<Quantity>)` | bool             | Returns true if and only if the receiver is greater than the operand |
| `compareTo(<Quantity>)`     | int              | Compares receiver to operand and returns 0 if they are equal, 1 if the receiver is greater, or -1 if the receiver is less than the operand |

Examples:

| CEL Expression                                                            | Purpose                                               |
|---------------------------------------------------------------------------|-------------------------------------------------------|
| `quantity("500000G").isInteger()`                                         | Test if conversion to integer would throw an error    |
| `quantity("50k").asInteger()`                                             | Precise conversion to integer                         |
| `quantity("9999999999999999999999999999999999999G").asApproximateFloat()` | Lossy conversion to float                              |
| `quantity("50k").add(quantity("20k"))`                                   | Add two quantities                                    |
| `quantity("50k").sub(20000)`                                              | Subtract an integer from a quantity                   |
| `quantity("50k").add(20).sub(quantity("100k")).sub(-50000)`               | Chain adding and subtracting integers and quantities  |
| `quantity("200M").compareTo(quantity("0.2G"))`                            | Compare two quantities                                |
| `quantity("150Mi").isGreaterThan(quantity("100Mi"))`                      | Test if a quantity is greater than the receiver       |
| `quantity("50M").isLessThan(quantity("100M"))`                            | Test if a quantity is less than the receiver          |

## Type checking

CEL is a [gradually typed language](https://github.com/google/cel-spec/blob/master/doc/langdef.md#gradual-type-checking).

Some Kubernetes API fields contain fully type checked CEL expressions. For example,
[CustomResourceDefinitions Validation Rules](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules)
are fully type checked.

Some Kubernetes API fields contain partially type checked CEL expressions. A
partially type checked expression is an expressions where some of the variables
are statically typed but others are dynamically typed. For example, in the CEL
expressions of
[ValidatingAdmissionPolicies](/docs/reference/access-authn-authz/validating-admission-policy/)
the `request` variable is typed, but the `object` variable is dynamically typed.
As a result, an expression containing `request.namex` would fail type checking
because the `namex` field is not defined. However, `object.namex` would pass
type checking even when the `namex` field is not defined for the resource kinds
that `object` refers to, because `object` is dynamically typed.

The `has()` macro in CEL may be used in CEL expressions to check if a field of a
dynamically typed variable is accessible before attempting to access the field's
value. For example:

```cel
has(object.namex) ? object.namex == 'special' : request.name == 'special'
```

## Type system integration

| OpenAPIv3 type                                     | CEL type |
|----------------------------------------------------|----------|
| 'object' with Properties                           | object / "message type" (`type(<object>)` evaluates to `selfType<uniqueNumber>.path.to.object.from.self`) |
| 'object' with AdditionalProperties                 | map |
| 'object' with x-kubernetes-embedded-type           | object / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema |
| 'object' with x-kubernetes-preserve-unknown-fields | object / "message type", unknown fields are NOT accessible in CEL expression |
| x-kubernetes-int-or-string                         | union of int or string, `self.intOrString < 100 \|\| self.intOrString == '50%'` evaluates to true for both `50` and `"50%"` |
| 'array'                                            | list |
| 'array' with x-kubernetes-list-type=map            | list with map based Equality & unique key guarantees |
| 'array' with x-kubernetes-list-type=set            | list with set based Equality & unique entry guarantees |
| 'boolean'                                          | boolean |
| 'number' (all formats)                             | double |
| 'integer' (all formats)                            | int (64) |
| _no equivalent_                                    | uint (64) |
| 'null'                                             | null_type |
| 'string'                                           | string |
| 'string' with format=byte (base64 encoded)         | bytes |
| 'string' with format=date                          | timestamp (google.protobuf.Timestamp) |
| 'string' with format=datetime                      | timestamp (google.protobuf.Timestamp) |
| 'string' with format=duration                      | duration (google.protobuf.Duration) |

Also see: [CEL types](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#values),
[OpenAPI types](https://swagger.io/specification/#data-types),
[Kubernetes Structural Schemas](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema).

Equality comparison for arrays with `x-kubernetes-list-type` of `set` or `map` ignores element
order. For example `[1, 2] == [2, 1]` if the arrays represent Kubernetes `set` values.

Concatenation on arrays with `x-kubernetes-list-type` use the semantics of the
list type:

`set`
: `X + Y` performs a union where the array positions of all elements in
  `X` are preserved and non-intersecting elements in `Y` are appended, retaining
  their partial order.

`map`
: `X + Y` performs a merge where the array positions of all keys in `X`
  are preserved but the values are overwritten by values in `Y` when the key
  sets of `X` and `Y` intersect. Elements in `Y` with non-intersecting keys are
  appended, retaining their partial order.

## Escaping

Only Kubernetes resource property names of the form
`[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible from CEL. Accessible property
names are escaped according to the following rules when accessed in the
expression:

| escape sequence   | property name equivalent                                                                     |
|-------------------|----------------------------------------------------------------------------------------------|
| `__underscores__` | `__`                                                                                |
| `__dot__`         | `.`                                                                                |
| `__dash__`        | `-`                                                                                |
| `__slash__`       | `/`                                                                                |
| `__{keyword}__`   | [CEL **RESERVED** keyword](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#syntax) |

When you escape any of CEL's **RESERVED** keywords you need to match the exact property name
use the underscore escaping
(for example, `int` in the word `sprint` would not be escaped and nor would it need to be).

Examples on escaping:

| property name | rule with escaped property name   |
|---------------|-----------------------------------|
| `namespace`   | `self.__namespace__ > 0`          |
| `x-prop`      | `self.x__dash__prop > 0`          |
| `redact__d`   | `self.redact__underscores__d > 0` |
| `string`      | `self.startsWith('kube')`         |

## Resource constraints

CEL is non-Turing complete and offers a variety of production safety controls to
limit execution time. CEL's _resource constraint_ features provide feedback to
developers about expression complexity and help protect the API server from
excessive resource consumption during evaluation. CEL's resource constraint
features are used to prevent CEL evaluation from consuming excessive API server
resources.

A key element of the resource constraint features is a _cost unit_ that CEL
defines as a way of tracking CPU utilization. Cost units are independent of
system load and hardware. Cost units are also deterministic; for any given CEL
expression and input data, evaluation of the expression by the CEL interpreter
will always result in the same cost.

Many of CEL's core operations have fixed costs. The simplest operations, such as
comparisons (e.g. `<`) have a cost of 1. Some have a higher fixed cost, for
example list literal declarations have a fixed base cost of 40 cost units.

Calls to functions implemented in native code approximate cost based on the time
complexity of the operation. For example: operations that use regular
expressions, such as `match` and `find`, are estimated using an approximated
cost of `length(regexString)*length(inputString)`. The approximated cost
reflects the worst case time complexity of Go's RE2 implementation.

### Runtime cost budget

All CEL expressions evaluated by Kubernetes are constrained by a runtime cost
budget. The runtime cost budget is an estimate of actual CPU utilization
computed by incrementing a cost unit counter while interpreting a CEL
expression. If the CEL interpreter executes too many instructions, the runtime
cost budget will be exceeded, execution of the expressions will be halted, and
an error will result.

Some Kubernetes resources define an additional runtime cost budget that bounds
the execution of multiple expressions. If the sum total of the cost of
expressions exceed the budget, execution of the expressions will be halted, and
an error will result. For example the validation of a custom resource has a
_per-validation_ runtime cost budget for all
[Validation Rules](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules)
evaluated to validate the custom resource.

### Estimated cost limits

For some Kubernetes resources, the API server may also check if worst case
estimated running time of CEL expressions would be prohibitively expensive to
execute. If so, the API server prevent the CEL expression from being written to
API resources by rejecting create or update operations containing the CEL
expression to the API resources. This feature offers a stronger assurance that
CEL expressions written to the API resource will be evaluated at runtime without
exceeding the runtime cost budget.

title  Common Expression Language in Kubernetes reviewers    jpbetz   cici37 content type  concept weight  35 min kubernetes server version  1 25           overview      The  Common Expression Language  CEL   https   github com google cel go  is used in the Kubernetes API to declare validation rules  policy rules  and other constraints or conditions   CEL expressions are evaluated directly in the   making CEL a convenient alternative to out of process mechanisms  such as webhooks  for many extensibility use cases  Your CEL expressions continue to execute so long as the control plane s API server component remains available        body         Language overview  The  CEL language  https   github com google cel spec blob master doc langdef md  has a straightforward syntax that is similar to the expressions in C  C    Java  JavaScript and Go   CEL was designed to be embedded into applications  Each CEL  program  is a single expression that evaluates to a single value  CEL expressions are typically short  one liners  that inline well into the string fields of Kubernetes API resources   Inputs to a CEL program are  variables   Each Kubernetes API field that contains CEL declares in the API documentation which variables are available to use for that field  For example  in the  x kubernetes validations i  rules  field of CustomResourceDefinitions  the  self  and  oldSelf  variables are available and refer to the previous and current state of the custom resource data to be validated by the CEL expression  Other Kubernetes API fields may declare different variables  See the API documentation of the API fields to learn which variables are available for that field   Example CEL expressions      Rule                                                                                Purpose                                                                                self minReplicas    self replicas    self replicas    self maxReplicas              Validate that the three fields defining replicas are ordered appropriately              Available  in self stateCounts                                                     Validate that an entry with the  Available  key exists in a map                         self list1 size      0      self list2 size      0                                 Validate that one of two lists is non empty  but not both                              self envars filter e  e name    MY ENV   all e  e value matches    a zA Z           Validate the  value  field of a listMap entry where key field  name  is  MY ENV        has self expired     self created   self ttl   self expired                         Validate that  expired  date is after a  create  date plus a  ttl  duration            self health startsWith  ok                                                          Validate a  health  string field has the prefix  ok                                    self widgets exists w  w key     x     w foo   10                                   Validate that the  foo  property of a listMap item with a key  x  is less than 10      type self     string   self     99     self    42                                   Validate an int or string field for both the int and string cases                      self metadata name     singleton                                                    Validate that an object s name matches a specific value  making it a singleton         self set1 all e    e in self set2                                                   Validate that two listSets are disjoint                                                self names size      self details size      self names all n  n in self details     Validate the  details  map is keyed by the items in the  names  listSet                self details all key  key matches    a zA Z                                         Validate the keys of the  details  map                                                 self details all key  self details key  matches    a zA Z                           Validate the values of the  details  map                                                 CEL options  language features  and libraries  CEL is configured with the following options  libraries and language features  introduced at the specified Kubernetes versions     CEL option  library or language feature   Included   Availablity                                                                           Standard macros  https   github com google cel spec blob v0 7 0 doc langdef md macros     has    all    exists    exists one    map    filter    All Kubernetes versions      Standard functions  https   github com google cel spec blob master doc langdef md list of standard definitions    See  official list of standard definitions  https   github com google cel spec blob master doc langdef md list of standard definitions    All Kubernetes versions      Homogeneous Aggregate Literals  https   pkg go dev github com google cel go v0 17 4 cel HomogeneousAggregateLiterals      All Kubernetes versions      Default UTC Time Zone  https   pkg go dev github com google cel go v0 17 4 cel DefaultUTCTimeZone      All Kubernetes versions      Eagerly Validate Declarations  https   pkg go dev github com google cel go v0 17 4 cel EagerlyValidateDeclarations      All Kubernetes versions      Extended strings library  https   pkg go dev github com google cel go ext Strings   Version 1    charAt    indexOf    lastIndexOf    lowerAscii    upperAscii    replace    split    join    substring    trim    All Kubernetes versions     Kubernetes list library   See  Kubernetes list library   kubernetes list library    All Kubernetes versions     Kubernetes regex library   See  Kubernetes regex library   kubernetes regex library    All Kubernetes versions     Kubernetes URL library   See  Kubernetes URL library   kubernetes url library    All Kubernetes versions     Kubernetes authorizer library   See  Kubernetes authorizer library   kubernetes authorizer library    All Kubernetes versions     Kubernetes quantity library   See  Kubernetes quantity library   kubernetes quantity library    Kubernetes versions 1 29      CEL optional types   See  CEL optional types  https   pkg go dev github com google cel go v0 17 4 cel OptionalTypes    Kubernetes versions 1 29      CEL CrossTypeNumericComparisons   See  CEL CrossTypeNumericComparisons  https   pkg go dev github com google cel go v0 17 4 cel CrossTypeNumericComparisons    Kubernetes versions 1 29     CEL functions  features and language settings support Kubernetes control plane rollbacks  For example   CEL Optional Values  was introduced at Kubernetes 1 29 and so only API servers at that version or newer will accept write requests to CEL expressions that use  CEL Optional Values   However  when a cluster is rolled back to Kubernetes 1 28 CEL expressions using  CEL Optional Values  that are already stored in API resources will continue to evaluate correctly      Kubernetes CEL libraries  In additional to the CEL community libraries  Kubernetes includes CEL libraries that are available everywhere CEL is used in Kubernetes       Kubernetes list library  The list library includes  indexOf  and  lastIndexOf   which work similar to the strings functions of the same names  These functions either the first or last positional index of the provided element in the list   The list library also includes  min    max  and  sum   Sum is supported on all number types as well as the duration type  Min and max are supported on all comparable types    isSorted  is also provided as a convenience function and is supported on all comparable types   Examples      CEL Expression                                                                       Purpose                                                                                names isSorted                                                                      Verify that a list of names is kept in alphabetical order      items map x  x weight  sum      1 0                                                 Verify that the  weights  of a list of objects sum to 1 0      lowPriorities map x  x priority  max     highPriorities map x  x priority  min      Verify that two sets of priorities do not overlap              names indexOf  should be first      1                                               Require that the first name in a list if a specific value      See the  Kubernetes List Library  https   pkg go dev k8s io apiextensions apiserver pkg apiserver schema cel library Lists  godoc for more information       Kubernetes regex library  In addition to the  matches  function provided by the CEL standard library  the regex library provides  find  and  findAll   enabling a much wider range of regex operations   Examples      CEL Expression                                                Purpose                                                                                abc 123  find   0 9                                         Find the first number in a string                              1  2  3  4  findAll   0 9     map x  int x   sum     100    Verify that the numbers in a string sum to less than 100     See the  Kubernetes regex library  https   pkg go dev k8s io apiextensions apiserver pkg apiserver schema cel library Regex  godoc for more information       Kubernetes URL library  To make it easier and safer to process URLs  the following functions have been added      isURL string   checks if a string is a valid URL according to the    Go s net url  https   pkg go dev net url URL  package  The string must be an   absolute URL     url string  URL  converts a string to a URL or results in an error if the   string is not a valid URL   Once parsed via the  url  function  the resulting URL object has  getScheme    getHost    getHostname    getPort    getEscapedPath  and  getQuery  accessors   Examples      CEL Expression                                                    Purpose                                                                                url  https   example com 80    getHost                           Gets the  example com 80  host part of the URL      url  https   example com path with spaces    getEscapedPath      Returns   path 20with 20spaces                     See the  Kubernetes URL library  https   pkg go dev k8s io apiextensions apiserver pkg apiserver schema cel library URLs  godoc for more information       Kubernetes authorizer library  For CEL expressions in the API where a variable of type  Authorizer  is available  the authorizer may be used to perform authorization checks for the principal  authenticated user  of the request   API resource checks are performed as follows   1  Specify the group and resource to check   Authorizer group string  resource string  ResourceCheck  1  Optionally call any combination of the following builder functions to further narrow the authorization check     Note that these functions return the receiver type and can be chained        ResourceCheck subresource string  ResourceCheck        ResourceCheck namespace string  ResourceCheck        ResourceCheck name string  ResourceCheck  1  Call  ResourceCheck check verb string  Decision  to perform the authorization check  1  Call  allowed   bool  or  reason   string  to inspect the result of the authorization check   Non resource authorization performed are used as follows   1  Specify only a path   Authorizer path string  PathCheck  1  Call  PathCheck check httpVerb string  Decision  to perform the authorization check  1  Call  allowed   bool  or  reason   string  to inspect the result of the authorization check   To perform an authorization check for a service account      Authorizer serviceAccount namespace string  name string  Authorizer      CEL Expression   Purpose                                   authorizer group     resource  pods   namespace  default   check  create   allowed      Returns true if the principal  user or service account  is allowed create pods in the  default  namespace       authorizer path   healthz   check  get   allowed      Checks if the principal  user or service account  is authorized to make HTTP GET requests to the  healthz API path       authorizer serviceAccount  default    myserviceaccount   resource  deployments   check  delete   allowed      Checks if the service account is authorized to delete deployments        With the alpha  AuthorizeWithSelectors  feature enabled  field and label selectors can be added to authorization checks      CEL Expression   Purpose                                   authorizer group     resource  pods   fieldSelector  spec nodeName mynode   check  list   allowed      Returns true if the principal  user or service account  is allowed to list pods with the field selector  spec nodeName mynode        authorizer group     resource  pods   labelSelector  example com mylabel myvalue   check  list   allowed      Returns true if the principal  user or service account  is allowed to list pods with the label selector  example com mylabel myvalue       See the  Kubernetes Authz library  https   pkg go dev k8s io apiserver pkg cel library Authz  and  Kubernetes AuthzSelectors library  https   pkg go dev k8s io apiserver pkg cel library AuthzSelectors  godoc for more information       Kubernetes quantity library  Kubernetes 1 28 adds support for manipulating quantity strings  ex 1 5G  512k  20Mi      isQuantity string   checks if a string is a valid Quantity according to    Kubernetes  resource Quantity  https   pkg go dev k8s io apimachinery pkg api resource Quantity      quantity string  Quantity  converts a string to a Quantity or results in an error if the   string is not a valid quantity   Once parsed via the  quantity  function  the resulting Quantity object has the following library of member functions      Member Function               CEL Return Value   Description                                                                       isInteger                    bool               Returns true if and only if asInteger is safe to call without an error      asInteger                    int                Returns a representation of the current value as an int64 if possible or results in an error if conversion would result in overflow or loss of precision       asApproximateFloat           float              Returns a float64 representation of the quantity which may lose precision  If the value of the quantity is outside the range of a float64  Inf  Inf will be returned       sign                         int                Returns  1  if the quantity is positive    1  if it is negative   0  if it is zero      add  Quantity                Quantity           Returns sum of two quantities      add  int                     Quantity           Returns sum of quantity and an integer       sub  Quantity                Quantity           Returns difference between two quantities      sub  int                     Quantity           Returns difference between a quantity and an integer      isLessThan  Quantity         bool               Returns true if and only if the receiver is less than the operand      isGreaterThan  Quantity      bool               Returns true if and only if the receiver is greater than the operand      compareTo  Quantity          int                Compares receiver to operand and returns 0 if they are equal  1 if the receiver is greater  or  1 if the receiver is less than the operand     Examples      CEL Expression                                                              Purpose                                                                                quantity  500000G   isInteger                                              Test if conversion to integer would throw an error         quantity  50k   asInteger                                                  Precise conversion to integer                              quantity  9999999999999999999999999999999999999G   asApproximateFloat      Lossy conversion to float                                   quantity  50k   add quantity  20k                                         Add two quantities                                         quantity  50k   sub 20000                                                  Subtract an integer from a quantity                        quantity  50k   add 20  sub quantity  100k    sub  50000                   Chain adding and subtracting integers and quantities       quantity  200M   compareTo quantity  0 2G                                  Compare two quantities                                     quantity  150Mi   isGreaterThan quantity  100Mi                            Test if a quantity is greater than the receiver            quantity  50M   isLessThan quantity  100M                                  Test if a quantity is less than the receiver                 Type checking  CEL is a  gradually typed language  https   github com google cel spec blob master doc langdef md gradual type checking    Some Kubernetes API fields contain fully type checked CEL expressions  For example   CustomResourceDefinitions Validation Rules   docs tasks extend kubernetes custom resources custom resource definitions  validation rules  are fully type checked   Some Kubernetes API fields contain partially type checked CEL expressions  A partially type checked expression is an expressions where some of the variables are statically typed but others are dynamically typed  For example  in the CEL expressions of  ValidatingAdmissionPolicies   docs reference access authn authz validating admission policy   the  request  variable is typed  but the  object  variable is dynamically typed  As a result  an expression containing  request namex  would fail type checking because the  namex  field is not defined  However   object namex  would pass type checking even when the  namex  field is not defined for the resource kinds that  object  refers to  because  object  is dynamically typed   The  has    macro in CEL may be used in CEL expressions to check if a field of a dynamically typed variable is accessible before attempting to access the field s value  For example      cel has object namex    object namex     special    request name     special          Type system integration     OpenAPIv3 type                                       CEL type                                                                        object  with Properties                             object    message type    type  object    evaluates to  selfType uniqueNumber  path to object from self        object  with AdditionalProperties                   map      object  with x kubernetes embedded type             object    message type    apiVersion    kind    metadata name  and  metadata generateName  are implicitly included in schema      object  with x kubernetes preserve unknown fields   object    message type   unknown fields are NOT accessible in CEL expression     x kubernetes int or string                           union of int or string   self intOrString   100      self intOrString     50    evaluates to true for both  50  and   50         array                                               list      array  with x kubernetes list type map              list with map based Equality   unique key guarantees      array  with x kubernetes list type set              list with set based Equality   unique entry guarantees      boolean                                             boolean      number   all formats                                double      integer   all formats                               int  64       no equivalent                                       uint  64       null                                                null type      string                                              string      string  with format byte  base64 encoded            bytes      string  with format date                            timestamp  google protobuf Timestamp       string  with format datetime                        timestamp  google protobuf Timestamp       string  with format duration                        duration  google protobuf Duration      Also see   CEL types  https   github com google cel spec blob v0 6 0 doc langdef md values    OpenAPI types  https   swagger io specification  data types    Kubernetes Structural Schemas   docs tasks extend kubernetes custom resources custom resource definitions  specifying a structural schema    Equality comparison for arrays with  x kubernetes list type  of  set  or  map  ignores element order  For example   1  2      2  1   if the arrays represent Kubernetes  set  values   Concatenation on arrays with  x kubernetes list type  use the semantics of the list type    set     X   Y  performs a union where the array positions of all elements in    X  are preserved and non intersecting elements in  Y  are appended  retaining   their partial order    map     X   Y  performs a merge where the array positions of all keys in  X    are preserved but the values are overwritten by values in  Y  when the key   sets of  X  and  Y  intersect  Elements in  Y  with non intersecting keys are   appended  retaining their partial order      Escaping  Only Kubernetes resource property names of the form   a zA Z      a zA Z0 9        are accessible from CEL  Accessible property names are escaped according to the following rules when accessed in the expression      escape sequence     property name equivalent                                                                                underscores                                                                                dot                                                                                dash                                                                                slash                                                                                keyword          CEL   RESERVED   keyword  https   github com google cel spec blob v0 6 0 doc langdef md syntax      When you escape any of CEL s   RESERVED   keywords you need to match the exact property name use the underscore escaping  for example   int  in the word  sprint  would not be escaped and nor would it need to be    Examples on escaping      property name   rule with escaped property name                                                              namespace       self   namespace     0                x prop          self x  dash  prop   0                redact  d       self redact  underscores  d   0       string          self startsWith  kube                   Resource constraints  CEL is non Turing complete and offers a variety of production safety controls to limit execution time  CEL s  resource constraint  features provide feedback to developers about expression complexity and help protect the API server from excessive resource consumption during evaluation  CEL s resource constraint features are used to prevent CEL evaluation from consuming excessive API server resources   A key element of the resource constraint features is a  cost unit  that CEL defines as a way of tracking CPU utilization  Cost units are independent of system load and hardware  Cost units are also deterministic  for any given CEL expression and input data  evaluation of the expression by the CEL interpreter will always result in the same cost   Many of CEL s core operations have fixed costs  The simplest operations  such as comparisons  e g       have a cost of 1  Some have a higher fixed cost  for example list literal declarations have a fixed base cost of 40 cost units   Calls to functions implemented in native code approximate cost based on the time complexity of the operation  For example  operations that use regular expressions  such as  match  and  find   are estimated using an approximated cost of  length regexString  length inputString    The approximated cost reflects the worst case time complexity of Go s RE2 implementation       Runtime cost budget  All CEL expressions evaluated by Kubernetes are constrained by a runtime cost budget  The runtime cost budget is an estimate of actual CPU utilization computed by incrementing a cost unit counter while interpreting a CEL expression  If the CEL interpreter executes too many instructions  the runtime cost budget will be exceeded  execution of the expressions will be halted  and an error will result   Some Kubernetes resources define an additional runtime cost budget that bounds the execution of multiple expressions  If the sum total of the cost of expressions exceed the budget  execution of the expressions will be halted  and an error will result  For example the validation of a custom resource has a  per validation  runtime cost budget for all  Validation Rules   docs tasks extend kubernetes custom resources custom resource definitions  validation rules  evaluated to validate the custom resource       Estimated cost limits  For some Kubernetes resources  the API server may also check if worst case estimated running time of CEL expressions would be prohibitively expensive to execute  If so  the API server prevent the CEL expression from being written to API resources by rejecting create or update operations containing the CEL expression to the API resources  This feature offers a stronger assurance that CEL expressions written to the API resource will be evaluated at runtime without exceeding the runtime cost budget


kubernetes reference title Kubernetes API health endpoints contenttype concept logicalhan reviewers The Kubernetes glossarytooltip termid kube apiserver text API server provides API endpoints to indicate the current status of the API server overview weight 50

Answer:
---
title: Kubernetes API health endpoints
reviewers:
- logicalhan
content_type: concept
weight: 50
---

<!-- overview -->
The Kubernetes  provides API endpoints to indicate the current status of the API server.
This page describes these API endpoints and explains how you can use them.

<!-- body -->

## API endpoints for health

The Kubernetes API server provides 3 API endpoints (`healthz`, `livez` and `readyz`) to indicate the current status of the API server.
The `healthz` endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific `livez` and `readyz` endpoints instead.
The `livez` endpoint can be used with the `--livez-grace-period` [flag](/docs/reference/command-line-tools-reference/kube-apiserver) to specify the startup duration.
For a graceful shutdown you can specify the `--shutdown-delay-duration` [flag](/docs/reference/command-line-tools-reference/kube-apiserver) with the `/readyz` endpoint.
Machines that check the `healthz`/`livez`/`readyz` of the API server should rely on the HTTP status code.
A status code `200` indicates the API server is `healthy`/`live`/`ready`, depending on the called endpoint.
The more verbose options shown below are intended to be used by human operators to debug their cluster or understand the state of the API server.

The following examples will show how you can interact with the health API endpoints.

For all endpoints, you can use the `verbose` parameter to print out the checks and their status.
This can be useful for a human operator to debug the current status of the API server, it is not intended to be consumed by a machine:

```shell
curl -k https://localhost:6443/livez?verbose
```

or from a remote host with authentication:

```shell
kubectl get --raw='/readyz?verbose'
```

The output will look like this:

    [+]ping ok
    [+]log ok
    [+]etcd ok
    [+]poststarthook/start-kube-apiserver-admission-initializer ok
    [+]poststarthook/generic-apiserver-start-informers ok
    [+]poststarthook/start-apiextensions-informers ok
    [+]poststarthook/start-apiextensions-controllers ok
    [+]poststarthook/crd-informer-synced ok
    [+]poststarthook/bootstrap-controller ok
    [+]poststarthook/rbac/bootstrap-roles ok
    [+]poststarthook/scheduling/bootstrap-system-priority-classes ok
    [+]poststarthook/start-cluster-authentication-info-controller ok
    [+]poststarthook/start-kube-aggregator-informers ok
    [+]poststarthook/apiservice-registration-controller ok
    [+]poststarthook/apiservice-status-available-controller ok
    [+]poststarthook/kube-apiserver-autoregistration ok
    [+]autoregister-completion ok
    [+]poststarthook/apiservice-openapi-controller ok
    healthz check passed

The Kubernetes API server also supports to exclude specific checks.
The query parameters can also be combined like in this example:

```shell
curl -k 'https://localhost:6443/readyz?verbose&exclude=etcd'
```

The output show that the `etcd` check is excluded:

    [+]ping ok
    [+]log ok
    [+]etcd excluded: ok
    [+]poststarthook/start-kube-apiserver-admission-initializer ok
    [+]poststarthook/generic-apiserver-start-informers ok
    [+]poststarthook/start-apiextensions-informers ok
    [+]poststarthook/start-apiextensions-controllers ok
    [+]poststarthook/crd-informer-synced ok
    [+]poststarthook/bootstrap-controller ok
    [+]poststarthook/rbac/bootstrap-roles ok
    [+]poststarthook/scheduling/bootstrap-system-priority-classes ok
    [+]poststarthook/start-cluster-authentication-info-controller ok
    [+]poststarthook/start-kube-aggregator-informers ok
    [+]poststarthook/apiservice-registration-controller ok
    [+]poststarthook/apiservice-status-available-controller ok
    [+]poststarthook/kube-apiserver-autoregistration ok
    [+]autoregister-completion ok
    [+]poststarthook/apiservice-openapi-controller ok
    [+]shutdown ok
    healthz check passed

## Individual health checks

Each individual health check exposes an HTTP endpoint and can be checked individually.
The schema for the individual health checks is `/livez/<healthcheck-name>` or `/readyz/<healthcheck-name>`, where `livez` and `readyz` can be used to indicate if you want to check the liveness or the readiness of the API server, respectively.
The `<healthcheck-name>` path can be discovered using the `verbose` flag from above and take the path between `[+]` and `ok`.
These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:

```shell
curl -k https://localhost:6443/livez/etcd
```

title  Kubernetes API health endpoints reviewers    logicalhan content type  concept weight  50           overview     The Kubernetes  provides API endpoints to indicate the current status of the API server  This page describes these API endpoints and explains how you can use them        body         API endpoints for health  The Kubernetes API server provides 3 API endpoints   healthz    livez  and  readyz   to indicate the current status of the API server  The  healthz  endpoint is deprecated  since Kubernetes v1 16   and you should use the more specific  livez  and  readyz  endpoints instead  The  livez  endpoint can be used with the    livez grace period   flag   docs reference command line tools reference kube apiserver  to specify the startup duration  For a graceful shutdown you can specify the    shutdown delay duration   flag   docs reference command line tools reference kube apiserver  with the   readyz  endpoint  Machines that check the  healthz   livez   readyz  of the API server should rely on the HTTP status code  A status code  200  indicates the API server is  healthy   live   ready   depending on the called endpoint  The more verbose options shown below are intended to be used by human operators to debug their cluster or understand the state of the API server   The following examples will show how you can interact with the health API endpoints   For all endpoints  you can use the  verbose  parameter to print out the checks and their status  This can be useful for a human operator to debug the current status of the API server  it is not intended to be consumed by a machine      shell curl  k https   localhost 6443 livez verbose      or from a remote host with authentication      shell kubectl get   raw   readyz verbose       The output will look like this          ping ok        log ok        etcd ok        poststarthook start kube apiserver admission initializer ok        poststarthook generic apiserver start informers ok        poststarthook start apiextensions informers ok        poststarthook start apiextensions controllers ok        poststarthook crd informer synced ok        poststarthook bootstrap controller ok        poststarthook rbac bootstrap roles ok        poststarthook scheduling bootstrap system priority classes ok        poststarthook start cluster authentication info controller ok        poststarthook start kube aggregator informers ok        poststarthook apiservice registration controller ok        poststarthook apiservice status available controller ok        poststarthook kube apiserver autoregistration ok        autoregister completion ok        poststarthook apiservice openapi controller ok     healthz check passed  The Kubernetes API server also supports to exclude specific checks  The query parameters can also be combined like in this example      shell curl  k  https   localhost 6443 readyz verbose exclude etcd       The output show that the  etcd  check is excluded          ping ok        log ok        etcd excluded  ok        poststarthook start kube apiserver admission initializer ok        poststarthook generic apiserver start informers ok        poststarthook start apiextensions informers ok        poststarthook start apiextensions controllers ok        poststarthook crd informer synced ok        poststarthook bootstrap controller ok        poststarthook rbac bootstrap roles ok        poststarthook scheduling bootstrap system priority classes ok        poststarthook start cluster authentication info controller ok        poststarthook start kube aggregator informers ok        poststarthook apiservice registration controller ok        poststarthook apiservice status available controller ok        poststarthook kube apiserver autoregistration ok        autoregister completion ok        poststarthook apiservice openapi controller ok        shutdown ok     healthz check passed     Individual health checks    Each individual health check exposes an HTTP endpoint and can be checked individually  The schema for the individual health checks is   livez  healthcheck name   or   readyz  healthcheck name    where  livez  and  readyz  can be used to indicate if you want to check the liveness or the readiness of the API server  respectively  The   healthcheck name   path can be discovered using the  verbose  flag from above and take the path between       and  ok   These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system      shell curl  k https   localhost 6443 livez etcd


kubernetes reference weight 20 title API Overview erictune contenttype concept card reviewers jbeda nolist true lavalamp

Answer:
---
title: API Overview
reviewers:
- erictune
- lavalamp
- jbeda
content_type: concept
weight: 20
no_list: true
card:
  name: reference
  weight: 50
  title: Overview of API
---

<!-- overview -->

This section provides reference information for the Kubernetes API.

The REST API is the fundamental fabric of Kubernetes. All operations and
communications between components, and external user commands are REST API
calls that the API Server handles. Consequently, everything in the Kubernetes
platform is treated as an API object and has a corresponding entry in the
[API](/docs/reference/generated/kubernetes-api//).

The [Kubernetes API reference](/docs/reference/generated/kubernetes-api//)
lists the API for Kubernetes version .

For general background information, read
[The Kubernetes API](/docs/concepts/overview/kubernetes-api/).
[Controlling Access to the Kubernetes API](/docs/concepts/security/controlling-access/)
describes how clients can authenticate to the Kubernetes API server, and how their
requests are authorized.

## API versioning

The JSON and Protobuf serialization schemas follow the same guidelines for
schema changes. The following descriptions cover both formats.

The API versioning and software versioning are indirectly related.
The [API and release versioning proposal](https://git.k8s.io/sig-release/release-engineering/versioning.md)
describes the relationship between API versioning and software versioning.

Different API versions indicate different levels of stability and support. You
can find more information about the criteria for each level in the
[API Changes documentation](https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions).

Here's a summary of each level:

- Alpha:
  - The version names contain `alpha` (for example, `v1alpha1`).
  - Built-in alpha API versions are disabled by default and must be explicitly enabled in the `kube-apiserver` configuration to be used.
  - The software may contain bugs. Enabling a feature may expose bugs.
  - Support for an alpha API may be dropped at any time without notice.
  - The API may change in incompatible ways in a later software release without notice.
  - The software is recommended for use only in short-lived testing clusters,
    due to increased risk of bugs and lack of long-term support.

- Beta:
  - The version names contain `beta` (for example, `v2beta3`).
  - Built-in beta API versions are disabled by default and must be explicitly enabled in the `kube-apiserver` configuration to be used
    (**except** for beta versions of APIs introduced prior to Kubernetes 1.22, which were enabled by default).
  - Built-in beta API versions have a maximum lifetime of 9 months or 3 minor releases (whichever is longer) from introduction
    to deprecation, and 9 months or 3 minor releases (whichever is longer) from deprecation to removal.
  - The software is well tested. Enabling a feature is considered safe.
  - The support for a feature will not be dropped, though the details may change.

  - The schema and/or semantics of objects may change in incompatible ways in
    a subsequent beta or stable API version. When this happens, migration
    instructions are provided. Adapting to a subsequent beta or stable API version
    may require editing or re-creating API objects, and may not be straightforward.
    The migration may require downtime for applications that rely on the feature.
  - The software is not recommended for production uses. Subsequent releases
    may introduce incompatible changes. Use of beta API versions is
    required to transition to subsequent beta or stable API versions
    once the beta API version is deprecated and no longer served.

  Please try beta features and provide feedback. After the features exit beta, it
  may not be practical to make more changes.

- Stable:
  - The version name is `vX` where `X` is an integer.
  - Stable API versions remain available for all future releases within a Kubernetes major version,
    and there are no current plans for a major version revision of Kubernetes that removes stable APIs.

## API groups

[API groups](https://git.k8s.io/design-proposals-archive/api-machinery/api-group.md)
make it easier to extend the Kubernetes API.
The API group is specified in a REST path and in the `apiVersion` field of a
serialized object.

There are several API groups in Kubernetes:

*  The *core* (also called *legacy*) group is found at REST path `/api/v1`.
   The core group is not specified as part of the `apiVersion` field, for
   example, `apiVersion: v1`.
*  The named groups are at REST path `/apis/$GROUP_NAME/$VERSION` and use
   `apiVersion: $GROUP_NAME/$VERSION` (for example, `apiVersion: batch/v1`).
   You can find the full list of supported API groups in
   [Kubernetes API reference](/docs/reference/generated/kubernetes-api//#-strong-api-groups-strong-).

## Enabling or disabling API groups   {#enabling-or-disabling}

Certain resources and API groups are enabled by default. You can enable or
disable them by setting `--runtime-config` on the API server.  The
`--runtime-config` flag accepts comma separated `<key>[=<value>]` pairs
describing the runtime configuration of the API server. If the `=<value>`
part is omitted, it is treated as if `=true` is specified. For example:

 - to disable `batch/v1`, set `--runtime-config=batch/v1=false`
 - to enable `batch/v2alpha1`, set `--runtime-config=batch/v2alpha1`
 - to enable a specific version of an API, such as `storage.k8s.io/v1beta1/csistoragecapacities`, set `--runtime-config=storage.k8s.io/v1beta1/csistoragecapacities`

When you enable or disable groups or resources, you need to restart the API
server and controller manager to pick up the `--runtime-config` changes.

## Persistence

Kubernetes stores its serialized state in terms of the API resources by writing them into
.

##

- Learn more about [API conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#api-conventions)
- Read the design documentation for
  [aggregator](https://git.k8s.io/design-proposals-archive/api-machinery/aggregated-api-servers.md)

title  API Overview reviewers    erictune   lavalamp   jbeda content type  concept weight  20 no list  true card    name  reference   weight  50   title  Overview of API           overview      This section provides reference information for the Kubernetes API   The REST API is the fundamental fabric of Kubernetes  All operations and communications between components  and external user commands are REST API calls that the API Server handles  Consequently  everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the  API   docs reference generated kubernetes api      The  Kubernetes API reference   docs reference generated kubernetes api    lists the API for Kubernetes version    For general background information  read  The Kubernetes API   docs concepts overview kubernetes api     Controlling Access to the Kubernetes API   docs concepts security controlling access   describes how clients can authenticate to the Kubernetes API server  and how their requests are authorized       API versioning  The JSON and Protobuf serialization schemas follow the same guidelines for schema changes  The following descriptions cover both formats   The API versioning and software versioning are indirectly related  The  API and release versioning proposal  https   git k8s io sig release release engineering versioning md  describes the relationship between API versioning and software versioning   Different API versions indicate different levels of stability and support  You can find more information about the criteria for each level in the  API Changes documentation  https   git k8s io community contributors devel sig architecture api changes md alpha beta and stable versions    Here s a summary of each level     Alpha      The version names contain  alpha   for example   v1alpha1        Built in alpha API versions are disabled by default and must be explicitly enabled in the  kube apiserver  configuration to be used      The software may contain bugs  Enabling a feature may expose bugs      Support for an alpha API may be dropped at any time without notice      The API may change in incompatible ways in a later software release without notice      The software is recommended for use only in short lived testing clusters      due to increased risk of bugs and lack of long term support     Beta      The version names contain  beta   for example   v2beta3        Built in beta API versions are disabled by default and must be explicitly enabled in the  kube apiserver  configuration to be used        except   for beta versions of APIs introduced prior to Kubernetes 1 22  which were enabled by default       Built in beta API versions have a maximum lifetime of 9 months or 3 minor releases  whichever is longer  from introduction     to deprecation  and 9 months or 3 minor releases  whichever is longer  from deprecation to removal      The software is well tested  Enabling a feature is considered safe      The support for a feature will not be dropped  though the details may change       The schema and or semantics of objects may change in incompatible ways in     a subsequent beta or stable API version  When this happens  migration     instructions are provided  Adapting to a subsequent beta or stable API version     may require editing or re creating API objects  and may not be straightforward      The migration may require downtime for applications that rely on the feature      The software is not recommended for production uses  Subsequent releases     may introduce incompatible changes  Use of beta API versions is     required to transition to subsequent beta or stable API versions     once the beta API version is deprecated and no longer served        Please try beta features and provide feedback  After the features exit beta  it   may not be practical to make more changes        Stable      The version name is  vX  where  X  is an integer      Stable API versions remain available for all future releases within a Kubernetes major version      and there are no current plans for a major version revision of Kubernetes that removes stable APIs      API groups   API groups  https   git k8s io design proposals archive api machinery api group md  make it easier to extend the Kubernetes API  The API group is specified in a REST path and in the  apiVersion  field of a serialized object   There are several API groups in Kubernetes      The  core   also called  legacy   group is found at REST path   api v1      The core group is not specified as part of the  apiVersion  field  for    example   apiVersion  v1      The named groups are at REST path   apis  GROUP NAME  VERSION  and use     apiVersion   GROUP NAME  VERSION   for example   apiVersion  batch v1       You can find the full list of supported API groups in     Kubernetes API reference   docs reference generated kubernetes api    strong api groups strong        Enabling or disabling API groups     enabling or disabling   Certain resources and API groups are enabled by default  You can enable or disable them by setting    runtime config  on the API server   The    runtime config  flag accepts comma separated   key    value    pairs describing the runtime configuration of the API server  If the    value   part is omitted  it is treated as if   true  is specified  For example      to disable  batch v1   set    runtime config batch v1 false     to enable  batch v2alpha1   set    runtime config batch v2alpha1     to enable a specific version of an API  such as  storage k8s io v1beta1 csistoragecapacities   set    runtime config storage k8s io v1beta1 csistoragecapacities    When you enable or disable groups or resources  you need to restart the API server and controller manager to pick up the    runtime config  changes       Persistence  Kubernetes stores its serialized state in terms of the API resources by writing them into           Learn more about  API conventions  https   git k8s io community contributors devel sig architecture api conventions md api conventions    Read the design documentation for    aggregator  https   git k8s io design proposals archive api machinery aggregated api servers md


kubernetes setup weight 20 vincepri title Container Runtimes contenttype concept reviewers bart0sh overview

Answer:
---
reviewers:
- vincepri
- bart0sh
title: Container Runtimes
content_type: concept
weight: 20
---
<!-- overview -->

You need to install a

into each node in the cluster so that Pods can run there. This page outlines
what is involved and describes related tasks for setting up nodes.

Kubernetes  requires that you use a runtime that
conforms with the
 (CRI).

See [CRI version support](#cri-versions) for more information.

This page provides an outline of how to use several common container runtimes with
Kubernetes.

- [containerd](#containerd)
- [CRI-O](#cri-o)
- [Docker Engine](#docker)
- [Mirantis Container Runtime](#mcr)

Kubernetes releases before v1.24 included a direct integration with Docker Engine,
using a component named _dockershim_. That special direct integration is no longer
part of Kubernetes (this removal was
[announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation)
as part of the v1.20 release).
You can read
[Check whether Dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/)
to understand how this removal might affect you. To learn about migrating from using dockershim, see
[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/).

If you are running a version of Kubernetes other than v,
check the documentation for that version.

<!-- body -->
## Install and configure prerequisites

### Network configuration

By default, the Linux kernel does not allow IPv4 packets to be routed
between interfaces. Most Kubernetes cluster networking implementations
will change this setting (if needed), but some might expect the
administrator to do it for them. (Some might also expect other sysctl
parameters to be set, kernel modules to be loaded, etc; consult the
documentation for your specific network implementation.)

### Enable IPv4 packet forwarding {#prerequisite-ipv4-forwarding-optional}

To manually enable IPv4 packet forwarding:

```bash
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system
```

Verify that `net.ipv4.ip_forward` is set to 1 with:

```bash
sysctl net.ipv4.ip_forward
```

## cgroup drivers

On Linux,
are used to constrain resources that are allocated to processes.

Both the  and the
underlying container runtime need to interface with control groups to enforce
[resource management for pods and containers](/docs/concepts/configuration/manage-resources-containers/)
and set resources such as cpu/memory requests and limits. To interface with control
groups, the kubelet and the container runtime need to use a *cgroup driver*.
It's critical that the kubelet and the container runtime use the same cgroup
driver and are configured the same.

There are two cgroup drivers available:

* [`cgroupfs`](#cgroupfs-cgroup-driver)
* [`systemd`](#systemd-cgroup-driver)

### cgroupfs driver {#cgroupfs-cgroup-driver}

The `cgroupfs` driver is the [default cgroup driver in the kubelet](/docs/reference/config-api/kubelet-config.v1beta1).
 When the `cgroupfs` driver is used, the kubelet and the container runtime directly interface with
 the cgroup filesystem to configure cgroups.

The `cgroupfs` driver is **not** recommended when
[systemd](https://www.freedesktop.org/wiki/Software/systemd/) is the
init system because systemd expects a single cgroup manager on
the system. Additionally, if you use [cgroup v2](/docs/concepts/architecture/cgroups), use the `systemd`
cgroup driver instead of `cgroupfs`.

### systemd cgroup driver {#systemd-cgroup-driver}

When [systemd](https://www.freedesktop.org/wiki/Software/systemd/) is chosen as the init
system for a Linux distribution, the init process generates and consumes a root control group
(`cgroup`) and acts as a cgroup manager.

systemd has a tight integration with cgroups and allocates a cgroup per systemd
unit. As a result, if you use `systemd` as the init system with the `cgroupfs`
driver, the system gets two different cgroup managers.

Two cgroup managers result in two views of the available and in-use resources in
the system. In some cases, nodes that are configured to use `cgroupfs` for the
kubelet and container runtime, but use `systemd` for the rest of the processes become
unstable under resource pressure.

The approach to mitigate this instability is to use `systemd` as the cgroup driver for
the kubelet and the container runtime when systemd is the selected init system.

To set `systemd` as the cgroup driver, edit the
[`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/)
option of `cgroupDriver` and set it to `systemd`. For example:

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
cgroupDriver: systemd
```

Starting with v1.22 and later, when creating a cluster with kubeadm, if the user does not set
the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`.

If you configure `systemd` as the cgroup driver for the kubelet, you must also
configure `systemd` as the cgroup driver for the container runtime. Refer to
the documentation for your container runtime for instructions. For example:

*  [containerd](#containerd-systemd)
*  [CRI-O](#cri-o)

In Kubernetes , with the `KubeletCgroupDriverFromCRI`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled and a container runtime that supports the `RuntimeConfig` CRI RPC,
the kubelet automatically detects the appropriate cgroup driver from the runtime,
and ignores the `cgroupDriver` setting within the kubelet configuration.

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation.
If the kubelet has created Pods using the semantics of one cgroup driver, changing the container
runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox
for such existing Pods. Restarting the kubelet may not solve such errors.

If you have automation that makes it feasible, replace the node with another using the updated
configuration, or reinstall it using automation.

### Migrating to the `systemd` driver in kubeadm managed clusters

If you wish to migrate to the `systemd` cgroup driver in existing kubeadm managed clusters,
follow [configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).

## CRI version support {#cri-versions}

Your container runtime must support at least v1alpha2 of the container runtime interface.

Kubernetes [starting v1.26](/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#cri-api-removal)
_only works_ with v1 of the CRI API. Earlier versions default
to v1 version, however if a container runtime does not support the v1 API, the kubelet falls back to
using the (deprecated) v1alpha2 API instead.

## Container runtimes

### containerd

This section outlines the necessary steps to use containerd as CRI runtime.

To install containerd on your system, follow the instructions on
[getting started with containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md).
Return to this step once you've created a valid `config.toml` configuration file.

You can find this file under the path `/etc/containerd/config.toml`.

You can find this file under the path `C:\Program Files\containerd\config.toml`.

On Linux the default CRI socket for containerd is `/run/containerd/containerd.sock`.
On Windows the default CRI endpoint is `npipe://./pipe/containerd-containerd`.

#### Configuring the `systemd` cgroup driver {#containerd-systemd}

To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc`, set

```
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
```

The `systemd` cgroup driver is recommended if you use [cgroup v2](/docs/concepts/architecture/cgroups).

If you installed containerd from a package (for example, RPM or `.deb`), you may find
that the CRI integration plugin is disabled by default.

You need CRI support enabled to use containerd with Kubernetes. Make sure that `cri`
is not included in the`disabled_plugins` list within `/etc/containerd/config.toml`;
if you made changes to that file, also restart `containerd`.

If you experience container crash loops after the initial cluster installation or after
installing a CNI, the containerd configuration provided with the package might contain
incompatible configuration parameters. Consider resetting the containerd configuration
with `containerd config default > /etc/containerd/config.toml` as specified in
[getting-started.md](https://github.com/containerd/containerd/blob/main/docs/getting-started.md#advanced-topics)
and then set the configuration parameters specified above accordingly.

If you apply this change, make sure to restart containerd:

```shell
sudo systemctl restart containerd
```

When using kubeadm, manually configure the
[cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver).

In Kubernetes v1.28, you can enable automatic detection of the
cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

#### Overriding the sandbox (pause) image {#override-pause-image-containerd}

In your [containerd config](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) you can overwrite the
sandbox image by setting the following config:

```toml
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.2"
```

You might need to restart `containerd` as well once you've updated the config file: `systemctl restart containerd`.

### CRI-O

This section contains the necessary steps to install CRI-O as a container runtime.

To install CRI-O, follow [CRI-O Install Instructions](https://github.com/cri-o/packaging/blob/main/README.md#usage).

#### cgroup driver

CRI-O uses the systemd cgroup driver per default, which is likely to work fine
for you. To switch to the `cgroupfs` cgroup driver, either edit
`/etc/crio/crio.conf` or place a drop-in configuration in
`/etc/crio/crio.conf.d/02-cgroup-manager.conf`, for example:

```toml
[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"
```

You should also note the changed `conmon_cgroup`, which has to be set to the value
`pod` when using CRI-O with `cgroupfs`. It is generally necessary to keep the
cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O
in sync.

In Kubernetes v1.28, you can enable automatic detection of the
cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default.

#### Overriding the sandbox (pause) image {#override-pause-image-cri-o}

In your [CRI-O config](https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md) you can set the following
config value:

```toml
[crio.image]
pause_image="registry.k8s.io/pause:3.6"
```

This config option supports live configuration reload to apply this change: `systemctl reload crio` or by sending
`SIGHUP` to the `crio` process.

### Docker Engine {#docker}

These instructions assume that you are using the
[`cri-dockerd`](https://mirantis.github.io/cri-dockerd/) adapter to integrate
Docker Engine with Kubernetes.

1. On each of your nodes, install Docker for your Linux distribution as per
  [Install Docker Engine](https://docs.docker.com/engine/install/#server).

2. Install [`cri-dockerd`](https://mirantis.github.io/cri-dockerd/usage/install), following the directions in the install section of the documentation.

For `cri-dockerd`, the CRI socket is `/run/cri-dockerd.sock` by default.

### Mirantis Container Runtime {#mcr}

[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is a commercially
available container runtime that was formerly known as Docker Enterprise Edition.

You can use Mirantis Container Runtime with Kubernetes using the open source
[`cri-dockerd`](https://mirantis.github.io/cri-dockerd/) component, included with MCR.

To learn more about how to install Mirantis Container Runtime,
visit [MCR Deployment Guide](https://docs.mirantis.com/mcr/20.10/install.html).

Check the systemd unit named `cri-docker.socket` to find out the path to the CRI
socket.

#### Overriding the sandbox (pause) image {#override-pause-image-cri-dockerd-mcr}

The `cri-dockerd` adapter accepts a command line argument for
specifying which container image to use as the Pod infrastructure container (“pause image”).
The command line argument to use is `--pod-infra-container-image`.

##

As well as a container runtime, your cluster will need a working
[network plugin](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-network-model).

reviewers    vincepri   bart0sh title  Container Runtimes content type  concept weight  20          overview        You need to install a  into each node in the cluster so that Pods can run there  This page outlines what is involved and describes related tasks for setting up nodes   Kubernetes  requires that you use a runtime that conforms with the   CRI    See  CRI version support   cri versions  for more information   This page provides an outline of how to use several common container runtimes with Kubernetes      containerd   containerd     CRI O   cri o     Docker Engine   docker     Mirantis Container Runtime   mcr    Kubernetes releases before v1 24 included a direct integration with Docker Engine  using a component named  dockershim   That special direct integration is no longer part of Kubernetes  this removal was  announced   blog 2020 12 08 kubernetes 1 20 release announcement  dockershim deprecation  as part of the v1 20 release   You can read  Check whether Dockershim removal affects you   docs tasks administer cluster migrating from dockershim check if dockershim removal affects you   to understand how this removal might affect you  To learn about migrating from using dockershim  see  Migrating from dockershim   docs tasks administer cluster migrating from dockershim     If you are running a version of Kubernetes other than v  check the documentation for that version         body        Install and configure prerequisites      Network configuration  By default  the Linux kernel does not allow IPv4 packets to be routed between interfaces  Most Kubernetes cluster networking implementations will change this setting  if needed   but some might expect the administrator to do it for them   Some might also expect other sysctl parameters to be set  kernel modules to be loaded  etc  consult the documentation for your specific network implementation        Enable IPv4 packet forwarding   prerequisite ipv4 forwarding optional   To manually enable IPv4 packet forwarding      bash   sysctl params required by setup  params persist across reboots cat   EOF   sudo tee  etc sysctl d k8s conf net ipv4 ip forward   1 EOF    Apply sysctl params without reboot sudo sysctl   system      Verify that  net ipv4 ip forward  is set to 1 with      bash sysctl net ipv4 ip forward         cgroup drivers  On Linux   are used to constrain resources that are allocated to processes   Both the  and the underlying container runtime need to interface with control groups to enforce  resource management for pods and containers   docs concepts configuration manage resources containers   and set resources such as cpu memory requests and limits  To interface with control groups  the kubelet and the container runtime need to use a  cgroup driver   It s critical that the kubelet and the container runtime use the same cgroup driver and are configured the same   There are two cgroup drivers available       cgroupfs    cgroupfs cgroup driver      systemd    systemd cgroup driver       cgroupfs driver   cgroupfs cgroup driver   The  cgroupfs  driver is the  default cgroup driver in the kubelet   docs reference config api kubelet config v1beta1    When the  cgroupfs  driver is used  the kubelet and the container runtime directly interface with  the cgroup filesystem to configure cgroups   The  cgroupfs  driver is   not   recommended when  systemd  https   www freedesktop org wiki Software systemd   is the init system because systemd expects a single cgroup manager on the system  Additionally  if you use  cgroup v2   docs concepts architecture cgroups   use the  systemd  cgroup driver instead of  cgroupfs        systemd cgroup driver   systemd cgroup driver   When  systemd  https   www freedesktop org wiki Software systemd   is chosen as the init system for a Linux distribution  the init process generates and consumes a root control group   cgroup   and acts as a cgroup manager   systemd has a tight integration with cgroups and allocates a cgroup per systemd unit  As a result  if you use  systemd  as the init system with the  cgroupfs  driver  the system gets two different cgroup managers   Two cgroup managers result in two views of the available and in use resources in the system  In some cases  nodes that are configured to use  cgroupfs  for the kubelet and container runtime  but use  systemd  for the rest of the processes become unstable under resource pressure   The approach to mitigate this instability is to use  systemd  as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system   To set  systemd  as the cgroup driver  edit the   KubeletConfiguration    docs tasks administer cluster kubelet config file   option of  cgroupDriver  and set it to  systemd   For example      yaml apiVersion  kubelet config k8s io v1beta1 kind  KubeletConfiguration     cgroupDriver  systemd       Starting with v1 22 and later  when creating a cluster with kubeadm  if the user does not set the  cgroupDriver  field under  KubeletConfiguration   kubeadm defaults it to  systemd     If you configure  systemd  as the cgroup driver for the kubelet  you must also configure  systemd  as the cgroup driver for the container runtime  Refer to the documentation for your container runtime for instructions  For example       containerd   containerd systemd      CRI O   cri o   In Kubernetes   with the  KubeletCgroupDriverFromCRI   feature gate   docs reference command line tools reference feature gates   enabled and a container runtime that supports the  RuntimeConfig  CRI RPC  the kubelet automatically detects the appropriate cgroup driver from the runtime  and ignores the  cgroupDriver  setting within the kubelet configuration    Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation  If the kubelet has created Pods using the semantics of one cgroup driver  changing the container runtime to another cgroup driver can cause errors when trying to re create the Pod sandbox for such existing Pods  Restarting the kubelet may not solve such errors   If you have automation that makes it feasible  replace the node with another using the updated configuration  or reinstall it using automation         Migrating to the  systemd  driver in kubeadm managed clusters  If you wish to migrate to the  systemd  cgroup driver in existing kubeadm managed clusters  follow  configuring a cgroup driver   docs tasks administer cluster kubeadm configure cgroup driver        CRI version support   cri versions   Your container runtime must support at least v1alpha2 of the container runtime interface   Kubernetes  starting v1 26   blog 2022 11 18 upcoming changes in kubernetes 1 26  cri api removal   only works  with v1 of the CRI API  Earlier versions default to v1 version  however if a container runtime does not support the v1 API  the kubelet falls back to using the  deprecated  v1alpha2 API instead      Container runtimes        containerd  This section outlines the necessary steps to use containerd as CRI runtime   To install containerd on your system  follow the instructions on  getting started with containerd  https   github com containerd containerd blob main docs getting started md   Return to this step once you ve created a valid  config toml  configuration file     You can find this file under the path   etc containerd config toml     You can find this file under the path  C  Program Files containerd config toml      On Linux the default CRI socket for containerd is   run containerd containerd sock   On Windows the default CRI endpoint is  npipe     pipe containerd containerd         Configuring the  systemd  cgroup driver   containerd systemd   To use the  systemd  cgroup driver in   etc containerd config toml  with  runc   set       plugins  io containerd grpc v1 cri  containerd runtimes runc           plugins  io containerd grpc v1 cri  containerd runtimes runc options      SystemdCgroup   true      The  systemd  cgroup driver is recommended if you use  cgroup v2   docs concepts architecture cgroups     If you installed containerd from a package  for example  RPM or   deb    you may find that the CRI integration plugin is disabled by default   You need CRI support enabled to use containerd with Kubernetes  Make sure that  cri  is not included in the disabled plugins  list within   etc containerd config toml   if you made changes to that file  also restart  containerd    If you experience container crash loops after the initial cluster installation or after installing a CNI  the containerd configuration provided with the package might contain incompatible configuration parameters  Consider resetting the containerd configuration with  containerd config default    etc containerd config toml  as specified in  getting started md  https   github com containerd containerd blob main docs getting started md advanced topics  and then set the configuration parameters specified above accordingly    If you apply this change  make sure to restart containerd      shell sudo systemctl restart containerd      When using kubeadm  manually configure the  cgroup driver for kubelet   docs tasks administer cluster kubeadm configure cgroup driver  configuring the kubelet cgroup driver    In Kubernetes v1 28  you can enable automatic detection of the cgroup driver as an alpha feature  See  systemd cgroup driver   systemd cgroup driver  for more details        Overriding the sandbox  pause  image   override pause image containerd   In your  containerd config  https   github com containerd containerd blob main docs cri config md  you can overwrite the sandbox image by setting the following config      toml  plugins  io containerd grpc v1 cri     sandbox image    registry k8s io pause 3 2       You might need to restart  containerd  as well once you ve updated the config file   systemctl restart containerd        CRI O  This section contains the necessary steps to install CRI O as a container runtime   To install CRI O  follow  CRI O Install Instructions  https   github com cri o packaging blob main README md usage         cgroup driver  CRI O uses the systemd cgroup driver per default  which is likely to work fine for you  To switch to the  cgroupfs  cgroup driver  either edit   etc crio crio conf  or place a drop in configuration in   etc crio crio conf d 02 cgroup manager conf   for example      toml  crio runtime  conmon cgroup    pod  cgroup manager    cgroupfs       You should also note the changed  conmon cgroup   which has to be set to the value  pod  when using CRI O with  cgroupfs   It is generally necessary to keep the cgroup driver configuration of the kubelet  usually done via kubeadm  and CRI O in sync   In Kubernetes v1 28  you can enable automatic detection of the cgroup driver as an alpha feature  See  systemd cgroup driver   systemd cgroup driver  for more details   For CRI O  the CRI socket is   var run crio crio sock  by default        Overriding the sandbox  pause  image   override pause image cri o   In your  CRI O config  https   github com cri o cri o blob main docs crio conf 5 md  you can set the following config value      toml  crio image  pause image  registry k8s io pause 3 6       This config option supports live configuration reload to apply this change   systemctl reload crio  or by sending  SIGHUP  to the  crio  process       Docker Engine   docker    These instructions assume that you are using the   cri dockerd   https   mirantis github io cri dockerd   adapter to integrate Docker Engine with Kubernetes    1  On each of your nodes  install Docker for your Linux distribution as per    Install Docker Engine  https   docs docker com engine install  server    2  Install   cri dockerd   https   mirantis github io cri dockerd usage install   following the directions in the install section of the documentation   For  cri dockerd   the CRI socket is   run cri dockerd sock  by default       Mirantis Container Runtime   mcr    Mirantis Container Runtime  https   docs mirantis com mcr 20 10 overview html   MCR  is a commercially available container runtime that was formerly known as Docker Enterprise Edition   You can use Mirantis Container Runtime with Kubernetes using the open source   cri dockerd   https   mirantis github io cri dockerd   component  included with MCR   To learn more about how to install Mirantis Container Runtime  visit  MCR Deployment Guide  https   docs mirantis com mcr 20 10 install html    Check the systemd unit named  cri docker socket  to find out the path to the CRI socket        Overriding the sandbox  pause  image   override pause image cri dockerd mcr   The  cri dockerd  adapter accepts a command line argument for specifying which container image to use as the Pod infrastructure container   pause image    The command line argument to use is    pod infra container image         As well as a container runtime  your cluster will need a working  network plugin   docs concepts cluster administration networking  how to implement the kubernetes network model