tekton explains This section gives an overview of how the affinity assistant is resilient to a cluster maintenance without losing how an affinity assistant is created when a is used as a volume source for a in a Please refer to the same section for more details on the affinity assistant the running Please refer to the issue https github com tektoncd pipeline issues 6586 for more details Affinity Assistant

Answer:
# Affinity Assistant

[Specifying `Workspaces` in a `Pipeline`](../workspaces.md#specifying-workspace-order-in-a-pipeline-and-affinity-assistants) explains
how an affinity assistant is created when a `persistentVolumeClaim` is used as a volume source for a `workspace` in a `pipelineRun`.
Please refer to the same section for more details on the affinity assistant.

This section gives an overview of how the affinity assistant is resilient to a cluster maintenance without losing
the running `pipelineRun`. (Please refer to the issue https://github.com/tektoncd/pipeline/issues/6586 for more details.)

When a list of `tasks` share a single workspace, the affinity assistant pod gets created on a `node` along with all
`taskRun` pods. It is very common for a `pipeline` author to design a long-running tasks with a single workspace.
With these long-running tasks, a `node` on which these pods are scheduled can be cordoned while the `pipelineRun` is
still running. The tekton controller migrates the affinity assistant pod to any available `node` in a cluster along with
the rest of the `taskRun` pods sharing the same workspace.

Let's understand this with a sample `pipelineRun`:

```yaml
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: pipeline-run-
spec:
  workspaces:
  - name: source
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Mi
  pipelineSpec:
    workspaces:
    - name: source
    tasks:
    - name: first-task
      taskSpec:
        workspaces:
        - name: source
        steps:
        - image: alpine
          script: |
            echo $(workspaces.source.path)
            sleep 60
      workspaces:
      - name: source
    - name: last-task
      taskSpec:
        workspaces:
        - name: source
        steps:
        - image: alpine
          script: |
            echo $(workspaces.source.path)
            sleep 60
      runAfter: ["first-task"]
      workspaces:
      - name: source
```

This `pipelineRun` has two long-running tasks, `first-task` and `last-task`. Both of these tasks are sharing a single
volume with the access mode set to `ReadWriteOnce` which means the volume can be mounted to a single `node` at any
given point of time.

Create a `pipelineRun` and determine on which `node` the affinity assistant pod is scheduled:

```shell
kubectl get pods -l app.kubernetes.io/component=affinity-assistant -o wide -w
NAME                              READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
affinity-assistant-c7b485007a-0   0/1     Pending   0          0s    <none>   <none>   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Pending   0          0s    <none>   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     ContainerCreating   0          0s    <none>   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     ContainerCreating   0          1s    <none>   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   1/1     Running             0          5s    10.244.1.144   kind-multinode-worker1   <none>           <none>
```

Now, `cordon` that node to mark it unschedulable for any new pods:

```shell
kubectl cordon kind-multinode-worker1
node/kind-multinode-worker1 cordoned
```

The node is cordoned:

```shell
kubectl get node
NAME                           STATUS                     ROLES           AGE   VERSION
kind-multinode-control-plane   Ready                      control-plane   13d   v1.26.3
kind-multinode-worker1         Ready,SchedulingDisabled   <none>          13d   v1.26.3
kind-multinode-worker2         Ready                      <none>          13d   v1.26.3
```

Now, watch the affinity assistant pod getting transferred onto other available node `kind-multinode-worker2`:

```shell
kubectl get pods -l app.kubernetes.io/component=affinity-assistant -o wide -w
NAME                              READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
affinity-assistant-c7b485007a-0   1/1     Running   0          49s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   1/1     Terminating   0          70s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   1/1     Terminating   0          70s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Terminating   0          70s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Terminating   0          70s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Terminating   0          70s   10.244.1.144   kind-multinode-worker1   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Pending       0          0s    <none>          <none>          <none>           <none>
affinity-assistant-c7b485007a-0   0/1     Pending       0          1s    <none>          kind-multinode-worker2   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     ContainerCreating   0          1s    <none>          kind-multinode-worker2   <none>           <none>
affinity-assistant-c7b485007a-0   0/1     ContainerCreating   0          2s    <none>          kind-multinode-worker2   <none>           <none>
affinity-assistant-c7b485007a-0   1/1     Running             0          4s    10.244.2.144    kind-multinode-worker2   <none>           <none>
```

And, the `pipelineRun` finishes to completion:

```shell
kubectl get pr
NAME                     SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
pipeline-run-r2c7k       True        Succeeded   4m22s       2m1s

kubectl get tr
NAME                            SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
pipeline-run-r2c7k-first-task   True        Succeeded   5m16s       4m7s
pipeline-run-r2c7k-last-task    True        Succeeded   4m6s        2m56s
```

Affinity Assistant    Specifying  Workspaces  in a  Pipeline      workspaces md specifying workspace order in a pipeline and affinity assistants  explains how an affinity assistant is created when a  persistentVolumeClaim  is used as a volume source for a  workspace  in a  pipelineRun   Please refer to the same section for more details on the affinity assistant   This section gives an overview of how the affinity assistant is resilient to a cluster maintenance without losing the running  pipelineRun    Please refer to the issue https   github com tektoncd pipeline issues 6586 for more details    When a list of  tasks  share a single workspace  the affinity assistant pod gets created on a  node  along with all  taskRun  pods  It is very common for a  pipeline  author to design a long running tasks with a single workspace  With these long running tasks  a  node  on which these pods are scheduled can be cordoned while the  pipelineRun  is still running  The tekton controller migrates the affinity assistant pod to any available  node  in a cluster along with the rest of the  taskRun  pods sharing the same workspace   Let s understand this with a sample  pipelineRun       yaml apiVersion  tekton dev v1 kind  PipelineRun metadata    generateName  pipeline run  spec    workspaces      name  source     volumeClaimTemplate        spec          accessModes              ReadWriteOnce         resources            requests              storage  10Mi   pipelineSpec      workspaces        name  source     tasks        name  first task       taskSpec          workspaces            name  source         steps            image  alpine           script                echo   workspaces source path              sleep 60       workspaces          name  source       name  last task       taskSpec          workspaces            name  source         steps            image  alpine           script                echo   workspaces source path              sleep 60       runAfter    first task         workspaces          name  source      This  pipelineRun  has two long running tasks   first task  and  last task   Both of these tasks are sharing a single volume with the access mode set to  ReadWriteOnce  which means the volume can be mounted to a single  node  at any given point of time   Create a  pipelineRun  and determine on which  node  the affinity assistant pod is scheduled      shell kubectl get pods  l app kubernetes io component affinity assistant  o wide  w NAME                              READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES affinity assistant c7b485007a 0   0 1     Pending   0          0s     none     none     none             none  affinity assistant c7b485007a 0   0 1     Pending   0          0s     none    kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     ContainerCreating   0          0s     none    kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     ContainerCreating   0          1s     none    kind multinode worker1    none             none  affinity assistant c7b485007a 0   1 1     Running             0          5s    10 244 1 144   kind multinode worker1    none             none       Now   cordon  that node to mark it unschedulable for any new pods      shell kubectl cordon kind multinode worker1 node kind multinode worker1 cordoned      The node is cordoned      shell kubectl get node NAME                           STATUS                     ROLES           AGE   VERSION kind multinode control plane   Ready                      control plane   13d   v1 26 3 kind multinode worker1         Ready SchedulingDisabled    none           13d   v1 26 3 kind multinode worker2         Ready                       none           13d   v1 26 3      Now  watch the affinity assistant pod getting transferred onto other available node  kind multinode worker2       shell kubectl get pods  l app kubernetes io component affinity assistant  o wide  w NAME                              READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES affinity assistant c7b485007a 0   1 1     Running   0          49s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   1 1     Terminating   0          70s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   1 1     Terminating   0          70s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     Terminating   0          70s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     Terminating   0          70s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     Terminating   0          70s   10 244 1 144   kind multinode worker1    none             none  affinity assistant c7b485007a 0   0 1     Pending       0          0s     none            none            none             none  affinity assistant c7b485007a 0   0 1     Pending       0          1s     none           kind multinode worker2    none             none  affinity assistant c7b485007a 0   0 1     ContainerCreating   0          1s     none           kind multinode worker2    none             none  affinity assistant c7b485007a 0   0 1     ContainerCreating   0          2s     none           kind multinode worker2    none             none  affinity assistant c7b485007a 0   1 1     Running             0          4s    10 244 2 144    kind multinode worker2    none             none       And  the  pipelineRun  finishes to completion      shell kubectl get pr NAME                     SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME pipeline run r2c7k       True        Succeeded   4m22s       2m1s  kubectl get tr NAME                            SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME pipeline run r2c7k first task   True        Succeeded   5m16s       4m7s pipeline run r2c7k last task    True        Succeeded   4m6s        2m56s


tekton This section provides guidelines for running Tekton on your local workstation via the following methods Using Docker Desktop Prerequisites Local Setup

Answer:
# Local Setup

This section provides guidelines for running Tekton on your local workstation via the following methods:

- [Docker for Desktop](#using-docker-desktop)
- [Minikube](#using-minikube)

## Using Docker Desktop

### Prerequisites

Complete these prerequisites to run Tekton locally using Docker Desktop:

- Install the [required tools](https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#requirements).
- Install [Docker Desktop](https://www.docker.com/products/docker-desktop)
- Configure Docker Desktop ([Mac](https://docs.docker.com/docker-for-mac/#resources), [Windows](https://docs.docker.com/docker-for-windows/#resources))to use six CPUs, 10 GB of RAM and 2GB of swap space.
- Set `host.docker.internal:5000` as an insecure registry with Docker for Desktop. See the [Docker insecure registry documentation](https://docs.docker.com/registry/insecure/).
  for details.
- Pass `--insecure` as an argument to your Kaniko tasks so that you can push to an insecure registry.
- Run a local (insecure) Docker registry as follows:

  `docker run -d -p 5000:5000 --name registry-srv -e REGISTRY_STORAGE_DELETE_ENABLED=true registry:2`

- (Optional) Install a Docker registry viewer to verify the images have been pushed:

`docker run -it -p 8080:8080 --name registry-web --link registry-srv -e REGISTRY_URL=http://registry-srv:5000/v2 -e REGISTRY_NAME=localhost:5000 hyper/docker-registry-web`

- Verify that you can push to `host.docker.internal:5000/myregistry/<image_name>`.

### Reconfigure logging

- You can keep your logs in memory only without sending them to a logging service
  such as [Stackdriver](https://cloud.google.com/logging/).
- You can deploy Elasticsearch, Beats, or Kibana locally to view logs. You can find an
  example configuration at <https://github.com/mgreau/tekton-pipelines-elastic-tutorials>.
- To learn more about obtaining logs, see [Logs](logs.md).

## Using Minikube

### Prerequisites

Complete these prerequisites to run Tekton locally using Minikube:

- Install the [required tools](https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#requirements).
- Install [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/) and start a session as follows:
```bash
minikube start --memory 6144 --cpus 2
```
- Point your shell to minikube's docker-daemon by running `eval $(minikube -p minikube docker-env)`
<!-- wokeignore:rule=master -->
- Set up a [registry on minikube](https://github.com/kubernetes/minikube/tree/master/deploy/addons/registry-aliases) by running `minikube addons enable registry` and `minikube addons enable registry-aliases`

### Reconfigure logging

See the information in the "Docker for Desktop" section

## Using kind and local docker registry

### Prerequisites

Complete these prerequisites to run Tekton locally using Kind:

- Install the [required tools](https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#requirements).
- Install [Docker](https://www.docker.com/get-started).
- Install [kind](https://kind.sigs.k8s.io/).

### Use local registry without authentication

See [Using KinD](https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#using-kind).

### Use local private registry

1. Create password file with basic auth.

```bash
export TEST_USER=testuser
export TEST_PASS=testpassword
if [ ! -f auth ]; then
    mkdir auth
fi
docker run \
 --entrypoint htpasswd \
 httpd:2 -Bbn $TEST_USER $TEST_PASS > auth/htpasswd
```

2. Start kind cluster and local private registry

Execute the script.

```shell
#!/bin/sh
set -o errexit

# create registry container unless it already exists
reg_name='kind-registry'
reg_port='5000'
running="$(docker inspect -f '' "${reg_name}" 2>/dev/null || true)"
if [ "${running}" != 'true' ]; then
 docker run \
   -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \
   -v "$(pwd)"/auth:/auth \
   -e "REGISTRY_AUTH=htpasswd" \
   -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
   -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
   registry:2
fi

# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
   endpoint = ["http://${reg_name}:5000"]
EOF

# connect the registry to the cluster network
# (the network may already be connected)
docker network connect "kind" "${reg_name}" || true
# Document the local registry
# <!-- wokeignore:rule=master -->
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
 name: local-registry-hosting
 namespace: kube-public
data:
 localRegistryHosting.v1: |
   host: "localhost:${reg_port}"
   help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

```

3. Install tekton [pipeline](https://github.com/tektoncd/pipeline/blob/main/docs/install.md) and create the secret in cluster.

```bash
kubectl create secret docker-registry secret-tekton \
  --docker-username=$TEST_USER \
  --docker-password=$TEST_PASS \
  --docker-server=localhost:5000 \
   --namespace=tekton-pipelines
```

4. Config [ko](https://github.com/google/ko#install) and add secret to service acount.

```bash
export KO_DOCKER_REPO='localhost:5000'
```

`200-serviceaccount.yaml`

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tekton-pipelines-controller
 namespace: tekton-pipelines
 labels:
   app.kubernetes.io/component: controller
   app.kubernetes.io/instance: default
   app.kubernetes.io/part-of: tekton-pipelines
imagePullSecrets:
 - name: secret-tekton
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tekton-pipelines-webhook
 namespace: tekton-pipelines
 labels:
   app.kubernetes.io/component: webhook
   app.kubernetes.io/instance: default
   app.kubernetes.io/part-of: tekton-pipelines
imagePullSecrets:
 - name: secret-tekton
```

Local Setup  This section provides guidelines for running Tekton on your local workstation via the following methods      Docker for Desktop   using docker desktop     Minikube   using minikube      Using Docker Desktop      Prerequisites  Complete these prerequisites to run Tekton locally using Docker Desktop     Install the  required tools  https   github com tektoncd pipeline blob main DEVELOPMENT md requirements     Install  Docker Desktop  https   www docker com products docker desktop    Configure Docker Desktop   Mac  https   docs docker com docker for mac  resources    Windows  https   docs docker com docker for windows  resources  to use six CPUs  10 GB of RAM and 2GB of swap space    Set  host docker internal 5000  as an insecure registry with Docker for Desktop  See the  Docker insecure registry documentation  https   docs docker com registry insecure      for details    Pass    insecure  as an argument to your Kaniko tasks so that you can push to an insecure registry    Run a local  insecure  Docker registry as follows      docker run  d  p 5000 5000   name registry srv  e REGISTRY STORAGE DELETE ENABLED true registry 2      Optional  Install a Docker registry viewer to verify the images have been pushed    docker run  it  p 8080 8080   name registry web   link registry srv  e REGISTRY URL http   registry srv 5000 v2  e REGISTRY NAME localhost 5000 hyper docker registry web     Verify that you can push to  host docker internal 5000 myregistry  image name         Reconfigure logging    You can keep your logs in memory only without sending them to a logging service   such as  Stackdriver  https   cloud google com logging      You can deploy Elasticsearch  Beats  or Kibana locally to view logs  You can find an   example configuration at  https   github com mgreau tekton pipelines elastic tutorials     To learn more about obtaining logs  see  Logs  logs md       Using Minikube      Prerequisites  Complete these prerequisites to run Tekton locally using Minikube     Install the  required tools  https   github com tektoncd pipeline blob main DEVELOPMENT md requirements     Install  minikube  https   kubernetes io docs tasks tools install minikube   and start a session as follows     bash minikube start   memory 6144   cpus 2       Point your shell to minikube s docker daemon by running  eval   minikube  p minikube docker env        wokeignore rule master       Set up a  registry on minikube  https   github com kubernetes minikube tree master deploy addons registry aliases  by running  minikube addons enable registry  and  minikube addons enable registry aliases       Reconfigure logging  See the information in the  Docker for Desktop  section     Using kind and local docker registry      Prerequisites  Complete these prerequisites to run Tekton locally using Kind     Install the  required tools  https   github com tektoncd pipeline blob main DEVELOPMENT md requirements     Install  Docker  https   www docker com get started     Install  kind  https   kind sigs k8s io         Use local registry without authentication  See  Using KinD  https   github com tektoncd pipeline blob main DEVELOPMENT md using kind        Use local private registry  1  Create password file with basic auth      bash export TEST USER testuser export TEST PASS testpassword if      f auth    then     mkdir auth fi docker run      entrypoint htpasswd    httpd 2  Bbn  TEST USER  TEST PASS   auth htpasswd      2  Start kind cluster and local private registry   Execute the script      shell    bin sh set  o errexit    create registry container unless it already exists reg name  kind registry  reg port  5000  running    docker inspect  f       reg name   2  dev null    true   if      running       true     then  docker run       d   restart always  p  127 0 0 1   reg port  5000    name    reg name         v    pwd   auth  auth       e  REGISTRY AUTH htpasswd        e  REGISTRY AUTH HTPASSWD REALM Registry Realm        e REGISTRY AUTH HTPASSWD PATH  auth htpasswd      registry 2 fi    create a cluster with the local registry enabled in containerd cat   EOF   kind create cluster   config   kind  Cluster apiVersion  kind x k8s io v1alpha4 containerdConfigPatches         plugins  io containerd grpc v1 cri  registry mirrors  localhost   reg port       endpoint     http     reg name  5000   EOF    connect the registry to the cluster network    the network may already be connected  docker network connect  kind     reg name      true   Document the local registry        wokeignore rule master        https   github com kubernetes enhancements tree master keps sig cluster lifecycle generic 1755 communicating a local registry cat   EOF   kubectl apply  f   apiVersion  v1 kind  ConfigMap metadata   name  local registry hosting  namespace  kube public data   localRegistryHosting v1       host   localhost   reg port      help   https   kind sigs k8s io docs user local registry   EOF       3  Install tekton  pipeline  https   github com tektoncd pipeline blob main docs install md  and create the secret in cluster      bash kubectl create secret docker registry secret tekton       docker username  TEST USER       docker password  TEST PASS       docker server localhost 5000        namespace tekton pipelines      4  Config  ko  https   github com google ko install  and add secret to service acount      bash export KO DOCKER REPO  localhost 5000        200 serviceaccount yaml      yaml apiVersion  v1 kind  ServiceAccount metadata   name  tekton pipelines controller  namespace  tekton pipelines  labels     app kubernetes io component  controller    app kubernetes io instance  default    app kubernetes io part of  tekton pipelines imagePullSecrets     name  secret tekton     apiVersion  v1 kind  ServiceAccount metadata   name  tekton pipelines webhook  namespace  tekton pipelines  labels     app kubernetes io component  webhook    app kubernetes io instance  default    app kubernetes io part of  tekton pipelines imagePullSecrets     name  secret tekton


tekton You may follow the existing Tekton feature flags demo for detailed reference Feature Versioning API driven features are features that are accessed via a specific field in pipeline API They comply to the and the specified in the For example is an API driven feature Adding feature gates for API driven features The stability levels of features feature versioning are independent of CRD

Answer:
# Feature Versioning

The stability levels of features (feature versioning) are independent of CRD [API versioning](./api-versioning.md).

You may follow the existing Tekton feature flags demo for detailed reference:
- [Tekton Per-feature Flag Demo Slides](https://docs.google.com/presentation/d/1MAwBTKYUN40SZcd6om6LMw217TtNSppbmA8MatMjyjk/edit?usp=sharing&resourcekey=0-JY7-QhCrWJrzFgsFbJGROg)
- [Tekton Per-feature Flag Demo Recording](https://drive.google.com/file/d/1myFHtqps3gt2I6wBkvGIghDaJElOYOq1/view?usp=sharing)

## Adding feature gates for API-driven features
API-driven features are features that are accessed via a specific field in pipeline API. They comply to the [feature gates](../../api_compatibility_policy.md#feature-gates) and the [feature graduation process](../../api_compatibility_policy.md#feature-graduation-process) specified in the [API compatibility policy](../../api_compatibility_policy.md). For example, [remote tasks](https://github.com/tektoncd/pipeline/blob/454bfd340d102f16f4f2838cf4487198537e3cfa/docs/taskruns.md#remote-tasks) is an API-driven feature.

## Adding feature gated API fields for API-driven features
### Per-feature flag

All new features added after [v0.53.0](https://github.com/tektoncd/pipeline/releases/tag/v0.53.0) will be enabled by their dedicated feature flags. To introduce a new per-feature flag, we will proceed as follows:
- Add default values to the new per-feature flag for the new API-driven feature following the `PerFeatureFlag` struct in [feature_flags.go](./../../pkg/apis/config/feature_flags.go).
- Write unit tests to verify the new feature flag and update all test cases that require the configMap setting, such as those related to provenance propagation.
- To add integration tests:
    - First, add the tests to `pull-tekton-pipeline-alpha-integration-test` by enabling the newly-introduced per-feature flag at [alpha test Prow environment](./../../test/e2e-tests-kind-prow-alpha.env).
    - When the flag is promoted to `beta` stability level, change the test to use [beta Prow environment setup](./../../test/e2e-tests-kind-prow-beta.env).
    - To add additional CI tests for combinations of feature flags, add tests for all alpha feature flags being turned on, with one alpha feature turned off at a time.
- Add the tested new per-feature flag key value to the [the pipeline configMap](./../../config/config-feature-flags.yaml).
- Update documentations for the new alpha feature at [alpha-stability-level](./../additional-configs.md#alpha-features).

#### Example of adding a new Per-feature flag
1. Add the default value following the Per-Feature flag struct
```golang
const enableExampleNewFeatureKey = 'example-new-feature'

var DefaultExampleNewFeatre = PerFeatureFlag {
        Name:      enableExampleNewFeatureKey,
        Stability: AlphaAPIFields,
        Enabled:   DefaultAlphaFeatureEnabled,
}
```
2. Add unit tests with the newly-introduced yamls `feature-flags-example-new-feature` and `feature-flags-invalid-example-new-feature` according to the existing testing framework.

3. For integration tests, add `example-new-feature: true` to [alpha test Prow environment](./../../test/e2e-tests-kind-prow-alpha.env).

4. Add `example-new-feature: false` to [the pipeline configMap](./../../config/config-feature-flags.yaml) with a release note.

5. Update documentations for the new alpha feature at [alpha-stability-level](./../additional-configs.md#alpha-features).

### `enable-api-fields`

Prior to [v0.53.0](https://github.com/tektoncd/pipeline/tree/release-v0.53.x),  we have had the global feature flag `enable-api-fields` in
[config-feature-flags.yaml file](../../config/config-feature-flags.yaml)
deployed as part of our releases.

_Note that the `enable-api-fields` flag will has been deprecated since [v0.53.0](https://github.com/tektoncd/pipeline/tree/release-v0.53.x) and we will transition to use [Per-feature flags](#per-feature-flag) instead._

This field can be configured either to be `alpha`, `beta`, or `stable`. This field is
documented as part of our
[install docs](../install.md#customizing-the-pipelines-controller-behavior).

For developers adding new features to Pipelines' CRDs we've got a couple of
helpful tools to make gating those features simpler and to provide a consistent
testing experience.

### Guarding Features with Feature Gates

Writing new features is made trickier when you need to support both the existing
stable behaviour as well as your new alpha behaviour.

In reconciler code you can guard your new features with an `if` statement such
as the following:

```go
alphaAPIEnabled := config.FromContextOrDefaults(ctx).FeatureFlags.EnableAPIFields == "alpha"
if alphaAPIEnabled {
  // new feature code goes here
} else {
  // existing stable code goes here
}
```

Notice that you'll need a context object to be passed into your function for
this to work. When writing new features keep in mind that you might need to
include this in your new function signatures.

### Guarding Validations with Feature Gates

Just because your application code might be correctly observing the feature gate
flag doesn't mean you're done yet! When a user submits a Tekton resource it's
validated by Pipelines' webhook. Here too you'll need to ensure your new
features aren't accidentally accepted when the feature gate suggests they
shouldn't be. We've got a helper function,
[`ValidateEnabledAPIFields`](../../pkg/apis/version/version_validation.go),
to make validating the current feature gate easier. Use it like this:

```go
requiredVersion := config.AlphaAPIFields
// errs is an instance of *apis.FieldError, a common type in our validation code
errs = errs.Also(ValidateEnabledAPIFields(ctx, "your feature name", requiredVersion))
```

If the user's cluster isn't configured with the required feature gate it'll
return an error like this:

```
<your feature> requires "enable-api-fields" feature gate to be "alpha" but it is "stable"
```

### Unit Testing with Feature Gates

Any new code you write that uses the `ctx` context variable is trivially unit
tested with different feature gate settings. You should make sure to unit test
your code both with and without a feature gate enabled to make sure it's
properly guarded. See the following for an example of a unit test that sets the
feature gate to test behaviour:

```go
// EnableAlphaAPIFields enables alpha features in an existing context (for use in testing)
func EnableAlphaAPIFields(ctx context.Context) context.Context {
	return setEnableAPIFields(ctx, config.AlphaAPIFields)
}

func setEnableAPIFields(ctx context.Context, want string) context.Context {
	featureFlags, _ := config.NewFeatureFlagsFromMap(map[string]string{
		"enable-api-fields": want,
	})
	cfg := &config.Config{
		Defaults: &config.Defaults{
			DefaultTimeoutMinutes: 60,
		},
		FeatureFlags: featureFlags,
	}
	return config.ToContext(ctx, cfg)
}
```

### Example YAMLs

Writing new YAML examples that require a feature gate to be set is easy. New
YAML example files typically go in a directory called something like
`examples/v1/taskruns` in the root of the repo. To create a YAML that
should only be exercised when the `enable-api-fields` flag is `alpha` just put
it in an `alpha` subdirectory so the structure looks like:

```
examples/v1/taskruns/alpha/your-example.yaml
```

This should work for both taskruns and pipelineruns.

**Note**: To execute alpha examples with the integration test runner you must
manually set the `enable-api-fields` feature flag to `alpha` in your testing
cluster before kicking off the tests.

When you set this flag to `stable` in your cluster it will prevent `alpha`
examples from being created by the test runner. When you set the flag to `alpha`
all examples are run, since we want to exercise backwards-compatibility of the
examples under alpha conditions.

### Integration Tests

For integration tests we provide the
[`requireAnyGate` function](../../test/gate.go) which should be passed to the
`setup` function used by tests:

```go
c, namespace := setup(ctx, t, requireAnyGate(map[string]string{"enable-api-fields": "alpha"}))
```

This will Skip your integration test if the feature gate is not set to `alpha`
with a clear message explaining why it was skipped.

**Note**: As with running example YAMLs you have to manually set the
`enable-api-fields` flag to `alpha` in your test cluster to see your alpha
integration tests run. When the flag in your cluster is `alpha` _all_
integration tests are executed, both `stable` and `alpha`. Setting the feature
flag to `stable` will exclude `alpha` tests.

Feature Versioning  The stability levels of features  feature versioning  are independent of CRD  API versioning    api versioning md    You may follow the existing Tekton feature flags demo for detailed reference     Tekton Per feature Flag Demo Slides  https   docs google com presentation d 1MAwBTKYUN40SZcd6om6LMw217TtNSppbmA8MatMjyjk edit usp sharing resourcekey 0 JY7 QhCrWJrzFgsFbJGROg     Tekton Per feature Flag Demo Recording  https   drive google com file d 1myFHtqps3gt2I6wBkvGIghDaJElOYOq1 view usp sharing      Adding feature gates for API driven features API driven features are features that are accessed via a specific field in pipeline API  They comply to the  feature gates        api compatibility policy md feature gates  and the  feature graduation process        api compatibility policy md feature graduation process  specified in the  API compatibility policy        api compatibility policy md   For example   remote tasks  https   github com tektoncd pipeline blob 454bfd340d102f16f4f2838cf4487198537e3cfa docs taskruns md remote tasks  is an API driven feature      Adding feature gated API fields for API driven features     Per feature flag  All new features added after  v0 53 0  https   github com tektoncd pipeline releases tag v0 53 0  will be enabled by their dedicated feature flags  To introduce a new per feature flag  we will proceed as follows    Add default values to the new per feature flag for the new API driven feature following the  PerFeatureFlag  struct in  feature flags go          pkg apis config feature flags go     Write unit tests to verify the new feature flag and update all test cases that require the configMap setting  such as those related to provenance propagation    To add integration tests        First  add the tests to  pull tekton pipeline alpha integration test  by enabling the newly introduced per feature flag at  alpha test Prow environment          test e2e tests kind prow alpha env         When the flag is promoted to  beta  stability level  change the test to use  beta Prow environment setup          test e2e tests kind prow beta env         To add additional CI tests for combinations of feature flags  add tests for all alpha feature flags being turned on  with one alpha feature turned off at a time    Add the tested new per feature flag key value to the  the pipeline configMap          config config feature flags yaml     Update documentations for the new alpha feature at  alpha stability level       additional configs md alpha features         Example of adding a new Per feature flag 1  Add the default value following the Per Feature flag struct    golang const enableExampleNewFeatureKey    example new feature   var DefaultExampleNewFeatre   PerFeatureFlag           Name       enableExampleNewFeatureKey          Stability  AlphaAPIFields          Enabled    DefaultAlphaFeatureEnabled        2  Add unit tests with the newly introduced yamls  feature flags example new feature  and  feature flags invalid example new feature  according to the existing testing framework   3  For integration tests  add  example new feature  true  to  alpha test Prow environment          test e2e tests kind prow alpha env    4  Add  example new feature  false  to  the pipeline configMap          config config feature flags yaml  with a release note   5  Update documentations for the new alpha feature at  alpha stability level       additional configs md alpha features         enable api fields   Prior to  v0 53 0  https   github com tektoncd pipeline tree release v0 53 x    we have had the global feature flag  enable api fields  in  config feature flags yaml file        config config feature flags yaml  deployed as part of our releases    Note that the  enable api fields  flag will has been deprecated since  v0 53 0  https   github com tektoncd pipeline tree release v0 53 x  and we will transition to use  Per feature flags   per feature flag  instead    This field can be configured either to be  alpha    beta   or  stable   This field is documented as part of our  install docs     install md customizing the pipelines controller behavior    For developers adding new features to Pipelines  CRDs we ve got a couple of helpful tools to make gating those features simpler and to provide a consistent testing experience       Guarding Features with Feature Gates  Writing new features is made trickier when you need to support both the existing stable behaviour as well as your new alpha behaviour   In reconciler code you can guard your new features with an  if  statement such as the following      go alphaAPIEnabled    config FromContextOrDefaults ctx  FeatureFlags EnableAPIFields     alpha  if alphaAPIEnabled        new feature code goes here   else        existing stable code goes here        Notice that you ll need a context object to be passed into your function for this to work  When writing new features keep in mind that you might need to include this in your new function signatures       Guarding Validations with Feature Gates  Just because your application code might be correctly observing the feature gate flag doesn t mean you re done yet  When a user submits a Tekton resource it s validated by Pipelines  webhook  Here too you ll need to ensure your new features aren t accidentally accepted when the feature gate suggests they shouldn t be  We ve got a helper function    ValidateEnabledAPIFields         pkg apis version version validation go   to make validating the current feature gate easier  Use it like this      go requiredVersion    config AlphaAPIFields    errs is an instance of  apis FieldError  a common type in our validation code errs   errs Also ValidateEnabledAPIFields ctx   your feature name   requiredVersion        If the user s cluster isn t configured with the required feature gate it ll return an error like this        your feature  requires  enable api fields  feature gate to be  alpha  but it is  stable           Unit Testing with Feature Gates  Any new code you write that uses the  ctx  context variable is trivially unit tested with different feature gate settings  You should make sure to unit test your code both with and without a feature gate enabled to make sure it s properly guarded  See the following for an example of a unit test that sets the feature gate to test behaviour      go    EnableAlphaAPIFields enables alpha features in an existing context  for use in testing  func EnableAlphaAPIFields ctx context Context  context Context    return setEnableAPIFields ctx  config AlphaAPIFields     func setEnableAPIFields ctx context Context  want string  context Context    featureFlags       config NewFeatureFlagsFromMap map string string     enable api fields   want       cfg     config Config    Defaults   config Defaults     DefaultTimeoutMinutes  60         FeatureFlags  featureFlags      return config ToContext ctx  cfg             Example YAMLs  Writing new YAML examples that require a feature gate to be set is easy  New YAML example files typically go in a directory called something like  examples v1 taskruns  in the root of the repo  To create a YAML that should only be exercised when the  enable api fields  flag is  alpha  just put it in an  alpha  subdirectory so the structure looks like       examples v1 taskruns alpha your example yaml      This should work for both taskruns and pipelineruns     Note    To execute alpha examples with the integration test runner you must manually set the  enable api fields  feature flag to  alpha  in your testing cluster before kicking off the tests   When you set this flag to  stable  in your cluster it will prevent  alpha  examples from being created by the test runner  When you set the flag to  alpha  all examples are run  since we want to exercise backwards compatibility of the examples under alpha conditions       Integration Tests  For integration tests we provide the   requireAnyGate  function        test gate go  which should be passed to the  setup  function used by tests      go c  namespace    setup ctx  t  requireAnyGate map string string  enable api fields    alpha          This will Skip your integration test if the feature gate is not set to  alpha  with a clear message explaining why it was skipped     Note    As with running example YAMLs you have to manually set the  enable api fields  flag to  alpha  in your test cluster to see your alpha integration tests run  When the flag in your cluster is  alpha   all  integration tests are executed  both  stable  and  alpha   Setting the feature flag to  stable  will exclude  alpha  tests


tekton If specified wait until the file has non zero size user provided binary for each container to manage the execution order of the containers Entrypoint rewriting and step ordering This doc describes how TaskRuns are implemented using pods The binary has the following arguments Tekton releases include a binary called the entrypoint which wraps the If specified file to wait for

Answer:
This doc describes how TaskRuns are implemented using pods.

## Entrypoint rewriting and step ordering

Tekton releases include a binary called the "entrypoint", which wraps the
user-provided binary for each `Step` container to manage the execution order of the containers.
The `entrypoint` binary has the following arguments:

- `wait_file` - If specified, file to wait for
- `wait_file_content` - If specified, wait until the file has non-zero size
- `post_file` - If specified, file to write upon completion
- `entrypoint` - The command to run in the image being wrapped

As part of the PodSpec created by `TaskRun` the entrypoint for each `Task` step
is changed to the entrypoint binary with the mentioned arguments and a volume
with the binary and file(s) is mounted.

If the image is a private registry, the service account should include an
[ImagePullSecret](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account)

For more details, see [entrypoint/README.md](../../cmd/entrypoint/README.md)
or the talk ["Russian Doll: Extending Containers with Nested Processes"](https://www.youtube.com/watch?v=iz9_omZ0ctk).

## How to access the exit code of a step from any subsequent step in a task

The entrypoint now allows exiting with an error and continue running rest of the
steps in a task i.e., it is possible for a step to exit with a non-zero exit
code. Now, it is possible to design a task with a step which can take an action
depending on the exit code of any prior steps. The user can access the exit code
of a step by reading the file pointed by the path variable
`$(steps.step-<step-name>.exitCode.path)` or
`$(steps.step-unnamed-<step-index>.exitCode.path)`. For example:

- `$(steps.step-my-awesome-step.exitCode.path)` where the step name is
  `my-awesome-step`.
- `$(steps.step-unnamed-0.exitCode.path)` where the first step in a task has no
  name.

The exit code of a step is stored in a file named `exitCode` under a directory
`/tekton/steps/step-<step-name>/` or `/tekton/steps/step-unnamed-<step-index>/`
which is reserved for any other step specific information in the future.

If you would like to use the tekton internal path, you can access the exit code
by reading the file (which is not recommended though since the path might change
in the future):

```shell
cat /tekton/steps/step-<step-name>/exitCode
```

And, access the step exit code without a step name:

```shell
cat /tekton/steps/step-unnamed-<step-index>/exitCode
```

Or, you can access the step metadata directory via symlink, for example, use
`cat /tekton/steps/0/exitCode` for the first step in a task.

## TaskRun Use of Pod Termination Messages

Tekton Pipelines uses a `Pod's`
[termination message](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/)
to pass data from a Step's container to the Pipelines controller. Examples of
this data include: the time that execution of the user's step began, contents of
task results, contents of pipeline resource results.

The contents and format of the termination message can change. At time of
writing the message takes the form of a serialized JSON blob. Some of the data
from the message is internal to Tekton Pipelines, used for book-keeping, and
some is distributed across a number of fields of the `TaskRun's` `status`. For
example, a `TaskRun's` `status.taskResults` is populated from the termination
message.

## Reserved directories

### /workspace

- `/workspace` - This directory is where volumes for [resources](#resources) and
  [workspaces](#workspaces) are mounted.

### /tekton

The `/tekton/` directory is reserved on containers for internal usage.

Here is an example of a directory layout for a simple Task with 2 script steps:

```
/tekton
|-- bin
    `-- entrypoint
|-- creds
|-- downward
|   |-- ..2021_09_16_18_31_06.270542700
|   |   `-- ready
|   |-- ..data -> ..2021_09_16_18_31_06.270542700
|   `-- ready -> ..data/ready
|-- home
|-- results
|-- run
    `-- 0
        `-- out
        `-- status
            `-- exitCode
|-- scripts
|   |-- script-0-t4jd8
|   `-- script-1-4pjwp
|-- steps
|   |-- 0 -> /tekton/run/0/status
|   |-- 1 -> /tekton/run/1/status
|   |-- step-foo -> /tekton/run/1/status
|   `-- step-unnamed-0 -> /tekton/run/0/status
`-- termination
```

| Path                | Description                                                                                |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| /tekton             | Directory used for Tekton specific functionality                                                                                |
| /tekton/bin         | Tekton provided binaries / tools                                                                                |
| /tekton/creds       | Location of Tekton mounted secrets. See [Authentication at Run Time](../auth.md) for more details.                                                                                |
| /tekton/debug       | Contains [Debug scripts](https://github.com/tektoncd/pipeline/blob/main/docs/debug.md#debug-scripts) used to manage step lifecycle during debugging at a breakpoint and the [Debug Info](https://github.com/tektoncd/pipeline/blob/main/docs/debug.md#mounts) mount used to assist for the same.         |                                                                                |
| /tekton/downward    | Location of data mounted via the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api).                                                                                |
| /tekton/home        | (deprecated - see https://github.com/tektoncd/pipeline/issues/2013) Default home directory for user containers.                                                                                |
| /tekton/results     | Where [results](#results) are written to (path available to `Task` authors via [`$(results.name.path)`](../variables.md))                                                                                |
| /tekton/run         | Runtime variable data. [Used for coordinating step ordering](#entrypoint-rewriting-and-step-ordering).                                                                                |
| /tekton/scripts     | Contains user provided scripts specified in the TaskSpec.                                                                                |
| /tekton/steps       | Where the `step` exitCodes are written to (path available to `Task` authors via [`$(steps.<stepName>.exitCode.path)`](../variables.md#variables-available-in-a-task))                                                                                |
| /tekton/termination | where the eventual [termination log message](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#writing-and-reading-a-termination-message) is written to [Sequencing step containers](#entrypoint-rewriting-and-step-ordering)                                     |

The following directories are covered by the
[Tekton API Compatibility policy](../../api_compatibility_policy.md), and can be
relied on for stability:

- `/tekton/results`

All other files/directories are internal implementation details of Tekton -
**users should not rely on specific paths or behaviors as it may change in the
future**.

## What and Why of `/tekton/run`

`/tekton/run` is a collection of implicit volumes mounted on a pod and created
for storing the step specific information/metadata. Steps can only write
metadata to their own `tekton/run` directory - all other step volumes are mounted as
`readonly`. The `tekton/run` directories are considered internal implementation details
of Tekton and are not bound by the API compatibility policy - the contents and
structure can be safely changed so long as user behavior remains the same.

### `/tekton/steps`

`/tekton/steps` are special subdirectories are created for each step in a task -
each directory is actually a symlink to a directory in the Step's corresponding
`/tekton/run` volume. This is done to ensure that step directories can only be
modified by their own Step. To ensure that these symlinks are not modified, the
entire `/tekton/steps` volume is initially populated by an initContainer, and
mounted `readonly` on all user steps.

These symlinks are created as a part of the `step-init` entrypoint subcommand
initContainer on each Task Pod.

### Entrypoint configuration

The entrypoint is modified to include an additional flag representing the step
specific directory where step metadata should be written:

```
step_metadata_dir - the dir specified in this flag is created to hold a step specific metadata
```

`step_metadata_dir` is set to `/tekton/run/<step #>/status` for the entrypoint
of each step.

### Example

Let's take an example of a task with two steps, each exiting with non-zero exit
code:

```yaml
kind: TaskRun
apiVersion: tekton.dev/v1beta1
metadata:
  generateName: test-taskrun-
spec:
  taskSpec:
    steps:
      - image: alpine
        name: step0
        onError: continue
        script: |
          exit 1
      - image: alpine
        onError: continue
        script: |
          exit 2
```

During `step-step0`, the first container is actively running so none of the
output files are populated yet. The `/tekton/steps` directories are symlinked to
locations that do not yet exist, but will be populated during execution.

```
/tekton
|-- run
|   |-- 0
|   `-- 1
|-- steps
    |-- 0 -> /tekton/run/0/status
    |-- 1 -> /tekton/run/1/status
    |-- step-step0 -> /tekton/run/0/status
    `-- step-unnamed1 -> /tekton/run/1/status
```

During `step-unnamed1`, the first container has now finished. The output files
for the first step are now populated, and the folder pointed to by
`/tekton/steps/0` now exists, and is populated with a file named `exitCode`
which contains the exit code of the first step.

```
/tekton
|-- run
|   |-- 0
|   |   |-- out
|   |   `-- status
|   |       `-- exitCode
|   `-- 1
|-- steps
    |-- 0 -> /tekton/run/0/status
    |-- 1 -> /tekton/run/1/status
    |-- step-step0 -> /tekton/run/0/status
    `-- step-unnamed1 -> /tekton/run/1/status
```

Notice that there are multiple symlinks showing under `/tekton/steps/` pointing
to the same `/tekton/run` location. These symbolic links are created to provide
simplified access to the step metadata directories i.e., instead of referring to
a directory with the step name, access it via the step index. The step index
becomes complex and hard to keep track of in a task with a long list of steps,
for example, a task with 20 steps. Creating the step metadata directory using a
step name and creating a symbolic link using the step index gives the user
flexibility, and an option to choose whatever works best for them.

## Handling of injected sidecars

Tekton has to take some special steps to support sidecars that are injected into
TaskRun Pods. Without intervention sidecars will typically run for the entire
lifetime of a Pod but in Tekton's case it's desirable for the sidecars to run
only as long as Steps take to complete. There's also a need for Tekton to
schedule the sidecars to start before a Task's Steps begin, just in case the
Steps rely on a sidecars behavior, for example to join an Istio service mesh. To
handle all of this, Tekton Pipelines implements the following lifecycle for
sidecar containers:

First, the
[Downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api)
is used to project an annotation on the TaskRun's Pod into the `entrypoint`
container as a file. The annotation starts as an empty string, so the file
projected by the downward API has zero length. The entrypointer spins, waiting
for that file to have non-zero size.

The sidecar containers start up. Once they're all in a ready state, the
annotation is populated with string "READY", which in turn populates the
Downward API projected file. The entrypoint binary recognizes that the projected
file has a non-zero size and allows the Task's steps to begin.

On completion of all steps in a Task the TaskRun reconciler stops any sidecar
containers. The `Image` field of any sidecar containers is swapped to the nop
image. Kubernetes observes the change and relaunches the container with updated
container image. The nop container image exits immediately _because it does not
provide the command that the sidecar is configured to run_. The container is
considered `Terminated` by Kubernetes and the TaskRun's Pod stops.

There are known issues with the existing implementation of sidecars:

- When the `nop` image does provide the sidecar's command, the sidecar will
  continue to run even after `nop` has been swapped into the sidecar container's
  image field. See
  [the issue tracking this bug](https://github.com/tektoncd/pipeline/issues/1347)
  for the issue tracking this bug. Until this issue is resolved the best way to
  avoid it is to avoid overriding the `nop` image when deploying the tekton
  controller, or ensuring that the overridden `nop` image contains as few
  commands as possible.

- `kubectl get pods` will show a Completed pod when a sidecar exits successfully
  but an Error when the sidecar exits with an error. This is only apparent when
  using `kubectl` to get the pods of a TaskRun, not when describing the Pod
  using `kubectl describe pod ...` nor when looking at the TaskRun, but can be
  quite confusing.

## Breakpoint on Failure

Halting a TaskRun execution on Failure of a step.

### Failure of a Step

The entrypoint binary is used to manage the lifecycle of a step. Steps are aligned beforehand by the TaskRun controller
allowing each step to run in a particular order. This is done using `-wait_file` and the `-post_file` flags. The former
let's the entrypoint binary know that it has to wait on creation of a particular file before starting execution of the step.
And the latter provides information on the step number and signal the next step on completion of the step.

On success of a step, the `-post-file` is written as is, signalling the next step which would have the same argument given
for `-wait_file` to resume the entrypoint process and move ahead with the step.

On failure of a step, the `-post_file` is written with appending `.err` to it denoting that the previous step has failed with
and error. The subsequent steps are skipped in this case as well, marking the TaskRun as a failure.

### Halting a Step on failure

The failed step writes `<step-no>.err` to `/tekton/run` and stops running completely. To be able to debug a step we would
need it to continue running (not exit), not skip the next steps and signal health of the step. By disabling step skipping,
stopping write of the `<step-no>.err` file and waiting on a signal by the user to disable the halt, we would be simulating a
"breakpoint".

In this breakpoint, which is essentially a limbo state the TaskRun finds itself in, the user can interact with the step
environment using a CLI or an IDE.

### Exiting onfailure breakpoint

To exit a step which has been paused upon failure, the step would wait on a file similar to `<step-no>.breakpointexit` which
would unpause and exit the step container. eg: Step 0 fails and is paused. Writing `0.breakpointexit` in `/tekton/run`
would unpause and exit the step container.

## Breakpoint before step

TaskRun will be stuck waiting for user debugging before the step execution.

### Halting a Step before execution

The step program will be executed after all the `-wait_file` monitoring ends. If want the user to enter the debugging before the step is executed,
need to pass a parameter `debug_before_step` to `entrypoint`,
and `entrypoint` will end the monitoring of `waitFiles` back pause,
waiting to listen to the `/tekton/run/0/out.beforestepexit` file

### Exiting before step breakpoint

`entrypoint` listening `/tekton/run//out.beforestepexit` or `/tekton/run//out.beforestepexit.err` to
decide whether to proceed this step, `out.beforestepexit` means continue with step,
`out.beforestepexit.err` means do not continue with the step

This doc describes how TaskRuns are implemented using pods      Entrypoint rewriting and step ordering  Tekton releases include a binary called the  entrypoint   which wraps the user provided binary for each  Step  container to manage the execution order of the containers  The  entrypoint  binary has the following arguments      wait file    If specified  file to wait for    wait file content    If specified  wait until the file has non zero size    post file    If specified  file to write upon completion    entrypoint    The command to run in the image being wrapped  As part of the PodSpec created by  TaskRun  the entrypoint for each  Task  step is changed to the entrypoint binary with the mentioned arguments and a volume with the binary and file s  is mounted   If the image is a private registry  the service account should include an  ImagePullSecret  https   kubernetes io docs tasks configure pod container configure service account  add imagepullsecrets to a service account   For more details  see  entrypoint README md        cmd entrypoint README md  or the talk   Russian Doll  Extending Containers with Nested Processes   https   www youtube com watch v iz9 omZ0ctk       How to access the exit code of a step from any subsequent step in a task  The entrypoint now allows exiting with an error and continue running rest of the steps in a task i e   it is possible for a step to exit with a non zero exit code  Now  it is possible to design a task with a step which can take an action depending on the exit code of any prior steps  The user can access the exit code of a step by reading the file pointed by the path variable    steps step  step name  exitCode path   or    steps step unnamed  step index  exitCode path    For example        steps step my awesome step exitCode path   where the step name is    my awesome step        steps step unnamed 0 exitCode path   where the first step in a task has no   name   The exit code of a step is stored in a file named  exitCode  under a directory   tekton steps step  step name    or   tekton steps step unnamed  step index    which is reserved for any other step specific information in the future   If you would like to use the tekton internal path  you can access the exit code by reading the file  which is not recommended though since the path might change in the future       shell cat  tekton steps step  step name  exitCode      And  access the step exit code without a step name      shell cat  tekton steps step unnamed  step index  exitCode      Or  you can access the step metadata directory via symlink  for example  use  cat  tekton steps 0 exitCode  for the first step in a task      TaskRun Use of Pod Termination Messages  Tekton Pipelines uses a  Pod s   termination message  https   kubernetes io docs tasks debug application cluster determine reason pod failure   to pass data from a Step s container to the Pipelines controller  Examples of this data include  the time that execution of the user s step began  contents of task results  contents of pipeline resource results   The contents and format of the termination message can change  At time of writing the message takes the form of a serialized JSON blob  Some of the data from the message is internal to Tekton Pipelines  used for book keeping  and some is distributed across a number of fields of the  TaskRun s   status   For example  a  TaskRun s   status taskResults  is populated from the termination message      Reserved directories       workspace      workspace    This directory is where volumes for  resources   resources  and    workspaces   workspaces  are mounted        tekton  The   tekton   directory is reserved on containers for internal usage   Here is an example of a directory layout for a simple Task with 2 script steps        tekton     bin         entrypoint     creds     downward           2021 09 16 18 31 06 270542700             ready           data      2021 09 16 18 31 06 270542700         ready      data ready     home     results     run         0             out             status                 exitCode     scripts         script 0 t4jd8         script 1 4pjwp     steps         0     tekton run 0 status         1     tekton run 1 status         step foo     tekton run 1 status         step unnamed 0     tekton run 0 status     termination        Path                  Description                                                                                tekton               Directory used for Tekton specific functionality                                                                                tekton bin           Tekton provided binaries   tools                                                                                tekton creds         Location of Tekton mounted secrets  See  Authentication at Run Time     auth md  for more details                                                                                tekton debug         Contains  Debug scripts  https   github com tektoncd pipeline blob main docs debug md debug scripts  used to manage step lifecycle during debugging at a breakpoint and the  Debug Info  https   github com tektoncd pipeline blob main docs debug md mounts  mount used to assist for the same                                                                                tekton downward      Location of data mounted via the  Downward API  https   kubernetes io docs tasks inject data application downward api volume expose pod information  the downward api                                                                                tekton home           deprecated   see https   github com tektoncd pipeline issues 2013  Default home directory for user containers                                                                                tekton results       Where  results   results  are written to  path available to  Task  authors via     results name path       variables md                                                                                tekton run           Runtime variable data   Used for coordinating step ordering   entrypoint rewriting and step ordering                                                                                tekton scripts       Contains user provided scripts specified in the TaskSpec                                                                                tekton steps         Where the  step  exitCodes are written to  path available to  Task  authors via     steps  stepName  exitCode path       variables md variables available in a task                                                                                tekton termination   where the eventual  termination log message  https   kubernetes io docs tasks debug application cluster determine reason pod failure  writing and reading a termination message  is written to  Sequencing step containers   entrypoint rewriting and step ordering                                         The following directories are covered by the  Tekton API Compatibility policy        api compatibility policy md   and can be relied on for stability       tekton results   All other files directories are internal implementation details of Tekton     users should not rely on specific paths or behaviors as it may change in the future        What and Why of   tekton run     tekton run  is a collection of implicit volumes mounted on a pod and created for storing the step specific information metadata  Steps can only write metadata to their own  tekton run  directory   all other step volumes are mounted as  readonly   The  tekton run  directories are considered internal implementation details of Tekton and are not bound by the API compatibility policy   the contents and structure can be safely changed so long as user behavior remains the same         tekton steps     tekton steps  are special subdirectories are created for each step in a task   each directory is actually a symlink to a directory in the Step s corresponding   tekton run  volume  This is done to ensure that step directories can only be modified by their own Step  To ensure that these symlinks are not modified  the entire   tekton steps  volume is initially populated by an initContainer  and mounted  readonly  on all user steps   These symlinks are created as a part of the  step init  entrypoint subcommand initContainer on each Task Pod       Entrypoint configuration  The entrypoint is modified to include an additional flag representing the step specific directory where step metadata should be written       step metadata dir   the dir specified in this flag is created to hold a step specific metadata       step metadata dir  is set to   tekton run  step    status  for the entrypoint of each step       Example  Let s take an example of a task with two steps  each exiting with non zero exit code      yaml kind  TaskRun apiVersion  tekton dev v1beta1 metadata    generateName  test taskrun  spec    taskSpec      steps          image  alpine         name  step0         onError  continue         script              exit 1         image  alpine         onError  continue         script              exit 2      During  step step0   the first container is actively running so none of the output files are populated yet  The   tekton steps  directories are symlinked to locations that do not yet exist  but will be populated during execution        tekton     run         0         1     steps         0     tekton run 0 status         1     tekton run 1 status         step step0     tekton run 0 status         step unnamed1     tekton run 1 status      During  step unnamed1   the first container has now finished  The output files for the first step are now populated  and the folder pointed to by   tekton steps 0  now exists  and is populated with a file named  exitCode  which contains the exit code of the first step        tekton     run         0             out             status                 exitCode         1     steps         0     tekton run 0 status         1     tekton run 1 status         step step0     tekton run 0 status         step unnamed1     tekton run 1 status      Notice that there are multiple symlinks showing under   tekton steps   pointing to the same   tekton run  location  These symbolic links are created to provide simplified access to the step metadata directories i e   instead of referring to a directory with the step name  access it via the step index  The step index becomes complex and hard to keep track of in a task with a long list of steps  for example  a task with 20 steps  Creating the step metadata directory using a step name and creating a symbolic link using the step index gives the user flexibility  and an option to choose whatever works best for them      Handling of injected sidecars  Tekton has to take some special steps to support sidecars that are injected into TaskRun Pods  Without intervention sidecars will typically run for the entire lifetime of a Pod but in Tekton s case it s desirable for the sidecars to run only as long as Steps take to complete  There s also a need for Tekton to schedule the sidecars to start before a Task s Steps begin  just in case the Steps rely on a sidecars behavior  for example to join an Istio service mesh  To handle all of this  Tekton Pipelines implements the following lifecycle for sidecar containers   First  the  Downward API  https   kubernetes io docs tasks inject data application downward api volume expose pod information  the downward api  is used to project an annotation on the TaskRun s Pod into the  entrypoint  container as a file  The annotation starts as an empty string  so the file projected by the downward API has zero length  The entrypointer spins  waiting for that file to have non zero size   The sidecar containers start up  Once they re all in a ready state  the annotation is populated with string  READY   which in turn populates the Downward API projected file  The entrypoint binary recognizes that the projected file has a non zero size and allows the Task s steps to begin   On completion of all steps in a Task the TaskRun reconciler stops any sidecar containers  The  Image  field of any sidecar containers is swapped to the nop image  Kubernetes observes the change and relaunches the container with updated container image  The nop container image exits immediately  because it does not provide the command that the sidecar is configured to run   The container is considered  Terminated  by Kubernetes and the TaskRun s Pod stops   There are known issues with the existing implementation of sidecars     When the  nop  image does provide the sidecar s command  the sidecar will   continue to run even after  nop  has been swapped into the sidecar container s   image field  See    the issue tracking this bug  https   github com tektoncd pipeline issues 1347    for the issue tracking this bug  Until this issue is resolved the best way to   avoid it is to avoid overriding the  nop  image when deploying the tekton   controller  or ensuring that the overridden  nop  image contains as few   commands as possible      kubectl get pods  will show a Completed pod when a sidecar exits successfully   but an Error when the sidecar exits with an error  This is only apparent when   using  kubectl  to get the pods of a TaskRun  not when describing the Pod   using  kubectl describe pod      nor when looking at the TaskRun  but can be   quite confusing      Breakpoint on Failure  Halting a TaskRun execution on Failure of a step       Failure of a Step  The entrypoint binary is used to manage the lifecycle of a step  Steps are aligned beforehand by the TaskRun controller allowing each step to run in a particular order  This is done using   wait file  and the   post file  flags  The former let s the entrypoint binary know that it has to wait on creation of a particular file before starting execution of the step  And the latter provides information on the step number and signal the next step on completion of the step   On success of a step  the   post file  is written as is  signalling the next step which would have the same argument given for   wait file  to resume the entrypoint process and move ahead with the step   On failure of a step  the   post file  is written with appending   err  to it denoting that the previous step has failed with and error  The subsequent steps are skipped in this case as well  marking the TaskRun as a failure       Halting a Step on failure  The failed step writes   step no  err  to   tekton run  and stops running completely  To be able to debug a step we would need it to continue running  not exit   not skip the next steps and signal health of the step  By disabling step skipping  stopping write of the   step no  err  file and waiting on a signal by the user to disable the halt  we would be simulating a  breakpoint    In this breakpoint  which is essentially a limbo state the TaskRun finds itself in  the user can interact with the step environment using a CLI or an IDE       Exiting onfailure breakpoint  To exit a step which has been paused upon failure  the step would wait on a file similar to   step no  breakpointexit  which would unpause and exit the step container  eg  Step 0 fails and is paused  Writing  0 breakpointexit  in   tekton run  would unpause and exit the step container      Breakpoint before step  TaskRun will be stuck waiting for user debugging before the step execution       Halting a Step before execution  The step program will be executed after all the   wait file  monitoring ends  If want the user to enter the debugging before the step is executed  need to pass a parameter  debug before step  to  entrypoint   and  entrypoint  will end the monitoring of  waitFiles  back pause  waiting to listen to the   tekton run 0 out beforestepexit  file      Exiting before step breakpoint   entrypoint  listening   tekton run  out beforestepexit  or   tekton run  out beforestepexit err  to decide whether to proceed this step   out beforestepexit  means continue with step   out beforestepexit err  means do not continue with the step