consul Controller Testing 1 Unit Tests These should live alongside the controller and utilize mocks and the controller TestController Where possible split out controller functionality so that other functions can be independently tested 3 Container based integration tests These tests live along with our other container based integration tests They utilize a full multi node cluster and sometimes client agents There are 3 types of tests that can be created here 2 Lightweight integration tests These should live in an internal api group api group test package These tests utilize the in memory resource service and the standard controller manager There are two types of tests that should be created Lifecycle Integration Tests These go step by step to modify resources and check what the controller did They are meant to go through the lifecycle of resources and how they are reconciled Verifications are typically intermingled with resource updates For every controller we want to enable 3 types of testing One Shot Integration Tests These tests publish a bunch of resources and then perform all the verifications These mainly are focused on the controller eventually converging given all the resources thrown at it and aren t as concerned with any intermediate states resources go through Lifecycle Integration Tests These are the same as for the lighweight integration tests

Answer:
# Controller Testing

For every controller we want to enable 3 types of testing.

1. Unit Tests - These should live alongside the controller and utilize mocks and the controller.TestController. Where possible split out controller functionality so that other functions can be independently tested.
2. Lightweight integration tests - These should live in an internal/<api group>/<api group>test package. These tests utilize the in-memory resource service and the standard controller manager. There are two types of tests that should be created.
	* Lifecycle Integration Tests - These go step by step to modify resources and check what the controller did. They are meant to go through the lifecycle of resources and how they are reconciled. Verifications are typically intermingled with resource updates.
	* One-Shot Integration Tests - These tests publish a bunch of resources and then perform all the verifications. These mainly are focused on the controller eventually converging given all the resources thrown at it and aren't as concerned with any intermediate states resources go through.
3. Container based integration tests - These tests live along with our other container based integration tests. They utilize a full multi-node cluster (and sometimes client agents). There are 3 types of tests that can be created here:
   * Lifecycle Integration Tests - These are the same as for the lighweight integration tests.
	* One-shot IntegrationTests - These are the same as for the lightweight integration tests.
	* Upgrade Tests - These are a special form of One-shot Integration tests where the cluster is brought up with some original version, data is pushed in, an upgrade is done and then we verify the consistency of the data post-upgrade.

Between the lightweight and container based integration tests there is a lot of duplication in what is being tested. For this reason these integration test bodies should be defined as exported functions within the apigroups test package. The container based tests can then import those packages and invoke the same functionality with minimal overhead.

For one-shot integration tests, functions to do the resource publishing should be split from functions to perform the verifications. This allows upgrade tests to publish the resources once pre-upgrade and then validate that their correctness post-upgrade without requiring rewriting them.

Sometimes it may also be a good idea to export functions in the test packages for running a specific controllers integration tests. This is a good idea when the controller will use a different version of a dependency in Consul Enterprise to allow for the enterprise implementations package to invoke the integration tests after setting up the controller with its injected dependency.

## Unit Test Template

These tests live alongside controller source.

```go
package foo

import (
	"testing"

	"github.com/stretchr/testif/mock"
	"github.com/stretchr/testif/require"
	"github.com/stretchr/testif/suite"
)

func TestReconcile(t *testing.T) {
	rtest.RunWithTenancies(func(tenancy *pbresource.Tenancy) {
		suite.Run(t, &reconcileSuite{tenancy: tenancy})
	})
}

type reconcileSuite struct {
	suite.Suite

	tenancy *pbresource.Tenancy

	ctx context.Context
	ctl *controller.TestController
	client *rtest.Client

	// Mock objects needed for testing
}

func (suite *reconcileSuite) SetupTest() {
	suite.ctx = testutil.TestContext(suite.T())

	// Alternatively it is sometimes useful to use a mock resource service. For that
	// you can use github.com/hashicorp/consul/grpcmocks.NewResourceServiceClient
	// to create the client.
	client := svctest.NewResourceServiceBuilder().
		// register this API groups types. Also register any other
		// types this controller depends on.
		WithRegisterFns(types.Register).
		WithTenancies(suite.tenancy).
		Run(suite.T())

	// Build any mock objects or other dependencies of the controller here.

	// Build the TestController
	suite.ctl = controller.NewTestController(Controller(), client)
	suite.client = rtest.NewClient(suite.ctl.Runtime().Client)
}

// Implement tests on the suite as needed.
func (suite *reconcileSuite) TestSomething() {
	// Setup Mock expectations

	// Push resources into the resource service as needed.

	// Issue the Reconcile call
	suite.ctl.Reconcile(suite.ctx, controller.Request{})
}
```

## Integration Testing Templates

These tests should live in internal/<api group>/<api group>test. For these examples, assume the API group under test is named `foo` and the latest API group version is v2.

### `run_test.go`

This file is how `go test` knows to execute the tests. These integration tests should
be executed against an in-memory resource service with the standard controller manager.

```go
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: BUSL-1.1

package footest

import (
	"testing"

	"github.com/hashicorp/consul/internal/foo"
	"github.com/hashicorp/consul/internal/controller/controllertest"
	"github.com/hashicorp/consul/internal/resource/reaper"
	rtest "github.com/hashicorp/consul/internal/resource/resourcetest"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

var (
	// This makes the CLI options available to control timing delays of requests. The
	// randomized timings helps to build confidence that regardless of resources writes
	// occurring in quick succession, the controller under test will eventually converge
	// on its steady state.
	clientOpts = rtest.ConfigureTestCLIFlags()
)

func runInMemResourceServiceAndControllers(t *testing.T) pbresource.ResourceServiceClient {
	t.Helper()

	return controllertest.NewControllerTestBuilder().
		// Register your types for the API group and any others that these tests will depend on
		WithResourceRegisterFns(types.Register).
		WithControllerRegisterFns(
			reaper.RegisterControllers,
			foo.RegisterControllers,
		).Run(t)
}

// The basic integration test should operate mostly in a one-shot manner where resources
// are published and then verifications are performed.
func TestControllers_Integration(t *testing.T) {
	client := runInMemResourceServiceAndControllers(t)
	RunFooV2IntegrationTest(t, client, clientOpts.ClientOptions(t)...)
}

// The lifecycle integration test is typically more complex and deals with changing
// some values over time to cause the controllers to do something differently.
func TestControllers_Lifecycle(t *testing.T) {
	client := runInMemResourceServiceAndControllers(t)
	RunFooV2LifecycleTest(t, client, clientOpts.ClientOptions(t)...)
}

```

### `test_integration_v2.go`

```go
package footest

import (
	"embed"
	"fmt"
	"testing"

	rtest "github.com/hashicorp/consul/internal/resource/resourcetest"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

var (
	//go:embed integration_test_data
	testData embed.FS
)

// Execute the full integration test
func RunFooV2IntegrationTest(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) {
	t.Helper

	PublishFooV2IntegrationTestData(t, client, opts...)
	VerifyFooV2IntegrationTestResults(t, client)
}

// PublishFooV2IntegrationTestData publishes all the data that needs to exist in the resource service
// for the controllers to converge on the desired state.
func PublishFooV2IntegrationTestData(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) {
	t.Helper()

	c := rtest.NewClient(client, opts...)

	// Publishing resources manually is an option but alternatively you can store the resources on disk
	// and use go:embed declarations to embed the whole test data filesystem into the test binary.
	resources := rtest.ParseResourcesFromFilesystem(t, testData, "integration_test_data/v2")
	c.PublishResources(t, resources)
}

func VerifyFooV2IntegrationTestResults(t *testing.T, client pbresource.ResourceServiceClient) {
	t.Helper()

	c := rtest.NewClient(client)

	// Perform verifications here. All verifications should be retryable except in very exceptional circumstances.
	// This could be in a retry.Run block or could be retryed by using one of the WaitFor* methods on the rtest.Client.
	// Having them be retryable will prevent flakes especially when the verifications are run in the context of
	// a multi-server cluster where a raft follower hasn't yet observed some change.
}
```

### `test_lifecycle_v2.go`

```go
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: BUSL-1.1

package footest

import (
	"testing"

	rtest "github.com/hashicorp/consul/internal/resource/resourcetest"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

func RunFooV2LifecycleIntegrationTest(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) {
	t.Helper()

	// execute tests.
}
```

Controller Testing  For every controller we want to enable 3 types of testing   1  Unit Tests   These should live alongside the controller and utilize mocks and the controller TestController  Where possible split out controller functionality so that other functions can be independently tested  2  Lightweight integration tests   These should live in an internal  api group   api group test package  These tests utilize the in memory resource service and the standard controller manager  There are two types of tests that should be created     Lifecycle Integration Tests   These go step by step to modify resources and check what the controller did  They are meant to go through the lifecycle of resources and how they are reconciled  Verifications are typically intermingled with resource updates     One Shot Integration Tests   These tests publish a bunch of resources and then perform all the verifications  These mainly are focused on the controller eventually converging given all the resources thrown at it and aren t as concerned with any intermediate states resources go through  3  Container based integration tests   These tests live along with our other container based integration tests  They utilize a full multi node cluster  and sometimes client agents   There are 3 types of tests that can be created here       Lifecycle Integration Tests   These are the same as for the lighweight integration tests     One shot IntegrationTests   These are the same as for the lightweight integration tests     Upgrade Tests   These are a special form of One shot Integration tests where the cluster is brought up with some original version  data is pushed in  an upgrade is done and then we verify the consistency of the data post upgrade      Between the lightweight and container based integration tests there is a lot of duplication in what is being tested  For this reason these integration test bodies should be defined as exported functions within the apigroups test package  The container based tests can then import those packages and invoke the same functionality with minimal overhead   For one shot integration tests  functions to do the resource publishing should be split from functions to perform the verifications  This allows upgrade tests to publish the resources once pre upgrade and then validate that their correctness post upgrade without requiring rewriting them   Sometimes it may also be a good idea to export functions in the test packages for running a specific controllers integration tests  This is a good idea when the controller will use a different version of a dependency in Consul Enterprise to allow for the enterprise implementations package to invoke the integration tests after setting up the controller with its injected dependency      Unit Test Template  These tests live alongside controller source      go package foo  import     testing      github com stretchr testif mock    github com stretchr testif require    github com stretchr testif suite     func TestReconcile t  testing T     rtest RunWithTenancies func tenancy  pbresource Tenancy      suite Run t   reconcileSuite tenancy  tenancy          type reconcileSuite struct    suite Suite    tenancy  pbresource Tenancy    ctx context Context  ctl  controller TestController  client  rtest Client       Mock objects needed for testing    func  suite  reconcileSuite  SetupTest      suite ctx   testutil TestContext suite T         Alternatively it is sometimes useful to use a mock resource service  For that     you can use github com hashicorp consul grpcmocks NewResourceServiceClient     to create the client    client    svctest NewResourceServiceBuilder         register this API groups types  Also register any other      types this controller depends on    WithRegisterFns types Register     WithTenancies suite tenancy     Run suite T           Build any mock objects or other dependencies of the controller here       Build the TestController  suite ctl   controller NewTestController Controller    client   suite client   rtest NewClient suite ctl Runtime   Client        Implement tests on the suite as needed  func  suite  reconcileSuite  TestSomething         Setup Mock expectations       Push resources into the resource service as needed        Issue the Reconcile call  suite ctl Reconcile suite ctx  controller Request              Integration Testing Templates  These tests should live in internal  api group   api group test  For these examples  assume the API group under test is named  foo  and the latest API group version is v2        run test go   This file is how  go test  knows to execute the tests  These integration tests should be executed against an in memory resource service with the standard controller manager      go    Copyright  c  HashiCorp  Inc     SPDX License Identifier  BUSL 1 1  package footest  import     testing     github com hashicorp consul internal foo    github com hashicorp consul internal controller controllertest    github com hashicorp consul internal resource reaper   rtest  github com hashicorp consul internal resource resourcetest    github com hashicorp consul proto public pbresource     var       This makes the CLI options available to control timing delays of requests  The     randomized timings helps to build confidence that regardless of resources writes     occurring in quick succession  the controller under test will eventually converge     on its steady state   clientOpts   rtest ConfigureTestCLIFlags      func runInMemResourceServiceAndControllers t  testing T  pbresource ResourceServiceClient    t Helper     return controllertest NewControllerTestBuilder         Register your types for the API group and any others that these tests will depend on   WithResourceRegisterFns types Register     WithControllerRegisterFns     reaper RegisterControllers     foo RegisterControllers      Run t        The basic integration test should operate mostly in a one shot manner where resources    are published and then verifications are performed   func TestControllers Integration t  testing T     client    runInMemResourceServiceAndControllers t   RunFooV2IntegrationTest t  client  clientOpts ClientOptions t            The lifecycle integration test is typically more complex and deals with changing    some values over time to cause the controllers to do something differently  func TestControllers Lifecycle t  testing T     client    runInMemResourceServiceAndControllers t   RunFooV2LifecycleTest t  client  clientOpts ClientOptions t                   test integration v2 go       go package footest  import     embed    fmt    testing    rtest  github com hashicorp consul internal resource resourcetest    github com hashicorp consul proto public pbresource     var      go embed integration test data  testData embed FS       Execute the full integration test func RunFooV2IntegrationTest t  testing T  client pbresource ResourceServiceClient  opts    rtest ClientOption     t Helper    PublishFooV2IntegrationTestData t  client  opts      VerifyFooV2IntegrationTestResults t  client        PublishFooV2IntegrationTestData publishes all the data that needs to exist in the resource service    for the controllers to converge on the desired state  func PublishFooV2IntegrationTestData t  testing T  client pbresource ResourceServiceClient  opts    rtest ClientOption     t Helper     c    rtest NewClient client  opts          Publishing resources manually is an option but alternatively you can store the resources on disk     and use go embed declarations to embed the whole test data filesystem into the test binary   resources    rtest ParseResourcesFromFilesystem t  testData   integration test data v2    c PublishResources t  resources     func VerifyFooV2IntegrationTestResults t  testing T  client pbresource ResourceServiceClient     t Helper      c    rtest NewClient client        Perform verifications here  All verifications should be retryable except in very exceptional circumstances      This could be in a retry Run block or could be retryed by using one of the WaitFor  methods on the rtest Client      Having them be retryable will prevent flakes especially when the verifications are run in the context of     a multi server cluster where a raft follower hasn t yet observed some change              test lifecycle v2 go      go    Copyright  c  HashiCorp  Inc     SPDX License Identifier  BUSL 1 1  package footest  import     testing    rtest  github com hashicorp consul internal resource resourcetest    github com hashicorp consul proto public pbresource     func RunFooV2LifecycleIntegrationTest t  testing T  client pbresource ResourceServiceClient  opts    rtest ClientOption     t Helper         execute tests


consul Note out the Consul 1 16 introduced a set of generic APIs for managing resources and a controller runtime for building functionality on top of them Overview generic APIs proto public pbresource resource proto Looking for guidance on adding new resources and controllers to Consul Check

Answer:
# Overview

> **Note**
> Looking for guidance on adding new resources and controllers to Consul? Check
> out the [developer guide](guide.md).

Consul 1.16 introduced a set of [generic APIs] for managing resources, and a
[controller runtime] for building functionality on top of them.

[generic APIs]: ../../../proto-public/pbresource/resource.proto
[controller runtime]: ../../../internal/controller

Previously, adding features to Consul involved making changes at every layer of
the stack, including: HTTP handlers, RPC handlers, MemDB tables, Raft
operations, and CLI commands.

This architecture made sense when the product was maintained by a small core
group who could keep the entire system in their heads, but presented significant
collaboration, ownership, and onboarding challenges when our contributor base
expanded to many engineers, across several teams, and the product grew in
complexity.

In the new model, teams can work with much greater autonomy by building on top
of a shared platform and own their resource types and controllers.

## Architecture Overview

![architecture diagram](architecture-overview.png)

[source](https://whimsical.com/state-store-v2-UKE6SaEPXNc4UrZBrZj4Kg)

Our resource-oriented architecture comprises the following components:

#### Resource Service

[Resource Service](../../../proto-public/pbresource/resource.proto) is a gRPC
service that contains the shared logic for creating, reading, updating,
deleting, and watching resources. It will be consumed by controllers, our
Kubernetes integration, the CLI, and mapped to an HTTP+JSON API.

#### Type Registry

[Type Registry](../../../internal/resource/registry.go) is where teams register
their resource types, along with hooks for performing structural validation,
authorization, etc.

#### Storage Backend

[Storage Backend](../../../internal/storage/storage.go) is an abstraction over
low-level storage primitives. Today, there are two implementations (Raft and
an in-memory backend for tests) but in the future, we envisage external storage
systems such as the Kubernetes API or an RDBMS could be supported which would
reduce operational complexity for our customers.

#### Controllers

[Controllers](../../../internal/controller/api.go) implement Consul's business
logic using asynchronous control loops that respond to changes in resources.
Please see [Controller docs](controllers.md) for more details about controllers

## Raft Storage Backend

Our [Raft Storage Backend](../../../internal/storage/raft/backend.go) integrates
with the existing Raft machinery (e.g. FSM) used by the [old state store]. It
also transparently forwards writes and strongly consistent reads to the leader
over gRPC.

There's quite a lot going on here, so to dig into the details, let's take a look
at how a write operation is handled.

[old state store]: ../persistence/

![raft storage backend diagram](raft-backend.png)

[source](https://whimsical.com/state-store-v2-UKE6SaEPXNc4UrZBrZj4Kg)

#### Steps 1 & 2

User calls the resource service's `Write` endpoint, on a Raft follower, which
in-turn calls the storage backend's `WriteCAS` method.

#### Steps 3 & 4

The storage backend determines that the current server is a Raft follower, and
forwards the operation to the leader via a gRPC [forwarding service] listening
on the multiplexed RPC port ([`ports.server`]).

[forwarding service]: ../../../proto/private/pbstorage/raft.proto
[`ports.server`]: https://developer.hashicorp.com/consul/docs/agent/config/config-files#server_rpc_port

#### Step 5

The leader's storage backend serializes the operation to protobuf and applies it
to the Raft log. As we need to share the Raft log with the old state store, we go
through the [`consul.raftHandle`](../../../agent/consul/raft_handle.go) and
[`consul.Server`](../../agent/consul/server/server.go) which applies a msgpack
envelope and type byte prefix.

#### Step 6

Raft consensus happens! Once the log has been committed, it is applied to the
[FSM](../../../agent/consul/fsm/fsm.go) which calls the storage backend's `Apply`
method to apply the protobuf-encoded operation to the [`inmem.Store`].

[`inmem.Store`]: ../../../internal/storage/inmem/store.go

#### Steps 7, 8, 9

At this point, the operation is complete. The forwarding service returns a
successful response, as does the follower's storage backend, and the user
gets a successful response too.

#### Steps 10 & 11

Asynchronously, the log is replicated to followers and applied to their storage
backends.

Overview      Note       Looking for guidance on adding new resources and controllers to Consul  Check   out the  developer guide  guide md    Consul 1 16 introduced a set of  generic APIs  for managing resources  and a  controller runtime  for building functionality on top of them    generic APIs            proto public pbresource resource proto  controller runtime            internal controller  Previously  adding features to Consul involved making changes at every layer of the stack  including  HTTP handlers  RPC handlers  MemDB tables  Raft operations  and CLI commands   This architecture made sense when the product was maintained by a small core group who could keep the entire system in their heads  but presented significant collaboration  ownership  and onboarding challenges when our contributor base expanded to many engineers  across several teams  and the product grew in complexity   In the new model  teams can work with much greater autonomy by building on top of a shared platform and own their resource types and controllers      Architecture Overview    architecture diagram  architecture overview png    sup  source  https   whimsical com state store v2 UKE6SaEPXNc4UrZBrZj4Kg   sup   Our resource oriented architecture comprises the following components        Resource Service   Resource Service           proto public pbresource resource proto  is a gRPC service that contains the shared logic for creating  reading  updating  deleting  and watching resources  It will be consumed by controllers  our Kubernetes integration  the CLI  and mapped to an HTTP JSON API        Type Registry   Type Registry           internal resource registry go  is where teams register their resource types  along with hooks for performing structural validation  authorization  etc        Storage Backend   Storage Backend           internal storage storage go  is an abstraction over low level storage primitives  Today  there are two implementations  Raft and an in memory backend for tests  but in the future  we envisage external storage systems such as the Kubernetes API or an RDBMS could be supported which would reduce operational complexity for our customers        Controllers   Controllers           internal controller api go  implement Consul s business logic using asynchronous control loops that respond to changes in resources  Please see  Controller docs  controllers md  for more details about controllers     Raft Storage Backend  Our  Raft Storage Backend           internal storage raft backend go  integrates with the existing Raft machinery  e g  FSM  used by the  old state store   It also transparently forwards writes and strongly consistent reads to the leader over gRPC   There s quite a lot going on here  so to dig into the details  let s take a look at how a write operation is handled    old state store      persistence     raft storage backend diagram  raft backend png    sup  source  https   whimsical com state store v2 UKE6SaEPXNc4UrZBrZj4Kg   sup        Steps 1   2  User calls the resource service s  Write  endpoint  on a Raft follower  which in turn calls the storage backend s  WriteCAS  method        Steps 3   4  The storage backend determines that the current server is a Raft follower  and forwards the operation to the leader via a gRPC  forwarding service  listening on the multiplexed RPC port    ports server       forwarding service            proto private pbstorage raft proto   ports server    https   developer hashicorp com consul docs agent config config files server rpc port       Step 5  The leader s storage backend serializes the operation to protobuf and applies it to the Raft log  As we need to share the Raft log with the old state store  we go through the   consul raftHandle            agent consul raft handle go  and   consul Server         agent consul server server go  which applies a msgpack envelope and type byte prefix        Step 6  Raft consensus happens  Once the log has been committed  it is applied to the  FSM           agent consul fsm fsm go  which calls the storage backend s  Apply  method to apply the protobuf encoded operation to the   inmem Store       inmem Store             internal storage inmem store go       Steps 7  8  9  At this point  the operation is complete  The forwarding service returns a successful response  as does the follower s storage backend  and the user gets a successful response too        Steps 10   11  Asynchronously  the log is replicated to followers and applied to their storage backends


consul Resource Schema Consul Adding a new resource type begins with defining the object schema as a protobuf message in the appropriate package under This is a whistle stop tour through adding a new resource type and controller to Resource and Controller Developer Guide

Answer:
# Resource and Controller Developer Guide

This is a whistle-stop tour through adding a new resource type and controller to
Consul 🚂

## Resource Schema

Adding a new resource type begins with defining the object schema as a protobuf
message, in the appropriate package under [`proto-public`](../../../proto-public).

```shell
$ mkdir proto-public/pbfoo/v1alpha1
```

```proto
// proto-public/pbfoo/v1alpha1/foo.proto
syntax = "proto3";

import "pbresource/resource.proto";
import "pbresource/annotations.proto";

package hashicorp.consul.foo.v1alpha1;

message Bar {
  option (hashicorp.consul.resource.spec) = {scope: SCOPE_NAMESPACE};

  string baz = 1;
  hashicorp.consul.resource.ID qux = 2;
}
```

```shell
$ make proto
```

Next, we must add our resource type to the registry. At this point, it's useful
to add a package (e.g. under [`internal`](../../../internal)) to contain the logic
associated with this resource type.

The convention is to have this package export variables for its type identifiers
along with a method for registering its types:

```Go
// internal/foo/types.go
package foo

import (
	"github.com/hashicorp/consul/internal/resource"
	pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:  pbv1alpha1.BarType,
		Scope: resource.ScopePartition,
		Proto: &pbv1alpha1.Bar{},
	})
}
```
Note that Scope reference the scope of the new resource, `resource.ScopePartition`
mean that resource will be at the partition level and have no namespace, while `resource.ScopeNamespace` mean it will have both a namespace
and a partition.

Update the `NewTypeRegistry` method in [`type_registry.go`] to call your
package's type registration method:

[`type_registry.go`]: ../../../agent/consul/type_registry.go

```Go
import (
	// …
	"github.com/hashicorp/consul/internal/foo"
	// …
)

func NewTypeRegistry() resource.Registry {
	// …
    foo.RegisterTypes(registry)
	// …
}
```

That should be all you need to start using your new resource type. Test it out
by starting an agent in dev mode:

```shell
$ make dev
$ consul agent -dev
```

You can now use [grpcurl](https://github.com/fullstorydev/grpcurl) to interact
with the [resource service](../../../proto-public/pbresource/resource.proto):

```shell
$ grpcurl -d @ \
  -plaintext \
  -protoset pkg/consul.protoset \
  127.0.0.1:8502 \
  hashicorp.consul.resource.ResourceService.Write \
<<EOF
  {
    "resource": {
      "id": {
        "type": {
          "group": "foo",
          "group_version": "v1alpha1",
          "kind": "bar"
        },
        "tenancy": {
          "partition": "default",
          "namespace": "default"
        }
      },
      "data": {
        "@type": "types.googleapis.com/hashicorp.consul.foo.v1alpha1.Bar",
        "baz": "Hello World"
      }
    }
  }
EOF
```

## Validation

Broadly, there are two kinds of validation you might want to perform against
your resources:

- **Structural** validation ensures the user's input is well-formed, for
  example: checking that a required field is provided, or that a port is within
  an acceptable range.
- **Semantic** validation ensures that the resource makes sense in the context
  of *other* resources, for example: checking that an L7 intention is not
  targeting an L4 service.

Structural validation should be done up-front, before the resource is admitted,
using a validation hook provided in the type registration:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:     pbv1alpha1.BarType,
		Proto:    &pbv1alpha1.Bar{},
		Scope:    resource.ScopeNamespace,
		Validate: validateBar,
	})
}

func validateBar(res *pbresource.Resource) error {
	var bar pbv1alpha1.Bar
	if err := res.Data.UnmarshalTo(&bar); err != nil {
		return resource.NewErrDataParse(&bar, err)
	}
	if bar.Baz == "" {
		return resource.ErrInvalidField{
			Name:    "baz",
			Wrapped: resource.ErrMissing,
		}
	}
	return nil
}
```

Semantic validation should be done asynchronously, after the resource is
written, by controllers ([covered below](#controllers)).

## Authorization

You can control how operations on your resource type are authorized by providing
a set of ACL hooks:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:  pbv1alpha1.BarType,
		Proto: &pbv1alpha1.Bar{},
		Scope: resource.ScopeNamespace,
		ACLs: &resource.ACLHooks{,
			Read:  authzReadBar,
			Write: authzWriteBar,
			List:  authzListBar,
		},
	})
}

func authzReadBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext, id *pbresource.ID,  _ *pbresource.Resource) error {
	return authz.ToAllowAuthorizer().
		BarReadAllowed(id.Name, authzContext)
}

func authzWriteBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext, res *pbresource.Resource) error {
	return authz.ToAllowAuthorizer().
		BarWriteAllowed(res.ID().Name, authzContext)
}

func authzListBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext) error {
	return authz.ToAllowAuthorizer().
		BarListAllowed(authzContext)
}
```

If you do not provide ACL hooks, `operator:read` and `operator:write`
permissions will be required.

## Mutation

Sometimes, it's necessary to modify resources before they're persisted. For
example, to set sensible default values or normalize user input. You can do this
by providing a mutation hook:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:   pbv1alpha1.BarType,
		Proto:  &pbv1alpha1.Bar{},
		Scope:  resource.ScopeNamespace,
		Mutate: mutateBar,
	})
}

func mutateBar(res *pbresource.Resource) error {
	var bar pbv1alpha1.Bar
	if err := res.Data.UnmarshalTo(&bar); err != nil {
		return resource.NewErrDataParse(&bar, err)
	}
	bar.Baz = strings.ToLower(bar.Baz)
	return res.Data.MarshalFrom(&bar)
}
```

## Controllers

Controllers are where the business logic of your resources will live. They're
asynchronous [reconciliation loops] that "wake up" whenever a resource is
modified to validate and realize the changes.

You can create a new controller using the [builder API]. Start by identifying
the resource type you want this controller to manage, and provide a reconciler
that will be called whenever a resource of that type is changed.

```Go
package foo

import (
	"context"

	"github.com/hashicorp/consul/internal/controller"
	pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithReconciler(barReconciler{})
}

type barReconciler struct{}

func (barReconciler) Reconcile(ctx context.Context, rt controller.Runtime, req controller.Request) error {
	rsp, err := rt.Client.Read(ctx, &pbresource.ReadRequest{Id: req.ID})
	switch {
	case status.Code(err) == codes.NotFound:
		return nil
	case err != nil:
		return err
	}

	var bar pbv1alpha1.Bar
	if err := rsp.Resource.Data.UnmarshalTo(&bar); err != nil {
		return err
	}
	rt.Logger.Debug("Hello from bar reconciler!", "baz", bar.Baz)

	return nil
}
```

[reconciliation loops]: https://www.oreilly.com/library/view/97-things-every/9781492050896/ch73.html
[builder API]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#Controller

Next, register your controller with the controller manager. Another common
pattern is to have your package expose a method for registering controllers,
which is called from `registerControllers` in [`server.go`].

[`server.go`]: ../../../agent/consul/server.go

```Go
package foo

func RegisterControllers(mgr *controller.Manager) {
	mgr.Register(barController())
}
```

```Go
package consul

func (s *Server) registerControllers() {
	// …
	foo.RegisterControllers(s.controllerManager)
	// …
}
```

### Retries

By default, if your reconciler returns an error, it will be retried with
exponential backoff. While this is correct in most circumstances, you can
override it by returning [`RequeueAfter`] or [`RequeueNow`].

[`RequeueAfter`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#RequeueAfter
[`RequeueNow`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#RequeueNow

```Go
func (barReconciler) Reconcile(context.Context, controller.Runtime, controller.Request) error {
	if time.Now().Hour() < 9 {
		return controller.RequeueAfter(1 * time.Hour)
	}
	return nil
}
```

### Status

Controllers can communicate the result of reconciling resource changes (e.g.
surfacing semantic validation issues) with users and other controllers by
updating the resource's status using the `WriteStatus` method.

Each resource can have multiple statuses, typically one per controller,
identified by a string key. Statuses are composed of a set of conditions, which
represent discreet observations about the resource in relation to the current
state of the system.

That all sounds a little abstract, so let's take a look at an example.

```Go
client.WriteStatus(ctx, &pbresource.WriteStatusRequest{
	Id:     res.Id,
	Key:    "consul.io/bar",
	Status: &pbresource.Status{
		ObservedGeneration: res.Generation,
		Conditions: []*pbresource.Condition{
			{
				Type:    "Healthy",
				State:   pbresource.Condition_STATE_TRUE,
				Reason:  "OK",
				Message: "All checks are passing",
			},
			{
				Type:    "ResolvedRefs",
				State:   pbresource.Condition_STATE_FALSE,
				Reason:  "INVALID_REFERENCE",
				Message: "Bar contained an invalid reference to qux",
				Resource: resource.Reference(bar.Qux, ""),
			},
		},
	},
})
```

In the previous example, the controller makes two observations about the
current state of the resource:

1. That it's "healthy" (whatever that means in this hypothetical scenario)
1. That it contains a reference that couldn't be resolved

The `Type` and `Reason` should be simple, machine-readable, strings, but there
aren't any strict rules about what are acceptable values. Over time, we
anticipate that common values will emerge that we'll standardize on for
consistency.

`Message` should be a human-readable explanation of the condition.

> **Warning**
> Writing a status to the resource will cause it to be re-reconciled. To avoid
> infinite loops, we recommend dirty checking the status before writing it with
> [`resource.EqualStatus`].

[`resource.EqualStatus`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/resource#EqualStatus

### Watching Other Resources

In addition to watching their "managed" resources, controllers can also watch
resources of different, related, types. For example, the service endpoints
controller also watches workloads and services.

```Go
func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithWatch(pbv1alpha1.BazType, controller.MapOwner)
		WithReconciler(barReconciler{})
}
```

The second argument to `WithWatch` is a [dependency mapper] function. Whenever a
resource of the watched type is modified, the dependency mapper will be called
to determine which of the controller's managed resources need to be reconciled.

[`dependency.MapOwner`] is a convenience function which causes the watched
resource's [owner](#ownership--cascading-deletion) to be reconciled.

[dependency mapper]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#DependencyMapper
[`dependency.MapOwner`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller/dependency#MapOwner

### Placement

By default, only a single, leader-elected, replica of each controller will run
within a cluster. Sometimes it's necessary to override this, for example when
you want to run a copy of the controller on each server (e.g. to apply some
configuration to the server whenever it changes). You can do this by changing
the controller's placement.

```Go
func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithPlacement(controller.PlacementEachServer)
		WithReconciler(barReconciler{})
}
```

> **Warning**
> Controllers placed with [`controller.PlacementEachServer`] generally shouldn't
> modify resources (as it could lead to race conditions).

[`controller.PlacementEachServer`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#PlacementEachServer

### Initializer

If your controller needs to execute setup steps when the controller
first starts and before any resources are reconciled, you can add an
Initializer.

If the controller has an Initializer, it will not start unless the
Initialize method is successful. The controller does not have retry
logic for the initialize method specifically, but the controller
is restarted on error. When restarted, the controller will attempt
to execute the initialization again.

The example below has the controller creating a default resource as
part of initialization.

```Go
package foo

import (
 "context"

 "github.com/hashicorp/consul/internal/controller"
 pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1"
 "github.com/hashicorp/consul/proto-public/pbresource"
)

func barController() controller.Controller {
 return controller.ForType(pbv1alpha1.BarType).
  WithReconciler(barReconciler{}).
  WithInitializer(barInitializer{})
}

type barInitializer struct{}

func (barInitializer) Initialize(ctx context.Context, rt controller.Runtime) error {
  _, err := rt.Client.Write(ctx,
    &pbresource.WriteRequest{
    Resource: &pbresource.Resource{
     Id: &pbresource.ID{
      Name: "default",
      Type: pbv1alpha1.BarType,
     },
    },
   },
  )
  if err != nil {
   return err
  }

  return nil
}
```

### Finalizer

A finalizer allows a controller to execute teardown logic before a
resource is deleted. This can be useful to perform cleanup or block
deletion until certain conditions are met.

Finalizers are encoded as keys within a resource's metadata map. It
is the responsibility of each controller that adds a finalizer to a
resource to remove the finalizer when it is marked for deletion.
Once a resource has no finalizers present, it is deleted by the
resource service.

When the `Delete` endpoint is called on a resource with one or more
finalizers, the resource is marked for deletion by adding an immutable
`deletionTimestamp` key to the resource's metadata map. The resource is
now effectively frozen and will only accept subsequent `Write`s
that remove finalizers. `WriteStatus` is still allowed.

The `resource` package API can be used to manage finalizers and
check whether a resource has been marked for deletion. You would
typically use this API within the logic of your controller's
`Reconcile` method to either put a finalizer in place or perform
cleanup and then remove a finalizer. Don't forget to `Write` your
changes once you add or remove finalizers.

```Go
package resource

// IsMarkedForDeletion returns true if a resource has been marked for deletion,
// false otherwise.
func IsMarkedForDeletion(res *pbresource.Resource) bool { ... }

// HasFinalizers returns true if a resource has one or more finalizers, false otherwise.
func HasFinalizers(res *pbresource.Resource) bool { ... }

// HasFinalizer returns true if a resource has a given finalizer, false otherwise.
func HasFinalizer(res *pbresource.Resource, finalizer string) bool { ... }

// AddFinalizer adds a finalizer to the given resource.
func AddFinalizer(res *pbresource.Resource, finalizer string) { ... }

// RemoveFinalizer removes a finalizer from the given resource.
func RemoveFinalizer(res *pbresource.Resource, finalizer string) { ... }

// GetFinalizers returns the set of finalizers for the given resource.
func GetFinalizers(res *pbresource.Resource) mapset.Set[string] { ... }
```

Example flow in a controller's `Reconcile` method
```Go
const finalizer = "consul.io/bar-finalizer"

func (barReconciler) Reconcile(ctx context.Context, rt controller.Runtime, req controller.Request) error {
	...
	// Check if resource is marked for deletion. If yes, perform cleanup, remove finalizer, and Write the resource
	if resource.IsMarkedForDeletion(res) {
		// Perform some cleanup...
		return EnsureFinalizerRemoved(ctx, rt, res, finalizer)
	}

	// Check if resource has finalizer. If not, add it and Write the resource
	if err := EnsureHasFinalizer(ctx, rt, res, finalizer); err != nil {
		return err
	}
}
```

## Ownership & Cascading Deletion

The resource service implements a lightweight `1:N` ownership model where, on
creation, you can mark a resource as being "owned" by another resource. When the
owner is deleted, the owned resource will be deleted too.

```Go
client.Write(ctx, &pbresource.WriteRequest{
	Resource: &pbresource.Resource{,
		Owner: ownerID,
		// …
	},
})
```

## Testing

Now that you have created your controller its time to test it. The types of tests each controller should have and boiler plat for test files is documented [here](./testing.md)

Resource and Controller Developer Guide  This is a whistle stop tour through adding a new resource type and controller to Consul       Resource Schema  Adding a new resource type begins with defining the object schema as a protobuf message  in the appropriate package under   proto public            proto public       shell   mkdir proto public pbfoo v1alpha1         proto    proto public pbfoo v1alpha1 foo proto syntax    proto3    import  pbresource resource proto   import  pbresource annotations proto    package hashicorp consul foo v1alpha1   message Bar     option  hashicorp consul resource spec     scope  SCOPE NAMESPACE        string baz   1    hashicorp consul resource ID qux   2            shell   make proto      Next  we must add our resource type to the registry  At this point  it s useful to add a package  e g  under   internal            internal   to contain the logic associated with this resource type   The convention is to have this package export variables for its type identifiers along with a method for registering its types      Go    internal foo types go package foo  import     github com hashicorp consul internal resource   pbv1alpha1  github com hashicorp consul proto public pbfoo v1alpha1    github com hashicorp consul proto public pbresource     func RegisterTypes r resource Registry     r Register resource Registration    Type   pbv1alpha1 BarType     Scope  resource ScopePartition    Proto   pbv1alpha1 Bar              Note that Scope reference the scope of the new resource   resource ScopePartition   mean that resource will be at the partition level and have no namespace  while  resource ScopeNamespace  mean it will have both a namespace  and a partition   Update the  NewTypeRegistry  method in   type registry go   to call your package s type registration method     type registry go             agent consul type registry go     Go import           github com hashicorp consul internal foo           func NewTypeRegistry   resource Registry             foo RegisterTypes registry               That should be all you need to start using your new resource type  Test it out by starting an agent in dev mode      shell   make dev   consul agent  dev      You can now use  grpcurl  https   github com fullstorydev grpcurl  to interact with the  resource service           proto public pbresource resource proto       shell   grpcurl  d        plaintext      protoset pkg consul protoset     127 0 0 1 8502     hashicorp consul resource ResourceService Write     EOF          resource            id              type                group    foo              group version    v1alpha1              kind    bar                      tenancy                partition    default              namespace    default                            data               type    types googleapis com hashicorp consul foo v1alpha1 Bar            baz    Hello World                    EOF         Validation  Broadly  there are two kinds of validation you might want to perform against your resources       Structural   validation ensures the user s input is well formed  for   example  checking that a required field is provided  or that a port is within   an acceptable range      Semantic   validation ensures that the resource makes sense in the context   of  other  resources  for example  checking that an L7 intention is not   targeting an L4 service   Structural validation should be done up front  before the resource is admitted  using a validation hook provided in the type registration      Go func RegisterTypes r resource Registry     r Register resource Registration    Type      pbv1alpha1 BarType    Proto      pbv1alpha1 Bar       Scope     resource ScopeNamespace    Validate  validateBar         func validateBar res  pbresource Resource  error    var bar pbv1alpha1 Bar  if err    res Data UnmarshalTo  bar   err    nil     return resource NewErrDataParse  bar  err      if bar Baz           return resource ErrInvalidField     Name      baz      Wrapped  resource ErrMissing          return nil        Semantic validation should be done asynchronously  after the resource is written  by controllers   covered below   controllers        Authorization  You can control how operations on your resource type are authorized by providing a set of ACL hooks      Go func RegisterTypes r resource Registry     r Register resource Registration    Type   pbv1alpha1 BarType    Proto   pbv1alpha1 Bar       Scope  resource ScopeNamespace    ACLs   resource ACLHooks      Read   authzReadBar     Write  authzWriteBar     List   authzListBar              func authzReadBar authz acl Authorizer  authzContext  acl AuthorizerContext  id  pbresource ID      pbresource Resource  error    return authz ToAllowAuthorizer      BarReadAllowed id Name  authzContext     func authzWriteBar authz acl Authorizer  authzContext  acl AuthorizerContext  res  pbresource Resource  error    return authz ToAllowAuthorizer      BarWriteAllowed res ID   Name  authzContext     func authzListBar authz acl Authorizer  authzContext  acl AuthorizerContext  error    return authz ToAllowAuthorizer      BarListAllowed authzContext         If you do not provide ACL hooks   operator read  and  operator write  permissions will be required      Mutation  Sometimes  it s necessary to modify resources before they re persisted  For example  to set sensible default values or normalize user input  You can do this by providing a mutation hook      Go func RegisterTypes r resource Registry     r Register resource Registration    Type    pbv1alpha1 BarType    Proto    pbv1alpha1 Bar       Scope   resource ScopeNamespace    Mutate  mutateBar         func mutateBar res  pbresource Resource  error    var bar pbv1alpha1 Bar  if err    res Data UnmarshalTo  bar   err    nil     return resource NewErrDataParse  bar  err      bar Baz   strings ToLower bar Baz   return res Data MarshalFrom  bar            Controllers  Controllers are where the business logic of your resources will live  They re asynchronous  reconciliation loops  that  wake up  whenever a resource is modified to validate and realize the changes   You can create a new controller using the  builder API   Start by identifying the resource type you want this controller to manage  and provide a reconciler that will be called whenever a resource of that type is changed      Go package foo  import     context     github com hashicorp consul internal controller   pbv1alpha1  github com hashicorp consul proto public pbfoo v1alpha1    github com hashicorp consul proto public pbresource     func barController   controller Controller    return controller NewController  bar   pbv1alpha1 BarType     WithReconciler barReconciler       type barReconciler struct    func  barReconciler  Reconcile ctx context Context  rt controller Runtime  req controller Request  error    rsp  err    rt Client Read ctx   pbresource ReadRequest Id  req ID    switch    case status Code err     codes NotFound    return nil  case err    nil    return err      var bar pbv1alpha1 Bar  if err    rsp Resource Data UnmarshalTo  bar   err    nil     return err     rt Logger Debug  Hello from bar reconciler     baz   bar Baz    return nil         reconciliation loops   https   www oreilly com library view 97 things every 9781492050896 ch73 html  builder API   https   pkg go dev github com hashicorp consul internal controller Controller  Next  register your controller with the controller manager  Another common pattern is to have your package expose a method for registering controllers  which is called from  registerControllers  in   server go       server go             agent consul server go     Go package foo  func RegisterControllers mgr  controller Manager     mgr Register barController              Go package consul  func  s  Server  registerControllers            foo RegisterControllers s controllerManager                   Retries  By default  if your reconciler returns an error  it will be retried with exponential backoff  While this is correct in most circumstances  you can override it by returning   RequeueAfter   or   RequeueNow       RequeueAfter    https   pkg go dev github com hashicorp consul internal controller RequeueAfter   RequeueNow    https   pkg go dev github com hashicorp consul internal controller RequeueNow     Go func  barReconciler  Reconcile context Context  controller Runtime  controller Request  error    if time Now   Hour     9     return controller RequeueAfter 1   time Hour      return nil            Status  Controllers can communicate the result of reconciling resource changes  e g  surfacing semantic validation issues  with users and other controllers by updating the resource s status using the  WriteStatus  method   Each resource can have multiple statuses  typically one per controller  identified by a string key  Statuses are composed of a set of conditions  which represent discreet observations about the resource in relation to the current state of the system   That all sounds a little abstract  so let s take a look at an example      Go client WriteStatus ctx   pbresource WriteStatusRequest   Id      res Id   Key      consul io bar    Status   pbresource Status    ObservedGeneration  res Generation    Conditions     pbresource Condition           Type      Healthy       State    pbresource Condition STATE TRUE      Reason    OK       Message   All checks are passing                  Type      ResolvedRefs       State    pbresource Condition STATE FALSE      Reason    INVALID REFERENCE       Message   Bar contained an invalid reference to qux       Resource  resource Reference bar Qux                              In the previous example  the controller makes two observations about the current state of the resource   1  That it s  healthy   whatever that means in this hypothetical scenario  1  That it contains a reference that couldn t be resolved  The  Type  and  Reason  should be simple  machine readable  strings  but there aren t any strict rules about what are acceptable values  Over time  we anticipate that common values will emerge that we ll standardize on for consistency    Message  should be a human readable explanation of the condition       Warning       Writing a status to the resource will cause it to be re reconciled  To avoid   infinite loops  we recommend dirty checking the status before writing it with     resource EqualStatus       resource EqualStatus    https   pkg go dev github com hashicorp consul internal resource EqualStatus      Watching Other Resources  In addition to watching their  managed  resources  controllers can also watch resources of different  related  types  For example  the service endpoints controller also watches workloads and services      Go func barController   controller Controller    return controller NewController  bar   pbv1alpha1 BarType     WithWatch pbv1alpha1 BazType  controller MapOwner    WithReconciler barReconciler           The second argument to  WithWatch  is a  dependency mapper  function  Whenever a resource of the watched type is modified  the dependency mapper will be called to determine which of the controller s managed resources need to be reconciled     dependency MapOwner   is a convenience function which causes the watched resource s  owner   ownership  cascading deletion  to be reconciled    dependency mapper   https   pkg go dev github com hashicorp consul internal controller DependencyMapper   dependency MapOwner    https   pkg go dev github com hashicorp consul internal controller dependency MapOwner      Placement  By default  only a single  leader elected  replica of each controller will run within a cluster  Sometimes it s necessary to override this  for example when you want to run a copy of the controller on each server  e g  to apply some configuration to the server whenever it changes   You can do this by changing the controller s placement      Go func barController   controller Controller    return controller NewController  bar   pbv1alpha1 BarType     WithPlacement controller PlacementEachServer    WithReconciler barReconciler               Warning       Controllers placed with   controller PlacementEachServer   generally shouldn t   modify resources  as it could lead to race conditions      controller PlacementEachServer    https   pkg go dev github com hashicorp consul internal controller PlacementEachServer      Initializer  If your controller needs to execute setup steps when the controller first starts and before any resources are reconciled  you can add an Initializer   If the controller has an Initializer  it will not start unless the Initialize method is successful  The controller does not have retry logic for the initialize method specifically  but the controller is restarted on error  When restarted  the controller will attempt to execute the initialization again   The example below has the controller creating a default resource as part of initialization      Go package foo  import     context     github com hashicorp consul internal controller   pbv1alpha1  github com hashicorp consul proto public pbfoo v1alpha1    github com hashicorp consul proto public pbresource     func barController   controller Controller    return controller ForType pbv1alpha1 BarType     WithReconciler barReconciler       WithInitializer barInitializer       type barInitializer struct    func  barInitializer  Initialize ctx context Context  rt controller Runtime  error        err    rt Client Write ctx       pbresource WriteRequest      Resource   pbresource Resource       Id   pbresource ID        Name   default         Type  pbv1alpha1 BarType                             if err    nil      return err        return nil            Finalizer  A finalizer allows a controller to execute teardown logic before a resource is deleted  This can be useful to perform cleanup or block deletion until certain conditions are met   Finalizers are encoded as keys within a resource s metadata map  It is the responsibility of each controller that adds a finalizer to a resource to remove the finalizer when it is marked for deletion  Once a resource has no finalizers present  it is deleted by the resource service   When the  Delete  endpoint is called on a resource with one or more finalizers  the resource is marked for deletion by adding an immutable  deletionTimestamp  key to the resource s metadata map  The resource is now effectively frozen and will only accept subsequent  Write s that remove finalizers   WriteStatus  is still allowed   The  resource  package API can be used to manage finalizers and check whether a resource has been marked for deletion  You would typically use this API within the logic of your controller s  Reconcile  method to either put a finalizer in place or perform cleanup and then remove a finalizer  Don t forget to  Write  your changes once you add or remove finalizers      Go package resource     IsMarkedForDeletion returns true if a resource has been marked for deletion     false otherwise  func IsMarkedForDeletion res  pbresource Resource  bool             HasFinalizers returns true if a resource has one or more finalizers  false otherwise  func HasFinalizers res  pbresource Resource  bool             HasFinalizer returns true if a resource has a given finalizer  false otherwise  func HasFinalizer res  pbresource Resource  finalizer string  bool             AddFinalizer adds a finalizer to the given resource  func AddFinalizer res  pbresource Resource  finalizer string              RemoveFinalizer removes a finalizer from the given resource  func RemoveFinalizer res  pbresource Resource  finalizer string              GetFinalizers returns the set of finalizers for the given resource  func GetFinalizers res  pbresource Resource  mapset Set string               Example flow in a controller s  Reconcile  method    Go const finalizer    consul io bar finalizer   func  barReconciler  Reconcile ctx context Context  rt controller Runtime  req controller Request  error            Check if resource is marked for deletion  If yes  perform cleanup  remove finalizer  and Write the resource  if resource IsMarkedForDeletion res         Perform some cleanup      return EnsureFinalizerRemoved ctx  rt  res  finalizer          Check if resource has finalizer  If not  add it and Write the resource  if err    EnsureHasFinalizer ctx  rt  res  finalizer   err    nil     return err              Ownership   Cascading Deletion  The resource service implements a lightweight  1 N  ownership model where  on creation  you can mark a resource as being  owned  by another resource  When the owner is deleted  the owned resource will be deleted too      Go client Write ctx   pbresource WriteRequest   Resource   pbresource Resource     Owner  ownerID                        Testing  Now that you have created your controller its time to test it  The types of tests each controller should have and boiler plat for test files is documented  here    testing md


consul Controllers This page describes how to write controllers in Consul s new controller architecture A controller consists of several parts Note This information is valid as of Consul 1 17 but some portions may change in future releases Controller Basics

Answer:
# Controllers

This page describes how to write controllers in Consul's new controller architecture.

-> **Note**: This information is valid as of Consul 1.17 but some portions may change in future releases.

## Controller Basics

A controller consists of several parts:

1. **The watched type** - This is the main type a controller is watching and reconciling.
2. **Additional watched types** - These are additional types a controller may care about in addition to the main watched type.
3. **Additional custom watches** - These are the watches for things that aren't resources in Consul.
4. **Reconciler** - This is the instance that's responsible for reconciling requests whenever there's an event for the main watched type or for any of the watched types.
5. **Initializer** - This is responsible for anything that needs to be executed when the controller is started.

A basic controller setup could look like this:

```go
func barController() controller.Controller {
    return controller.NewController("bar", pbexample.BarType).
        WithReconciler(barReconciler{})
}
```

barReconciler needs to implement the `Reconcile` method of the `Reconciler` interface.
It's important to note that the `Reconcile` method only gets the request with the `ID` of the main
watched resource and so it's up to the reconcile implementation to fetch the resource and any relevant information needed
to perform the reconciliation. The most basic reconciler could look as follows:

```go
type barReconciler struct {}

func (b *barReconciler) Reconcile(ctx context.Context, rt Runtime, req Request) error {
    ...
}
```

## Watching Additional Resources

Most of the time, controllers will need to watch more resources in addition to the main watched type.
To set up an additional watch, the main thing we need to figure out is how to map additional watched resource to the main
watched resource. Controller-runtime allows us to implement a mapper function that can take the additional watched resource
as the input and produce reconcile `Requests` for our main watched type.

To figure out how to map the two resources together, we need to think about the relationship between the two resources.

There are several common relationship types between resources that are being used currently:
1. Name-alignment: this relationship means that resources are named the same and live in the same tenancy, but have different data. Examples: `Service` and `ServiceEndpoints`, `Workload` and `ProxyStateTemplate`.
2. Selector: this relationship happens when one resource selects another by name or name prefix. Examples: `Service` and `Workload`, `ProxyConfiguration` and `Workload`.
3. Owner: in this relationship, one resource is the owner of another resource. Examples: `Service` and `ServiceEndpoints`, `HealthStatus` and `Workload`.
4. Arbitrary reference: in this relationship, one resource may reference another by some sort of reference. This reference could be a single string in the resource data or a more composite reference containing name, tenancy, and type. Examples: `Workload` and `WorkloadIdentity`, `HTTPRoute` and `Service`.

Note that it's possible for the two watched resources to have more than one relationship type simultaneously.
For example, `FailoverPolicy` type is name-aligned with a service to which it applies, however, it also contains
references to destination services, and for a controller that reconciles `FailoverPolicy` and watches `Service`
we need to account for both type 1 and type 4 relationship whenever we get an event for a `Service`.

### Simple Mappers

Let's look at some simple mapping examples.

#### Name-aligned resources
If our resources only have a name-aligned relationship, we can map them with a built-in function:

```go
func barController() controller.Controller {
    return controller.NewController("bar", pbexample.BarType).
        WithWatch(pbexample.FooType, controller.ReplaceType(pbexample.BarType)).
        WithReconciler(barReconciler{})
}
```

Here, all we need to do is replace the type of the `Foo` resource whenever we get an event for it.

#### Owned resources

Let's say our `Foo` resource owns `Bar` resources, where any `Foo` resource can own multiple `Bar` resources.
In this case, whenever we see a new event for `Foo`, all we need to do is get all `Bar` resources that `Foo` currently owns.
For this, we can also use a built-in function to set up our watch:

```go
func MapOwned(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) {
    resp, err := rt.Client.ListByOwner(ctx, &pbresource.ListByOwnerRequest{Owner: res.Id})
    if err != nil {
        return nil, err
    }

    var result []controller.Request
    for _, r := range resp.Resources {
        result = append(result, controller.Request{ID: r.Id})
    }

    return result, nil
}

func barController() controller.Controller {
    return controller.NewController("bar", pbexample.BarType).
        WithWatch(pbexample.FooType, MapOwned).
        WithReconciler(barReconciler{})
}
```

### Advanced Mappers and Caches

For selector or arbitrary reference relationships, the mapping that we choose may need to be more advanced.

#### Naive mapper implementation

Let's first consider what a naive mapping function could look like in this case. Let's say that the `Bar` resource
references `Foo` resource by name in the data. Now to watch and map `Foo` resources, we need to be able to find all relevant `Bar` resources
whenever we get an event for a `Foo` resource.

```go
func MapFoo(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) {
    resp, err := rt.Client.List(ctx, &pbresource.ListRequest{Type: pbexample.BarType, Tenancy: res.Id.Tenancy})
    if err != nil {
        return nil, err
    }

    var result []controller.Request
    for _, r := range resp.Resources {
        decodedResource, err := resource.Decode[*pbexample.Bar](r)
        if err != nil {
            return nil, err
        }

        // Only add Bar resources that match Foo by name.
        if decodedResource.GetData().GetFooName() == res.Id.Name {
            result = append(result, controller.Request{ID: r.Id})
        }
    }
}
```

This approach is fine for cases when the number of `Bar` resources in a cluster is relatively small. If it's not,
then we'd be doing a large `O(N)` search on each `Bar` event which could be too expensive.

#### Caching Mappers

For cases when `N` is too large, we'd want to use a caching layer to help us make lookups more efficient so that they
don't require an `O(N)` search of potentially all cluster resources.

The controller runtime contains a controller cache and the facilities to keep the cache up to date in response to watches. Additionally there are dependency mappers provided for querying the cache.

_While it is possible to not use the builtin cache and manage state in dependency mappers yourself, this can get quite complex and reasoning about the correct times to track and untrack relationships is tricky to get right. Usage of the cache is therefore the advised approach._

At a high level, the controller author provides the indexes to track for each watchedtype and can then query thosfunc fooFromArgs(args ...any) ([]byte, error)e indexes in the  {

}future. The querying can occur during both dependency mapping and during resource reconciliation.

The following example shows how to configure the "bar" controller to rereconcile a Bar resource whenever a Foo resource is changed that references the Bar

```go
func fooReferenceFromBar(r *resource.DecodedResource[*pbexample.Bar]) (bool, []byte, error) {
    idx := index.IndexFromRefOrID(&pbresource.ID{
        Type: pbexample.FooType,
        Tenancy: r.Id.Tenancy,
        Name: r.Data.GetFooName(),
    })

    return true, idx, nil
}

func barController() controller.Controller {
    fooIndex := indexers.DecodedSingleIndexer(
        "foo",
        index.ReferenceOrIDFromArgs,
        fooReferenceFromBar,
    )

    return controller.NewController("bar", pbexample.BarType, fooIndex).
        WithWatch(
            pbexample.FooType,
            dependency.CacheListMapper(pbexample.BarType, fooIndex.Name()),
        ).
        WithReconciler(barReconciler{})
}
```

The controller will now reconcile Bar type resources whenever the Foo type resources they reference are updated. No further tracking is necessary as changes to all Bar types will automatically update the cache.

One limitation of the cache is that it only has knowledge about the current state of resources. That specifically means that the previous state is forgotten once the cache observes a write. This can be problematic when you want to reconcile a resource to no longer take into account something that previously reference it.

Lets say there are two types: `Baz` and `ComputedBaz` and a controller that will aggregate all `Baz` resource with some value into a single `ComputedBaz` object. When
a `Baz` resource gets updated to no longer have a value, it should not be represented in the `ComputedBaz` resource. The typical way to work around this is to:

1. Store references to the resources that were used during reconciliation within the computed/reconciled resource. For types computed by controllers and not expected to be written directly by users a `bound_references` field should be added to the top level resource types message. For other user manageable types the references may need to be stored within the Status field.

2. Add a cache index to the watch of the computed type (usually the controllers main managed type). This index can use one of the indexers specified within the  [`internal/controller/cache/indexers`](../../../internal/controller/cache/indexers/) package. That package contains some builtin functionality around reference indexing.

3. Update the dependency mappers to query the cache index *in addition to* looking at the current state of the dependent resource. In our example above the `Baz` dependency mapper could use the [`MultiMapper`] to combine querying the cache for `Baz` types that currently should be associated with a `ComputedBaz` and querying the index added in step 2 for previous references.

#### Footgun: Needing Bound References

When an interior (mutable) foreign key pointer on watched data is used to
determine the resources's applicability in a dependency mapper, it is subject
to the "orphaned computed resource" problem.

(An example of this would be a ParentRef on an xRoute, or the Destination field
of a TrafficPermission.)

When you edit the mutable pointer to point elsewhere, the DependencyMapper will
only witness the NEW value and will trigger reconciles for things derived from
the NEW pointer, but side effects from a prior reconcile using the OLD pointer
will be orphaned until some other event triggers that reconcile (if ever).

This applies equally to all varieties of controller:

- creates computed resources
- only updates status conditions on existing resources
- has other external side effects (xDS controller writes envoy config over a stream)

To solve this we need to collect the list of bound references that were
"ingredients" into a computed resource's output and persist them on the newly
written resource. Then we load them up and index them such that we can use them
to AUGMENT a mapper event with additional maps using the OLD data as well.

We have only actively worked to solve this for the computed resource flavor of
controller:

1. The top level of the resource data protobuf needs a
   `BoundReferences []*pbresource.Reference` field.

2. Use a `*resource.BoundReferenceCollector` to capture any resource during
   `Reconcile` that directly contributes to the final output resource data
   payload.

3. Call `brc.List()` on the above and set it to the `BoundReferences` field on
   the computed resource before persisting.

4. Use `indexers.BoundRefsIndex` to index this field on the primary type of the
   controller.

5. Create `boundRefsMapper := dependency.CacheListMapper(ZZZ, boundRefsIndex.Name())`

6. For each watched type, wrap its DependencyMapper with
   `dependency.MultiMapper(boundRefsMapper, ZZZ)`

7. That's it.

This will cause each reconcile to index the prior list of inputs and augment
the results of future mapper events with historical references.

### Custom Watches

In some cases, we may want to trigger reconciles for events that aren't generated from CRUD operations on resources, for example
when Envoy proxy connects or disconnects to a server. Controller-runtime allows us to setup watches from
events that come from a custom event channel. Please see [xds-controller](https://github.com/hashicorp/consul/blob/ecfeb7aac51df8730064d869bb1f2c633a531522/internal/mesh/internal/controllers/xds/controller.go#L40-L41) for examples of custom watches.

## Statuses

In many cases, controllers would need to update statuses on resources to let the user know about the successful or unsuccessful
state of a resource.

These are the guidelines that we recommend for statuses:

* While status conditions is a list, the Condition type should be treated as a key in a map, meaning a resource should not have two status conditions with the same type.
* Controllers need to both update successful and unsuccessful conditions states. This is because we need to make sure that we clear any failed status conditions.
* Status conditions should be named such that the `True` state is a successful state and `False` state is a failed state.

## Best Practices

Below is a list of controller best practices that we've learned so far. Many of them are inspired by [kubebuilder](https://book.kubebuilder.io/reference/good-practices).

* Avoid monolithic controllers as much as possible. A single controller should only manage a single resource to avoid complexity and race conditions.
* If using cached mappers, aim to write (update or delete entries) to mappers in the `Reconcile` method and read from them in the mapper functions used by watches.
* Fetch all data in the `Reconcile` method and avoid caching it from the mapper functions. This ensures that we get the latest data for each reconciliation.

Controllers  This page describes how to write controllers in Consul s new controller architecture        Note    This information is valid as of Consul 1 17 but some portions may change in future releases      Controller Basics  A controller consists of several parts    1    The watched type     This is the main type a controller is watching and reconciling  2    Additional watched types     These are additional types a controller may care about in addition to the main watched type  3    Additional custom watches     These are the watches for things that aren t resources in Consul   4    Reconciler     This is the instance that s responsible for reconciling requests whenever there s an event for the main watched type or for any of the watched types  5    Initializer     This is responsible for anything that needs to be executed when the controller is started   A basic controller setup could look like this      go func barController   controller Controller       return controller NewController  bar   pbexample BarType           WithReconciler barReconciler           barReconciler needs to implement the  Reconcile  method of the  Reconciler  interface   It s important to note that the  Reconcile  method only gets the request with the  ID  of the main watched resource and so it s up to the reconcile implementation to fetch the resource and any relevant information needed to perform the reconciliation  The most basic reconciler could look as follows      go type barReconciler struct     func  b  barReconciler  Reconcile ctx context Context  rt Runtime  req Request  error                     Watching Additional Resources  Most of the time  controllers will need to watch more resources in addition to the main watched type   To set up an additional watch  the main thing we need to figure out is how to map additional watched resource to the main watched resource  Controller runtime allows us to implement a mapper function that can take the additional watched resource as the input and produce reconcile  Requests  for our main watched type   To figure out how to map the two resources together  we need to think about the relationship between the two resources   There are several common relationship types between resources that are being used currently  1  Name alignment  this relationship means that resources are named the same and live in the same tenancy  but have different data  Examples   Service  and  ServiceEndpoints    Workload  and  ProxyStateTemplate   2  Selector  this relationship happens when one resource selects another by name or name prefix  Examples   Service  and  Workload    ProxyConfiguration  and  Workload   3  Owner  in this relationship  one resource is the owner of another resource  Examples   Service  and  ServiceEndpoints    HealthStatus  and  Workload   4  Arbitrary reference  in this relationship  one resource may reference another by some sort of reference  This reference could be a single string in the resource data or a more composite reference containing name  tenancy  and type  Examples   Workload  and  WorkloadIdentity    HTTPRoute  and  Service    Note that it s possible for the two watched resources to have more than one relationship type simultaneously   For example   FailoverPolicy  type is name aligned with a service to which it applies  however  it also contains references to destination services  and for a controller that reconciles  FailoverPolicy  and watches  Service  we need to account for both type 1 and type 4 relationship whenever we get an event for a  Service         Simple Mappers  Let s look at some simple mapping examples         Name aligned resources If our resources only have a name aligned relationship  we can map them with a built in function      go func barController   controller Controller       return controller NewController  bar   pbexample BarType           WithWatch pbexample FooType  controller ReplaceType pbexample BarType             WithReconciler barReconciler           Here  all we need to do is replace the type of the  Foo  resource whenever we get an event for it        Owned resources  Let s say our  Foo  resource owns  Bar  resources  where any  Foo  resource can own multiple  Bar  resources   In this case  whenever we see a new event for  Foo   all we need to do is get all  Bar  resources that  Foo  currently owns  For this  we can also use a built in function to set up our watch      go func MapOwned ctx context Context  rt controller Runtime  res  pbresource Resource     controller Request  error        resp  err    rt Client ListByOwner ctx   pbresource ListByOwnerRequest Owner  res Id       if err    nil           return nil  err            var result   controller Request     for    r    range resp Resources           result   append result  controller Request ID  r Id              return result  nil    func barController   controller Controller       return controller NewController  bar   pbexample BarType           WithWatch pbexample FooType  MapOwned            WithReconciler barReconciler               Advanced Mappers and Caches  For selector or arbitrary reference relationships  the mapping that we choose may need to be more advanced         Naive mapper implementation  Let s first consider what a naive mapping function could look like in this case  Let s say that the  Bar  resource references  Foo  resource by name in the data  Now to watch and map  Foo  resources  we need to be able to find all relevant  Bar  resources whenever we get an event for a  Foo  resource      go func MapFoo ctx context Context  rt controller Runtime  res  pbresource Resource     controller Request  error        resp  err    rt Client List ctx   pbresource ListRequest Type  pbexample BarType  Tenancy  res Id Tenancy       if err    nil           return nil  err            var result   controller Request     for    r    range resp Resources           decodedResource  err    resource Decode  pbexample Bar  r          if err    nil               return nil  err                               Only add Bar resources that match Foo by name           if decodedResource GetData   GetFooName      res Id Name               result   append result  controller Request ID  r Id                          This approach is fine for cases when the number of  Bar  resources in a cluster is relatively small  If it s not  then we d be doing a large  O N   search on each  Bar  event which could be too expensive         Caching Mappers  For cases when  N  is too large  we d want to use a caching layer to help us make lookups more efficient so that they don t require an  O N   search of potentially all cluster resources   The controller runtime contains a controller cache and the facilities to keep the cache up to date in response to watches  Additionally there are dependency mappers provided for querying the cache     While it is possible to not use the builtin cache and manage state in dependency mappers yourself  this can get quite complex and reasoning about the correct times to track and untrack relationships is tricky to get right  Usage of the cache is therefore the advised approach    At a high level  the controller author provides the indexes to track for each watchedtype and can then query thosfunc fooFromArgs args    any     byte  error e indexes in the          future  The querying can occur during both dependency mapping and during resource reconciliation   The following example shows how to configure the  bar  controller to rereconcile a Bar resource whenever a Foo resource is changed that references the Bar     go func fooReferenceFromBar r  resource DecodedResource  pbexample Bar    bool    byte  error        idx    index IndexFromRefOrID  pbresource ID          Type  pbexample FooType          Tenancy  r Id Tenancy          Name  r Data GetFooName                    return true  idx  nil    func barController   controller Controller       fooIndex    indexers DecodedSingleIndexer           foo            index ReferenceOrIDFromArgs          fooReferenceFromBar              return controller NewController  bar   pbexample BarType  fooIndex           WithWatch              pbexample FooType               dependency CacheListMapper pbexample BarType  fooIndex Name                        WithReconciler barReconciler           The controller will now reconcile Bar type resources whenever the Foo type resources they reference are updated  No further tracking is necessary as changes to all Bar types will automatically update the cache   One limitation of the cache is that it only has knowledge about the current state of resources  That specifically means that the previous state is forgotten once the cache observes a write  This can be problematic when you want to reconcile a resource to no longer take into account something that previously reference it    Lets say there are two types   Baz  and  ComputedBaz  and a controller that will aggregate all  Baz  resource with some value into a single  ComputedBaz  object  When a  Baz  resource gets updated to no longer have a value  it should not be represented in the  ComputedBaz  resource  The typical way to work around this is to   1  Store references to the resources that were used during reconciliation within the computed reconciled resource  For types computed by controllers and not expected to be written directly by users a  bound references  field should be added to the top level resource types message  For other user manageable types the references may need to be stored within the Status field   2  Add a cache index to the watch of the computed type  usually the controllers main managed type   This index can use one of the indexers specified within the    internal controller cache indexers            internal controller cache indexers   package  That package contains some builtin functionality around reference indexing   3  Update the dependency mappers to query the cache index  in addition to  looking at the current state of the dependent resource  In our example above the  Baz  dependency mapper could use the   MultiMapper   to combine querying the cache for  Baz  types that currently should be associated with a  ComputedBaz  and querying the index added in step 2 for previous references        Footgun  Needing Bound References  When an interior  mutable  foreign key pointer on watched data is used to determine the resources s applicability in a dependency mapper  it is subject to the  orphaned computed resource  problem    An example of this would be a ParentRef on an xRoute  or the Destination field of a TrafficPermission    When you edit the mutable pointer to point elsewhere  the DependencyMapper will only witness the NEW value and will trigger reconciles for things derived from the NEW pointer  but side effects from a prior reconcile using the OLD pointer will be orphaned until some other event triggers that reconcile  if ever    This applies equally to all varieties of controller     creates computed resources   only updates status conditions on existing resources   has other external side effects  xDS controller writes envoy config over a stream   To solve this we need to collect the list of bound references that were  ingredients  into a computed resource s output and persist them on the newly written resource  Then we load them up and index them such that we can use them to AUGMENT a mapper event with additional maps using the OLD data as well   We have only actively worked to solve this for the computed resource flavor of controller   1  The top level of the resource data protobuf needs a     BoundReferences    pbresource Reference  field   2  Use a   resource BoundReferenceCollector  to capture any resource during     Reconcile  that directly contributes to the final output resource data    payload   3  Call  brc List    on the above and set it to the  BoundReferences  field on    the computed resource before persisting   4  Use  indexers BoundRefsIndex  to index this field on the primary type of the    controller   5  Create  boundRefsMapper    dependency CacheListMapper ZZZ  boundRefsIndex Name      6  For each watched type  wrap its DependencyMapper with     dependency MultiMapper boundRefsMapper  ZZZ    7  That s it   This will cause each reconcile to index the prior list of inputs and augment the results of future mapper events with historical references       Custom Watches  In some cases  we may want to trigger reconciles for events that aren t generated from CRUD operations on resources  for example when Envoy proxy connects or disconnects to a server  Controller runtime allows us to setup watches from events that come from a custom event channel  Please see  xds controller  https   github com hashicorp consul blob ecfeb7aac51df8730064d869bb1f2c633a531522 internal mesh internal controllers xds controller go L40 L41  for examples of custom watches      Statuses  In many cases  controllers would need to update statuses on resources to let the user know about the successful or unsuccessful state of a resource   These are the guidelines that we recommend for statuses     While status conditions is a list  the Condition type should be treated as a key in a map  meaning a resource should not have two status conditions with the same type    Controllers need to both update successful and unsuccessful conditions states  This is because we need to make sure that we clear any failed status conditions    Status conditions should be named such that the  True  state is a successful state and  False  state is a failed state       Best Practices  Below is a list of controller best practices that we ve learned so far  Many of them are inspired by  kubebuilder  https   book kubebuilder io reference good practices      Avoid monolithic controllers as much as possible  A single controller should only manage a single resource to avoid complexity and race conditions    If using cached mappers  aim to write  update or delete entries  to mappers in the  Reconcile  method and read from them in the mapper functions used by watches    Fetch all data in the  Reconcile  method and avoid caching it from the mapper functions  This ensures that we get the latest data for each reconciliation