Managing Secrets Across Kubernetes Clusters with Fleet

Originally published on the River Point Technology Blog

For organizations leveraging Kubernetes and Rancher, efficient secret management across multiple clusters is a common concern. This blog post uses SUSE’s Fleet to explore a custom operator solution that streamlines secret distribution in multi-cluster environments.

The Challenge of Multi-Cluster Secret Management

As organizations scale Kubernetes infrastructure, managing secrets across multiple clusters becomes increasingly complex. While it’s possible to configure each downstream cluster to communicate directly with a central secret store like HashiCorp Vault, this approach can become unwieldy as the number of clusters grows. Using the Kubernetes JWT authentication method with Vault requires careful management of roles and policies for each cluster. Alternatively, using the AppRole authentication method, while more straightforward to set up, falls short of providing the needed level of security.

Leveraging Rancher as a Central Secrets Manager

Clusters managed by Rancher leverage the Fleet agent for various aspects of configuration. Using a custom operator, we can enable replication of secrets in targeted Rancher managed clusters. This accounts for secrets created manually in the Rancher cluster or externally managed secrets from tools like HashiCorp Vault or External Secrets.

Introducing the Fleet Handshake Operator

We’ve developed a customer Kubernetes operator that works with Fleet to distribute secrets across clusters. This operator, the Fleet Handshake Operator, listens for defined Kubernetes secrets and creates Fleet Bundles to distribute to downstream clusters.

Fleet Handshake utilizes the cluster hosting the Fleet controller, typically deployed within Rancher, as the central source of truth for secrets. Fleet agents then consume these secrets in downstream clusters. This approach solves the “secret zero” problem by leveraging Rancher as the JWT authentication to Vault and Fleet to manage the connectivity to downstream clusters. Moreover, it provides a centralized point of control for secret distribution while maintaining security.

Advantages of This Approach

Automation: The operator automates the process of secret distribution, reducing manual intervention and potential errors.
Scalability: Adding new clusters is as simple as updating the FleetHandshake resource with new targets or tagging the resource with the appropriate targeted label selector.
Consistency: Ensures that secrets are consistently distributed across all specified clusters.
Leverage Existing Infrastructure: Utilizes Rancher’s Fleet for secret distribution, using its secure communication channels with downstream clusters.
Flexibility: Works with Kubernetes secrets regardless of source, allowing integration with various secret management tools and workflows.

Technical Implementation

The Fleet Handshake Operator is built around a custom resource definition (CRD) called FleetHandshake. This CRD defines the structure for specifying which secrets should be synchronized and to which target clusters. The main controller, FleetHandshakeReconciler, handles the reconciliation loop for these custom resources. When a secret is created or updated, the reconciler, which listens for the changes, will then update the Bundle resource it manages to distribute downstream.

Let’s dive into the critical components of the operator:

Reconciliation Process

The Reconcile function implements the operator’s core logic. It first gets the secret resource and the custom FleetHandshake. It then upserts the bundle attribute. If all is successful, it updates the fleet handshake to synced.

func (r *FleetHandshakeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Fetch the FleetHandshake resource
    var fleetHandshake rancherv1.FleetHandshake
    if err := r.Get(ctx, req.NamespacedName, &fleetHandshake); err != nil {
        // Handle error or return if not found
    }

    // Retrieve the target secret
    var secret corev1.Secret
    if err := r.Get(ctx, types.NamespacedName{Namespace: fleetHandshake.Namespace, Name: fleetHandshake.Spec.SecretName}, &secret); err != nil {
        // Handle error or return if not found. if not found, a status of mising will be set
    }

    // Check to see if a fleet bundle exists
    var existingBundle fleetv1alpha1api.Bundle
    err := r.Get(ctx, types.NamespacedName{Name: bundle.Name, Namespace: bundle.Namespace}, existingBundle)
    if err != nil && errors.IsNotFound(err) {
       // Update existing Bundle if content has changed
         if !reflect.DeepEqual(existingBundle.Spec, bundle.Spec) {
            if err := r.Update(ctx, bundle); err != nil {
                // Handle bundle error
            }
        }
    } else {
        if err := r.Create(ctx, bundle); err != nil {
                // Handle bundle error
            }
        }
    }

   fleetHandshake.Status.Status = "Synced"
    if err := r.Status().Update(ctx, &fleetHandshake); err != nil {
        // Handle error
    }

    return ctrl.Result{}, nil
}

A Bundle–a resource housing the secret’s content–is distributed by Fleet to the respective targets. The owner reference plays a significant role here, tying the bundle to the fleet handshake resource. This synchronization ensures that the lifecycles align seamlessly, enhancing the efficiency of the process. Importantly, deleting the handshake prompts the Kubernetes API to remove the secrets downstream.

bundle := &fleetv1alpha1api.Bundle{
    ObjectMeta: metav1.ObjectMeta{
        Name:      fleetHandshake.Name,
        Namespace: fleetHandshake.Namespace,
        OwnerReferences: []metav1.OwnerReference{{
            APIVersion: fleetHandshake.APIVersion,
            Kind:       fleetHandshake.Kind,
            Name:       fleetHandshake.Name,
            UID:        fleetHandshake.UID,
        }},
    },
    Spec: fleetv1alpha1api.BundleSpec{
        Resources: []fleetv1alpha1api.BundleResource{
            {
                Name:    fmt.Sprintf("%s.json", secret.Name),
                Content: string(jsonSecret),
            },
        },
        Targets: fleetHandshake.Spec.Targets,
    },
}

Conclusion

The Fleet Handshake Operator provides a powerful solution for organizations seeking to streamline their secret management across multiple Kubernetes clusters. By leveraging Suse’s Fleet and implementing a custom operator, we can achieve a scalable, secure, and automated approach to secret distribution. This implementation serves as a testament to the pivotal role custom operators play in extending and enhancing the capabilities of existing Kubernetes ecosystem tools, providing tailored solutions for complex operational challenges. As Kubernetes environments become complex, the role of such custom operators in maintaining operational efficiency and security becomes increasingly significant.

To explore this solution and install it within your rancher instance, visit our GitHub repository at GitHub - rptcloud/fleet-handshake: A custom resource for syncing secrets with rancher., or checkout our visual walkthrough of the Fleet Handshake Operator’s capabilities