Broadleaf's microservices architecture uses the Cluster Singleton Pattern to ensure certain operations run exclusively across your deployment, even when you have multiple replicas. This guide covers how to configure the cluster service implementation for different environments—from local development to production Kubernetes clusters.

What is the Cluster Singleton Pattern?

Some tasks should only run on one node at a time, even when you have multiple replicas running.

The Cluster Singleton Pattern guarantees exclusive execution across your cluster. Some examples within Broadleaf that utilize this pattern include:

• Broadleaf’s Messaging Resiliency: allows resilient asynchronous communication between microservices 
• Scheduled Job Co-ordination: To ensure a trigger event is only emitted once. e.g. so that a job isn't inadvertently run multiple times

How Broadleaf Handles This

Broadleaf uses Apache's CamelClusterService to implement this pattern. Broadleaf provides configuration support for these backing implementations:

• FileLockClusterService - For local development
• KubernetesClusterService - For cloud Kubernetes deployments
• ZookeeperClusterService - Alternative for non-Kubernetes environments

Local Development Configuration

Broadleaf services use FileLockClusterService by default, which is the appropriate implementation for local development. Your manifest file configures this per environment:

schema:
  version: '1.0'
project:
  groupId: com.example.microservices
  packageName: com.example.microservices
  starterParentVersion: x.x.x-GA
  version: 1.0.0-SNAPSHOT
  clusterServiceType:
local: file
docker: file
cloud: kubernetes

The file-based implementation works well for single-node development environments. For multi-node production deployments, you'll want to configure either Kubernetes or Zookeeper as your cluster service implementation.

Kubernetes Production Configuration

For Kubernetes deployments, Broadleaf provides support for KubernetesClusterService, which uses Kubernetes's native Lease API to coordinate leader election across replicas.

Step 1: Environment Variables

Add these environment variables to your Kubernetes deployment:

env:
  - name: BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE
    value: "kubernetes"
  - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
    value: "auth"  # Use unique names: auth, cart, catalog, etc.
  
  - name: LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK
    value: "DEBUG"  # Optional but helpful for initial setup

The CLUSTERK8SAPPLABELVALUE should be unique for each microservice deployment. For the Balanced Flex Package Composition, use: auth, browse, cart, processing, or supporting. For Granular Composition, use service-specific names like cart, catalog, offer, etc.

Step 2: Kubernetes RBAC Configuration

The KubernetesClusterService needs permissions to read and write Lease objects. Create these Kubernetes resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lease-access
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - create
      - update
      - list
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lease-access
subjects:
  - kind: ServiceAccount
    name: your-service-account
    namespace: your-namespace
roleRef:
  kind: ClusterRole
  name: lease-access
  apiGroup: rbac.authorization.k8s.io

Step 3: Verify It Works

Check your logs with DEBUG enabled. You should see messages about lease acquisition and renewal from the org.apache.camel.component.kubernetes.cluster.lock package.

You can also check the Lease objects directly:

kubectl get leases -n your-namespace

Expected output will show leases with names like:

NAME                                                    HOLDER                      AGE
leaders-batch-index-request-processing-batch-index...  processing-5b4d98c557-...   136m
leaders-cartnotification-cart-checkout-completion       cart-85bb888d-vzj6f         138m
leaders-changesummary-supporting-change                 supporting-67f8946db9-...   138m

Each lease shows which pod currently holds the leadership lock for that specific cluster singleton.

Zookeeper Configuration

Zookeeper is a supported alternative to Kubernetes for cluster service coordination. This can be useful for non-Kubernetes deployments or for large-scale granular deployments where centralized lock management may be preferred. Since Zookeeper is part of the Broadleaf reference stack, it's readily available.

Configuration is straightforward via environment variables:

env:
  - name: BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE
    value: "zookeeper"
  # Optional: Zookeeper ACL settings (defaults shown)
  - name: BROADLEAF_LOCK_ZKACL_ENABLED
    value: "true"  # Default: true
  - name: BROADLEAF_LOCK_ZKACL_USERNAME
    value: "readonly-user"  # Default: readonly-user
  - name: BROADLEAF_LOCK_ZKACL_PASSWORD
    value: "your-secure-password"  # Supply via secure mechanism

Verify it's working by running the dump command against Zookeeper:

kubectl exec -it zookeeper-pod -- curl 
http://localhost:8080/commands/dump

Look for /camel in the output—that's where cluster service locks live.

Configuration Considerations

Environment-Specific Implementation

The cluster service implementation should match your deployment environment. File-based is appropriate for local development, while Kubernetes or Zookeeper implementations are designed for multi-node production deployments.

RBAC Permissions

The KubernetesClusterService requires specific permissions to manage Lease objects. Ensure your ClusterRole and ClusterRoleBinding are properly configured with the permissions shown above.

Unique Label Values

Each microservice deployment needs a unique CLUSTERK8SAPPLABELVALUE. This label distinguishes different services and enables proper leader election per service type.

API Connectivity

Pods must be able to reach the Kubernetes API server for lease management. Consider this when configuring network policies.

Performance Tuning for Large Deployments

If you have a large topology (e.g., deploying multiple replicas of a full granular deployment) and you're seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane, additional configuration is likely warranted. The default settings can cause high CPU utilization due to the underlying Fabric8 client library making frequent "LIST" API calls.

To mitigate this, you can tune these Apache Camel Kubernetes configuration options:

• camel.component.kubernetes-cluster-service.lease-duration-millis - Duration that leaders hold the lease (milliseconds)
• camel.component.kubernetes-cluster-service.renew-deadline-millis - Duration that non-leader candidates wait to force acquire leadership (milliseconds)
• camel.component.kubernetes-cluster-service.retry-period-millis - Time between subsequent attempts to check and acquire leadership, randomized using jitter factor (milliseconds)
• camel.component.kubernetes-cluster-service.jitter-factor - Jitter factor applied to the renew deadline

Increasing these values reduces the frequency of API calls to the Kubernetes master.

Here's an example configuration that has proven effective for Broadleaf microservices clusters under load:

env:
  - name: CAMEL_CLUSTER_KUBERNETES_LEASE_DURATION_MILLIS
    value: "60000"
  - name: CAMEL_CLUSTER_KUBERNETES_RENEW_DEADLINE_MILLIS
    value: "40000"
  - name: CAMEL_CLUSTER_KUBERNETES_RETRY_PERIOD_MILLIS
    value: "10000"
  - name: CAMEL_CLUSTER_KUBERNETES_JITTER_FACTOR
    value: "2"

Environment-Based Configuration via Manifest

The manifest file provides a clean way to specify cluster service implementations per environment (available since Broadleaf 2.0.2-GA / 2.1.0-GA):

clusterServiceType:
  local: file      # Local development
  docker: file     # Docker Compose
  cloud: kubernetes # Production Kubernetes

Valid values for each environment are file, zookeeper, and kubernetes. Any omitted environment defaults to file.

This approach allows you to configure the appropriate implementation for each deployment target in one place, and the correct implementation is automatically selected based on where the application runs.

Summary

Broadleaf's Camel Cluster Service supports multiple implementations to suit different deployment environments:

• FileLockClusterService for local development
• KubernetesClusterService for Kubernetes production deployments
• ZookeeperClusterService as an alternative for non-Kubernetes environments

The manifest-based configuration makes it straightforward to specify the appropriate implementation for each environment. Once configured, the cluster service automatically handles leader election and failover across your deployment.

Note: All configuration examples and technical details in this guide are sourced from the official Broadleaf documentation.

Resources:

• Broadleaf Camel Cluster Service Documentation
• Apache Camel Clustering Guide
• Kubernetes Lease API Documentation

For more details, see the official Broadleaf documentation linked above.