client-go Deep Dive: Informer — The Core of Kubernetes Controller Development

If you've ever built a Kubernetes controller — or wondered how kube-controller-manager keeps thousands of resources in sync — the answer is Informer. It's the single most important component in client-go, and mastering it is the foundation of all Kubernetes development.

1. The Problem Informer Solves

Kubernetes components need to constantly watch and query cluster resources. Take the Deployment controller: it must track every Deployment and its ReplicaSets in real time, continuously reconciling actual state toward desired state.

Doing this naively creates serious problems:

Problem	Impact
Polling the API Server repeatedly	Crushes API Server and etcd under query load
No middleware between components	Must guarantee message reliability, ordering, and real-time delivery over raw HTTP
Multiple controllers watching the same resource	Duplicate List/Watch calls, redundant serialization/deserialization

Informer solves all three — with a local cache, a shared watch connection, and an event subscription model.

2. How Informer Works: List/Watch

The core mechanism is List/Watch:

Informer starts
     │
     ▼
LIST  ── fetch ALL resources of interest from API Server
     │    store full snapshot in local cache (Indexer)
     ▼
WATCH ── open a long-lived HTTP connection to API Server
     │    receive incremental change events (Add/Update/Delete)
     ▼
Update local cache on every event
(cache stays in sync with etcd without polling)

Benefits:

Controllers query the local cache (Indexer) — extremely fast, zero API Server load
A single Watch connection replaces thousands of polling requests
Cache is always consistent with etcd state

3. The Informer Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Informer Framework                       │
│                                                                 │
│  API Server                                                     │
│      │  List / Watch                                            │
│      ▼                                                          │
│  ┌──────────┐    events    ┌─────────────┐                      │
│  │ Reflector│ ──────────► │  DeltaFIFO  │                      │
│  │(List/Watch)│           │  (event queue)│                     │
│  └──────────┘             └──────┬──────┘                      │
│                                  │ Pop                          │
│                                  ▼                              │
│                          ┌───────────────┐                      │
│                          │   Controller  │ (Informer internal)  │
│                          │ processLoop() │                      │
│                          └──────┬────────┘                      │
│                    ┌────────────┴──────────────┐                │
│                    ▼                           ▼                │
│             ┌────────────┐           ┌──────────────────┐       │
│             │  Indexer   │           │  ShareInformer   │       │
│             │(local cache│           │  distribute()    │       │
│             │ ThreadSafe)│           └────────┬─────────┘       │
│             └────────────┘                    │                 │
│                    ▲                          ▼                 │
│                    │ Lister           AddEventHandler           │
│             ┌──────┴──────┐          (user callbacks)          │
│             │ Your        │                  │                  │
│             │ Controller  │◄─── WorkQueue ───┘                  │
│             └─────────────┘                                     │
└─────────────────────────────────────────────────────────────────┘

4. The Complete Data Flow (Step by Step)

① Controller starts Reflector
   Reflector calls List → fetches ALL resources → stores in DeltaFIFO
   Reflector calls Watch → streams future change events into DeltaFIFO

② Controller's processLoop() calls DeltaFIFO.Pop()
   Pops events one by one → passes to HandleDeltas callback

③ HandleDeltas: if event type is Add / Update / Delete
   → stores/updates/removes object in Indexer (local cache)
   → Indexer is backed by ThreadSafeMap (concurrent-safe in-memory store)

④ HandleDeltas also calls distribute()
   → fans out event to all registered ShareInformer listeners

⑤ Your controller registers via informer.AddEventHandler()
   On event: adds resource key to WorkQueue

⑥ Your controller's worker goroutine calls WorkQueue.Get()
   → runs reconciliation logic
   → if error: re-enqueues with rate limiting for retry

⚠️ Important distinction: The "Controller" in steps ①② is an Informer-internal component (k8s.io/client-go/tools/cache/controller.go) that manages the Informer's startup flow. The "controller" in steps ⑤⑥ is your application controller that reconciles resource state.

5. Best Practice: Clientset + Informer

The recommended pattern in Kubernetes development:

┌─────────────────────────────────────────────────┐
│  Use Clientset for:  Create / Update / Delete   │
│  Use Informer for:   Read / List / Watch        │
└─────────────────────────────────────────────────┘

Never use clientset.CoreV1().Pods().List() in a hot path — always query the Informer's local cache via Lister.

6. Code Walkthrough

Step 1 — Build a Clientset from kubeconfig

func GetKubeClient(cfgpath string) (*kubernetes.Clientset, *restclient.Config) {
    kubeconfig, err := clientcmd.BuildConfigFromFlags("", cfgpath)
    if err != nil {
        panic(err)
    }
    clientset, err := kubernetes.NewForConfig(kubeconfig)
    if err != nil {
        panic(err)
    }
    return clientset, kubeconfig
}

Step 2 — Define the Controller struct

type KubeController struct {
    kubeConfig  *restclient.Config
    status      int32
    clientset   *kubernetes.Clientset
    factory     informers.SharedInformerFactory

    // One Informer + Lister + HasSynced per resource type
    podInformer coreinformers.PodInformer
    podsLister  corelisters.PodLister
    podsSynced  cache.InformerSynced
    // ... deployments, services, etc.
}

Step 3 — Initialize with SharedInformerFactory

func NewKubeController(
    kubeConfig *restclient.Config,
    clientset *kubernetes.Clientset,
    defaultResync time.Duration,
) *KubeController {
    kc := &KubeController{kubeConfig: kubeConfig, clientset: clientset}

    // SharedInformerFactory ensures all informers for the same resource
    // share ONE Reflector — no duplicate List/Watch calls
    // defaultResync=0: List once at startup, no periodic re-sync
    kc.factory = informers.NewSharedInformerFactory(clientset, defaultResync)

    kc.podInformer = kc.factory.Core().V1().Pods()
    kc.podsLister  = kc.podInformer.Lister()
    kc.podsSynced  = kc.podInformer.Informer().HasSynced

    return kc
}

Step 4 — Start the factory and wait for cache sync

func (kc *KubeController) Run(stopCh chan struct{}) {
    defer utilruntime.HandleCrash()

    // Start all informers registered with this factory
    kc.factory.Start(stopCh)

    // Block until the initial List is complete and cache is populated
    if !cache.WaitForCacheSync(stopCh,
        kc.podsSynced,
        kc.deploymentsSynced,
        kc.servicesSynced,
    ) {
        utilruntime.HandleError(fmt.Errorf("timed out waiting for caches to sync"))
        return
    }

    kc.status = 1 // signal: cache is ready, safe to query
    <-stopCh
}

Step 5 — Query the cache via Lister (never hit the API Server)

func main() {
    clientset, kubeConfig := GetKubeClient("conf/config")
    kc := NewKubeController(kubeConfig, clientset, time.Second*3000)

    stopCh := make(chan struct{})
    go kc.Run(stopCh)

    // Wait for cache to be ready before querying
    for kc.status != 1 {
        time.Sleep(time.Second)
    }

    // Query from local cache — fast, no API Server call
    pod, err := kc.podsLister.Pods("tech-daily").Get("hello-omega-deployment-7d8ff89d87-25kb4")
    if err != nil {
        log.Fatalf("get pod err: %s", err)
    }
    log.Printf("pod is running on node: %s", pod.Spec.NodeName)
}

7. SharedInformer: Why It Matters

If you instantiate multiple Informers for the same resource type, each one creates its own Reflector and makes its own List/Watch call — wasting resources and adding API Server load.

SharedInformer solves this by maintaining a single Reflector per resource type, shared across all consumers:

// Internal structure of SharedInformerFactory
type sharedInformerFactory struct {
    client           kubernetes.Interface
    lock             sync.Mutex
    defaultResync    time.Duration
    customResync     map[reflect.Type]time.Duration

    // Key insight: one SharedIndexInformer per resource type
    // All calls to factory.Core().V1().Pods() return the SAME informer
    informers        map[reflect.Type]cache.SharedIndexInformer
    startedInformers map[reflect.Type]bool
}

Without SharedInformer:
Controller A → Reflector A → List/Watch Pods
Controller B → Reflector B → List/Watch Pods   ← duplicate!
Controller C → Reflector C → List/Watch Pods   ← duplicate!

With SharedInformerFactory:
Controller A ──┐
Controller B ──┼──► Single Reflector → List/Watch Pods once  ✅
Controller C ──┘

8. Summary

Component	Role
Reflector	Calls List/Watch against API Server; feeds events into DeltaFIFO
DeltaFIFO	FIFO queue storing resource change events with their delta type
Controller (internal)	Drives the Informer loop; pops from DeltaFIFO via processLoop
HandleDeltas	Routes events to Indexer (for storage) and ShareInformer (for callbacks)
Indexer	Thread-safe in-memory cache; backed by ThreadSafeMap
Lister	Read interface over Indexer — fast local queries, no API Server calls
ShareInformer	Fan-out mechanism; delivers events to all AddEventHandler subscribers
WorkQueue	Rate-limited, deduplicating queue; bridges event callbacks and controller workers

Informer is the backbone of every Kubernetes controller. Once you understand its List/Watch → DeltaFIFO → Indexer → WorkQueue pipeline, the architecture of kube-controller-manager, custom controllers, and Operators all become clear.

Next in this series: Reflector: How client-go Syncs with the API Server (Part 2)

Follow the series for more deep dives into Kubernetes development.

推荐订阅源

DEV Community