Netflix has reworked its Conductor workflow orchestration engine to address growing scale across the company. Conductor now supports roughly 200,000 workflow definitions across 150 applications and executes about 420 million workflows per month. The redesign increases the supported workflow size from approximately 2,500 to 30,000 tasks while reducing p99 workflow evaluation latency by about 40%.

Conductor architecture evolution from versions 1.0 through 3.0 (Source: Netflix Blog Post)

Conductor orchestrates distributed workflows across Netflix’s Content and Studio Engineering, Ads, and Games organizations. Its evolution included migrating execution data from Dynomite to Cassandra, moving large task inputs and outputs to Amazon S3, and replacing DynoQueues with Timestone. Kafka was later introduced to decouple indexing from the execution path, with Elasticsearch and Iceberg supporting indexing and long-term storage.

With Conductor 4.0, Netflix identified workflow evaluation as a remaining bottleneck. Earlier versions loaded complete workflow state into memory during evaluation, creating pressure as workflows grew. The new design separates workflow metadata from task and user data, storing tasks independently. The evaluator uses a lightweight workflow blueprint and loads only the task data needed for the next decision.

Earlier Conductor community discussions had surfaced some of these scaling challenges. In 2022, a user reported 55,745 workflows and 310,000 tasks driving JVM heap usage to 5 GB. Aravind Ramkumar, a Conductor maintainer at Netflix, explained that the engine loaded the entire running workflow into memory for evaluation and that loading only the necessary portions was on the roadmap. Another 2022 discussion described a 1.7 MB workflow containing nearly 5,000 task entries. The user reported that storing the workflow definition took close to two minutes and that repeatedly loading the definition from the database affected execution time

Conductor 4.0 Architecture (Source: Netflix Blog Post)

The redesign also removes locking from task state coordination by storing pending and terminal task states separately and reconciling them in the application layer, with terminal state taking precedence. Workflow evaluation was moved out of the synchronous request path, with updates placed into exclusive Timestone queues for sequential asynchronous processing. Netflix reports that failed lock acquisition attempts, which reached about 2,700 per interval during contention, dropped to essentially zero.

Earlier discussions also highlighted worker scaling constraints. In one Conductor discussion, a user reported 25,000 to 30,000 running workflows with HTTP task queues building up. Ramkumar advised that increasing polling counts would not improve processing and could overload the system, recommending horizontal scaling instead.

Conductor 4.0 adds native concurrency controls, dynamic worker allocation, and a type-safe Java Workflow SDK, used by Netflix Ads for creative ingestion and Data Clean Room workflows. Conductor uses task-based orchestration with workflow definitions and a separate execution engine. Both target long-running distributed workflows.

Netflix expects workflow demand to potentially grow fivefold as it expands into areas including live content, games, and podcasts. Netflix discontinued maintenance of its public Conductor OSS repository in December 2023, citing a shift toward its internal Conductor fork, while Community-contributed modules and extensions remain in the separate Conductor community repository.