python: Pipeline Pattern

博客园 - ®Geovin Du Dream Park™

go: Functional Options Pattern go:Timing Functions Pattern python: Timing Functions Pattern go: Steady-State Pattern python: Steady-State Pattern go: Handshaking Pattern python: Handshaking Pattern go: Fail-Fast Pattern python: Fail-Fast Pattern I go: Deadline Pattern python: Deadline Pattern go: Circuit-Breaker Pattern python: Circuit-Breaker Pattern go: Bulkheads Pattern python: Bulkheads Pattern go: Push & Pull Pattern python: Push & Pull Pattern python: Publish/Subscribe Pattern II go: Publish/Subscribe Pattern python: Publish/Subscribe Pattern go: Futures & Promises Pattern python: Futures & Promises Pattern go: Worker Pool Pattern go：Pipeline Pattern python: Worker Pool Pattern go: Fan-Out Pattern python: Fan-Out Pattern go: Fan-In Pattern python: Fan-In Pattern Fan-In go: Producer Consumer Pattern python: Producer Consumer Pattern go: Parallelism Pattern python: Parallelism Pattern go: Reactor Pattern python: Reactor Pattern go: Generators Pattern python: speech to text offline python: Generators Pattern go: Coroutines Pattern python:Coroutines Pattern go: Broadcast Pattern python: Broadcast Pattern python: Bounded Parallelism Pattern go: Bounded Parallelism Pattern python: N-Barrier Pattern go: N-Barrier Pattern python: Semaphore Pattern go: Semaphore Pattern python: Read-Write Lock Pattern go: Read-Write Lock Pattern python: Monitor Pattern go: Monitor Pattern python: Mutex Pattern go: Lock/Mutex Pattern Python: Condition Variable Pattern go:Condition Variable Pattern python: Registry Pattern python: Interpreter Pattern go: Interpreter Pattern go: Registry Pattern go: Memento Pattern go: Command Pattern go: State Pattern go:Template Method Pattern go: Strategy Pattern go: Visitor Pattern go: Observer Pattern go: Mediator Pattern go: Iterator Pattern go: Chain of Responsibility Pattern go: Proxy Pattern go:Decorator Pattern go: Facade Pattern go: Flyweight Pattern go: Composite Pattern go: Singleton Pattern go: Prototype Pattern go: Bridge Pattern go: Adapter Pattern go: Builder Pattern 密码进行加盐哈希 using CSharp,Python,Go,Java go: Abstract Factory Pattern go: Model，Interface，DAL ，Factory，BLL using mysql go: Simple Factory Pattern go: Factory Method Pattern go: goLang 在Windows环境搭建Go语言开发环境 CSharp: Parallel Extensions cpp: class python: 蝴蝶跟随鼠标 javascript: 中国历史人物热力分布图using echart javascript: 中国历史人物热力图 python: 初养龙虾微信纯文字自动回复using workBuddy python: object 入门 python: Factory Method Pattern python: Simple Factory Pattern python: Abstract Factory Pattern 区域化代码 python: Null Object Pattern python: Builder Pattern python: Adapter Pattern

®Geovin Du Dream Park™ · 2026-06-20 · via 博客园 - ®Geovin Du Dream Park™

项目结构：

多阶段链式处理（原材料→加工→质检→镶嵌→成品→销售）
每个阶段独立运行（并发执行）
阶段间通过 ** 通道（Channel）** 传递数据
每个阶段读取输入通道、写入输出通道
处理完成后自动关闭输出通道

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:25 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : config.py

import logging

# 流水线全局配置
PIPELINE_TIMEOUT = 1.0
STAGE_RETRY_TIMES = 2
MAX_REWORK_COUNT = 2
PROCESS_SLEEP_SEC = 0.2

# 日志配置
LOG_FORMAT = "%(asctime)s [%(levelname)s] %(message)s"
LOG_DATE_FMT = "%Y-%m-%d %H:%M:%S"

def init_logger():
    logging.basicConfig(
        format=LOG_FORMAT,
        level=logging.INFO,
        datefmt=LOG_DATE_FMT
    )
    return logging.getLogger("JewelryEnterprisePipe")

logger = init_logger()




# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:42 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : defect.py
from dataclasses import dataclass
from typing import Any

@dataclass
class DefectItem:
    """
    次品记录模型
    """
    sn: int
    fail_stage: str
    error_msg: str
    can_rework: bool
    origin_data: Any


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:40 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : diamond.py
from dataclasses import dataclass
from PipelinePattern.model.raw_material import RawStone

@dataclass
class CutDiamond:
    """

    """
    sn: int
    carat: float
    cut_grade: str
    raw_data: RawStone
    rework_count: int = 0

@dataclass
class ClarityCheckResult:
    """

    """
    sn: int
    clarity_grade: str
    pass_clarity: bool
    diamond: CutDiamond  # 绑定原始裸石数据

@dataclass
class ColorCheckResult:
    """

    """
    sn: int
    color_grade: str
    pass_color: bool
    diamond: CutDiamond  # 绑定原始裸石数据

@dataclass
class QCMergedResult:
    """

    """
    sn: int
    carat: float
    cut: str
    clarity: str
    color: str
    pass_all_qc: bool
    diamond: CutDiamond


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:41 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : jewelry_product.py
from dataclasses import dataclass
from PipelinePattern.model.diamond import QCMergedResult

@dataclass
class InlaidJewelry:
    """
     镶嵌、成品模型
    """
    sn: int
    metal_material: str
    diamond_qc: QCMergedResult
    labor_cost: float

@dataclass
class FinishedProduct:
    """
    成品模型
    """
    sn: int
    product_name: str
    sale_price: float
    inlaid_info: InlaidJewelry


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:40 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : raw_material.py
from dataclasses import dataclass

@dataclass
class RawStone:
    """
    原石原料模型
    """
    sn: int
    carat: float
    origin: str

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:41 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : sale.py
from dataclasses import dataclass
from PipelinePattern.model.jewelry_product import FinishedProduct

@dataclass
class SaleOrder:
    """
    销售单据
    """
    sn: int
    goods_name: str
    final_price: float
    product: FinishedProduct

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:45 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : cut_service.py
import asyncio
from PipelinePattern.model.raw_material import RawStone
from PipelinePattern.model.diamond import CutDiamond
from PipelinePattern.config.config import PROCESS_SLEEP_SEC

async def cut_raw_stone(raw: RawStone) -> CutDiamond:
    """
    原石切割打磨业务逻辑，单一职责：仅转换原石为裸石
    :param raw:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC)
    grade = "EX" if raw.carat >= 1.0 else "VG"
    return CutDiamond(
        sn=raw.sn,
        carat=raw.carat,
        cut_grade=grade,
        raw_data=raw
    )


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:44 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : finish_service.py
import asyncio
from PipelinePattern.model.jewelry_product import InlaidJewelry, FinishedProduct
from PipelinePattern.config.config import PROCESS_SLEEP_SEC

async def finish_product(inlay: InlaidJewelry) -> FinishedProduct:
    """
    成品定价包装
    :param inlay:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC)
    base_cost = inlay.labor_cost + inlay.diamond_qc.carat * 8500
    sell_price = round(base_cost * 1.75, 2)
    name = f"{inlay.metal_material}钻戒-{inlay.sn}"
    return FinishedProduct(
        sn=inlay.sn,
        product_name=name,
        sale_price=sell_price,
        inlaid_info=inlay
    )

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:43 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : inlay_service.py
import asyncio
import random
from PipelinePattern.model.diamond import QCMergedResult
from PipelinePattern.model.jewelry_product import InlaidJewelry
from PipelinePattern.config.config import PROCESS_SLEEP_SEC

async def metal_inlay(qc_data: QCMergedResult) -> InlaidJewelry:
    """
    贵金属镶嵌，仅处理通过质检钻石
    :param qc_data:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC)
    if not qc_data.pass_all_qc:
        raise Exception("钻石质检未通过，禁止镶嵌加工")
    metal_list = ["18K黄金", "PT950铂金", "S925银"]
    metal = random.choice(metal_list)
    labor = qc_data.carat * 1200
    return InlaidJewelry(
        sn=qc_data.sn,
        metal_material=metal,
        diamond_qc=qc_data,
        labor_cost=labor
    )


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:41 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : qc_service.py
import asyncio
import random
from PipelinePattern.model.diamond import CutDiamond, ClarityCheckResult, ColorCheckResult, QCMergedResult
from PipelinePattern.config.config import PROCESS_SLEEP_SEC


async def clarity_detect(dia: CutDiamond) -> ClarityCheckResult:
    """
    净度检测分支业务
    :param dia:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC)
    grade = random.choice(["VS1", "VS2", "SI1"])
    pass_flag = grade != "SI1" or dia.carat > 0.7
    return ClarityCheckResult(
        sn=dia.sn,
        clarity_grade=grade,
        pass_clarity=pass_flag,
        diamond=dia
    )

async def color_detect(dia: CutDiamond) -> ColorCheckResult:
    """

    :param dia:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC)
    grade = random.choice(["D", "G", "J"])
    pass_flag = grade in ("D", "G")
    return ColorCheckResult(
        sn=dia.sn,
        color_grade=grade,
        pass_color=pass_flag,
        diamond=dia
    )

def merge_qc_result(dia: CutDiamond, clarity: ClarityCheckResult, color: ColorCheckResult) -> QCMergedResult:
    """

    :param dia:
    :param clarity:
    :param color:
    :return:
    """
    all_pass = clarity.pass_clarity and color.pass_color
    return QCMergedResult(
        sn=dia.sn,
        carat=dia.carat,
        cut=dia.cut_grade,
        clarity=clarity.clarity_grade,
        color=color.color_grade,
        pass_all_qc=all_pass,
        diamond=dia
    )




# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:45 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : sale_service.py
import asyncio
from PipelinePattern.model.jewelry_product import FinishedProduct
from PipelinePattern.model.sale import SaleOrder
from PipelinePattern.config.config import PROCESS_SLEEP_SEC

async def create_sale_order(goods: FinishedProduct) -> SaleOrder:
    """
    销售出库
    :param goods:
    :return:
    """
    await asyncio.sleep(PROCESS_SLEEP_SEC / 2)
    return SaleOrder(
        sn=goods.sn,
        goods_name=goods.product_name,
        final_price=goods.sale_price,
        product=goods
    )

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:27 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : channel.py
import asyncio
from typing import Optional

# 通道终止标记，统一约定
STOP_SIGNAL = None

class PipeChannel:
    """
    通道包装类，统一队列创建规范
    """
    @staticmethod
    def new_queue(maxsize: int = 0) -> asyncio.Queue:
        """

        :param maxsize:
        :return:
        """
        return asyncio.Queue(maxsize=maxsize)

    @staticmethod
    async def close_channel(q: asyncio.Queue):
        """
        写入终止信号关闭通道
        :param q:
        :return:
        """
        await q.put(STOP_SIGNAL)


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:38 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : defect_rework.py
import asyncio
from PipelinePattern.core.channel import STOP_SIGNAL
from PipelinePattern.config.config import logger
from PipelinePattern.model.defect import DefectItem

async def defect_rework_consumer(
    defect_q: asyncio.Queue,
    rework_input_q: asyncio.Queue,
    scrap_q: asyncio.Queue,
    max_rework: int
):
    """
     次品与返工通用处理器
    通用次品分拣：区分返工/报废，回流到上游切割通道
    :param defect_q:
    :param rework_input_q:
    :param scrap_q:
    :param max_rework:
    :return:
    """
    logger.info("次品返工分拣中心启动")
    while True:
        try:
            raw_data = await asyncio.wait_for(defect_q.get(), 2)
            if raw_data == STOP_SIGNAL:
                break
            item, stage, err = raw_data
            current_rework_cnt = getattr(item, "rework_count", 0)

            if current_rework_cnt < max_rework:
                item.rework_count = current_rework_cnt + 1
                logger.info(f"【返工】SN:{item.sn} 第{item.rework_count}次回流切割工序")
                await rework_input_q.put(item)
            else:
                defect_record = DefectItem(
                    sn=item.sn,
                    fail_stage=stage,
                    error_msg=err,
                    can_rework=False,
                    origin_data=item
                )
                logger.warning(f"【报废】SN:{item.sn} 超出最大返工次数")
                await scrap_q.put(defect_record)
            defect_q.task_done()
        except asyncio.TimeoutError:
            continue
    await scrap_q.put(STOP_SIGNAL)
    logger.info("次品分拣协程退出")


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:37 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : fork_join.py
import asyncio
from PipelinePattern.core.channel import STOP_SIGNAL
from PipelinePattern.config.config import logger

async def split_stream(source_q: asyncio.Queue, dest_q_list: list[asyncio.Queue]):
    """
    单输入流复制分发到N个输出队列（解决多分支争抢同一数据源）
    :param source_q:
    :param dest_q_list:
    :return:
    """
    logger.info(f"数据流分流器启动，目标分支数：{len(dest_q_list)}")
    while True:
        try:
            item = await asyncio.wait_for(source_q.get(), 1)
            if item == STOP_SIGNAL:
                for q in dest_q_list:
                    await q.put(STOP_SIGNAL)
                break
            # 复制数据下发所有分支队列
            for q in dest_q_list:
                await q.put(item)
            source_q.task_done()
        except asyncio.TimeoutError:
            continue
    logger.info("数据流分流器执行完毕")

async def fork_branch_worker(
    input_q: asyncio.Queue,
    output_q: asyncio.Queue,
    branch_func
):
    """
    通用分支协程，用于并行检测
    :param input_q:
    :param output_q:
    :param branch_func:
    :return:
    """
    logger.info("启动并行分支检测协程")
    while True:
        try:
            item = await asyncio.wait_for(input_q.get(), 1)
            if item == STOP_SIGNAL:
                break
            res = await branch_func(item)
            await output_q.put(res)
            input_q.task_done()
        except asyncio.TimeoutError:
            continue
    await output_q.put(STOP_SIGNAL)

async def join_merger(
    clarity_q: asyncio.Queue,
    color_q: asyncio.Queue,
    out_merge_q: asyncio.Queue,
    merge_func
):
    """
    通用结果合并器，按SN匹配两路并行输出（移除对原切割队列依赖）
    :param clarity_q:
    :param color_q:
    :param out_merge_q:
    :param merge_func:
    :return:
    """
    logger.info("启动并行结果合并协程")
    clarity_map = {}
    color_map = {}

    # 消费两路质检分支全部数据
    async def drain_queue(q, data_map):
        while True:
            val = await q.get()
            if val == STOP_SIGNAL:
                break
            data_map[val.sn] = val

    await asyncio.gather(
        drain_queue(clarity_q, clarity_map),
        drain_queue(color_q, color_map)
    )

    # 按SN匹配两路质检结果，还原钻石基础数据
    all_sn = set(clarity_map.keys()) & set(color_map.keys())
    for sn in all_sn:
        c = clarity_map[sn]
        col = color_map[sn]
        merged = merge_func(c.diamond, c, col)
        await out_merge_q.put(merged)

    await out_merge_q.put(STOP_SIGNAL)
    logger.info("并行质检结果合并完成")


# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:27 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : metrics.py
from typing import Dict

class PipelineMetrics:
    """
    指标统计
    """
    def __init__(self):
        self._data: Dict[str, Dict[str, int]] = {}

    def init_stage(self, stage_name: str):
        """

        :param stage_name:
        :return:
        """
        if stage_name not in self._data:
            self._data[stage_name] = {
                "total": 0,
                "success": 0,
                "fail": 0,
                "retry": 0
            }

    def inc_total(self, stage_name: str):
        """

        :param stage_name:
        :return:
        """
        self.init_stage(stage_name)
        self._data[stage_name]["total"] += 1

    def inc_success(self, stage_name: str):
        """

        :param stage_name:
        :return:
        """
        self.init_stage(stage_name)
        self._data[stage_name]["success"] += 1

    def inc_fail(self, stage_name: str):
        """

        :param stage_name:
        :return:
        """
        self.init_stage(stage_name)
        self._data[stage_name]["fail"] += 1

    def inc_retry(self, stage_name: str):
        """

        :param stage_name:
        :return:
        """
        self.init_stage(stage_name)
        self._data[stage_name]["retry"] += 1

    def get_all(self) -> Dict:
        """

        :return:
        """
        return self._data.copy()

# 全局单例指标
global_metrics = PipelineMetrics()

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 12:34 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : pipeline_stage.py
import asyncio
from typing import Callable, Optional, Awaitable
from PipelinePattern.core.channel import PipeChannel, STOP_SIGNAL
from PipelinePattern.core.metrics import global_metrics
from PipelinePattern.config.config import logger, PIPELINE_TIMEOUT

async def base_pipeline_stage(
    stage_name: str,
    input_q: asyncio.Queue,
    output_q: asyncio.Queue,
    defect_q: asyncio.Queue,
    process_func: Callable[[any], Awaitable[any]],
    retry_times: int
):
    """
     通用管道阶段执行器，纯底层逻辑，无业务耦合
    :param stage_name:
    :param input_q:
    :param output_q:
    :param defect_q:
    :param process_func:
    :param retry_times:
    :return:
    """
    logger.info(f"【阶段启动】{stage_name}")
    has_data = True

    while has_data:
        try:
            item = await asyncio.wait_for(input_q.get(), timeout=PIPELINE_TIMEOUT)
            if item == STOP_SIGNAL:
                has_data = False
                input_q.task_done()
                break

            global_metrics.inc_total(stage_name)
            process_ok = False
            result = None
            err_msg = ""

            for attempt in range(retry_times + 1):
                try:
                    result = await asyncio.wait_for(process_func(item), timeout=PIPELINE_TIMEOUT)
                    process_ok = True
                    break
                except Exception as e:
                    err_msg = str(e)
                    global_metrics.inc_retry(stage_name)
                    logger.warning(f"【{stage_name}】SN:{item.sn} 第{attempt+1}次失败: {err_msg}")
                    await asyncio.sleep(0.1)

            if process_ok:
                global_metrics.inc_success(stage_name)
                await output_q.put(result)
            else:
                global_metrics.inc_fail(stage_name)
                await defect_q.put((item, stage_name, err_msg))

            input_q.task_done()

        except asyncio.TimeoutError:
            # 超时无数据不阻塞，继续轮询，不会卡死
            continue

    await PipeChannel.close_channel(output_q)
    logger.info(f"【阶段结束】{stage_name}")

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述：Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:46 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : jewelry_pipe_factory.py
import asyncio
from PipelinePattern.core.channel import PipeChannel
from PipelinePattern.core.pipeline_stage import base_pipeline_stage
from PipelinePattern.core.fork_join import fork_branch_worker, join_merger,split_stream
from PipelinePattern.core.defect_rework import defect_rework_consumer
from PipelinePattern.service import cut_service, qc_service, inlay_service, finish_service, sale_service
from PipelinePattern.config.config import STAGE_RETRY_TIMES, MAX_REWORK_COUNT

class JewelryPipelineFactory(object):
    """
    重构流水线编排，增加分流器，消除队列争抢
    """
    @staticmethod
    async def create_full_pipeline(source_queue: asyncio.Queue):
        """

        :param source_queue:
        :return:
        """
        # 1. 通道初始化
        q_cut_out = PipeChannel.new_queue()
        # 并行分支独立输入队列
        q_clarity_in = PipeChannel.new_queue()
        q_color_in = PipeChannel.new_queue()
        q_clarity_out = PipeChannel.new_queue()
        q_color_out = PipeChannel.new_queue()
        q_qc_merge = PipeChannel.new_queue()
        q_inlay_out = PipeChannel.new_queue()
        q_finish_out = PipeChannel.new_queue()
        q_sale_out = PipeChannel.new_queue()

        q_defect_pool = PipeChannel.new_queue()
        q_scrap_pool = PipeChannel.new_queue()

        # 任务列表统一管理
        all_tasks = []

        # 阶段1：原石切割
        task_cut = asyncio.create_task(base_pipeline_stage(
            stage_name="1-原石切割",
            input_q=source_queue,
            output_q=q_cut_out,
            defect_q=q_defect_pool,
            process_func=cut_service.cut_raw_stone,
            retry_times=STAGE_RETRY_TIMES
        ))
        all_tasks.append(task_cut)

        # 新增：切割输出分流器 → 复制数据给净度、颜色两个分支，解决数据争抢
        task_splitter = asyncio.create_task(split_stream(q_cut_out, [q_clarity_in, q_color_in]))
        all_tasks.append(task_splitter)

        # 并行Fork两路质检
        task_clarity_branch = asyncio.create_task(fork_branch_worker(
            input_q=q_clarity_in,
            output_q=q_clarity_out,
            branch_func=qc_service.clarity_detect
        ))
        task_color_branch = asyncio.create_task(fork_branch_worker(
            input_q=q_color_in,
            output_q=q_color_out,
            branch_func=qc_service.color_detect
        ))
        all_tasks.extend([task_clarity_branch, task_color_branch])

        # Join合并两路质检（不再依赖切割队列）
        task_qc_merge = asyncio.create_task(join_merger(
            clarity_q=q_clarity_out,
            color_q=q_color_out,
            out_merge_q=q_qc_merge,
            merge_func=qc_service.merge_qc_result
        ))
        all_tasks.append(task_qc_merge)

        # 串行后置工序
        task_inlay = asyncio.create_task(base_pipeline_stage(
            stage_name="2-贵金属镶嵌",
            input_q=q_qc_merge,
            output_q=q_inlay_out,
            defect_q=q_defect_pool,
            process_func=inlay_service.metal_inlay,
            retry_times=STAGE_RETRY_TIMES
        ))
        task_finish = asyncio.create_task(base_pipeline_stage(
            stage_name="3-成品定价",
            input_q=q_inlay_out,
            output_q=q_finish_out,
            defect_q=q_defect_pool,
            process_func=finish_service.finish_product,
            retry_times=STAGE_RETRY_TIMES
        ))
        task_sale = asyncio.create_task(base_pipeline_stage(
            stage_name="4-销售出库",
            input_q=q_finish_out,
            output_q=q_sale_out,
            defect_q=q_defect_pool,
            process_func=sale_service.create_sale_order,
            retry_times=STAGE_RETRY_TIMES
        ))
        all_tasks.extend([task_inlay, task_finish, task_sale])

        # 次品返工协程
        task_defect_handle = asyncio.create_task(defect_rework_consumer(
            defect_q=q_defect_pool,
            rework_input_q=source_queue,
            scrap_q=q_scrap_pool,
            max_rework=MAX_REWORK_COUNT
        ))
        all_tasks.append(task_defect_handle)

        return q_sale_out, q_scrap_pool, all_tasks

调用：

# encoding: utf-8 
# 版权所有  2026 ©涂聚文有限公司™ ®
# 许可信息查看：言語成了邀功盡責的功臣，還需要行爲每日來值班嗎
# 描述： Pipeline Pattern 管道模式
# Author    : geovindu,Geovin Du 涂聚文.
# IDE       : PyCharm 2024.3.6 python 3.11
# os        : windows 10
# database  : mysql 9.0 sql server 2019, postgreSQL 17.0  Oracle 21c Neo4j
# Datetime  : 2026/6/20 21:48 
# User      :  geovindu
# Product   : PyCharm
# Project   : pydesginpattern
# File      : PipelineBll.py
import asyncio
from PipelinePattern.core.channel import PipeChannel, STOP_SIGNAL
from PipelinePattern.core.metrics import global_metrics
from PipelinePattern.factory.jewelry_pipe_factory import JewelryPipelineFactory
from PipelinePattern.model.raw_material import RawStone
from PipelinePattern.config.config import logger

class  PipelineBll(object):
    """

    """
    async def demo(self):
        """

        :return:
        """
        logger.info("===== 企业级珠宝异步Pipeline启动（并行质检+次品返工）=====")
        source_q = PipeChannel.new_queue()

        # 灌入测试原石原料
        raw_list = [
            RawStone(sn=2001, carat=0.62, origin="南非"),
            RawStone(sn=2002, carat=1.15, origin="博茨瓦纳"),
            RawStone(sn=2003, carat=0.48, origin="俄罗斯"),
            RawStone(sn=2004, carat=1.32, origin="南非"),
            RawStone(sn=2005, carat=0.75, origin="澳大利亚"),
            RawStone(sn=2007, carat=0.90, origin="纳米比亚"),
        ]
        for stone in raw_list:
            await source_q.put(stone)
            logger.info(f"原料进料 SN:{stone.sn}")
        # 写入终止信号，告知上游无更多原料
        await source_q.put(STOP_SIGNAL)

        # 工厂创建完整流水线
        sale_output_q, scrap_q, pipe_tasks = await JewelryPipelineFactory.create_full_pipeline(source_q)

        # 消费销售出库结果
        sale_records = []
        while True:
            order = await sale_output_q.get()
            if order == STOP_SIGNAL:
                break
            sale_records.append(order)
            logger.info(f"✅ 销售单据 SN:{order.sn} 售价:{order.final_price}")

        # 等待所有流水线协程执行完毕
        await asyncio.gather(*pipe_tasks)

        # 输出全局监控报表
        logger.info("\n========== 流水线执行统计报表 ==========")
        logger.info(f"总投入原石：{len(raw_list)} 件")
        logger.info(f"成功销售出库：{len(sale_records)} 件")
        all_metrics = global_metrics.get_all()
        for stage, stat in all_metrics.items():
            logger.info(f"{stage} 指标：{stat}")

输出：

哲学管理(学)人生, 文学艺术生活, 自动(计算机学)物理(学)工作, 生物(学)化学逆境, 历史(学)测绘(学)时间, 经济(学)数学金钱(理财), 心理(学)医学情绪, 诗词美容情感, 美学建筑(学)家园, 解构建构(分析)整合学习, 智商情商(IQ、EQ)运筹(学)生存.---Geovin Du(涂聚文）

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

博客园 - ®Geovin Du Dream Park™