惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Microsoft Azure Blog
Microsoft Azure Blog
Google Online Security Blog
Google Online Security Blog
D
Darknet – Hacking Tools, Hacker News & Cyber Security
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Simon Willison's Weblog
Simon Willison's Weblog
T
Threat Research - Cisco Blogs
C
CXSECURITY Database RSS Feed - CXSecurity.com
L
Lohrmann on Cybersecurity
www.infosecurity-magazine.com
www.infosecurity-magazine.com
Forbes - Security
Forbes - Security
P
Palo Alto Networks Blog
Schneier on Security
Schneier on Security
S
Schneier on Security
T
Tor Project blog
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
WordPress大学
WordPress大学
The Hacker News
The Hacker News
Hacker News - Newest:
Hacker News - Newest: "LLM"
罗磊的独立博客
Application and Cybersecurity Blog
Application and Cybersecurity Blog
F
Fortinet All Blogs
博客园 - 三生石上(FineUI控件)
小众软件
小众软件
C
Check Point Blog
Stack Overflow Blog
Stack Overflow Blog
Blog — PlanetScale
Blog — PlanetScale
雷峰网
雷峰网
S
Security @ Cisco Blogs
PCI Perspectives
PCI Perspectives
Spread Privacy
Spread Privacy
W
WeLiveSecurity
SecWiki News
SecWiki News
A
About on SuperTechFans
H
Help Net Security
博客园 - 司徒正美
Recent Commits to openclaw:main
Recent Commits to openclaw:main
爱范儿
爱范儿
S
Securelist
M
MIT News - Artificial intelligence
云风的 BLOG
云风的 BLOG
月光博客
月光博客
Jina AI
Jina AI
博客园 - 叶小钗
Vercel News
Vercel News
阮一峰的网络日志
阮一峰的网络日志
Recent Announcements
Recent Announcements
S
Secure Thoughts
The Cloudflare Blog
美团技术团队
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More

Yesterday17's Blog

2026 新年解密红包 / Melody Flag | Yesterday17's Blog 谈谈 Iori 的设计思路(二):如何实现一个 Showroom 录制工具? | Yesterday17's Blog 谈谈 Iori 的设计思路(一):从 Nico Timeshift 说起 | Yesterday17's Blog Iori Minyami 0.1.0 发布 | Yesterday17's Blog 2025 新年解密红包 / Melody Flag | Yesterday17's Blog 使用 Cloudflare Warp 解决罗森票务的海外登录问题 | Yesterday17's Blog How To Blog 04: The Astro v5 Era | Yesterday17's Blog 谈谈 tokio::select! 的公平性 | Yesterday17's Blog Learning Pingora 05 - Connect with TLS | Yesterday17's Blog Leaving Bytedance | Yesterday17's Blog 大橋彩香 AsiaTour「Reflection」上海公演 个人向记录 & Repo | Yesterday17's Blog Recoving from burnout - What happened? | Yesterday17 Yubikey 重建手册 | Yesterday17's Blog How To Blog 03: Heimus | Yesterday17's Blog 🪧 Blog Migration Accouncement | Yesterday17's Blog Learn Your IDE - VSCode 是如何仅重启插件的? | Yesterday17's Blog How To Blog 02: Astro❤️Password | Yesterday17's Blog How To Blog 01: Why, How, and the Future | Yesterday17's Blog Learning Pingora 04 - Establish L4 Connection | Yesterday17's Blog Learning Pingora 03 - Upstreams and Peers | Yesterday17's Blog Learning Pingora 01 - Getting Started | Yesterday17's Blog 2024 新年解密红包 / Melody Flag | Yesterday17's Blog 向新的一年飞驰——记录 2023 | Yesterday17's Blog 「サクラノ刻」对话选摘(2) | Yesterday17's Blog PGP Key Revocation 注销声明 | Yesterday17's Blog 「サクラノ刻」对话选摘(1) | Yesterday17's Blog 2023 新年解密红包 / Melody Flag | Yesterday17's Blog 『蒼の彼方のフォーリズム』通关感想 | Yesterday17's Blog 单显卡直通教程 | Yesterday17's Blog 对博客与笔记的思考 | Yesterday17's Blog Project Anni 之旅(3)自动化 Flutter 应用 CI/CD 上架流程 | Yesterday17's Blog AsobiStage 直接播放链接 | Yesterday17 如何在后分P时代进行投稿——sswa使用详解 | Yesterday17's Blog JSON RPC 与 LSP 协议基础 | Yesterday17's Blog Grajapa Shueisha / BookEnd 加密方式调查 | Yesterday17's Blog 【2022篇+WriteUp】如何再收一个新年红包? | Yesterday17's Blog 如何将良心云的良心功能清理干净 | Yesterday17's Blog 【油猴脚本】bilibili 投稿页面返回旧版+旧版页面强制允许分P上传 | Yesterday17's Blog Cloudr1v1 授权方式分析 | Yesterday17 Typora 1.0.2 逆向实录 | Yesterday17's Blog Project Anni 之旅(2)ValueAfterTable——toml-rs的实现与限制 | Yesterday17's Blog IPv4透明代理+IPv6 Passthrough——树莓派单臂软路由折腾记 | Yesterday17's Blog Chaos; Child 汉化补丁 神秘编码探索 | Yesterday17's Blog 镣铐与舞蹈——个性与共性之迷思 | Yesterday17's Blog Go 学习笔记 02 - 找准 io 之道 | Yesterday17's Blog NAT Slipstreaming v1 原理浅析 | Yesterday17's Blog 绕过「9-nine-」的 CDKEY 验证——KrkrPlugin 正(?)向实录 | Yesterday17's Blog 静流的青春纪念册——「サクラノ刻 -櫻の森の下を歩む-」体验版感言 | Yesterday17's Blog Project Anni 之旅 01 - 从 clap-builder 到 derive | Yesterday17's Blog [Google CTF 2021] CPP WriteUp | Yesterday17's Blog 获取 アソビステージ 的实际播放链接 | Yesterday17's Blog 90 行 Rust 代码实现 AsyncTeeReader | Yesterday17's Blog 或许还算有价值一读的文章列表 | Yesterday17's Blog 从零开始的 Seedbox 之旅 | Yesterday17's Blog [随笔]技术型博客行文迷思(1) | Yesterday17's Blog 浅谈 git fetch 的工作方式 | Yesterday17's Blog 『ソーサレス*アライヴ! ~the World's End Fallen Star~』通关感想" | Yesterday17's Blog Rust std::fmt 格式语法简述 | Yesterday17's Blog 日亚修改居住国的解决方案 | Yesterday17's Blog [Windows/Linux] GC553 的 Switch 完美采集之路 | Yesterday17's Blog 【翻译】Subtyping and Variance / 子类型与变型 | Yesterday17's Blog Berd's Red Envelope 2021 WriteUp | Yesterday17's Blog 【中英对照】ALSA 音频 API 使用教程/A Tutorial on Using the ALSA Audio API | Yesterday17's Blog 从 cue_scanner.l 看 CUE Sheet 的词法单元 | Yesterday17's Blog Postman 历史记录导出的解决方案 | Yesterday17's Blog 《恋爱绮谭 不存在的夏天》通关感想 | Yesterday17's Blog [微机实验/TD-PITE] 微机接口综合实验 | Yesterday17's Blog [微机实验/TD-PITE] 键盘扫描及数码管显示实验 | Yesterday17's Blog [微机实验/TD-PITE] 数码管显示实验 | Yesterday17's Blog Airsonic Advanced+Google Drive+Caddy 部署纪实 | Yesterday17's Blog X-NUCA 2020 - hellowasm 题解 | Yesterday17's Blog [微机实验/TD-PITE] 8251 串行接口实验 | Yesterday17's Blog Node.js child_process.fork 与 env 污染 RCE | Yesterday17's Blog EP.01 「夜の向日葵」 | Yesterday17's Blog [微机实验/TD-PITE] 8254 定时/计数器实验+选做实验 | Yesterday17's Blog [JLU CTF/2020] babywasm WriteUp | Yesterday17's Blog PHP 反序列化与经典利用 | Yesterday17's Blog WebAssembly 逆向简述 | Yesterday17's Blog 『彼女、お借りします』一期完结点评 | Yesterday17's Blog [微机实验/TD-PITE] D/A 转换实验+选做实验 | Yesterday17's Blog [微机实验/TD-PITE] A/D 转换实验+选做实验 | Yesterday17's Blog 开源项目申请 JetBrains Open Source License 简单流程 | Yesterday17's Blog 微软拼音与 JetBrains 搜索快捷键冲突的解决方案 | Yesterday17's Blog [微机实验/TD-PITE] 8259 中断优先级实验+选做实验 | Yesterday17's Blog IFTTT 测试(续) | Yesterday17 IFTTT 测试 | Yesterday17's Blog [微机实验/TD-PITE] 存储器扩展实验+选做实验 | Yesterday17's Blog 新版 GCC 针对 -fdump-translation-unit 的替代方案 | Yesterday17's Blog 一次 HSTS 策略配置的排错之旅 | Yesterday17's Blog YukiNative 踩坑记——Windows 的消息队列 | Yesterday17's Blog 我是我自己——论获取 HTTPS 证书时的验证步骤 | Yesterday17's Blog 【设计文档】对 PUG 的大规模设计修订(1.1) | Yesterday17's Blog GS65 折腾记(2)加装固态,分区,Grub2 引导 Manjaro LiveCD | Yesterday17's Blog 「さくら、もゆ。」的空白字体列表——一次逆向问题定位过程实录 | Yesterday17's Blog GSuite 探索篇(1)使用 Service Account 向 Google Drive 传输文件 | Yesterday17's Blog 『サクラノ詩 -櫻の森の上を舞う-』通关感想 | Yesterday17's Blog 《ATRI -My Dear Moments-》通关感想 | Yesterday17's Blog [工具][VSCode 扩展] AegiKit——方便 Aegisub 使用的工具箱 | Yesterday17's Blog 贝塞尔曲线、字体矢量化与曲线运算 | Yesterday17's Blog NAT 类型初探 | Yesterday17's Blog
Learning Pingora 02 - A Simple HTTP Server | Yesterday17's Blog
Yesterday17 · 2024-03-30 · via Yesterday17's Blog

比较有趣的是,作为一个通常意义上用于「反向代理」的工具,Pingora 中也有可以实现 HTTP Server 的部分。与大部分 HTTP Server 类似,但也有 Pingora 的醍醐味在里面。话不多说,赶紧端上来吧(

ToC

  • Example: Logging
  • trait ServeHttp
  • struct HttpServer
  • HttpModule
    • trait HttpModule
    • trait HttpModuleBuilder
    • HttpModules
    • HttpModuleCtx
  • 回到 Prometheus
  • 最后看看……ResponseCompression

Example: Logging

官方示例里有一个 logging 的部分,代码如下:

pub struct MyGateway {

req_metric: prometheus::IntCounter,

}

#[async_trait]

impl ProxyHttp for MyGateway {

...

async fn logging(

&self,

session: &mut Session,

_e: Option<&pingora::Error>,

ctx: &mut Self::CTX,

) {

let response_code = session

.response_written()

.map_or(0, |resp| resp.status.as_u16());

// access log

info!(

"{} response code: {response_code}",

self.request_summary(session, ctx)

);

self.req_metric.inc();

}

fn main() {

...

let mut prometheus_service_http =

pingora::services::listening::Service::prometheus_http_service();

prometheus_service_http.add_tcp("127.0.0.1:6192");

my_server.add_service(prometheus_service_http);

my_server.run_forever();

}

可以看到,通过高亮部分的代码,这段 logging 的代码在 127.0.0.1:6192 上启动了一个 Prometheus metric server。这个 Server 是按什么逻辑去实现的呢?

trait ServeHttp

我们自底向上分解一下。首先来看 ServeHttp

/// This trait defines how to map a request to a response

#[cfg_attr(not(doc_async_trait), async_trait)]

pub trait ServeHttp {

/// Define the mapping from a request to a response.

/// Note that the request header is already read, but the implementation needs to read the

/// request body if any.

///

/// # Limitation

/// In this API, the entire response has to be generated before the end of this call.

/// So it is not suitable for streaming response or interactive communications.

/// Users need to implement their own [`super::HttpServerApp`] for those use cases.

async fn response(&self, http_session: &mut ServerSession) -> Response<Vec<u8>>;

}

这是一个标准又简单的 HTTP Server,将 Request 映射为 Response。这个 Trait 的定义也比较简单,不支持 streaming,只能一次性返回全部 Body 内容。

struct HttpServer

HttpServer 则是一个捆绑了 ServeHttp 和一系列 HttpModules 中间件的存在:

/// A helper struct for HTTP server with http modules embedded

pub struct HttpServer<SV> {

app: SV,

modules: HttpModules,

}

针对 SV: ServeHttp,它实现了 HttpServerApp。这实际也是约束了 app 必须是一个实现了 ServeHttpServer 实现:

#[cfg_attr(not(doc_async_trait), async_trait)]

impl<SV> HttpServerApp for HttpServer<SV>

where

SV: ServeHttp + Send + Sync,

{

async fn process_new_http(

self: &Arc<Self>,

mut http: ServerSession,

shutdown: &ShutdownWatch,

) -> Option<Stream> {

// ...

HttpModule

HttpModule 可以简单理解成用于修改 ServeHttp 接收和返回的中间件,用于修改请求和返回的内容。Pingora 中有一系列用于构造/使用 HttpModule 的方式。

trait HttpModule

我们先从最基本的 Trait 定义来看:

pub trait HttpModule {

fn request_header_filter(&mut self, _req: &mut RequestHeader) -> Result<()> {

Ok(())

}

fn request_body_filter(&mut self, body: Option<Bytes>) -> Result<Option<Bytes>> {

Ok(body)

}

fn response_filter(&mut self, _t: &mut HttpTask) -> Result<()> {

Ok(())

}

fn as_any(&self) -> &dyn Any;

fn as_any_mut(&mut self) -> &mut dyn Any;

}

type Module = Box<dyn HttpModule + 'static + Send + Sync>

可以看到,HttpModule 对请求和返回都可以进行修改。对于请求,Trait 中定义了 request_header_filter()用于修改 header,以及 request_body_filter() 用来修改 Body;而对于返回,HttpTask 中的所有部分都可以更改。HttpTask 定义如下:

/// An enum to hold all possible HTTP response events.

#[derive(Debug)]

pub enum HttpTask {

/// the response header and the boolean end of response flag

Header(Box<pingora_http::ResponseHeader>, bool),

/// A piece of response header and the end of response boolean flag

Body(Option<bytes::Bytes>, bool),

/// HTTP response trailer

Trailer(Option<Box<http::HeaderMap>>),

/// Signal that the response is already finished

Done,

/// Signal that the reading of the response encounters errors.

Failed(pingora_error::BError),

}

基本上涵盖了 Response 返回的所有内容。

trait HttpModuleBuilder

定义了模块后,我们还不能直接使用。我们需要为这个模块创建一个 Builder,符合以下 Trait 的定义:

/// Trait to init the http module ctx for each request

pub trait HttpModuleBuilder {

/// The order the module will run

///

/// The lower the value, the later it runs relative to other filters.

/// If the order of the filter is not important, leave it to the default 0.

fn order(&self) -> i16 {

0

}

/// Initialize and return the per request module context

fn init(&self) -> Module;

}

pub type ModuleBuilder = Box<dyn HttpModuleBuilder + 'static + Send + Sync>

HttpModuleBuilder 中,Pingora 规定了 HttpModule 加载与应用的优先级。

HttpModules

最后,回到我们 HttpServer 中使用的 HttpModules 类型。这其实是一个很像 Builder 的结构,定义如下:

/// The object to hold multiple http modules

pub struct HttpModules {

modules: Vec<ModuleBuilder>,

module_index: OnceCell<Arc<HashMap<TypeId, usize>>>,

}

impl HttpModules {

/// Create a new [HttpModules]

pub fn new() -> Self {

HttpModules {

modules: vec![],

module_index: OnceCell::new(),

}

}

/// Add a new [ModuleBuilder] to [HttpModules]

///

/// Each type of [HttpModule] can be only added once.

/// # Panic

/// Panic if any [HttpModule] is added more than once.

pub fn add_module(&mut self, builder: ModuleBuilder) {

if self.module_index.get().is_some() {

// We use a shared module_index the index would be out of sync if we

// add more modules.

panic!("cannot add module after ctx is already built")

}

self.modules.push(builder);

// not the most efficient way but should be fine

// largest order first

self.modules.sort_by_key(|m| -m.order());

}

/// Build the contexts of all the modules added to this [HttpModules]

pub fn build_ctx(&self) -> HttpModuleCtx {

let module_ctx: Vec<_> = self.modules.iter().map(|b| b.init()).collect();

let module_index = self

.module_index

.get_or_init(|| {

let mut module_index = HashMap::with_capacity(self.modules.len());

for (i, c) in module_ctx.iter().enumerate() {

let exist = module_index.insert(c.as_any().type_id(), i);

if exist.is_some() {

panic!("duplicated filters found")

}

}

Arc::new(module_index)

})

.clone();

HttpModuleCtx {

module_ctx,

module_index,

}

}

}

在构造期间,我们需要通过 HttpModules::add_module 添加新的 ModuleBuilder。在每次添加时,都会以 HttpModuleBuilder::order() 进行排序,确定最终模块的执行顺序([2])。

在每次 HTTP 请求来临时,build_ctx() 都会被调用。在这个过程中,Pingora 会通过 HttpModuleBuilder::init() 创建对应的 HttpModule([3]),并且将顺序信息以 HashMap 的形式存储来生成一个 index([4])。顺序信息只会创建一次,之后便通过 OnceCell 的形式保留在 module_index 中,供后续复用([1])。

HttpModuleCtx

HttpModule 的最后一环就是 HttpModuleCtx 了。其实这个部分已经没什么可讲的了,就是很简单的顺序执行 Module 的内容,代码一看就懂了x

/// The Contexts of multiple modules

///

/// This is the object that will apply all the included modules to a certain HTTP request.

/// The modules are ordered according to their `order()`.

pub struct HttpModuleCtx {

// the modules in the order of execution

module_ctx: Vec<Module>,

// find the module in the vec with its type ID

module_index: Arc<HashMap<TypeId, usize>>,

}

impl HttpModuleCtx {

/// Create a placeholder empty [HttpModuleCtx].

///

/// [HttpModules] should be used to create nonempty [HttpModuleCtx].

pub fn empty() -> Self {

HttpModuleCtx {

module_ctx: vec![],

module_index: Arc::new(HashMap::new()),

}

}

/// Get a ref to [HttpModule] if any.

pub fn get<T: 'static>(&self) -> Option<&T> {

let idx = self.module_index.get(&TypeId::of::<T>())?;

let ctx = &self.module_ctx[*idx];

Some(

ctx.as_any()

.downcast_ref::<T>()

.expect("type should always match"),

)

}

/// Get a mut ref to [HttpModule] if any.

pub fn get_mut<T: 'static>(&mut self) -> Option<&mut T> {

let idx = self.module_index.get(&TypeId::of::<T>())?;

let ctx = &mut self.module_ctx[*idx];

Some(

ctx.as_any_mut()

.downcast_mut::<T>()

.expect("type should always match"),

)

}

/// Run the `request_header_filter` for all the modules according to their orders.

pub fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {

for filter in self.module_ctx.iter_mut() {

filter.request_header_filter(req)?;

}

Ok(())

}

/// Run the `request_body_filter` for all the modules according to their orders.

pub fn request_body_filter(&mut self, mut body: Option<Bytes>) -> Result<Option<Bytes>> {

for filter in self.module_ctx.iter_mut() {

body = filter.request_body_filter(body)?;

}

Ok(body)

}

/// Run the `response_filter` for all the modules according to their orders.

pub fn response_filter(&mut self, t: &mut HttpTask) -> Result<()> {

for filter in self.module_ctx.iter_mut() {

filter.response_filter(t)?;

}

Ok(())

}

}

回到 Prometheus

看完一圈底层的实现,我们再回到 Prometheus。现在再来看 PrometheusServer 的实现就非常简单清晰了:

/// A HTTP application that reports Prometheus metrics.

///

/// This application will report all the [static metrics](https://docs.rs/prometheus/latest/prometheus/index.html#static-metrics)

/// collected via the [Prometheus](https://docs.rs/prometheus/) crate;

pub struct PrometheusHttpApp;

#[cfg_attr(not(doc_async_trait), async_trait)]

impl ServeHttp for PrometheusHttpApp {

async fn response(&self, _http_session: &mut ServerSession) -> Response<Vec<u8>> {

let encoder = TextEncoder::new();

let metric_families = prometheus::gather();

let mut buffer = vec![];

encoder.encode(&metric_families, &mut buffer).unwrap();

Response::builder()

.status(200)

.header(http::header::CONTENT_TYPE, encoder.format_type())

.header(http::header::CONTENT_LENGTH, buffer.len())

.body(buffer)

.unwrap()

}

}

/// The [HttpServer] for [PrometheusHttpApp]

///

/// This type provides the functionality of [PrometheusHttpApp] with compression enabled

pub type PrometheusServer = HttpServer<PrometheusHttpApp>;

impl PrometheusServer {

pub fn new() -> Self {

let mut server = Self::new_app(PrometheusHttpApp);

// enable gzip level 7 compression

server.add_module(ResponseCompressionBuilder::enable(7));

server

}

}

整个过程中最主要的步骤只有:

  1. PrometheusHttpApp 实现 ServeHttp
  2. 定义一个 PrometheusServer,将 PrometheusHttpApp 构造进去,并且添加上需要的 HttpModuleBuilder

真的是⑨都能看懂的程度,这里就不过多赘述了。

最后看看……ResponseCompression

PrometheusServer 中用到了 ResponseCompressionBuilder,而这也是 Pingora 目前在其仓库内实现的唯一一个 HttpModule。这里我们简单看看它在 pingora-core 中的基本实现,详情留到以后再展开(咕):

/// HTTP response compression module

pub struct ResponseCompression(ResponseCompressionCtx);

impl HttpModule for ResponseCompression {

fn as_any(&self) -> &dyn std::any::Any {

self

}

fn as_any_mut(&mut self) -> &mut dyn std::any::Any {

self

}

fn request_header_filter(&mut self, req: &mut RequestHeader) -> Result<()> {

self.0.request_filter(req);

Ok(())

}

fn response_filter(&mut self, t: &mut HttpTask) -> Result<()> {

self.0.response_filter(t);

Ok(())

}

}

/// The builder for HTTP response compression module

pub struct ResponseCompressionBuilder {

level: u32,

}

impl ResponseCompressionBuilder {

/// Return a [ModuleBuilder] for [ResponseCompression] with the given compression level

pub fn enable(level: u32) -> ModuleBuilder {

Box::new(ResponseCompressionBuilder { level })

}

}

impl HttpModuleBuilder for ResponseCompressionBuilder {

fn init(&self) -> Module {

Box::new(ResponseCompression(ResponseCompressionCtx::new(

self.level, false,

)))

}

fn order(&self) -> i16 {

// run the response filter later than most others filters

i16::MIN / 2

}

}

嘛,就是这样(x