惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Hacker News: Ask HN
Hacker News: Ask HN
Recent Commits to openclaw:main
Recent Commits to openclaw:main
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
C
Check Point Blog
S
Security Affairs
Hacker News - Newest:
Hacker News - Newest: "LLM"
S
Secure Thoughts
Recorded Future
Recorded Future
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
T
The Blog of Author Tim Ferriss
B
Blog
C
Cybersecurity and Infrastructure Security Agency CISA
Google DeepMind News
Google DeepMind News
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
A
Arctic Wolf
T
The Exploit Database - CXSecurity.com
Stack Overflow Blog
Stack Overflow Blog
T
Threat Research - Cisco Blogs
GbyAI
GbyAI
AWS News Blog
AWS News Blog
MongoDB | Blog
MongoDB | Blog
Y
Y Combinator Blog
Google Online Security Blog
Google Online Security Blog
T
Troy Hunt's Blog
I
InfoQ
L
LINUX DO - 热门话题
WordPress大学
WordPress大学
C
Cisco Blogs
G
GRAHAM CLULEY
The Register - Security
The Register - Security
A
About on SuperTechFans
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
Schneier on Security
Schneier on Security
Project Zero
Project Zero
H
Hackread – Cybersecurity News, Data Breaches, AI and More
P
Privacy & Cybersecurity Law Blog
Cloudbric
Cloudbric
H
Hacker News: Front Page
小众软件
小众软件
雷峰网
雷峰网
The Hacker News
The Hacker News
www.infosecurity-magazine.com
www.infosecurity-magazine.com
T
Tor Project blog
博客园 - 聂微东
N
Netflix TechBlog - Medium
V
Vulnerabilities – Threatpost
The GitHub Blog
The GitHub Blog
腾讯CDC
P
Palo Alto Networks Blog
Scott Helme
Scott Helme

kmcd.dev

gRPC-Web Should Have Fixed gRPC Making Dynamic Protobuf Fast in Go Proxy, Record, and Mock gRPC APIs with FauxRPC Exploring Protocol Buffers Interactively Introducing ProtoDocs Ghost in the Shell: The Manga Behind the Anime The Hidden Cost of google.protobuf.Value Why Networking Built Its Own Data Modeling Language Zero-Friction Demos with WASM Let's Learn About BGP ConnectRPC: Where is it now? Building APIs with Contracts The Case for Greppable Code Unknown Fields in Protobuf IRC Log: Reactionary Faking protobuf data in Go Y'all are Sleeping on Mise-en-Place IRC Log: Standup 2 HTTP/2 From Scratch: Part 4 IRC Log: rm -rf /var/opt/gitlab/postgresql/data HTTP/2 From Scratch: Part 3 Building a Live BGP Map HTTP/2 From Scratch: Part 2 IRC Log: The Cloud Scale Incident Visualizing the Internet (2026) Shell Log: Namaste HTTP/2 From Scratch: Part 1 IRC Log: Standup HTTP/1.1 From Scratch WHOIS is dead, long live RDAP Months Considered Harmful Encryption vs. Compression On Creating My Own Cover Art Traceroute Tool from Scratch in Go My Favorite Interview Question From JSON to Protobuf Breaking gRPC Morse Code Can You Hack a Phone with Your Voice? Visualizing the Internet (2025) HTTP QUERY and Go I made a daily word game Protovalidate: Can Input Validation Be This Easy? Behold! The Barcode Scanner Mixing CEL and Protobuf for Fun FauxRPC and Protovalidate The Call of the Monolithic Codebase FauxRPC + Test Containers Self-Documenting Connect Services gRPC Over HTTP/3: Followup JSON to Protobuf Conversion gRPC: The Ugly Parts Working with Protobuf in 2024 Introducing FauxRPC HTTP/1.0 From Scratch Y'all are sleeping on HTTP/3 HTTP/0.9 From Scratch What version of HTTP are you using? Texans in Denmark gRPC Over HTTP/3 gRPC: The Good Parts Leaving Texas for Greener Pastures gRPC: The Bad Parts Unit Testing ConnectRPC Servers Daily Prompts Adding chart.js to Hugo Why I'm Rebranding Benchmarking gRPC (golang) Blog Update gRPC From Scratch: Part 3 - Protobuf Encoding Tracking the Wins Visualizing the Internet (2024) Dropping Unknown Fields in ConnectRPC RESTless: Web APIs After REST Introducing unknownconnect-go Making gRPC more approachable with ConnectRPC Inspecting Protobuf Messages Introducing protoc-gen-connect-openapi gRPC From Scratch: Part 2 - Server gRPC From Scratch: Part 1 - Client Why you should use gNMI over SNMP in 2026 The Rollercoaster of Productivity in Side Projects Lessons from a Decades-Long Project How I learned to code Economists with (virtual) Guns Visualizing the Internet (2023) softlayer-python: language bindings/CLI for a cloud company SwFTP: SFTP/FTP Server For Openstack Swift Video: Morning Copenhagen Commute Goodbye Evepraisal Visualizing the spectrum of the sun (Part 2) Visualizing the Internet (2022) Evepraisal: A price estimation tool for Eve Online Visualizing the spectrum of the sun
Beating Go
2026-07-21 · via kmcd.dev

In my previous article, I explored the performance bottlenecks of using dynamic JSON structures like google.protobuf.Value and official protojson in Go. The benchmark results pointed at the same culprit over and over: reflection, pointer chasing, and descriptor traversal were showing up as real latency and allocation costs.

ProtoJSON sits in an awkward place. It gives Protobuf-backed APIs a JSON representation that human operators, browsers, API gateways, and debugging tools can work with, but Go’s official protojson implementation pays heavily for this generality. To resolve field names, enum values, presence semantics, and message structure at runtime, it has to walk message descriptors using Protobuf reflection (protoreflect) and allocate intermediate values.

This is the right tradeoff for correctness and compatibility. But it made me wonder: how much of this cost is fundamental to ProtoJSON, and how much comes from repeatedly resolving schema mappings at runtime?

So I built protojsonx, partly as an experiment and partly because I wanted to know whether ProtoJSON was inherently slow or whether Go’s implementation was paying too much runtime bookkeeping cost. It implements two strategies: compiling descriptors into flat offset layout tables once at startup (Runtime Table Mode), and generating static, reflection-free parsing routines via a protoc plugin (Generated Plugin Mode).

In this post, we’ll look at the benchmarks comparing standard protojson, raw struct-based JSON serializers (encoding/json and github.com/go-json-experiment/json), protojsonx in both modes, and raw binary Protobuf (proto and the optimized reflection-free vtproto). The interesting part is that moving schema work out of the hot path gets ProtoJSON much closer to ordinary JSON and binary protobuf than I expected.

An experimental faster ProtoJSON encoder and decoder for Go.

Warning: protojsonx is highly experimental at this stage. It does pass the official Protobuf conformance tests, which gives me confidence that it follows the expected ProtoJSON behavior, but I would still treat it as early-stage software.


What is protojsonx?

protojsonx is designed as a mostly API-compatible experimental replacement for Go’s official google.golang.org/protobuf/encoding/protojson library. It supports core ProtoJSON behaviors like field names, enums, presence semantics, unknown-field handling, json_name, oneofs, maps, and standard marshaling/unmarshaling options. However, it does not yet cover every edge case or configuration option of the official library (such as dynamic Any resolving).

To make serialization faster, it implements two key optimization strategies:

  • Runtime Table Mode: This mode implements the startup compilation strategy. It builds flat offset tables at initialization, allowing it to marshal and unmarshal structures using fast, sequential offset arithmetic instead of per-call descriptor traversal.
  • Generated Plugin Mode: For the highest-performance path, protojsonx provides a protoc plugin (protoc-gen-go-protojsonx) that generates type-specific marshaling and unmarshaling methods directly. At that point, a lot of the runtime bookkeeping disappears. The remaining cost is less about “what field is this?” and more about the unavoidable work of reading or writing JSON.

The Benchmark Setup

I used the benchmark suite from my previous article, running on Go 1.26 on an Apple M1 Pro. The configurations are:

  • Small: A flat object with 4 fields (string ID, status boolean, age integer, score float).
  • Medium: A nested user signup event containing actor object, string tags, and metadata map.
  • Large: An array repeating the Medium object 100 times.

The Rules of the Game

All tests use equivalent payload shapes and measure end-to-end marshal/unmarshal cost, including allocations. The generic JSON cases use plain Go structs with similar fields, while the protobuf cases use generated protobuf messages.

This is not a perfect apples-to-apples comparison. ProtoJSON has extra rules around field names, presence, enums, well-known types, and numeric encoding. The point is narrower: if you already need ProtoJSON-shaped output, how much does that cost compared with the JSON tools Go developers normally reach for? These benchmarks compare the cost of serving similar application-shaped JSON payloads, not identical semantics.

I also included hyperpb, a descriptor/layout-driven dynamic protobuf parser built around read-oriented offset decoding, including hyperpb.Shared where the benchmark can reuse its arena. It is not a ProtoJSON library, so its numbers should be read as a binary protobuf reference point rather than a direct competitor.

Payload Sizes

Serialization output size matters, especially when comparing JSON and binary protobuf. These are the payload sizes produced by the benchmark fixtures:

PayloadProtoJSON bytesgeneric JSON bytesbinary proto bytes
Small55 B55 B25 B
Medium293 B291 B162 B
Large29,412 B29,201 B16,500 B

The ProtoJSON and generic JSON sizes are close, but not byte-identical. Binary protobuf is much smaller for these payloads, so the binary rows should be read as a compact binary reference point rather than a JSON-format comparison.

Headline Results

Before the full tables, here is the short version for the fastest JSON path in each group:

OperationBest JSON optionCompared with official protojsonCompared with encoding/json
Small marshalprotojsonx generated6.0x faster1.8x faster
Medium marshalprotojsonx generated7.0x faster1.6x faster
Large marshalprotojsonx generated8.1x faster1.5x faster
Small unmarshalprotojsonx generated6.9x faster5.5x faster
Medium unmarshalprotojsonx generated5.1x faster4.1x faster
Large unmarshalprotojsonx generated5.1x faster3.7x faster

Marshaling Performance

Marshaling is the happier path here. The encoder already has typed Go values in memory, so the main question is how quickly each implementation can walk those values and write JSON bytes.

Show complete data table
Format / Serializerns/opMemory (B/op)Allocations/opSpeed vs protojson
protojson2,227 ns1,722 B341.0x (Baseline)
encoding/json521 ns464 B2-
go-json-experiment/json803 ns608 B3-
protojsonx (Runtime Tables)404 ns320 B15.5x faster
protojsonx (Generated Plugin)318 ns320 B17.0x faster
proto.Marshal285 ns176 B1-
vtproto103 ns176 B1-
hyperpb + Shared1,062 ns744 B17-

Takeaways: Marshaling

Most of the marshaling performance gain comes from compiling descriptors up-front. Standard protojson spends its cycles querying Go’s reflection and descriptor trees on the fly, creating a continuous stream of allocations. Compiling those layout mappings once at startup (Runtime Table Mode) allows the serialization process to bypass runtime lookup loops entirely, converting the fields to JSON via sequential offset math.

The generated plugin goes one step further. Instead of building layout tables at startup, it writes the field access code directly into the generated Go file. That removes another layer of lookup and assertion work from the hot path.

The tradeoff is the usual one for generated code: in a large schema repository, generating specialized JSON routines for hundreds of message types can increase compiled binary size.

For these fixtures, schema-guided ProtoJSON marshaling beats both encoding/json and github.com/go-json-experiment/json, and the generated encoder lands close to standard binary proto.Marshal. hyperpb + Shared is slower on marshal here, which fits its design: it is much more interesting as a read-oriented parser than as a serializer.


Unmarshaling Performance

Unmarshaling is the harder side of the benchmark. The decoder has to turn strings, tokens, object keys, maps, and repeated fields back into typed Go values. The tables below show both the runtime-table path and the generated path.

Show complete data table
Format / Serializerns/opMemory (B/op)Allocations/opSpeed vs protojson
protojson3,703 ns1,304 B581.0x (Baseline)
encoding/json2,937 ns688 B19-
go-json-experiment/json1,133 ns256 B4-
protojsonx (Runtime Tables)1,108 ns576 B163.3x faster
protojsonx (Generated Plugin)720 ns528 B145.1x faster
proto.Unmarshal571 ns560 B15-
vtproto322 ns432 B14-
hyperpb638 ns1,446 B5-
hyperpb + Shared290 ns357 B1-

Takeaways: Unmarshaling

Decode is where the generated code earns its keep. If you’ve ever looked at Go’s standard protojson decoder, it spends a lot of time matching string keys against descriptors, allocating intermediate maps, and resolving types at runtime. Compiling direct field assignments into the generated decoder cuts out a lot of that work: 5.1x to 6.9x faster than official protojson on these static payloads. It also reduces heap churn substantially; unmarshaling the Large payload drops from over 5,700 allocations down to just 1,407.

The interesting part is that the generated JSON decoder beats both encoding/json and go-json-experiment/json in these fixtures. It still does not catch binary protobuf, but it gets close enough that ProtoJSON stops looking like an automatic performance disaster.

That said, unmarshaling still has a wider performance gap relative to binary Protobuf than marshaling does. This is a fundamental constraint of the JSON format itself. A binary decoder can skip fields using length prefixes and parse integer types with very little overhead. A JSON decoder, by contrast, is forced to parse string layouts, handle token boundaries, and instantiate nested objects and map fields on the fly. But even with these structural constraints, the reflection-free generated code narrows the gap significantly, turning what used to be a large bottleneck into a much smaller and more predictable cost.


How protojsonx Compares to Generic JSON & Binary Formats

The basic idea held up: moving schema work out of the hot path can make ProtoJSON faster than Go’s standard library encoding/json package for these fixtures.

The external github.com/go-json-experiment/json package also looks very strong. Under unmarshaling, it cuts a lot of latency and allocation overhead compared with encoding/json (e.g. unmarshaling the Large payload drops from 272,103 ns/op down to 108,811 ns/op).

The experimental JSON package gets close to protojsonx Runtime Table Mode, but the generated protojsonx path was still the fastest JSON option in these benchmarks.

A rough summary of the benchmark results looks like this:

Format / SerializerRole in these benchmarksNotes
Official protojsonSlowest JSON pathMost general and most dynamic
encoding/jsonStandard JSON baselineGeneral-purpose reflection-based JSON
go-json-experiment/jsonFaster generic JSON baselineMuch better on unmarshal
protojsonx Runtime Table ModeFast dynamic ProtoJSONDescriptor work moved to startup
protojsonx Generated Plugin ModeFastest JSON path hereType-specific generated code
Standard binary protoBinary protobuf baselineCompact protobuf wire format
vtprotoFast generated binary baselineStrongest generated marshal path here
hyperpb + SharedBinary decode referenceVery fast reusable-storage decode path

For static-message marshaling, protojsonx with the plugin gets much closer to standard binary Protobuf than official protojson, and in these benchmarks it beats generic encoding/json and go-json-experiment/json.


Methodology

To ensure these results are reproducible, here are the environment parameters and test details used for this benchmark run:

  • Go Version: go version go1.26.3 darwin/arm64
  • Machine Details: Apple M1 Pro (10-core CPU, 16GB unified memory, macOS)
  • Target Library Version: github.com/sudorandom/[email protected]
  • Benchmark Commit: b8fff78c
  • Benchmark Source: The benchmark code is available in the benchmarks/ folder.
  • Test Execution Command:
    go test -bench=. -benchmem -benchtime=5s -count=5 > results.txt
    

All benchmarks were run on an otherwise idle machine using a multi-run sequence to reduce noise. All runs were performed on an AC-powered, thermally-settled machine to prevent thermal throttling or low-power state interference. The command writes the five raw runs for each benchmark to results.txt; the article tables report arithmetic means for ns/op, B/op, and allocs/op computed from those raw rows. For stricter statistical comparison, I would increase the run count and summarize with benchstat.

I left GOMAXPROCS at Go’s default for this machine, which was 8.

As always with microbenchmarks, the exact numbers matter less than the overall shape of the results. You should benchmark your own schemas and payloads under production-realistic environments before drawing final design conclusions.


Try it Out & Give Feedback!

protojsonx is still highly experimental and early-stage software. It passes the official Protobuf conformance tests, which gives me confidence that the core serialization and parsing behavior is on the right track, but I would not treat it as boring infrastructure yet.

If you are serving JSON APIs backed by Protobuf and protojson is showing up in your profiles, I’d love for you to try it against your own schemas.

I’m especially interested in results from real production-shaped messages: large repeated fields, maps, oneofs, well-known types, custom json_name usage, and other cases that are more interesting than tiny benchmark fixtures.

You can find the code and instructions on GitHub:

Please file issues or share benchmark results there. The more weird schemas, the better.