Apache Data Lakehouse Weekly: May 21-27, 2026

The week after a major release tends to look quiet on a project's dev list. This one did not. With Iceberg 1.11.0 and 1.10.2 both out the door the week before, and Polaris 1.5.0 shipping right at the start of this window, you might have expected the lakehouse projects to take a breath. Instead the conversation shifted from "what are we shipping" to "what do we build on top of what we just shipped," and that turned out to be a busier and more interesting set of threads. Encryption moved from Iceberg core into the catalog layer. The REST spec picked up two new client-facing extensions. Arrow took a donation across the finish line and started arguing about whether a bot should review its pull requests. Parquet finally voted on a statistics change that had been circling the list for years. Taken together, the week was about the connective tissue of the lakehouse: the catalog, the protocol, the client contract, and the unglamorous governance work that keeps four independent projects interoperable.

Apache Iceberg

The single most consequential thread of the Iceberg week was procedural rather than technical. Ryan Blue posted the RESULT for the vote to add an unregister endpoint to the REST spec, which passed with 16 +1 votes, 9 of them binding, and no dissent. That is a strong mandate for what is a deceptively important capability. Until now the REST catalog has had no standard way to drop a table's registration without also deleting its data and metadata, which matters enormously for the migration and multi-catalog scenarios that Iceberg increasingly has to support. As tables move between catalogs, or as a catalog needs to hand off ownership of a table to another system, unregister is the clean primitive that makes that safe. The breadth of binding support tells you this was not controversial in substance, only in getting the wording precise enough to standardize.

Around that vote, a cluster of REST spec discussions showed the catalog protocol entering a more mature phase where the arguments are about extensibility and forward compatibility rather than core mechanics. Prashant Singh opened a discussion on adding an X-Iceberg-Client-Capabilities header to the REST spec, which came out of the Read Restrictions community sync on May 12. The idea is to give clients a standard way to advertise what they support so that servers can adapt their responses, which is exactly the kind of negotiation mechanism a protocol needs once it has many independent implementations that ship on different schedules. In a related vein, Alexandre Dutra summarized the outcomes from the catalog sync in a thread on passing arbitrary information to request signers, refining the language around how clients hand context to the components that sign storage requests. Both threads point at the same underlying reality: the REST catalog is now the integration surface that the whole ecosystem leans on, and the community is carefully building the seams that let it evolve without breaking clients.

Release work did not stop with 1.11.0. The non-Java implementations are all moving in parallel. Matt Topol opened the vote on Apache Iceberg Go v0.6.0 RC2, which drew nine participants and active testing, while Junwang Zhao started a discussion on releasing Iceberg C++ 0.3.0 and Alex Stephen kicked off planning for PyIceberg 0.12.0 by pointing contributors at the milestone to flag anything still missing. Three client libraries in three languages all cycling toward releases in the same week is a good illustration of how the project has decoupled its implementations so they can each move at their own pace rather than waiting on the Java reference.

The post-1.11.0 design conversation also got going. Steven Wu opened a discussion on Flink version support after Iceberg 1.11.0, working through how the project should manage the matrix of supported Flink versions going forward, a recurring maintenance question that every engine integration eventually has to confront. Stepan Stepanishchev proposed adding Flink SQL procedure support to Iceberg, which would let users invoke Iceberg maintenance and management operations through Flink's CALL statement the way they already can in Spark. Noritaka Sekiyama proposed adding an OpenTelemetry-based MetricsReporter to iceberg-core that would export ScanReport and CommitReport data to any OTLP-compatible backend, a genuinely useful piece of observability plumbing that drew seven participants and nine replies. Iceberg already ships metrics reporting interfaces, but standardizing on OpenTelemetry would let operators wire table-level telemetry into the same monitoring stack they use for everything else.

Two threads dealt with the practical grind of running a large open source project. Robert Thomson, writing on behalf of ASF infrastructure, raised Iceberg's consumption of the shared GitHub-hosted Actions runners, part of a foundation-wide effort to keep CI usage within the shared pool's limits. It drew eight participants quickly, because every active committer feels the pain of CI queueing. Max Konstantinov opened a discussion on sunsetting MkDocs for the project's versioned documentation, noting that the MkDocs project itself appears effectively abandoned with no new contributions in roughly 18 months, which makes it a risky foundation for the docs that the whole community depends on. These are not glamorous threads, but they are the kind of maintenance the project has to stay on top of to keep growing.

Apache Polaris

Polaris had the cleanest headline of the week. Jean-Baptiste Onofré announced the release of Apache Polaris 1.5.0, following the vote that passed with binding +1s from Robert Stupp, François Papon, Dmitri Bourlatchkov, Yong Zheng, and Onofré himself. The release headlines Apache Ranger support as an external authorizer, alongside CLI improvements and Helm chart work. Ranger integration is a meaningful step for enterprise adoption, because it lets organizations that already run Ranger for their broader data platform extend those same authorization policies to their Iceberg catalog rather than maintaining a separate access model. That this is the first release since the project's graduation discussions, and that it shipped on a clean RC0, says good things about where Polaris is in its maturity curve.

The more forward-looking work was about authorization and delegation, which are clearly the project's center of gravity right now. Yufei Gu posted updates on the Delegation Service design document, noting that he and Onofré will co-author it and that the pull request reflects the latest direction on the pull-versus-push modes question. Sung Yun followed up on the dedicated sync on Polaris authorization, proposing to fold earlier authorization discussion into the new authorization SPI work. The throughline is that Polaris is building a pluggable authorization architecture rather than hardcoding a single model, which is the right call for a catalog that has to serve organizations with wildly different security requirements. Ranger landing in 1.5.0 is the first external authorizer; the SPI work is what makes the second and third ones tractable.

Two threads showed Polaris reaching across project boundaries. Alexandre Dutra opened an Iceberg 1.11 feature branch retrospective, evaluating the experience of maintaining a feature/iceberg-1.11 branch to stay ahead of upcoming Iceberg enhancements and deciding what to do with it now that 1.11 has shipped. Adam Szita started a discussion on Iceberg table encryption support in Polaris, picking up directly from the base encryption implementation that landed in Iceberg 1.11. This is the most important cross-project signal of the week and worth dwelling on. Iceberg shipped KMS-based key wrapping and encrypted data, delete, manifest, and manifest-list files in 1.11, but encryption is only useful end to end if the catalog knows how to manage and hand out keys. The fact that Polaris opened this thread within days of the Iceberg release shows how tightly the catalog and the table format now move together. The encryption story does not work unless both halves cooperate, and the community is treating it that way.

There was also a healthy run of operational and integration discussion. Adnan Hemani followed up on an OpenLineage proposal for lineage tracking, Bill Bejeck floated a diagnostics shell prototype to answer simple operational questions like how many tables a bootstrapped Polaris instance is managing, and Dmitri Bourlatchkov pushed on the practical question of how to land generic table delegation in the Polaris SparkCatalog. Alexandre Dutra also proposed forbidding special characters in entity names that most cloud providers reject or discourage, the kind of guardrail that prevents a class of cryptic failures down the line. And in a thread that captures where the industry's head is at, Dennis Huo proposed an agentic eval meta-skill for extensibility and maintainability, exploring how the project should think about agentic development as a first-class tool rather than something contributors do off to the side.

Apache Arrow

Arrow's week was anchored by a donation reaching its conclusion. Sutou Kouhei posted the RESULT for the vote to donate the Apache Arrow Erlang implementation, which carried with four binding +1s from Sutou, Curt Hagenlocher, Matt Topol, and David Li. The next step is the IP clearance process and a vote on the incubator general list. Benjamin Philip, who has been shepherding the contribution, had earlier worked through the grant documents for the Erlang implementation, filling out the IP clearance template, the contributor license agreement, and the software grant. The Erlang library is built on bindings to the Rust implementation, which is itself a nice illustration of how Arrow's investment in a strong Rust core is now letting new language bindings come together faster than a from-scratch implementation ever could. Every new language Arrow speaks is another place its columnar format becomes the default interchange layer, and getting there by wrapping Rust rather than reimplementing C++ is the efficient path.

The release engineering that has become Arrow's signature continued without drama. Andrew Lamb posted RESULT threads for three Rust releases in close succession: arrow-rs 56.2.1, 57.3.1, and 58.3.0, each approved with five +1 votes. Shipping three point releases across three maintenance lines in a single stretch is the kind of cadence that signals a project with its release automation thoroughly sorted out. Rok Mihevc also closed the loop on the pyarrow-stubs donation, confirming that the software grant has been formally filed and published, which concludes that process and brings type stubs for PyArrow into the project proper.

The design discussions leaned toward type system and protocol extensions. Florian Hölzlwimmer proposed an arrow.range canonical extension type for bounded ranges, filling a gap in Arrow's type vocabulary. Tornike Gurgenidze opened two threads on the Flight and ADBC side: a partitioned bulk ingest API for ADBC that would mirror the existing ExecutePartitions and ReadPartition read-side primitives on the write side, and a proposal to add dialect-related SqlInfo codes to FlightSQL so clients have a standard way to learn what SQL features a backend supports. There was also continued work on a Flight SQL field to signal whether a prepared statement is an update, with Jean-Baptiste Onofré suggesting the vote be extended to give more people time to weigh in.

The thread that will resonate beyond Arrow was Sutou Kouhei's discussion on enabling automatic GitHub Copilot review on apache/arrow pull requests. His framing was pragmatic: the project does not have enough human review bandwidth, and a Copilot pass could catch trivial problems before a human reviewer ever looks. This is the same underlying tension that Iceberg has been working through with its AI contribution guidelines, just approached from the reviewer side rather than the contributor side. It is a question every large open source project is going to have to answer, and seeing Arrow debate it openly on the dev list, weighing the value against the noise, is exactly how these norms should get set.

There was also a low-level performance thread worth a mention for the systems-minded: Dan Mattheiss opened a discussion on AVX2 SBBF probe for parquet/bloom_filter.cc, noting that arrow-go already shipped SIMD bloom filter probes and proposing the C++ side catch up. It is a reminder that the cross-language consistency Arrow promises also means keeping performance optimizations roughly in step across implementations.

Apache Parquet

Parquet's headline was the close of a long-running saga. Gang Wu opened the vote to adopt the format change for PARQUET-2249, covering IEEE 754 total order and NaN-counts, drawing eight participants. This proposal has been circulating in various forms for a long time, and the discussion thread on adding nan_count to handle NaNs in statistics had a "bumping this one last time" quality to it before the vote finally opened. The substance matters more than the procedural relief. Floating-point NaN values break the assumptions that min/max statistics rely on, which means engines either produce wrong results when pruning row groups that contain NaNs or disable statistics-based pruning entirely on float columns to be safe. Standardizing a total order and a NaN count fixes that at the format level, so every engine can prune float columns correctly. It is the kind of fix that sounds narrow and is actually load-bearing for query performance on a very common data type.

The other major structural conversation was the footer. Jiayi Wang posted the kickoff for the Parquet Footer Working Group, setting up a dedicated forum to move the footer redesign forward more efficiently after it was discussed at the Parquet sync. The footer is where Parquet stores its metadata, and how it is structured determines how quickly a reader can open a file and figure out what is in it, which is increasingly a bottleneck as files get larger and workloads get more selective. Pierre Lacave contributed to the related discussion on an alternative to the FlatBuffer footer, a lightweight byte-offset index, sharing that a similar pattern is in use in a custom file format his team is migrating toward Parquet. Standing up a working group is a signal that the community sees footer evolution as a multi-release effort that deserves focused attention rather than ad hoc threads.

Release planning got going too. Fokko Driesprong opened a discussion on an Apache Parquet 1.18.0 release, noting that a lot of work has accumulated since the last major release and that it is overdue. Ismaël Mejía proposed bumping the minimum Java version for Parquet Java to 17, pointing out that Java 17 has been the baseline LTS since September 2021 and that holding the floor at 11 is increasingly costly. Mejía was also active on the performance front, sharing encoding and decoding hot-path optimizations and asking for code reviews on work he presented at the Parquet community sync. On the safety side, Steve Loughran circulated a pull request hardening the variant readers, noting that while a malformed 1KB file triggering a multi-gigabyte allocation is not strictly a security issue, it is close enough to be worth fixing. And the community recognized that contribution work with an announcement that Ed Seidl has accepted an invitation to become a committer. There was also continued interest in the geospatial story, with Dewey Dunnington noting in a thread on geography test files with statistics that he had added geography statistics writing to SedonaDB via arrow-rs, closing a gap that had been flagged when the geospatial types blog post came out.

Cross-Project Themes

Two patterns connect these four lists this week, and both say something about where the lakehouse stack is heading.

The first is that encryption has become a cross-project program rather than a single project's feature. Iceberg shipped the base table encryption implementation in 1.11, and within the same week Polaris opened a thread on how the catalog should support it. Encryption that protects data files but leaks key management to every client is not real protection, so the catalog has to be the trusted party that wraps, unwraps, and hands out keys under policy. You cannot reason about Iceberg encryption by reading the Iceberg list alone; the design only closes when you read the Polaris thread next to it. That is the lakehouse working as a coordinated platform, where a capability is split across the format and the catalog by design and the two communities build their halves in step.

The second is that every one of these projects is now wrestling, openly and on the record, with how AI fits into its development process. Arrow debated whether to let Copilot review pull requests. Polaris explored an agentic eval meta-skill as first-class project tooling. Iceberg has its AI contribution guidelines work, and the word "agentic" is showing up in the Polaris topic cloud. These are not the same question, but they rhyme. The community is deciding, in the open, what role AI tools should play in producing and reviewing the code that underpins the open data stack, and it is doing so transparently rather than letting individual contributors quietly make those choices alone. The decisions made over the next few months will set norms that stick for years, and it matters that they are being made on public dev lists where the whole community can see the reasoning.

There is a quieter third theme worth naming: protocol and format extensibility. Iceberg's client capabilities header, Arrow's FlightSQL dialect codes, and Parquet's footer working group are all the same instinct expressed in three places. Each project has reached the point where it has many independent implementations on different release schedules, and the central task is no longer adding features but building the negotiation and versioning seams that let those implementations evolve without breaking each other. That is what maturity looks like for an interoperability standard.

Looking Ahead

The Iceberg Go v0.6.0 and PyIceberg 0.12.0 releases should close in the coming days, and the C++ 0.3.0 discussion will likely firm up into a release plan. Watch the X-Iceberg-Client-Capabilities header thread, because if it gains traction it becomes the mechanism through which a lot of future REST evolution gets negotiated. On the Polaris side, the encryption support thread is the one to follow, since it is the catalog half of the story Iceberg started in 1.11, and the Delegation Service design doc should continue to take shape. Arrow's Erlang donation moves to the incubator general list for IP clearance, and the Copilot review discussion is worth watching as a bellwether for how Apache data projects handle AI in their workflows. For Parquet, the PARQUET-2249 vote should close and move into implementation across engines, the Footer Working Group will likely publish a charter and cadence, and the 1.18.0 release planning plus the Java 17 baseline proposal will shape what the next Parquet Java looks like. The through-line for the weeks ahead is the same one that defined this week: the interesting work is increasingly in the layers that connect the projects to each other.