devops

Got big binaries? Tired of other version control systems that treat them like inferior files? Lore might be worth a look

Fortnite maker and Apple nemesis Epic Games has decided to git good all on its own with the open-source release of its homemade version control system, dubbed Lore.

The project began life as Unreal Revision Control, and was used by internal teams and as the version control system (VCS) built into Unreal Editor for Fortnite. Now, Epic is ready to share its handiwork with the world. 

Lore is a centralized, content-addressed VCS that’s meant to be more flexible for developers, as it's licensed under the less restrictive MIT License instead of the copyleft requirements inherent in the GNU standard. MIT is generally considered more permissive because, unlike GNU, it doesn't require derivatives to be licensed in the same way (e.g., a fork of Lore could be proprietary). 

Lore can be installed on macOS, Windows, and Linux and its server side is designed to be transportable into different cloud services as well. The biggest difference between Lore and other VCS is its equal treatment of text files – e.g., code – and binaries. 

“All content is treated as opaque byte streams on the hot path,” Epic explains in its system design explanation document. “Text-aware features are layered on top, never assumed by the storage or transport paths. Binary content gets the same first-class treatment as text.”

With that in mind, it’s obvious who Epic is targeting with the release: Game developers. 

Lore is purpose-built for projects that use large binary files such as games, Epic said, but that doesn’t preclude other use cases with heavy binary loads, like AI model builders, systems developers, and others who work with large amounts of machine-readable data alongside their own code.

We have lots of VCS data, so why do we need Lore?

There are plenty of VCS options out there: Git, Perforce, Mercurial (and its descendent Sapling) are all mentioned by Epic as alternatives that resemble Lore in its design and use. So, why a new VCS? That’s easy, says the Fortnite studio: None of ‘em do it all. 

Git, says Epic, has great revision graphing, but treats binaries as “second class citizens” and lacks multi-tenant isolation that ensures users on the same infrastructure can't access each others work. Perforce requires multiple server round trips to conduct standard operations, making it too slow. Mercurial and Sapling elegantly solve “the scale of source repositories” via their distributed architecture, but again treat text as king and everything else as second-class data. 

“The motivation is not that prior systems are bad,” Epic explained. “What Lore offers that the prior art does not is the union” of all those features, and some others too.

Key design goals Epic had in mind when designing Lore included the aforementioned binary-first design, a sparse-by-construction architecture that only downloads necessary fragments from the server to clients to ensure fewer round trips, the elimination of partially-applied revisions, in-between states are invisible to readers, and a full-surface API that allows Lore to work with a variety of programming languages. 

If you want to give Lore a spin Epic has published a thorough quickstart guide, and pre-built binaries are available, ironically enough, on GitHub. ®