One of the challenges I faced while building my LaTeX Writer project was implementing version control and project forking in a storage-efficient way.
A typical LaTeX project contains multiple files. Even a simple project usually has a "main.tex" file, bibliography files, images, style files, and other supporting documents. If I stored a complete copy of every file for every version or fork, storage requirements would grow rapidly.
Imagine a project with four files and ten versions. Storing the entire project for every version would mean storing the same files repeatedly, even when only one line changed. Forking would create an even bigger problem because every fork would require another complete copy of the project.
Instead of accepting this inefficiency, I started researching how large platforms solve the same problem. GitHub was the obvious inspiration.
Learning from GitHub
GitHub does not store a complete copy of a repository every time a change is made. Instead, it stores content separately and uses references to connect files, commits, and repositories.
This idea became the foundation for my own implementation.
Project Structure
Whenever a new project is created, a default file called "main.tex" is generated automatically.
The project itself does not directly contain file contents. Instead, it stores metadata such as:
- Project ID
- Owner ID
- Root Folder ID
- File References
Each file also has its own metadata record containing:
- File ID
- File Name
- Blob ID
- Project ID
- Owner ID
- Folder ID
The actual content is not stored inside the file metadata.
Instead, the content lives inside a separate entity called a Blob.
Loading a Project
When the editor loads a project, it reconstructs the directory structure using metadata.
The process works like this:
- Retrieve the project's Root Folder ID.
- Find all folders belonging to that folder hierarchy.
- Find all files belonging to each folder.
- Build the directory tree for the frontend.
Because files and folders are stored independently, the entire project structure can be recreated efficiently without duplicating data.
Blob-Based Content Storage
The most important optimization is the Blob system.
A Blob represents the actual content of a file.
When a user edits a file:
- The content is hashed.
- A unique Blob ID is generated.
- If a Blob with identical content already exists, the existing Blob is reused.
- The file metadata simply points to the appropriate Blob.
This means identical file contents are stored only once, regardless of how many files, versions, or forks reference them.
How Forking Works
Forking was designed with the same principle.
When a user forks a project:
- A new Project ID is created.
- New file metadata records are created.
- New folder metadata records are created.
- Ownership information changes to the new user.
However, the file content is not copied.
The new file metadata points to the exact same Blob IDs as the original project.
As a result, a fork initially consumes very little additional storage because only metadata is duplicated.
What Happens When a Fork Is Edited?
Suppose User A creates a project and User B forks it.
Initially, both projects reference the same Blobs.
If User B edits a file:
- The updated content is hashed.
- The system checks whether a Blob with that content already exists.
- If it exists, the existing Blob is reused.
- If it does not exist, a new Blob is created.
- User B's file metadata is updated to point to the new Blob.
The original project remains unchanged because its file metadata still points to the old Blob.
This creates behavior similar to Git's object storage model while keeping storage usage minimal.
The Result
By separating metadata from content and introducing Blob-based storage, I achieved:
- Efficient version management
- Storage-friendly project forking
- Content deduplication
- Faster project cloning
- Reduced database growth
- GitHub-inspired architecture without implementing Git itself
Instead of storing the same file hundreds of times, the system stores content once and uses references everywhere else.
This approach allows the platform to scale much more efficiently while supporting features like version history, project cloning, and collaborative workflows.