Building a better Perforce

JamHub lets you collaborate on game projects with your team. It's like Perforce, but free and open-source.

It's time for a free and open source collaboration system for game developers. Current closed-source systems, like Perforce and Plastic/Unity DevOps, are expensive and complex. Current open-source solutions, like Git or SVN, are difficult to use and do not scale with large projects containing large files. Game developers want to focus on their game, not their version control.

Jam is an AGPL-licensed version control system that efficiently tracks changes to large files and large projects. You can try it out by logging in or check out the AGPL-licensed source on GitHub.

Join the Discord for updates or to get support.

Problem

Third generation version control systems have enabled code collaboration and sharing on a massive scale. Most notably Git in combination with hosting platforms like GitHub have improved our ability to reuse and share code. However, game developers have mostly been removed from this movement due to Git's poor support for large files and large projects. There are several problems that will also continue to get worse.

Third generation version control systems, almost all of which were created in the mid-2000's, were created in a widely different storage, network and build environment than what we have today. These changes, as they continue to evolve over the next decade, will create a need for a "fourth generation" version control system.

Fourth Generation Version Control

JamHub

JamHub is an in-development fourth generation version control system that is being built for game developers. You can currently do things most that you would expect from a current version control system, like pulling, pushing, and merging, but it's not quite ready for production use. The core algorithm has been implemented but there's more work remaining to build out features that developers expect from a full collaboration platform. Please join the Discord and star the repo in GitHub to follow future development!

Terminology

Since JamHub works a little differently than most version control systems, it's necessary to define some words since they may have slightly different meaning than other systems.

Benchmarks

This section compares JamHub upload and download speed for a directory to Git. Note that these numbers are not final and future features will give JamHub ways to make typical workflows faster, like directory mounting over NFS. Also, these are raw file measurements, meaning no previous versions are uploaded or downloaded (which is to Git's advantage).

Measurements from JamHub hosted in us-west-2 from Seattle and measurements from Git repositories hosted by GitHub. Obviously, there are more operations that matter in version control than just uploading and downloading an entire repository, but this is just a demonstration to show what improvements can be made in our current systems.

Algorithm

The idea behind JamHub based off of the rsync algorithm and Content Defined Chunking (CDC). If you haven't read these, I would highly recommend them!

How JamHub uses Rsync and CDC

The main idea behind JamHub is that we can store the operations sent by the sender in an rsync-like stream to track changes to a file. This means we treat rsync operations like a delta chain that we can use later to regenerate the file. The storage of deltas and their usage to regenerate a file is similar to the Mercurial concept of a Revlog. However, the advantage of using rsync blocks is that we can efficiently store changes to, and regenerate, arbitrarily large files since these blocks can be streamed and regenerated independently.

Data pointers

In each block, we can store the location of the last data block to regenerate the file efficiently. By using blocks instead of an xdelta approach, we can store pointers in each block find the last actual data block to use in the file, rather than regenerating the file through a delta chain which Mercurial does. Mercurial essentially caches the entire file at certain points and uses this later to have a smaller regeneration length.

Workspaces

A chain of changes, formed by the process above, can be used to regenerate every file in a project. Workspaces can be automatically rebased on top of the mainline. This means that every workspace will always be up-to-date. If conflicts occur during the rebase, a workspace will need manual merging.

Limitations

The goal is to be able to handle over 100M files and over 1TB-sized files in a single repository. We're not there yet in the current implementation (~1M files with 16GB-sized files) but should be there in the next couple months.

Implementation

JamHub is being written from scratch in Golang and uses mattn/go-sqlite3 to store projects and change information. gRPC and Protocol buffers are used for service definitions and data serialization.

Acknowledgements

This awesome site theme is made by @panr and adapted to this site.