scottchacon.com all posts
gitbutler.com — What I'm building now: a better version control system

Git Wire Protocol v2: A Deep Dive

today 8 min read

Recently, I’ve been messing about with Git server stuff for some experimental work for GitButler. In doing that, one of the things that I’ve been digging into is the bundle-uri and packfile-uri capabilities.

Looking more into it made me realize that I’m not as familiar with how Git’s server works these days as I thought I was. Specifically, I had never really learned about Git’s new (8 years old) “v2” server protocol, which is what those capabilities are built on. So I did a bit of a deep dive.

Smart HTTP, First Edition

When I published the first version of the Pro Git book in August of 2009, Git barely had HTTP support at all. What it had then is now referred to as the “dumb” protocol, just serving static files directly over HTTP.

In December of 2009, Git 1.6.6 landed with the first “smart” HTTP protocol - a CGI script you could load into Apache or whatever to handle ref advertisement and object negotiation via the same git-upload-pack and git-receive-pack binaries that SSH runs via git-shell for communication.

That first “smart” HTTP support landed as part of a huge 28-patch series from Shawn Pearce — the origin of git-http-backend and the smart client and server we still use:

Shawn O. Pearce
Shawn O. Pearce · Oct 30, 2009
lore.kernel.org · 3 messages
Smart HTTP transport

Hopefully the final spin of the smart HTTP series. I think we are down to bike shedding, which I take it means folks are otherwise reasonably happy with the code.

Read the first message →

Since this landed after Pro Git (ed 1) was published, one of the bigger updates in the second edition of the book was to describe HTTP as a first-class protocol to use. I actually wrote a blog post here in 2010 outlining the new “smart” HTTP.

HTTP Protocol V2

I published the second edition of Pro Git in November of 2014 updated with these changes, and a few years later the HTTP transport got smarter again with the v2 protocol.

Since I wasn’t as heavily into Git stuff for a bit, I sort of missed what this meant and why it was done, so I’ve recently dug into it. If you’re curious how Git’s modern HTTP protocol works, this is the blog post for you.

Why a New Protocol?

The first version of the smart protocol would basically do a dump of every reference when you first connected, so the client could determine what objects it had or needed and continue to the negotiation or object push.

That’s fine for most projects, but when your repository has hundreds of thousands of refs, it becomes a lot of data to send over the wire every time you want to do anything at all. So one of the first things the new protocol could do was filter the refs advertised with the ls-refs command.

Another problem was capabilities advertisement. The first version sort of shoehorned them in to the first reference line, hidden behind a NUL byte, which means a server could never advertise more than the size of a pkt-line.

Finally, instead of multiple service names, in v2, multiple commands can be supported by a single service. This means that instead of adding new commands by adding new http endpoints, the git-upload-pack service can now be asked to do lots of different things other than producing packfiles (like listing filtered references, for example).

To lay the foundation to address these issues, Brandon Williams of Google introduced git-serve in March 2018, along with the first two capabilities, ls-refs and fetch.

@bmwill ed10cb95 serve: introduce git-serve @bmwill · Mar 15, 2018

The introduction of the v2 capable server

It finally became the default protocol for the client in Git 2.29, released in October 2020.

Anatomy of a Git HTTP conversation

OK, so what did an HTTP conversation look like with the first protocol and what does it look like with v2? Let’s walk through a clone of a small repository to see the difference.

The First Protocol

Originally, the “Smart” HTTP conversation went something like this.

For a clone, the client would make two HTTP round trips. One to figure out what the reference tips (branches) are. The second to tell the server what it wanted so the server could generate a packfile of just the data needed.

Let’s look at each HTTP request in a bit of detail. The first roundtrip is a GET request that asks for references and server capabilities.

Roundtrip 1 · GET /schacon/soe.git/info/refs?service=git-upload-pack
  • ?service=
    git-upload-pack     Selects the service. git-upload-pack means "I want to fetch.", git-receive-pack means "I want to push".
  • 001e# service=… A smart-HTTP-only preamble that lets the client confirm it reached a smart server (not a dumb file server), followed by a 0000 flush.
  • HEAD\0…caps Here you can see the capabilities list is just tucked behind a NUL byte after the first ref. This allowed the addition of capabilities without technically changing the protocol, but it's why the capability list was size-limited and awkward to grow.
  • symref=HEAD:
    refs/heads/main     Another sort of hack that allows the smart protocol to smuggle the fact that HEAD is symbolic, because there was nowhere else to put this information.
  • refs/pull/1/head Finally, notice that it's advertising every reference. On a large repo this can result in a huge amount of data over the wire before anything else.

The second request is a POST body that says what heads we want data for and the response is the packfile and sideband information to allow the client to show progress as it’s downloading the streamed body.

Roundtrip 2 · POST /schacon/soe.git/git-upload-pack
  • want <oid> caps The client asks for objects by object id (not by name). It wants both tips — ee93348… (main/HEAD) and 9cb163a… (sc-branch-1). The capabilities it actually intends to use ride on the first want line.
  • 0000 then done A flush ends the want list; done tells the server to stop negotiating and just send a pack. On a fresh clone there are no have lines — the client has nothing yet. On a fetch we would also send the stuff we already have so the server can determine the smallest packfile needed.
  • NAK "No common commit found". Because this is a clone. In an incremental fetch this is where ACK <oid> lines would appear as the two sides find shared history.
  • \x01 / \x02 The packfile is multiplexed on a side-band: a leading byte per chunk marks channel 1 (pack bytes), 2 (progress), or 3 (error). This is so the client can do the "Receiving objects…" meter that you see on a clone.

The v2 Protocol

Here’s that same clone, this time in v2. This time it’s three round trips rather than two. The first just to determine capabilities. Then a ref advertisement using the ls-refs command. Then a packfile request using the fetch command.

Let’s take a look at each of them.

Roundtrip 1 · GET /schacon/soe.git/info/refs?service=git-upload-pack
  • Git-Protocol:
    version=2     This new header lets the server know that we're dealing with a v2 capable client so it knows to respond differently.
  • version 2 The server confirms it speaks v2.
  • ls-refs= / fetch= These capabilities are commands that the client can invoke. Their values advertise sub-features: fetch here supports shallow, wait-for-done, and filter (partial clone).

OK, so now the client knows what the server can do, including what object format the repository is in (sha1 or sha256).

Now that we know that, we can then start POSTing commands to run. For a clone, we start with ls-refs, which is similar to the first GET we did in the previous protocol, but crucially we can filter the returned values.

This is a little funny, because now in v2 the GET to /info/refs does not return refs, it returns capabilities. Then a POST to /git-upload-pack with a ls-refs command is used to actually return the refs we want to inspect (as upload-pack is now just a service endpoint used to run any of several commands).

Roundtrip 2 · POST /schacon/soe.git/git-upload-pack · command=ls-refs
  • command=ls-refs The first line of the POST body is the actual command to run. Here we're asking to run ls-refs
  • 0001 delimiter Separates the capabilities the client is using from the command's arguments. (A 0000 flush ends the whole request.)
  • peel Show peeled tags — appends peeled: to a tag's line (the commit the tag ultimately dereferences to).
  • symrefs For a symbolic ref, also show what it points at — appends symref-target: (e.g. HEAD symref-target:refs/heads/main).
  • ref-prefix … Here is the filtering argument. The client asks only for refs/heads/, refs/tags/, and HEAD. It could also only ask for a single ref.
  • symref-target: The symbolic-ref info now has a proper place in the response, instead of being smuggled into a capability string as it was in v0.
  • 0002 The response-end-pkt, so the client can tell the response is complete over a stateless HTTP POST.

Now we have a list of the objects we want and can ask for the packfile with the fetch command (basically the only thing git-upload-pack did before v2).

Roundtrip 3 · POST /schacon/soe.git/git-upload-pack · command=fetch
  • command=fetch Again, the command to run. This time we're going to tell the server what we want and have and it will generate a custom packfile for us.
  • done A fresh clone has nothing to negotiate, so the client sends done. Because it did, the server skips the acknowledgments section and jumps straight to the pack data
  • packfile A named response section. v2 responses are structured into labelled sections (acknowledgments, shallow-info, wanted-refs, packfile); here only the pack is needed.
  • \x01 … 0002 The pack is side-band-multiplexed just like the old protocol, then 0002 ends the response.

In this simple example, in practice, there is actually a lot more going over the wire in v2 than in the first version of the protocol. There are 3 round trips consisting of 456 bytes, instead of 2 and 234 bytes.

However, in practical terms, v2 normally saves a lot of traffic because it can filter the references it cares about. Some larger repositories have many megabytes of reference data, so that first refs advertisement is killer overhead.

What all can a Git server do nowadays?

So this is really mostly what I didn’t know. What can a git server do these days outside of listing references and negotiating packfiles? What all commands can you tell it to advertise and allow clients to invoke?

The reality is that out of the box, it’s pretty much nothing. Outside of ls-refs and fetch the vanilla server only advertises server-option and object-format, which aren’t actually commands, they’re just simple ways to pass other data (to send arbitrary values to custom servers like Gerrit, and tell the client the hash-format).

There are other commands (like the bundle-uri and packfile-uri I mentioned at the start of the article), but they’re all gated behind server (and sometimes also client-side) config options.

Here is a full list of the current potential capabilities of a v2 Git server (green rows are enabled by default, gray must be configured on) and when they were introduced.

Capability Since Does Purpose

Now, there is a lot to potentially dig into here (perhaps a future blog post), but what’s really interesting is how old some of these capabilities are and how very, very rarely used they are.

Nearly all of them landed over 5 years ago and nearly none of them are available on any major code host. GitHub and GitLab, for example, only enable fetch.filter so you can do blobless clones. None of the rest of these capabilities are enabled (well, GitLab can do bundle-uris on self-hosted instances, I think, but not by default).

Honestly, most of this is by and for Google. 11 of the 14 capabilities were contributed by Google and nearly all of the configurable options were also contributed by them. So, most of v2 seems to be so Google can scale Git weirdly internally.

But it’s cool.

See it for yourself

If you want to check any of this out yourself, it’s actually pretty easy to see what is going over the wire. If you set the GIT_TRACE_PACKET environment variable and run Git networking stuff, it will show you what it’s doing.

$ GIT_TRACE_PACKET=1 git ls-remote https://github.com/schacon/soe
12:37:45.271670 pkt-line.c:85           packet:          git< # service=git-upload-pack
12:37:45.272656 pkt-line.c:85           packet:          git< 0000
12:37:45.272691 pkt-line.c:85           packet:          git< version 2
12:37:45.272716 pkt-line.c:85           packet:          git< agent=git/github-cda1d7094a30-Linux
12:37:45.272732 pkt-line.c:85           packet:          git< ls-refs=unborn
12:37:45.272747 pkt-line.c:85           packet:          git< fetch=shallow wait-for-done filter
12:37:45.272760 pkt-line.c:85           packet:          git< server-option
12:37:45.272773 pkt-line.c:85           packet:          git< object-format=sha1
12:37:45.272785 pkt-line.c:85           packet:          git< 0000
12:37:45.272956 pkt-line.c:85           packet:    ls-remote< version 2
12:37:45.274390 pkt-line.c:85           packet:    ls-remote< agent=git/github-cda1d7094a30-Linux
12:37:45.274425 pkt-line.c:85           packet:    ls-remote< ls-refs=unborn
12:37:45.274446 pkt-line.c:85           packet:    ls-remote< fetch=shallow wait-for-done filter
12:37:45.274463 pkt-line.c:85           packet:    ls-remote< server-option
12:37:45.274477 pkt-line.c:85           packet:    ls-remote< object-format=sha1
12:37:45.274491 pkt-line.c:85           packet:    ls-remote< 0000
12:37:45.274508 pkt-line.c:85           packet:    ls-remote> command=ls-refs
12:37:45.274552 pkt-line.c:85           packet:    ls-remote> agent=git/2.52.0-Darwin
12:37:45.274576 pkt-line.c:85           packet:    ls-remote> object-format=sha1
12:37:45.274594 pkt-line.c:85           packet:    ls-remote> 0001
12:37:45.274561 pkt-line.c:85           packet:          git< command=ls-refs
12:37:45.274612 pkt-line.c:85           packet:    ls-remote> peel
12:37:45.274656 pkt-line.c:85           packet:          git< agent=git/2.52.0-Darwin
12:37:45.274714 pkt-line.c:85           packet:    ls-remote> symrefs
12:37:45.274744 pkt-line.c:85           packet:          git< object-format=sha1
12:37:45.274779 pkt-line.c:85           packet:    ls-remote> unborn
12:37:45.274819 pkt-line.c:85           packet:          git< 0001
12:37:45.274893 pkt-line.c:85           packet:          git< peel
12:37:45.274919 pkt-line.c:85           packet:          git< symrefs
12:37:45.274860 pkt-line.c:85           packet:    ls-remote> 0000
12:37:45.274968 pkt-line.c:85           packet:          git< unborn
12:37:45.275097 pkt-line.c:85           packet:          git< 0000
12:37:45.490316 pkt-line.c:85           packet:          git> 0002
12:37:45.490332 pkt-line.c:85           packet:    ls-remote< ee93348afd51e0d634d9e84a702af190d45cf9e3 HEAD symref-target:refs/heads/main
12:37:45.490569 pkt-line.c:85           packet:    ls-remote< ee93348afd51e0d634d9e84a702af190d45cf9e3 refs/heads/main
12:37:45.490611 pkt-line.c:85           packet:    ls-remote< 9cb163aaa43273f49de0182416d3a621b3ae8292 refs/heads/sc-branch-1
12:37:45.490646 pkt-line.c:85           packet:    ls-remote< 9cb163aaa43273f49de0182416d3a621b3ae8292 refs/pull/1/head
12:37:45.490671 pkt-line.c:85           packet:    ls-remote< 0000
12:37:45.490874 pkt-line.c:85           packet:    ls-remote< 0002
ee93348afd51e0d634d9e84a702af190d45cf9e3	HEAD
ee93348afd51e0d634d9e84a702af190d45cf9e3	refs/heads/main
9cb163aaa43273f49de0182416d3a621b3ae8292	refs/heads/sc-branch-1
9cb163aaa43273f49de0182416d3a621b3ae8292	refs/pull/1/head

Well, that’s it. Hope you learned something, because I certainly did.