Optimize your git clone / fetch strategy for CI pipelines
- 15 September 2021
- •
- 1 min
During the migration of our Frontend Monorepo from an on-premise Jenkins pipeline to a cloud based solution, I had to overcome a big obstacle: dealing with the sheer size of our monorepo. If we include the whole history (all branches on remote, all tags, full history), the size of our repo amounts to more than 1GB. Downloading that much data on each run is a waste of bandwidth and time, so I've dived into the git documentation and found several possibilities that I’d like to share with you.
You only need to checkout / build a single branch
git clone –single-branch
This simple flag (–single-branch) makes sure you only fetch the history of your main branch. The more active branches live on your remote, the bigger the benefit and with more than 400 active contributors in our repo, that helps!
git clone –no-tags
Another simple flag! This one omits the tags from the git history. Tags are often version numbers such as @1.2.67. Although not a huge data saver, it helps. Be careful though, some CI pipelines need tags, so double check if you need it.
git clone –depth=
Oh yes, now we’re talking! This flag restricts the history to the latest commits. Really useful if you only care about the latest state of a certain branch. For example, if you know you need to build a feature branch (or your main branch) and you don’t need all its history! Potential huge data savings ?.
Combining the flags git clone –no-tags –single-branch –depth=1
If you just need the latest state of a single branch, then this is your command! You can also modify this command to checkout a specific branch by adding –branch= if you are building a branch that is not your main.
You need to more than one branch (but not all)
This is where things become a bit more tricky. In this scenario git clone might not be our best bet because it only allows for cloning all branches or one branch.
This was also the scenario which was most tricky for me, because I needed to determine a single “ancestor” commit where a feature branch started to branch of the main branch.
git clone –depth=
As we saw, this command fetches all active branches with limited history. This might work for you if you’re interested in the most recent state of two (or more branches). Downsides of this command is that it still fetches all branches and that determining “ancestor” commits is difficult/impossible (depends on depth, but can still be inaccurate).
Custom git config
This one might seem scary, but can actually be really effective. You have full control over the number of branches you want to track, if you would like tags and still control depth of the history
Steps:
1 – Initialize an empty git repository by running git init
2 – Add the following snippet to your .git/config file
# .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote \"origin\"]
# replace with your own repo!
url = git@github.com:patricksevat/foo.git
# list the branches that you want to track
fetch = +refs/heads/main:refs/remotes/origin/main
fetch = +refs/heads/development:refs/remotes/origin/development
fetch = +refs/heads/feature:refs/remotes/origin/feature
# Replace main with master or whatever your main branch name is
[branch \"main\"]
remote = origin
merge = refs/heads/main3 – Now you can git fetch with whatever flags you like. Do not add any flags for full history of the listed branches, or use –no-tags or –depth=1 to limit history.
4 – Don’t forget to run git checkout with one of your fetched branches or you’ll have an empty working directory!
Conclusion
I hope these commands help you in optimizing your own git CI strategy. If you have any useful tips or improvements please leave them in the comments!
About the author
Patrick is crazy about JS/TS and has the small ambition to transform Rabobank from a bank with an IT department to a tech company with a banking license.
Related articles
Do you really need that feature?
These 6 solutions are making our lives as easy as we can
