Today, I will show how to perform custom git clone on Azure DevOps build pipeline and be able to manage source code cloning on Azure DevOps build engine in a more flexible way.

The problem: In one of my projects I had relatively large git repository due to handling big binary files (design images, documentation, etc.) within git repository. These files were not directly related to source code. Therefore, overall git size become relatively huge, but source code was still fairly small.

I used Azure DevOps build pipelines to build my app. Azure DevOps pipelines allow relatively small control over source code cloning process. Normally, whole deep clone is executed every time the build is triggered. Because of this, my build took a lot of time just to download the source code to build machine.

Possible solution: It would be nice to clone only the last commit on specific branch (e.g. development, stage, prod). This is very simple with raw git command, I just need to figure out how to overcome this limitation on on Azure DevOps.

As said, out-of-the-box Azure DevOps does not allow this kind of flexibility. Basically, Azure DevOps only allows simple full git cloning or small area of tuning.

The problem

I have relative large git repository and I need to build my app on my Azure DevOps. Let’s first check straightforward approach by simply cloning my git repository into Azure DevOps pipeline. I just want to check how long this action takes.

For brevity, I create empty build pipeline which only clones source code from my custom git repository. When I trigger the build I get this situation:

So, it takes almost 8 minutes to clone my git repository to the build machine. Data transferred from my custom git server to the Azure DevOps is 2.25 GB. Can this be optimize in any way?

What would I like to achieve is to run git command like this:

In short: I would like to clone dev branch from my git repository with only latest commit – no git history, tags, other branches or anything else. Just current, latest snapshot my source code repository.

Solution

The idea is to try to invoke git from batch or shell task on Azure DevOps pipeline. So, initial (obligatory) git clone will only be for fake purposes. In order to build my application, I need latest commit from my git on specific branch. So let’s check how I can do this.

Azure DevOps allows two approaches to define build pipeline: YAML (code) or classic (visual) editor approach. I will show both ways, but first start with YAML.

YAML Azure DevOps pipeline definition

When I create new pipeline, I select default – YAML approach – to define build pipeline.

Azure DevOps provide several git/subversion providers from where I can pull the code. It enables pulling code from Azure DevOps git repository, Bitbucket, GitHub, GitHub Enterprise, Custom git and Subversion.

As said before, here I will use “fake” git repository just for obligatory demands of Azure DevOps pipelines. After initial clone I will use custom git command inside shell or batch task to clone my code.

First, I select empty git repository.

Next, I create simple YAML build pipeline definition from prefabricated options.

Then, I update my YAML build pipeline definition with two tasks:

• task for cleaning existing artifacts in Azure DevOps $(Build.SourcesDirectory) folder.
• custom git clone command. Again, inside shell/bash/powershell task.

Anyway, long story short, let’s take a look on my YAML definition:

I save everything, start the build and my result output is:

Wow, from full git clone which took approx 7 minutes I reduced clone task to 11 secs. Yes, from 7 minutes to 11 seconds. Awesome, right?

And data transfer reduction: from 2.25GB to 22.1MB. Amazing.

This way I have latest snapshot of my source code to bi build on my build machine. Let’s see how this looks on classical Azure DevOps pipeline editor.

Classical Azure DevOps pipeline editor

Procedure is analogues as with YAML: I create new pipeline, select classical editor, select dummy/fake Azure DevOps git repository and add two “Command line” tasks.

First is for cleaning sources folder, the other one clones my git repository. Images below depicts these two tasks with script executed.

Cleaning task:

Custom git clone build pipeline task:

Result is basically the same as with YAML version. Huge reduction in git clone time and amount of data transfer.

Conclusion

Azure DevOps is amazing tool. For developers, build pipelines are very useful tools to automate building processes. Nevertheless, sometimes some out-of-the-box features are just not enough flexible. Well, with some tricks and workarounds, Azure DevOps can be tuned quite a bit.

In this blog post, I presented how to trick Azure DevOps to clone source code with custom git command. I presented an easy and useful trick how to clone source code on Azure DevOps build pipeline engine.

Until the next reading, happy coding.