[enhancement]: Support Sparse Checkout

[bdovaz]

Describe your feature request here

For monorepo repositories this is a must to be able to speed up builds as much as possible.

I see that this very thing has been recently merged by @dscho into the GitHub checkout action: actions/checkout#1369

Would this same functionality be possible in azure pipelines?

Thanks!

[KonstantinTyukalov]

Hi @bdovaz thanks for the reporting! We're currently working on higher priority issues, but we'll get back this this one soon.

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[bdovaz]

No stale

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

Note that partial clones are now supported by Azure Repos!

[bdovaz]

Although it is an improvement, it has nothing to do with the purpose of this issue.

The big improvement would be that at a high level through the checkout task we can make a sparse checkout indicating the directories to include as you can do in Github from your PR from half a year ago.

[sanghel-orbyta]

any way to bump priority on this one?

monorepos consume agent time uselessly doing full checkouts...

[ToreDemant]

Seems to be related to these:
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
https://developercommunity.visualstudio.com/t/Support-Gits-sparse-checkout-in-the-Azu/10472337

[Khayoann2]

I proposed this one
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
So i’m also waiting this feature …
Please 🙏

[mggrand]

If anyone is interested, I wrote the sparse checkout as a template, which can then be used as follows:

- template: sparse-checkout.yml
    parameters:
      folders:
        - frontend
        - resources

with the template sparse-checkout.yml

parameters:
  - name: folders
    type: object
    default:

  - name: fetchLfs
    type: boolean
    default: true

  - name: persistCredentials
    type: boolean
    default: false

steps:
  - checkout: none
  - pwsh: |
      function Start-Group($name){ Write-Host "##[group]$name"}
      function End-Group { Write-Host "##[endgroup]"}

      function Run-Cmd($command){
          Write-Host "##[command]$command"
          Invoke-Expression $command
      }


      $folders = "${{ join(' ',parameters.folders) }}"
      $repositoryUri = "$(Build.Repository.Uri)"
      Write-Host "Sync repository folders '$folders' to '$pwd'"
            
      Start-Group "Getting git version info"
      Run-Cmd "git version"
      Run-Cmd "git lfs version"
      End-Group

      if(Test-Path ".git"){
        Start-Group "Cleanup existing repository"
        # Existing directory
        $null = Remove-Item ".git/index.lock" -Force -ErrorAction SilentlyContinue
        $null = Remove-Item ".git/shallow.lock" -Force -ErrorAction SilentlyContinue

        $currentRemote = Run-Cmd "git remote get-url origin"
        if($currentRemote -ne $repositoryUri){
            Write-Host "Current fetch url: $currentRemote"
            Write-Host "New repository fetch url: $repositoryUri"
            Write-Host "##[warning]Fetch urls do not match. Clean the whole directory"
            $null = Remove-Item ./* -Force -Recurse
        }
        else{
            Run-Cmd "git remote remove origin"
            Run-Cmd "git clean -ffxd"
            Run-Cmd "git reset --hard"
        }
        End-Group
      }

      Start-Group "Initialize the repository"
      Run-Cmd "git init"
      Run-Cmd "git sparse-checkout init --cone"
      if($folders.Length -gt 1){
        Run-Cmd "git sparse-checkout add $folders"
      } else {
        Write-Host "No folders were provided for the sparse checkout"
      }
      Run-Cmd "git lfs install --local"
      End-Group

      Start-Group "Configure the repository"
      Run-Cmd "git remote add origin $repositoryUri"
      Run-Cmd "git config core.sparsecheckout true"
      Run-Cmd "git config gc.auto 0"
      Run-Cmd "git config --get-all http.$($repositoryUri).extraheader"
      Run-Cmd "git config --get-all http.proxy"
      Run-Cmd "git config http.version HTTP/1.1"
      End-Group


      Start-Group "Fetching the repository"
      Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader fetch --force --no-tags --progress --prune --no-recurse-submodules --verbose --depth=1 origin +$(Build.SourceVersion):refs/remotes/origin/$(Build.SourceVersion)"
      if("${{ parameters.fetchLfs }}" -eq "true"){
        Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader lfs fetch origin refs/remotes/origin/$(Build.SourceVersion)"
      } else {
        Write-Host "Skipping lfs fetch"
      }
      End-Group


      Start-Group "Checkout the repository"
      Run-Cmd "git checkout --force --progress refs/remotes/origin/$(Build.SourceVersion)"
      End-Group

      if("${{ parameters.persistCredentials }}" -eq "true"){
        Start-Group "Persist credentials in the repository"
        Run-Cmd "git config http.$($repositoryUri).extraheader `"$($env:env_var_http_extraheader)`""
        End-Group
      }

      if("$($env:env_var_vsts_debug)" -eq "true"){ 
        Start-Group "Print directory outputs"
        Run-Cmd "Get-ChildItem -Recurse"
        End-Group
      }

    displayName: "Clone partial git repo"
    env:
      env_var_http_extraheader: "AUTHORIZATION: bearer $(System.AccessToken)"
      env_var_vsts_debug: $[eq(variables['System.Debug'], true)]

[bdovaz]

https://developercommunity.visualstudio.com/t/Support-Gits-sparse-checkout-in-the-Azu/10472337#T-N10776291

[bdovaz]

Added in #5015 and will be in release v3.247.1

Closing...

[bdovaz]

There is no documentation for this feature yet?

https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines

cc @jnilau @merlynomsft

[PaulVrugt]

docs would be really helpful

[PaulVrugt]

so, trying to use this in azure devops gives me:

(Line: 91, Col: 9): Unexpected value 'sparseCheckoutDirectories'

and another Object reference not set to an instance of an object. error

could it be that azure devops still needs an update before this is supported?

[ChristopherHX]

ADO has exactly two syntaxes, the less familiar one is working right know in dev.azure.com, the short notation needs server side updates to the schema and eventually more.

6d15af64-176c-496d-b583-fd2ae21d4df4 is the taskid of the agents checkout task, version 1 the only available version.

variables:
- name: AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK
  value: true
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4@1
  inputs:
    sparseCheckoutDirectories: .vscode
    repository: self
- script: ls -la

If you need AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK or not has not been investigated

You would see the git sparse commands in the shell.

 * branch            6ccd4af5a724d5b17713a6280806877f23a8fabe -> FETCH_HEAD
git sparse-checkout init --cone
git sparse-checkout set .vscode
git checkout --progress --force refs/remotes/origin/6ccd4af5a724d5b17713a6280806877f23a8fabe

[jnilau]

@bdovaz documentation is out of my hands but it is in the works

[jnilau]

@PaulVrugt there was a holiday deployment freeze but it has ended now and we will be continuing rollout on the ADO side

[baterja]

It's good to finally have it! I'm still waiting until it's documented before I show it to my teams. Is there an issue (or anything else that may be tracked) on the docs side?

[PaulVrugt]

I just logged issue #5113 @bdovaz

[arielmoraes]

How to set the agent knob when using the managed agent?

[ChristopherHX]

@arielmoraes if you really need the knob do this like (this worked a long time ago #4325 (comment) on hosted agents (managed agent could be the same))
add this to your pipeline root

variables:
- name: AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK
  value: true

Some knobs are setable from variables, others as env variable

Or did you try this already?

[arielmoraes]

@ChristopherHX yes, I've tried that, but it seems we can't change system variables:

Cannot override system variable 'AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK'

[ChristopherHX]

@arielmoraes Are you trying to use this command https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-variables-scripts?view=azure-devops&tabs=bash? Then this is your problem

This only works in the pipeline yaml file in the declarative variables block, I can set it to true and false and get feedback in logs

Initialize job should show the value e.g.

There is no message like you get.

If not I have honestly no idea, then is your azurepipelines deployment different than mine, I'm using a https://dev.azure.com/cloud version

[arielmoraes]

@ChristopherHX yes I'm using the variables block, are you using a Microsoft hosted agent or self hosted?

[arielmoraes]

I don't know how you are managing to change the variable because the documentation clearly states Predefined variables are readonly and can't be changed, with the exception of Build.Clean and System.Debug.

https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

EDIT:

Turns out the variable is already set to true without any variables, but the checkout task is not working.

[ChristopherHX]

are you using a Microsoft hosted agent or self hosted?

Microsoft hosted agent

Ok since for me AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK is not present by default, you have a newer deployment that I have in my instance.
Which also implies AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK is not readonly for me.
Weirdness of rolling deployment (I have no idea how to even view the version of what I have)

but the checkout task is not working

I guess if this is the case, report it as new issue in this repo with the log attached