Programmatically export project, build and release definitions to new ones in VSTS / Azure DevOps via REST API

Let’s assume we have an existing project, with its own git repository and some very complex build and release definitions. Let’s assume that you want to create a service able to recreate the exact copy of this structure; it is not unlikely that you could have to deal with git repository conflict, custom extensions not being loaded or service endpoints not existing in the new project. How to deal with these issues and properly achieve our goal?

Before starting, you may be thinking: “why do I have to do this?”

Well, this approach could be powerful in scenarios where you, as DevOps team, may want to export a self-provision-able template for spreading build/release best practices in your work environment. Think about it: your developers want only to think about coding and not dealing with Azure DevOps powerful complexity.

Checklist (a.k.a. “What I should check before getting started”)

Do you have any task from a custom extension, in your build/release pipeline definition?
Are you missing what “service endpoint” concept is?
Are you connecting to external providers different than other VSTS/ADO git repositories or Azure Resource Manager?

If any of the following questions has a “yes” as answer, before continuing, please scroll down and go to the F.A.Q. section

Ready? Let’s start

myOldProject is a project for an ASP.Net MVC application, with a Dockerfile for its containerization and a Helm chart definition for its deployment on AKS (cool managed Kubernetes cluster offering on Azure)

In the next paragraphs you will learn how to migrate myOldProject to myNewProject, i.e.:

Create a new Azure DevOps/VSTS project
Create “gitexternal” and “azurerm” Service Endpoints
Import an existing Azure DevOps git repository
Export, parametrize, import a build definition
Export, parametrize, import a release definition

REMEMBER:
For all the following calls to Azure DevOps REST API, I am assuming an a basic authentication with Personal Access Tokens. Check here for other possibilities

Azure DevOps project creation

First of all, you will need to create a new Azure DevOps Project. The request body will contain a name, a description but above all the processTemplate you want to use

AGILE = “adcc42ab-9882-485e-a3ed-7678f01f66bc”
SCRUM = “6b724908-ef14-45cf-84f8-768b5384da45”
CMMI = “27450541-8e31-4150-9947-dc59f998fc01”

Request:

1
POST https://dev.azure.com/{organization}/_apis/projects?api-version=6.0

Body:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "name": "myNewProject",
  "description": "myNewProject",
  "capabilities": {
      "versioncontrol": {
          "sourceControlType": "Git"
      },
      "processTemplate": {
          "templateTypeId": "adcc42ab-9882-485e-a3ed-7678f01f66bc"
      }
  }
}

Create “gitexternal” and “azurerm” service endpoints

You will be importing the existing git repository of myOldProject, and the build/release pipelines will need access to some Azure Resources like Azure Container Registry (docker push) or Azure Kubernetes Service (helm install).

For this reason, in this example, you will need to create two service endpoints:

A “gitexternal” service endpoint, with an authentication scheme “UsernamePassword” (if you want to use a Personal Access Token)
A “azurerm” service endpoint, with an authentication scheme “ServicePrincipal” (using appId and password from a ServicePrincipal with an appropriate role in your Azure Active Directory)

Request:

1
POST https://dev.azure.com/{organization}/_apis/serviceendpoint/endpoints?api-version=6.0-preview.4

Body for gitexternal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
    "name": "gitexternal",
    "type": "git",
    "authorization": {
        "parameters": {
            "username": "",
            "password": "base64 encoding of PAT"
        },
        "scheme": "UsernamePassword"
    },
    "url": "https://{organization}@dev.azure.com/{organization}/myOldProject/_git/{repositoryName}"
}

N.B.: This will return a serviceEndpointId you will need later, take note of it

Body for azurerm:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
    "name": "myServiceEndpointSP",
    "type": "azurerm",
    "url": "https://management.azure.com/",
    "authorization": {
        "parameters": {
            "serviceprincipalid": serviceprincipalId,
            "serviceprincipalkey": serviceprincipalKey,
            "tenantid": aadTenantId
        },
        "scheme": "ServicePrincipal"
    },
    "data": {
        "SubscriptionId": subscriptionId,
        "SubscriptionName": subscriptionName
    }
}

Import an existing Azure DevOps git repository

You can submit a POST request to start the async operation of importing the existing git repository.

Request:

1
POST https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/importRequests?api-version=6.0-preview.1

Body:

1
2
3
4
5
6
7
8
{
    "parameters": {
        "gitSource": {
            "url": "https://{organization}@dev.azure.com/{organization}/myOldProject/_git/{repository}"
        },
        "serviceEndpointId": SERVICE_ENDPOINT_ID
    }
}

As it is an async operation, it is very useful to check the progress of the import process. You will definitely want to move on when the “status” property is equal to “completed”:

Get Git import request status:

1
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/importRequests/{importRequestId}?api-version=6.0-preview.1

Export/import a build definition

List Build Definitions and Get a specific one by its id:

1
2
GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=5.0-preview.7
GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=5.0-preview.7

This is actually everything you need to get and export a build definition, simple right?

By the way, what you just exported, it contains a lot of “project/organization/user”-specific information that you have to delete/modify for a template creation.

The fields you can safely delete are:

_links
authoredBy
queue._links
queue.url
queue.id
url
uri
revision
createdDate
id

and then change

name
“project” property (returned by project creation or GET https://dev.azure.com/{organization}/_apis/projects/{projectId}?api-version=5.0-preview.3)
“git repository”-specific information (returned by GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories?api-version=5.0-preview.1):
- repository.id with value[0].id
- repository.name with value[0].name
- repository.url with value[0].remoteUrl
task specific information (such as Azure Subscription or Azure Container Registry service endpoint references)
process.phases[0].steps[2].inputs.azureSubscriptionEndpoint
process.phases[0].steps[2].inputs.azureContainerRegistry

With these changes you will have a JSON of a “importable” build definition. Of course, you could do more modifications to it, depending on what you want to achieve.

When you are done with parametrizing your JSON definition, you can submit it to your new project with a simple POST request

1
POST https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=5.0-preview.7

Please, take note of “id” and “queue.id” properties of the response. They could be needed for the release definition if, in your scenario, the release uses the outputs of a build.

Export/import a release definition

As for build definition, you can retrieve the list of myOldProject release definitions and then export the one desired:

1
2
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.0-preview.3
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions/{definitionId}?api-version=5.0-preview.3

You will have to remove old specific information like:

createdby
createdon
modifiedby
modifiedon
revision
environments[0].currentRelease
environments[0].badgeUrl
artifacts[0].definitionReference.artifactSourceDefinitionUrl
url
_links
id

and properly change:

artifacts[0].sourceId has to be projectId:buildDefinitionId
artifacts[0].definitionReference.project.id
artifacts[0].definitionReference.repository.id
artifacts[0].definitionReference.definition.id
environments[0].deployPhases[0].deploymentInput.queueId
environments[0].owner.id (could be the “createdby.Id” property from build definition create POST)
task specific information environments[0].deployPhases[0].workflowTasks[2].inputs.azureSubscriptionEndpoint
environments[0].deployPhases[0].workflowTasks[2].inputs.azureResourceGroup
environments[0].deployPhases[0].workflowTasks[2].inputs.kubernetesCluster

We are now ready to submit your release definition to myNewProject with another POST request:

1
POST https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.0-preview.3

Conclusions

You are finally done, Azure DevOps Services REST APIs for programmatically cloning projects are no more a mystery for you!

We achieved having a “clone” of myOldProject to myNewProject, with original git repository code and history, original build and release definitions.

F.A.Q.:

I am using one or more custom extensions, will I have problems? Custom extensions are installed at the organization level. For this reason, if you are exporting to a new project in the same organization you will have no issues. Otherwise, if you are exporting to a new project in a different organization you have to check if the extension is already there.I suggest you to:
- GET all installed extensions on the source organization
  1
  GET https://extmgmt.dev.azure.com/{organization}/_apis/extensionmanagement/installedextensions?api-version=5.0-preview.1
- Take note of the publisherId and extensionId from the custom extensions, by excluding the ones with the “builtin” flag
- POST to install custom extensions you need on the target organization
  1
  POST https://extmgmt.dev.azure.com/{organization}/_apis/extensionmanagement/installedextensionsbyname/{publisherId}/{extensionId}/{version}?api-version=5.0-preview.1
What a “service endpoint” is? From ADO docs: “For Azure DevOps Services and Team Foundation Server to be able to connect to the external service, in addition to using the credentials, there is also need to know how to set the credentials in the HTTP request header when calling the external endpoint. Azure DevOps Services supports a closed set of authentication schemes that can be utilized by a custom service endpoint type”
I am using a private Nuget feed, a private npm repo, a Service Fabric Cluster, a Service Bus or some other different endpoint. How do I create a proper service endpoint to authenticate and authorize the requests? Doing the following GET request you can retrieve the list of the available service endpoints, with matching URLs and authentication schemas:
1
GET https://dev.azure.com/{organization}/_apis/serviceendpoint/types?api-version=5.0-preview.1

Checklist (a.k.a. “What I should check before getting started”)#

Ready? Let’s start#

Azure DevOps project creation#

Create “gitexternal” and “azurerm” service endpoints#

Import an existing Azure DevOps git repository#

Export/import a build definition#

Export/import a release definition#

Conclusions#

F.A.Q.:#

Checklist (a.k.a. “What I should check before getting started”)

Ready? Let’s start

Azure DevOps project creation

Create “gitexternal” and “azurerm” service endpoints

Import an existing Azure DevOps git repository

Export/import a build definition

Export/import a release definition

Conclusions

F.A.Q.: