Programmatically export project, build and release definitions to new ones in VSTS / Azure DevOps via REST API

Let’s assume we have an existing project, with its own git repository and some very complex build and release definitions.
Let’s assume that you want to create a service able to recreate the exact copy of this structure; it is not unlikely that you could have to deal with git repository conflict, custom extensions not being loaded or service endpoints not existing in the new project. How to deal with these issues and properly achieve our goal?

Before starting, you may be thinking: “why do I have to do this?”

Well, this approach could be powerful in scenarios where you, as DevOps team, may want to export a self-provision-able template for spreading build/release best practices in your work environment. Think about it: your developers want only to think about coding and not dealing with Azure DevOps powerful complexity.

Checklist (a.k.a. “What I should check before getting started”)

  • Do you have any task from a custom extension, in your build/release pipeline definition?
  • Are you missing what “service endpoint” concept is?
  • Are you connecting to external providers different than other VSTS/ADO git repositories or Azure Resource Manager?

If any of the following questions has a “yes” as answer, before continuing, please scroll down and go to the F.A.Q. section

Ready? Let’s start

myOldProject is a project for an ASP.Net MVC application, with a Dockerfile for its containerization and a Helm chart definition for its deployment on AKS (cool managed Kubernetes cluster offering on Azure)

In the next paragraphs you will learn how to migrate myOldProject to myNewProject, i.e.:

  1. Create a new Azure DevOps/VSTS project
  2. Create “gitexternal” and “azurerm” Service Endpoints
  3. Import an existing Azure DevOps git repository
  4. Export, parametrize, import a build definition
  5. Export, parametrize, import a release definition

REMEMBER:

For all the following calls to Azure DevOps REST API, I am assuming an a basic authentication with Personal Access Tokens.
Check here for other possibilities

Azure DevOps project creation

First of all, you will need to create a new Azure DevOps Project.
The request body will contain a name, a description but above all the processTemplate you want to use

AGILE = “adcc42ab-9882-485e-a3ed-7678f01f66bc”
SCRUM = “6b724908-ef14-45cf-84f8-768b5384da45”
CMMI = “27450541-8e31-4150-9947-dc59f998fc01”

POST https://dev.azure.com/{organization}/_apis/projects?api-version=5.0-preview.3

Body:
{
    "name": "myNewProject",
    "description": "myNewProject",
    "capabilities": {
        "versioncontrol": {
            "sourceControlType": "Git"
        },
        "processTemplate": {
            "templateTypeId": "adcc42ab-9882-485e-a3ed-7678f01f66bc"
        }
    }
}
Azure DevOps project creation

Create “gitexternal” and “azurerm” service endpoints

You will be importing the existing git repository of myOldProject, and the build/release pipelines will need access to some Azure Resources like Azure Container Registry (docker push) or Azure Kubernetes Service (helm install).

For this reason, in this example, you will need to create two service endpoints:

  1. A “gitexternal” service endpoint, with an authentication scheme “UsernamePassword” (if you want to use a Personal Access Token)
  2. A “azurerm” service endpoint, with an authentication scheme “ServicePrincipal” (using appId and password from a ServicePrincipal with an appropriate role in your Azure Active Directory)
https://dev.azure.com/{organization}/{project}/_apis/serviceendpoint/endpoints?api-version=5.0-preview.2
POST for ServiceEndpoint creation
"url" parameter should be the from git repository to import

{
    "name": "gitexternal",
    "type": "git",
    "authorization": {
        "parameters": {
            "username": "",
            "password": "base64 encoding of PAT"
        },
        "scheme": "UsernamePassword"
    },
    "url": "https://{organization}@dev.azure.com/{organization}/myOldProject/_git/{repositoryName}"
}

N.B.: This will return a serviceEndpointId you will need later, take note of it
Request body: gitexternal ServiceEndpoint
{
    "name": "myServiceEndpointSP",
    "type": "azurerm",
    "url": "https://management.azure.com/",
    "authorization": {
        "parameters": {
            "serviceprincipalid": serviceprincipalId,
            "serviceprincipalkey": serviceprincipalKey,
            "tenantid": aadTenantId
        },
        "scheme": "ServicePrincipal"
    },
    "data": {
        "SubscriptionId": subscriptionId,
        "SubscriptionName": subscriptionName
    }
}
Request body: azurerm ServiceEndpoint

Import an existing Azure DevOps git repository

You can submit a POST request to start the async operation of importing the existing git repository.

POST https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/importRequests?api-version=5.0-preview.1

Body: 
{
    "parameters": {
        "gitSource": {
            "url": "https://{organization}@dev.azure.com/{organization}/myOldProject/_git/{repository}"
        },
        "serviceEndpointId": SERVICE_ENDPOINT_ID
    }
}
Create Git import request

As it is an async operation, it is very useful to check the progress of the import process. For sure, you will want to move on when the “status” property is equal to “completed”:

GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/importRequests/{importRequestId}?api-version=5.0-preview.1
Get Git import request status

Export/import a build definition

List Build Definitions: GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=5.0-preview.7
Get specific Build def: GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=5.0-preview.7
List and Get Build Definition

This is actually everything you need to get and export a build definition, simple right?

By the way, what you just exported, it contains a lot of “project/organization/user”-specific information that you have to delete/modify for a template creation.

The fields you can safely delete are:

_links
authoredBy
queue._links
queue.url
queue.id
url
uri
revision
createdDate
id
and then change
name
“project” property (returned by project creation or GET https://dev.azure.com/{organization}/_apis/projects/{projectId}?api-version=5.0-preview.3)
“git repository”-specific information
(returned by GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories?api-version=5.0-preview.1):
   repository.id with value[0].id
   repository.name with value[0].name
   repository.url with value[0].remoteUrl
task specific information (such as Azure Subscription or Azure Container Registry service endpoint references)
process.phases[0].steps[2].inputs.azureSubscriptionEndpoint
process.phases[0].steps[2].inputs.azureContainerRegistry
With these changes you will have a JSON of a “importable” build definition. Of course, you could do more modifications to it, depending on what you want to achieve.
When you are done with parametrizing your JSON definition, you can submit it to your new project with a simple POST request
POST https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=5.0-preview.7
Create a build definition

Please, take note of “id” and “queue.id” properties of the response. They could be needed for the release definition if, in your scenario, the release uses the outputs of a build.

Export/import a release definition

As for build definition, you can retrieve the list of myOldProject release definitions and then export the one desired:

GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.0-preview.3
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions/{definitionId}?api-version=5.0-preview.3
List and Get release definition

You will have to remove old specific information like:

createdby
createdon
modifiedby
modifiedon
revision
environments[0].currentRelease
environments[0].badgeUrl
artifacts[0].definitionReference.artifactSourceDefinitionUrl
url
_links
id

and properly change:

artifacts[0].sourceId has to be projectId:buildDefinitionId
artifacts[0].definitionReference.project.id
artifacts[0].definitionReference.repository.id
artifacts[0].definitionReference.definition.id
environments[0].deployPhases[0].deploymentInput.queueId
environments[0].owner.id (could be the “createdby.Id” property from build definition create POST)
task specific information
environments[0].deployPhases[0].workflowTasks[2].inputs.azureSubscriptionEndpoint
environments[0].deployPhases[0].workflowTasks[2].inputs.azureResourceGroup
environments[0].deployPhases[0].workflowTasks[2].inputs.kubernetesCluster
We are now ready to submit your release definition to myNewProject with another POST request:
POST https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.0-preview.3
Create release definition

Conclusions

You are finally done, Azure DevOps Services REST APIs for programmatically cloning projects are no more a mystery for you!

We achieved having a “clone” of myOldProject to myNewProject, with original git repository code and history, original build and release definitions.


F.A.Q.:

  • I am using one or more custom extensions, will I have problems?
    Custom extensions are installed at the organization level.
    For this reason, if you are exporting to a new project in the same organization you will have no issues.
    Otherwise, if you are exporting to a new project in a different organization you have to check if the extension is already there.I suggest you to:

    • GET all installed extensions on the source organization
      GET https://extmgmt.dev.azure.com/{organization}/_apis/extensionmanagement/installedextensions?api-version=5.0-preview.1
    • Take note of the publisherId and extensionId from the custom extensions, by excluding the ones with the “builtin” flag
    • POST to install custom extensions you need on the target organization
      POST https://extmgmt.dev.azure.com/{organization}/_apis/extensionmanagement/installedextensionsbyname/{publisherId}/{extensionId}/{version}?api-version=5.0-preview.1
  • What a “service endpoint” is?
    From ADO docs: “For Azure DevOps Services and Team Foundation Server to be able to connect to the external service, in addition to using the credentials, there is also need to know how to set the credentials in the HTTP request header when calling the external endpoint. Azure DevOps Services supports a closed set of authentication schemes that can be utilized by a custom service endpoint type”
  • I am using a private Nuget feed, a private npm repo, a Service Fabric Cluster, a Service Bus or some other different endpoint. How do I create a proper service endpoint to authenticate and authorize the requests?
    Doing the following GET request you can retrieve the list of the available service endpoints, with matching URLs and authentication schemas:
    GET https://dev.azure.com/{organization}/_apis/serviceendpoint/types?api-version=5.0-preview.1

     

Spread this article

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.