API Management DTAP Pipeline with VSTS

In order to handle a full lifecycle, we need to be able to export API’s from our API Management Development instance to the Test instance, if we have a Acceptance Test instance and later on the Production instance.

There are several ways of doing this and here we will go thru the GIT option. Out of Box the API Management instance comes with the function to export the configuration to a dedicated GIT repository (comes with the product, read more about it here) you can the use this saved state to restore your API instance or export API’s to other instances.

Note the GIT exportion will not export users, Named Values (Properties) or subscriptions, and if Named Values (Properties) are used these need’s to be added manually before import to the new istance.

The workflow is simple we develop new API’s and updating old ones in the Development instance first, then the code is saved to the GIT repository and a Build is initalized in VSTS to publish the artifacts for use during deployment. Release/Deployment is then made to next instance by cloning that instance’s repository and merge the changes, replacing URL’s and so on with powershell scripts.

The Build Step

The build is all about creating artifacts for deployment to the next instance.

So this we do by setting up a new build in VSTS, for more information about how to do this in VSTS follow this link.

The build is rather small, we just create a new build with an Empty Process, pick the “Remote Repo” and create a New Service Connection shows up in a popup window and here you shall provide the credentials found at you APIM instance, don’t forgett to generate the password. (annoying is that this expires every 30 days and need’s to be replaced). Read more about how to find credentials and URL’s for GIT repository here

Next step is to publish the artifacts, now we have the possibility to either push everything or add masks to prevent saving unnessisary settings. In this image we show how masking is used to only publish artifacts for a subset of API’s in the API Management instance (isolating the deployment to these API’s).

In order to make these masks you need to understand the GIT structure, read more about how to understand APIM Source Code here in my earlier post

After an initated build the files related to the API are published and ready to be used, as you can see we want to make sure that products, groups and polices are moved with aswell so that we can manage the ground set from Development instance. (since we merge added special products or groups in later instances will be left untouched)

The build part of this is now done and we hav our artifacts published and ready to use with our Releases.

The Release Steps

As stated in the begginign the release steps are also working with GIT, we will now clone the next instance’s GIT repository, merge the changes and publish it all back to the same GIT Repository they came from, thereafter we will initiate a deployment from the GIT repository to the API Management instance.

So how do we get the next instance’s source code and merge it? Using Powershell!

The Release definition is actually only powershell and will be looking like this.

Cloning the target APIM Instance GIT repository

To make this simpler we have created a few powershell scripts that help’s out with these steps, first one is cloning the repository. (tip we are using a shared repository for our powershell script and then links it to the release via Artifacts->Link an artifact source)

param([string] $apimname, [string] $urlencodedkey, [string] $user = "apim", [string] $copyfrompath = ".\APIM\*")

"Cloning repository"

#create the paths
$gitpath = $apimname + ".scm.azure-api.net"
$destpath = ".\" + $gitpath + "\api-management\"

$url = "https://" + $user +":"+ $urlencodedkey + "@" + $apimname + ".scm.azure-api.net/";
$url

git clone $url -q

"Start Copy files"


Copy-Item -Path $copyfrompath -Destination $destpath –Recurse -force

"Files copied"

As you can see from the above script we just need some parameters to get started, and we provide these via the Arguments textbox on the powershell script step, and for easy reuse we use variables. Note that in this script the GIT key need’s to be base64 encoded when provided as variable.

-apimname $(apiname) -urlencodedkey $(apikey)  -copyfrompath ".\INT001 Work Orders APIM\APIM\*"

The copyfrompath url can be found via the button on the script path to browse the path to where you want to start copy files from. (image shows sample)

For easy reuse we provide the varibales on Environment level (easy to clone the environment or the whole release and we are 80% done just changing the variables, search paths and replacement scripts).

Changing URL’s and other values in the files

When the code is extracted we need to do some manipulation, almost always it’s changing the URL and some small adjustments. There are several ways of doing this but we have thought that the easiest way is to just use Regex with the replace function. Here is a commonly used Powershell script for replacing a text via Regex expressions in powershell and a sample on how we use it with parameters.

param([string] $filepath ,[string] $regex, [string] $value)

(Get-Content $filepath) -replace $regex,$value | Out-File $filepath

And the arguments provided need’s to point to the exact file location (on the disk where the release agent can find it) the regex on what it should match (in the sample bellow we change th URL on the backend setting for the SOAP API we are using to the one matching the new instance).

-filepath "$(SYSTEM.ARTIFACTSDIRECTORY)\$(apiname).scm.azure-api.net\api-management\backends\BizTalkServiceInstance_Soa107AV4D\configuration.json" -regex '"url":\W*[0-9a-zA-Z:/\."]*' -value '"url": "https://newurlformysoapservice.com"'

The $(SYSTEM.ARTIFACTSDIRECTORY) is vital for finding the correct path since this parameter will contain the path to the folder where the where we can find the file.

This is then repeated until all URL’s or other settings thas need’s to be changed.

Create a commit and push the changes

When all changes have been made we need to push the changes to the targeted APIM instance GIT repository, this is also done via Powershell and here is a script we are using for this.

param([string] $apimname)
#create the paths
$gitpath = $apimname + ".scm.azure-api.net"

#Part publish Start
"Start Publishing"

#move to git folder
cd $gitpath

#set a git user
git config user.email "deployment@vsts.se"
git config user.name "Visual Studio Team Services"

#Add all new files
git add *

#comit and push
git commit -m "Deployment updates"
git push -q

"Publish done"

#return to original path
cd..

As you can see this script is simpler since we have all GIT settings prepared allready, and it does is commiting localy and then pushing to the . When all this is done it’s easy to clone this and reuse on a new instance/environment or a new Release step for another API.

Deploy the changes from GIT to the APIM instnace

Since we often want to have control over the deployment we normaly do the deployment step manually, entering the Azure Portal and browsing the APIM instance going to the Repositories blade and pressing the Deploy to API Management button but this can also be automated via powershell or the REST API.

Summary

Our experience of the GIT deployment setup is good, altough some minor issues have been found with new functionality as when the backend part was added with the SOAP support. But that is only natural and the product group has been really helpful in finding work arounds on problems and fixing the core problem.

With that said there are room for improvements around the experience, using regex to replace URL’s is not optimal and parameter files would help out improving the experience and make it easier to automate URL changes and so on. The closest thing we have is the Named Values but these can’t be used everywhere and another down sides are that these values need’s to be provided manuall before deployment otherwise it fails with a cryptic error and also the visibility of what URL the API is using, it’s not so fun to start using traces just to make sure the URL is properly set.

This option is working good for us but we are looking forward for the upcoming ARM Template support to see if we can get better and easier ways of handling parameters.

Posted in: •Azure API Management  •Integration  | Tagged: •DTAP  •API Management  •Deployments  •Azure  •GIT