Category Archives: Cloud Computing

Querying Azure VM state with PowerShell

I was recently given the task of identifying the state of an Azure VM so that an automation script using the az vm run-command invoke would not fail if the VM was down or under a reboot.

I initially thought the task would be really easy and a simple query of the VM state using Get-AzVM would provide us with a running state property of the VM, but as it happens the state is a little abstracted.

Using Get-AzVM cmdlet

We will query our Virtual Machine running state using the Get-AzVM cmdlet using the -Status switch.

Get-AzVM -resourcegroupname "Demo" -name "server1" -Status

Running this command will return an object of type PSVirtualMachineInstanceView that we can use to test the status of the VM. The output to the above command is as follows:

ResourceGroupName          : Demo
Name                       : server1
HyperVGeneration           : V1
BootDiagnostics            : 
  ConsoleScreenshotBlobUri : https://ortodiag.blob.core.windows.net/bootdiagnostics-server1-cb292d11-c4c2-4abd-8f28-c0de31e68
010/server1.cb292d11-c4c2-4abd-8f28-c0de31e68010.screenshot.bmp
  SerialConsoleLogBlobUri  : https://ortodiag.blob.core.windows.net/bootdiagnostics-server1-cb292d11-c4c2-4abd-8f28-c0de31e68
010/server1.cb292d11-c4c2-4abd-8f28-c0de31e68010.serialconsole.log
Disks[0]                   : 
  Name                     : server1_OsDisk_1_6311f5e2cf24419eb27ac5c3734e952b
  Statuses[0]              : 
    Code                   : ProvisioningState/succeeded
    Level                  : Info
    DisplayStatus          : Provisioning succeeded
    Time                   : 12/05/2020 12:37:11
Disks[1]                   : 
  Name                     : server1_log_drive
  Statuses[0]              : 
    Code                   : ProvisioningState/succeeded
    Level                  : Info
    DisplayStatus          : Provisioning succeeded
    Time                   : 12/05/2020 12:37:11
Statuses[0]                : 
  Code                     : ProvisioningState/succeeded
  Level                    : Info
  DisplayStatus            : Provisioning succeeded
  Time                     : 12/05/2020 12:37:11
Statuses[1]                : 
  Code                     : PowerState/deallocated
  Level                    : Info
  DisplayStatus            : VM deallocated

This is quite good in that we have lots of information about our VM and associated resource states. We can see that it is a Generation 1 VM and see the state of our disks and our VM. More specifically we see the Statuses[1] element and it’s Code property to get our VM status.

The possible Code states are:
– PowerState/deallocated (VM is stopped and deallocated)
– PowerState/starting (VM is starting)
– PowerState/running (VM is running – the Azure Portal may show a VM as running before this state returns)
– PowerState/deallocating (VM is stopping)

We now have enough information to write our PowerShell code.

Querying Azure VM state

As touched upon earlier, we need to access the PSVirtualMachineInstanceView object and access the Statuses[1] element and it’s Code property. This gives us a $provisioningState value that we can test against our static state (in our case PowerState/running).

If the state is not running then we will keep looping with a 5 second wait. The previous state is tracked only for output sugar so that we will only write to the screen on a state change:

$vmName = "server1"
$resourceGroup = "Demo"
$lastProvisioningState = ""
$provisioningState = (Get-AzVM -resourcegroupname $resourceGroup -name $vmName -Status).Statuses[1].Code
$condition = ($provisioningState -eq "PowerState/running")
while (!$condition){
    if ($lastProvisioningState -ne $provisioningState){
        write-host $vmName "under" $resourceGroup "is" $provisioningState "(waiting for state change)"
    }
    $lastProvisioningState = $provisioningState

    sleep -Seconds 5
    $provisioningState = (Get-AzVM -resourcegroupname $resourceGroup -name $vmName -Status).Statuses[1].Code

    $condition = ($provisioningState -eq "PowerState/running")
}
write-host $vmName "under" $resourceGroup "is" $provisioningState -fore green

Results

The script is executed and we stop and (ultimately) start the server1 VM. The output from the script was as follows:
VM State
As can be seen, this simple backoff script correctly reported on shutdown and startup of our VM.

Conclusion

Whilst I do not pretend for one moment that this script is particularly clever (or even optimally written), it does at least demonstrate how easy it is to query our Azure VM state (with appropriate backoffs) through PowerShell so that we can accurately and predictably perform follow up actions on them.

Obviously this can be adapted for your specific use-case.

Removing and maintaining Azure resource group deployments based upon deployment count

Whenever you create or update an Azure resource, a new deployment is created under the resources’ configured resource group. This deployment history is retained ad-infinitum until you eventually hit the hard limit of 800 deployments (per resource group). You may think this figure is more than enough to accommodate all the possible resource changes that could ever be made in a resource group, but if you are running CICD pipelines to push out your Infrastructure as Code (IaC) (or create lots of resources per resource group) then it is very likely you will exhaust this figure very quickly.

Every time a release pipeline runs, regardless of whether you are changing resources or not, all configured and enabled deployments in the pipeline will result in a new deployment record. You can view all historic deployments in the Azure Portal for each resource group by selecting its Deployments item under the Settings pane (see below).


In the example above you will note that we only have 4 deployments that have been created in this resource group. When the hard limit is eventually hit, all subsequent deployments to that specific resource group will fail.

Microsoft’s solution

Microsoft provide a solution to this in the MS doc titled Resolve error when deployment count exceeds 800 which allows you to programmatically remove deployments (through Azure CLI or PowerShell Az) based upon a deployment date and this is made possible because of the Timestamp property. I have also seen many blog posts that simply seem to regurgitate this Microsoft code giving really just one solution – to maintain deployments based on date.

This is all well and good if your deployments span many weeks or months and that the counts are predictable, based on date-time, but what happens if you have highly active, highly unpredictable, or high number of resources per resource group?

Deployment count solution

Perhaps a far better solution would be to maintain a set deployment count that will allow each release to succeed each time. If you are only deploying a single resource, then clearly you would only have to ensure a spare deployment slot is available. If you are deploying resources through a CICD pipeline then you simply need to ensure that you have at least that number of resources in your pipeline available.

Running from Azure CLI or PowerShell Az command-line

If you are manually running the maintenance code either from a remote command-line session or directly within the Portal command-line itself, you will have to set your subscription context that you wish to maintain. We can do this easily in PowerShell by running the following code (ensuring that you change the subscription text to the one you wish to target):

$subscriptionName =  "ACMEPRODSUB"
$subscription = Get-AzSubscription -SubscriptionName $subscriptionName
Set-AzContext -Subscription $subscription | Out-Null

Once you have set your subscription you can then use the subsequent code (detailed in the Running from within an Azure DevOps Pipeline section).

Running from within an Azure DevOps Pipeline

I have generally found that running a maintenance step at the start of any infrastructure Release Pipeline is a good point of execution. It will reduce the time to cycle through and delete any excess deployments to a minimum, and will also ensure there are enough deployment slots to prevent release failure. For our pipelines, maintaining a deployment count of 700 is a good compromise -it leaves 100 spare slots for each run and plenty of past deployment history.

In the release pipeline, we can create an Azure PowerShell task within a release stage.

For convenience we can use our code as an inline script -though you may ultimately decide to parameterize the $retainCount variable and publish the script from a repo instead.

Use your common sense when setting the number of deployments you wish to retain.

$retainCount = 700

Since our Azure PowerShell task has explicit settings for the subscription that we wish to execute the script against, we do not need to worry about changing subscription context. All we are concerned about is the functionality of the code itself.

In the code below we are first looping through all resource groups in the current subscription context. For each resource group we return all its deployments, and for any deployment that is above the iterator threshold set by $retainCount (assuming there are any) -it will be deleted.

foreach ($resourceGroup in Get-AzResourceGroup){
    $resourceGroupName = $resourceGroup.ResourceGroupName
    $deployments = Get-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName

    write-host "Resource group" $resourceGroupName "has" $deployments.Count "deployments..."

    $iterationCount = 1
    foreach ($deployment in $deployments) { #deployments are returned sorted by age desc
        if ($iterationcount -gt $retainCount){

            Remove-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -Name $deployment.DeploymentName | Out-Null
            write-host "   Deleted deployment on" $deployment.Timestamp -fore magenta
        }

        $iterationCount = $iterationCount + 1
    }
    Write-Host "   Resource group deployment maintenance complete." -fore green
}

This results in the following output:

If you are using an Azure DevOps release task to execute this code you will not see coloured text in the task output.

Conclusion

If you are manually maintaining your resource group deployments or wish to automate it through Azure DevOps, the timestamped solution provided by Microsoft may not fit your requirements given the frequency of your deployments or other considerations. Given that the deployments are returned in a time sorted descending order, we can easily delete deployments based upon the deployment count -always leaving enough space for future deployments and not removing those based upon date alone. Ensuring that this maintenance task is run prior to any automated infrastructure releases can improve the success rate of your release pipelines in highly active environments.

How to restore a deleted Azure DevOps repository

If you are using Azure DevOps, you might be comforted that your Git repo is “in the Cloud” and automatically has availability and disaster guarantees. However you (or someone else) still have the ability to accidentally (or maliciously) delete repos from Azure DevOps Repos. Surprisingly, at the time of writing, there is no GUI based option to restore your repo. This might initially instill a sense of panic as you frantically search for the latest local clone to replace your remote – but there is a better way.

When you delete an Azure DevOps repository, it is initially soft-deleted to the “recycle bin”. After a period of time (oddly I have failed to find an offical Microsoft reference stating exactly what this but I believe it is 28 days) it is automatically purged and hard-deleted. Although there is no GUI support to restore your soft-deleted repositories, that ability is exposed through the Azure DevOps REST API, but frustratingly the Microsoft Azure DevOps Services REST API Reference does not provide a worked example in the Repositories – Restore Repository From Recycle Bin API call page.

To make your life easier, I will provide the solution below!


Getting started with Azure DevOps REST API and PAT token

Within my blog so far I have provided several worked examples of making a REST API call to Azure DevOps. If you are new to this, I suggest you first check out my post titled Querying Azure DevOps REST API with PowerShell.

Once you have assigned your $header variable from an encoded PAT token (as documented in the aforementioned article) you are ready to roll!

Set your repository’s Organisation and project

Each project will contain its own set of Azure repositories. Ensure you provide the correct values for your organization and project- and ensure that for any names with spaces are correctly replaced using %20 (so that a valid url can be formed).

$organization = "retracement"
$project = "ACME%20Corp"

REST API call to list repositories in the recycleBin

From the Microsoft Azure DevOps Services REST API Reference we can call the Repositories – List REST API call to return a list of all deleted repositories in our recycleBin for our organization’s project.

We will first build up our $url using the variables set earlier.

$url = "https://dev.azure.com/$organization/$project/_apis/git/recycleBin/repositories?api-version=5.1-preview.1"

Now that all variables are set we can make our REST API call and iterate over all deleted repositories

$deletedRepos = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
  # for each repository
Write-Host "Deleted repositories"
Write-Host "--------------------"

$deletedRepos.value | ForEach-Object {
    $repoId = $_.id
    $repoName = $_.name
    $deletedBy = $_.deletedBy.displayName
    $deletedDate = $_.deletedDate
    Write-Host "repoId:" $repoId $repoName "deleted on" $deletedDate "deleted by" $deletedBy
}

The following output is returned:

Deleted repositories
--------------------
repoId: 3b1bbfe0-470d-4724-bc69-6ec29ff88cb5 SuperImportantRepo deleted on 2020-04-08T13:44:21.807Z deleted by Mark Broadbent
repoId: 4c3abef0-520a-2461-ac70-1ad30ef11ab7 NotImportantRepo deleted on 2020-04-12T10:00:01.201Z deleted by Mark Broadbent

We have now identified that someone (me!) has deleted a super important repository by accident. Using the repoId we can use this to restore it from the recycleBin.


Recover soft deleted repository

First we need to set a variable $repoId to the deleted repository (SuperImportantRepo) repoId that we identified earlier. This will be used in our next REST API call.

$repoId = "3b1bbfe0-470d-4724-bc69-6ec29ff88cb5"

Now we can return back to the Repositories – Restore Repository From Recycle Bin REST API call page as use this to build out our new url.
As you will see, the url contains our $repoId and we will also create a $body variable set to a JSON key value pair setting the deleted property to false. This JSON body is passed into our REST API call using the Patch Method.

$url = "https://dev.azure.com/$organization/$project/_apis/git/recycleBin/repositories/" + $repoId +"?api-version=5.1-preview.1"
$body = ConvertTo-Json @{“deleted”= "false"}
Invoke-RestMethod -Uri $url -Method Patch -Body ($body) -ContentType "application/json" -Headers $header

The output of our final REST API call results in:

id            : 3b1bbfe0-470d-4724-bc69-6ec29ff88cb5
name          : SuperImportantRepo
url           : https://dev.azure.com/retracement/e6fa212f-3520-4c30-8c28-d6bd88926ff2/_apis/git/repositories/3b1bbfe0-470d-4724-bc69-6ec29ff88cb5
project       : @{id=e6fa212f-3520-4c30-8c28-d6bd88926ff2; name=ACME%20Corp; description=Super Important Repository for mission critical systems; url=https://dev.azure.com/retracement/_apis/projects/e6fa212f-3520-4c30-8c28-d6bd88926ff2; state=wellFormed; revision=626; 
                visibility=private; lastUpdateTime=2019-11-20T15:49:09.773Z}
defaultBranch : refs/heads/master
size          : 731
remoteUrl     : https://retracement@dev.azure.com/retracement/ACME%20Corp/_git/SuperImportantRepo
sshUrl        : git@ssh.dev.azure.com:v3/retracement/ACME%20Corp/SuperImportantRepo
webUrl        : https://dev.azure.com/retracement/ACME%20Corp/_git/SuperImportantRepo

As we can see from the above output success!


Summary

As I have shown, deleting a repository by accident in Azure DevOps does not have to be a disaster recovery situation since the recycleBin and Azure DevOps REST API makes it relatively simply to view and restore (when you know how!). However it is worth pointing out that for Git repositories, no similar situation exists if you delete a branch (unlike with Tfs Repos in Azure DevOps). So the moral of the story is to ensure you periodically back up all your remote repositories AND set branch policies to protect them against accidental deletion.

Hope you enjoyed the post!

Cannot delete old build definitions in Azure DevOps

I have been experiencing a problem for quite a while now in my current environment, in that some of our old builds cannot be deleted. When you attempt to do so it results in the following error:

One or more builds associated with the requested pipeline(s) are retained by a release. The pipeline(s) and builds will not be deleted.

Many of our pipelines have undergone a lot of change over time to the degree it is not even obvious anymore why (or indeed where) these builds are being prevented from being dropped. The only thing that is clear is that until they can be, the old build definitions will remain.

I have tried to set the Stop retaining the build setting for all builds associated with a build definition to no avail. The setting just does not seem to want to take in most cases.

I have also tried playing around with build retention policies and even tried tidying up the release pipelines (and releases) themselves. Unfortunately for me, those darn build pipelines do not want to delete.

Today I decided to put some of my recent Powershell and Azure DevOps REST API experiences (see previous posts in this blog) to the test and attempt to get to the bottom of the problem. As it turns out there is a build property called retainedByRelease that is exposed through the REST API which is the reason why a build cannot be removed -resulting in our irritating error.

Using the same technique that I wrote about in Querying Azure DevOps REST API with PowerShell I first decided to try an report on this property. Please refer back to the post above for more explanation on utilizing the REST API, but I realized I would need to make two REST API calls. The first would be to query one or more build definitions and the second would be to query all builds for each build definition. More specifically, with this last call I would report on the retainedByRelease property.


Querying the build definition builds

In the first piece of code we create our authorization token.

$personalToken = "tiksj25oumfavuzr4316vhpxw2mywzbapxj7sw3x2xet3dml1ygy"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalToken)"))
$header = @{authorization = "Basic $token"}

Next we set our organization and project variables.

$organization = "retracement"
$project = "ACME%20Corp"

Our first REST API call queries all build definitions within the project.

#all build definitions
$url = "https://dev.azure.com/$organization/$project/_apis/build/definitions?api-version=6.0-preview.7"
$builddefinitions = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$builddefinitions.value | Sort-Object id|ForEach-Object {
Write-Host $_.id $_.name $_.queueStatus

#all builds for a definition
$url = "https://dev.azure.com/$organization/$project/_apis/build/builds?definitions=" + $_.id + "&api-version=6.0-preview.5"
$builds = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header

$builds.value | Sort-Object id|ForEach-Object {
#report on retain status
Write-Host " BuildId" $_.id "- retainedByRelease:" $_.retainedByRelease
}
Write-Host
}

For brevity I provide only a subset of the results:

339 SQL Dacpac Build enabled
BuildId 43045 - retainedByRelease: False
BuildId 43051 - retainedByRelease: False
BuildId 43053 - retainedByRelease: True
BuildId 43307 - retainedByRelease: True
BuildId 43325 - retainedByRelease: True

366 Databricks Notebooks Build enabled
BuildId 45338 - retainedByRelease: False
BuildId 45340 - retainedByRelease: False
BuildId 45346 - retainedByRelease: True
BuildId 46032 - retainedByRelease: True

375 ARM Templates Build enabled
BuildId 46452 - retainedByRelease: False
BuildId 46454 - retainedByRelease: True

As you can see, from the three active build definitions listed, each one has at least one build that is marked for retention by release.


Setting the build retainedByRelease property

Now we have a procedure in place to query the retainedByRelease property, it is just as easy to set it. If you are trying to remove a specific Build Definition (or builds), you can implement a filter in the builddefinitions iterator. So:

$builddefinitions.value | Sort-Object id|ForEach-Object {

Would now become:

$builddefinitions.value | where {$_.name -eq "ARM Templates Build"}|Sort-Object id|ForEach-Object {

In the above example we are filtering on a single build definition, but feel free to use the filter of your choosing.

The final thing we need to do is make a REST API call to update each build returned by this filtered build definition. We can so this as follows by adding the following line inside our build iterator:

Invoke-RestMethod -Uri $url -Method Patch -Body (ConvertTo-Json @{"retainedByRelease"='false'}) -ContentType "application/json" -Headers $header

You will note the use of -Method Patch within this call rather than -Method Get and the JSON body. The patch method allows us to partially update resources (in this case one field) with the JSON body provided.


Putting it all together

So if we wanted to update the builds of one specific Build Definition called ARM Templates Build we would run the following code:

$personalToken = "tiksj25oumfavuzr4316vhpxw2mywzbapxj7sw3x2xet3dml1ygy"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalToken)"))
$header = @{authorization = "Basic $token"}

$organization = "retracement"
$project = "ACME%20Corp"

#all build definitions
$url = "https://dev.azure.com/$organization/$project/_apis/build/definitions?api-version=6.0-preview.7"
$builddefinitions = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header

$builddefinitions.value | where {$_.name -eq "ARM Templates Build"}|Sort-Object id|ForEach-Object {
Write-Host $_.id $_.name $_.queueStatus

#all builds for a definition
$url = "https://dev.azure.com/$organization/$project/_apis/build/builds?definitions=" + $_.id + "&api-version=6.0-preview.5"
$builds = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header

$builds.value | Sort-Object id|ForEach-Object {
#report on retain status
Write-Host " BuildId" $_.id "- retainedByRelease:" $_.retainedByRelease

#api call for a build
$url = "https://dev.azure.com/$organization/$project/_apis/build/builds/" + $_.id + "?api-version=6.0-preview.5"

#set retainedByRelease property to false
Invoke-RestMethod -Uri $url -Method Patch -Body (ConvertTo-Json @{"retainedByRelease"='false'}) -ContentType "application/json" -Headers $header
}
Write-Host
}

Now that all your builds for the ARM Templates Build Build Definition are deleted, you should be able to remove this build definition without further error (you do not need to first remove its builds).


Summary

There are certain issues that you might experience in Azure DevOps which seem almost impossible to resolve through the GUI, but yet again the Azure DevOps API can come to our rescue. In this specific example we have easily queried aspects of DevOps through PowerShell, and this time even changed information through it to resolve our problem.

I hope you find this post useful for this rather frustrating problem!

Querying Azure DevOps REST API with PowerShell

In previous posts we have talked about trying to use and consume Azure DevOps using PowerShell and utilizing the Azure CLI. In particular, my post titled Use PowerShell to consume your Azure CLI DevOps result set painted a rather frustrating picture when trying to manipulate the tabular dataset from the Azure CLI. Furthermore, our functionality is restricted to only those commands implemented by the Azure CLI Azure DevOps add-in -as will become increasingly obvious, this is limited to say the least.

There is a better way to query Azure Devops – Azure DevOps REST API to the rescue.


Probably the first thing you will want to do is understand what kind of queries and actions you can make against the Azure DevOps REST API. These are not limited to reporting upon existing configurations, it can also be used to change configuration. For example, through the REST API we could POST a call to create a brand new release. For the purposes of simplicity we will simply query Azure DevOps in this article.

In order to understand all the potential Azure DevOps queries and actions you can make through the Azure DevOps REST API you can refer to the Microsoft Azure DevOps Services REST API Reference. We will return back to this reference when we look to make specific calls but before we get there, we will first break down the steps that you will need to take in order to successfully make your call.

Create your PAT token

In order to securely communicate with Azure DevOps, you will first need to create a PAT token which will allow your code to make an authorized call to the REST API. This can be created by clicking the configuration icon from the toolbar of Azure DevOps.

image.png

In my specific example I am going to create a PAT token with Full access, but it is recommended that you should create a Custom defined scope to limit the security surface area. Also note that you must set an expiration date to this token and once it expires, you will need either regenerate it, or create a new one to meet your personal requirements.

image.png

We can now use this PAT in your REST API call, but it is important to ensure this string uses Base64 encoding.

Assign and encode your PAT token

$personalToken = "tiksj25oumfavuzr4316vhpxw2mywzbapxj7sw3x2xet3dml1ygy"

#Write-Host "Initialize authentication context" -ForegroundColor Yellow
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalToken)"))
$header = @{authorization = "Basic $token"}

Make your REST API call

From the Microsoft Azure DevOps Services REST API Reference select the REST API call URI that you need to use.

In this first example, the URI chosen is used to query all existing Azure DevOps Projects. The following code invokes the Azure DevOps REST API call and iterates through each project.

For example, to make a call to query all projects in your Azure DevOps organisation you can call the following:

$url = "https://retracement.visualstudio.com/_apis/projects?api-version=5.0"

$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header

$output.value | ForEach-Object {
    Write-Host $_.name
}

In my case I get the following Projects returned:

Parts Unlimited
ACME Corp
Main Project
My Test Project

Other examples

In the following examples I will perform some common queries against our Azure DevOps project. I will do my best to expand and implement new REST API calls over time in follow up posts.

Query all build definitions

In this example we will return the results in descending order. There is also a bit of further work needed to parse the definition output to improve the quality of the result set. I’ve left a few extra fields commented out for brevity.

# Builds API call
$url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/build/builds?api-version=5.0"
$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$output.value | Sort-Object id -Descending|ForEach-Object {
    Write-Host $_.buildNumber - $_.status - $_.reason# - $_.definition - $_.url
}

This returns the following builds:

26740 - completed - schedule
20021 - completed - schedule
17436 - completed - schedule
14701 - completed - manual

Query all release pipeline definitions

In this example we will pull back a sorted list of all release pipeline definitions.

# Release Definitions API call
$url = "https://retracement.vsrm.visualstudio.com/ACME%20Corp/_apis/release/definitions?api-version=5.0"
$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$output.value | Sort-Object name|ForEach-Object {
    Write-Host $_.name
}

And we get all current release definitions:

Big Daddy Release Pipeline
Little Tom Release Pipeline
Widgets Release Pipeline

Query all repositories in a project

# Repositories API call
$url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories?api-version=6.0-preview.1"
$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$output.value | ForEach-Object {
    Write-Host $_.id,$_.name
}

The following repositories are returned:

ba008565-118a-41e6-878c-d7a8180bf734 Widget Database
49ebc167-8b48-4202-af4e-f8fd885aede1 Widget Notebooks
682c7ebf-11d1-443f-b0b0-fbd7f2bfdd71 ACME dotNet master
112635ba-c5e7-4c91-bae7-ff014cf36be4 ACME Helper

Query a repository branches

In this next example we will take a repository id and use this in our REST API call to query it’s branches.

# Repository API call
$repoId = "ba008565-118a-41e6-878c-d7a8180bf734"
$url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories/$repoId/refs?api-version=6.0-preview.1"
$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$output.value | ForEach-Object {
    Write-Host $_.name
}

The branches returned for this repo are as follows:

ba008565-118a-41e6-878c-d7a8180bf734 refs/heads/mybrillfeature
ba008565-118a-41e6-878c-d7a8180bf734 refs/heads/development
ba008565-118a-41e6-878c-d7a8180bf734 refs/heads/master

Query all branches for all repositories in a project

We can put the previous two API calls together to query all branches for all repositories.

#branches for each repo
$url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories?api-version=6.0-preview.1"
$repo = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$repo.value | ForEach-Object {
    $repoId = $_.id
    $repoName = $_.name
    $url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories/$repoId/refs?api-version=6.0-preview.1"
    $output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
    $output.value | ForEach-Object {
        Write-Host $repoId - $repoName - $_.name
    }
}

This returns:

ba008565-118a-41e6-878c-d7a8180bf734 - Widget Database - refs/heads/mybrillfeature
ba008565-118a-41e6-878c-d7a8180bf734 - Widget Database - refs/heads/development
ba008565-118a-41e6-878c-d7a8180bf734 - Widget Database - refs/heads/master
49ebc167-8b48-4202-af4e-f8fd885aede1 - Widget Notebooks - refs/heads/development
49ebc167-8b48-4202-af4e-f8fd885aede1 - Widget Notebooks - refs/heads/master
682c7ebf-11d1-443f-b0b0-fbd7f2bfdd71 - ACME dotNet master - refs/heads/development
682c7ebf-11d1-443f-b0b0-fbd7f2bfdd71 - ACME dotNet master - refs/heads/master
112635ba-c5e7-4c91-bae7-ff014cf36be4 - ACME Helper - refs/heads/development
112635ba-c5e7-4c91-bae7-ff014cf36be4 - ACME Helper - refs/heads/master

Tabular query of all branches for all repositories in a project

And finally, our Azure DevOps REST API result set is far more useful as a tabular object so that we can manipulate it further in PowerShell (should we so wish) and perform various filters and sorts against it. So extending the previous example we will put our result set into a table object.

#branches for each repo in a table
$table = New-Object System.Data.DataTable #create table and columns
$table.Columns.Add("Id")
$table.Columns.Add("Repository")
$table.Columns.Add("Branch")

$url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories?api-version=6.0-preview.1"
$repo = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$repo.value | ForEach-Object {
    $repoId = $_.id
    $repoName = $_.name
    $url = "https://retracement.visualstudio.com/ACME%20Corp/_apis/git/repositories/$repoId/refs?api-version=6.0-preview.1"
    $output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
    $output.value | ForEach-Object {
        $table.Rows.Add($repoId, $repoName, $_.name)|Out-Null
    }
}
$table | select Repository, Branch | Sort-Object Repository, Branch| ft

Querying the table object and filtering on two columns returns:

Repository          Branch                                         
----------          ------
ACME dotNet master  refs/heads/development
ACME dotNet master  refs/heads/master
ACME Helper         refs/heads/development
ACME Helper         refs/heads/master
Widget Database     refs/heads/mybrillfeature
Widget Database     refs/heads/development
Widget Database     refs/heads/master
Widget Notebooks    refs/heads/development
Widget Notebooks    refs/heads/master

Conclusion

As you have seen, the Azure DevOps API is not only very easy to use and consume through PowerShell (when you know how), but provides a much more comprehensive route to interface with Azure DevOps than the other techniques I have previously talked about (such as the Azure CLI).

In future posts I will talk about implementing other queries and actions such listing all outstanding pull requests across repositories and even how to create a release.

Hope you find this post useful, please leave your comments below!

Use PowerShell to consume your Azure CLI DevOps result set

Before we get going it is probably first worth me pointing out (in case you are wondering) that the whole premise of this post stems from a lack of native Azure DevOps PowerShell Module. Yes there are a few solutions out there, but at the time of writing there is no official Microsoft PowerShell DevOps module, so we are stuck with using the CLI if you want to avoid using these other solutions.

In my post Using Azure CLI to query Azure DevOps I explained how you can use the Azure CLI to query Azure DevOps so you can obtain useful information on builds, releases, and other useful information. The solution required a certain level of skill with JMESPath to manipulate your result sets -which as explained can be a little confusing.

However once you have a bare bones result set, it is likely that you will want to consume these results in a more user-friendly environment such as PowerShell so that you can build upon these data sets. I thought this would be an easy thing to do, but as you will see below it was anything but.

A simple example

Lets write a simple Azure CLI query with JMESPath and assign the results to a PowerShell variable:
$releases = az pipelines release list --query "[].{name:name,pipeline:releaseDefinition.name}" --output table

If we take a quick look at the contents of the $releases variable we get the following result set:

PS C:\> $releases

Name          Pipeline
------------  ----------------------------------------------
Release-12    Databricks Pipeline
Release-11    Databricks Pipeline
Release-10    Databricks Pipeline

The problem is that this result set is coming across as an array of values rather than a tabular data-set and we can see this if we try to select a column:

PS C:\> $releases|Select-Object Name

Name
----

ConvertFrom-String

After lots of trial and error and frustrating attempts to arrive at a solution which I would have expected to be easy to get around, I was given a great tip from Jonathan Allen (t) to try and use the ConvertFrom-String cmdlet. Using this on its own gives us the following:

PS C:\> $releases | ConvertFrom-String -PropertyNames Name, Pipeline

Name          Pipeline
------------  --------
Name          Pipeline
------------  ----------------------------------------------
Release-12    Databricks
Release-11    Databricks
Release-10    Databricks
Release-12    Pipeline
Release-11    Pipeline
Release-10    Pipeline

Clearly this looks like we are getting closer so I tried removing the header and adding an explicit delimiter into this cmdlet:

PS C:\> $releases = $releases | where-object {($_.tostring() -ne "------------  ----------------------------------------------" -And $_.tostring() -ne "Name          Pipeline")}
PS C:\> $releases | ConvertFrom-String -PropertyNames Name, Pipeline -Delimiter "  "

Name          Pipeline
----          --------
Release-12    Databricks Pipeline
Release-1      Databricks Pipeline

Unfortunately due to the fact we do not have a consistent delimiter due to the change in the first column width when our result number changes in digit size we get inconsistent formatting results as exampled above.

Back to basics

I have always been told that you should try and use PowerShell pipeline functionality to keep code brief and optimal, but try as I might I could not arrive at an acceptable solution. In the end I decided to try and return to programming basics and write code to format the result sets as I would in C# or other procedural languages.

So in a nutshell, I decided to do the following:

  • Create a table, with correct number of columns
  • Create DevOps result set array and strip the header
  • Iterate through the array and add a new row to our table containing data for each column
  • Manipulate our table in PowerShell as we want

The full solution was as follows:

$releases = az pipelines release list --query "[].{name:name,pipeline:releaseDefinition.name}" --output table
$table = New-Object System.Data.DataTable
$table.Columns.Add("Name")
$table.Columns.Add("Pipeline")

# strip headers and column string
$releases = $releases | where-object {($_.tostring() -ne "------------  ----------------------------------------------" -And $_.tostring() -ne "Name          Pipeline")}
$break = 14 # where 14 is the breakpoint between columns
foreach ($row in $releases)
{
    #$row
    $column1 = $row.ToString().Substring(0,$break)
    $column2 = $row.ToString().Substring($break,$row.ToString().Length - $break)
    $table.Rows.Add($column1, $column2) | Out-Null # Out-Null required to avoid echo to screen
}
$table | select Name | where {($_.Pipeline -eq "Databricks Pipeline")} | ft

And the last line of this code simply selects the Name of the last three releases where the pipeline was called Databricks Pipeline (proving that our result set is tabular).

Name          
------------
Release-12
Release-11
Release-10

Conclusion

As you can see, in those situations where our result sets are not tabular at the point of assignment to a PowerShell variable, it is still possible to convert them as you see fit. Whilst this is a little painful to initially set up (depending upon the number of columns you might have), it is a fairly repeatable pattern. It is disappointing that I could not find a pure pipeline driven solution (and I did try many different techniques including formatting strings), but if you are aware of a better solution, please let me know! If I come across a better solution myself I will update this post accordingly!

Rewrite your PowerShell AzureRm to Az for Azure DevOps deployments?

I just wanted to get a quick post out on specific problem that I have just encountered from one of our Azure DevOps release pipelines in which one of the release stage tasks started failing for no reason (despite no obvious changes having been made). Hopefully this post will help others if you are also experiencing this problem and don’t know what to do.

The error message was as follows for our task “Enable Firewall on Data Lake”

2020-02-13T12:30:51.4672549Z ##[section]Starting: Enable Firewall on Data Lake
2020-02-13T12:30:51.4779190Z ==============================================================================
2020-02-13T12:30:51.4779279Z Task : Azure PowerShell
2020-02-13T12:30:51.4779364Z Description : Run a PowerShell script within an Azure environment
2020-02-13T12:30:51.4779425Z Version : 3.153.2
2020-02-13T12:30:51.4779503Z Author : Microsoft Corporation
2020-02-13T12:30:51.4779571Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-powershell
2020-02-13T12:30:51.4779656Z ==============================================================================
2020-02-13T12:30:53.6479788Z ##[command]Import-Module -Name C:\Modules\azurerm_6.7.0\AzureRM\6.7.0\AzureRM.psd1 -Global
2020-02-13T12:31:00.6580029Z ##[command]Clear-AzureRmContext -Scope Process
2020-02-13T12:31:01.3041473Z ##[command]Disable-AzureRmContextAutosave -ErrorAction Stop
2020-02-13T12:31:01.9336559Z ##[command]"D:\a\_tasks\AzurePowerShell_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\3.153.2\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in d:\a\_temp\clientcertificate.pem -out d:\a\_temp\clientcertificate.pfx -password file:"d:\a\_temp\clientcertificatepassword.txt"
2020-02-13T12:31:02.1959058Z ##[command]Add-AzureRMAccount -ServicePrincipal -Tenant *** -CertificateThumbprint ****** -ApplicationId *** -Environment AzureCloud @processScope
2020-02-13T12:31:02.6748632Z ##[command] Select-AzureRMSubscription -SubscriptionId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -TenantId ***
2020-02-13T12:31:03.1996315Z ##[command]& 'd:\a\_temp\xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.ps1'
2020-02-13T12:31:05.7869733Z ##[command]Disconnect-AzureRmAccount -Scope Process -ErrorAction Stop
2020-02-13T12:31:06.1648969Z ##[command]Clear-AzureRmContext -Scope Process -ErrorAction Stop
2020-02-13T12:31:06.9334190Z ##[error]Exception of type 'Microsoft.Rest.Azure.CloudException' was thrown.
2020-02-13T12:31:07.0104724Z ##[section]Finishing: Enable Firewall on Data Lake

The task was configured as follows

As you can see we are simply trying to execute the following Azure RM command to configure an existing DataLakeStorage v1 resource:
Set-AzureRmDataLakeStoreAccount -ResourceGroupName myazureresourcegroup -Name myazuredlresource-AllowAzureIpState Enabled -FirewallState Enabled

As several failed attempts to re-run this task with the same error and confirmation that nothing else had changed in our environment I decided to rewrite the now deprecated AzureRm code (I have discussed this elsewhere in this blog) to Azure PowerShell Az module. The command was now as follows:
Set-AzDataLakeStoreAccount -Name myazuredlresource -FirewallState Enabled -ResourceGroupName myazureresourcegroup -AllowAzureIpState Enabled

Also, in order to use Azure PowerShell Az module we also need to change the task version to 4.* and above.

This time upon release we get success!

Conclusion

While it is still not clear to me that the failures we were experiencing are as a direct result of AzureRm being slowly phased out (doubtful at this stage), it is clear this particular problem is perhaps a wake up call for us all to slowly start considering converting our existing code to Az in order to future proof it in our pipelines. I will check with Microsoft for feedback on this issue to understand if there is official advise with regards to our legacy AzureRm code and update this post accordingly.

For now, I hope this resolution fixes your problem too!