Microsoft Azure Resource Manager

What is it?

In the early days of Azure, Microsoft developed a REST API to interface with the Azure public cloud called the Red Dog Front End (RDFE), most commonly called the Azure Service Management API (ASM). This API is an XML-driven, imperative API that enabled provisioning of Azure cloud resources from the Azure Management Portal.

Due to some concerns around performance, concurrency, and user experience, Microsoft developed a new API called the Azure Resource Manager (ARM) API. Some of the key benefits of this new API are as follows:

There are other new features as well, but these are the most critical ones for most businesses who are leveraging the Microsoft Azure cloud.

Why do I need to understand ARM?

When you’re provisioning and managing cloud resources, within the Microsoft Azure platform, all of your resources will be provisioned using Azure Resource Manager (ARM). Regardless of the interface you’re using to manage resources, it’s important to understand the ARM architecture, so that you can be more effective in your work. Someone who doesn’t understand ARM might find the various management interfaces confusing, and challenging to use, because they don’t understand the platform’s terminology.

You might be inclined to think “I’m not performing any automation, so I don’t need to know ARM.” Well, that is a fallacy, because even the work that you do inside of the Microsoft Azure Portal is dependent on ARM. You’ll see common terms like “Resource Group” or “RBAC” or “Resource Provider” pop up, and if you don’t understand the ARM architecture, these concepts might seem foreign.

You might also think “I can just use the old portal and the Service Management (ASM) API, because I already understand it.” While that might be true, in the short term, given enough time, the classic interface to Microsoft Azure will no longer exist. Although no one knows exactly when this will happen, it is being phased out slowly, and will be someday be fully replaced by ARM.

By having a solid foundation of knowledge to build on top of, you ensure that you are equipped with the resources to perform work inside of Microsoft Azure effectively, quickly, and with reduced friction.

Architecture

Here is a diagram that shows the high-level relationship between entities in the Azure Resource Manager (ARM) API. The concepts portrayed in the image will be explained, in more depth, under the Concepts heading.

Azure Resource Manager API Architecture

Concepts

Resource Provider

The Azure Resource Manager (ARM) REST API itself enables generic access to cloud resources. Each cloud resource is exposed to the ARM API through a layer known as a “Resource Provider.” Each Resource Provider exposes a related set of functionality as individual Resource Types. For example, Resource Providers exist for Compute, Network, Storage, App Service, Caching, SQL Server, and more.

You can use the List All Resource Providers REST API to obtain a list of Resource Providers, or if you prefer a simpler experience, use the Get-AzureRmResourceProvider PowerShell command. Below is the invocation of the PowerShell command, and the resulting output. As you can see from the output, each Azure Provider (“Provider Namespace”) exposes a series of different Resource Types.

Resource Provider Registration

In some cases, you may be required to “register” a Resource Provider with your Azure subscription, before you can utilize the Resource Types it contains. This scenario commonly occurs with Azure features that are in preview / pre-release, that Microsoft wants users to explicitly subscribe to, with the understanding that it’s still in preview. The ARM PowerShell module provides a command called Register-AzureProvider that enables the user to register the Resource Provider with the currently selected Azure subscription. All of the Resource Providers that are currently unregistered can be discovered using the following command:

Get-AzureRmResourceProvider -ListAvailable | Where-Object -FilterScript { $PSItem.RegistrationState -ne 'Registered'; }.

Resource Group

An ARM Resource Group is a container, created in a user-specified Azure Region, that hosts zero or more Azure cloud resources. While Resource Groups are generally intended to maintain cloud resources that share a common lifecycle — for example, a set of PaaS services for a single application — this is not a requirement. A Resource Group can be locked down via Role-based Access Control (RBAC) rules, such that only the specified users, or applications, can access it. Deploying resources to a Resource Group can be accomplished using imperative ARM APIs, or through an ARM JSON template. Resource Groups can have tags associated with them, which are explained under the Tags heading.

Resource

In the context of the Azure Resource Manager (ARM) API, a “resource” is an instance of a Resource Type, which is deployed into an ARM Resource Group, and is a generic representation of a Microsoft Azure cloud resource (eg. Storage Blob, Virtual Machine, Web App). Resources can be managed as simple JSON objects, however they must conform to a certain set of common properties in order to be properly recognized by the ARM REST API. These common JSON properties are listed below:

  • API Version – the API version that is used to manage the resource. This is documented in the ARM REST API reference.
  • Name – The name of the Resource within the Resource Group.
  • Type – The Resource Provider and Resource Type of the resource (eg. Microsoft.Compute/virtualMachines).
  • Location – The Azure Region (aka. Location) that the resource should be deployed to. Use the Get-AzureLocation command to obtain an authoritative list of Azure Regions.
  • Properties – This is the “property bag” that contains the resource definition. The required and optional JSON properties will be different for each Resource Type. Although documentation is incomplete, you can generally find the authoritative resource reference on MSDN.

In addition to managing Azure cloud resources using ARM Templates, resources can also be managed using imperative statements. The ARM PowerShell module and cross-platform (xPlat) CLI Tool include commands to perform create, read, update, and delete (CRUD) operations on most types of cloud resources.

Tags

Tags are key-value pairs of metadata that can be applied to individual ARM Resources, or to ARM Resource Groups. The value of an ARM tag can be null or non-null. Each Resource or Resource Group can have up to 15 tags associated to them. Many of the ARM PowerShell commands have a -Tag parameter on them, which accepts a HashTable (set of key-value pairs). You can use the built-in discovery mechanisms in PowerShell to find out which commands support the -Tag parameter by calling Get-Command -Module AzureResourceManager -ParameterName Tag.

Search for Tagged Resources

After you’ve tagged Resources and Resource Groups, you can search for them using the Find-AzureRmResource and Find-AzureRmResourceGroup commmands. For example, to find all Resource Groups that are tagged with a Department of Finance, you would use this command: Find-AzureRmResourceGroup -Tag @{ Name = ‘Department’; Value = ‘Finance’; }. By searching for resources, instead of memorizing Resource Group and Resource names, you can manage your resources much more efficiently.

ARM PowerShell

The ARM PowerShell module is designed to deploy and manage cloud resources to a Microsoft Azure subscription using the Azure Resource Manager API. When the ARM PowerShell module was initially released, it implemented a Switch-AzureMode command that “switched” back and forth between the Azure Service Management (ASM) PowerShell module and the ARM module. This behavior of “module switching” impedes the core design tenet of “discoverability” in PowerShell, by preventing the commands in either the ASM or ARM module from being discovered, while the other one is loaded. This mechanism was implemented, because the two PowerShell modules have conflicting command names. Instead of offering guidance to users, to simply use the standard PowerShell module prefix during command invocation (eg. AzureRM\Get-AzureVM vs. Azure\Get-AzureVM), Microsoft chose to rename all of the ARM PowerShell commands to the format: <Verb>-AzureRM<Noun>. This creates a new problem, however, because commands like Get-AzureRMVM are not as human-readable as the more straightforward Get-AzureVM.

Azure Cross-Platform (xPlat) CLI Tool

If you’re using Windows or non-Windows systems, and either don’t have access to PowerShell, or prefer not to use it, you can perform many management commands against your Azure subscription by using the Azure Cross-Platform (xPlat) CLI Tool. The Azure xPlat CLI is a Node.js application that communicates with both the Azure Service Management (ASM) API as well as the Azure Resource Manager (ARM) API. I’ve been using this tool fairly extensively, on Mac OS X, to manage ARM Resource Groups and deploy ARM JSON Templates.

Policy

Azure Resource Manager (ARM) supports policy rules that control whether a particular request will be allowed or denied. In some ways, this is similar to Role-Based Access Control (RBAC), but the features were actually designed to work together. Each request against the ARM API is evaluated against these rules.

You can use ARM Policy to limit the scope of API calls, for use cases similar to the following:

  • Resources can only be provisioned in certain regions
  • Only certain types of cloud resources can be provisioned (eg. a SQL Database, but not a Virtual Machine)
  • Resources must be provisioned, and managed, with a specific tag, for internal tracking purposes
  • Cloud resources must be provisioned using a specific naming convention

Once a new policy definition has been created, it can be assigned to an Azure subscription, a Resource Group, or an individual cloud resource. Policies are inherited at all levels, so a policy applied at the subscription level will also apply to all of the Resource Groups and individual cloud resources that it contains.

It’s important to note that ARM Policy only works when the API call uses the PUT HTTP verb.

Roles

In the older Azure Service Management (ASM) API, the only way to grant access to manage Azure cloud resources was to give someone “co-administrator” access to an Azure subscription. Because this role effectively gave “god” access to all Azure resources, Microsoft developed a more fine-grained mechanism to control access to cloud resources, called Role-Based Access Control (RBAC). RBAC is only available in the Azure Resource Management (ARM) API, and cannot be applied to resources created in the ASM API. If an Azure subscription has been configured with co-administrators in the ASM API, the co-administrator accounts will automatically inherit the built-in “owner” role in ARM. The owner role definition is described by the platform as: “Lets you manage everything, including access to resources.”

Role Definitions

A Role Definition in ARM is a set of permissions (“actions”) that member users and service principals have to a set of resources. Each Role Definition contains an “Actions” property, which contains an array of string values that declares the actions that can be performed. The structure of an “action” is as follows: <ResourceProvider>/<ResourceType>/<Action>. The Role Definition also contains a “NotActions” property which explicitly prevents certain actions from being performed, using a similar structure as the “Actions” property. The ARM PowerShell module provides a command called New-AzureRoleDefinition that enables you to create custom ARM Role Definitions. This command might initially look a little cryptic, with limited documentation, but it’s actually pretty easy to use, once you understand the properties that make up a Role Definition. I also wrote a helper function that makes it easier to create a role definition.

Once a Role Definition has been created, it can be applied at the subscription level, or to an ARM Resource Group, or individual Resources, using Role Assignments.

Role Assignments

Once a role definition has been created, an Azure Active Directory (AAD) user, group, or application (service principal) can be associated with the role definition. By default, a role assignment associates the user at the subscription level, but role assignments can also be scoped to a Resource Group or individual Resources. The ARM PowerShell module contains a command called New-AzureRoleAssignment that assists with associating a Role Definition to a user and scope.

ARM JSON Templates

The Azure Resource Manager API uses the concept of “JSON templates” (aka. “ARM templates”) to deploy and manage cloud resources from a declarative perspective. An ARM template is deployed into a Resource Group using the incremental or complete modes. The incremental mode means that newly provisioned resources will be deployed alongside (merged) with existing resources in the same Resource Group. The complete mode means that the ARM template contains all of the cloud resource definitions that should make up the “desired state” of the target Resource Group, thereby resulting in a complete elimination and overwrite of any existing resource in the target Resource Group.

2015-08-22 20_16_11-Presentation1 - PowerPoint

ARM Template Deployment Process

Structure

Each ARM template file must follow a specific structure, in order to be successfully deployed to the ARM REST API. There are two root-level JSON properties that must be declared, $schema and contentVersion.

The main body structure of an ARM JSON template contains the following four major sections.

  • Input Parameters
  • Variables
  • Resources
  • Output Parameters

Finding a Template

To speed the provisioning of cloud resources, Microsoft has provided a repository of ARM templates on a GitHub “quickstart templates” project. At the project’s root, there are a series of folders that contain different ARM templates, that target specific scenarios. These templates include a metadata file (metadata.json) that describes the purpose of the ARM template in plain English. The metadata contained in this file is also used to drive the searchable “Azure Quickstart Templates” web front-end on azure.microsoft.com.

The main files behind each of these ARM JSON templates are the azuredeploy.json and azuredeploy-parameters.json files. The azuredeploy.json file is responsible for declaring the cloud resources that make up the template, as well as any variables that will be used inside the template, and input & output parameters. ARM templates can accept input parameters, and can also emit output for values that are computed inside the template engine. The azuredeploy-parameters.json file is a simple file that declares the input parameter values that will be passed into the ARM template during a deployment operation.

Creating a Template

When you create an ARM template, try not to think of creating one from the “ground up.” While ideally it would work this way, reality is a bit different. Creating an ARM template, without prior knowledge of the structure, can be very challenging and cause significant mental frustration. To avoid losing the hair on your head, you’ll want to look at existing ARM templates, and copy the bits of JSON that you need from each one. For example, if you need to deploy an Azure Web App and a Virtual Network, you might need to look at two separate ARM templates, and grab just those pieces from each one, and paste it into your own, custom template.

If you do want to tackle authoring an ARM JSON template yourself, Microsoft released an Azure Resource Manager extension for Visual Studio Code. You can install this very easily by opening VS Code, pressing F1, searching for the “Extensions: Install Extension” command, and then performing a partial search for “azure.” Just hit Enter, restart VS Code, and you’ll have the extension installed! As long as you have the $schema property defined in a fresh JSON file, VS Code will give you Intellisense / auto-completion results for most of the ARM JSON Template, so you can build from the “ground up!”

Deployment

Now that we’ve reviewed what the ARM API is, and how ARM templates work, let’s talk about deploying cloud resources using ARM. There are a few different ways to deploy resources using the Azure Resource Manager API. In the following sections, we will explore each of these in depth.

  • Deploy to Azure” Button (via Azure Portal)
  • Using ARM PowerShell
  • Azure Resource Group” Visual Studio Project template
  • Azure Cross-Platform (xPlat) CLI Tool
  • Call the ARM REST API directly

Deploy to Azure Button

The “deploy to Azure” button appears on the README.md pages on the Azure Quickstart Templates GitHub repository. This button enables the rapid deployment of ARM Templates directly from the source. Once the button is clicked, the user is redirected to the Microsoft Azure “Ibiza” Preview Portal, where they are prompted to enter the input parameters for the JSON template.

Deploying ARM Templates using PowerShell

The Azure Resource Manager PowerShell module offers a variety of generic “resource” commands that enable management of ARM resources, provided the user understands ARM implementation details. Once you’ve authenticated to Azure from PowerShell, and selected the appropriate Azure subscription, you can use a single command to create a new Resource Group, and then deploy an ARM template into it.

The New-AzureRmResourceGroup command is the ARM PowerShell command that can create a Resource Group and deploy it. The command offers a few essential parameters to successfully deploy.

  • Name – The name of the Azure Resource Group that will be deployed
  • Location – The Azure Region where the Azure Resource Group will be managed from
  • TemplateUri – The URL to the ARM JSON template file that will be deployed. You can plug in the full GitHub URL to the JSON file
  • TemplateParameterObject – A PowerShell HashTable that contains the parameter names-value pairs that will be passed into the ARM JSON template

PowerShell Automation

The ARM PowerShell module offers a wide variety of commands to manage cloud resources. The following sections will discuss the topic in more depth. To get started using the ARM PowerShell module, you will follow a process roughly similar to the following:

  1. Authenticate using Add-AzureRmAccount.
  2. Select the subscription you want to operate on, using Select-AzureRmSubscription.
  3. Perform cloud automation tasks.

Authentication

While many existing Azure users are familiar with the term “PublishSettings file,” this concept no longer exists in the Azure Resource Manager (ARM) API. The old “PublishSettings file” was an auto-generated file that contained Azure subscription metadata, and a self-signed certificate that was used to authenticate against the Azure Service Management (ASM) REST API from PowerShell. There were several problems with this model, the least of which is that obtaining the file was a manual process requiring the use of a web browser. Another major problem with this model is that the use of certificates for authentication made it nearly impossible to determine who performed management tasks against the Azure Subscription.

Going forward, the Add-AzureRmAccount PowerShell command is used to authenticate against your Microsoft Azure Active Directory (AAD) tenant, or using a Microsoft Account (MSA). The Add-AzureRmAccount command has an optional -Credential parameter, which enables you to pass in a PSCredential instance to authenticate, thereby removing the interactive authentication requirement. It is important to note that the -Credential parameter cannot be used with Microsoft Accounts.

You can construct a PSCredential instance by calling the New-Object PowerShell command. The PSCredential object constructor has two overloads. The first accepts a single parameter, which is a generic PSObject instance, which must have a Username and Password property. You can use a PowerShell HashTable for this purpose. The second constructor accepts a String parameter for the username, and a SecureString parameter for the password. The latter constructor is the safer one to use, since it works with the SecureString class, instead of the plain-text String class.

With this background information, let’s look at how to authenticate to Microsoft Azure, using the PowerShell module.

### Invoking this command, with no parameters
### will result in an authentication dialog.
Add-AzureRmAccount;

### You can specify an "environment" (eg. China Cloud, Public Cloud, Government 
### Cloud, Azure Stack) to authenticate against.
Get-AzureEnvironment;
Add-AzureRmAccount -Environment AzureCloud;

### You can specify a PSCredential instance for the -Credential parameter
### to perform silent authentication.
$AzureCredential = Get-Credential;
Add-AzureRmAccount -Credential $AzureCredential;

### If you want to construct a PSCredential instance, instead of
### being prompted for it, you can use the New-Object command.
### (TAKE NOTE of the line continuation character)
$AzureCredential = New-Object -TypeName PSCredential `
    -ArgumentList @{ Username = 'ps@trevorsullivan.net'; Password = 'S3cur1ty!!'; };
Add-AzureRmAccount -Credential $AzureCredential;

Subscription Management

Once a user has authenticated to the Azure platform from PowerShell, the next step is to determine which Azure subscription you want to operate on. The user account that you authenticated with might have access to more than one Azure subscription. For example, your organization may grant you access to an enterprise subscription, and you may separately sign up for an Azure subscription personally using MSDN credits or a pay-as-you-go subscription with a credit card.

The ARM PowerShell module provides a Get-AzureRmSubscription command to list out all of the subscriptions that you have currently available. Even if you don’t have an up-to-date authentication token, the ARM PowerShell module keeps a local cache of subscription metadata. If you have a large number of subscriptions, you can pipe the output of Get-AzureRmSubscription into the PowerShell Out-GridView command, which enables you to filter and sort the data easily, in a friendly graphical interface.

Once you’ve found a subscription that you’d like to use, you use the Select-AzureRmSubscription command to change contexts into a specific one. Only a single Azure subscription can be “selected” at a time. Any ARM PowerShell commands you issue, at that point, will operate on the “selected” subscription.

Profiles

An Azure Profile is returned when you authenticate to Microsoft Azure. Each Azure Profile object can have more than one Azure subscription associated with it. Each Azure Profile object has a “current” subscription. You use the Select-AzureRmSubscription command to change the subscription that you want to operate on.

To create an Azure Profile, you call the Add-AzureRmAccount cmdlet, and specify the -Credential (a PSCredential instance) parameter. Assign the resulting object to a variable, so you can reference it. Now that you have created an Azure Profile object, you can change into that Profile’s context by using the Set-AzureRmContext command. If you’re only dealing with a single profile, then you don’t need to worry about changing Azure Profile contexts.

 

Imperative Deployment with PowerShell

While ARM templates can be used to deploy Azure cloud resources declaratively, you can also use imperative PowerShell statements to create resources. Each Azure platform feature (eg. SQL database, Redis Cache, App Service, etc.) has PowerShell commands that enable you to provision and manage those resources within your Azure subscription.

Azure SQL Database

If your application uses Microsoft SQL Server as its back-end data storage mechanism, you might be able to move your SQL database to the Microsoft Azure public cloud platform. There are some limitations of Azure SQL that might prevent you from leveraging a cloud-based SQL instance, but in many cases, applications can easily be migrated to an Azure SQL database.

Provisioning Azure SQL databases is incredibly easy using the Microsoft Azure Resource Manager (ARM) API and the ARM PowerShell module. You can use ARM JSON templates to provision Azure SQL Servers and SQL databases, however, you can also leverage the ARM PowerShell module’s imperative commands to create and manage these entities.

To create a new Azure SQL Server, use the New-AzureRmSqlServer PowerShell command:

$SqlServer = @{
    ResourceGroupName = 'contoso';
    ServerName = 'contoso';
    SqlAdministratorCredentials = Get-Credential;
    ServerVersion = '12.0';
    Location = 'West Europe';
    };

New-AzureRmSqlServer @SqlServer;

Once you’ve provisioned a new SQL Server instance, you can begin provisioning SQL Server databases on top of the SQL Server instance, using the New-AzureRmSqlDatabase command. There are a variety of parameters that help you to configure your Azure SQL Databases, including the -MaxSizeBytes parameter, which specifies the Azure SQL DB’s maximum size, and the -Edition parameter, which allows you to specify the edition of Azure SQL Database that should be used for this particular SQL Database.

$SqlDatabase = @{
    ResourceGroupName = 'contoso';
    SqlServer = 'contoso';
    DatabaseName = 'contoso';
    MaxSizeBytes = 500MB;
    Edition = 'Basic';
 };
New-AzureRmSqlDatabase @SqlDatabase;

After provisioning the Azure SQL Database, you can connect to the database instance and issue standard T-SQL commands against it. From PowerShell, you can easily achieve this using Microsoft .NET Framework types.

class SqlConnectionString {
    [string] $Server;
    [string] $Database;
    [pscredential] $Credential;

    SqlConnectionString([string] $Server, [string] $Database, [pscredential] $Credential) {
        $this.Server = $Server;
        $this.Database = $Database;
        $this.Credential = $Credential;
    }
    
    [string] GetConnectionString() {
        return 'Server={0};Database = {1}; User Id = {2}; Password = {3}' -f $this.Server, $this.Database, $this.Credential.UserName, $this.Credential.GetNetworkCredential().Password;
    }
}

$ConnString = [SqlConnectionString]::new('contoso.database.windows.net', 'contoso', (Get-Credential));

$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $ConnString.GetConnectionString();
$SqlConnection.Open();

$SqlCommandText = @'
CREATE USER contosouser WITH PASSWORD = '!c0nt0s0!';
ALTER ROLE db_owner ADD MEMBER contosouser;
CREATE TABLE contoso.VirtualMachines
'@
$SqlCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList @($SqlCommandText, $SqlConnection);
$SqlCommand.ExecuteNonQuery();

Virtual Networks

Virtual Networks can be provisioned with relative ease, using imperative PowerShell commands. First, you create the Subnet configurations, that subdivide the IP address space of the Virtual Network, using the New-AzureRmVirtualNetworkSubnetConfig command. Next, you create a Virtual Network by passing in those Subnet configurations, using the New-AzureRmVirtualNetwork command. You can optionally assign a Network Security Group (NSG) to a subnet, which is essentially a collection of layer 4 network security (firewall) rules. Each subnet can only have a single Network Security Group associated with it.

Here is a working example of creating a new Resource Group, with a single Virtual Network, containing a single Subnet. You must first authenticate to Microsoft Azure, before running this script, and ensure that you have selected the appropriate subscription.

### 1. Create a Resource Group
$ResourceGroup = @{
    Name = 'Contoso';
    Location = 'West US';
    Force = $true;
    };
New-AzureRmResourceGroup @ResourceGroup;

### 2. Create one or more Subnet configurations
$Subnet = @{
    Name = 'Hosts';
    AddressPrefix = '10.9.1.0/24';
    };
$SubnetConfig = New-AzureRmVirtualNetworkSubnetConfig @Subnet;

### 3. Create the Virtual Network
$VirtualNetwork = @{
    ResourceGroupName = $ResourceGroup.Name;
    Name = 'contoso';
    Location = 'Central US';
    AddressPrefix = @('10.9.0.0/16', '10.10.0.0/16');
    DnsServer = @('8.8.8.8', '8.8.4.4');
    Subnet = $SubnetConfig;
    Force = $true;
    };
New-AzureRmVirtualNetwork @VirtualNetwork;

Virtual Machine Extensions

In Microsoft Azure, Virtual Machine Extensions enable you to configure various settings on your Virtual Machine instances. For example, you can configure PowerShell Desired State Configuration (DSC), deploy a custom PowerShell script, configure SQL Server, and much more. Virtual Machine Extensions are, to some extent, treated like Virtual Machine images. To discover all of the Virtual Machine Extensions available, use the following command.

Get-AzureRmVMImagePublisher -Location westus| % { Get-AzureRmVMExtensionImageType -Location westus -PublisherName $PSItem.PublisherName } | Out-GridView

Azure Automation

The Microsoft Azure Automation service is a feature of the Microsoft Azure platform. However, this platform feature can sometimes be ambiguous, and therefore confused with, the generic concept of automating the Microsoft Azure cloud platform from an external system (eg. a Windows laptop or management server). The Azure Automation feature exposes a couple of key, useful management functions, including:

  • A cloud-based PowerShell Desired State Configuration (DSC) Pull Server
  • Runbook execution environment and scheduling

In order to utilize Azure Automation features, you must first create a top-level element called an Automation Account. An Automation Account must be deployed into an Azure Resource Manager (ARM) Resource Group, and must also be deployed into an Azure Region (aka. Location). Once you have created an Automation Account, you can upload Runbooks and set up a DSC Pull Service.

Runbooks are simply PowerShell scripts that perform automation tasks. In previous iterations of this feature, and Service Management Automation (SMA), a “runbook” was required to be implemented as a PowerShell Workflow. This requirement no longer exists in Azure Automation. Developing Runbooks locally, and synchronizing them with your Azure Automation Account can be simplified by using the Azure Automation Integrated Scripting Editor (ISE) add-on for PowerShell ISE.

Assets are data fragments that can be stored securely inside of an Azure Automation Account, and can be used from within Runbooks. The types of assets that can be created are: variables, schedules, credentials, certificates, and connections. Variables and credentials are simple key-value pairs that can be referenced within Runbooks. Schedules are associated with automation Runbooks, instead of being consumed by them, and enable the Runbooks to be automatically deployed on a periodic basis.

Run the following command to find a list of PowerShell commands related to the Azure Automation feature.

### Get Azure Automation commands
Get-Command -Module AzureResourceManager -Name *Automation*;

Here is a working PowerShell script example of how to create an Automation Account inside of a new Resource Group.

### Create a Resource Group
$ResourceGroup = @{
 Name = 'AzureAutomation';
 Location = 'West US';
 Force = $true;
 };
New-AzureRmResourceGroup @ResourceGroup;

### Validate that the Microsoft.Automation Resource Provider has been registered
Get-AzureRmResourceProvider -ProviderNamespace Microsoft.Automation;

### Find supported locations for this Resource Provider
(Get-AzureRmLocation | Where-Object -FilterScript { $PSItem.Name -match 'Microsoft\.Automation'; }).Locations;

$AutomationAccount = @{
 ResourceGroupName = $ResourceGroup.Name;
 Name = 'ContosoAutomation';
 Location = 'West Europe';
 Plan = 'Free';
 };
New-AzureRmAutomationAccount @AutomationAccount;

After creating an Automation Account, you can begin importing Runbooks into it, creating Assets, and leveraging the DSC Pull Server service.

Importing Runbooks, from your local filesystem, can be achieved using the Import-AzureRmAutomationRunbook command. After importing the Runbooks into the Automation Account, use the Get-AzureRmAutomationRunbook command to retrieve a list of Runbooks that have been created / imported into the Automation Account.  The Start-AzureRmAutomationRunbook command is used to kick off Runbook execution immediately.

### Create a Runbook (PowerShell script)
Set-Content -Path runbook.ps1 -Value Get-Process;

### Import a Runbook
$Runbook = @{
 ResourceGroupName = $ResourceGroup.Name;
 AutomationAccountName = $AutomationAccount.Name;
 Path = '.\runbook.ps1';
 Type = 'PowerShell';
 Description = 'Does some cool stuff!';
 Force = $true;
 };
Import-AzureRmAutomationRunbook @Runbook;

### Get a list of Runbooks in the Automation Account
Get-AzureRmAutomationRunbook -ResourceGroupName $ResourceGroup.Name -AutomationAccountName $AutomationAccount.Name;

Azure xPlat CLI Automation

The Microsoft Azure Cross-Platform (xPlat) CLI Tool is a Node.js application that enables you to automate the Microsoft Azure cloud platform from Windows and non-Windows systems. In this section, we’ll talk about how to use this tool to automate various aspects of the Microsoft Azure platform.

JSON Output

Although PowerShell users are accustomed to working with objects, the Unix command line deals with plain text instead. Although it is quite common to use tools like grep, sed, and awk to parse command line output, the Azure xPlat CLI Tool can optionally emit JSON objects. Because JSON objects can be interpreted using tools such as jsawk and jq, we don’t have to worry about writing our own, custom text parsers.

xPlat Operational Modes

Because the Azure xPlat CLI Tool works with both the ASM and ARM APIs, there are two exclusive modes that it can operate in. The azure config mode sub-command enables you to switch between these modes.

### Change into Azure Service Management (ASM) mode
azure config mode asm
### Change into Azure Resource Manager (ARM) mode
azure config mode arm

Authentication

Before you can manage your Azure cloud resources with the Azure xPlat CLI Tool, you need to authenticate against Azure Active Directory (AAD) or your Microsoft Account. You can simply call azure login, with no parameters, to authenticate, but you will have to open a web browser, navigate to http://aka.ms/devicelogin, and type in a code that is provided to you at the command line. Instead of doing that, I recommend that you use the azure login –username user@domain.com command. This way, you’ll be prompted to type your password at the command line, instead of having to leave the terminal to authenticate.

### Authenticate with your Azure Active Directory (AAD) account
azure login --username user@domain.com

Locations

The Azure xPlat CLI Tool enables us to list out the Azure Regions (aka. Locations) that are available for us to deploy resources into. The azure location list command is all you need to run.

> azure location list

info:    Executing command location list
warn:    The "location list" commands is changed to list subscription's locations. For old information, use "provider list or show" commands.
info:    Getting locations...
data:    Name                Display Name         Latitude  Longitude
data:    ------------------  -------------------  --------  ---------
data:    eastasia            East Asia            22.267    114.188
data:    southeastasia       Southeast Asia       1.283     103.833
data:    centralus           Central US           41.5908   -93.6208
data:    eastus              East US              37.3719   -79.8164
data:    eastus2             East US 2            36.6681   -78.3889
data:    westus              West US              37.783    -122.417
data:    northcentralus      North Central US     41.8819   -87.6278
data:    southcentralus      South Central US     29.4167   -98.5
data:    northeurope         North Europe         53.3478   -6.2597
data:    westeurope          West Europe          52.3667   4.9
data:    japanwest           Japan West           34.6939   135.5022
data:    japaneast           Japan East           35.68     139.77
data:    brazilsouth         Brazil South         -23.55    -46.633
data:    australiaeast       Australia East       -33.86    151.2094
data:    australiasoutheast  Australia Southeast  -37.8136  144.9631
data:    southindia          South India          12.9822   80.1636
data:    centralindia        Central India        18.5822   73.9197
data:    westindia           West India           19.088    72.868
info:    location list command OK

Resource Groups

The Azure xPlat CLI Tool enables you to manage your Azure Resource Manager (ARM) Resource Groups. To do this, you use the azure group sub-command.

The azure group create sub-command allows you to create a new ARM Resource Group inside your Azure subscription.

> azure group create --location westus --name MyCoolResourceGroup --tags Company=ContosoCorp
info:    Executing command group create
+ Getting resource group MyCoolResourceGroup
+ Creating resource group MyCoolResourceGroup
info:    Created resource group MyCoolResourceGroup
data:    Id:                  /subscriptions/cb8b24b7-63fb-41b8-bc9b-a9e92cfa7ae9/resourceGroups/MyCoolResourceGroup
data:    Name:                MyCoolResourceGroup
data:    Location:            westus
data:    Provisioning State:  Succeeded
data:    Tags: Company=ContosoCorp
data:
info:    group create command OK

Deploy ARM JSON Templates

One of the simplest and fastest methods of deploying cloud resources into your Azure subscription, is to use Azure Resource Manager (ARM) JSON Templates. These JSON template files contain declarations of cloud resources, which are provisioned when you deploy the template into an ARM Resource Group. To deploy an ARM JSON Template using the Azure xPlat CLI Tool, we use the azure group deployment create sub-command. If you add the –help parameter, you’ll see a list of all the parameters that you can specify for this sub-command.

Here are some of the common parameters that you’ll come across:

  • –resource-group – This parameter accepts the name of the ARM Resource Group that you want to deploy the template into. An ARM template can currently only be deployed into a single Resource Group. If you need to deploy resources into different Resource Groups, you will need to break up your deployment into multiple ARM JSON Templates.
  • –template-uri – This parameter accepts the full URI of the ARM JSON Template that you want to deploy. You don’t even need to download the template, in order to deploy it! The tool will handle that for you.
  • –template-file – This parameter is mutually exclusive to the –template-uri parameter. You would use this parameter to specify the path to the ARM JSON Template file on your local filesystem, if that is where you are deploying the template from.
  • –name – This parameter is entirely optional, but if you do specify it, the “deployment” object that’s created inside the targeted ARM Resource Group, will be assigned this value as its “name.”
  • –parameters – This parameter is very important, because it specifies the input parameters that will be sent to the ARM JSON Template that you’re deploying. The value of this input parameter should be a singleton JSON-formatted string that declares the input parameters as follows: “{ “<paramname>”: { “value”: “<paramvalue>” }, “<paramname>”: { “value”: “<paramvalue>” } }“.
  • –mode – For this parameter, you can specify one of two values: Incremental or Complete. If you use Incremental, then the command will append the resources defined in the ARM JSON Template with any pre-existing resources in the ARM Resource Group. If you specify Complete, then the command will replace any pre-existing resources in the ARM Resource Group with the resources that are declared in the ARM JSON Template.
  • –parameters-file – This parameter accepts a filesystem path, which contains the JSON-formatted string, providing input parameters.
  • –subscription – The Microsoft Azure Subscription Name or Subscription ID that contains the Resource Group that you are deploying the ARM JSON Template into. This parameter is useful if you don’t want to change contexts into the desired subscription, using the azure account set sub-command.

Now that we’ve reviewed the command to deploy an ARM JSON Template, give it a try yourself! The below example deploys a simple Azure Availability Set into a Resource Group.

ResourceGroupName='MyCoolResourceGroup'
TemplateURI='https://github.com/Azure/azure-quickstart-templates/blob/master/101-create-availability-set/azuredeploy.json'
TemplateParams='{ "location": { "value": "westus" } }'
azure group deployment create --resource-group $ResourceGroupName --template-uri $TemplateURI --parameters

Availability Sets

The Microsoft Azure Cross-Platform (xPlat) CLI Tool has a sub-command that helps you manage Availability Sets. In Microsoft Azure, an Availability Set helps you to increase your application or service availability, by ensuring that at least one IaaS Virtual Machine is available to service requests. In fact, in order to be eligible for the 99.95% up-time SLA, you must deploy your application or service in an Availability Set.

Here’s a simple Bash script that will create a new ARM Resource Group, and create an Availability Set (containing no Azure Virtual Machines) inside the Resource Group.

Location='westus'
ResourceGroupName='avsettest'
AVSetName='WebServers'
azure group create --location $Location --name $ResourceGroupName
azure availset create --resource-group $ResourceGroupName --location $Location --name $AVSetName

Storage

The Microsoft Azure Cross-Platform (xPlat) CLI Tool has a sub-command that helps you manage Azure Storage. The Microsoft Azure Storage service actually has four, distinct components within it, all of which can be managed from the Azure xPlat CLI:

  • Blob Storage (Page, Block, and Append Blobs) – storage for unstructured data, and also provides the backing storage for Azure VM Virtual Hard Disks (VHDs)
  • Queues – Messaging queues that enable asynchronous communication between applications
  • NoSQL Tables – A key-attribute, schema-less storage mechanism
  • Storage Shares – Cloud-based SMB 3.x shares that can be mounted in Microsoft Azure Virtual Machines.

Using the azure storage sub-command in Azure xPlat CLI, we can perform nearly any operation against the Azure management plane (Azure Resource Manager (ARM)), and the Azure Storage service’s data plane, which has its own management endpoints.

Storage Account

In order to utilize any Azure Storage services, a Storage Account must be created within an ARM Resource Group. Storage Accounts are available in two types: Standard and Premium. Premium Storage Accounts are designed to be utilized with Azure Virtual Machines, and therefore only offer the Page Blob service. In order to access the other Azure Storage services, you’ll need to use a Standard Storage Account. Standard Storage Accounts offer several different types of redundancy, to ensure high availability of your data, which is transparent to the user.

Here is a list of the Storage Account types that are supported by Microsoft Azure:

  • Locally Redundant Storage (LRS) – Three copies of data are stored within the Azure Region that the Storage Account exists in.
  • Zone Redundant Storage (ZRS)
  • Geo Redundant Storage (GRS) – Three copies of data are stored within the Azure Region that the Storage Account exists in, in addition to another three copies in a “sister region.”
  • Read-Access Geo Redandant Storage (RAGRS) – Same as Geo Redundant, but the copies of data in the “sister region” are read-only.
  • Premium Storage (PLRS) – SSD-backed, high-throughput, low-latency Page Blob storage for Microsoft Azure Virtual Machines.

In the Azure xPlat CLI Tool, we manage Storage Accounts using the azure storage account sub-command. Let’s take a look at how to create a new Storage Account, using a simple Bash script, and review the parameters.

  • –resource-group – This parameter accepts the Resource Group Name that the Storage Account will be created in.
  • –type – This is the type of Storage Account that will be provisioned. Use the acronyms from the above list of Storage Account types.
  • –location – This parameter accepts an Azure Region (aka. Location) where the Storage Account will be created.
  • –name – This parameter accepts the name of the Storage Account. It must start with a lower-case letter, and can only contain lower-case characters, numbers, and must be 3 to 24 characters in length.
Location='westus'
ResourceGroupName='storagetest'
StorageAccountName='testacct20'
azure group create --location $Location --name $ResourceGroupName
azure storage account create --resource-group --location $Location --name $StorageAccountName --type LRS

Logging

The Azure Resource Manager (ARM) API provides a logging mechanism whereby status logs can be retrieved for a specific resource, or a Resource Provider. The ARM PowerShell module also provides commands that make accessing the REST API significantly easier. To get a list of logging-related commands in the ARM PowerShell module, run the following command: Get-Command -Module AzureResourceManager -Name log.

You can view logs for the past hour by simply calling Get-AzureRmLog without any parameters. You can retrieve logs for a maximum time period of 15 days by specifying the -StartTime and/or -EndTime parameters, and passing in DateTime objects as the values to those parameters. For example, to see the logs for the past 24 hours, you can simply specify the -StartTime parameter in the following example: Get-AzureRmLog -StartTime (Get-Date).AddDays(-1). If you want more detailed output, you can tack on the -DetailedOutput switch parameter; this causes the command to include the Azure Active Directory (AAD) claims that were presented along with each Azure Resource Manager (ARM) request. If there’s a specific Resource Group that you want to retrieve the logs for, you can use the -ResourceGroup parameter to specify its name.

Errors

While you’re working with Microsoft Azure Resource Manager (ARM), you’ll probably run into some error messages. Sometimes they’re very helpful errors, but sometimes they can be a little confusing to understand. We’ll explore some of the errors in this section, to hopefully unblock your efforts and get you on your way!

Template Errors

The errors documented in this section pertain to the deployment of Azure Resource Manager (ARM) JSON Templates.

Resource Not Defined in Template

When you attempt to deploy an ARM JSON template, you might receive an error message, similar to the following one.

Deployment template validation failed: ‘The resource ‘Microsoft.Network/virtualNetworks/mycoolvirtualnetwork’ is not defined in the template. Please see https://aka.ms/arm-template for usage details.’

If you receive this error message, it is most likely being caused by an invalid reference to a resource, perhaps in the DependsOn property of one of the resources that’s declared in your ARM JSON template. If you’re referencing an existing resource, that isn’t declared in the template that you’re trying to deploy, then you must use the resourceId() template function to reference it. You can store a reference to the resource in a variable, which would be the preferred, reusable approach, or you can simply use the resourceId() function in-line with your template.

Template Variable Not Found

When you’re deploying an ARM JSON Template, you might encounter the following error.

Unable to process template language expressions for resource ‘/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/mycoolresourcegroup/providers/Microsoft.Network/virtualNetworkGateways/VPNGateway’ at line ‘162’ and column ’10’. ‘The template variable ‘gatewaySubnetRef’ is not found. Please see https://aka.ms/arm-template/#variables for usage details.’

This error is caused by using a variable that you haven’t defined in your ARM JSON template. To solve this problem, use one of the following solutions:

  1. Search your ARM JSON template for the text variables(‘variablename’) to locate instances where you’ve referenced the variable, and remove them
  2. Go to the top-level variables section of the template and add the variable definition

After adding the variable declaration or removing the references to the variable, re-submit your ARM JSON template deployment.

Template Parameter Not Found

You might receive this error message during an ARM JSON template deployment:

Unable to process template language expressions for resource ‘/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/mycoolresourcegroup/providers/Microsoft.Network/virtualNetworkGateways/VPNGateway’ at line ‘163’ and column ’10’. ‘The template parameter ‘vpnType’ is not found. Please see https://aka.ms/arm-template/#parameters for usage details.’

This can happen if you utilized the parameters() template function in your resource or variable definitions, but did not actually add the input parameter to the parameters section of the ARM JSON template file. To resolve this, add a new input parameter to your ARM JSON template with the appropriate name.

Deployment Still Active

If you have given your ARM JSON template deployment a specific name (eg. New-AzureRmResourceGroupDeployment -Name HDInsightSecureCluster), and you’ve previously kicked off the same deployment, then you might encounter the following error message.

Unable to edit or replace deployment ‘HDInsightSecureCluster’: previous deployment from ‘7/15/2016 7:59:04 PM’ is still active (expiration time is ‘7/22/2016 7:59:02 PM’). Please see https://aka.ms/arm-deploy for usage details.

This message indicates that there’s another deployment going on, with the same name, in the same ARM Resource Group. In order to perform your new deployment, you’ll need to either:

  • Wait for the existing deployment to complete
  • Stop the in-progress / active deployment

You can stop an active deployment by using the Stop-AzureRmResourceGroupDeployment PowerShell command or the azure group deployment stop command in the xPlat CLI tool.

 

  • Graham Bunce

    OMG – Thanks for this. I have spent days wading through out-of-date or incomplete Microsoft blogs trying to work out how to generate a script to create a very complex microservice architecture with powershell using Resource Manager. Nothing they have ever written is as readable or as simple to understand as this. Thanks for taking the time to write it up.

  • HI following a (genius) tip from Graham Smith, I have been visiting this blog on a daily basis as there a lot of info to harvest.
    Q: creating resource groups against a subscription: do we now need to to Select-AzureProfile rather than Select-AzureSubscrption ?
    Q: Given I want to have all my resources to be located in the same region (West Europe) , do I need to set each Resources’ location individually in each JSON config file or is their an implicit default. Finally, I do like the json syntax for the ARM but you have to make “search and replace” changes rather than use a single PS var change and rerun the PS script.

    • Hey Daniel,

      I apologize for responding to you late.

      I’m still a little unclear about the relationships between profiles, contexts, subscriptions, and whatever else.

      With respect to Azure Regions, there is no implicit default. If you’re using ARM JSON Templates (declarative provisioning), then you would create a variable, that stores the desired Azure Region, and reference that variable for each resource instance that is in your ARM JSON Template.

      I think the ARM JSON Template variable will solve your “search and replace” issue in ARM JSON Templates. 🙂

      Cheers,
      Trevor Sullivan
      Microsoft MVP: PowerShell

      • Thanks for replying .. Yes the they need to clarify the semantics re the profiles and subscriptions.
        I agree with you, with ARM JSON Template variables is the way to parametrise the script .
        I have started using the templates and then the I am told that SharePoint needs to be configured by DSC JSON files( WMF 4.0 / 5.0 ) is this what you have been told?

  • Pingback: Getting Started with Azure Resource Manager - Please Release Me()

  • Pingback: Azure Portal vs classic | Microsoft Enterprise Technologies()

  • Pingback: Azure CLI - the cross-platform way - Shared Points - Site Home - TechNet Blogs()

  • Pingback: Introduction to Azure Resource Manager Templates for the IT Pro – Techdiction()

  • Pingback: My Azure bookmarks – Niklas Lundgren()

  • Pingback: Change Allocation Method for Azure Network Interface - Trevor Sullivan()

  • Ewin Kiser

    Awesome article!! I can’t find anything on ARM in book form and the MS articles I read are out of date. Some of the Quickstarts will NOT run and I get all types of errors………so frustrating! Write a book on ARM and you will make a fortune! 🙂