Friday, August 21, 2015

Azure Resource Manager Templates, The Missing Parts

Azure Resource Manager (ARM) is a fantastic addition to the Azure ecosystem. The fact that you can create a template for your environment is all the better. Basically, it allows you to describe all of the resources you need in an environment and have an amazing amount of configuration.

ARM templates are merely JSON files that use JSONSchema. Visual Studio gives you validation and lets you know if you reference something that is not known. The support is nice, but there are several things that do not validate, but work against Azure regardless.

A good place to start looking for template examples is the Azure GitHub repo azure-quickstart-templates. There are many examples, but from a development standpoint, good examples putting them all together are difficult to come by.

All is not rainbows and butterflies, there are limitations. To list a few:
  • If you require a GUI, you will be greatly dismayed by the offering in Visual Studio. It is extremely simplistic and you will quickly out grow it. There is, however, a very nice JSON Outline pane which will help you navigate the JSON file.
  • Database servers cannot be shared across Resource Groups.
  • Does not handle Cloud Services (Web/Worker Roles)
  • Does not handle Service Bus namespaces
  • There are a number of other services that are not supported on the new portal and not in the Resource Manager.
I am sure that in the coming months the unhandled services will be supported. It is possible to get around the these limitations via Powershell, but you don't don't get the template deployment goodness.

Build your connection strings

Assuming you are describing an Azure Web App, you can configure the configuration connectionstrings. You can build them based on other resources described in the template.
Here are examples of the ones I have been able to find:
"DefaultConnection": {
    "value": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', parameters('serverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', parameters('databaseName'), ';User Id=', parameters('administratorLogin'), '@', parameters('serverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
    "type": "SQLAzure"
},
"variables": {
    "storageAccountId": "[concat('/subscriptions/',subscription().subscriptionId,'/resourceGroups/',resourceGroup().name,'/providers/','Microsoft.Storage/storageAccounts/', parameters('storageAccountName'))]",
...
"AzureWebJobsDashboard": {
    "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('storageAccountName'), ';AccountKey=', listKeys(variables('storageAccountId'),'2015-05-01-preview').key1)]",
    "type": "custom"
},
So this next one is a bit cheating, but it is presently the only way to make it happen (for now). I will dive more into this later.
"AzureWebJobsServiceBus": {
    "value": "[parameters('serviceBusConnectionString')]",
    "type": "custom"
},
"WebDocDb": {
    "value": "[concat('AccountEndpoint=', reference(concat('Microsoft.DocumentDb/databaseAccounts/', parameters('databaseName'))).documentEndpoint, ';AccountKey=', listKeys(resourceId('Microsoft.DocumentDb/databaseAccounts', parameters('databaseName')), '2015-04-08').primaryMasterKey, ';')]",
    "type": "custom"
},
"RedisCache": {
    "value": "[listKeys(resourceId('Microsoft.Cache/Redis', parameters('redisName')), '2014-04-01').primaryKey]",
    "type": "custom"
}

Web App With Staging Slot

Here is a good example of how to create a web application with a staging slot both containing the correct connection strings.
"variables": {
    "siteNameStage": "[concat(parameters('siteName'),'stage')]",
    "databaseNameStage": "[concat(parameters('databaseName'),'stage')]",

    "storageAccountId": "[concat('/subscriptions/',subscription().subscriptionId,'/resourceGroups/',resourceGroup().name,'/providers/','Microsoft.Storage/storageAccounts/', parameters('storageAccountName'))]",
    "storageAccountIdStage": "[concat('/subscriptions/',subscription().subscriptionId,'/resourceGroups/',resourceGroup().name,'/providers/','Microsoft.Storage/storageAccounts/', variables('storageAccountNameStage'))]",

    "storageAccountNameStage": "[concat(parameters('storageAccountName'),'stage')]"
},
"resources": [

...

    /*** Web App ***/
    {
      "apiVersion": "2015-06-01",
      "name": "[parameters('siteName')]",
      "type": "Microsoft.Web/Sites",
      "location": "[parameters('siteLocation')]",
      "dependsOn": [ "[concat('Microsoft.Web/serverFarms/', parameters('hostingPlanName'))]" ],
      "tags": {
        "[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "empty"
      },
      "properties": {
        "name": "[parameters('siteName')]",
        "serverFarmId": "[parameters('hostingPlanName')]"
      },
      "resources": [
        {
          "apiVersion": "2014-11-01",
          "type": "config",
          "name": "connectionstrings",
          "dependsOn": [
            "[concat('Microsoft.Web/Sites/', parameters('siteName'))]",
            "[resourceId('Microsoft.Sql/servers', parameters('serverName'))]",
            "[resourceId('Microsoft.Cache/Redis', parameters('redisName'))]"
          ],
          "properties": {
            "DefaultConnection": {
              "value": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', parameters('serverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', parameters('databaseName'), ';User Id=', parameters('administratorLogin'), '@', parameters('serverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
              "type": "SQLAzure"
            },
            "AzureWebJobsDashboard": {
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('storageAccountName'), ';AccountKey=', listKeys(variables('storageAccountId'),'2015-05-01-preview').key1)]",
              "type": "custom"
            },
            "AzureWebJobsStorage": {
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('storageAccountName'), ';AccountKey=', listKeys(variables('storageAccountId'),'2015-05-01-preview').key1)]",
              "type": "custom"
            },
            "AzureWebJobsServiceBus": {
              "value": "[parameters('serviceBusConnectionString')]",
              "type": "custom"
            },
            "WebDocDb": {
              "value": "[concat('AccountEndpoint=', reference(concat('Microsoft.DocumentDb/databaseAccounts/', parameters('databaseName'))).documentEndpoint, ';AccountKey=', listKeys(resourceId('Microsoft.DocumentDb/databaseAccounts', parameters('databaseName')), '2015-04-08').primaryMasterKey, ';')]",
              "type": "custom"
            },
            "RedisCache": {
              "value": "[listKeys(resourceId('Microsoft.Cache/Redis', parameters('redisName')), '2014-04-01').primaryKey]",
              "type": "custom"
            }
          }
        },
        {
          "apiVersion": "2015-04-01",
          "name": "appsettings",
          "type": "config",
          "dependsOn": [
            "[concat('Microsoft.Web/Sites/', parameters('siteName'))]"
          ],
          "properties": {
            "Demo:Environment": "PROD",
            "Test:Environment": ""
          }
        },
        {
          "apiVersion": "2014-11-01",
          "name": "slotconfignames",
          "type": "config",
          "dependsOn": [
            "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
          ],
          "properties": {
            "connectionStringNames": [ "DefaultConnection", "AzureWebJobsDashboard", "AzureWebJobsStorage", "AzureWebJobsServiceBus", "WebDocDb", "RedisCache" ],
            "appSettingNames": [ "Demo:Environment", "Test:Environment" ]
          }
        },
    
        /*** Web App STAGING SLOT ***/
        {
          "apiVersion": "2015-04-01",
          "name": "Staging",
          "type": "slots",
          "location": "[parameters('siteLocation')]",
          "dependsOn": [
            "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
          ],
          "properties": {
          },
          "resources": [
            {
              "apiVersion": "2014-11-01",
              "type": "config",
              "name": "connectionstrings",
              "dependsOn": [
                "[resourceId('Microsoft.Web/Sites/slots', parameters('siteName'), 'Staging')]",
                "[resourceId('Microsoft.Sql/servers', parameters('serverName'))]",
                "[resourceId('Microsoft.Cache/Redis', parameters('redisName'))]"
              ],
              "properties": {
                "DefaultConnection": {
                  "value": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', parameters('serverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', variables('databaseNameStage'), ';User Id=', parameters('administratorLogin'), '@', parameters('serverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
                  "type": "SQLAzure"
                },
                "AzureWebJobsDashboard": {
                  "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountNameStage'), ';AccountKey=', listKeys(variables('storageAccountIdStage'),'2015-05-01-preview').key1)]",
                  "type": "custom"
                },
                "AzureWebJobsStorage": {
                  "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountNameStage'), ';AccountKey=', listKeys(variables('storageAccountIdStage'),'2015-05-01-preview').key1)]",
                  "type": "custom"
                },
                "AzureWebJobsServiceBus": {
                  "value": "[parameters('serviceBusConnectionStringStage')]",
                  "type": "custom"
                },
                "WebDocDb": {
                  "value": "[concat('AccountEndpoint=', reference(concat('Microsoft.DocumentDb/databaseAccounts/', variables('databaseNameStage'))).documentEndpoint, ';AccountKey=', listKeys(resourceId('Microsoft.DocumentDb/databaseAccounts', variables('databaseNameStage')), '2015-04-08').primaryMasterKey, ';')]",
                  "type": "custom"
                },
                "RedisCache": {
                  "value": "[listKeys(resourceId('Microsoft.Cache/Redis', parameters('redisName')), '2014-04-01').primaryKey]",
                  "type": "custom"
                }
              }
            },
            {
              "apiVersion": "2015-04-01",
              "name": "appsettings",
              "type": "config",
              "dependsOn": [
                "[resourceId('Microsoft.Web/Sites/slots', parameters('siteName'), 'Staging')]"
              ],
              "properties": {
                "Demo:Environment": "TEST",
                "Test:Environment": "TEST"
              }
            }
          ]
        }
      ]
    },

...

That is a lot of JSON, but very useful.

Service Buses

Services buses don't seem to be receiving the love that other Azure resources have received, but it doesn't make them any less useful.

The trick to using/maintaining Service Buses is to not use the Resource Manager template. Basically, you can use a Powershell script to create the Service Bus(es), grab the connection string(s), and then pass the connection string into the ARM template deployment as a parameter.
function Create-AzureServiceBusQueue($Namespace, $Location) {
 # Query to see if the namespace currently exists
 $CurrentNamespace = Get-AzureSBNamespace -Name $Namespace;

 # Check if the namespace already exists or needs to be created
 if ($CurrentNamespace)
 {
  Write-Host "The namespace [$Namespace] already exists in the [$($CurrentNamespace.Region)] region.";
 }
 else
 {
  Write-Host "The [$Namespace] namespace does not exist.";
  Write-Host "Creating the [$Namespace] namespace in the [$Location] region...";
  New-AzureSBNamespace -Name $Namespace -Location $Location -CreateACSNamespace $false -NamespaceType Messaging;
  $CurrentNamespace = Get-AzureSBNamespace -Name $Namespace;
  Write-Host "The [$Namespace] namespace in the [$Location] region has been successfully created.";
 }
 return $CurrentNamespace.ConnectionString;
}
You may want to dig a little deeper and this page MSDN page, Use PowerShell to manage Service Bus and Event Hubs resources, is pretty useful.

Putting It Together

With the web application resource section in a template and the Service Bus(es) created via Powershell, how do we deploy the template to put it together?
# Create the Services Buses
$serviceBusConnectionStrings = @{"Prod"=$(Create-AzureServiceBusQueue $ServiceBusName $ResourceGroupLocation);
     "Stage"=$(Create-AzureServiceBusQueue "$($ServiceBusName)stage" $ResourceGroupLocation);
     "Dev"=$(Create-AzureServiceBusQueue "$($ServiceBusName)dev" $ResourceGroupLocation);}

...

$rg = Get-AzureResourceGroup | ? { $_.ResourceGroupName -eq $ResourceGroupName };
if ($rg -eq $null) {
 # Create the Resource Group
 New-AzureResourceGroup -Name $ResourceGroupName -Location $ResourceGroupLocation;
}
# Start a Resource Group deployment
$results = New-AzureResourceGroupDeployment `
  -Name WebAppDeployment `
  -ResourceGroupName $ResourceGroupName `
  -TemplateFile $TemplateFile `
  -TemplateParameterFile $TemplateParameterFile `
  -storageAccountNameFromTemplate $DefaultStorage `
  -serviceBusConnectionString $($serviceBusConnectionStrings.Prod) `
  -serviceBusConnectionStringStage $($serviceBusConnectionStrings.Stage);
Write-Output $results;
Write-Output "ServiceBus Prod: $($serviceBusConnectionStrings.Prod)";
Write-Output "ServiceBus Stage: $($serviceBusConnectionStrings.Stage)";
Write-Output "ServiceBus Dev: $($serviceBusConnectionStrings.Dev)";
This will configure the web app and populate the correct connection strings on the correct slot. Hopefully, Microsoft will add the capability to maintain Cloud Services and Service Buses soon, but until then, this will be helpful.