The Sec in DevSecOps

Obviously, the Sec stands for Secure (or Security if you like), which means, that we’re trying to use the DevOps mechanism as secure as possible. When developing, all your systems are tied together using secrets. If you like to communicate with a SQL Database, you need a password to connect. A password that you don’t like to be spread throughout the entire internet I guess. So that password is a secret. You know the secret, maybe some others may know it or can access it, but it’s not public, and you like to keep it that way.

That’s one part of the Sec. The second part would be that the infrastructure you deploy. It should deploy a ‘safe’ infrastructure and protect vulnerable endpoints from being reachable by ‘unwanted’ visitors. For example, when you’re developing a Web API that uses SQL Server as a storage mechanism, you may want to create a VNet and create that SQL Server inside that VNet. You can then allow access to that VNet from your API. Now nobody can access your SQL Server, except your own API, in case it has the correct secrets. Beautiful!

It’s all about the secrets

This part is not going to cover the VNet part. So if you’re looking for how to create VNets and allow access from A to B, go ahead and google it, or wait for one of my future posts to complete. In case you’re looking for part one, safely handling secrets… Stay with me and write along ;)

Now when I start writing a pipeline for a new project, I always create a new and separate resource group in Azure, for just this deployment. That resource group contains a storage account and a Key Vault. This Key Vault will contain all my ‘input’ secrets. And with input, I mean secrets that I use to provision my resources. So if you’re using a SQL Server instance, your admin password will be stored here.

On the side 1/2

Please note, that when you’re creating a resource that does not require a secret to be set, you don’t have to store anything in that Key Vault (you don’t need it). For example, when you need a Storage Account. When you need to access your storage account, you need to have a valid key or connection string to access it. This key or connection string can be retrieved from your ARM template, immediately after creation. In other words, there are no secrets required as input, to create the resource.

On the side 2/2

Also note, that the secrets I use in my system are not stored in that same Key Vault. If I need my ARM Template to store Connection Strings, Keys, or other secrets that you need to run the system, I first create a new instance of a Key Vault and store the secrets there. My example project for this BLOG will contain an ASP.NET Core Web API connecting to an Azure SQL Server. The password for my SQL Server is stored in my Deployment Key Vault. My ARM Template deploys stuff, and stores a connection string to the SQL Server in a different Key Vault, which is used by my API. I like to use Key Vault References for this.

So Let’s start

We’re going to create two new resource groups in Azure. Let’s call the first one ‘Deployments’ and the second one whichever name you like that corresponds with the system you like to deploy. I also like to include the environment name in the resource group name, for example, ‘systemx-test-api’, where systemx is replaced by some common shorthand name that I gave to the system. Now in the deployments group, let’s add a Key Vault, I called this ‘shared-deployments’. This is basically because I want to store the secrets of all my deployments here. This is because I run an Azure subscription that comes with my VS.NET Premium account which allows me to burn 130 euros monthly. I don’t want to provision several kinds of Key Vaults per system, but if your running production systems, please do!

Now in this Key Vault, create a new secret called systemx-test-sql-admin-password and give it a value containing a strong password.

Configure Azure DevOps

Now go to your Azure DevOps environment and open your projects. In the lower-left corner, click ‘Project Settings’ and look for ‘Service Connections’. Create a new service connection of type ‘Azure Resource Manager’. Authorize your Azure Subscription and point to the resource group ‘deployments’ and name this Service Connection ‘deployments’. You now have a service connection to the resource group ‘deployments’ in your Azure environment. Now create a new service connection, pointing to the other resource group and (also) name this according to your system and environment, so ‘systemx-test-api’ for example.

In Azure DevOps, navigate to [ Pipelines > Libraries ] and create a new Variable Group. Again, we’re going to name this ‘systemx-test-api’, add an optional subscription if you like. Leave the switch ‘Allow access to all pipelines’ on and toggle the ‘Link secrets from an Azure key value as variables’ to on if it’s switched off. This allows you, to point out secrets from your key vault, to use them as variables in your pipeline. Once you have turned this switch on, two new fields become available. In the Azure Subscription, do not select one of your Azure subscriptions, but select the Service Connection called ‘deployments’ instead. Then in the Key vault name field, select ‘shared-deployments’ or whatever you named your key vault. Once you did that, you notice that you cannot save your variable group. This is because you need to select at least one secret from the key vault to use. So under the ‘Variables’ table, click Add and select one or more secrets you like to be available in your pipeline. You can now save the variable group.

Using the variables

In one of my previous posts I explained multi-stage pipelines, and how to build and use ARM Templates. I assume you already got this, so in this part, things go pretty fast. In case it’s too fast for you, please read the ARM Templates from A to Z to learn and create a fully working API Project with a functioning multi-stage pipeline using an ARM Template to provision your infra on Azure.

The pipeline

So you first need a pipeline. I created this multi-stage pipeline Gist which builds and deploys the API Project and your deployment project and published the pipeline artifacts. In the following stages, these artifacts may be used to deploy the infrastructure and then the system itself.

Overriding parameters

Now as you may have noticed, I’m using the ‘overrideParameters’ when deploying ARM Templates. This allows me to pass variables from the pipeline to the ARM Template.

overrideParameters: '-sqlServerAdminPassword "$(systemx-test-sql-admin-password)"'

So the password is stored somewhere save in Key Vault. I granted the pipeline to use this secret. The pipeline runs and passes the secret as a parameter to the ARM Template, and the ARM Template starts provisioning stuff with that password… Just awesome…

Building the ARM Template

So first, let’s create a SQL Server :

{
  "type": "Microsoft.Sql/servers",
  "apiVersion": "2019-06-01-preview",
  "name": "[variables('sqlServerName')]",
  "location": "[resourceGroup().location]",
  "kind": "v12.0",
  "properties": {
    "administratorLogin": "[variables('sqlServerName')]",
    "administratorLoginPassword": "[parameters('sqlServerAdminPassword')]",
    "version": "12.0",
    "publicNetworkAccess": "Enabled"
  }
}

As you can see, the sqlServerAdminPassword parameter is used here to define the password for SQL Server. Then let’s add a database:

{
  "type": "Microsoft.Sql/servers/databases",
  "apiVersion": "2019-06-01-preview",
  "name": "[concat(variables('sqlServerName'), '/', variables('sqlDatabaseName'))]",
  "location": "[resourceGroup().location]",
  "dependsOn": [
    "[resourceId('Microsoft.Sql/servers', variables('sqlServerName'))]"
  ],
  "sku": {
    "name": "Standard",
    "tier": "Standard",
    "capacity": 10
  },
  "kind": "v12.0,user",
  "properties": {
    "collation": "SQL_Latin1_General_CP1_CI_AS",
    "maxSizeBytes": 268435456000,
    "catalogCollation": "SQL_Latin1_General_CP1_CI_AS",
    "zoneRedundant": false,
    "readScale": "Disabled",
    "readReplicaCount": 0,
    "storageAccountType": "GRS"
  }
}

And that’s all there is to it… Your database server and database are now successfully provisioned and waiting for your API to connect to it!

Let’s go API!

So the Web App requires a service plan since this is just a test I’ll provision a small one:

{
  "type": "Microsoft.Web/serverfarms",
  "apiVersion": "2018-02-01",
  "name": "[variables('webApiAppServicePlan')]",
  "location": "[resourceGroup().location]",
  "sku": {
    "name": "D1",
    "tier": "Shared",
    "size": "D1",
    "family": "D",
    "capacity": 0
  },
  "kind": "app",
  "properties": {
    "perSiteScaling": false,
    "maximumElasticWorkerCount": 1,
    "isSpot": false,
    "reserved": false,
    "isXenon": false,
    "hyperV": false,
    "targetWorkerCount": 0,
    "targetWorkerSizeId": 0
  }
}

And then when done, the Web App itself :

{
  "type": "Microsoft.Web/sites",
  "apiVersion": "2018-11-01",
  "name": "[variables('webApiAppService')]",
  "location": "[resourceGroup().location]",
  "kind": "app",
  "dependsOn": [
    "[resourceId('Microsoft.Web/serverfarms', variables('webApiAppServicePlan'))]"
  ],
  "identity": {
    "type": "SystemAssigned"
  },
  "properties": {
    "enabled": true,
    "hostNameSslStates": [
      {
        "name": "[concat(variables('webApiAppService'), '.azurewebsites.net')]",
        "sslState": "Disabled",
        "hostType": "Standard"
      },
      {
        "name": "[concat(variables('webApiAppService'), '.scm.azurewebsites.net')]",
        "sslState": "Disabled",
        "hostType": "Repository"
      }
    ],
    "serverFarmId": "[variables('webApiAppServicePlan')]",
    "reserved": false,
    "isXenon": false,
    "hyperV": false,
    "siteConfig": {},
    "scmSiteAlsoStopped": false,
    "clientAffinityEnabled": true,
    "clientCertEnabled": false,
    "hostNamesDisabled": false,
    "containerSize": 0,
    "dailyMemoryTimeQuota": 0,
    "httpsOnly": false,
    "redundancyMode": "None"
  },
  "resources": [
    {
      "name": "appsettings",
      "type": "config",
      "apiVersion": "2018-11-01",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.Web/sites', variables('webApiAppService'))]",
        "[resourceId('Microsoft.KeyVault/vaults', variables('keyVaultNamespace'))]",
        "[resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultNamespace'), variables('secret_sql_connectionstring'))]"
      ],
      "properties": {
        "WEBSITE_RUN_FROM_PACKAGE": "1",
        "ConnectionStrings:SqlServer": "[concat('@Microsoft.KeyVault(SecretUri=', reference(resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultNamespace'), variables('secret_sql_connectionstring'))).secretUriWithVersion, ')')]"
      }
    }
  ]
}

PRO TIP

Note line 10, it says SystemAssigned. This means that the API itself becomes a system-assigned identity in Active Directory, meaning that it’s something we can grant permissions. Because I use Key Vault References, I want to configure my Web App to reference the key vault secrets and fill them in for me while the app is being configured. This way, I don’t have to include any Key Vault magic in my code, just fetch the values from appsettings.json as you always do, but the values come from a key vault instead of the app settings file itself.

OK, now we’re getting somewhere, the SQL Server is there, the database is there, and the App Service Plan and Web App are there. Now as you can see, the Web App has some nested resources, which are the app settings. I want to store an app setting ConnectionStrings:SqlServer with the value of the actual SQL Connection String. This value @Microsoft.KeyVault() is that Key Vault Reference. These settings, depending on the Web App, The Key Vault, and the secret in that Key Vault. This means that the nested resource creating the app settings will wait to deploy those app settings, after the deployment of all dependencies completed successfully.

The Key Vault

So now it’s time to deploy the Key Vault:

{
    "type": "Microsoft.KeyVault/vaults",
    "name": "[variables('keyVaultNamespace')]",
    "apiVersion": "2015-06-01",
    "location": "[resourceGroup().location]",
    "dependsOn": [
    "[resourceId('Microsoft.Web/sites', variables('functionsAppService'))]"
    ],
    "properties": {
    "enabledForTemplateDeployment": false,
    "tenantId": "[subscription().tenantId]",
    "accessPolicies": [
        {
            "tenantId": "[subscription().tenantId]",
            "objectId": "[reference(concat(resourceId('Microsoft.Web/sites', variables('webApiAppService')),'/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-preview').principalId]",
            "permissions": {
                "secrets": [
                "get"
                ]
            }
        }
    }
    ]
}

As you can see, I created this Key Vault and grant get permissions on secrets, to the Managed Identity of the Web App. This means that the Web App now has permission to read the secrets. This is a requirement if you like to use Key Vault References. Now, all we need to do is store the Connection String to our SQL Server Database as a secret in the Key Vault.

{
  "type": "Microsoft.KeyVault/vaults/secrets",
  "name": "[concat(variables('keyVaultNamespace'), '/', variables('secret_sql_connectionstring'))]",
  "apiVersion": "2016-10-01",
  "location": "[resourceGroup().location]",
  "dependsOn": [
    "[resourceId('Microsoft.KeyVault/vaults/', variables('keyVaultNamespace'))]",
    "[resourceId('Microsoft.Sql/servers', variables('sqlServerName'))]"
  ],
  "properties": {
    "value": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', variables('sqlServerName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', variables('sqlDatabaseName'), ';User Id=', variables('sqlServerName'), '@', variables('sqlServerName'), ';Password=', parameters('sqlServerAdminPassword'), ';')]"
  }
}

And yes, this secret, of course, depends on the existence of the SQL Server instance and the Key Vault. Although you could store the connection string as a secret even when deployment of the SQL Server instance failed, in my opinion, it’s neat to add this dependency and make sure it’s deployed successfully.