Scale up Kubernetes cluster in Azure

Problem

Unexpectedly your Web Applications became a hit and you need to scale up your application. Your application is being hosted on a Kubernetes cluster. The cluster was created using `acs-engine` with X nodes but now it has to be scaled up to Y nodes.

Conditions that need to be met:

  • Zero Downtime.
  • Current nodes (VMs) need to go unchanged.

Solution

In case that the cluster was created using AKS (Azure Container Services) the solution is simple and doesn’t warrant another article. Just execute the command:

az acs scale -g myResourceGroup -n containerservice-myACSName --new-agent-count 10

The solution described is for clusters created using the acs-engine. This requires manual intervention in the generated ARM templates.

Let’s get started.

  • Open your `azuredeploy.json` and delete the resource Microsoft.Network/networkSecurityGroups. This includes the entire section below:
{
  "apiVersion": "[variables('apiVersionDefault')]",
  "location": "[variables('location')]",
  "name": "[variables('nsgName')]",
  "properties": {
    "securityRules": [
      {
        "name": "allow_ssh",
        "properties": {
            # SHORTENED FOR BREVITY
        }
      },
      {
        "name": "allow_kube_tls",
        "properties": {
            # SHORTENED FOR BREVITY
        }
      }
    ]
  },
  "type": "Microsoft.Network/networkSecurityGroups"
},
  • Delete dependency of the VNet resource o the networkSecurityGroup, by deleting the following line:
"dependsOn": [
    "[concat('Microsoft.Network/networkSecurityGroups/', variables('nsgName'))]" ## DELETE THIS LINE <<<<<<<
],
  • Open the file `azuredeploy.parameters.json` and replace the following part:
"agentpool1Count": {
  "value": X          
},

With:

"agentpool1Count": {
  "value": Y          
},
"agentpool1Offset": { 
  "value": X          
},

agentPool1Count will not necessarily be the name of your pool though that one needs to be changed to the new number of nodes desired. Meanwhile agentPool1Offset represents the old state.

  • Re-execute the deployment command with the new templates:
az group deployment create --name [DEPLOYMENT_NAME] --resource-group [K8S_CLUSTER_RG] --template-file "[path to updated ./azuredeploy.json]" --parameters "[path to updated azuredeploy.parameters.json]"

To verify execute ` kubectl get nodes ` and verify that old nodes are unchanged and just recently new nodes were added.

Additional advice

To ensure that the ARM Templates do not get lost and do not have to be recreated save them in a repository. (Be careful to not expose sensitive information). This will minimize the time needed to re-scale your application to meet your future demand. 😉

If you enjoyed the article, please share and comment below!