How do I use the Kubernetes Horizontal Pod Autoscaler?

With a Horizontal Pod Autoscaler (HPA), you can let Kubernetes automatically adjust the number of pods for a workload based on the current load. This allows you to add capacity during high load and reduce the number of pods when the load decreases. 

In this guide, you configure autoscaling based on CPU usage. You check the Metrics API, create an example deployment and test how the HPA scales up and down under load.

The Horizontal Pod Autoscaler automatically adjusts the number of pods. To also automatically adjust the number of nodes, we recommend using the Cluster Autoscaler.

How a Horizontal Pod Autoscaler works

The HPA controller is part of the Kubernetes control plane. This controller periodically reads metrics and compares the current value with the configured target value. With CPU autoscaling, Kubernetes calculates the average CPU load as a percentage of the CPU requests of the relevant pods.

Suppose a deployment has one pod with a CPU request of 200m and the HPA uses a target value of 50 per cent. The target load is then an average of 100m CPU per pod. If the average load rises above this value, the HPA increases the number of replicas up to the configured maximum.

Scaling up generally happens relatively quickly. For scaling pods down, Kubernetes uses a default stabilisation period of five minutes. This prevents the HPA from constantly changing the number of pods because of short-term fluctuations.

Requirements

For this guide, you need the following:

• A cluster: a Kubernetes cluster.
• kubectl: kubectl, configured for the correct cluster.
• Kubernetes version: version 1.23 or newer.

The commands in this guide work the same way in Linux, macOS and Windows PowerShell.

Checking and installing Metrics Server

An HPA requires a metrics source (for collecting data, in this case CPU usage). For CPU and memory metrics, Kubernetes generally uses Metrics Server. This component collects resource metrics from the kubelets and makes them available through the Metrics API.

Step 1

Check the active context and Kubernetes version:

kubectl config current-context
kubectl version

Only continue if the displayed context belongs to the intended cluster and the server version is 1.23 or newer.

Step 2

Check whether the Metrics API is already available:

kubectl top nodes

Do you see CPU and memory usage per node? Then continue with the section ‘Deploying an example application’.

Do you receive the message ‘Metrics API not available’? Then install the current official Metrics Server configuration:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

The -f option tells kubectl to apply the configuration at the specified address. For production environments, always first check the current Metrics Server release and the contents of the manifest.

Step 3

Wait until Metrics Server is available and then check the Metrics API:

kubectl rollout status deployment/metrics-server --namespace kube-system --timeout=180s
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top nodes

The --namespace option selects the kube-system namespace. With --timeout=180s, kubectl stops waiting after 180 seconds. It may take approximately another minute after the rollout before the first metrics become available. In the APIService output, the value under AVAILABLE should be True.

Deploying an example application

For the test, you use Kubernetes' official php-apache example application. This application consumes CPU when processing requests and has a CPU request of 200m.

Step 1

Create a separate namespace for the test:

kubectl create namespace hpa-tutorial

Step 2

Create the deployment and associated Service using the official example .yaml manifest:

kubectl apply --namespace hpa-tutorial -f https://k8s.io/examples/application/php-apache.yaml

The --namespace option creates the resources in hpa-tutorial. The -f option refers to the manifest that kubectl applies.

Step 3

Wait until the deployment is available:

kubectl rollout status deployment/php-apache --namespace hpa-tutorial --timeout=180s
kubectl get pods --namespace hpa-tutorial

The first command waits up to 180 seconds for a successful rollout. The second command should show one active php-apache pod.

Creating a Horizontal Pod Autoscaler

Step 1

Create an HPA that targets an average of 50 per cent of the requested CPU:

kubectl autoscale deployment php-apache --namespace hpa-tutorial --cpu=50% --min=1 --max=10

The options in this command have the following functions:

• --namespace hpa-tutorial: creates the HPA in the same namespace as the deployment.
• --cpu=50%: sets the average target load to 50 per cent of the CPU request.
• --min=1: keeps at least one replica active.
• --max=10: allows a maximum of ten replicas.

Step 2

Check the HPA and the received metrics:

kubectl get hpa --namespace hpa-tutorial
kubectl describe hpa php-apache --namespace hpa-tutorial

Immediately after creation, TARGETS may temporarily show <unknown>/50%. Wait until Metrics Server has collected a measurement and run the command again. In the detailed output, ScalingActive indicates whether the HPA is using a valid metric.

Testing the Horizontal Pod Autoscaler under load

Step 1

Start a temporary pod that continuously sends requests to the example application:

kubectl run load-generator --namespace hpa-tutorial --image=busybox:1.37.0 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

The --image option selects the container image to use. With --restart=Never, kubectl creates a separate pod instead of a deployment. Everything after -- is the command that is executed inside the container.

Step 2

Monitor how the HPA responds in your terminal:

kubectl get hpa php-apache --namespace hpa-tutorial --watch

The --watch option continues to display changes. Once CPU usage rises above 50 per cent, the HPA increases the desired number of replicas. Stop monitoring with ctrl + c.

Also check the created pods:

kubectl get deployment,pods --namespace hpa-tutorial

Step 3

Stop the load test:

kubectl delete pod load-generator --namespace hpa-tutorial

Monitor the HPA again:

kubectl get hpa php-apache --namespace hpa-tutorial --watch

Once the load disappears, the HPA reduces the number of replicas back to the configured minimum. Because of the default stabilisation period for scaling down, this generally takes approximately five minutes.

Configuring an HPA for your own workload

For a persistent configuration, use a .yaml manifest with API version autoscaling/v2. This allows you to store the HPA configuration together with the rest of your application's configuration.

Step 1

On Linux or macOS, open a new file:

nano hpa.yaml

On Windows PowerShell, use Notepad:

notepad.exe hpa.yaml

Add the following configuration:

apiVersion: autoscaling/v2
kind: HorizontalpodAutoscaler
metadata:
  name: <hpa-naam>
  namespace: <namespace>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: deployment
    name: <deployment-naam>
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Replace <hpa-naam>, <namespace> and <deployment-naam> with the names of your HPA, namespace and deployment. minReplicas and maxReplicas determine the lower and upper limits. averageUtilization is the desired average CPU percentage relative to the CPU request. stabilizationWindowSeconds prevents the HPA from immediately removing pods after a brief drop in load.

Step 2

Check that every container used for CPU autoscaling has a CPU request. For example, a container configuration may contain:

resources:
  requests:
    cpu: 200m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

The HPA uses the CPU request as the basis for the percentage. A value of 200m represents 0.2 CPU core.

Step 3

Apply the HPA configuration and check its status:

kubectl apply -f hpa.yaml
kubectl get hpa --namespace <namespace>
kubectl describe hpa <hpa-naam> --namespace <namespace>

Replace <namespace> and <hpa-naam> with the values from your manifest. Use the conditions and events in the detailed output to check whether the HPA is receiving metrics and is allowed to scale the workload.

Troubleshooting a Horizontal Pod Autoscaler

• TARGETS shows <unknown>: use kubectl top pods to check whether the Metrics API is returning values. Then check whether all relevant containers have a request configured for the metric you are scaling on.
• ScalingActive is False: check the events with kubectl describe hpa. Common causes include missing metrics, an incorrect scaleTargetRef or insufficient permissions.
• The HPA does not scale up: check whether the average load actually rises above the target value. The controller ignores small fluctuations around the target value.
• New pods remain Pending: use kubectl describe pod to check whether the nodes have sufficient CPU and memory available. An HPA does not create new nodes.
• The HPA does not scale down immediately: this is usually caused by the scale-down stabilisation period. The default value is 300 seconds.

Use the following commands for the most important checks:

kubectl top pods --namespace <namespace>
kubectl describe hpa <hpa-naam> --namespace <namespace>
kubectl get events --namespace <namespace> --sort-by=.metadata.creationTimestamp

Replace <namespace> and <hpa-naam> with the namespace and name of your HPA. The --sort-by option sorts the events by creation time.

Removing the test resources

Delete the namespace when you have finished testing. This also removes the example deployment, Service, HPA and temporary pods:

kubectl delete namespace hpa-tutorial

Metrics Server is a cluster-wide component that may also be used by other workloads. Therefore, do not remove this component as part of the test clean-up.

You have created a Kubernetes Horizontal Pod Autoscaler based on CPU usage and tested it under load. For production, use realistic resource requests, appropriate minimum and maximum values and sufficient node capacity, and continue to monitor scaling behaviour and application performance.