Kubernetes Horizontal Pod Autoscaling: Configure, Tune, and Troubleshoot HPA

Overview

HPA is a Kubernetes control loop. It periodically reads one or more metrics, compares the current value with a target, and changes the replica count between configured minimum and maximum values. It works with scalable workload resources such as Deployments and StatefulSets through the resource's scale subresource.

For a basic CPU target, HPA estimates the desired replica count using the relationship between current utilization and the configured target. If average CPU utilization is above the target, the controller generally requests more replicas; if it is below the target, it may request fewer. The actual decision is affected by metric availability, rounding, stabilization, tolerance, configured behavior policies, and the time required for new pods to become ready.

HPA is not a substitute for capacity planning. It can add pods only when the cluster has enough schedulable capacity, and it cannot make an application handle load efficiently if requests are blocked, dependencies are saturated, or startup is slow. Pair HPA with appropriate resource requests and limits, workload health checks, cluster capacity management, and application-level observability. The Kubernetes resource requests and limits guide is a useful companion for establishing those baselines.

Before adopting a configuration, check the Kubernetes version and the feature support of the metrics pipeline in your environment. The autoscaling/v2 API is the usual choice for multiple metrics and configurable scaling behavior, but exact behavior can vary by release and installed components.

What to track

Configuration and prerequisites

Track these values for every HPA-managed workload:

• Target reference: Confirm that the HPA points to the intended Deployment or StatefulSet and that the target exposes a usable scale subresource.
• Replica bounds: Record minReplicas, maxReplicas, and the reason for each limit. A low maximum can hide capacity problems; an unnecessarily high maximum can increase dependency load and cost.
• Metric source: Identify whether the HPA uses resource metrics, custom metrics, or external metrics. Resource metrics commonly come from Metrics Server, while custom and external metrics require a compatible adapter and provider.
• Resource requests: CPU utilization targets are calculated relative to CPU requests. If requests are missing or poorly sized, a CPU-based HPA can produce misleading decisions. Memory utilization has the same relationship to memory requests when configured as a resource metric.
• Readiness and startup: Track image pull time, container startup time, readiness probe behavior, and time from scheduling to serving traffic. These determine how quickly added replicas contribute capacity.

HPA configuration reference

Baseline manifest

This example uses CPU and memory resource metrics. It assumes the workload has matching resource requests and that the cluster's resource metrics pipeline is working.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 75

Multiple metrics are evaluated when deciding the desired replica count. Treat memory carefully: unlike CPU, memory is not always released promptly, and a rising memory signal may indicate a leak, cache growth, or a workload that should be restarted or redesigned rather than simply replicated.

Custom metrics

Application metrics are often more useful than CPU when demand is defined by queue depth, requests per second, active sessions, or work waiting to be processed. For example, a queue worker could scale from a metric representing messages per pod. The metric adapter must expose the metric in a form HPA can query, and the target must be defined consistently with the metric's units and scope.

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "40"

Document the metric's source, aggregation window, labels, unit, and failure behavior. A metric that disappears during an exporter outage should not be treated as zero without careful consideration.

Cadence and checkpoints

Use a repeatable review schedule rather than tuning HPA only during an incident. A monthly or quarterly review is appropriate for stable services; review sooner after a traffic change, major release, dependency change, or cluster capacity adjustment.

Before enabling or changing HPA

1. Confirm the workload has realistic CPU and memory requests.
2. Check that Metrics Server or the required custom-metrics adapter returns data for the target pods.
3. Define the application's useful scaling signal and its acceptable range.
4. Set minimum and maximum replicas from availability, capacity, and dependency constraints.
5. Verify readiness probes prevent new pods from receiving traffic before they are ready.
6. Check that PodDisruptionBudgets, topology rules, quotas, and node capacity do not prevent scheduling.

During each review

Record the following in a sizing and tuning worksheet:

• Observed peak and typical metric values.
• Replica count at peak, during low traffic, and during recovery.
• Time from the first scaling signal to a ready pod.
• How often the HPA reached its minimum or maximum.
• Periods of pending pods, failed scheduling, throttling, elevated latency, or error rates.
• Whether scale-down removed capacity before traffic was reliably low.
• Changes in request volume, payload size, dependency latency, and deployment behavior.

Validate changes with a controlled test where possible. Generate a representative request or queue workload, observe the metric and replica timeline, then continue long enough to see recovery and scale-down. A short burst may test only startup behavior; it may not reveal oscillation or a maximum-replica ceiling.

How to interpret changes

Start with the HPA status rather than changing thresholds immediately. Inspect the object and its events with commands such as kubectl describe hpa checkout-api -n production and kubectl get hpa checkout-api -n production -o yaml. Compare those results with pod status, workload events, metrics, and application latency.

Common symptoms and likely causes

Scaling too slowly is often a time-to-capacity problem, not simply a target problem. Break the timeline into signal collection, HPA decision, scheduling, image pull, container startup, readiness, and traffic arrival. Improving the slowest stage may be safer than lowering the CPU target. Conversely, scaling too early can happen when requests are over-sized, a metric includes background work, or a custom metric has an unsuitable aggregation window.

Use HPA alongside observability tools that correlate metrics, logs, traces, and incidents. The observability stack guide provides broader guidance for connecting those signals. For deployment-related changes, use the Kubernetes deployment checklist and validate that a new image or probe has not altered startup and readiness behavior.

When to revisit

Revisit this HPA configuration on a monthly or quarterly cadence, and immediately after any event that changes the workload's scaling assumptions. Useful triggers include a sustained traffic pattern change, a new endpoint or background job, a change in request or limit sizing, a new container image, repeated scaling alerts, a dependency migration, a change in node pools, or a cluster capacity rebalance.

At each review, compare the worksheet with the previous period. Ask whether the selected metric still represents user demand, whether the target is reached without harming latency or error rates, whether replicas become ready before they are needed, and whether maximum capacity remains realistic. Record the reason for each adjustment in version control, including the expected result and the observation window.

For a practical validation cycle, first capture a baseline, then apply one change at a time, run representative load, and monitor both scaling and service-level outcomes. Keep rollback instructions for the HPA manifest and alert on conditions such as unavailable metrics, prolonged maximum replicas, pending pods, and latency that rises while replica count increases. HPA should be treated as an operational control that is measured and reviewed—not a one-time YAML setting.