#Introduction

I started using Kubernetes professionally back in January 2019. I was the only person responsible for keeping it running in production, and I learned a lot over the years. If you are just getting started with Kubernetes, here are some tips for managing a cluster.

#Start Small and Simple

Why: Overcomplicating your Kubernetes setup too early introduces unnecessary complexity and can cause cascading failures.

Tip: Focus on the basics first. Use Kubernetes’ core features like Deployments, Services, and Ingress before jumping into advanced tools like Helm, Istio or Operators. Once your infrastructure is stable and your team is familiar with Kubernetes management, then consider adding these tools as needed. Try to go as long as possible without adding additional complexity.

#Be Cautious with the aws-auth Config

Why: This is specific to AWS and EKS: Mismanaging the aws-auth config can lock you or your team out of your Kubernetes cluster, especially if manual changes lead to incorrect IAM role mappings.

Update: EKS now supports access entries, which manage cluster access through the EKS API instead of the aws-auth ConfigMap. If you are setting up a new cluster, use those instead and most of this goes away.

Tip: Automate aws-auth management using infrastructure-as-code tools like ArgoCD. I know this is immediately counter to Start Small and Simple, but a broken cluster is not fun to deal with. Automating this process reduces the risk of manual errors and provides a version-controlled way to manage access. The user/role that creates the EKS cluster will always be an admin, so try to use a shared role whenever possible. This way, you’ll always have access to the cluster, even if aws-auth is misconfigured.

#Match Your Resource Requests and Limits

Why: Setting different resource requests and limits (especially for memory) can lead to unexpected pod terminations. Some containers may not perform garbage collection correctly if they see more memory than requested.

Tip: Keep your memory requests and limits the same to ensure garbage collection runs properly within the container. This will prevent Kubernetes from killing your pods unexpectedly due to memory pressure.

#Use PodDisruptionBudgets (PDBs) for Safe Rollouts and Drains

Why: PDBs ensure that a certain number or percentage of replicas are available during voluntary disruptions (e.g., rolling updates, node drains).

Tip: Set PDBs to prevent evictions of critical pods during cluster upgrades or scaling activities. Without it, Kubernetes could kill too many pods, impacting application availability and stability.

#Use Topology Spread Constraints

Why: Spreading pods across failure domains like nodes and zones means one failure doesn’t take out every replica.

Tip: Use topology spread constraints to spread pods across nodes and availability zones. Here’s an example of spreading pods across both nodes and AWS availability zones:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: "kubernetes.io/hostname"  # Spread across nodes
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: "topology.kubernetes.io/zone"  # Spread across AWS availability zones
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app

That way no single node or availability zone ends up with all of your pods, and losing one doesn’t take the app down.

#Use Liveness, Readiness, and Startup Probes Properly

Why: Misconfigured probes can cause cascading failures and application downtime.

Tip: Kubernetes provides three types of probes—liveness, readiness, and startup—each serving a different purpose.

• Liveness Probe: Ensures that your application is still running. If it fails, Kubernetes will restart the pod.
• Readiness Probe: Ensures that your application is ready to receive traffic. Kubernetes will only send traffic to pods that pass the readiness check.
• Startup Probe: Used when your application has a long startup time. It prevents Kubernetes from killing the container prematurely during initialization.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

startupProbe:
  httpGet:
    path: /start
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

Use startup probes for slow-starting applications, which can prevent unnecessary restarts while the app is still initializing. Readiness probes should be used to ensure that traffic is only routed to fully functional pods, while liveness probes can be more aggressive but should be tuned carefully to avoid killing slow but healthy pods.

#Limit NodePort Usage

Why: NodePort services expose specific ports on all nodes in the cluster, which can lead to security risks and inefficient use of resources.

Tip: Use LoadBalancer services or Ingress for external traffic handling. Limit NodePort usage to development environments or internal traffic routing.

#Limit Secret Exposure with RBAC and Cloud Secret Managers

Why: Secrets in Kubernetes are only base64-encoded, which is not secure. Exposing secrets to unauthorized users or applications can lead to security breaches.

Tip: Instead of storing sensitive secrets directly in Kubernetes, use tools like HashiCorp Vault, AWS Secrets Manager, or another cloud-based secret manager. They handle encryption and access to secrets much better than Kubernetes does on its own.

Also use Role-Based Access Control (RBAC) in Kubernetes to limit who can access secrets, and always enable secret encryption at rest for an extra layer of security.

#Use Horizontal Pod Autoscaling with Custom Metrics

Why: Default autoscaling is based on CPU or memory utilization, which may not be the best indicator of your app’s performance. Especially for workloads based on queues like RabbitMQ.

Tip: Use custom metrics for Horizontal Pod Autoscaling (HPA) to scale based on more relevant indicators, such as queue length, request latency, or other custom metrics. Datadog provides an easy way to define and use custom metrics.

Example: Datadog has great documentation on how to achieve this

Scaling on custom metrics means you scale on what actually matters for your app, like how backed up a queue is, instead of CPU or memory.