⬡ Hub
Skip to content

Kustomize Interview Questions and Answers (Enhanced with Practical Examples)

1. What is Kustomize, and how does it fundamentally differ from other Kubernetes configuration management tools like Helm?

Answer:

Kustomize is a template-free tool for customizing Kubernetes configurations. It allows you to manage your Kubernetes applications by layering environment-specific configurations on top of a common base configuration. Instead of templating, Kustomize uses a declarative approach where you define your customizations in a kustomization.yaml file, which then merges and patches standard Kubernetes YAML files.

The fundamental difference between Kustomize and Helm is that:

  • Kustomize is template-free and declarative: It works directly with raw Kubernetes YAML manifests. You define how to modify existing YAMLs (e.g., add a label, change an image, apply a patch) rather than generating YAML from templates. It's a "build-time" tool that outputs plain YAML.
  • Helm is a templating engine and package manager: It uses Go templating to generate Kubernetes YAML manifests from .yaml files with placeholders. Helm charts are packages that can be versioned, shared, and deployed.

Key Distinction: * Kustomize: "I have a set of YAMLs, and I want to apply these specific changes to them." (Focus on modification) * Helm: "I have a template, and I want to fill in these values to generate my YAMLs." (Focus on generation)

2. Explain the core components of Kustomize: bases, overlays, and kustomization.yaml files.

Answer:

Kustomize organizes configurations around three core concepts:

  • bases:

    • A base is a directory containing a kustomization.yaml file and a set of standard Kubernetes resource YAML files (e.g., deployment.yaml, service.yaml).
    • It represents the common, shared configuration for an application or component that will be deployed across multiple environments.
    • Bases are typically kept clean and generic, without environment-specific details.

  • overlays:

    • An overlay is a directory that contains its own kustomization.yaml file and a set of customizations (patches, generators, transformers) that are specific to a particular environment (e.g., development, staging, production).
    • An overlay's kustomization.yaml file explicitly references one or more bases and then applies its unique modifications on top of them.
    • Overlays allow for environment-specific variations without altering the base configuration.

  • kustomization.yaml:

    • This is the central configuration file for Kustomize. Every directory that Kustomize operates on must contain a kustomization.yaml file.
    • It declares:
      • The resources (raw YAML files) to include.
      • The bases it depends on (if it's an overlay).
      • Generators (e.g., secretGenerator, configMapGenerator) to create resources.
      • Transformers (e.g., namePrefix, nameSuffix, commonLabels, commonAnnotations) to modify resources.
      • Patches (e.g., patchesStrategicMerge, patchesJson6902) to apply specific changes.

Example Directory Structure:

├── base
│   ├── deployment.yaml       # Generic Deployment
│   ├── service.yaml          # Generic Service
│   └── kustomization.yaml    # Defines base resources
└── overlays
    ├── development
    │   ├── kustomization.yaml  # References base, applies dev-specific patches
    │   └── dev-patch.yaml      # Patch for dev environment
    └── production
        ├── kustomization.yaml  # References base, applies prod-specific patches
        └── prod-patch.yaml     # Patch for prod environment

3. Describe the concept of "patches" in Kustomize. What are the different types of patches, and when would you use each?

Answer:

Patches are a core mechanism in Kustomize used to modify the base Kubernetes resources without directly editing the original files. They allow you to apply environment-specific changes, add new fields, or update existing ones.

There are two main types of patches in Kustomize:

  1. Strategic Merge Patch (SMP):

    • Mechanism: This is the default and most commonly used patching mechanism. It merges the patch with the base resource based on a set of Kubernetes-specific merging rules. It's a "smart" merge that understands Kubernetes YAML structure.
    • When to Use: Ideal for making simple, additive, or modifying changes like:
      • Updating a container image.
      • Changing a replica count.
      • Adding a new label or annotation.
      • Adding a new container to a deployment.
    • Example (deployment.yaml in base): yaml # base/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app labels: app: my-app spec: replicas: 1 template: metadata: labels: app: my-app spec: containers: - name: my-app-container image: my-app:1.0.0 ports: - containerPort: 80
    • Example (prod-patch.yaml in overlay): yaml # overlays/production/prod-patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app # Must match the name of the resource in the base spec: replicas: 3 # Change replica count template: spec: containers: - name: my-app-container image: my-app:1.1.0 # Update image version env: # Add new environment variable - name: ENV_TYPE value: production
    • Kustomization (kustomization.yaml in overlay): ```yaml # overlays/production/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases:
      • ../../base patchesStrategicMerge:
      • prod-patch.yaml ```

  2. JSON Patch (RFC 6902):

    • Mechanism: This uses the JSON Patch standard to apply a sequence of operations (add, remove, replace, move, copy, test) to a resource. It's more explicit and powerful for complex, structural changes.
    • When to Use: Useful for making more complex or precise changes that SMP might not handle well, such as:
      • Removing a specific element from a list.
      • Inserting an element at a specific index in a list.
      • Renaming a field.
    • Example (deployment.yaml in base - same as above):
    • Example (dev-json-patch.yaml in overlay to remove a port): ```yaml # overlays/development/dev-json-patch.yaml
      • op: remove path: /spec/template/spec/containers/0/ports/0 # Path to the containerPort: 80 ```
    • Kustomization (kustomization.yaml in overlay): ```yaml # overlays/development/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases:
      • ../../base patchesJson6902:
      • target: group: apps version: v1 kind: Deployment name: my-app path: dev-json-patch.yaml ```

Choosing between them: Prefer Strategic Merge Patch for most common modifications due to its simplicity and Kubernetes-native understanding. Use JSON Patch when SMP is insufficient for highly specific or structural changes.

4. As a Solutions Architect, how would you structure a Kustomize project to manage configurations for multiple environments (e.g., development, staging, production) in a scalable and maintainable way?

Answer:

For a scalable and maintainable Kustomize project across multiple environments, I would adopt a clear, hierarchical directory structure that separates common configurations from environment-specific customizations. This approach promotes reusability, reduces duplication, and simplifies management.

Recommended Directory Structure:

my-k8s-config/
├── base/
│   ├── my-app/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   ├── ingress.yaml
│   │   └── kustomization.yaml  # Defines common resources for 'my-app'
│   ├── another-app/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── kustomization.yaml  # Defines common resources for 'another-app'
│   └── kustomization.yaml      # Optional: Aggregates multiple bases if needed
└── overlays/
    ├── development/
    │   ├── my-app/
    │   │   ├── kustomization.yaml  # References base/my-app, applies dev-specific patches
    │   │   └── dev-replicas-patch.yaml
    │   │   └── dev-env-vars.yaml
    │   ├── another-app/
    │   │   ├── kustomization.yaml
    │   │   └── dev-resource-limits.yaml
    │   └── kustomization.yaml      # Optional: Aggregates dev overlays for all apps
    ├── staging/
    │   ├── my-app/
    │   │   ├── kustomization.yaml  # References base/my-app, applies staging-specific patches
    │   │   └── staging-ingress-patch.yaml
    │   └── another-app/
    │       └── kustomization.yaml
    └── production/
        ├── my-app/
        │   ├── kustomization.yaml  # References base/my-app, applies prod-specific patches
        │   ├── prod-hpa.yaml       # Horizontal Pod Autoscaler for prod
        │   └── prod-secrets.yaml   # Secret generator for prod
        └── another-app/
            └── kustomization.yaml

Explanation:

  1. base/ Directory:

    • Contains the most generic, common Kubernetes manifests for each application or component (e.g., my-app, another-app).
    • Each application within base/ has its own kustomization.yaml listing its core resources.
    • This ensures that all environments start from the same foundational configuration.

  2. overlays/ Directory:

    • Contains subdirectories for each environment (development, staging, production).
    • Within each environment subdirectory, there's a further subdirectory for each application (e.g., overlays/development/my-app).
    • Each application's overlay kustomization.yaml references its corresponding base (e.g., ../../base/my-app) and then applies environment-specific customizations.

Example kustomization.yaml for overlays/production/my-app:

# my-k8s-config/overlays/production/my-app/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference the base configuration for 'my-app'
bases:
- ../../../base/my-app

# Apply environment-specific patches
patchesStrategicMerge:
- prod-replicas-patch.yaml # Increase replicas for production
- prod-resource-limits.yaml # Set higher resource limits

# Add production-specific resources (e.g., HPA)
resources:
- prod-hpa.yaml

# Generate secrets/configmaps for production
secretGenerator:
- name: my-app-prod-secret
  literals:
  - API_KEY=prod-api-key-123
  - DB_PASSWORD=prod-db-password-xyz

# Apply common labels/annotations for production
commonLabels:
  env: production
  app.kubernetes.io/instance: my-app-prod

namePrefix: prod- # Prefix all resource names with 'prod-'

Benefits of this Structure:

  • Reusability: The base configurations are reused across all environments.
  • Maintainability: Changes to the core application logic only need to be made in the base. Environment-specific changes are isolated to their respective overlays.
  • Scalability: Easily add new environments or applications by creating new overlay directories.
  • Clarity: The structure clearly shows what is common and what is environment-specific.
  • GitOps Friendly: This structure integrates seamlessly with GitOps tools like Argo CD or Flux CD, where each overlay can represent a deployable unit for a specific environment.

5. How does Kustomize facilitate managing environment-specific configurations for sensitive data like secrets and ConfigMaps?

Answer:

Kustomize provides built-in generators, secretGenerator and configMapGenerator, to manage sensitive data (Secrets) and non-sensitive configuration (ConfigMaps) in an environment-specific manner. This allows you to keep your sensitive data and configuration separate from your main Kubernetes manifests and inject them securely at build time.

Key Features:

  1. secretGenerator:

    • Purpose: Creates Kubernetes Secret resources from literals, files, or environment variables.
    • Mechanism: Kustomize generates a Secret with a unique name (often including a hash of its content for immutability and automatic rollout on change).
    • Benefit: You don't commit raw secret values to your Git repository. Instead, you commit references to files or use literals that are processed by Kustomize. The generated Secret name changes when its content changes, triggering a rollout of dependent Deployments.

    Example: * overlays/production/kustomization.yaml: ```yaml # overlays/production/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base/my-app

    secretGenerator:
- name: my-app-db-credentials
  files:
  - db_username.txt # File containing username
  - db_password.txt # File containing password
  type: Opaque
- name: my-app-api-key
  literals:
  - API_KEY=prod-api-key-xyz-123
```
    • overlays/production/db_username.txt: produser
    • overlays/production/db_password.txt: supersecretprodpassword
    • When kustomize build overlays/production is run, it will generate Secret YAMLs with base64 encoded values.

  2. configMapGenerator:

    • Purpose: Creates Kubernetes ConfigMap resources from literals, files, or environment variables.
    • Mechanism: Similar to secretGenerator, it generates a ConfigMap with a unique name (including a content hash).
    • Benefit: Centralizes non-sensitive configuration for an environment, allowing easy updates and versioning.

    Example: * overlays/development/kustomization.yaml: ```yaml # overlays/development/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base/my-app

    configMapGenerator:
- name: my-app-config
  literals:
  - LOG_LEVEL=DEBUG
  - FEATURE_TOGGLE_A=true
  files:
  - application.properties # File containing properties
```
    • overlays/development/application.properties: properties db.url=jdbc:h2:mem:testdb server.port=8080

How it Facilitates Management:

  • Separation of Concerns: Keeps sensitive data out of the main application manifests.
  • Version Control: The kustomization.yaml and the source files for secrets/config maps can be version-controlled.
  • Automatic Rollout: When the content of a generated Secret or ConfigMap changes, its name (due to the hash suffix) also changes. This automatically triggers a rollout of any Deployments that reference it, ensuring applications pick up the new configuration.
  • Environment-Specific Values: Each overlay can have its own secretGenerator and configMapGenerator definitions, providing distinct values for different environments.
  • Security: While secretGenerator doesn't encrypt secrets at rest in Git, it prevents accidental exposure of plaintext values in YAML manifests and ensures that secrets are managed distinctly per environment. For true encryption at rest in Git, tools like sops are often used in conjunction with Kustomize.

6. Discuss how Kustomize helps in achieving consistency across different environments while still allowing for necessary variations. Provide an example.

Answer:

Kustomize achieves a powerful balance between consistency and variation by leveraging its base and overlay structure.

  • Consistency through base:

    • The base directory contains the core, common Kubernetes manifests that define the fundamental structure and behavior of an application. This base is shared and referenced by all environment-specific overlays.
    • Any changes made to the base automatically propagate to all environments, ensuring that the core application configuration remains consistent. This is crucial for maintaining architectural standards and reducing configuration drift.

  • Variation through overlays and patches:

    • Each overlay (e.g., development, staging, production) references the base and then applies its unique set of customizations. These customizations can include:
      • Patches: Modifying replica counts, container images, resource limits, adding environment variables.
      • Generators: Creating environment-specific ConfigMaps or Secrets.
      • Transformers: Applying namePrefix, nameSuffix, commonLabels, commonAnnotations.
    • These variations are applied non-destructively on top of the base, allowing each environment to have its specific settings without forking the entire configuration.

Example Scenario:

Let's consider a simple web application (my-webapp) that needs to be deployed to development and production environments.

1. base/my-webapp/kustomization.yaml and deployment.yaml:

# base/my-webapp/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webapp
  labels:
    app: my-webapp
spec:
  replicas: 1 # Default replica count
  selector:
    matchLabels:
      app: my-webapp
  template:
    metadata:
      labels:
        app: my-webapp
    spec:
      containers:
      - name: my-webapp-container
        image: my-webapp:latest # Default image
        ports:
        - containerPort: 80
        env:
        - name: LOG_LEVEL
          value: INFO
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
---
# base/my-webapp/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-webapp
spec:
  selector:
    app: my-webapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

# base/my-webapp/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
commonLabels:
  app.kubernetes.io/part-of: my-application

2. overlays/development/my-webapp/kustomization.yaml and dev-patch.yaml:

# overlays/development/my-webapp/dev-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webapp
spec:
  replicas: 1 # Ensure 1 replica for dev
  template:
    spec:
      containers:
      - name: my-webapp-container
        image: my-webapp:dev-build # Use dev-specific image
        env:
        - name: LOG_LEVEL
          value: DEBUG # Debug logging for dev
        resources:
          limits:
            cpu: "100m" # Lower limits for dev
            memory: "128Mi"

# overlays/development/my-webapp/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../../base/my-webapp
patchesStrategicMerge:
- dev-patch.yaml
namePrefix: dev- # Prefix all resource names with 'dev-'
commonLabels:
  env: development

3. overlays/production/my-webapp/kustomization.yaml and prod-patch.yaml:

# overlays/production/my-webapp/prod-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webapp
spec:
  replicas: 3 # Scale up for production
  template:
    spec:
      containers:
      - name: my-webapp-container
        image: my-webapp:1.2.0 # Use stable production image
        env:
        - name: LOG_LEVEL
          value: ERROR # Error logging for prod
        resources:
          requests:
            cpu: "200m" # Higher requests for prod
            memory: "256Mi"
          limits:
            cpu: "500m" # Higher limits for prod
            memory: "512Mi"

# overlays/production/my-webapp/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../../base/my-webapp
patchesStrategicMerge:
- prod-patch.yaml
namePrefix: prod- # Prefix all resource names with 'prod-'
commonLabels:
  env: production

Result: * Both development and production deployments are based on the same base/my-webapp configuration (consistency). * Each environment gets its specific replica count, image, log level, resource limits, and name prefix (variation). * If a new port needs to be added to my-webapp, it's changed once in base/my-webapp/deployment.yaml, and both environments inherit it.

7. Explain how you would use Kustomize to inject a sidecar container (e.g., a service mesh proxy) into all deployments within a specific overlay without modifying the original deployment manifests.

Answer:

Injecting a sidecar container into existing deployments using Kustomize is a common and powerful use case, especially for service mesh integration (like Istio or Linkerd) or adding logging/monitoring agents. This can be achieved using a Strategic Merge Patch.

Scenario: We want to add a monitoring-agent sidecar container to my-app deployment in the production environment.

1. Base Deployment (base/my-app/deployment.yaml): (Same as the deployment.yaml example in Q6)

2. Sidecar Patch (overlays/production/my-app/sidecar-patch.yaml):

# overlays/production/my-app/sidecar-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webapp # Must match the name of the resource in the base
spec:
  template:
    spec:
      containers:
      - name: my-webapp-container # Reference the existing container to ensure correct merging
      - name: monitoring-agent # This is the new sidecar container
        image: my-monitoring-agent:1.0.0
        ports:
        - containerPort: 9090
        env:
        - name: APP_NAME
          value: my-webapp
        resources:
          limits:
            cpu: "50m"
            memory: "64Mi"
  • Explanation: The patch targets the my-webapp Deployment. Within spec.template.spec.containers, it defines a new container named monitoring-agent. Kustomize's Strategic Merge Patch rules understand that if a container with a new name is added to the containers list, it should be appended. The existing my-webapp-container is included to ensure the patch correctly targets the Deployment and doesn't accidentally replace the entire containers list (though in this case, just adding the new container would also work due to SMP rules for lists of objects with keys).

3. Kustomization File (overlays/production/my-app/kustomization.yaml):

# overlays/production/my-app/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../../base/my-app
patchesStrategicMerge:
- prod-patch.yaml # Existing production patch (e.g., for replicas, image)
- sidecar-patch.yaml # Inject the new sidecar
namePrefix: prod-
commonLabels:
  env: production

Result: When kustomize build overlays/production/my-app is executed, the output YAML for the prod-my-webapp Deployment will include both the original my-webapp-container and the newly injected monitoring-agent sidecar container, along with all other production-specific customizations. The original base/my-app/deployment.yaml remains untouched.

8. Describe a scenario where you would use namePrefix or nameSuffix in Kustomize, and what problem it solves.

Answer:

namePrefix and nameSuffix are powerful Kustomize transformers used to modify the names of all resources generated by a kustomization.yaml file. They are particularly useful in scenarios where you need to deploy the same application multiple times within the same Kubernetes namespace or across different namespaces, without naming conflicts.

Problem Solved: Kubernetes resource names must be unique within a namespace for a given resource type. If you try to deploy the same Deployment named my-app twice in the same namespace, the second deployment will fail. namePrefix and nameSuffix solve this by automatically prepending or appending a string to the name of every resource, making them unique.

Scenario: Deploying multiple instances of the same application (e.g., my-app) for different tenants or feature branches within the same Kubernetes cluster, possibly even the same namespace.

Example:

Let's say we have a base/my-app configuration. We want to deploy two instances: one for tenant-a and one for tenant-b.

1. Base Configuration (base/my-app/deployment.yaml): (Same as the deployment.yaml example in Q6, with name: my-webapp)

2. Overlay for tenant-a (overlays/tenant-a/kustomization.yaml):

# overlays/tenant-a/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base/my-app # Reference the common base

namePrefix: tenant-a- # All resources will be prefixed with 'tenant-a-'
commonLabels:
  tenant: tenant-a

3. Overlay for tenant-b (overlays/tenant-b/kustomization.yaml):

# overlays/tenant-b/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base/my-app # Reference the common base

namePrefix: tenant-b- # All resources will be prefixed with 'tenant-b-'
commonLabels:
  tenant: tenant-b

Result:

  • kustomize build overlays/tenant-a would generate a Deployment named tenant-a-my-webapp and a Service named tenant-a-my-webapp.
  • kustomize build overlays/tenant-b would generate a Deployment named tenant-b-my-webapp and a Service named tenant-b-my-webapp.

Both instances can now coexist in the same namespace without naming conflicts. This is incredibly useful for:

  • Multi-tenant deployments: Each tenant gets their own isolated set of resources.
  • Feature branch deployments: Deploying multiple versions of an application for different feature branches for testing.
  • Environment differentiation: As shown in Q6, using prod- or dev- prefixes to clearly distinguish resources by environment, even if they are in different namespaces.

nameSuffix works similarly, appending the string to the resource names. You can use both namePrefix and nameSuffix together if needed.

9. How can Kustomize be integrated into a GitOps workflow, particularly with tools like Argo CD or Flux CD?

Answer:

Kustomize is an ideal fit for GitOps workflows because it aligns perfectly with the core principles: declarative configuration, version control (Git as the single source of truth), and automated reconciliation. Tools like Argo CD and Flux CD natively understand and integrate with Kustomize.

Integration Mechanism:

  1. Git Repository as Source of Truth:

    • The entire Kustomize project structure (bases, overlays, kustomization.yaml files, patches, etc.) is stored in a Git repository. This repository becomes the single source of truth for the desired state of the Kubernetes cluster.

  2. GitOps Controller (Argo CD / Flux CD):

    • Argo CD / Flux CD are deployed within the Kubernetes cluster.
    • They are configured to monitor specific paths within the Git repository (e.g., my-k8s-config/overlays/production/my-app).
    • When a change is pushed to the Git repository, the GitOps controller detects it.

  3. Kustomize Build and Apply:

    • The GitOps controller internally performs a kustomize build operation on the specified overlay path. This generates the final, flattened Kubernetes YAML manifests for that environment.
    • The controller then compares these generated manifests with the current state of the resources in the Kubernetes cluster.
    • If there's a divergence, the controller automatically applies the necessary changes to the cluster to bring it into the desired state (reconciliation).

Example Integration with Argo CD:

Let's assume the Kustomize project from Q4 is in a Git repository at https://github.com/my-org/my-k8s-config.git.

Argo CD Application Manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/my-k8s-config.git
    targetRevision: HEAD
    path: overlays/production/my-app # Argo CD points directly to the Kustomize overlay
    kustomize: {} # This tells Argo CD to use Kustomize for this path
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app-prod # Deploy to 'my-app-prod' namespace
  syncPolicy:
    automated: # Enable automatic synchronization
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true # Create the namespace if it doesn't exist

Workflow:

  1. A developer makes a change to overlays/production/my-app/prod-patch.yaml (e.g., updates the image version) and pushes it to main branch.
  2. Argo CD detects the change in the Git repository.
  3. Argo CD runs kustomize build overlays/production/my-app internally.
  4. Argo CD compares the generated YAML with the live state in the my-app-prod namespace.
  5. Argo CD detects the image version difference in the prod-my-webapp Deployment.
  6. Argo CD automatically applies the change to the Kubernetes cluster, initiating a rollout of the Deployment.

Benefits in a GitOps Context:

  • Single Source of Truth: Git repository is the definitive record of the desired state.
  • Auditability: Every change to the cluster state is a Git commit, providing a clear audit trail.
  • Rollback: Rolling back to a previous cluster state is as simple as reverting a Git commit.
  • Self-Healing: The GitOps controller continuously reconciles the cluster state with Git, automatically correcting any drift.
  • Developer Experience: Developers interact with Git, not directly with kubectl apply.

10. You have a large microservices architecture with many teams. How would you recommend using Kustomize to manage the Kubernetes configurations for these services, ensuring team autonomy while maintaining overall consistency?

Answer:

Managing Kubernetes configurations for a large microservices architecture with multiple teams requires a strategy that balances team autonomy with organizational consistency and governance. Kustomize is well-suited for this by promoting a federated approach.

Recommended Structure and Strategy:

  1. Centralized base for Common Infrastructure/Standards:

    • Location: A dedicated, centrally managed Git repository (e.g., platform-k8s-bases).
    • Content: This repository would contain:
      • Organizational Standards: Common labels, annotations, network policies, resource quotas.
      • Shared Infrastructure: Base configurations for ingress controllers, service meshes, logging agents, monitoring agents (which teams might then patch into their apps).
      • Security Baselines: Pod Security Standards (PSS) enforcement, default RBAC roles.
    • Management: Managed by a central platform or SRE team. Teams consume these bases but cannot directly modify them.

  2. Team-Owned Application bases:

    • Location: Each microservice team maintains its own Git repository for their application's Kubernetes manifests (e.g., team-a/user-service-k8s, team-b/product-service-k8s).
    • Content: This repository contains the generic, environment-agnostic Kubernetes manifests for their specific microservice (Deployment, Service, ConfigMaps, etc.). This is the "base" for their application.
    • Autonomy: Teams have full control over their application's base configuration, allowing them to iterate quickly on their service.

  3. Centralized overlays for Environments:

    • Location: Another dedicated, centrally managed Git repository (e.g., env-k8s-overlays).
    • Content: This repository contains the environment-specific overlays (development, staging, production).
    • Mechanism: Each overlay's kustomization.yaml would:
      • Reference the centralized platform bases (from step 1) to inherit organizational standards.
      • Reference the team-owned application bases (from step 2) for each microservice.
      • Apply environment-specific patches (e.g., replica counts, image tags, resource limits, environment variables, secretGenerator for prod secrets).
      • Apply common transformers like namePrefix (e.g., dev-, prod-) and commonLabels (e.g., env: development).
    • Consistency: This layer ensures that all applications conform to environment-specific requirements and standards. Managed by the platform/SRE team, with input from application teams for specific environment variables or resource needs.

Example Structure:

# Git Repo 1: platform-k8s-bases (Managed by Platform Team)
├── common-labels/
│   └── kustomization.yaml
├── network-policies/
│   └── kustomization.yaml
└── service-mesh-injection/
    └── kustomization.yaml # Patch to inject sidecar

# Git Repo 2: team-a/user-service-k8s (Managed by Team A)
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml

# Git Repo 3: env-k8s-overlays (Managed by Platform/SRE Team)
├── development/
│   ├── user-service/
│   │   ├── kustomization.yaml # References team-a/user-service-k8s/base, platform-k8s-bases
│   │   └── dev-patch.yaml
│   ├── product-service/
│   │   └── kustomization.yaml # References team-b/product-service-k8s/base, platform-k8s-bases
│   └── kustomization.yaml # Aggregates all dev services
├── production/
│   ├── user-service/
│   │   ├── kustomization.yaml # References team-a/user-service-k8s/base, platform-k8s-bases
│   │   └── prod-patch.yaml
│   │   └── prod-secrets.yaml
│   ├── product-service/
│   │   └── kustomization.yaml
│   └── kustomization.yaml # Aggregates all prod services

Ensuring Autonomy and Consistency:

  • Autonomy: Teams own their application's base manifests. They can make changes to their service's core definition without needing central approval for every line of YAML.
  • Consistency: The central platform-k8s-bases and env-k8s-overlays repositories enforce organizational standards, security policies, and environment-specific configurations.
  • Clear Responsibilities: Clear ownership boundaries for different parts of the configuration.
  • GitOps Integration: Argo CD or Flux CD would monitor the env-k8s-overlays repository, building and applying the appropriate environment overlays.

This federated Kustomize approach allows large organizations to scale their Kubernetes deployments effectively, empowering teams while maintaining necessary control and consistency.

11. What are some potential challenges or limitations you've encountered or foresee with Kustomize in a large-scale enterprise environment, and how would you mitigate them?

Answer:

While Kustomize is powerful, especially in large enterprises, it comes with its own set of challenges.

1. Learning Curve for Newcomers: * Challenge: Developers new to Kubernetes or Kustomize might find the "patching" and "merging" concepts less intuitive than traditional templating. Understanding Strategic Merge Patch rules can be tricky. * Mitigation: * Comprehensive Documentation: Provide clear, internal documentation with examples specific to the organization's Kustomize patterns. * Training: Offer workshops or training sessions for new team members. * Standardized Bases/Overlays: Enforce a consistent project structure and naming conventions to reduce cognitive load. * Helper Scripts: Provide simple scripts to run kustomize build for different environments.

2. Verbosity of Patches for Complex Changes: * Challenge: For very complex or deeply nested changes, Strategic Merge Patches can become long and difficult to read. JSON Patches, while powerful, are even more verbose and less human-readable. * Mitigation: * Keep Bases Generic: Design base manifests to be as generic as possible to minimize the need for complex patches. * Break Down Patches: Split large patches into smaller, more focused patch files. * Use commonLabels/commonAnnotations: Leverage Kustomize's built-in transformers for common modifications instead of manual patches. * Consider Custom Kustomize Plugins (Advanced): For highly repetitive, complex transformations, custom plugins can abstract away complexity (though this adds maintenance overhead).

3. Lack of True Templating/Conditional Logic: * Challenge: Kustomize is intentionally template-free. This means it lacks the conditional logic (if/else) or loops (for) that templating engines like Helm provide. This can make certain dynamic configurations difficult. * Mitigation: * Design for Immutability: Embrace the declarative nature. If a resource needs to be fundamentally different, it might warrant a separate base or a more extensive patch. * Leverage generators and transformers: Use secretGenerator, configMapGenerator, namePrefix, commonLabels for common variations. * External Tools for Pre-processing: For extreme cases, a small pre-processing script (e.g., Python, Go) could generate parts of the kustomization.yaml or resource files before Kustomize runs, but this should be a last resort to avoid complexity. * Multiple Overlays: Create distinct overlays for variations that would otherwise require conditional logic.

4. Managing External Bases/Dependencies: * Challenge: When referencing bases from external Git repositories or remote URLs, managing versions and ensuring stability can be tricky. * Mitigation: * Pin Revisions: Always pin external bases to specific Git commits or tags (targetRevision in GitOps tools, or ref in Kustomize git references) to ensure reproducible builds. * Internal Mirroring: For critical external dependencies, consider mirroring them internally to protect against upstream changes or outages. * Automated Updates: Implement automated processes (e.g., Renovate Bot, Dependabot) to propose updates to external bases.

5. Debugging Complex Merges: * Challenge: When multiple patches and transformers are applied, understanding the final merged output can be difficult if something goes wrong. * Mitigation: * kustomize build --output-directory: Generate the final YAML and inspect it. * kustomize build | kubectl diff -f -: Use kubectl diff to see the exact changes that would be applied. * Incremental Builds: Debug by applying patches one by one or building smaller subsets of the configuration. * Clear Patch Naming: Name patch files descriptively (e.g., increase-replicas-patch.yaml, add-env-vars-patch.yaml).

By proactively addressing these challenges with good practices, clear documentation, and strategic use of Kustomize's features, it can be a highly effective tool in large enterprise environments.

12. How would you approach migrating an existing Kubernetes deployment that uses raw YAML files (potentially with some manual modifications per environment) to a Kustomize-based solution?

Answer:

Migrating from raw, potentially duplicated, YAML files to a Kustomize-based solution is a common refactoring task aimed at improving maintainability, consistency, and scalability. I would approach this systematically:

Phase 1: Discovery and Analysis

  1. Inventory Existing Configurations:
    • Identify all current Kubernetes YAML files across all environments (development, staging, production).
    • Note their locations, naming conventions, and how they are currently deployed.
  2. Identify Commonalities:
    • Compare YAML files across environments for the same application.
    • Identify common resources (Deployments, Services, ConfigMaps) and common fields (labels, selectors, container names, ports). This will form the base.
  3. Identify Variations:
    • Pinpoint environment-specific differences:
      • Replica counts, image tags, resource limits.
      • Environment variables, ConfigMap/Secret values.
      • Ingress hostnames, annotations.
      • Existence of certain resources (e.g., HPA only in production).
    • These variations will become the patches and generators in overlays.
  4. Map Dependencies:
    • Understand how different applications or components relate to each other.

Phase 2: Establish Base Configurations

  1. Create base Directories:
    • For each distinct application or component, create a base/<app-name> directory.
    • Copy the most generic version of its Kubernetes manifests (e.g., deployment.yaml, service.yaml) into this directory. Remove all environment-specific values, replacing them with generic defaults or placeholders if necessary.

  2. Create base/kustomization.yaml:

    • In each base/<app-name> directory, create a kustomization.yaml file.
    • List all the resources (deployment.yaml, service.yaml) within this base.
    • Add any common labels or annotations that apply to all environments using commonLabels and commonAnnotations.

    Example: ```yaml

    base/my-app/kustomization.yaml

    apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml commonLabels: app.kubernetes.io/part-of: my-application ```

Phase 3: Create Overlays for Each Environment

  1. Create overlays Directories:
    • For each environment (e.g., development, staging, production), create an overlays/<env-name>/<app-name> directory.
  2. Create overlays/kustomization.yaml:
    • In each environment-specific application directory, create a kustomization.yaml file.
    • Reference the corresponding base using the bases field.
    • Apply environment-specific transformers:
      • namePrefix: Add dev-, stg-, prod- prefixes.
      • commonLabels: Add env: development, env: production.

  3. Create Patches and Generators:

    • For each identified variation, create a separate patch file (e.g., dev-replicas-patch.yaml, prod-image-patch.yaml).
    • Use patchesStrategicMerge for most changes.
    • Use secretGenerator and configMapGenerator for environment-specific secrets and config maps.

    Example: ```yaml

    overlays/production/my-app/kustomization.yaml

    apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../../base/my-app # Relative path to the base patchesStrategicMerge: - prod-replicas-patch.yaml - prod-image-patch.yaml secretGenerator: - name: my-app-prod-secret literals: - API_KEY=prod-api-key namePrefix: prod- commonLabels: env: production ```

Phase 4: Testing and Validation

  1. Build and Inspect:
    • For each overlay, run kustomize build overlays/<env-name>/<app-name> and carefully inspect the generated YAML output.
    • Compare the generated YAML with the original, manually modified YAML for that environment to ensure fidelity.
  2. Dry Run Deployments:
    • Use kustomize build overlays/<env-name>/<app-name> | kubectl diff -f - to see the exact changes that would be applied to the cluster.
    • Perform dry runs: kustomize build overlays/<env-name>/<app-name> | kubectl apply -f - --dry-run=client.
  3. Phased Rollout:
    • Start with a less critical environment (e.g., development).
    • Gradually roll out to staging and then production, closely monitoring for any issues.

Phase 5: Refinement and Documentation

  1. Refactor and Optimize: As you gain experience, refactor bases and overlays to improve reusability and reduce complexity.
  2. Document: Create clear documentation for the new Kustomize structure, including how to make changes, add new resources, and troubleshoot.
  3. Integrate with CI/CD/GitOps: Integrate the Kustomize build step into your CI/CD pipelines or GitOps tools (like Argo CD/Flux CD) for automated deployments.

This systematic approach ensures a smooth transition to Kustomize, leading to a more robust and manageable Kubernetes configuration management system.

13. When would you choose Kustomize over Helm, or vice-versa, for a Kubernetes deployment strategy?

Answer:

The choice between Kustomize and Helm often depends on the specific use case, the nature of the applications being deployed, and the organizational context. Both are excellent tools, but they excel in different scenarios.

Choose Kustomize When:

  1. You are managing in-house applications:
    • Reason: You have full control over the source code and the base Kubernetes manifests. Kustomize is perfect for applying environment-specific tweaks to your own application's YAMLs.
  2. You prefer a template-free, declarative approach:
    • Reason: You want to work directly with standard Kubernetes YAML and avoid templating languages. Kustomize's patching mechanism feels more native to Kubernetes.
  3. You need simple, additive, or modifying customizations:
    • Reason: For common tasks like changing image tags, replica counts, adding labels/annotations, or injecting sidecars, Kustomize's Strategic Merge Patches are very efficient and readable.
  4. You want to avoid the overhead of a package manager:
    • Reason: You don't need the full lifecycle management (install, upgrade, rollback) that Helm charts provide. You just want to generate final YAMLs.
  5. You are integrating with GitOps tools:
    • Reason: Kustomize's declarative nature makes it a natural fit for GitOps. Tools like Argo CD and Flux CD have native Kustomize support.
  6. You need to compose configurations from multiple sources:
    • Reason: Kustomize excels at layering configurations, allowing you to combine a base application config with a base security config, and then apply environment-specific changes.

Choose Helm When:

  1. You are deploying third-party applications:
    • Reason: Many popular open-source applications (e.g., Prometheus, Grafana, Nginx Ingress Controller) provide official Helm charts. These charts are maintained by the community and offer a standardized way to deploy and configure complex applications.
  2. You need a package manager for Kubernetes:
    • Reason: Helm provides a complete packaging solution, including versioning, dependency management between charts, and easy installation, upgrade, and rollback of releases.
  3. You require complex conditional logic or loops in your configurations:
    • Reason: Helm's Go templating engine allows for advanced conditional logic, loops, and functions that are not possible with Kustomize's declarative patching. This is useful for highly dynamic configurations.
  4. You want to abstract away Kubernetes YAML complexity for end-users:
    • Reason: Helm charts can expose a simple values.yaml file, allowing users to customize deployments without needing deep Kubernetes YAML knowledge.
  5. You need to manage application dependencies within Kubernetes:
    • Reason: Helm charts can declare dependencies on other charts, simplifying the deployment of complex application stacks.

Hybrid Approach:

It's also common to use both tools together:

  • Use Helm to deploy third-party applications (e.g., a database, a message queue).
  • Use Kustomize to customize the Helm chart's output (e.g., apply a namePrefix or add a specific label to a resource generated by a Helm chart). This is done by treating the output of helm template as a Kustomize base.

Example Hybrid Workflow:

# 1. Generate raw YAML from a Helm chart
helm template my-release stable/nginx-ingress --output-dir ./nginx-ingress-base

# 2. Create a Kustomize overlay to customize the Helm output
# overlays/production/nginx-ingress/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../../nginx-ingress-base/nginx-ingress # Reference the Helm output
namePrefix: prod-
patchesStrategicMerge:
- prod-ingress-patch.yaml # Add prod-specific annotations to Ingress

This allows you to leverage the packaging power of Helm while retaining the declarative customization capabilities of Kustomize.

14. What are some best practices for organizing Kustomize directories and files to ensure clarity, reusability, and ease of maintenance?

Answer:

Effective organization of Kustomize directories and files is crucial for managing Kubernetes configurations in a clear, reusable, and maintainable way, especially in large projects.

  1. Adopt the base/overlays Structure:

    • Practice: Always separate your common, generic configurations (base) from your environment-specific customizations (overlays).
    • Benefit: Promotes reusability of the base and isolates environment-specific changes, making updates easier.

  2. Granular Bases:

    • Practice: Create a separate base directory for each distinct application or microservice. Avoid monolithic bases.
    • Benefit: Improves modularity, allows teams to manage their own application bases, and makes it easier to compose different applications.

  3. Consistent Overlay Structure:

    • Practice: Within overlays, maintain a consistent structure, typically overlays/<environment>/<app-name>.
    • Benefit: Predictable navigation and understanding of where to find environment-specific configurations for any given application.

  4. Descriptive File Naming:

    • Practice: Use clear, descriptive names for your YAML files and patches (e.g., deployment.yaml, service.yaml, prod-replicas-patch.yaml, dev-env-vars.yaml).
    • Benefit: Improves readability and makes it immediately clear what each file's purpose is.

  5. Small, Focused Patches:

    • Practice: Each patch file should ideally address a single, logical change (e.g., one patch for replica count, another for image tag, another for resource limits).
    • Benefit: Makes patches easier to understand, debug, and reuse. Avoid "god patches" that do too many things.

  6. Leverage Kustomize Transformers:

    • Practice: Use built-in transformers like namePrefix, nameSuffix, commonLabels, commonAnnotations in your kustomization.yaml files instead of creating manual patches for these common modifications.
    • Benefit: Reduces verbosity and makes the kustomization.yaml more concise and readable.

  7. Version Control Separation (Optional but Recommended for Large Orgs):

    • Practice: Consider placing base configurations and overlays in separate Git repositories, especially in large organizations with distinct platform and application teams.
      • platform-k8s-bases (for shared infrastructure, standards)
      • team-a-app-k8s-base (for Team A's application base)
      • env-k8s-overlays (for environment-specific overlays that reference the above)
    • Benefit: Enforces clear ownership, promotes autonomy for application teams, and centralizes governance for platform teams.

  8. Use resources for Environment-Specific Resources:

    • Practice: If a resource (e.g., an HPA, a specific NetworkPolicy) only exists in a particular environment, define it directly in the overlay's kustomization.yaml using the resources field, rather than patching it into a base.
    • Benefit: Keeps the base clean and ensures resources are only deployed where needed.

  9. Document Your Structure:

    • Practice: Provide a README.md at the root of your Kustomize project and within key directories explaining the structure, conventions, and how to build/deploy.
    • Benefit: Essential for onboarding new team members and ensuring long-term maintainability.

  10. Automate kustomize build:

    • Practice: Integrate kustomize build into your CI/CD pipelines or GitOps tools. Avoid manual kubectl apply commands.
    • Benefit: Ensures consistency, reduces human error, and provides an auditable trail.

Example of a well-organized kustomization.yaml:

# overlays/production/my-app/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 1. Reference the base configuration
bases:
- ../../../base/my-app

# 2. Apply common transformations
namePrefix: prod-
commonLabels:
  env: production
  app.kubernetes.io/instance: my-app-prod

# 3. Generate environment-specific secrets/configmaps
secretGenerator:
- name: my-app-db-credentials
  files:
  - db_username.txt
  - db_password.txt

# 4. Apply specific patches
patchesStrategicMerge:
- prod-replicas-patch.yaml
- prod-resource-limits.yaml
- sidecar-injection-patch.yaml

# 5. Add environment-specific resources
resources:
- prod-hpa.yaml

By adhering to these best practices, Kustomize can be a highly effective tool for managing complex Kubernetes configurations in any enterprise setting.

Troubleshooting

15. How do you troubleshoot an issue with a Kustomize overlay?

Answer:

Troubleshooting Kustomize overlays involves systematically inspecting the generated output and verifying each layer of customization. The goal is to understand what Kustomize is actually producing versus what you expect it to produce.

  1. Use kustomize build to Inspect Output:

    • Command: kustomize build overlays/<env-name>/<app-name>
    • Action: This is the most fundamental step. It generates the final, flattened Kubernetes YAML manifests. Pipe the output to less or save it to a file for detailed inspection.
    • Focus: Look for missing resources, incorrect values, unexpected prefixes/suffixes, or patches that didn't apply as intended.

  2. Check the Overlay's kustomization.yaml:

    • Action: Verify that the kustomization.yaml file in your overlay is correctly configured.
    • Check for:
      • bases: Is the path to the base correct? (e.g., ../../../base/my-app).
      • resources: Are all environment-specific resources listed?
      • patchesStrategicMerge / patchesJson6902: Are all patches correctly listed and do their paths resolve?
      • generators: Are secretGenerator and configMapGenerator correctly defined and referencing the right files/literals?
      • transformers: Are namePrefix, commonLabels, etc., applied as expected?

  3. Inspect Individual Patches:

    • Action: If a patch isn't working, examine the patch file itself.
    • Check for:
      • Target Matching: For Strategic Merge Patches, does the apiVersion, kind, and metadata.name in the patch exactly match the resource it's supposed to modify in the base (or the output of previous transformations)?
      • Syntax: Is the YAML syntax correct?
      • Content: Is the content of the patch correct for the desired change?

  4. Verify Base Configuration:

    • Action: Sometimes the issue isn't with the overlay but with the base it's trying to modify.
    • Check: Run kustomize build base/<app-name> to see the raw base output. Ensure it's what you expect before any overlays are applied.

  5. Use kubectl diff for Live Comparison:

    • Command: kustomize build overlays/<env-name>/<app-name> | kubectl diff -f -
    • Action: This command shows you the exact differences between the Kustomize-generated manifests and the currently deployed resources in your cluster.
    • Focus: Pinpoint exactly what changes Kustomize would make, which can highlight unexpected modifications or omissions.

  6. Check Kustomize Version:

    • Action: Ensure you are using a consistent and up-to-date version of Kustomize, as behavior can sometimes vary between versions.

16. How do you troubleshoot a patching issue in Kustomize?

Answer:

Patching issues are common in Kustomize, especially with Strategic Merge Patches. Here's a systematic approach:

  1. Generate and Compare Output:

    • Command: kustomize build overlays/<env-name>/<app-name> > generated.yaml
    • Action: Generate the full output and then manually inspect the generated.yaml file.
    • Focus: Did the patch apply at all? Did it apply partially? Did it apply incorrectly (e.g., overwriting instead of merging)?

  2. Verify Patch Target:

    • Strategic Merge Patch: The apiVersion, kind, and metadata.name (and metadata.namespace if applicable) in your patch file must exactly match the resource you intend to modify in the base (or the resource after namePrefix/nameSuffix has been applied).
    • JSON Patch: Ensure the target block correctly identifies the resource.

  3. Understand Strategic Merge Patch Rules:

    • Action: Familiarize yourself with how Kubernetes (and thus Kustomize) merges lists and maps.
    • Key Rule: Lists of primitive types are replaced. Lists of objects with a defined "key" (like containers by name, ports by containerPort) are merged by matching the key. If no key, the list is replaced.
    • Example: If you want to add a container, you list the existing container(s) by name and then add your new container. If you just list the new container, it might replace the entire list if the list doesn't have a defined key.

  4. Simplify the Patch:

    • Action: If a complex patch isn't working, simplify it. Try to apply a minimal change (e.g., just change one field). Once that works, gradually add more complexity.

  5. Use kubectl diff with the Generated Output:

    • Command: kustomize build overlays/<env-name>/<app-name> | kubectl diff -f -
    • Action: This will show you the exact line-by-line differences between what Kustomize wants to apply and what's currently in the cluster. This is invaluable for seeing why a patch might not be taking effect or is causing unintended changes.

  6. Check for Conflicts:

    • Action: If multiple patches or transformers modify the same field, Kustomize's behavior can be complex.
    • Focus: Ensure your patches are not conflicting with each other or with other Kustomize features like commonLabels.

  7. JSON Patch Specifics:

    • Action: For patchesJson6902, ensure the JSON path (/path/to/field) is absolutely correct. Tools like jsonpatch.com can help validate JSON patches.
    • Operations: Double-check the op (add, remove, replace) and its arguments.

17. How do you troubleshoot a configuration error in a Kustomize base?

Answer:

Troubleshooting errors in a Kustomize base is generally simpler than overlays because there are fewer layers of abstraction. The primary goal is to ensure the base resources are valid Kubernetes YAML and that the base kustomization.yaml correctly references them.

  1. Validate Individual Resource YAMLs:

    • Command: kubectl dry-run -f base/my-app/deployment.yaml --validate=true --output=yaml
    • Action: Before even involving Kustomize, ensure each raw YAML file in your base is syntactically correct and valid against the Kubernetes API schema.
    • Focus: Look for typos, incorrect indentation, or invalid Kubernetes fields.

  2. Build the Base Independently:

    • Command: kustomize build base/my-app
    • Action: Generate the output of just the base. This will show you if Kustomize itself has issues processing the base kustomization.yaml or its resources.
    • Focus: Check for errors reported by Kustomize (e.g., "resource not found," "invalid YAML").

  3. Check the Base kustomization.yaml:

    • Action: Ensure the kustomization.yaml in the base directory correctly lists all the resources it intends to manage.
    • Check for:
      • resources: Are all deployment.yaml, service.yaml, etc., correctly listed? Are there any typos in the filenames?
      • commonLabels / commonAnnotations: Are these correctly formatted?

  4. Review Kustomize Build Errors:

    • Action: Kustomize often provides helpful error messages if it encounters issues during the build process (e.g., "unable to find resource," "invalid YAML"). Read these messages carefully.

  5. Version Compatibility:

    • Action: Ensure the apiVersion and kind in your base resources are compatible with your target Kubernetes cluster version.

By isolating the base and validating its components individually, you can quickly identify and resolve configuration errors before they are complicated by overlay-specific issues.