Hi,
IMHO, to address the (secret sauce) is a combination of proactive resource management, smart scaling strategies, and ongoing monitoring and adjustment. Here is a course of action.
(1) Adjusting Pod Requests and Limits
You would need to update your Kubernetes deployment YAML to adjust the requests and limits:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
replicas: 3 # Adjust this number based on your scaling needs
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: api-server
image: dependencytrack/apiserver
ports:
- containerPort: 8080
resources:
limits:
cpu: "3" # Adjusted from 4 to 3 based on observed usage
memory: "8Gi" # Adjusted from 16Gi to 8Gi
requests:
cpu: "1" # Adjusted from 2 to 1
memory: "4Gi" # Adjusted from 5Gi to 4Gi
volumeMounts:
- mountPath: /data
name: dependency-track
volumes:
- name: dependency-track
persistentVolumeClaim:
claimName: <your-pvc-name>
(2) Implement Horizontal Pod Autoscaler (HPA)
Set up HPA to scale based on CPU and memory usage.
kubectl autoscale deployment api-server --cpu-percent=50 --min=1 --max=10
(3) Using Affinity and Anti-Affinity
To set up pod affinity and anti-affinity, you would modify your deployment configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- api-server
topologyKey: "kubernetes.io/hostname"
(4) Setting Budgets and Cost Allocation Tags
To create budgets and set up cost alerts, you would use the GCP console or gcloud CLI, not Kubernetes configuration files. However, to use labels for cost allocation:
kubectl label pods <pod-name> team=finance
(5) Configure Cluser Autoscaler
Ensure your cluster autoscaler is set up correctly for your node pools. Here is the Terraform (IaC)
resource "google_container_cluster" "primary" {
# ... other cluster specs
node_pool {
# ... other node pool specs
autoscaling {
min_node_count = 1
max_node_count = 10
}
}
}
(6) Optimize Node Pool Management
Create multiple node pools for different workload needs.
resource "google_container_node_pool" "secondary" {
# ... other node pool specs
autoscaling {
min_node_count = 1
max_node_count = 5
}
management {
auto_repair = true
auto_upgrade = true
}
node_config {
# Specify a different machine type, disk size, etc.
}
}
(7) Monitor Autoscaling Events
Review the autoscaling events to understand the scaling behavior.
gcloud logging read "resource.type=\"k8s_cluster\" AND jsonPayload.message: \"ClusterAutoscaler\"" --limit 10 --format "table(timestamp, jsonPayload.message)"
I hope it helps
Best Regards
Mahmoud
