Site Reliability Engineer

The Site Reliability Engineer ensures our production services remain highly available, scalable, and efficient on Google Cloud Platform. You will bridge the gap between development and operations by diving deep into application source code and cloud infrastructure to permanently engineer away underlying issues, rather than just patching symptoms. This role focuses on owning complex alert triage, reading and debugging code to solve root causes, GitOps application deployments via ArgoCD, and leveraging Grafana observability alongside advanced AIOps platforms to drive down operational toil.

Key Responsibilities:

• Code-Level Alert Resolution: Act as the ultimate owner of complex alerts by investigating stack traces, reading application source code, and submitting code-level fixes alongside infrastructure adjustments to permanently resolve chronic issues.
• Infrastructure Optimization: Diagnose and resolve deep OS and distributed system bottlenecks—including CPU throttling, memory leaks, and storage constraints—across GKE worker nodes and critical data infrastructure (e.g., Kafka, Datastream).
• GitOps & Deployments: Deploy, roll back, and manage the lifecycle of containerized applications using ArgoCD pipeline workflows, ensuring safe and reliable release rollouts.
• AIOps & Automation: Utilize AI-driven operations tools and build custom Python/Go automation (e.g., PagerDuty API integrations) to interpret correlated events, reduce alert noise, and automate manager/triage workflows.
• Network Troubleshooting: Diagnose complex connectivity and latency issues across all network layers, isolating problems between cloud VPCs, Kubernetes overlays, and microservices.
• Incident Response: Participate in on-call rotations, using Grafana dashboards, AIOps suggestions, and code-level tracing to rapidly mitigate and permanently fix production container issues.

What You'll Actually Do (Example Scenario):

• The Alert: PagerDuty pages you for elevated consumer lag on a critical Kafka topic and latency spikes in a downstream microservice.
• The Investigation: You use Grafana to correlate the latency with CPU throttling on specific GKE worker nodes. You drop into the command line, run top and tcpdump, and notice the application pods are churning through memory and network connections.
• The Code Dive: You pull the Python application source code and discover a recently merged commit introduced an inefficient retry loop when failing to parse certain Kafka payloads, causing a memory leak.
• The Fix: You temporarily scale up the GCP machine type or rollback via ArgoCD to restore service. Then, you write a Python PR to fix the exception handling, adjust the Kubernetes resource limits in the Git repo, and update your AIOps definitions to catch this specific anomaly automatically in the future.

Required Technical Skills:

• Software Engineering: Strong proficiency in reading, debugging, and modifying software in Python or Go to fix production bugs, build APIs, and create automation scripts.
• Cloud & Orchestration: Deep functional knowledge of deploying, scaling, and managing workloads in GKE, with a strong grasp of GCP infrastructure and machine type optimization.
• Operating Systems: Strong foundational knowledge of Linux internals, process management, file systems, kernel parameters, and how application performance triggers OS-level constraints (e.g., OOM killers).
• Networking: Practical troubleshooting skills in Kubernetes Networking (Pod-to-Pod communication, CNI plugins, Ingress) as well as L3-L7 mechanics (TCP/UDP, gRPC, DNS, IP routing).
• Observability & Alerting: Deep experience with the Grafana Labs ecosystem (Grafana, Mimir/Prometheus, Loki, Tempo) and advanced incident management platforms like PagerDuty.
• CI/CD & GitOps: Practical experience managing applications using ArgoCD and Git version control systems.
• System & Network Utilities: Proficiency with command-line diagnostic tools (e.g., tcpdump, curl, dig, traceroute, top, iostat).
• ML/AIOps (Nice to have): Familiarity with anomaly detection concepts, log-based ML models, and automated noise reduction.

Soft Skills & Qualifications:

• Engineering Mindset: A relentless drive to engineer away toil and fix root causes at the architectural or code layer rather than relying on manual runbooks.
• Autonomous Problem Solving: Ability to systematically troubleshoot complex, highly distributed microservice issues under pressure without needing a playbook.
• Urgency & Prioritization: Strong sense of ownership and ability to prioritize alerts based on business and customer impact.
• Communication: Clear written and verbal communication during high-stress incident responses and blameless Root Cause Analyses (RCAs).

Location:

• India – (Flexible hybrid work model - work from Bangalore office for 20 days in a quarter)