Senior Observability Engineer (Splunk / SignalFx) - US Citizen or Green Card only

**Overview** We are seeking an experienced Observability Engineer specializing in Splunk Enterprise and Splunk Observability Cloud (formerly SignalFx). In this role, you will design, implement, and maintain our enterprise-wide logging, metrics, and application performance monitoring (APM) infrastructure. You will work closely with DevOps, SRE, and software engineering teams to ensure real-time visibility into cloud-native microservices, hybrid infrastructure, and critical business applications. **Key Responsibilities** * **Platform Architecture**: Architect, deploy, and optimize Splunk Enterprise log management and Splunk Observability Cloud (SignalFx) environments. * **Data Ingestion & Pipelines**: Build and maintain telemetry data pipelines using the OpenTelemetry (OTel) Collector, SignalFx Smart Agents, and Splunk Universal/Heavy Forwarders. * **Monitoring & Alerting**: Design high-performance dashboards, real-time streaming charts, and dynamic alerting mechanisms using SignalFlow or advanced SPL (Search Processing Language) to eliminate alert fatigue. * **Distributed Tracing & APM**: Instrument cloud-native microservices applications (Java, Python, Go, Node.js) to capture distributed traces, metrics, and logs for end-to-end transaction visibility. * **Collaboration & Coaching**: Partner with SRE and Development teams to establish Service Level Indicators (SLIs) and Service Level Objectives (SLOs), providing training on root-cause analysis using the modern Splunk ecosystem. * **Best Practice**: Maintain comprehensive documentation of system architecture, configurations, and troubleshooting procedures aligned with SDLC (Software Development Life Cycle) best practices. * **Up-to-date Security Posture**: Stay current with emerging trends in cloud security, vulnerability management, threat intelligence, and system hardening to enhance monitoring strategies. **Qualifications** * Experience: 4+ years of dedicated engineering experience in platform **Observability**, Site Reliability Engineering (SRE), or DevOps. * Proven experience in solution architecture for large-scale enterprise systems with a focus on observability tools such as **Splunk Enterprise Security** or **SignalFx (Splunk Observability Cloud)**. * Tooling Expertise: Proven, hands-on experience with Splunk Cloud/Enterprise and SignalFx (Splunk Infrastructure Monitoring, Splunk APM, and Splunk Log Observer). * OpenTelemetry: Deep familiarity with OpenTelemetry standards, including configuring OTel Collectors for distributed tracing and streaming metrics. * Cloud & Containerization: Strong expertise monitoring AWS, Azure, or GCP infrastructure alongside container orchestration platforms like Kubernetes (EKS/ GKE) and Docker. * Infrastructure as Code (IaC): Experience automating monitoring deployments via Terraform, Ansible, or CloudFormation. * Strong background in IT infrastructure including cloud platforms (AWS, Azure), virtualization (VMware), operating systems (Linux/Windows), and network fundamentals (DNS, VPNs, firewalls). * Expertise in scripting languages like Python, Bash shell scripting; familiarity with version control systems such as Git for managing codebases. * Deep understanding of system design principles including microservices architecture, load balancing, RESTful APIs, and system integration techniques. * Knowledge of database systems including SQL (MySQL, PostgreSQL), NoSQL databases (MongoDB), and data management best practices. * Experience with DevOps practices such as CI/CD pipelines (Jenkins), container orchestration (Kubernetes), configuration management (GPO), and system hardening for security compliance. * Strong analytical skills in requirements analysis, business analysis, and technical troubleshooting support across diverse IT environments. * Excellent cross-functional communication skills to translate technical concepts into clear documentation for stakeholders at all levels. **Preferred Qualifications** * Splunk Enterprise Certified Admin, Splunk Cloud Certified Admin, or official Splunk Observability Cloud certifications. * Proficiency in scripting or coding languages (Python, Go, or Bash) to write custom automation scripts or OpenTelemetry plug-ins. Join us to be at the forefront of observability innovation—empowering organizations through smarter monitoring solutions that safeguard their digital assets while enabling seamless operational excellence! Pay: $60.00 - $80.00 per hour Benefits: * Flexible schedule Work Location: Hybrid remote in Fairfax, VA 22031