apache-kafka jobs in San Diego

The Site Reliability Engineer owns the reliability, scalability, and operational readiness of production services running on AWS and Kubernetes. Responsibilities include designing highly available infrastructure with Terraform and managed AWS services, building CI/CD pipelines with GitHub Actions and Argo CD, defining SLOs and implementing observability with Prometheus and Grafana, leading incident response, and automating operational work with Python, Go, or Bash.…See more The role requires 3–8 years of hands-on SRE or DevOps experience, production Kubernetes expertise, strong AWS and infrastructure-as-code knowledge, and proficiency with observability and deployment strategies.

Linux Infrastructure Engineer responsible for designing, building, and maintaining ultra-high-performance data center infrastructure supporting scientific computing and operational technology systems. Core duties include architecting and tuning software-defined storage (ZFS, BeeGFS, NVMe), managing 100GbE networking with RDMA, automating bare-metal provisioning via Ansible and Bash, and ensuring OT security through segmented, firewalled network architecture.…See more Requires 15+ years of systems administration experience, expert-level Linux (RHEL) and storage knowledge, and hands-on data center operations including physical deployment, diagnostics, and hardware lifecycle management.

A Software Engineer will design, build, test, and deliver secure, mission-critical software solutions for the U.S. Department of Defense, working with cross-functional teams to develop front-end and back-end features.…See more The role requires proficiency in multiple programming languages (C++, Python, Java, JavaScript), version control (Git), containerization (Docker, Kubernetes), relational databases (SQL/Postgres), and Linux systems, with additional value from front-end frameworks like React, DevSecOps practices, cloud/virtualization expertise, and scripting automation. The engineer will maintain CI/CD pipelines using Jenkins, integrate third-party software, and adhere to DoD and industry security and reliability standards.

Entry-level software engineer role at BAE Systems supporting C4ISR and geospatial intelligence platforms for defense customers. Candidates must hold or be able to obtain a security clearance and have a Bachelor's degree in Computer Science or related field (or Master's for Associate II level).…See more Required skills include Java, Object-Oriented Programming (OOP), and Object-Oriented Analysis & Design (OOAD); preferred experience includes full-stack development, multiple programming languages (C/C++, C#, Python, JavaScript), cloud platforms (AWS, Azure), containerization (Docker, Kubernetes), CI/CD pipelines, and familiarity with frameworks like Spring, REST APIs, and microservices architecture.

Software Engineering Intern III position for Summer 2027 at BAE Systems, supporting defense and aerospace projects. Interns will work alongside experienced engineers on impactful projects using modern full-stack technologies including React/Angular, AWS/Azure/Kubernetes, Spring, and various databases (SQL and NoSQL).…See more The role requires proficiency in Java, C++, C#, Python, or JavaScript, strong fundamentals in data structures and object-oriented programming, and a drive to learn. Participation in BAE Systems' LEAP Internship Program includes career development, mentorship, and networking opportunities.

Design, build, and maintain scalable data pipelines and platforms that power high-volume, data-driven applications. Develop and optimize ETL/ELT processes for large, complex datasets using Databricks and Spark, and build backend data services in C# / .NET.…See more Lead technical design decisions, mentor engineers, and leverage AI-assisted coding tools to improve engineering efficiency. Requires 5+ years of data engineering experience, strong expertise with data pipelines and ETL frameworks, and hands-on experience with AWS-based data platforms.

Lead technical architecture and design for banking and payments platforms, guiding engineering teams through standards, design reviews, and build activities to deliver secure, scalable systems. Collaborate with client and Deloitte stakeholders to align architecture decisions with program objectives.…See more Evaluate solution options, shape implementation roadmaps, and drive execution across technology transformation initiatives. Requires 6+ years in technical architecture or lead engineering, with 3+ years architecting enterprise-scale systems using microservices and event-driven architectures in production.

As an IC5 Senior Staff Engineer, you will architect and deliver large-scale distributed data platform components centered on Kafka, Apache Iceberg, and Apache Spark. You will lead complex technical initiatives, design high-performance data ingestion pipelines, and define engineering best practices across the organization.…See more The role demands deep expertise in distributed systems, JVM performance tuning, stream processing, and full-stack Data Lake solutions, with hands-on delivery of production systems and mentorship of engineering teams.

A Platform Engineer role focused on deploying, managing, and optimizing critical infrastructure for defense and government clients. Responsibilities include implementing system automation with Ansible and Puppet across RHEL and Windows platforms, deploying and maintaining Kubernetes clusters with RKE2 and Rancher, and managing open-source applications (NiFi, Kafka, Zookeeper, MinIO, Keycloak, Longhorn, Grafana, Prometheus, Loki, Promtail, GitLab).…See more The role requires 7+ years of hands-on enterprise Linux and Windows systems management, strong troubleshooting skills in distributed environments, and an active Top Secret/SCI clearance.

Staff Engineer leading the design and operation of Shield AI's data platform to support autonomous systems, ML workflows, and edge-to-cloud data flows. You will architect distributed storage and compute solutions, build first-party integrations for heterogeneous data sources, and establish operational standards for reliability, security, and observability.…See more The role requires deep expertise in data modeling, API design, distributed systems, Kubernetes infrastructure, and the ability to guide technical direction across autonomy, ML, test, and infrastructure teams.

Design, build, and maintain real-time data ingestion pipelines that reliably stream data from diverse sources into a production data platform. You will ensure data quality, observability, and scalability while monitoring pipeline health, building resilient systems with proper error handling and backpressure strategies, and automating monitoring and alerting for timeliness and data issues.…See more Partner with data engineers, platform engineers, and analytics teams to configure pipelines for reliability, perform root cause analysis on outages, and work with security teams on encryption and data governance. Proficiency required in Python and bash scripting, plus hands-on experience with tools like Kafka, NiFi, Flink, Spark, Snowflake, Grafana, and Prometheus.

Design, build, and maintain real-time data ingestion pipelines that reliably stream data from multiple sources into a centralized data platform. You will ensure data quality, observability, and low-latency delivery while collaborating with data engineers, platform engineers, and analytics teams.…See more Responsibilities include building resilient pipelines with error handling, automating monitoring and alerting for data issues, performing root cause analysis on outages, and partnering with security teams on encryption and data governance. You must demonstrate proficiency in Python and bash scripting, familiarity with tools like Kafka, NiFi, Spark Streaming, Snowflake, and Elasticsearch, and 3+ years of experience in data engineering or DevOps roles supporting production pipelines.

SHEIN is seeking a Senior Data Engineer to build and productionize GenAI/LLM solutions for data engineering workflows, including code assistance, metadata discovery, and knowledge access. The role focuses on developing RAG and agentic workflows, automating incident triage and log analysis, and integrating AI capabilities into internal developer tools and services.…See more You'll own scoped projects from problem definition through production support, requiring 3+ years of hands-on production experience with strong Python/SQL fundamentals, proven GenAI/LLM application development, and practical knowledge of databases, APIs, data pipelines, and distributed systems.

The SW Test Development Engineer 2 will design, develop, and maintain automated and manual test scripts for Dexcom's CGM software and data platform systems. You'll write API and UI tests, create verification and validation test plans, analyze failures with development teams, and contribute to release improvements.…See more The role requires 2–5 years of hands-on software or test development experience with Python or OOP languages, SQL/NoSQL databases, API testing frameworks (Robot framework), and cloud platforms (GCP/AWS); additional experience with Docker, Kubernetes, Kafka, BigQuery, and Airflow is highly valued. You'll mentor junior engineers and work cross-functionally in an Agile environment.

As a SW Test Development Engineer 2 at Dexcom, you will design, develop, and maintain automated and manual test scripts for CGM software systems, including API and UI testing. You will create verification and validation test plans, traceability matrices, and track issues while collaborating with development teams on root cause analysis.…See more The role requires 2–5 years of hands-on software test development experience with Python or other OOP languages, SQL/NoSQL databases, API testing frameworks (including Robot Framework), and cloud platforms (GCP/AWS). You will also mentor junior engineers and contribute to release process improvements.

Staff Data Engineer role at Credit Karma building petabyte-scale data infrastructure and streaming platforms on GCP. You will design and operate Kafka-based streaming infrastructure, build cloud-native data pipelines using Dataflow/Beam, Flink, and Spark, and create persistence frameworks across Spanner, MySQL, and BigQuery.…See more The role requires 7+ years backend/data systems experience in JVM languages (Scala/Java), expertise with high-throughput distributed systems, and deep knowledge of streaming platforms, Apache Beam, CDC patterns, encryption, and data governance frameworks. You'll also integrate AI/ML and generative AI technologies including LLMs, RAG, semantic search, and knowledge graphs into the data platform.

Senior Data Engineer role focused on building and productionizing GenAI/LLM solutions for data engineering workflows. You will develop RAG and agentic workflows, automate incident triage and root-cause analysis, integrate AI capabilities into internal developer tools, and own projects from problem definition through production support.…See more Required: 3+ years of production systems experience, strong Python/SQL, hands-on GenAI/LLM application building, and software engineering fundamentals including testing, CI/CD, and monitoring.

Senior Software Engineer working on AI-integrated expeditionary systems for the Navy, designing and deploying tactical data pipelines that connect sensors, unmanned systems, and command-and-control platforms. Requires 10+ years in software engineering, data engineering, or related roles, with expert Python skills, containerization experience (Docker/Kubernetes), and hands-on expertise in tactical data formats, event streaming platforms, and system integration.…See more Will collaborate with ML engineers, data engineers, and systems engineers to normalize tactical data, integrate machine learning models, and ensure robust data flow in field-tested operational environments.

A Senior Site Reliability Engineer at SHEIN will own and operate mission-critical, large-scale distributed systems (Kubernetes, Kafka, Elasticsearch, Redis, APISIX, Nginx) running 24/7/365, participating in on-call rotations and driving fast incident response using AI-assisted log analysis and anomaly detection. The role requires strong software engineering expertise in Python or Go, deep Linux and networking knowledge, and hands-on experience with observability platforms (Prometheus, Grafana) and configuration management tools.…See more Responsibilities include designing resilient monitoring and alerting infrastructure, automating operational workflows to eliminate toil, capacity planning, and collaborating with global teams to improve system reliability and performance.

Staff-level data engineer who designs, develops, and maintains scalable ETL/ELT pipelines using Databricks, PySpark, and Python for enterprise analytics and AI use cases. Builds curated data layers following Lakehouse Medallion architecture, develops Databricks-native applications (notebooks, dashboards, APIs), and optimizes Apache Spark workloads for performance and cost efficiency.…See more Owns production data pipelines end-to-end, ensures data quality and reliability, and acts as a technical leader mentoring teams on data engineering best practices and modern Databricks capabilities.

The Backend Engineer will design, develop, and scale core backend services and APIs for high-throughput applications using Node.js, Python, or Go. Responsibilities include architecting relational and NoSQL databases for high availability, building data pipelines and asynchronous processing systems, and deploying cloud infrastructure on AWS with Docker, Kubernetes, and Terraform.…See more The role requires 3–6 years of experience building distributed systems at scale, with deep expertise in database design, caching strategies, and cloud platforms. You'll also own monitoring, performance optimization, and comprehensive testing across unit, integration, and end-to-end scenarios.

As a Senior AI Engineer at Teradata, you will design and build agent-based AI solutions that enhance developer productivity, working on scalable, marketplace-ready platforms across the Software Development Lifecycle. You'll develop high-quality backend services and APIs, integrate AI/ML technologies including LLMs and agentic AI, and collaborate with cross-functional teams on Teradata's AI platform.…See more The role requires 5+ years of experience in backend services, distributed systems, or data platforms, with strong proficiency in Java, Go, or Python, and hands-on expertise in RESTful APIs, SQL/NoSQL databases, Kubernetes, and cloud platforms. You will also mentor engineers and stay current with emerging AI/ML and cloud-native technologies.

Design, implement, and scale core backend services and APIs using Python, Go, Node.js, or Java to support high-throughput applications. Architect and optimize relational (PostgreSQL) and NoSQL (Redis) databases, deploy containerized services on AWS/GCP using Docker and Kubernetes, and build CI/CD pipelines with comprehensive automated testing.…See more Monitor production systems, troubleshoot incidents, and collaborate cross-functionally to deliver reliable, low-latency features. Requires 3–6 years of distributed systems experience and proficiency in at least one backend language.

Sr. Director role overseeing SIE's data and AI platform ecosystem, including data infrastructure (streaming, ingestion, warehouse), experimentation platforms, and governance tools.…See more Responsible for defining long-term vision, leading global engineering teams, and ensuring platforms process massive data volumes to deliver insights across gaming, content creation, and enterprise operations. Must drive scalability, cost efficiency, security, and operational excellence while collaborating with senior leadership to align platform strategy with business objectives. Requires proven Sr. Director-level experience and thought leadership in platform architecture, real-time data processing, and ML/AI integration.

Design and implement Infrastructure as Code to automate provisioning, monitoring, and lifecycle management of NoSQL, Streaming, and Caching platforms (Cassandra, Aerospike, Kafka, Redis) across AWS and GCP. Build highly available, self-healing systems with automated failover and scaling, develop comprehensive observability solutions, and lead incident response for critical data platform issues.…See more Drive automation-first practices and apply AI/ML approaches such as anomaly detection and predictive scaling to enhance reliability and reduce manual toil. Partner with engineering and platform teams to ensure resilient infrastructure supporting billions of transactions and millions of players globally.