apache-kafka jobs in San Diego

SHEIN is seeking a Senior Data Engineer to build and productionize GenAI/LLM solutions for data engineering workflows, including code assistance, metadata discovery, and knowledge access. The role focuses on developing RAG and agentic workflows, automating incident triage and log analysis, and integrating AI capabilities into internal developer tools and services.…See more You'll own scoped projects from problem definition through production support, requiring 3+ years of hands-on production experience with strong Python/SQL fundamentals, proven GenAI/LLM application development, and practical knowledge of databases, APIs, data pipelines, and distributed systems.

The SW Test Development Engineer 2 will design, develop, and maintain automated and manual test scripts for Dexcom's CGM software and data platform systems. You'll write API and UI tests, create verification and validation test plans, analyze failures with development teams, and contribute to release improvements.…See more The role requires 2–5 years of hands-on software or test development experience with Python or OOP languages, SQL/NoSQL databases, API testing frameworks (Robot framework), and cloud platforms (GCP/AWS); additional experience with Docker, Kubernetes, Kafka, BigQuery, and Airflow is highly valued. You'll mentor junior engineers and work cross-functionally in an Agile environment.

As a SW Test Development Engineer 2 at Dexcom, you will design, develop, and maintain automated and manual test scripts for CGM software systems, including API and UI testing. You will create verification and validation test plans, traceability matrices, and track issues while collaborating with development teams on root cause analysis.…See more The role requires 2–5 years of hands-on software test development experience with Python or other OOP languages, SQL/NoSQL databases, API testing frameworks (including Robot Framework), and cloud platforms (GCP/AWS). You will also mentor junior engineers and contribute to release process improvements.

Staff Data Engineer role at Credit Karma building petabyte-scale data infrastructure and streaming platforms on GCP. You will design and operate Kafka-based streaming infrastructure, build cloud-native data pipelines using Dataflow/Beam, Flink, and Spark, and create persistence frameworks across Spanner, MySQL, and BigQuery.…See more The role requires 7+ years backend/data systems experience in JVM languages (Scala/Java), expertise with high-throughput distributed systems, and deep knowledge of streaming platforms, Apache Beam, CDC patterns, encryption, and data governance frameworks. You'll also integrate AI/ML and generative AI technologies including LLMs, RAG, semantic search, and knowledge graphs into the data platform.

Senior Data Engineer role focused on building and productionizing GenAI/LLM solutions for data engineering workflows. You will develop RAG and agentic workflows, automate incident triage and root-cause analysis, integrate AI capabilities into internal developer tools, and own projects from problem definition through production support.…See more Required: 3+ years of production systems experience, strong Python/SQL, hands-on GenAI/LLM application building, and software engineering fundamentals including testing, CI/CD, and monitoring.

Senior Software Engineer working on AI-integrated expeditionary systems for the Navy, designing and deploying tactical data pipelines that connect sensors, unmanned systems, and command-and-control platforms. Requires 10+ years in software engineering, data engineering, or related roles, with expert Python skills, containerization experience (Docker/Kubernetes), and hands-on expertise in tactical data formats, event streaming platforms, and system integration.…See more Will collaborate with ML engineers, data engineers, and systems engineers to normalize tactical data, integrate machine learning models, and ensure robust data flow in field-tested operational environments.

A Senior Site Reliability Engineer at SHEIN will own and operate mission-critical, large-scale distributed systems (Kubernetes, Kafka, Elasticsearch, Redis, APISIX, Nginx) running 24/7/365, participating in on-call rotations and driving fast incident response using AI-assisted log analysis and anomaly detection. The role requires strong software engineering expertise in Python or Go, deep Linux and networking knowledge, and hands-on experience with observability platforms (Prometheus, Grafana) and configuration management tools.…See more Responsibilities include designing resilient monitoring and alerting infrastructure, automating operational workflows to eliminate toil, capacity planning, and collaborating with global teams to improve system reliability and performance.

Staff-level data engineer who designs, develops, and maintains scalable ETL/ELT pipelines using Databricks, PySpark, and Python for enterprise analytics and AI use cases. Builds curated data layers following Lakehouse Medallion architecture, develops Databricks-native applications (notebooks, dashboards, APIs), and optimizes Apache Spark workloads for performance and cost efficiency.…See more Owns production data pipelines end-to-end, ensures data quality and reliability, and acts as a technical leader mentoring teams on data engineering best practices and modern Databricks capabilities.

js, Python, or Go. Responsibilities include architecting relational and NoSQL databases for high availability, building data pipelines and asynchronous processing systems, and deploying cloud infrastructure on AWS with Docker, Kubernetes, and Terraform.…See more pipelines and asynchronous processing systems, and deploying cloud infrastructure on AWS with Docker, Kubernetes, and Terraform. The role requires 3–6 years of experience building distributed systems at scale, with deep expertise in database design, caching strategies, and cloud platforms. You'll also own monitoring, performance optimization, and comprehensive testing across unit, integration, and end-to-end scenarios.

As a Senior AI Engineer at Teradata, you will design and build agent-based AI solutions that enhance developer productivity, working on scalable, marketplace-ready platforms across the Software Development Lifecycle. You'll develop high-quality backend services and APIs, integrate AI/ML technologies including LLMs and agentic AI, and collaborate with cross-functional teams on Teradata's AI platform.…See more The role requires 5+ years of experience in backend services, distributed systems, or data platforms, with strong proficiency in Java, Go, or Python, and hands-on expertise in RESTful APIs, SQL/NoSQL databases, Kubernetes, and cloud platforms. You will also mentor engineers and stay current with emerging AI/ML and cloud-native technologies.

js, or Java to support high-throughput applications. Architect and optimize relational (PostgreSQL) and NoSQL (Redis) databases, deploy containerized services on AWS/GCP using Docker and Kubernetes, and build CI/CD pipelines with comprehensive automated testing.…See more ker and Kubernetes, and build CI/CD pipelines with comprehensive automated testing. Monitor production systems, troubleshoot incidents, and collaborate cross-functionally to deliver reliable, low-latency features. Requires 3–6 years of distributed systems experience and proficiency in at least one backend language.

Sr. Director role overseeing SIE's data and AI platform ecosystem, including data infrastructure (streaming, ingestion, warehouse), experimentation platforms, and governance tools.…See more Responsible for defining long-term vision, leading global engineering teams, and ensuring platforms process massive data volumes to deliver insights across gaming, content creation, and enterprise operations. Must drive scalability, cost efficiency, security, and operational excellence while collaborating with senior leadership to align platform strategy with business objectives. Requires proven Sr. Director-level experience and thought leadership in platform architecture, real-time data processing, and ML/AI integration.

Design and implement Infrastructure as Code to automate provisioning, monitoring, and lifecycle management of NoSQL, Streaming, and Caching platforms (Cassandra, Aerospike, Kafka, Redis) across AWS and GCP. Build highly available, self-healing systems with automated failover and scaling, develop comprehensive observability solutions, and lead incident response for critical data platform issues.…See more Drive automation-first practices and apply AI/ML approaches such as anomaly detection and predictive scaling to enhance reliability and reduce manual toil. Partner with engineering and platform teams to ensure resilient infrastructure supporting billions of transactions and millions of players globally.

Senior Python Backend Engineer building and maintaining Python-based APIs and services in an AWS environment. The role focuses on developing Kafka-based data pipelines, managing data movement across systems, and ensuring reliability and performance of containerized services.…See more You'll need strong production Python experience, familiarity with streaming or message-based systems, and comfort working with AWS and Docker.

Staff Software Engineer leading the design, architecture, and operations of PlayStation's Real-Time Analytics Platform (RTAP), a large-scale distributed data system handling high-throughput, low-latency analytics and stream processing. The role involves architecting and evolving streaming and analytics platforms using Apache Flink, Spark, ClickHouse, and Druid; integrating batch and streaming workloads via lakehouse architectures; and embedding AI capabilities to automate engineering workflows and improve platform reliability.…See more You will own critical platform capabilities end-to-end from design through production operations, mentor engineers, establish technical standards, and influence cross-functional technical strategy across PlayStation's data infrastructure.

Staff Software Engineer leading design and evolution of PlayStation's Real-Time Analytics Platform (RTAP), a large-scale distributed data system handling high-throughput, low-latency analytics and stream processing. Responsibilities include architecting stream processing pipelines, optimizing data services using Apache Flink, Spark, ClickHouse, and Druid, integrating batch and streaming workloads with lakehouse architectures, and building AI-powered capabilities for platform automation.…See more The role requires deep expertise in distributed systems, real-time analytics, and mentoring engineering teams while establishing technical standards and best practices across the organization.

Staff-level engineer designing and operating large-scale distributed data platforms powering real-time analytics across PlayStation. You will lead the architecture and evolution of the Real-Time Analytics Platform (RTAP), own critical capabilities end-to-end from design through production operations, and mentor engineers.…See more Key responsibilities include building and optimizing distributed services using Apache Flink, Spark Structured Streaming, ClickHouse, and Apache Druid; integrating batch and streaming workloads with lakehouse architectures; and establishing technical standards for distributed systems, stream processing, and data modeling.

SAIC seeks a Senior Software Engineer to integrate AI capabilities into expeditionary systems for the Navy, including sensor kits, unmanned systems, command & control, and TAK. The role spans hands-on development—maintaining Zenoh data mesh in Rust, building React frontends in JavaScript, integrating machine learning models, converting tactical message formats—and technical leadership with senior management.…See more Required qualifications include 9+ years as a software engineer, data engineer, or data scientist; expert Python; intermediate Rust or Go; experience with containerization, event streaming platforms (Kafka, Kinesis, RabbitMQ, Zenoh), tactical data standardization, and data pipeline deployment. Successful candidates will collaborate with ML engineers, systems engineers, and field teams to ensure high-quality data flow and operational performance.

Design, build, and scale ETL/ELT pipelines and cloud data warehouse infrastructure using Python, SQL, and modern orchestration tools like Apache Airflow or Prefect. Own the end-to-end data infrastructure that powers business intelligence and operational reporting, including data quality checks, anomaly detection, and performance optimization.…See more Collaborate with analytics and software engineering teams to define data modeling standards and maintain infrastructure-as-code deployments using Terraform and CI/CD. Requires 3–6 years of data engineering or backend development experience with advanced SQL, Python, and hands-on work with cloud data warehouses like Snowflake or BigQuery.

Staff Site Reliability Engineer at SHEIN responsible for operating and evolving large-scale, mission-critical production systems with 24/7/365 on-call participation. Design, build, and maintain observability solutions (metrics, logs, traces, alerting) with AI-powered anomaly detection; own and operate core open-source infrastructure (APISIX, Nginx, Kubernetes, Kafka, Elasticsearch, Redis, Consul, Etcd, Zookeeper).…See more Automate operational workflows, reduce incident frequency and MTTR, and provide technical leadership across global engineering teams. Requires strong software engineering skills, deep Linux/networking/distributed systems expertise, and passion for solving problems at scale.

NET Core in Fremont, California (onsite). Responsibilities include developing and maintaining REST APIs, building responsive Angular UIs, designing microservices, optimizing SQL queries, and integrating caching (Redis) and event-driven messaging (Kafka).…See more , optimizing SQL queries, and integrating caching (Redis) and event-driven messaging (Kafka). Requires 3–5 years of .NET experience with enterprise applications and distributed systems.

This role provides Tier 2 technical support for Windows and Linux systems, network infrastructure, and enterprise applications in a classified DoD environment. The administrator will configure and maintain operating systems, troubleshoot hardware and software issues, support distributed systems (Hadoop, Kafka, HBase/Accumulo, Spark), and utilize automation tools including Bash, PowerShell, Python, Ansible, Puppet, and Terraform.…See more Responsibilities include maintaining compliance with DoD and Intelligence Community security requirements, collaborating with engineering and cybersecurity teams, and documenting technical procedures. An active TS/SCI clearance and DoD 8570/8140 IAT Level I certification are required.