apache-spark-streaming jobs in San Diego

Design, build, and maintain real-time data ingestion pipelines that reliably flow streaming data from multiple sources into a data platform while ensuring quality, observability, and scalability. Monitor data health, develop resilient pipelines with proper error-handling and backpressure strategies, and automate monitoring of real-time feeds to alert on timeliness, volume, and distribution issues.…See more Collaborate with data, platform, and software engineers to optimize pipeline configuration, perform root-cause analysis on data outages, and work with security teams on encryption and data classification. Requires 3+ years in data engineering, data operations, or DevOps roles; proficiency with Python and bash; and hands-on experience with tools like Kafka, NiFi, Spark Streaming, Snowflake, Elasticsearch, Grafana, and Prometheus.

Design, build, and maintain real-time data ingestion pipelines that reliably stream data from multiple sources into a centralized data platform. You will ensure data quality, observability, and low-latency delivery while collaborating with data engineers, platform engineers, and analytics teams.…See more Responsibilities include building resilient pipelines with error handling, automating monitoring and alerting for data issues, performing root cause analysis on outages, and partnering with security teams on encryption and data governance. You must demonstrate proficiency in Python and bash scripting, familiarity with tools like Kafka, NiFi, Spark Streaming, Snowflake, and Elasticsearch, and 3+ years of experience in data engineering or DevOps roles supporting production pipelines.