devin jobs in San Diego

Build and scale evaluation tooling to measure how well AI-powered software development tools perform. Develop evaluation harnesses, automate benchmark runs, and ensure reproducibility through versioned processes with pinned dependencies and containerized environments.…See more Validate evaluation approaches against human judgment, analyze results for variance and failure patterns, and work with engineering teams to improve automation workflows. This role requires strong software engineering foundations in automation and testing, hands-on experience with AI coding agents, and proficiency in Python, Java, or JavaScript.

Build and scale evaluation harnesses and automation for AI-powered software development tools, turning real engineering artifacts into repeatable benchmark tasks. Develop versioned, reproducible processes with containerized environments to measure AI tool performance across quality, productivity, and efficiency metrics.…See more Validate evaluation approaches against human judgment, analyze results for variance and failure patterns, and work with engineering and data teams to improve tooling and document findings.

Build and scale evaluation harnesses and automation to measure how well AI-powered software development tools perform. You'll develop reproducible benchmarking systems, validate evaluation approaches against human judgment, and analyze results across repeated runs to identify variance, failure patterns, and optimization opportunities.…See more The role requires strong software engineering expertise in automation and testing, proficiency in Python and at least one other language (Java, JavaScript), and solid knowledge of Git, Docker, CI/CD pipelines, and AI/LLM evaluation methodologies.