llm-as-judge jobs in San Diego

Build and scale evaluation harnesses and automation to measure how well AI-powered software development tools perform. You'll develop reproducible benchmarking systems, validate evaluation approaches against human judgment, and analyze results across repeated runs to identify variance, failure patterns, and optimization opportunities.…See more The role requires strong software engineering expertise in automation and testing, proficiency in Python and at least one other language (Java, JavaScript), and solid knowledge of Git, Docker, CI/CD pipelines, and AI/LLM evaluation methodologies.

Design, build, and ship LLM-powered capabilities end to end—from prototyping and fine-tuning models to deploying production agents and retrieval systems. Own the full stack: prompt and context engineering, multi-step agent design with tool calling, RAG systems (embeddings, chunking, hybrid search, reranking), fine-tuning on multi-GPU with LoRA/QLoRA, evaluation and tracing infrastructure, and clean APIs for other engineers.…See more Work in a secure, distributed environment where the platform runs on customer compute, cloud, or hybrid setups, requiring expertise with both commercial and self-hosted models.

Build the foundational agentic AI layer for a materials-science platform, including multi-model provider abstraction, agent orchestration with stateful checkpoints, retrieval systems, prompt versioning, and comprehensive tracing and evaluation frameworks. You'll design agents that plan and reason over tool calls in production, implement human-in-the-loop safety gates, and ensure all LLM behavior remains auditable and cost-tracked across customers' secure environments.…See more The role demands deep production experience with agentic and LLM systems: async Python, structured outputs, memory and context management, multi-step workflow orchestration, and evaluation harnesses that catch regressions before deployment.