Gen AI Developer & Tester (RARR Job 5708)

We are looking for a GenAI Developer & Tester who can design, build, and validate enterprise-grade Retrieval-Augmented Generation (RAG) Agents, Agentic AI, and Generative AI applications on Azure.
This hybrid role combines hands-on development with rigorous testing and evaluation to deliver grounded, reliable, and production-ready AI solutions at scale.

Key Responsibilities

• Design and develop RAG bots, Agentic AI bots, and GenAI applications leveraging:

  • Azure OpenAI, Azure AI Search, Python, Flask / FastAPI, Azure Functions, and APIM.

  • Secure configurations using environment variables and Azure-managed identities.

• Implement scalable data layers using Cosmos DB and Azure Storage (Blob, Queue, Table).

  • Optimize chunking strategies, embeddings, and hybrid/vector search pipelines.

• Lead prompt engineering and tuning across:

  • System prompts, retrieval prompts, and tool-use prompts.

  • Design guardrails to minimize hallucinations and enhance factual grounding.

• Automate testing and evaluation using pytest and RAG evaluation frameworks (Ragas).

  • Evaluate faithfulness, precision, recall, and answer correctness.

• Implement observability and telemetry using Application Insights, advanced logging, and end-to-end traceability.

• Orchestrate intelligent agent workflows using LangChain, and optionally LangGraph for stateful and multi-agent orchestration.

• Ensure production readiness by:

  • Managing rate limits, optimizing cost-performance, and scaling AI Search tiers.

  • Meeting defined SLA and latency benchmarks for production environments

Must-Have Skills

• Strong expertise in Python, Flask / FastAPI, Azure Functions, and APIM.

• Proven experience with Azure OpenAI, Azure AI Search, Cosmos DB, and Azure Storage (Blob / Queue / Table).

• Hands-on proficiency in prompt engineering, retrieval optimization, and prompt tuning.

• Experience in test automation (pytest) and evaluation frameworks (Ragas).

• Deep understanding of grounding techniques, hybrid search, semantic ranking, and context filtering for RAG systems.

Nice to Have

• Experience with LangGraph for advanced agentic flow orchestration.

• Familiarity with CI/CD pipelines using Azure DevOps or GitHub Actions.

• Exposure to structured output validation and fine-tuning workflows.

Preferred Qualifications

• Bachelor’s or Master’s degree in Computer Science, AI/ML, or a related field.

• 2–5 years of experience building AI applications with Azure OpenAI and related Azure services.

• Demonstrated experience in production-grade RAG and GenAI implementations.