AI Software Development Engineer in Test (AI-SDET)

KEY RESPONSIBILITIES

1. Change leadership and team enablement

• Champion the adoption of AI-powered testing tools and practices across the QA Guild.

• Identify gaps in current QA processes and introduce AI-driven solutions to address them, from automated test generation to self-healing frameworks.

• Run workshops, lunch-and-learns, and knowledge-sharing sessions to upskill QA engineers on AI tools and methodologies.

• Serve as the internal subject matter expert on AI quality standards, emerging tooling, and best practices.

• Collaborate with QA leads, engineering managers, and product teams to embed AI quality practices into the SDLC.

2. AI and LLM quality engineering

• Design and maintain evaluation (Evals) frameworks covering hallucination detection, groundedness, toxicity, bias, prompt regression, determinism, and safety guardrail validation.

• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, and embedding-based metrics.

• Validate RAG architectures including chunking strategies, embedding quality, retrieval accuracy, re-ranking, and vector search relevance.

• Test multi-agent orchestration workflows, agent memory and context retention, and MCP (Model Context Protocol) integrations.

• Create synthetic and benchmark datasets for AI testing scenarios.

• Evaluate latency, token usage, model cost, throughput, and scalability of AI systems.

3. Automation engineering

• Develop and maintain scalable UI and API automation frameworks using Python, Robot Framework, and Playwright/Selenium/Cypress.

• Automate testing across REST APIs, GraphQL, event-driven systems, streaming services, webhooks, microservices, and micro-frontend (MFE) architectures.

• Build reusable automation libraries, utilities, and AI-assisted testing workflows.

• Integrate automation and quality gates into CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI/CD, Azure DevOps).

4. AI platform and agent validation

• Validate AI agents built on LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.

• Work with vector databases including Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.

• Validate AI workflows involving tool calling, function calling, agent planning and execution, structured outputs, and multi-modal systems.

• Monitor AI system behavior using observability, logging, and tracing platforms.

5. DevOps, cloud, and reliability

• Work within Docker and Kubernetes environments.

• Perform cloud-based testing and deployments across AWS, Azure, OCI, and GCP.

• Validate distributed system reliability and resiliency.

• Contribute to shift-left and quality engineering best practices across the organization.

• Strong programming experience in Python.

• Deep understanding of software testing and quality engineering principles.

• Hands-on experience with AI/LLM testing, prompt engineering, context engineering, and AI Evals.

• Practical experience with RAG systems, AI agents, and vector databases.

• Experience with CI/CD pipelines, Docker, Kubernetes, and at least one major cloud platform (AWS, Azure, OCI, or GCP).

• Experience with API testing, microservices testing, and UI automation.

• SQL proficiency and data validation experience.

• Demonstrated ability to influence engineering teams and drive adoption of new tools and processes.

• Strong analytical, debugging, and troubleshooting skills.

• Experience working in Agile/Scrum environments.

Preferred

• Experience with Robot Framework and Playwright.

• Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, or Semantic Kernel.

• Familiarity with MCP (Model Context Protocol).

• Hands-on experience with developer AI tools such as Cursor, GitHub Copilot, Claude, or Gemini.

• Exposure to MLOps workflows, performance testing, security testing, or AI governance and compliance requirements.

• Experience with contract testing, chaos testing, or resilience engineering.

• AI Red Teaming or LLM security testing experience (prompt injection, jailbreaks, data leakage).

• Familiarity with accessibility testing and cross-browser/device testing.

EDUCATION

Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, or a related field preferred. Equivalent hands-on experience and a demonstrable portfolio of projects are highly valued.

SUCCESS PROFILE

The right candidate for this role is more than a strong engineer. We are looking for someone who:

• Actively experiments with AI tools and builds side projects, prototypes, or open-source contributions.

• Can bridge traditional quality engineering practices with modern AI-native approaches.

• Communicates clearly, influences without authority, and earns trust across engineering, product, and leadership teams.

• Brings a continuous improvement mindset and a bias toward action.

• Rapidly learns and applies emerging AI frameworks, tools, and architectures.