Why This Job is Featured on The SaaS Jobs
Agentic AI is rapidly becoming a core feature set across SaaS products, and it introduces a distinct quality challenge: validating systems that mix conventional UI and API behavior with probabilistic, multi-step decisioning. This SDET role sits directly in that intersection, focusing on how agent workflows behave end to end across services, rather than treating AI as a separate research surface.
For a long-term SaaS career, the work maps closely to how modern subscription products are built and shipped: automation-first quality, CI-driven release confidence, and test strategies that scale with microservices. Experience defining approaches for non-deterministic behavior, observability, and contract-style validation is increasingly portable across SaaS teams adopting LLM features, even when the underlying domain differs.
The position is best suited to an engineer who prefers writing production-grade automation code and influencing system design through testability discussions. It will fit someone comfortable translating ambiguous failure modes into repeatable checks and metrics, and who enjoys collaborating with both platform engineers and AI or ML specialists to tighten release readiness in complex, integrated products.
The section above is editorial commentary from The SaaS Jobs, provided to help SaaS professionals understand the role in a broader industry context.
Job Description
Your role
As a SDET focused on Agentic AI, you are a strong software engineer with deep test automation skills and a passion for quality in AI-powered systems. You will: design and build scalable Python + Playwright automation for UI, API, and end-to-end flows; test complex agentic workflows, including multi-step reasoning, tool invocation, and orchestration across multiple services; telp define testing strategies for deterministic and non-deterministic systems, and contribute best practices for LLM and Agentic AI quality across the organization; you’ll be hands-on in code, involved in design and architecture discussions, and a key voice for testability, observability, and quality.
What you’ll do
- Design, develop, and maintain scalable automation frameworks using Playwright and Python for UI, API, and end-to-end testing.
- Build automated tests for Agentic AI workflows, including multi-step reasoning, tool invocation, and autonomous decision flows.
- Validate AI-driven systems using deterministic testing approaches, mocks, simulators, and contract testing where applicable.
- Develop test strategies for LLM-based features, including prompt validation, response evaluation, guardrails, and failure modes.
- Implement API, integration, and system-level tests across microservices and AI orchestration layers.
- Collaborate with AI/ML engineers to test model integrations, inference pipelines, and agent execution frameworks.
- Set up and maintain CI/CD pipelines using GitHub Actions, enabling fast feedback and reliable releases.
- Analyze test results, derive quality KPIs, and clearly communicate risks and release readiness.
- Provide detailed root-cause and failure analysis to accelerate defect resolution.
- Participate in design and architecture reviews, advocating for testability, observability, and quality-first design.
- Mentor junior SDETs and QA engineers, raising the bar on automation, AI testing practices, and engineering excellence.
Skills you’ll bring
- 4+ years of experience as an SDET, Software Engineer, or Test Automation Engineer in SaaS environments.
- Strong programming experience in Python (preferred) and familiarity with JavaScript/TypeScript.
- Hands-on experience with Playwright (preferred) or modern UI automation frameworks.
- Solid experience testing API-first and microservices-based architectures.
- Practical understanding of AI/ML or LLM-based systems, including:
- Agentic AI concepts (agents, tools, planners, memory, orchestration).
- Deterministic vs non-deterministic testing strategies.
- Prompt-based systems and response evaluation.
- Experience with mocking, stubbing, and simulation to test complex integrations.
- Strong knowledge of REST APIs, server-side testing, and integration testing.
- Experience working with CI/CD pipelines, preferably GitHub Actions.
- Familiarity with Cloud technologies and containerized environments (Docker).
- Excellent written and verbal communication skills; comfortable working across geographies.
Nice to Have
- Experience testing Agentic AI frameworks (e.g., LangChain, LlamaIndex, or similar concepts).
- Knowledge of AI evaluation techniques (golden datasets, semantic similarity, rule-based validation, replay testing).
- Experience with communication platforms, contact centers, or VoIP technologies.
- Experience with performance, load, and reliability testing for AI-powered systems.
Technologies We Use
- Languages: Python, Java, JavaScript.
- Automation: Playwright, Selenium, TestNG.
- API Testing: REST Assured, Postman.
- CI/CD: GitHub Actions (GHA).
- Dev Tools: Git, GitHub, Jira.
- Infrastructure: Docker, Cloud-based services.
