Why This Job is Featured on The SaaS Jobs
SaaS products are increasingly shipping AI features where behaviour is probabilistic, distributed, and difficult to validate with traditional QA alone. This SDET role stands out because it treats quality as an engineering problem across UI, APIs, and end to end agentic workflows, including tool invocation and orchestration across services. The focus on Python and Playwright also signals a modern web delivery surface typical of subscription software.
From a SaaS career perspective, the work maps closely to how reliable software gets delivered at scale: automation frameworks, contract and integration testing, CI driven feedback loops, and release readiness signals. The added layer is AI evaluation, where teams must define what “correct” looks like, build simulators and mocks, and develop guardrails for failure modes. That combination is increasingly transferable across SaaS categories as LLM features become part of core product flows.
This role is likely to suit engineers who prefer hands on coding and enjoy partnering with product and platform stakeholders to improve testability and observability. It also fits professionals who want to move beyond scripted QA into engineering led quality, and who are comfortable reasoning about systems where determinism is not guaranteed.
The section above is editorial commentary from The SaaS Jobs, provided to help SaaS professionals understand the role in a broader industry context.
Job Description
Your role
As a SDET focused on Agentic AI, you are a strong software engineer with deep test automation skills and a passion for quality in AI-powered systems. You will: design and build scalable Python + Playwright automation for UI, API, and end-to-end flows; test complex agentic workflows, including multi-step reasoning, tool invocation, and orchestration across multiple services; telp define testing strategies for deterministic and non-deterministic systems, and contribute best practices for LLM and Agentic AI quality across the organization; you’ll be hands-on in code, involved in design and architecture discussions, and a key voice for testability, observability, and quality.
What you’ll do
- Design, develop, and maintain scalable automation frameworks using Playwright and Python for UI, API, and end-to-end testing.
- Build automated tests for Agentic AI workflows, including multi-step reasoning, tool invocation, and autonomous decision flows.
- Validate AI-driven systems using deterministic testing approaches, mocks, simulators, and contract testing where applicable.
- Develop test strategies for LLM-based features, including prompt validation, response evaluation, guardrails, and failure modes.
- Implement API, integration, and system-level tests across microservices and AI orchestration layers.
- Collaborate with AI/ML engineers to test model integrations, inference pipelines, and agent execution frameworks.
- Set up and maintain CI/CD pipelines using GitHub Actions, enabling fast feedback and reliable releases.
- Analyze test results, derive quality KPIs, and clearly communicate risks and release readiness.
- Provide detailed root-cause and failure analysis to accelerate defect resolution.
- Participate in design and architecture reviews, advocating for testability, observability, and quality-first design.
- Mentor junior SDETs and QA engineers, raising the bar on automation, AI testing practices, and engineering excellence.
Skills you’ll bring
- 4+ years of experience as an SDET, Software Engineer, or Test Automation Engineer in SaaS environments.
- Strong programming experience in Python (preferred) and familiarity with JavaScript/TypeScript.
- Hands-on experience with Playwright (preferred) or modern UI automation frameworks.
- Solid experience testing API-first and microservices-based architectures.
- Practical understanding of AI/ML or LLM-based systems, including:
- Agentic AI concepts (agents, tools, planners, memory, orchestration).
- Deterministic vs non-deterministic testing strategies.
- Prompt-based systems and response evaluation.
- Experience with mocking, stubbing, and simulation to test complex integrations.
- Strong knowledge of REST APIs, server-side testing, and integration testing.
- Experience working with CI/CD pipelines, preferably GitHub Actions.
- Familiarity with Cloud technologies and containerized environments (Docker).
- Excellent written and verbal communication skills; comfortable working across geographies.
Nice to Have
- Experience testing Agentic AI frameworks (e.g., LangChain, LlamaIndex, or similar concepts).
- Knowledge of AI evaluation techniques (golden datasets, semantic similarity, rule-based validation, replay testing).
- Experience with communication platforms, contact centers, or VoIP technologies.
- Experience with performance, load, and reliability testing for AI-powered systems.
Technologies We Use
- Languages: Python, Java, JavaScript.
- Automation: Playwright, Selenium, TestNG.
- API Testing: REST Assured, Postman.
- CI/CD: GitHub Actions (GHA).
- Dev Tools: Git, GitHub, Jira.
- Infrastructure: Docker, Cloud-based services.
