# Senior Test Analyst

**Company:** [Smartstream Limited](http://jobs.workable.com/companies/xrHunSPFFf6yxYNqT477c2.md)
**Location:** Mumbai, India
**Workplace:** hybrid
**Department:** 610-Quality Assurance

[Apply for this job](http://jobs.workable.com/view/faeb7344-5e03-41dd-a830-37d9bb787d26)

## Description

The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.

The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

### Job Responsibilities

-   Write automation alongside development to test unit level, API and e2e UI.
-   Implement backend services supporting agent runtimes, tool execution, and memory layers
-   Integrate LLMs into business workflows and internal systems
-   Contribute to orchestration logic connecting tools, APIs, and external services
-   Write clean, maintainable, well-tested code
-   Participate in architectural discussions
-   Collaborate with AI engineers to productionize experimental features
-   Contribute to monitoring, logging, and performance optimization

## Requirements

-   Strong hands-on experience testing LLM-based or agentic AI applications.
-   Expertise in Python automation (PyTest, Playwright, Postman).
-   Understanding of prompts, embeddings, vector stores, tokenization. • Experience testing RAG pipelines and grounding data workflows.
-   Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP).
-   Ability to evaluate unstructured AI output using structured scoring techniques.
-   Strong analytical and debugging skills in complex AI-driven workflows.

### Desirable Skills

-   Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;
-   Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI.
-   Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming.
-   Knowledge of model fine-tuning, RLHF, and observability pipelines.
-   Exposure to distributed architecture and enterprise platforms.
-   Ability to mentor QA engineers on AI testing methods.

### Qualifications

-   5+ years of professional software engineering experience
-   Hands-on work with generative AI and agentic systems.
-   Experience working in cross-functional product teams
-   Degree in Computer Science, Engineering, or related field (or equivalent experience)
-   Strong problem-solving skills and ability to work in fast-evolving technical domains
-   Interest in AI-native software architectures and continuous learning mindset
-   Certifications such as ISTQB Advanced, Azure/AWS/Google AI certifications (optional).

### Equality Statement

_SmartStream is an equal opportunities employer. We are committed to promoting equality of opportunity and following practices which are free from unfair and unlawful discrimination._