# AI Quality Analyst

**Company:** [Texas Sports Academy Main](http://jobs.workable.com/companies/msWwuDKBiDwbttWn9TomUh.md)
**Location:** Remote
**Workplace:** remote

[Apply for this job](http://jobs.workable.com/view/8ffad75e-e3fa-42a1-a758-25fc74343718)

## Description

We run an AI-powered SMS bot and an AI-graded call training system for a private school

network. Both systems have extensive test suites (400+ test files) and a RAG-based knowledge

base that drives bot behavior

We need someone to own the full quality cycle:

\- Review production conversations to find bad bot responses

\- Create test files (JSON) that reproduce the issue

\- Fix the knowledge base content (Markdown files, WRONG/RIGHT examples)

\- Run test suites to verify fixes and catch regressions

\- Maintain and expand the knowledge base as our program evolves

This is NOT a software engineering role. The infrastructure is built. You're operating it — editing

JSON, writing Markdown, running CLI scripts, reading test reports.

What You'll Work With

\- JSON test files (you'll write and edit these daily)

\- Markdown knowledge base documents

\- Terminal commands (copy-paste and run scripts)

\- Git (commit, push, basic branching)

\- VS Code or similar editor

## Requirements

\- Native-level English fluency (non-negotiable). You'll be writing realistic SMS conversations

between parents and our school. The language has to sound like a real person texting, not a

corporate chatbot. You'll also be writing precise pass/fail evaluation criteria.

\- Comfortable editing JSON and Markdown in an IDE

\- Can run commands in a terminal without hand-holding

\- Extreme attention to detail

\- Ability to learn a complex domain quickly (education, state government programs, etc)

Nice-to-Haves

\- Experience writing test cases or QA documentation

\- Experience with chatbot QA, conversational AI testing, or LLM evaluation

Hours & Ramp-Up

\- 30 hrs/week to start, ramping up as needed. US time zones required.

\- The first 1-2 weeks will be focused on learning our domain — how our school works, how the

state voucher program works, compliance rules, etc. We have extensive internal documentation.