QA Engineer - Gen AI

Company Overview:

Sustainment is a software company that builds Supplier Discovery and Supplier Relationship Management (SRM) solutions for the manufacturing industry. Sustainment helps procurement and supply chain teams in manufacturing companies solve sourcing and procurement challenges to confidently build and manage their supply chains and get work done faster with manufacturing suppliers. We help manufacturers and suppliers get work done together more easily, leveraging software and AI to remove much of the friction that harms productivity. Sustainment has become the solution of choice for manufacturers to work with thousands of suppliers across the country.

Job Overview:

We are seeking a QA Engineer to help ensure the reliability, accuracy, and robustness of our AI Agents. This role will focus on data quality, model evaluation, and regression testing frameworks to identify and mitigate common LLM failure modes. You will be responsible for designing automated and scalable quality assurance systems while working in an AWS-based infrastructure. If you have a strong background in LLM testing, data validation, and automated QA frameworks, this role is an excellent opportunity to contribute to cutting-edge AI systems.

Responsibilities: 

• Develop and maintain regression test frameworks for LLM evaluation.
• Identify and track LLM failure modes, including hallucinations, biases, factual inconsistencies, and logical errors.
• Design data quality assessment pipelines to evaluate training and test datasets.
• Automate LLM performance monitoring with advanced metrics and validation strategies.
• Implement best practices for prompt engineering testing, fine-tuning validation, and output consistency analysis.
• Collaborate with ML engineers, data scientists, and product teams to align on quality benchmarks.
• Work within an AWS ecosystem, leveraging services such as S3, SageMaker, Lambda, and ECS for model testing and deployment.
• Develop tools and dashboards for LLM quality tracking over time.

Qualifications:

• 3+ years in software testing, quality assurance, and ML evaluation, with a focus on NLP, LLMs, VLMs, etc.
• Familiarity with RAG and RAGAS.
• Deep understanding of LLM data quality challenges and common failure modes.
• Experience in building automated testing frameworks for AI/ML models.
• Familiarity in Python and testing frameworks such as PyTest, Hypothesis, or similar.
• Familiarity with AWS cloud services (S3, Lambda, ECS, Step Functions, SageMaker).
• Knowledge of evaluation metrics for LLMs (BLEU, ROUGE, perplexity, factual consistency).
• Hands-on experience with automated data validation techniques.
• Strong debugging and analytical skills.

Preferred Qualifications

• Experience with LLM fine-tuning and prompt engineering.
• Understanding of human-in-the-loop (HITL) evaluation strategies.
• Familiarity with LLM APIs (OpenAI, Anthropic, Cohere, or similar).
• Background in statistical analysis or model interpretability.
• Experience with MLOps practices and CI/CD pipelines for ML models