Validation Engineer, AI & Data

• Design and implement the testing strategy for the Domain Context Model, Business Context Repository, ontology mappings, and AI response evaluation.
  
• Develop representative ground truth question and answer test sets with defined expected results and tolerances, validated by data owners.
  
• Build and maintain automated evaluation tests for AI responses generated through the client’s Context Injection toolkit.
  
• Evaluate AI responses with and without domain context to establish baseline accuracy and measure improvements.
  
• Validate numeric answers, SQL and query results, business logic, aggregations, join relationships, and narrative responses.
  
• Identify and classify failures, including hallucinations, stale information, incorrect schemas, faulty joins, missing business context, and persona mismatches.
  
• Develop repeatable evaluation scripts, scoring rubrics, regression tests, and accuracy reporting dashboards or scorecards.
  
• Verify context repository data completeness, classification consistency, access control behavior, and changes introduced through automated context updates.
  
• Document test evidence, defects, remediation results, AI readiness assessments, and validation outcomes.
  
• Support release readiness, weekly demonstrations, and handover of reusable evaluation frameworks and documentation.
  

Requirements