Sr. AI Quality Engineer
full-time•United States
About this role
The Company:
Groundtruth is building a first-of-its-kind AI billing platform backed by one of the largest companies in the transportation and logistics industry. Our platform automates accuracy and trust in complex freight transactions by transforming messy, unstructured data – like complex email chains, documents, telematics, and contracts – into a clear, reliable system of record that drives accurate billing.
The Role:
We’re hiring a hybrid AI QA + Product Analyst to own end-to-end quality for our AI-powered inference system. This role sits at the intersection of LLM inference quality, event-driven backend state-machines, and freight domain logic.
You will define what “correct” means, build the quality measurement and regression approach to enforce it, and lead deep-dive investigations when edge cases or customer-specific rules break downstream behavior. The goal is to make our system more accurate, more diagnosable, and more reliable as email volume and customer complexity scales.
What you’ll do:
Own end-to-end system quality
- Develop and maintain a quality rubric for key use cases and exception types. (what “right” looks like, and what failure looks like).
- Build and curate golden datasets (representative emails + expected structured output + expected final outcome), including customer-specific variations.
- Own ongoing quality review in dev and production: regularly inspect high-volume outputs, diagnose what’s breaking and why, and convert discoveries into concrete roadmap items and regression coverage.
- Define and execute regression tests for new model changes, backend logic changes, or customer-specific use cases.
Investigate and diagnose issues across the full stack of the product
- Triage quality incidents and ambiguous failures by tracing through:
- email ingestion/parsing
- prompts / model outputs / normalization steps / data contracts
- intermediate structured representations
- event streams and state-machine transitions
- final audit exception generation and downstream reporting
- Use logs, traces, event histories, and data queries to isolate root cause.
- Produce high-signal findings reports: minimal reproduction, suspected component, evidence, impact, and recommended fix.
Build scalable quality operations
- Create a repeatable triage playbook and classification system for quality issues
- Define monitoring & dashboards for quality signals (volume anomalies, exception drift, per-customer error hotspots).
- Partner with engineering/AI to improve observability (correlation IDs, structured logging, traceability from email → state transitions).
Act as a product/domain translator
- Understand freight billing workflows and how real-world documents and communication map to our system’s model of “truth”.
- Convert customer-specific requirements into testable rules and expected outcomes.
- Identify systemic gaps where “reality” doesn’t fit the current schema, and propose product changes.
Required qualifications:
- Experience in roles that blend quality + investigation + systems thinking (examples: QA engineer in distributed systems, product analyst with deep debugging, LLM quality analyst, solutions engineer owning incident triage).
- Demonstrated experience evaluating AI/LLM output quality (extraction/classification, structured outputs, tool calling, RAG, prompt-driven pipelines, or similar).
- Strong technical ability to debug production issues using:
- log/trace tools (Datadog, ELK, Honeycomb, OpenTelemetry/Jaeger, etc.)
- SQL and/or Python for analysis and repro
- event-driven architectures and workflows/state machines (or similar distributed workflow systems)
- Ability to write crisp requirements and acceptance criteria, and translate ambiguity into test cases.
- Comfort operating in messy, high-volume, edge-case-heavy environments.
Nice-to-have qualifications:
- Freight/logistics/audit/billing domain experience (carrier invoices, accessorials, detention, lumper, fuel surcharge, tenders, BOLs, rate confirmations, PODs, etc.).
- Experience designing evaluation metrics (precision/recall, drift detection, per-customer or per-use-case scorecards).
- Familiarity with workflow engines/state machines and distributed systems failure modes (event ordering, retries, dedupe, idempotency, partial failure).
- Experience with annotation/labeling workflows, taxonomy design, and building human-in-the-loop QA processes.
Traits that matter in this role:
- High ownership: you don’t stop at “it’s broken,” you drive it to root cause and resolution.
- Comfortable with ambiguity and edge cases; systematic in building clarity.
- Able to communicate across product, engineering, ML, and operations.
Other facts
Tech stack
Quality Assurance,AI,LLM,Systems Thinking,Debugging,SQL,Python,Event-Driven Architectures,Data Analysis,Triage,Freight Billing,Monitoring,Observability,Communication,Root Cause Analysis,Regression Testing
About UP.Labs
UP.Labs is a first-of-its-kind venture lab unlocking the future of transportation and mobility. We work with global corporate partners to identify the most pressing challenges that they, and broader society, face. Inspired by these complex problems, we launch startups built by proven entrepreneurs, product leaders and technologists that use their agility and talent to develop transformative solutions. After these companies have matured and proven market fit, our corporate partners are able to acquire them, reaping strategic value while enriching their culture and core business. We believe this to be the shortest road to a faster, cleaner, safer, and more accessible future.
Team size: 11-50 employees
Website: https://up.partners/labs/
LinkedIn: Visit
Industry: Venture Capital and Private Equity Principals
What you'll do
- Own end-to-end system quality and develop a quality rubric for key use cases. Investigate and diagnose issues across the full stack of the product to ensure accuracy and reliability.
Ready to join UP.Labs?
Take the next step in your career journey
Frequently Asked Questions
What does a Sr. AI Quality Engineer do at UP.Labs?
As a Sr. AI Quality Engineer at UP.Labs, you will: own end-to-end system quality and develop a quality rubric for key use cases. Investigate and diagnose issues across the full stack of the product to ensure accuracy and reliability..
Why join UP.Labs as a Sr. AI Quality Engineer?
UP.Labs is a leading Venture Capital and Private Equity Principals company.
Is the Sr. AI Quality Engineer position at UP.Labs remote?
The Sr. AI Quality Engineer position at UP.Labs is based in United States, United States. Contact the company through Clera for specific work arrangement details.
How do I apply for the Sr. AI Quality Engineer position at UP.Labs?
You can apply for the Sr. AI Quality Engineer position at UP.Labs directly through Clera. Click the "Apply Now" button above to start your application. Clera's AI-powered platform will help match your profile with this opportunity and guide you through the application process. You can also learn more about UP.Labs on their website.