Generative AI Analyst - California
Welo Data is a global transformation partner that specializes in multilingual content transformation services. They are seeking a Generative AI Analyst to support the evaluation and quality review of AI-generated videos, focusing on identifying mismatches and visual errors in video content compared to text prompts.
Responsibilities
- Review text prompts and corresponding AI-generated video clips to identify mismatches, inconsistencies, and visual errors
- Tag specific parts of the prompt that do not match the video by identifying only the incorrect prompt text, rather than the full sentence unless needed
- Add visual or behavioral errors as annotation instances in annotation platform
- Write brief, clear descriptions for each error instance
- Avoid duplicate annotations by ensuring errors already captured as prompt mismatches are not added again as separate instances
- Evaluate video content for actions, context, motion, scene consistency, object behavior, and prompt alignment
- Apply project guidelines consistently across repetitive, detail-oriented tasks
- Use critical thinking and judgment to handle ambiguous scenarios and determine the most accurate ground truth
- Identify recurring or systematic errors across tasks and document examples for review
- Perform self-QA on completed work and correct errors before submission
- Participate in calibration sessions to align interpretation of guidelines and reduce annotator-to-annotator variance
- Incorporate feedback from quality reviews into subsequent work
- Support throughput and quality targets while maintaining accuracy at scale
Skills
- Experience in video annotation, multimedia annotation, content quality review, data labeling, computer vision labeling, Generative AI evaluation, or a closely related field
- Strong attention to detail and ability to identify subtle errors, mismatches, and inconsistencies between prompts and videos
- Solid understanding of US driving rules, road behavior, traffic scenarios, and roadway conventions
- Good reading comprehension and ability to accurately compare written prompts against visual outputs
- Strong written communication skills in English, with the ability to describe errors clearly and concisely
- Ability to follow detailed guidelines consistently and maintain high accuracy across repetitive tasks
- Critical thinking and sound judgment when reviewing ambiguous or complex scenarios
- Basic understanding of robotics behavior in real-world environments
- Comfortable working in structured annotation platforms or similar tools
- Ability to maintain focus and quality while reviewing multimedia content for extended periods
- Prior experience evaluating AI-generated video, synthetic media, autonomous driving data, robotics scenarios, or traffic-related visual content
- Familiarity with US traffic rules, road signage, lane behavior, vehicle interactions, pedestrian behavior, and common driving scenarios
- Experience with QA, audit, or second-pass review workflows, including calibration, sampling, defect tracking, or error taxonomy development
- Ability to identify recurring model failure patterns across tasks or batches
Benefits
- Remote with the option to work onsite in Santa Clara, CA area.
- Employment Type: W2 Full-Time Employee
Company Overview