Senior Software Engineer - AI Code Testing & Evaluation
Varna, Varna, Bulgaria · Remote Solely
Senior
Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving…
Job Description: About DXC Bulgaria We are DXC – a Fortune 500 global IT services leader. In Bulgaria, we are among the largest IT employers, delivering the full spectrum of the company’s diverse technical portfolio. Our…
Skills: Linux, Windows, CI/CD, Git, Infrastructure as Code
Senior Data Engineer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
At myPOS, we’re all about helping businesses grow and get paid. We make payments simple, smart, and accessible for everyone, but we’re more than just payment solutions - myPOS is a partner in growth. From free multicurre…
At myPOS, we’re all about helping businesses grow and get paid. We make payments simple, smart, and accessible for everyone, but we’re more than just payment solutions - myPOS is a partner in growth. From free multicurre…
Data Architect - Data & AI Platforms (Databricks Focus) - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Senior+
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Skills: Databricks, Apache Spark, PySpark, Delta Lake, Data Architecture
Data Architect - Data & AI Platforms (Databricks Focus) - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Senior+
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Skills: Databricks, Apache Spark, PySpark, Delta Lake, Data Architecture
Salesforce CPQ Specialist - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Mid level
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Salesforce CPQ Specialist - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Mid level
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Salesforce Engineer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Mid level
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Skills: Salesforce Development, Salesforce Field Service, LWC, Salesforce APIs, DevOps
Salesforce Engineer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Mid level
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Skills: Salesforce Development, Salesforce Field Service, LWC, Salesforce APIs, CI/CD
Senior Network Engineer (Messaging) – Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Hybrid
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026), is looking for a highly skilled and motivated Senior Network Engineer (Messaging), a professional who will join a project within a Telecom Team for ou…
Senior Network Engineer (Messaging) – Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Hybrid
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026), is looking for a highly skilled and motivated Senior Network Engineer (Messaging), a professional who will join a project within a Telecom Team for ou…
Senior Java Developer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Hybrid
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Senior Java Developer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Hybrid
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. With a strong heritage of Nordic …
Senior Data Engineer - Tieto Tech Consulting (m/f/d)
Varna, Varna, Bulgaria · Remote OK
Senior
Job Description Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide. Key responsibilities Design and d…
About us: Perfomalis specializes in performance marketing services, creating innovative market-leading solutions. Focused on cutting-edge digital strategy in web development and content marketing, the company’s aim is to…
Skills: Data Engineering, Data Architecture, Database Design, Data Warehousing, Sql
About us: Perfomalis specializes in performance marketing services, creating innovative market-leading solutions. Focused on cutting-edge digital strategy in web development and content marketing, the company’s aim is to…
Skills: Data Engineering, SQL, Python, Data Warehousing, Big Data Technologies
At myPOS, we’re all about helping businesses grow and get paid. We make payments simple, smart, and accessible for everyone, but we’re more than just payment solutions - myPOS is a partner in growth. From free multicurre…
At myPOS, we’re all about helping businesses grow and get paid. We make payments simple, smart, and accessible for everyone, but we’re more than just payment solutions - myPOS is a partner in growth. From free multicurre…
Sign up with Clera and we'll reach out the moment a role actually fits you — no more spraying applications into the void.
Part-time
Posted 33d ago
~40 hrs/week
Remote in Varna, Varna, Bulgaria
Responsibilities
Create challenging tasks and evaluation criteria to test AI coding agents within simulated developer environments. Write and iterate on tests to verify agent solutions and ensure robust evaluation of AI performance.
Requirements
Requires over 5 years of software development experience with a core stack including Python, JavaScript/TypeScript, and various infrastructure tools. Must possess strong English proficiency (B2+) and experience in functional and integration testing.
Full job description
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based.
What this opportunity involves
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What this is NOT
Not data labeling
Not prompt engineering
Not writing code from scratch - the agent writes most of the code; you guide and evaluate
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Effort estimate
Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.
Compensation
Up to $40/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.
Related keywords
AI Coding AgentsPythonFastAPIJavaScriptTypeScriptReactDockerPostgresKafkaRedisFunctional TestingIntegration TestingAI EvaluationDataset BuildingSoftware DevelopmentSimulated Environments
Be the “I” in AI with Mindrift. Work from anywhere on groundbreaking AI projects
Industry
Technology, Information and Internet
Company size
501-1,000 employees
LinkedIn followers
825,343
Mindrift (mindrift.ai) is the expert-sourcing platform built by Toloka, a Nebius Group (Nasdaq: NBIS) company and a global leader in data services for AI since 2014. Mindrift’s CEO and co-founder, Olga Megorskaya, launched the platform to connect domain experts — from software engineers and physicians to linguists and lawyers — with AI training projects for the world's leading technology companies.
Through Mindrift, over 10,000 professionals across 50+ domains and 15+ languages contribute to supervised fine-tuning, reinforcement learning from human feedback (RLHF), model evaluation, and red-teaming for frontier AI models developed by leading global technology companies.
Operated by Toloka from Amsterdam, Netherlands, Mindrift offers flexible, fully remote, project-based opportunities — empowering experts to shape the future of generative AI, one task at a time.