Senior Python Engineer - AI Coding Agent Evaluation (Freelance)
MindriftJob Description
About Mindrift
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What This Opportunity Involves
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. You'll create challenging tasks and evaluation criteria within realistic simulated environments.
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What This Is NOT
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What We Look For
- 8+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Why This Is Hard
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How It Works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Effort Estimate
Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.
Compensation
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.
How to Apply
Please submit your CV in English and indicate your level of English proficiency.
Get notified of similar jobs
We'll send you an email when jobs similar to "Senior Python Engineer - AI Coding Agent Evaluation (Freelance)" are posted.
Related Jobs You Might Like
View all jobs →
Supervisor - WH
LANDMARK GROUP
About Landmark GroupOur journey started in 1973 with a single store in Bahrain. Since then, we have grown into a global retail and hospitality group. The proud creator of 25 plus value led, own-built brands across retail, hospitality, food, and leisure.Over the years, our UAE - based group has evolved into a comprehensive retail and hospitality entity, with over 2200 retail stores, leisure, and hospitality outlets, panning the GCC, Middle East, India, Southeast Asia, and Africa. Today, Landmark Group is one of GCC's largest omnichannel retailers and India's top home and fashion retailer.We take immense pride in the organic growth of our retail brands, which have blossomed into household names. From Max, Splash, Babyshop, Centrepoint, Shoemart, Homecentre, Emax, Fitness First, to Funcity, to name a few, our brands span across a multitude of categories, enriching the lives of countless families over the past decade....Job SummaryLandmark Group is seeking an experienced and dedicated Warehouse Supervisor to join our team in Bahrain. The ideal candidate will be responsible for overseeing daily warehouse operations, ensuring efficient inventory management, leading a team of warehouse staff, and maintaining high standards of safety and productivity.Key ResponsibilitiesOversee daily warehouse activities, including receiving, storage, picking, packing, and dispatch of goods.Lead, motivate, train, and supervise a team of warehouse staff to achieve operational targets.Implement and enforce robust inventory control procedures to ensure accuracy and minimize discrepancies.Ensure strict adherence to all health, safety, and environmental regulations within the warehouse.Coordinate with logistics, procurement, and other internal departments to optimize workflow and resolve issues.Monitor warehouse performance metrics, prepare reports, and identify areas for improvement.Manage warehouse equipment maintenance and ensure optimal operational efficiency.Conduct regular stock checks and audits.Qualifications & ExperienceHigh School Diploma or equivalent; a Diploma or Degree in Logistics, Supply Chain Management, or a related field is a plus.Minimum of 2-3 years of proven experience in warehouse operations, with at least 1 year in a supervisory role.Strong knowledge of warehouse management systems (WMS) and inventory control principles.Excellent leadership, communication, and interpersonal skills.Ability to work effectively in a fast-paced environment and manage multiple priorities.Proficiency in Microsoft Office Suite (Excel, Word).BenefitsCompetitive salary package.Comprehensive health insurance.Annual leave as per company policy.Employee discounts across Landmark Group brands.Opportunities for professional growth and career development.
Warranty Administrator – Automotive
Urban Ridge Supplies
Job SummaryWe are hiring a Warranty Administrator – Automotive to manage the administrative coordination of automotive warranty claims and ensure timely, accurate submissions. The role will act as a key link between the workshop, service teams, customers, manufacturers, and suppliers throughout the claims process.This position is suited to someone with 2+ years of automotive warranty experience who is highly organized, detail-oriented, and comfortable following up on multiple claims and resolving documentation issues.Key ResponsibilitiesCoordinate the preparation, submission, and processing of vehicle warranty claims.Verify claim details, repair documentation, invoices, and supporting information before submission.Maintain an accurate and organized warranty claims database and supporting records.Communicate with manufacturers and suppliers regarding claim approvals, queries, and rejections.Track open claims and proactively follow up to ensure timely resolution.Keep customers informed about warranty claim progress where required.Coordinate with technicians, workshop staff, and service advisors to obtain accurate claim information.Ensure warranty submissions comply with relevant manufacturer policies and procedures.Flag inconsistencies, incomplete documentation, unusual claims, or potential fraudulent activity.Support the team in improving warranty administration workflows and reducing claim errors.Compile basic reports covering claim volumes, pending claims, rejections, and other warranty activity.RequirementsHigh school diploma or equivalent.2+ years of experience in automotive warranty administration, claims processing, or a similar role.Solid knowledge of automotive warranty processes and manufacturer requirements.Excellent attention to detail and documentation accuracy.Strong organizational and follow-up skills.Good analytical and problem-solving abilities.Strong written and verbal communication skills.Proficiency in Microsoft Office and automotive warranty/service systems.Ability to prioritize multiple claims and meet strict deadlines.Key SkillsWarranty CoordinationAutomotive ClaimsClaims AdministrationManufacturer LiaisonClaim DocumentationWorkshop CoordinationCustomer SupportReportingProblem Solving
Warranty Claims Officer
TradesApp
SummaryWe are seeking a detail-oriented Warranty Claims Officer to manage the end-to-end processing of automotive warranty claims and ensure compliance with manufacturer requirements. The successful candidate will work closely with customers, manufacturers, suppliers, and internal automotive teams to ensure claims are accurately documented, processed efficiently, and resolved within the required timelines.Key ResponsibilitiesManage warranty claims from initial submission through approval, resolution, and closure.Verify claim information, supporting documents, and repair details for accuracy and completeness.Maintain comprehensive and up-to-date records of all warranty claims.Liaise with manufacturers and suppliers regarding claim approvals, rejections, documentation, and outstanding issues.Provide customers with timely and professional updates regarding their warranty claims.Coordinate with service advisors, technicians, and other automotive team members to obtain required information and documentation.Ensure all claims comply with applicable manufacturer warranty policies and procedures.Monitor claims for discrepancies, inconsistencies, or potential fraudulent activity.Escalate complex or disputed claims to the appropriate stakeholders.Support the implementation and improvement of warranty claim procedures and workflows.Stay informed about changes to warranty policies, processes, and manufacturer requirements.QualificationsHigh school diploma or equivalent required.At least 2 years of experience in automotive warranty claims or a similar automotive support function.Good knowledge of automotive warranty processes, policies, and claim documentation.Strong attention to detail and ability to identify errors or discrepancies.Excellent organizational and time-management skills.Strong analytical, communication, and interpersonal skills.Proficient in Microsoft Office and relevant business or warranty management software.Ability to work effectively in a fast-paced automotive environment.Strong customer-service orientation and ability to manage multiple cases simultaneously.