Talent.com
Gramian Consulting Group
AI Evaluation Engineer - Planning & OperationsGramian Consulting Group • NG
AI Evaluation Engineer - Planning & Operations

AI Evaluation Engineer - Planning & Operations

Gramian Consulting Group • NG
16 days ago
Job type
  • Quick Apply
Job description

About Us

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

Role overview

We are looking for an AI Evaluation Engineer specialized in planning and operations to design and build benchmark tasks that simulate real-world scenarios such as scheduling, logistics, and resource allocation.

This role focuses on planning, scheduling, and operational optimization problems, where multiple agents must collaborate to solve constraint-rich scenarios involving resources, timelines, and dependencies.

Commitments Required: 8 hours per day with an overlap of 4 hours with PST.

Employment type: Contractor assignment (no medical/paid leave)

Duration of contract: 4 weeks+

Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam

Interview: take home assessment (60min) + short interview

Responsibilities

  • Design and build multi-agent benchmark tasks involving:
    • Planning, scheduling, and resource allocation
    • Operational decision-making (logistics, project planning, incident response, capacity planning)
  • Create constraint-rich problem statements with multiple interacting variables
  • Develop verification scripts to evaluate:
    • Feasibility (all constraints satisfied)
    • Completeness (all requirements met)
    • Optimality (efficiency of solutions)
  • Define task decomposition strategies across specialized sub-agents (e.g., resource allocation, constraint resolution, optimization)
  • Model realistic operational systems with dependencies, timelines, and constraints
  • Implement validation logic and evaluation pipelines using Python
  • Work with Docker environments for reproducibility and execution
  • Collaborate with internal teams to improve task quality, coverage, and evaluation rigor

Requirements

  • 5+ years of experience in operations, project management, logistics, or supply chain
  • Strong ability to formalize constraints, dependencies, and scheduling logic
  • Proficiency in Python for building validation and verification scripts
  • Experience with optimization techniques (linear programming, constraint satisfaction, scheduling algorithms)
  • Strong structured problem-solving and decomposition skills
  • Experience with AI benchmarks or evaluation frameworks (e.g., SWE-bench or similar)
  • Hands-on experience with Docker (Dockerfiles, image builds, debugging)

Nice to Have

  • Background in operations research or optimization-heavy domains
  • Experience with simulation or modeling tools
  • Familiarity with AI planning systems or automated reasoning
  • Project management experience or certifications (PMP, Agile, etc.)
Create a job alert for this search

AI Evaluation Engineer - Planning & Operations • NG

Similar jobs

Creative AI Designer

Hire OverseasNG
Quick Apply

AI creative tools and open to learning and integrating them into their workflow.The core skills are video editing, motion design, and graphic design.Experience with AI creative tools is a plus but ... Show more

AI Evaluation Engineer (Knowledge & Research)

Gramian Consulting GroupNG
Quick Apply

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions.With a strong background in software engineering and leadership, we help comp... Show more

Research Scientist - Computational Mathematics (Scientific Coding / AI Evaluatio

Gramian Consulting GroupNG
Quick Apply

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions.With a strong background in software engineering and leadership, we help comp... Show more

59418523934 - On-Call AI Video Editor

Activate TalentNG
Quick Apply

We’re looking for a Video Editor who thrives at the intersection of creativity, speed, and data.This is not a traditional editing role.You will be responsible for producing high-converting, direct ... Show more

Senior DevOps Engineer (Africa)

Talent HackersNG
Quick Apply

Job Summary: We are seeking a skilled Senior DevOps Engineer with strong AWS expertise to build and maintain secure, scalable infrastructure, CI/CD pipelines, observability systems, and multi-stage... Show more

AI Prompt Engineer (Nigeria)

Talent HackersNG
Quick Apply

Job Overview: About the role: We’re hiring an AI Prompt Engineer who knows how to turn AI into real operational leverage.This is not a “casual AI user” role.We need someone who un... Show more

AI Engineer (Full-Stack)

Hire OverseasNG
Quick Apply

AI systems using Claude Code as a primary development tool.This role is for someone who combines strong full-stack engineering fundamentals with hands-on experience shipping AI-powered products, au... Show more

Digital Analytics Specialist (MarTech)

Hire OverseasNG
Quick Apply

This role goes well beyond basic pixel setup.It requires strong technical judgment, QA discipline, and comfort working directly with code when needed.You'll partner closely with account managers an... Show more

AI & Automation Specialist (n8n & ElevenLabs)

Remote VANG
Quick Apply

We are seeking a highly creative and technically proficient.AI, specifically focusing on voice synthesis and conversational AI.In this role, you will be responsible for turning manual bottlenecks i... Show more

DevOps Engineer

OpenFnNG
Quick Apply

Full-time contractor (hours negotiable).Remote - Africa/Europe time zones (strong preference for candidates based in Africa).Our platform automates data exchange and digital workflows so that gover... Show more

Vision Data Intelligence Specialist (Worldwide)

Toloka AnnotatorsNG
Quick Apply

The Toloka Annotators connects individuals with Generative AI projects from leading tech innovators.Our mission is to unlock the full potential of AI by involving real people from around the world ... Show more

AI Training Specialist (Global Egocentric Video)

Toloka AnnotatorsNG
Quick Apply

The Toloka Annotators connects individuals with Generative AI projects from leading tech innovators.Our mission is to unlock the full potential of AI by involving real people from around the world ... Show more

AI Evaluation Engineer - Mathematics & Algorithms

Gramian Consulting GroupNG
Quick Apply

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions.With a strong background in software engineering and leadership, we help comp... Show more

Web Analytics Developer

Hire OverseasNG
Quick Apply

This role goes well beyond basic pixel setup.It requires strong technical judgment, QA discipline, and comfort working directly with code when needed.You'll partner closely with account managers an... Show more

Executive Assistant (Claude AI)

Hire OverseasNG
Quick Apply

AI-Powered Executive Assistant.This is a high-ownership, all-rounder role that blends executive support, operations management, and strategic assistance into one position.You will use Claude AI, Cl... Show more

AI Engineer

Hire OverseasNG
Quick Apply

Reasonable Magic is an AI product development studio.We bring together business strategy, design thinking, and AI engineering to build intelligent software systems with assessable business value at... Show more

Biology Experts - AI Training

Gramian Consulting GroupNG
Quick Apply

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions.With a strong background in software engineering and leadership, we help comp... Show more

Machine Learning Engineer

KoraNG
Quick Apply

Kora is the marketplace for everything payments.We offer a robust payment API for payment collections, disbursements, and conversions for businesses anywhere in Africa.Our vision, which is at the c... Show more