# 🏖️Agent Evals Specialist - Job Code: 1778280366787

**Company:** [Coconut](http://jobs.workable.com/companies/x6z38HwsPLkhT5xJjKb7BJ.md)
**Location:** Remote
**Workplace:** remote

[Apply for this job](http://jobs.workable.com/view/077d6542-e1fd-477f-a06b-c8a6b3d67e33)

## Description

At **Coconut 🥥**, we connect founders and businesses with top-tier remote talent while prioritizing meaningful, reliable partnerships. We believe success happens when everyone wins. Clients achieve their goals, and our virtual professionals find purpose and growth in their work. We're currently looking for **top talent** to fill the role below:

**  
Role**: Agent Evals Specialist

**Monthly Rate**: $900-1450

**Availability**: 40 hours/week

**Time Zone**: Preferred overlap anytime during 10AM to 10 PM PST

\==================================================

To streamline your application, please carefully review and complete **ALL THREE** items below before proceeding to the application form.

**What to Prepare:**

1️⃣ **English Proficiency Rating** – You will need to enter your CEFR English proficiency level in the application form. If you’re unsure of your rating, you can take a quick assessment here: [**EF SET Quick Check**](https://www.efset.org/quick-check/).

2️⃣ **Coconut VA Profile (Resume)** – Before submitting your application, please create your Coconut VA profile using this template: [**Coconut VA Profile Template**](https://bit.ly/CVAProfileTemplate). ⚠️⚠️⚠️ **YOUR RESUME MUST FOLLOW THIS FORMAT AND SHOULD BE IN ENGLISH.** **This is a test of how keen you are with details and how well you follow instructions. ⚠️⚠️⚠️**

3️⃣ **1-Minute Introduction Video** – Record a short introduction video (preferably using [**Loom**](https://www.loom.com/)) and provide the link in your application. Before you do, check out these [**video tips**](https://bit.ly/introvideotips) and [**sample videos**](https://bit.ly/IntroVidGuideAndSamples). **Important:** By submitting your video, you grant Coconut permission to share it on our platforms to help increase your chances of being matched with a client. Videos with broken or dummy links will result in disqualification.

## Requirements

**Experience Needed**:

1\. Read the source and the AI agent's output side by side. Verify the content was captured accurately.

2\. Review what the AI agent did. What it created, changed, or left out.

3\. Score a short rubric covering accuracy, coverage, organization, and rule adherence. Full rubric provided at onboarding.

4\. Write detailed feedback about the mistake. This is the most important thing you produce since we use it to improve the agent.

5\. Submit. Move to the next task.

**Good to have:**

• Prior work as an AI trainer, tutor, or evaluator (Outlier, DataAnnotation, xAI, Surge, Mercor, Invisible, Toloka, etc.)

• Technical writing, editing, QA, translation, paralegal, or research-assistant background

• Markdown familiarity

**Non-negotiable Requirements:**

\- Strong written English. Can read dense technical content for hours without losing focus

\- Consistent — your scoring on Monday matches your scoring on Friday

\-  Clear, specific feedback:"section 4 dropped the key requirement from page 17", not "this is confusing"

**Industry/Tools Experience:**

\- Slack and internal platform

## Benefits

We’re all about uplifting lives in Coconut and we make sure we provide benefits that support work-life balance, well-being, and growth. Here’s what you can look forward to as part of the Coconut Community! ✨

✅ **Competitive Salary** – We recognize your skills and experience with a great compensation package.  
✅ **13th Month Pay** – Enjoy an extra month’s salary—because who doesn’t love a little bonus?

✅ **Paid Time Off** – Recharge with **12 days of PTO** to use however you like—vacation, self-care, or just a break from it all!  
✅ **US Holidays Off** – Enjoy **12 paid US holidays** throughout the year.  
✅ **Maternity & Paternity Leave** – We support growing families with dedicated leave for new parents.

✅ **Comprehensive Healthcare** – We’ve got you and your family covered with quality healthcare benefits.  
✅ **Life Insurance**\* – Our full-time HMO plan includes life insurance for extra peace of mind.  
✅ **Mental Health Support** – Prioritize your well-being with access to mental health consultations and wellness resources.

✅ **Milestone Gifts & Birthday Treats** – We celebrate your achievements and make your special day even better!  
✅ **Exciting Team Experiences** – Think **island getaways, virtual town halls, and regional meetups**—because work should be fun, too!

✅ **Stay Connected** – Be part of a supportive network where you can learn, grow, and connect with amazing people.

_Some benefits may vary. Full details will be provided during the onboarding process._