Prompt Engineer & AI Evaluator

📋 Role Overview

Design, test, and optimize system prompts for enterprise LLMs, conducting systematic evals for accuracy, safety, and output consistency.

🎯 Key Deliverables & Responsibilities

• Craft high-level system prompts (few-shot, chain-of-thought, structured JSON output) for specialized domain use cases.
• Build robust eval datasets to test model hallucinations, factual correctness, tone, and formatting constraints.
• Perform comparative benchmarking across models (OpenAI GPT-4o, Claude 3.5 Sonnet, Llama 3, DeepSeek R1).
• Optimize prompt length and token usage for maximum cost and latency efficiency.
• Establish AI safety guidelines and guardrails against prompt injections.

🎓 Qualifications & Requirements

• Indonesian citizen residing in Indonesia (100% Full Remote position).
• 1+ years of systematic prompt engineering or AI evaluation experience in production environments.
• Fluent/Native-level English proficiency (C1-C2) as the majority of prompting and evals are in English.
• Sharp analytical and critical thinking skills to detect subtle LLM errors or hallucinations.
• Basic understanding of JSON, Markdown, and Python scripting.

✨ Preferred Qualifications (Nice-to-Have)

• Experience with evaluation platforms like Braintrust or Ragas
• Linguistics or CS background