# QA Agentic Engineer - Company: [Coderoad]() - Location: Latin America - Team: CodeRoad - Posted: August 17, 2026 ## Job description ### ## **About CodeRoad** CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our nearshore technology services empower businesses to thrive in an ever-evolving digital landscape. ## **About the Role** We are looking for a **Junior AI Agent Quality Engineer** to help us build and validate the next generation of autonomous systems. In this role, you won't just be checking for bugs; you'll be evaluating the "brain" of our AI. You will focus on the reliability of **Agentic AI** and **RAG (Retrieval-Augmented Generation)**. You’ll be responsible for ensuring our agents can plan tasks, use tools accurately, and recover from errors without "hallucinating." This is a perfect role for someone with a QA mindset who is passionate about Python and the future of LLM-based orchestration. ## **Key Responsibilities** * **Agent Evaluation:** Create question templates and Python scripts to test how well AI Agents and RAG instances solve complex tasks. * **KPI Tracking:** Evaluate Agent performance using specific metrics: **Success Rate, Tool Use Accuracy, Planning Quality, and Autonomy**. * **Dataset Creation:** Collaborate with AI Engineers to build "Ground Truth" datasets and **Agentic Task Datasets** to benchmark model improvements. * **API & Logic Testing:** Write unit and integration tests for Python-based RESTful APIs and agent endpoints. * **Performance Testing:** Run load and bulk testing (using **Locust or JMeter**) to see how our AI handles high-volume requests. * **Input/Output Validation:** Perform rigorous testing of agent payloads to catch prompt injection risks and logic failures. --- ## **Requirements** * **Agentic AI Focus:** Basic understanding of **Agentic workflows**, prompt engineering (ReAct prompts), and LLM orchestration. * **AI Observability:** Familiarity with (or a strong desire to learn) tools like **LangSmith, Langfuse, or OpenTelemetry**. * **Python Skills:** Proficiency in **Python 3.10+** for automation and data manipulation. * **API Fundamentals:** Strong experience testing and validating RESTful APIs and JSON structures. * **Analytical Thinking:** A "breaker" mindset—the ability to find edge cases where an AI might fail to follow instructions. * **Language:** Advanced English (B2/C1) for global team collaboration and technical documentation. --- ## **Nice to Have** * **QA Experience:** 1–3 years of experience in Software QA (Manual or Automation). * **Automation Tooling:** Exposure to **Pytest**, Playwright, or Selenium. * **AI Frameworks:** Exposure to **LangChain, LangGraph, or Pydantic AI**. * **Infrastructure:** Basic knowledge of **Docker** and Vector Databases. * **Performance:** Experience with **Locust or JMeter** for bulk testing. ## **What You’ll Love** * 100% Remote * Holidays off * Paid Time Off * Health insurance assistance program * Competitive USD compensation * Strong team culture and collaborative environment * Ongoing training and growth opportunities ### ## Apply [Apply on Coderoad]() Canonical job page: