AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards. If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you! WHAT YOU WILL DO - Architect and oversee end-to-end test strategies across different layers of the testing pyramid; - Define, evaluate, and monitor model validation criteria, including accuracy thresholds, confidence scoring, fallback behaviors, and more; - Drive automation of QA processes, ensuring robust, scalable solutions for data ingestion, extraction, and classification; - Lead validation of integration points between AI components and business logic, identifying risks and ensuring seamless operation; - Own, maintain, and evolve test infrastructure, specifically CI/CD quality gates, data generation, and automated validation scripts; - Establish and monitor key quality metrics aligned with business goals, including turnaround time, confidence scores, and user corrections; - Act as a strategic partner to Product and Engineering teams, advocating for quality from design through delivery; - Stay up to date with the latest AI technologies and explore how to apply advancements to solve previously impossible challenges for our customers; - Take ownership of code quality through exploratory and automated testing. MUST HAVES - Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent experience; - 2–3 years of experience in AQA, Software Test Engineering, or related roles, including leadership responsibilities; -Experience in AQA JavaScript; - Solid knowledge of the software development life cycle (SDLC), Agile methodologies, and QA/QE best practices; - Proven experience building and scaling test automation frameworks (e.g., Pytest, Cypress, Selenium, Postman); - Strong technical skills in working with REST APIs, data pipelines, backend services, and cloud platforms (e.g., AWS, Azure); - Exceptional attention to detail and a mindset for building robust systems that catch issues early and protect the user experience; - Demonstrated ability to learn quickly and proactively—especially important in the rapidly evolving field of AI; - Upper-Intermediate English level. NICE TO HAVES - Experience testing AI/ML systems in production environments; - Familiarity with GenAI model evaluation metrics; - Experience setting up proactive monitoring and alerting systems to detect model drift, performance degradation, or workflow failures. THE BENEFITS OF JOINING US - Professional growth: Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps. - Competitive compensation: We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities. - A selection of exciting projects: Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands. - Flextime: Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive. Your application doesn't end here! To unlock the next steps, check your email and complete your registration on our Applicant Site. The incomplete registration results in the termination of your process.