Testlio
Location: This is a 100% remote role open to those who reside in the United States. About the job Testlio's fully managed crowdsourced testing platform, powered by our proprietary intelligence technology - LeoCore™, integrates expert, on-demand testers directly into your release process. Ship faster and more confidently with global coverage across 600,000+ devices, 800+ payment methods, 150+ countries, and 100+ languages. To learn more, visit testlio.com. We are hiring a visionary and hands-on AI Solutions Architect to own the definition, packaging, technical design, and delivery methodology for our AI testing offerings. This role brings together deep practitioner expertise with a strong business perspective. You will design the testing approach for complex AI-powered applications (including agentic systems), and you will advance Testlio's proprietary framework combining automated and human-in-the-loop evaluation while defining what Testlio sells as a service—setting the catalog, staffing models, and pricing structure. As the technical authority, you will decompose architectures into structured testing approaches and collaborate with sales and delivery to drive our AI business forward. Why will you love this job? • Evaluate at a Scale That Exists Nowhere Else: Every vendor has LLM-as-a-judge. Only Testlio can put thousands of vetted experts across 150 countries and 100+ languages behind a grading rubric. You will design evaluation systems that combine model graders and human experts. • Innovate at the Frontier of QA: Shape the industry standards and operational playbooks for validating agentic AI, large language models, and the data pipelines behind them. • High-Impact Consultative Visibility: Collaborate with global market leaders across diverse verticals. Interface directly with data scientists, machine learning engineers, and executive sponsors at market-leading global brands to architect their AI quality blueprints. • Decompose Complex Ecosystems: Turn "