Global IT QA Software Engineer Senior Manager – Gen AI/ LLM Full-time

WHAT YOU’LL DO

This is a GenAI Quality Engineer Senior Manager position within BCG’s Quality Engineering Center of Expertise (QE CoE). The QE CoE resides within BCG’s Cloud & Platform Engineering (CPE) Portfolio. The role of CPE includes developing and supporting new and enhanced applications, as well as integrating and configuring third party software applications used by our Global HR, Finance, Knowledge, Marketing, Legal, etc. functions. The responsibilities of CPE include enabling efficient processes within the Global Functions; leading and implementing system change initiatives; and supporting the functional user community to optimize the management of the business.

As a GenAI Quality Engineer Senior Manager, you will oversee the quality engineering processes for AI/ML/LLM models and systems. You will be responsible for developing and implementing testing frameworks to ensure the performance, accuracy, and reliability of generative AI solutions. Key duties include designing comprehensive test plans and utilizing advanced testing methodologies to identify and resolve defects.

You will collaborate with cross-functional teams including data scientists, software engineers, and product owners to ensure that AI/ML/LLM models meet specified standards and user requirements. Additionally, you will monitor and analyze model performance, provide feedback for continuous improvement, and stay updated on the latest advancements in AI technology and quality engineering practices.

This role requires strong technical expertise in AI/ML/LLMs, programming, and testing tools, along with leadership and project management skills to drive the quality engineering strategy and ensure the successful deployment of AI/ML/LLM solutions.

WHO WE ARE

BCG pioneered strategy consulting more than 50 years ago, and we continue to innovate and redefine the industry. We offer multiple career paths for the world’s best talent to have a real impact on business and society. As part of our team, you will benefit from the breadth and diversity of what we are doing today and where we are headed next. We count on your authenticity, exceptional work, and strong integrity. In return we are committed to supporting you in discovering the most fulfilling career journey possible-and unlocking your potential to advance the world. Our team called Global Services (GS) provides corporate support to business areas such as Finance, Legal, HR, Marketing and IT. This diverse team of experts, operators and specialists represent all levels from Partner to entry level staff, operating across the globe in multiple countries. Global Services is in short, the backbone of BCG.

YOU BRING (EXPERIENCE & QUALIFICATIONS)

• Bachelor’s degree in computer science, Engineering, or a related field
• Extensive experience in quality engineering or software testing
• Proven Experience in testing Gen AI, LLMs, Artificial Intelligence, Machine Learning, or Natural Language Processing based systems for scale and robustness.
• Strong background in AI and machine learning
• Proficiency in programming languages (e.g., Python, Java)
• Experience with AI/ML/LLM testing tools and frameworks e.g. DeepEval or equivalent.
• Solid understanding of AI model development and deployment
• Proven leadership and team management skills
• Excellent problem-solving and analytical abilities
• Strong communication and collaboration skills
• Knowledge of industry standards and best practices in AI/LLM quality engineering
• Experience with performance and reliability testing

EQUAL OPPORTUNITY

BCG is an Equal Employment Opportunity employer and is committed to a policy of administering all employment decisions and actions without regard to race, national origin, religion, age, color, sex, sexual orientation, gender identity, disability, or protected veteran status, or any other characteristic protected by local, state, or federal laws, rules, or regulations.

YOU’RE GOOD AT

• Developing comprehensive test plans
• Implementing advanced testing methodologies
• Identifying and resolving defects
• Collaborating with cross-functional teams
• Ensuring AI model performance and reliability
• Analyzing model performance metrics
• Providing feedback for continuous improvement
• Staying updated on AI technology advancements
• Applying strong technical expertise in AI/ML/LLMs
• Utilizing programming and testing tools
• Managing quality engineering strategies
• Overseeing successful AI/ML/LLM solution deployment
• Communicating effectively with stakeholders
• Problem-solving and critical thinking
• Project management
• Understanding AI/ML/LLM model standards and requirements
• Conducting risk assessments
• Enhancing testing frameworks
• Maintaining high standards of AI/ML/LLM model accuracy and performance

YOU’LL WORK WITH

BCG’s information technology group collaboratively delivers the latest digital technologies that enable our consultants to lead and our business to grow. For our IT jobs, we seek individuals with expertise in the areas of IT infrastructure, application development, business systems, collaborative and social technologies, information security, and project leadership.