← Back to Research
2019 IBM Research Sesame Street

Automatic Generation of Visual Assessments for Early Childhood Learning

This first of its kind industry collaboration brought together IBM Research and Sesame Workshop to build a cognitive vocabulary learning app for young children. As a Machine Learning research engineer, I worked alognside primary school teachers and early childhood learning experts to automatically generate picture-based quiz questions at varying difficulty levels. This project pioneered the use of AI to generate pedagogically appropriate visual assessments that adapt to each child's vocabulary range.

Methods

Semi-structured interviews with primary school teachers to understand the process and challenges of creating visual multiple choice questions (MCQs) at varying difficulty levels
Developed a novel automated approach to generate visual MCQs — where answer options are presented as images — using an image corpus and a semantic network of curated vocabulary words connected by their relationships extracted from Children's Book Test database Leveraged image captioning, open information extraction, and neural embedding based semantic similarity matching
Contributed a novel metric for measuring semantic similarity between images, used to source and rank MCQ options at controlled difficulty levels

Key Findings

Creating visual MCQs at varying difficulty levels is highly time-consuming for educators, with sourcing pedagogically appropriate images consuming a lot of their time. Our approach automated this end-to-end, by developing technology that could automatically generate picture-based assessment questions at well-calibrated difficulty levels without sacrificing pedagogical quality.

Impact

    Contributed to industry's first cognitive vocabulary learning app for young children, in collaboration between IBM Research and Sesame Workshop.