StudyToolkitVid

A DDLitLab-funded toolkit for studying human perception of video material, supporting research in audio-visual translation quality.

StudyToolkitVid is a student research project funded by the DDLitLab (D³ Data Literacy Lab) at the University of Hamburg for 10,000 €. It set out to build a reusable toolkit for designing and running user studies that measure human perception of video material - with a particular focus on audio-visual translations such as dubbing and lip-synced video.

Evaluating audio-visual translation quality is hard: automated metrics like BLEU cannot capture whether translated speech feels natural when paired with the visual content. Human perception studies are the gold standard, but designing and running them requires significant tooling infrastructure. StudyToolkitVid aimed to lower that barrier by providing a reusable, configurable toolkit for this purpose.

The project received funding of 10,000 € from the DDLitLab (D³ Data Literacy Lab) initiative at the University of Hamburg - the first funding round of the programme. It was mentored by Prof. Dr. Timo Baumann.

StudyToolkitVid grew out of research on audio-visual speech processing and lip-sync evaluation conducted in collaboration with Timo Baumann's group. It provided tooling infrastructure for the neural synchrony evaluation work published at ICMI 2022 and the Linux Magazine article on NLP.