Material

Data

A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction

In this corpus of multiparty situated interaction participants collaborated on moving virtual objects on a large touch screen. A moderator facilitated the discussion and directed the interaction. The corpus contains recordings of a variety of multimodal data, in that we captured speech, eye gaze and gesture data using a multisensory setup (wearable eye trackers, motion capture and audio/video). In the multimodal corpus, we investigate four different types of social gaze: referential gaze, joint attention, mutual gaze and gaze aversion by both perspectives of a speaker and a listener. We annotated the groups’ object references during object manipulation tasks and analysed the group’s proportional referential eye-gaze with regards to the referent object. When investigating the distributions of gaze during and before referring expressions we could corroborate the differences in time between speakers’ and listeners’ eye gaze found in earlier studies. This corpus is of particular interest to researchers who are interested in social eye-gaze patterns in turn-taking and referring language in situated multi-party interaction.

language

For more information on the corpus look at: kth.se/profile/diko/page/material.

Citation
For a detailed description of the corpus refer to our paper:

@inproceedings{kontogiorgos2018lrec,
author = {Dimosthenis Kontogiorgos, Vanya Avramova, Simon Alexandersson, Patrik Jonell, Catharine Oertel, Jonas Beskow, Gabriel Skantze and Joakim Gustafson},
title = {{A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction}},
year = {2018},
booktitle = {Language Resources and Evaluation Conference LREC 2018},
}

Code

Real-time eye-gaze & mocap visualisation

In this visualisation tool we use WebGL and Three.js to visualise in real time motion capture and eye tracking data, combined with syntactic parsing in natural language.

Github

—————————————————————————————————————————–

Multisensory processing architecture

We present a processing architecture used to collect multimodal sensor data, both for corpora collection and real-time processing. The code for the implemented architecture is available as an open-source repository under Apache License v2. The architecture is agnostic to the choice of hardware (e.g. microphones, cameras, etc.) and programming languages, although our framework implementation is mostly written in Python. The architecture is of particular interest for researchers who are interested in the collection of multi-party, richly recorded corpora and the design of conversational systems. Moreover for researchers who are interested in human-robot interaction the available modules offer the possibility to easily create both autonomous and wizarded interactions.

Github

Citation
For a detailed description of the architecture and framework refer to our paper:

@inproceedings{jonell2018lrec,
author = {Patrik Jonell, Mattias Bystedt, Per Fallgren, Dimosthenis Kontogiorgos, José Lopes, Zofia Malisz, Samuel Mascarenhas, Catharine Oertel, Eran Raveh and Todd Shore},
title = {{ARMI: An Architecture for Recording Multimodal Interactions}},
year = {2018},
booktitle = {Language Resources and Evaluation Conference LREC 2018},
}

—————————————————————————————————————————–

Multimodal corpora crowdsourcing collection

We are proposing a novel tool which will enable researchers to rapidly gather large amounts of multimodal data spanning a wide demographic range. The code is released under an Apache License 2.0 and available as an open-source repository which will allow researchers to set-up their own multimodal data collection system quickly and create their own multimodal corpora.

GitHub

Citation
For a detailed description of the tool refer to our paper:

@inproceedings{jonell2018lrec,
author = {Patrik Jonell, Catharine Oertel, Dimosthenis Kontogiorgos, Jonas Beskow and Joakim Gustafson},
title = {{Crowdsourced Multimodal Corpora Collection Tool}},
year = {2018},
booktitle = {Language Resources and Evaluation Conference LREC 2018},
}