Gibbs Sampling with People

Using collective human behavior to implement probabilistic inference and generative modeling.

As cognitive scientists, we are often interested in mapping the relationship between external stimuli (e.g., spoken sentences, musical chords, faces) and semantic features that the mind derives from these stimuli (e.g., happiness, sadness, pleasantness). Traditional methods for studying such relationships (e.g., non-adaptive rating experiments) work well when the stimulus spaces are low-dimensional and the semantic features are simple, but struggle when applied to more complex cognitive domains.

We have developed a new technique, termed Gibbs Sampling with People (GSP), designed to overcome this problem. This is an adaptive technique, inspired by the Gibbs Sampling method from computational statistics and machine learning, where many participants collaborate to navigate a stimulus space and identify regions associated with a given semantic concept, for example “pleasantness.”

In each trial, the participant is presented with a stimulus and a slider, where the slider is coupled to a particular dimension of the stimulus space that changes from trial to trial. The participant is instructed to move the slider to find the stimulus most associated with the target semantic concept. The resulting stimulus is then passed along the chain of participants, with each successive participant optimizing a different dimension. Under our cognitive decision model, the emergent process corresponds to a Gibbs sampler that maps the relationship between the stimulus space and the target semantic concept.

Example faces generated by combining GSP with the StyleGAN image synthesis model
Example faces generated by combining GSP with the StyleGAN image synthesis model (Karras et al., 2018).

In a recent paper (Harrison et al., NeurIPS 2021), we demonstrate this technique in four different domains that vary substantially in complexity: colors, musical chords, emotional prosody, and faces. We show that the method generalizes well to these different domains, generating interpretable and interesting results. Further, we show that the method can be coupled with state-of-the-art deep neural synthesis models to study the perception of very high-dimensional stimulus spaces, such as natural images.

The slider interface

Slider interface applied to colors
Example of the slider interface as applied to colors.
Slider interface applied to faces
Example of the slider interface as applied to faces.

GSP versus prior methods

Color samples from MCMCP, GSP, and aggregated GSP
Color samples generated by the prior state-of-the-art technique (MCMCP), GSP, and aggregated GSP.
Performance of MCMCP / GSP / aggregated GSP as a function of iteration
Performance of MCMCP / GSP / aggregated GSP as a function of iteration number.

Sound examples

(Harrison et al., 2020; van Rijn et al., 2021; Van Geert & Jacoby, 2024)

Related Publications

2024

  1. Using Gibbs Sampling with People to characterize perceptual and aesthetic evaluations in multidimensional visual stimulus space
    Eline Van Geert and Nori Jacoby
    In Proceedings of the Annual Meeting of the Cognitive Science Society, 2024
    DOI URL

2021

  1. Exploring emotional prototypes in a high dimensional TTS latent space
    Pol Rijn, Silvan Mertes, Dominik Schiller, and 4 more authors
    arXiv preprint arXiv:2105.01891, 2021
    DOI URL arXiv

2020

  1. Gibbs sampling with people
    Peter Harrison, Raja Marjieh, Federico Adolfi, and 5 more authors
    Advances in Neural Information Processing Systems, 2020
    DOI URL