Visual Sciences Society Conference 2022

Earlier this year I attented the Visual Science Society 2022 conference where I gave a talk titled “A Beta-Variational Auto-Encoder Model of Human Visual Representation Formation in Utility-Based Learning”. The entire abstract I submitted is copied at the bottom of this blog post if you would like to read it.

The VSS experience was very positive, I was somewhat worried or hesitant that I would not get a lot of the posters and talks since my experience is much more in the learning domain rather than vision. While there were many talks and posters that I didn’t fully understand, all of the participants and presenters made a good effort to explain things for a general audience and I didn’t have too much trouble getting the general gist of projects.

Apart from the general vision research, there was, to me, a suprising number of Deep Neural Network techniques used in various vision related projects, as well as a fair number of tasks that involved either learning or decision making, which is the focus of my thesis. I was able to meet with a few PIs who were presenting posters related to their work that I was very interested in and discuss how my work relates to there.

Another good experience was the industry meeting. While I do have experience in industry research somewhat through my internship with IBM, it was interesting to see how different companies handle AI and Human learning research. There was a good combination of bigger companies (Apple and Google), as well as smaller companies and start ups that have very different structures when it comes to research.

The other great aspect of VSS is of course the location, St.Pete’s beach Florida which had amazing weather. I had heard that in previous years the weather had been very hot and somewhat muggy/humid. This year there was probably only one day where I felt too hot/humid while walking around. I did get to stay in the same hotel where the conference was so I was able to pop back to my room if things got too sweaty, which I would recommend for anyone able to when attending either VSS or another conference in a hot area in the summer. So all in all I had a great experience and I’m excited to head to my next conference Reinforcement Learning and Decision Making next month!

Here is the link VSS Presentation for just my talk, for you to view.

Abstract: Tyler Malloy (mallot@rpi.edu), Chris R. Sims; Rensselaer Polytechnic Institute

The human brain is capable of forming informationally constrained representations of complex visual stimuli in order to achieve its behavioural goals, such as utility-based learning. Recently, methods borrowed from machine learning have demonstrated a close connection between the mechanisms of visual representation formation in primate brains with the latent representations formed by Beta-Variational Auto-Encoders (Beta-VAEs). While auto-encoder models capture some aspects of visual representations, they fail to explain how visual representations are adapted in a task-directed manner. We developed a model of visual representation formation in learning environments based on a modified Beta-VAE model that simultaneously learns the task-specific utility of visual information. We hypothesized that humans update their visual representations as they learn which visual features are associated with utility in learning tasks. To test this hypothesis, we applied the proposed model onto the data from a visual contextual bandit learning task [Niv et al. 2015; J. Neuroscience]. The experiment involved humans (N=22) learning the utility associated with 9 possible visual features (3 colors, shapes or textures). Critically, our model takes in as input the same visual information that is presented to participants, instead of the hand-crafted features typically used to model human learning. A comparison of predictive accuracy between our proposed model and models using hand-crafted features demonstrated a similar correlation to human learning. These results show that representations formed by our Beta-VAE based model can predict human learning from complex visual information. Additionally, our proposed model makes predictions of how visual representations adapt during human learning in a utility-based task. Further, we performed a comparison of our proposed model across a range of parameters such as information-constraint, utility-weight, and number of training steps between predictions. Results from this comparison give insight into how the human brain adjusts its visual representation formation during learning.

Acknowledgements: This work was supported by NSF research grant DRL-1915874 to CRS and an IBM AIRC scholarship to TJM