I am a UC Chancellor’s Postdoctoral Fellow at UCSD, working with Brad Voytek and Ben Bergen in the Department of Cognitive Science, where I completed my PhD in 2023.
Throughout my research career, I have always been interested in how neural circuits adapt to variance and patterns of regularity in the world around us. In my graduate work, I explored this question within the rodent hippocampus, asking how inhibitory interneurons varied their spike timing as a function of changes to an environment’s “reward landscape”. More recently, I have pivoted my research efforts towards the study of learning in artificial neural networks, with a focus on transformer-based language models and how they vary their weights over the course of pre-training.
I am broadly interested in questions that get at the training dynamics of language models:
-
what is the shape of the trajectory of performance on a particular task, over pre-training (e.g. is there a sharp increase, or gradual ramp-up in performance? are increases relatively smooth, or is there a lot of variance from checkpoint to checkpoint?)
-
for a complex capability (e.g. mental state modeling: Alma thinks the ball is in the box.), is the onset of improvement preceded by the onset of improvement in a capability that we would deem more basic (e.g. having a robust situation model: Alma moved the ball to the box.)?
-
can we use the charted trajectory of training dynamics to isolate the internal mechanisms that give rise to these trajectories of performance?
contact information
pdrivier@ucsd.edu