Dynamical phases of short-term memory mechanisms in RNNs

Bariscan Kurtkaya*1,2, Fatih Dinc*2,3,4, Mert Yuksekgonul5, Marta Blanco-Pozo2,6, Ege Cirakman2, Mark Schnitzer2,6,7, Yucel Yemez1, Hidenori Tanaka✟ 8,9, Peng Yuan✟ 10, Nina Miolane✟ 3,
1Koc University
2CNC Program, Stanford University
3Geometric Intelligence Lab, UC Santa Barbara
4Kavli Institute for Theoretical Physics, UC Santa Barbara
5Computer Science, Stanford University
6James H. Clark Center for Biomedical Engineering & Sciences, Stanford University
7Howard Hughes Medical Institute, Stanford University
8Phi Lab, NTT Research
9Center for Brain Science, Harvard University
10Institute for Translational Brain Research, Fudan University
ICML 2025
*Indicates Equal Contribution
✟Supervision
Paper Code arXiv Dataset ICML

Abstract

Short-term memory is essential for cognitive processing, yet our understanding of its neural mechanisms remains unclear. Neuroscience has long focused on how sequential activity patterns, where neurons fire one after another within large networks, can explain how information is maintained. While recurrent connections were shown to drive sequential dynamics, a mechanistic understanding of this process still remains unknown. In this work, we introduce two unique mechanisms that can support this form of short-term memory: slow-point manifolds generating direct sequences or limit cycles providing temporally localized approximations. Using analytical models, we identify fundamental properties that govern the selection of each mechanism. Precisely, on short-term memory tasks (delayed cue-discrimination tasks), we derive theoretical scaling laws for critical learning rates as a function of the delay period length, beyond which no learning is possible. We empirically verify these results by training and evaluating approximately 80,000 recurrent neural networks (RNNs), which are publicly available for further analysis. Overall, our work provides new insights into short-term memory mechanisms and proposes experimentally testable predictions for systems neuroscience.

Paper Summary

What is the problem?

Short-term memory is essential for cognition, but its neural mechanisms remain poorly understood. While neuroscience has suggested that recurrent dynamics support sequential neural activity, the precise mechanisms remain elusive—particularly given the rapid reorganization of activity following stimuli or rewards. Biologically interpretable recurrent neural networks (RNNs) offer a promising avenue for generating hypotheses about these mechanisms.

What did we do?

We studied how the learning rate (inspired by dopamine's role in learning), the delay interval, and the post-reaction period influence short-term memory dynamics in analytical models, low-rank RNNs, and full-rank RNNs. We focused on delayed-response tasks such as delayed activation and delayed cue-discrimination.

MY ALT TEXT
MY ALT TEXT

What did we find?

  • We identified two distinct mechanisms for maintaining short-term memory: slow-point manifolds that support smooth sequences, and limit cycles that generate temporally localized dynamics.
  • We derived theoretical scaling laws linking the learning rate and delay length, predicting the emergence of each mechanism and defining critical thresholds beyond which learning fails.
  • Small changes in task structure—such as adding a post-response period—can significantly alter the underlying phase space and learned dynamics.
  • We propose testable predictions to guide wet-lab neuroscience experiments in identifying memory-related dynamics in the brain.
MY ALT TEXT

What are these emergent short-term memory mechanisms?

This video illustrates three distinct behaviors: unable to learn, limit cycle dynamics, and slow-point manifolds—shown in that order. In this experiment, we trained full-rank RNNs with 100 neurons on a delayed cue-discrimination task with a 160-millisecond delay. The learning rate was set to 1 for the network that unable to learn, 0.01 for the limit cycle case, and 0.001 for the slow-point manifold. While limit cycles and slow-point manifolds produce similar activity patterns during the trial, they begin to diverge in the post-reaction period—a phase that is especially difficult to capture in wet-lab experiments due to rapid synaptic reorganization.

Poster

Bibtex

@inproceedings{
          kurtkaya2025dynamical,
          title={Dynamical phases of short-term memory mechanisms in {RNN}s},
          author={Bariscan Kurtkaya and Fatih Dinc and Mert Yuksekgonul and Marta Blanco-Pozo and Ege Cirakman and Mark Schnitzer and Yucel Yemez and Hidenori Tanaka and Peng Yuan and Nina Miolane},
          booktitle={Forty-second International Conference on Machine Learning},
          year={2025},
          url={https://openreview.net/forum?id=ybBuwgOPOd}
          }