Self-Communicating Deep Reinforcement Learning Agents Develop External Number Representations
DOI:
https://doi.org/10.7557/18.6291Keywords:
deep learning, reinforcement learning, self-communication, neural networks, numerical cognition, mathematical cognition, external representationsAbstract
Symbolic numbers are a remarkable product of human cultural development. The developmental process involved the creation and progressive refinement of material representational tools, such as notched tallies, knotted strings, and counting boards. In this paper, we introduce a computational framework that allows the investigation of how material representations might support number processing in a deep reinforcement learning scenario. In this framework, agents can use an external, discrete state to communicate information to solve a simple numerical estimation task. We find that different perceptual and processing constraints result in different emergent representations, whose specific characteristics can facilitate the learning and communication of numbers.
References
B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528, 2019. https://doi.org/10.1016/j.cognition.2012.05.005.
N. Barhate. Minimal PyTorch Implementation of Proximal Policy Optimization. https://github.com/nikhilbarhate99/PPO-PyTorch, 2021.
R. Bellman. A markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679–684, 1957. https://doi.org/10.1512/iumj.1957.6.56038.
A. Bender and S. Beller. Nature and culture of finger counting: Diversity and representational effects of an embodied cognitive tool. Cognition, 124(2):156–182, 2012. https://doi.org/10.1016/j.cognition.2012.05.005.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41–48, 2009. https://doi.org/10.1145/1553374.1553380.
S. Carey and D. Barner. Ontogenetic origins of human integer representations. Trends in Cognitive Sciences, 23(10):823–835, 2019. https://doi.org/10.1016/j.tics.2019.07.004.
R. Chaabouni, E. Kharitonov, E. Dupoux, and M. Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 118(12), 2021. https://doi.org/10.1073/pnas.2016569118.
A. Clark et al. Supersizing the mind: Embodiment, action, and cognitive extension. OUP USA, 2008. https://doi.org/10.1093/acprof:oso/9780195333213.001.0001.
C. Creatore, S. Sabathiel, and T. Solstad. Learning exact enumeration and approximate estimation in deep neural network models. Cognition, 215:104815, 2021. https://doi.org/10.1016/j.cognition.2021.104815.
S. Dehaene. The number sense: How the mind creates mathematics. OUP USA, 2011.
S. Dehaene, M. Piazza, P. Pinel, and L. Cohen. Three parietal circuits for number processing. Cognitive Neuropsychology, 20(3-6): 487–506, 2003. https://doi.org/10.1080/02643290244000239.
G. Lakoff and R. E. N´u˜nez. Where mathematics comes from: How the embodied mind brings mathematics into being. AMC, 10(12): 720–733, 2000.
A. Lazaridou and M. Baroni. Emergent multiagent communication in the deep learning era. arXiv preprint arXiv:2006.02419, 2020.
A. Nieder. Number faculty is rooted in our biological heritage. Trends in Cognitive Sciences, 21(6):403–404, 2017. https://doi.org/10.1016/j.tics.2017.03.014.
R. E. N´u˜nez. No innate number line in the human brain. Journal of Cross-Cultural Psychology, 42(4):651–668, 2011. https://doi.org/10.1177/0022022111406097. ISSN 00220221. doi: 10.1177/0022022111406097.
K. A. Overmann. Constructing a concept of number. Journal of Numerical Cognition, 4(2):464–493, 2018. https://doi.org/10.5964/jnc.v4i2.161. ISSN 2363-8761. doi:10.5964/jnc.v4i2.161.
S. Sabathiel, J. McClelland, and T. Solstad. A computational model of learning to count in a multimodal, interactive environment. In CogSci, 2020.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
I. Stoianov and M. Zorzi. Emergence of a’visual number sense’in hierarchical generative models. Nature Neuroscience, 15(2):194, 2012. https://doi.org/10.1038/nn.2996.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
A. Testolin. The Challenge of Modeling the Acquisition of Mathematical Concepts. Frontiers in Human Neuroscience, 14(March):1–9, 2020. https://doi.org/10.3389/fnhum.2020.00100. ISSN 16625161. doi: 10.3389/fnhum.2020.00100.
A. Testolin, S. Dolfi, M. Rochus, and M. Zorzi. Visual sense of number vs. sense of magnitude in humans and machines. Scientific Reports, 10(1):1–13, 2020. https://doi.org/10.1038/s41598-020-66838-5.
A. Testolin, W. Y. Zou, and J. L. McClelland. Numerosity discrimination in deep neural networks: Initial competence, developmental refinement and experience statistics. Developmental Science, 23(5):e12940, 2020. https://doi.org/10.1111/desc.12940.
K. Wynn. An evolved capacity for number. in: Cummins D.D. Allen C. The evolution of mind, page 107–126, 1998.
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Silvester Sabathiel, Trygve Solstad, Alberto Testolin, Flavio Petruzzellis
This work is licensed under a Creative Commons Attribution 4.0 International License.
