Robin Pre-Training for the Deep Ritz Method
DOI:
https://doi.org/10.7557/18.6800Keywords:
Deep Ritz Method, Neural Networks, Variational Problems, Essential Boundary ConditionsAbstract
We analyze the training process of the Deep Ritz Method for elliptic equations with Dirichlet boundary conditions and highlight problems arising from essential boundary values. Typically, one employs a penalty approach to enforce essential boundary conditions, however, the naive approach to this problem becomes unstable for large penalizations. A novel method to compensate this problem is proposed, using a small penalization strength to pre-train the model before the main training on the target penalization strength is conducted. We present numerical evidence that the proposed method is beneficial.
References
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.
I. Babuˇska. The finite element method with penalty. Mathematics of computation, 27(122): 221–228, 1973.
E. C. Cyr, M. A. Gulian, R. G. Patel, M. Perego, and N. A. Trask. Robust Training and Initialization of Deep Neural Networks: An adaptive Basis Viewpoint. In Mathematical and Scientific Machine Learning, pages 512–536. PMLR, 2020.
M. Dissanayake and N. Phan-Thien. Neural-Network-based Approximations for solving Partial Differential Equations. Communications in Numerical Methods in Engineering, 10 (3):195–201, 1994.
W. E and B. Yu. The Deep Ritz method: a Deep Learning-based numerical Algorithm for solving Variational Problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018. doi: 10.1007/s40304-018-0127-z. URL https://doi.org/ 10.1007/s40304-018-0127-z.
I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial Neural Networks for solving Ordinary and Partial Differential Equations. IEEE transactions on neural networks, 9(5):987–1000, 1998. doi: 10.1109/72.712178. URL https://doi.org/10.110/72.712178.
K. Lee, N. A. Trask, R. G. Patel, M. A. Gulian, and E. C. Cyr. Partition of unity networks: Deep hp-approximation. In CEUR Workshop Proceedings, volume 2964, page 180. CEURWS, 2021.
Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations, 2021.
J. Mueller and M. Zeinhofer. Error Estimates for the Deep Ritz Method with
Boundary Penalty. In 3rd Conference on Mathematical and Scientific Machine Learning (MSML2022), 2022.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
M. Raissi and G. E. Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125–141, 2018. doi: 10.1016/j.jcp.2017.11.039. URL https://doi.org/10.1016/j.jcp.2017.11.039.
J. Sirignano and K. Spiliopoulos. DGM: A Deep Learning Algorithm for solving Partial Differential Equations. Journal of computational physics, 375:1339–1364, 2018. doi: 10.1016/j.jcp.2018.08.029. URL https://doi.org/10.1016/j.jcp.2018.08.029.
R. van der Meer, C. W. Oosterlee, and A. Borovykh. Optimally weighted loss functions for solving pdes with neural networks. Journal of Computational and Applied Mathematics, 405:113887, 2022. doi: 10.1016/j.cam.2021.113887. URL https://doi.org/10.1016/j.cam.2021.113887.
S. Wang, Y. Teng, and P. Perdikaris. Understanding and mitigating gradient flow
pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021. doi: 10.1137/20M1318043. URL https://doi.org/10.1137/20M1318043.
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Luca Courte, Marius Zeinhofer
This work is licensed under a Creative Commons Attribution 4.0 International License.
