Optimizer or Precision? Understanding Optimizer–Precision Interactions in PINNs

Physics-informed neural networks can reach low training loss and still miss the true solution, and earlier work blamed some of that on low arithmetic precision. Those studies only trained with L-BFGS, though. We separated the two factors, running eight optimizer and precision configurations across four PDE benchmarks.

Precision mostly matters under L-BFGS: doubling it changed Adam's error by at most 1.25×, and L-BFGS's by up to 17.7×. Every run that reached low error finished with L-BFGS, and starting in FP32 Adam before switching to FP64 L-BFGS beat full FP64 L-BFGS on three of the four benchmarks.

Read the paper on OpenReview ↗︎