Journal of Machine Learning
type: channel
description: Journal of Machine Learning (JML) publishes high quality research papers in all areas of machine learning, including innovative algorithms of machine learning, theories of machine learning, important applications of machine learning in AI, natural sciences, social sciences, and engineering etc. The journal emphasizes a balanced coverage of both theory and practice. The journal is published in a timely fashion in electronic form. All articles in JML are open-access and there is no charge for the authors.
publisher: Global Science Press
language: en
publication name: Journal of Machine Learning
issn: 2790-203X
items: Anchor Function: A Type of Benchmark Functions for Studying Language Models
items: Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learning
items: Spectral Neural Networks: Approximation Theory and Optimization Landscape
items: Neural Stochastic Volterra Equations: Learning Path-Dependent Dynamics
items: Soft-Constrained Distance Preserving t-SNE
items: A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions
items: Progressive Optimal Path Sampling for Closed-Loop Optimal Control Design with Deep Neural Networks
items: Optimistic Estimate Uncovers the Potential of Nonlinear Models
items: A Deep Uzawa-Lagrange Multiplier Approach for Boundary Conditions in PINNs and Deep Ritz Methods
items: Tangent Differential Privacy
items: Cluster-Based Classification with Neural ODEs via Control
items: Convergence Analysis of Discrete Diffusion Model: Exact Implementation Through Uniformization
items: Convergence of Stochastic Gradient Descent under a Local Łojasiewicz Condition for Deep Neural Networks
items: Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow
items: Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces
items: A Note on Continuous-Time Online Learning
items: MAFE-Net: A Multi-Level Attention Feature Extraction Network for Pancreas Segmentation
items: Variational Formulations of ODE-Net as a Mean-Field Optimal Control Problem and Existence Results
items: Enhancing Accuracy in Deep Learning Using Random Matrix Theory
items: Memory$^3$: Language Modeling with Explicit Memory
items: Fast Gradient Computation for Gromov-Wasserstein Distance
items: Convergence of Stochastic Gradient Descent Schemes for Łojasiewicz-Landscapes
items: Solving Bivariate Kinetic Equations for Polymer Diffusion Using Deep Learning
items: Approximation Results for Gradient Flow Trained Neural Networks
items: Mean-Field Neural Networks-Based Algorithms for McKean-Vlasov Control Problems
items: Semi-Supervised Clustering of Sparse Graphs: Crossing the Information-Theoretic Threshold
items: Stochastic Delay Differential Games: Financial Modeling and Machine Learning Algorithms
items: On the Existence of Optimal Shallow Feedforward Networks with ReLU Activation
items: Embedding Inequalities for Barron-Type Spaces
items: Prompt Engineering Through the Lens of Optimal Control

Anchor Function: A Type of Benchmark Functions for Studying Language Models
type: item
description: Understanding transformer-based language models is becoming increasingly crucial, particularly as they play pivotal roles in advancing towards artificial general intelligence. However, language model research faces significant challenges, especially for academic research groups with constrained resources. These challenges include complex data structures, unknown target functions, high computational costs and memory requirements, and a lack of interpretability in the inference process, etc. Drawing a parallel to the use of simple models in scientific research, we propose the concept of an anchor function. This is a type of benchmark function designed for studying language models in learning tasks that follow an "anchor-key" pattern. By utilizing the concept of an anchor function, we can construct a series of functions to simulate various language tasks. The anchor function plays a role analogous to that of mice in diabetes research, particularly suitable for academic research. We demonstrate the utility of the anchor function with an example, revealing two basic operations by attention structures in language models: shifting tokens and broadcasting one token from one position to many positions. These operations are also commonly observed in large language models. The anchor function framework, therefore, opens up a series of valuable and accessible research questions for further exploration, especially for theoretical study.
subject: Articles
  topic: section
subject: Language models
  topic: keywords
subject: Transformer
  topic: keywords
subject: Anchor function
  topic: keywords
subject: Machine learning theory
  topic: keywords
creator: Zhongwang Zhang
creator: Zhiwei Wang
creator: Junjie Yao
creator: Zhangchen Zhou
creator: Xiaolong Li
creator: Weinan E
creator: Zhi-Qin John Xu
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2026-03-06
publication date: 2026-03-06
doi: 10.4208/jml.250723

Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learning
type: item
description: We develop a novel framework for uncertainty quantification in operator learning, namely the stochastic operator network (SON). SON combines the stochastic optimal control concepts of the stochastic neural network (SNN) with the deep operator network. By formulating the branch net as an SDE and backpropagating through the adjoint BSDE, we replace the gradient of the loss function with the gradient of the Hamiltonian from the stochastic maximum principle in the stochastic gradient descent (SGD) update. This allows SON to learn the uncertainty present in operators through its diffusion parameters. We then demonstrate the effectiveness of SON when replicating several noisy operators in 2D and 3D.
subject: Articles
  topic: section
subject: Stochastic operator learning
  topic: keywords
subject: Deep operator network
  topic: keywords
subject: Stochastic maximum principle
  topic: keywords
subject: Backward stochastic differential equation
  topic: keywords
subject: Uncertainty quantification
  topic: keywords
creator: Ryan Bausback
creator: Jingqiao Tang
creator: Lu Lu
creator: Feng Bao
creator: Phuoc-Toan Huynh
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2026-01-28
publication date: 2026-01-28
starting page: 71
ending page: 96
doi: 10.4208/jml.250709

Spectral Neural Networks: Approximation Theory and Optimization Landscape
type: item
description: There is a large variety of machine learning methodologies that are based on the extraction of spectral geometric information from data. However, the implementations of many of these methods often depend on traditional eigensolvers, which present limitations when applied in practical online big data scenarios. To address some of these challenges, researchers have proposed different strategies for training neural networks as alternatives to traditional eigensolvers, with one such approach known as spectral neural network (SNN). In this paper, we investigate key theoretical aspects of SNN. First, we present quantitative insights into the tradeoff between the number of neurons and the amount of spectral geometric information a neural network learns. Second, we initiate a theoretical exploration of the optimization landscape of SNN’s objective to shed light on the training dynamics of SNN. Unlike typical studies of convergence to global solutions of NN training dynamics, SNN presents an additional complexity due to its non-convex ambient loss function, a feature that is common in unsupervised learning settings.
subject: Articles
  topic: section
subject: Spectral neural networks
  topic: keywords
subject: Manifold learning
  topic: keywords
subject: Riemannian optimization
  topic: keywords
subject: Approximation with neural networks
  topic: keywords
subject: Graph Laplacian
  topic: keywords
creator: Chenghui Li
creator: Rishi Sonthalia
creator: Nicolás García Trillos
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2026-01-06
publication date: 2026-01-06
starting page: 19
ending page: 70
doi: 10.4208/jml.240322

Neural Stochastic Volterra Equations: Learning Path-Dependent Dynamics
type: item
description: Stochastic Volterra equations (SVEs) serve as mathematical models for the time evolutions of random systems with memory effects and irregular behaviour. We introduce neural stochastic Volterra equations as a physics-inspired architecture, generalizing the class of neural stochastic differential equations, and provide some theoretical foundation. Numerical experiments on various SVEs, like the disturbed pendulum equation, the generalized Ornstein-Uhlenbeck process, the rough Heston model and a monetary reserve dynamics, are presented, comparing the performance of neural SVEs, neural stochastic differential equations (SDEs) and Deep Operator Networks (DeepONets).
subject: Articles
  topic: section
subject: Feedforward neural network
  topic: keywords
subject: Deep operator network
  topic: keywords
subject: Neural stochastic differential equation
  topic: keywords
subject: Stochastic Volterra equation
  topic: keywords
subject: Supervised learning
  topic: keywords
creator: Martin Bergerhausen
creator: David J. Prömel
creator: David Scheffels
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2025-12-17
publication date: 2025-12-17
starting page: 264
ending page: 289
doi: 10.4208/jml.240926

Soft-Constrained Distance Preserving t-SNE
type: item
description: Dimension reduction is a crucial tool for high-dimensional data analysis. Many dimension reduction techniques have been proposed for preserving different properties of a given dataset. For data visualization, t-distributed stochastic neighborhood embedding (t-SNE) is a popular method due to its ability to produce nicely separated clusters. However, t-SNE suffers from some major drawbacks. In this paper, a distance preserving t-SNE (DPt-SNE) is proposed, aiming to capture the global structure of the data and simultaneously maintain its local cluster separation. The basic idea is to incorporate a set of soft constraints, i.e. relaxing expected pairwise distance preserving constraints in order to regulate the low-dimensional embedding to preserve global structure encoded in the given distance metric of input data. In addition, we introduce a scaling optimization parameter to alleviate potential issues that arise when the difference between high and low dimensional distances is too large to overcome. Experimental results on six datasets positively confirm that our DPt-SNE can better reveal global structure than t-SNE, while retaining competitive clustering separation.
subject: Articles
  topic: section
subject: t-SNE
  topic: keywords
subject: Distance preservation
  topic: keywords
subject: Dimensionality reduction
  topic: keywords
subject: Data visualization
  topic: keywords
creator: Joseph Balderas
creator: Li Wang
creator: Andrzej Korzeniowski
creator: Ren-Cang Li
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2025-12-10
publication date: 2025-12-10
starting page: 1
ending page: 18
doi: 10.4208/jml.250414

A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions
type: item
description: Neural networks are one tool for approximating non-linear differential equations used in scientific computing tasks such as surrogate modeling, real-time predictions, and optimal control. PDE foundation models utilize neural networks to train approximations to multiple differential equations simultaneously and are thus a general purpose solver that can be adapted to downstream tasks. Current PDE foundation models focus on either learning general solution operators and/or the governing system of equations, and thus only handle numerical or symbolic modalities. However, real-world applications may require more flexible data modalities, e.g. text analysis or descriptive outputs. To address this gap, we propose a novel multimodal deep learning approach that leverages a transformer-based architecture to approximate solution operators for a wide variety of ODEs and PDEs. Our method integrates numerical inputs, such as equation parameters and initial conditions, with text descriptions of physical processes or system dynamics. This enables our model to handle settings where symbolic representations may be incomplete or unavailable. In addition to providing accurate numerical predictions, our approach generates interpretable scientific text descriptions, offering deeper insights into the underlying dynamics and solution properties. The numerical experiments show that our model provides accurate solutions for in-distribution data (with average relative error less than 3.3%) and out-of-distribution data (average relative error less than 7.8%) together with precise text descriptions (with correct descriptions generated 100% of times). In certain tests, the model is also shown to be capable of extrapolating solutions in time.
subject: Articles
  topic: section
subject: Multimodal AI
  topic: keywords
subject: Generative AI
  topic: keywords
subject: PDE foundation model
  topic: keywords
subject: Transformer-based operator learning
  topic: keywords
subject: Scientific text generation
  topic: keywords
creator: Elisa Negrini
creator: Yuxuan Liu
creator: Liu Yang
creator: Stanley J. Osher
creator: Hayden Schaeffer
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2025-11-14
publication date: 2025-11-14
starting page: 290
ending page: 317
doi: 10.4208/jml.250214

Progressive Optimal Path Sampling for Closed-Loop Optimal Control Design with Deep Neural Networks
type: item
description: Closed-loop optimal control design for high-dimensional nonlinear systems has been a long-standing challenge. Traditional methods, such as solving the associated Hamilton-Jacobi-Bellman equation, suffer from the curse of dimensionality. Recent literature proposed a new promising approach based on supervised learning, by leveraging powerful open-loop optimal control solvers to generate training data and neural networks as efficient high-dimensional function approximators to fit the closed-loop optimal control. This approach successfully handles certain high-dimensional optimal control problems but still performs poorly on more challenging problems. One of the crucial reasons for the failure is the so-called distribution mismatch phenomenon brought by the controlled dynamics. In this paper, we investigate this phenomenon and propose the progressive optimal path sampling method to mitigate this problem. We theoretically prove that this enhanced sampling strategy outperforms both the vanilla approach and the widely used dataset aggregation method on the classical linear-quadratic regulator by a factor proportional to the total time duration. We further numerically demonstrate that the proposed sampling strategy significantly improves the performance on tested control problems, including the optimal landing problem of a quadrotor and the optimal reaching problem of a 7-DoF manipulator.
subject: Articles
  topic: section
subject: Closed-loop optimal control
  topic: keywords
subject: Distribution mismatch
  topic: keywords
subject: Adaptive sampling
  topic: keywords
subject: Supervised learning
  topic: keywords
creator: Xuanxi Zhang
creator: Jihao Long
creator: Wei Hu
creator: Weinan E
creator: Jiequn Han
rights: Copyright (c) 2025 Journal of Machine Learning
date: 2025-10-31
publication date: 2025-10-31
starting page: 223
ending page: 263
doi: 10.4208/jml.250213

Optimistic Estimate Uncovers the Potential of Nonlinear Models
type: item
description: We propose an optimistic estimate framework to evaluate the potential of nonlinear models in fitting target functions at overparameterization. In our framework, such potential is quantified by estimating the smallest possible sample size needed for a model to recover a target function, referred to as an optimistic sample size. Following the framework, we derive the optimistic sample sizes for matrix factorization models, deep models, and two-layer neural networks (NNs) with fully-connected or convolutional architectures. For each nonlinear model, we confirm via experiments that the target functions can be fitted at overparameterization as predicted by our analysis. Our results suggest a hierarchical inductive bias of nonlinear models towards simple functions with smaller optimistic sample sizes intrinsic to their architecture. The dynamical realization of the suggested inductive bias remains an open problem for the further study.
subject: Articles
  topic: section
subject: Nonlinear regression
  topic: keywords
subject: Optimistic sample size estimate
  topic: keywords
subject: Matrix factorization model
  topic: keywords
subject: Two-layer neural network
  topic: keywords
subject: Hierarchical inductive bias.
  topic: keywords
subject:
  topic: subjects
creator: Yaoyu Zhang
creator: Zhongwang Zhang
creator: Leyang Zhang
creator: Zhiwei Bai
creator: Tao Luo
creator: Zhi-Qin John Xu
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-09-12
publication date: 2025-09-12
starting page: 192
ending page: 222
doi: 10.4208/jml.231109

A Deep Uzawa-Lagrange Multiplier Approach for Boundary Conditions in PINNs and Deep Ritz Methods
type: item
description: We introduce a deep learning-based framework for weakly enforcing boundary conditions in the numerical approximation of partial differential equations. Building on existing physics-informed neural network and deep Ritz methods, we propose the Deep Uzawa algorithm, which incorporates Lagrange multipliers to handle boundary conditions effectively. This modification requires only a minor computational adjustment but ensures enhanced convergence properties and provably accurate enforcement of boundary conditions, even for singularly perturbed problems. We provide a comprehensive mathematical analysis demonstrating the convergence of the scheme and validate the effectiveness of the Deep Uzawa algorithm through numerical experiments, including high dimensional, singularly perturbed problems and those posed over non-convex domains.
subject: Articles
  topic: section
subject: Physics-informed neural networks
  topic: keywords
subject: Deep Ritz method
  topic: keywords
subject: Uzawa algorithm
  topic: keywords
subject: Lagrange multipliers
  topic: keywords
subject: Boundary condition enforcement.
  topic: keywords
subject:
  topic: subjects
creator: Charalambos G. Makridakis
creator: Aaron Pim
creator: Tristan Pryer
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-09-12
publication date: 2025-09-12
starting page: 166
ending page: 191
doi: 10.4208/jml.250107

Tangent Differential Privacy
type: item
description: Differential privacy is a framework for protecting the identity of individual data points in the decision-making process. In this note, we propose a new form of differential privacy, known as tangent differential privacy. Compared to the usual differential privacy, which is defined uniformly across data distributions, tangent differential privacy is tailored to a specific data distribution of interest. It also allows for general distribution distances such as total variation distance and Wasserstein distance. In the context of risk minimization, we demonstrate that entropic regularization ensures tangent differential privacy under relatively general conditions on the risk function.
subject: Articles
  topic: section
subject: Differential privacy
  topic: keywords
subject: Entropic regularization.
  topic: keywords
subject:
  topic: subjects
creator: Lexing Ying
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-09-12
publication date: 2025-09-12
starting page: 157
ending page: 165
doi: 10.4208/jml.240928

Cluster-Based Classification with Neural ODEs via Control
type: item
description: We address binary classification using neural ordinary differential equations from the perspective of simultaneous control of $N$ data points. We consider a single-neuron architecture with parameters fixed as piecewise constant functions of time. In this setting, the model complexity can be quantified by the number of control switches. Previous work has shown that classification can be achieved using a point-by-point strategy that requires $\mathcal{O}(N)$ switches. We propose a new control method that classifies any arbitrary dataset by sequentially steering clusters of $d$ points, thereby reducing the complexity to $\mathcal{O}(N/d)$ switches. The optimality of this result, particularly in high dimensions, is supported by some numerical experiments. Our complexity bound is sufficient but often conservative because same-class points tend to appear in larger clusters, simplifying classification. This motivates studying the probability distribution of the number of switches required. We introduce a simple control method that imposes a collinearity constraint on the parameters, and analyze a worst-case scenario where both classes have the same size and all points are i.i.d. Our results highlight the benefits of high-dimensional spaces, showing that classification using constant controls becomes more probable as $d$ increases.
subject: Articles
  topic: section
subject: Classification
  topic: keywords
subject: Complexity
  topic: keywords
subject: Neural ODEs
  topic: keywords
subject: Separability
  topic: keywords
subject: Controllability.
  topic: keywords
subject:
  topic: subjects
creator: Antonio Álvarez-Lόpez
creator: Rafael Orive-Illera
creator: Enrique Zuazua
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-06-03
publication date: 2025-06-03
starting page: 128
ending page: 156
doi: 10.4208/jml.241114

Convergence Analysis of Discrete Diffusion Model: Exact Implementation Through Uniformization
type: item
description: Diffusion models have achieved huge empirical success in data generation tasks. Recently, some efforts have been made to adapt the framework of diffusion models to discrete state space, providing a more natural approach for modeling intrinsically discrete data, such as language and graphs. This is achieved by formulating both the forward noising process and the corresponding reversed process as continuous time Markov chains. In this paper, we investigate the theoretical properties of the discrete diffusion model. Specifically, we introduce an algorithm leveraging the uniformization of continuous Markov chains, implementing transitions on random time points. Under reasonable assumptions on the learning of the discrete score function, we derive total variation distance and Kullback–Leibler divergence guarantees for sampling from any distribution on a hypercube. Our results align with state-of-the-art achievements for diffusion models in $\mathbb{R}^d$ and further underscore the advantages of discrete diffusion models in comparison to the $\mathbb{R}^d$ setting.
subject: Articles
  topic: section
subject: Diffusion model
  topic: keywords
subject: Sampling
  topic: keywords
subject: Machine learning theory.
  topic: keywords
subject:
  topic: subjects
creator: Hongrui Chen
creator: Lexing Ying
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-06-03
publication date: 2025-06-03
starting page: 108
ending page: 127
doi: 10.4208/jml.240812

Convergence of Stochastic Gradient Descent under a Local Łojasiewicz Condition for Deep Neural Networks
type: item
description: We study the convergence of stochastic gradient descent (SGD) for non-convex objective functions. We establish the local convergence with positive probability under the local Łojasiewicz condition introduced by Chatterjee [arXiv:2203.16462, 2022] and an additional local structural assumption of the loss function landscape. A key component of our proof is to ensure that the whole trajectories of SGD stay inside the local region with a positive probability. We also provide examples of neural networks with finite widths such that our assumptions hold.
subject: Articles
  topic: section
subject: Non-convex optimization
  topic: keywords
subject: Stochastic gradient descent
  topic: keywords
subject: Convergence analysis.
  topic: keywords
subject:
  topic: subjects
creator: Jing An
creator: Jianfeng Lu
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-06-03
publication date: 2025-06-03
starting page: 89
ending page: 107
doi: 10.4208/jml.240724

Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow
type: item
description: This paper presents a method for finding a sparse representation of Barron functions. Specifically, given an $L^2$ function $f,$ the inverse scale space flow is used to find a sparse measure $\mu$ minimising the $L^2$ loss between the Barron function associated to the measure $\mu$ and the function $f.$ The convergence properties of this method are analysed in an ideal setting and in the cases of measurement noise and sampling bias. In an ideal setting the objective decreases strictly monotone in time to a minimizer with $\mathcal{O}(1/t),$ and in the case of measurement noise or sampling bias the optimum is achieved up to a multiplicative or additive constant. This convergence is preserved on discretization of the parameter space, and the minimizers on increasingly fine discretizations converge to the optimum on the full parameter space.
subject: Articles
  topic: section
subject: Barron Space
  topic: keywords
subject: Bregman Iterations
  topic: keywords
subject: Sparse Neural Networks
  topic: keywords
subject: Inverse Scale Space
  topic: keywords
subject: Optimization.
  topic: keywords
subject:
  topic: subjects
creator: Tjeerd Jan Heeringa
creator: Tim Roith
creator: Christoph Brune
creator: Martin Burger
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-03-12
publication date: 2025-03-12
starting page: 48
ending page: 88
doi: 10.4208/jml.240123

Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces
type: item
description: We present the development and analysis of a reinforcement learning algorithm designed to solve continuous-space mean field game (MFG) and mean field control (MFC) problems in a unified manner. The proposed approach pairs the actor-critic (AC) paradigm with a representation of the mean field distribution via a parameterized score function, which can be efficiently updated in an online fashion, and uses Langevin dynamics to obtain samples from the resulting distribution. The AC agent and the score function are updated iteratively to converge, either to the MFG equilibrium or the MFC optimum for a given mean field problem, depending on the choice of learning rates. A straightforward modification of the algorithm allows us to solve mixed mean field control games. The performance of our algorithm is evaluated using linear-quadratic benchmarks in the asymptotic infinite horizon framework.
subject: Articles
  topic: section
subject: Actor-critic
  topic: keywords
subject: Linear-quadratic control
  topic: keywords
subject: Mean field game
  topic: keywords
subject: Mean field control
  topic: keywords
subject: Mixed mean field control game
  topic: keywords
subject: Score matching
  topic: keywords
subject: Reinforcement learning
  topic: keywords
subject: Timescales.
  topic: keywords
subject:
  topic: subjects
creator: Andrea Angiuli
creator: Jean-Pierre Fouque
creator: Ruimeng Hu
creator: Alan Raydan
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-03-12
publication date: 2025-03-12
starting page: 11
ending page: 47
doi: 10.4208/jml.230919

A Note on Continuous-Time Online Learning
type: item
description: In online learning, the data is provided in sequential order, and the goal of the learner is to make online decisions to minimize overall regrets. This note is concerned with continuous-time models and algorithms for several online learning problems: online linear optimization, adversarial bandit, and adversarial linear bandit. For each problem, we extend the discrete-time algorithm to the continuous-time setting and provide a concise proof of the optimal regret bound.
subject: Articles
  topic: section
subject: Online learning
  topic: keywords
subject: Online optimization
  topic: keywords
subject: Adversarial bandits
  topic: keywords
subject: Adversarial linear bandits.
  topic: keywords
subject:
  topic: subjects
creator: Lexing Ying
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2025-03-12
publication date: 2025-03-12
starting page: 1
ending page: 10
doi: 10.4208/jml.240605

MAFE-Net: A Multi-Level Attention Feature Extraction Network for Pancreas Segmentation
type: item
description: Accurate and automatic segmentation of the pancreas from abdominal computed tomography (CT) scans is crucial for diagnosing and treating pancreatic diseases. However, the pancreas is a tiny target abdominal organ with high anatomical variability and low tissue contrast in CT scans, making segmentation tasks challenging. To address this challenge, we propose a multilevel attention feature extraction network to segment the pancreas in abdominal CT images. Specifically, a multi-field attention convolution module (MFAC) and a connection feature fusion module (CFF) are added to the encoding and decoding structure to improve the extraction of pancreatic features. To further enhance the segmentation network’s extraction of pancreatic edge features, we propose a decoding feature recall module (DFC), which can be migrated to other encoding and decoding structures and pruned to capture pancreatic edge information better. We compared the performance of our method with that of the most advanced method on the NIH pancreatic segmentation dataset and the challenging pancreatic cancer CT image dataset collected by the Zhujiang Hospital of Southern Medical University. The experimental results show that the DSC of our method on NIH dataset and pancreatic cancer dataset is 84.69% and 78.18% respectively, which is superior to the existing methods.
subject: Articles
  topic: section
subject: Pancreas segmentation
  topic: keywords
subject: Multi-field attention convolution
  topic: keywords
subject: Connection feature fusion
  topic: keywords
subject: Decoding feature recall.
  topic: keywords
subject:
  topic: subjects
creator: Jiawei Chen
creator: Wenjie Chen
creator: Zhipeng Zhu
creator: Qi Ye
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-11-07
publication date: 2024-11-07
starting page: 445
ending page: 463
doi: 10.4208/jml.231101

Variational Formulations of ODE-Net as a Mean-Field Optimal Control Problem and Existence Results
type: item
description: This paper presents a mathematical analysis of ODE-Net, a continuum model of deep neural networks (DNNs). In recent years, machine learning researchers have introduced ideas of replacing the deep structure of DNNs with ODEs as a continuum limit. These studies regard the “learning” of ODE-Net as the minimization of a “loss” constrained by a parametric ODE. Although the existence of a minimizer for this minimization problem needs to be assumed, only a few studies have investigated the existence analytically in detail. In the present paper, the existence of a minimizer is discussed based on a formulation of ODE-Net as a measure-theoretic mean-field optimal control problem. The existence result is proved when a neural network describing a vector field of ODE-Net is linear with respect to learnable parameters. The proof employs the measure-theoretic formulation combined with the direct method of calculus of variations. Secondly, an idealized minimization problem is proposed to remove the above linearity assumption. Such a problem is inspired by a kinetic regularization associated with the Benamou-Brenier formula and universal approximation theorems for neural networks.
subject: Articles
  topic: section
subject: Deep learning
  topic: keywords
subject: ResNet
  topic: keywords
subject: ODE-Net
  topic: keywords
subject: Benamou-Brenier formula
  topic: keywords
subject: Mean-field game.
  topic: keywords
subject:
  topic: subjects
creator: Noboru Isobe
creator: Mizuho Okumura
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-11-07
publication date: 2024-11-07
starting page: 413
ending page: 444
doi: 10.4208/jml.231210

Enhancing Accuracy in Deep Learning Using Random Matrix Theory
type: item
description: We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that reduces the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and convolutional neural network (CNNs). Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models.
subject: Articles
  topic: section
subject: Deep learning
  topic: keywords
subject: Marchenko-Pastur distribution
  topic: keywords
subject: Random matrix theory
  topic: keywords
subject: Increasing accuracy
  topic: keywords
subject: Pruning.
  topic: keywords
subject:
  topic: subjects
creator: Leonid Berlyand
creator: Etienne Sandier
creator: Yitzchak Shmalo
creator: Lei Zhang
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-11-07
publication date: 2024-11-07
starting page: 347
ending page: 412
doi: 10.4208/jml.231220

Memory$^3$: Language Modeling with Explicit Memory
type: item
description: The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining “abstract knowledge”. As a preliminary proof of concept, we train from scratch a 2.4 B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named ${\rm Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.
subject: Articles
  topic: section
subject: Large language model
  topic: keywords
subject: Explicit memory
  topic: keywords
subject: Large-scale pretraining
  topic: keywords
subject: Efficient inference
  topic: keywords
subject: AI database.
  topic: keywords
subject:
  topic: subjects
creator: Hongkang Yang
creator: Zehao Lin
creator: Wenjin Wang
creator: Hao Wu
creator: Zhiyu Li
creator: Bo Tang
creator: Wenqiang Wei
creator: Jinbo Wang
creator: Zeyun Tang
creator: Shichao Song
creator: Chenyang Xi
creator: Yu Yu
creator: Kai Chen
creator: Feiyu Xiong
creator: Linpeng Tang
creator: Weinan E
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-09-21
publication date: 2024-09-21
starting page: 300
ending page: 346
doi: 10.4208/jml.240708

Fast Gradient Computation for Gromov-Wasserstein Distance
type: item
description: The Gromov-Wasserstein distance is a notable extension of optimal transport. In contrast to the classic Wasserstein distance, it solves a quadratic assignment problem that minimizes the pair-wise distance distortion under the transportation of distributions and thus could apply to distributions in different spaces. These properties make Gromov-Wasserstein widely applicable to many fields, such as computer graphics and machine learning. However, the computation of the Gromov-Wasserstein distance and transport plan is expensive. The well-known Entropic Gromov-Wasserstein approach has a cubic complexity since the matrix multiplication operations need to be repeated in computing the gradient of Gromov-Wasserstein loss. This becomes a key bottleneck of the method. Currently, existing methods accelerate the computation focus on sampling and approximation, which leads to low accuracy or incomplete transport plans. In this work, we propose a novel method to accelerate accurate gradient computation by dynamic programming techniques, reducing the complexity from cubic to quadratic. In this way, the original computational bottleneck is broken and the new entropic solution can be obtained with total quadratic time, which is almost optimal complexity. Furthermore, it can be extended to some variants easily. Extensive experiments validate the efficiency and effectiveness of our method.
subject: Articles
  topic: section
subject: Optimal transport
  topic: keywords
subject: Gromov-Wasserstein distance
  topic: keywords
subject: Fast gradient computation algorithm
  topic: keywords
subject: Fast algorithm.
  topic: keywords
subject:
  topic: subjects
creator: Wei Zhang
creator: Zihao Wang
creator: Jie Fan
creator: Hao Wu
creator: Yong Zhang
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-09-21
publication date: 2024-09-21
starting page: 282
ending page: 299
doi: 10.4208/jml.240416

Convergence of Stochastic Gradient Descent Schemes for Łojasiewicz-Landscapes
type: item
description: In this article, we consider convergence of stochastic gradient descent schemes (SGD), including momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is only a countable number of critical points or if the objective function satisfies Łojasiewicz-inequalities around all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of staying bounded, if the random variables modelling the signal and response in the training are compactly supported.
subject: Articles
  topic: section
subject: Stochastic gradient descent
  topic: keywords
subject: Stochastic approximation
  topic: keywords
subject: Robbins-Monro
  topic: keywords
subject: Almost sure convergence
  topic: keywords
subject: Łojasiewicz-inequality.
  topic: keywords
subject:
  topic: subjects
creator: Steffen Dereich
creator: Sebastian Kassing
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-09-21
publication date: 2024-09-21
starting page: 245
ending page: 281
doi: 10.4208/jml.240109

Solving Bivariate Kinetic Equations for Polymer Diffusion Using Deep Learning
type: item
description: In this paper, we derive a class of backward stochastic differential equations (BSDEs) for infinite-dimensionally coupled nonlinear parabolic partial differential equations, thereby extending the deep BSDE method. In addition, we introduce a class of polymer dynamics models that accompany polymerization and depolymerization reactions, and derive the corresponding Fokker-Planck equations and Feynman-Kac equations. Due to chemical reactions, the system exhibits a Brownian yet non-Gaussian phenomenon, and the resulting equations are infinitely dimensionally coupled. We solve these equations numerically through our new deep BSDE method, and also solve a class of high-dimensional nonlinear equations, which verifies the effectiveness and shows approximation accuracy of the algorithm.
subject: Articles
  topic: section
subject: BSDEs
  topic: keywords
subject: Deep BSDE method
  topic: keywords
subject: Polymer dynamics
  topic: keywords
subject: Brownian yet non-Gaussian.
  topic: keywords
subject:
  topic: subjects
creator: Heng Wang
creator: Weihua Deng
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-06-27
publication date: 2024-06-27
starting page: 215
ending page: 244
doi: 10.4208/jml.240124

Approximation Results for Gradient Flow Trained Neural Networks
type: item
description: The paper contains approximation guarantees for neural networks that are trained with gradient flow, with error measured in the continuous $L_2(\mathbb{S}^{d−1 )}$-norm on the $d$-dimensional unit sphere and targets that are Sobolev smooth. The networks are fully connected of constant depth and increasing width. We show gradient flow convergence based on a neural tangent kernel (NTK) argument for the non-convex optimization of the second but last layer. Unlike standard NTK analysis, the continuous error norm implies an under-parametrized regime, possible by the natural smoothness assumption required for approximation. The typical over-parametrization re-enters the results in form of a loss in approximation rate relative to established approximation methods for Sobolev smooth functions.
subject: Articles
  topic: section
subject: Deep neural networks
  topic: keywords
subject: Approximation
  topic: keywords
subject: Gradient descent
  topic: keywords
subject: Neural tangent kernel.
  topic: keywords
subject:
  topic: subjects
creator: Gerrit Welper
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-06-27
publication date: 2024-06-27
starting page: 107
ending page: 175
doi: 10.4208/jml.230924

Mean-Field Neural Networks-Based Algorithms for McKean-Vlasov Control Problems
type: item
description: This paper is devoted to the numerical resolution of McKean-Vlasov control problems via the class of mean-field neural networks introduced in our companion paper [Pham and Warin, Neural Netw., 168, 2023] in order to learn the solution on the Wasserstein space. We propose several algorithms either based on dynamic programming with control learning by policy or value iteration, or backward stochastic differential equation SDE from stochastic maximum principle with global or local loss functions. Extensive numerical results on different examples are presented to illustrate the accuracy of each of our eight algorithms. We discuss and compare the pros and cons of all the tested methods.
subject: Articles
  topic: section
subject: McKean-Vlasov control
  topic: keywords
subject: Mean-field neural networks
  topic: keywords
subject: Learning on Wasserstein space
  topic: keywords
subject: Dynamic programming
  topic: keywords
subject: Backward SDE.
  topic: keywords
subject:
  topic: subjects
creator: Huyên Pham
creator: Xavier Warin
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-06-27
publication date: 2024-06-27
starting page: 176
ending page: 214
doi: 10.4208/jml.230106

Semi-Supervised Clustering of Sparse Graphs: Crossing the Information-Theoretic Threshold
type: item
description: The stochastic block model is a canonical random graph model for clustering and community detection on network-structured data. Decades of extensive study on the problem have established many profound results, among which the phase transition at the Kesten-Stigum threshold is particularly interesting both from a mathematical and an applied standpoint. It states that no estimator based on the network topology can perform substantially better than chance on sparse graphs if the model parameter is below a certain threshold. Nevertheless, if we slightly extend the horizon to the ubiquitous semi-supervised setting, such a fundamental limitation will disappear completely. We prove that with an arbitrary fraction of the labels revealed, the detection problem is feasible throughout the parameter domain. Moreover, we introduce two efficient algorithms, one combinatorial and one based on optimization, to integrate label information with graph structures. Our work brings a new perspective to the stochastic model of networks and semidefinite program research.
subject: Articles
  topic: section
subject: Clustering
  topic: keywords
subject: Semi-supervised learning
  topic: keywords
subject: Stochastic block model
  topic: keywords
subject: Kesten-Stigum threshold
  topic: keywords
subject: Semidefinite programming.
  topic: keywords
subject:
  topic: subjects
creator: Junda Sheng
creator: Thomas Strohmer
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-03-21
publication date: 2024-03-21
starting page: 64
ending page: 106
doi: 10.4208/jml.230624

Stochastic Delay Differential Games: Financial Modeling and Machine Learning Algorithms
type: item
description: In this paper, we propose a numerical methodology for finding the closed-loop Nash equilibrium of stochastic delay differential games through deep learning. These games are prevalent in finance and economics where multi-agent interaction and delayed effects are often desired features in a model, but are introduced at the expense of increased dimensionality of the problem. This increased dimensionality is especially significant as that arising from the number of players is coupled with the potential infinite dimensionality caused by the delay. Our approach involves parameterizing the controls of each player using distinct recurrent neural networks. These recurrent neural network-based controls are then trained using a modified version of Brown’s fictitious play, incorporating deep learning techniques. To evaluate the effectiveness of our methodology, we test it on finance-related problems with known solutions. Furthermore, we also develop new problems and derive their analytical Nash equilibrium solutions, which serve as additional benchmarks for assessing the performance of our proposed deep learning approach.
subject: Articles
  topic: section
subject: Stochastic delay differential games
  topic: keywords
subject: Deep fictitious play
  topic: keywords
subject: Recurrent neural networks
  topic: keywords
subject: Portfolio games.
  topic: keywords
subject:
  topic: subjects
creator: Robert Balkin
creator: Hector D. Ceniceros
creator: Ruimeng Hu
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-03-21
publication date: 2024-03-21
starting page: 23
ending page: 63
doi: 10.4208/jml.230713

On the Existence of Optimal Shallow Feedforward Networks with ReLU Activation
type: item
description: We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network.
subject: Articles
  topic: section
subject: Neural Networks
  topic: keywords
subject: Shallow Networks
  topic: keywords
subject: Best Approximation
  topic: keywords
subject: ReLU Activation
  topic: keywords
subject: Approximatively Compact.
  topic: keywords
subject:
  topic: subjects
creator: Steffen Dereich
creator: Sebastian Kassing
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2024-03-21
publication date: 2024-03-21
starting page: 1
ending page: 22
doi: 10.4208/jml.230903

Embedding Inequalities for Barron-Type Spaces
type: item
description: An important problem in machine learning theory is to understand the approximation and generalization properties of two-layer neural networks in high dimensions. To this end, researchers have introduced the Barron space $\mathcal{B}_s(Ω)$ and the spectral Barron space $\mathcal{F}_s(Ω),$ where the index $s ∈ [0, ∞)$ indicates the smoothness of functions within these spaces and $Ω ⊂ \mathbb{R}^d$ denotes the input domain. However, the precise relationship between the two types of Barron spaces remains unclear. In this paper, we establish a continuous embedding between them as implied by the following inequality: For any $\delta∈ (0, 1),s ∈ \mathbb{N}^+$ and $f : Ω\mapsto \mathbb{R},$ it holds that $$\delta||f||_{\mathcal{F}_{s−\delta}(Ω)}\lesssim_s||f||_{\mathcal{B}_s(Ω)}\lesssim_s||f||_{\mathcal{F}_{s+1}(Ω).}$$ Importantly, the constants do not depend on the input dimension $d,$ suggesting that the embedding is effective in high dimensions. Moreover, we also show that the lower and upper bound are both tight.
subject: Articles
  topic: section
subject: Barron space
  topic: keywords
subject: Two-layer neural network
  topic: keywords
subject: High-dimensional approximation
  topic: keywords
subject: Embedding theorem
  topic: keywords
subject: Fourier transform.
  topic: keywords
subject:
  topic: subjects
creator: Lei Wu
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2023-12-28
publication date: 2023-12-28
starting page: 259
ending page: 270
doi: 10.4208/jml.230530

Prompt Engineering Through the Lens of Optimal Control
type: item
description: Prompt engineering (PE) has emerged as a critical technique for guiding large language models (LLMs) in solving intricate tasks. Its importance is highlighted by its potential to significantly enhance the efficiency and effectiveness of human-machine interaction. As tasks grow increasingly complex, recent advanced PE methods have extended beyond the limitations of single-round interactions to embrace multi-round interactions, which allows for a deeper and more nuanced engagement with LLMs. In this paper, we propose an optimal control framework tailored for multi-round interactions with LLMs. This framework provides a unified mathematical structure that not only systematizes the existing PE methods but also sets the stage for rigorous analytical improvements. Furthermore, we extend this framework to include PE via ensemble methods and multi-agent collaboration, thereby enlarging the scope of applicability. By adopting an optimal control perspective, we offer fresh insights into existing PE methods and highlight theoretical challenges that warrant future research. Besides, our work lays a foundation for the development of more effective and interpretable PE methods.
subject: Articles
  topic: section
subject: Large language models
  topic: keywords
subject: Prompt engineering
  topic: keywords
subject: Optimal control.
  topic: keywords
subject:
  topic: subjects
creator: Yifan Luo
creator: Yiming Tang
creator: Chengfeng Shen
creator: Zhenan Zhou
creator: Bin Dong
rights: Copyright (c) 2025 COPYRIGHT: © Global Science Press
date: 2023-12-28
publication date: 2023-12-28
starting page: 241
ending page: 258
doi: 10.4208/jml.231023