Convex and Nonconvex Optimization Geometries

As many machine learning and signal processing problems are fundamentally nonconvex and too expensive/difficult to be convexified, my research is focused on understanding the optimization landscapes of their fundamentally nonconvex formulations. After understanding their optimization landscapes, we can develop optimization algorithms to efficiently navigate these optimization landscapes and achieve the global optimality convergence. So, the main theme of this thesis would be optimization, with an emphasis on nonconvex optimization and algorithmic developments for these popular optimization problems. This thesis can be conceptually divided into four parts: Part 1: Convex Optimization. In the first part, we apply convex relaxations to several popular nonconvex problems in signal processing and machine learning (e.g. line spectral estimation problem and tensor decomposition problem) and prove that the solving the new convex relaxation problems is guaranteed to achieve the globally optimal solutions of their original nonconvex formulations. Part 2: Nonconvex Optimization. Though convex relaxation is an elegant approach to deal with nonconvex optimization problems and often achieves information theoretical complexity, still many nonconvex machine learning and signal processing problems are too expensive/difficult to be convexified. Therefore, in the second part, we are focused on the fundamentally nonconvex optimization landscapes for several low-rank matrix optimization problems with general objective functions, which covers a massive number of popular problems in signal processing and machine learning. In particular, we develop mild conditions for these general low-rank matrix optimization problems to have a benign landscape: all second-order stationary points are global optimal solutions and all saddle points are strict saddles (i.e. Hessian matrix has a negative eigenvalue at these points). Part 3: Algorithms. In this part, we are focused on developing optimization algorithms with provable second-order optimality convergence for general nonconvex and non-Lipschitz problems. Further, in this part, we also solve an open problem for the second-order convergence of alternating minimization algorithms that have been widely used in practice to solve large-scale nonconvex problems due to their simple implementation, fast convergence, and superb empirical performance. Then the second-order convergence guarantees, along with the knowledge (see Part 2) that a massive number of nonconvex optimization problems have been shown to have a benign landscape (all second-order stationary points are global minima ensure that the proposed algorithms can find global minima for a class of nonconvex problems. Part 4: Applications. In this part, we combine the developed algorithms and landscape information to deal with several popular applications in signal processing and machine learning, e.g., the low-rank tensor recovery problem and the spherical Principal Component Analysis (PCA)