Low-rank methods in reinforcement learning

The increasing complexity of modern decision-making systems has propelled reinforcement learning (RL) into the spotlight, offering a framework for adaptive control and optimization in high-dimensional environments. Despite its successes, RL faces significant challenges, including the curse of dimensionality and high sample complexity, which hinder scalability and efficiency. Traditional non-parametric methods often fall short in addressing these challenges, necessitating novel approaches that balance expressiveness and computational feasibility. This dissertation introduces low-rank matrix and tensor-based methodologies as a unifying framework to address core challenges in RL, particularly for value function (VF) approximation and policy learning. By leveraging low-rank structures, we achieve scalable representations that reduce computational demands while maintaining theoretical rigor and practical applicability. The contributions of this thesis are organized as follows.

In the first part, we tackle VF approximation in unconstrained Markov Decision Processes (MDPs), a cornerstone of RL, by developing stochastic low-rank algorithms for matrix and tensor representations. Classical non-parametric VF estimation suffers from exponential growth in complexity with increasing state and action dimensions. To overcome this, we propose a shift from the traditional matrix-based representation of VFs to tensors, which better accommodate the multi-dimensional nature of state and action spaces. To promote parsimony, we enforce low rank on the matrix and tensor representation of VFs. Then, the VF matrix or tensor is estimated using block-coordinate methods from online samples of the MDP. By replacing traditional VF matrix representations with tensors, we capture higher-order interactions among state-action pairs. Theoretical analysis and numerical results demonstrate the efficacy of the proposed methods in standardized RL environments and in a more realistic wireless communication network problem.

In the second part, and following the same philosophy, we address the problem of learning multiple MDPs, or tasks, concurrently. To that end, we exploit task similarities by modeling the VFs of the multiple tasks jointly as a low-rank tensor. Recognizing that related tasks often share structural patterns, we impose the low-rank condition to simultaneously infer task similarities and estimate the VFs. This enables efficient learning from limited data while circumventing the need to explicitly predefine task relationships. Our methods are evaluated in two environments, a collection of inverted pendulums, and a collection of wireless communication networks, showcasing their robustness and scalability.

The third part extends these ideas to finite-horizon MDPs, where the non-stationarity of the policies exacerbates the challenges of high-dimensional VFs. We model the VFs as low-rank tensors, with time being an extra dimension of the tensor. Next, we develop an optimization-based framework for solving the finite-horizon Bellman equations. This framework includes block-coordinate methods, which offer convergence guarantees. Furthermore, when the dynamics are unknown, the proposed framework is extended to stochastic setups where learning takes place from sampled trajectories. Numerical experiments validate the computational and sample efficiency of our approach in two resource allocation scenarios: the wireless communication network and a battery charging problem.

Lastly, we investigate the use of low-rank tensors as multi-linear mappings for policy gradient (PG) methods. While neural networks dominate policy learning, they often face challenges related to convergence and redundancy in state-action representations. We propose tensor-based low-rank policies, where the policy parameters are structured as tensors and optimized using tensor-completion techniques. This approach enforces multi-linear mappings, reducing computational and sample complexities without compromising performance. Theoretical guarantees and numerical experiments underscore the advantages of low-rank tensor policies in a suite of standard RL problems, as well as in the wireless communication network problem.

In summary, this dissertation presents a cohesive framework for incorporating low-rank structures into RL to address fundamental challenges in VF approximation, policy learning, and multitask optimization. Through rigorous theoretical analysis and extensive experimental validation, we demonstrate that the proposed methods achieve significant reductions in computational and sample complexity while preserving accuracy and scalability. The findings highlight the potential of low-rank methodologies to enable practical RL systems, paving the way for more efficient and adaptive decision-making frameworks.