Efficient Reinforcement Learning Through Uncertainties

Download Efficient Reinforcement Learning Through Uncertainties full books in PDF, epub, and Kindle. Read online Efficient Reinforcement Learning Through Uncertainties ebook anywhere anytime directly on your device. Fast Download speed and no annoying ads. We cannot guarantee that every ebooks is available!

Efficient Reinforcement Learning Through Uncertainties

Author : Dongruo Zhou
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (138 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning Through Uncertainties by : Dongruo Zhou

Download or read book Efficient Reinforcement Learning Through Uncertainties written by Dongruo Zhou and published by . This book was released on 2023 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: This dissertation is centered around the concept of uncertainty-aware reinforcement learning (RL), which seeks to enhance the efficiency of RL by incorporating uncertainty. RL is a vital mathematical framework in the field of artificial intelligence (AI) for creating autonomous agents that can learn optimal behaviors through interaction with their environments. However, RL is often criticized for being sample inefficient and computationally demanding. To tackle these challenges, the primary goals of this dissertation are twofold: to offer theoretical understanding of uncertainty-aware RL and to develop practical algorithms that utilize uncertainty to enhance the efficiency of RL. Our first objective is to develop an RL approach that is efficient in terms of sample usage for Markov Decision Processes (MDPs) with large state and action spaces. We present an uncertainty-aware RL algorithm that incorporates function approximation. We provide theoretical proof that this algorithm achieves near minimax optimal statistical complexity when learning the optimal policy. In our second objective, we address two specific scenarios: the batch learning setting and the rare policy switch setting. For both settings, we propose uncertainty-aware RL algorithms with limited adaptivity. These algorithms significantly reduce the number of policy switches compared to previous baseline algorithms while maintaining a similar level of statistical complexity. Lastly, we focus on estimating uncertainties in neural network-based estimation models. We introduce a gradient-based method that effectively computes these uncertainties. Our approach is computationally efficient, and the resulting uncertainty estimates are both valid and reliable. The methods and techniques presented in this dissertation contribute to the advancement of our understanding regarding the fundamental limits of RL. These research findings pave the way for further exploration and development in the field of decision-making algorithm design.

Efficient Reinforcement Learning Using Gaussian Processes

Author : Marc Peter Deisenroth
Publisher : KIT Scientific Publishing
ISBN 13 : 3866445695
Total Pages : 226 pages
Book Rating : 4.8/5 (664 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning Using Gaussian Processes by : Marc Peter Deisenroth

Download or read book Efficient Reinforcement Learning Using Gaussian Processes written by Marc Peter Deisenroth and published by KIT Scientific Publishing. This book was released on 2010 with total page 226 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book examines Gaussian processes in both model-based reinforcement learning (RL) and inference in nonlinear dynamic systems.First, we introduce PILCO, a fully Bayesian approach for efficient RL in continuous-valued state and action spaces when no expert knowledge is available. PILCO takes model uncertainties consistently into account during long-term planning to reduce model bias. Second, we propose principled algorithms for robust filtering and smoothing in GP dynamic systems.

Model-Based Reinforcement Learning

Author : Milad Farsi
Publisher : John Wiley & Sons
ISBN 13 : 1119808596
Total Pages : 276 pages
Book Rating : 4.1/5 (198 download)

DOWNLOAD NOW!

Book Synopsis Model-Based Reinforcement Learning by : Milad Farsi

Download or read book Model-Based Reinforcement Learning written by Milad Farsi and published by John Wiley & Sons. This book was released on 2022-12-02 with total page 276 pages. Available in PDF, EPUB and Kindle. Book excerpt: Model-Based Reinforcement Learning Explore a comprehensive and practical approach to reinforcement learning Reinforcement learning is an essential paradigm of machine learning, wherein an intelligent agent performs actions that ensure optimal behavior from devices. While this paradigm of machine learning has gained tremendous success and popularity in recent years, previous scholarship has focused either on theory—optimal control and dynamic programming – or on algorithms—most of which are simulation-based. Model-Based Reinforcement Learning provides a model-based framework to bridge these two aspects, thereby creating a holistic treatment of the topic of model-based online learning control. In doing so, the authors seek to develop a model-based framework for data-driven control that bridges the topics of systems identification from data, model-based reinforcement learning, and optimal control, as well as the applications of each. This new technique for assessing classical results will allow for a more efficient reinforcement learning system. At its heart, this book is focused on providing an end-to-end framework—from design to application—of a more tractable model-based reinforcement learning technique. Model-Based Reinforcement Learning readers will also find: A useful textbook to use in graduate courses on data-driven and learning-based control that emphasizes modeling and control of dynamical systems from data Detailed comparisons of the impact of different techniques, such as basic linear quadratic controller, learning-based model predictive control, model-free reinforcement learning, and structured online learning Applications and case studies on ground vehicles with nonholonomic dynamics and another on quadrator helicopters An online, Python-based toolbox that accompanies the contents covered in the book, as well as the necessary code and data Model-Based Reinforcement Learning is a useful reference for senior undergraduate students, graduate students, research assistants, professors, process control engineers, and roboticists.

Posterior Sampling for Efficient Reinforcement Learning

Author : Vikranth Reddy Dwaracherla
Publisher :
ISBN 13 :
Total Pages : pages
Book Rating : 4.:/5 (128 download)

DOWNLOAD NOW!

Book Synopsis Posterior Sampling for Efficient Reinforcement Learning by : Vikranth Reddy Dwaracherla

Download or read book Posterior Sampling for Efficient Reinforcement Learning written by Vikranth Reddy Dwaracherla and published by . This book was released on 2021 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Reinforcement learning has shown tremendous success over the past few years. Much of this recent success can be attributed to agents learning from an inordinate amount of data in simulated environments. In order to achieve similar success in real environments, it is crucial to address data efficiency. Uncertainty quantification plays a prominent role in designing an intelligent agent which exhibits data efficiency. An agent which has a notion of uncertainty can trade-off between exploration and exploitation and explore in an intelligent manner. Such an agent should not only consider immediate information gain from an action but also its consequences on future learning prospects. An agent which has this capability is said to exhibit deep exploration. Algorithms that tackle deep exploration, so far, have relied on epistemic uncertainty representation through ensembles or other hypermodels, exploration bonuses, or visitation count distributions. An open question is whether deep exploration can be achieved by an incremental reinforcement learning algorithm that tracks a single point estimate, without additional complexity required to account for epistemic uncertainty. We answer this question in the affirmative. In this dissertation, we develop Langevin DQN, a variation of DQN that differs only in perturbing parameter updates with Gaussian noise, and demonstrate through a computational study that Langevin DQN achieves deep exploration. This is the first algorithm that demonstratively achieves deep exploration using a single-point estimate. We also present index sampling, a novel method for efficiently generating approximate samples from a posterior over complex models such as neural networks, induced by a prior distribution over the model family and a set of input-output data pairs. In addition, we develop posterior sampling networks, a new approach to model this distribution over models. We are particularly motivated by the application of our method to tackle reinforcement learning problems, but it could be of independent interest to the Bayesian deep learning community. Our method is especially useful in RL when we use complex exploration schemes, which make use of more than a single sample from the posterior, such as information directed sampling. Finally, we present some preliminary results demonstrating that the Langevin DQN update rule could be used to train posterior sampling networks, as an alternative to index sampling, and further improve data efficiency.

TEXPLORE: Temporal Difference Reinforcement Learning for Robots and Time-Constrained Domains

Author : Todd Hester
Publisher : Springer
ISBN 13 : 3319011685
Total Pages : 170 pages
Book Rating : 4.3/5 (19 download)

DOWNLOAD NOW!

Book Synopsis TEXPLORE: Temporal Difference Reinforcement Learning for Robots and Time-Constrained Domains by : Todd Hester

Download or read book TEXPLORE: Temporal Difference Reinforcement Learning for Robots and Time-Constrained Domains written by Todd Hester and published by Springer. This book was released on 2013-06-22 with total page 170 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents and develops new reinforcement learning methods that enable fast and robust learning on robots in real-time. Robots have the potential to solve many problems in society, because of their ability to work in dangerous places doing necessary jobs that no one wants or is able to do. One barrier to their widespread deployment is that they are mainly limited to tasks where it is possible to hand-program behaviors for every situation that may be encountered. For robots to meet their potential, they need methods that enable them to learn and adapt to novel situations that they were not programmed for. Reinforcement learning (RL) is a paradigm for learning sequential decision making processes and could solve the problems of learning and adaptation on robots. This book identifies four key challenges that must be addressed for an RL algorithm to be practical for robotic control tasks. These RL for Robotics Challenges are: 1) it must learn in very few samples; 2) it must learn in domains with continuous state features; 3) it must handle sensor and/or actuator delays; and 4) it should continually select actions in real time. This book focuses on addressing all four of these challenges. In particular, this book is focused on time-constrained domains where the first challenge is critically important. In these domains, the agent’s lifetime is not long enough for it to explore the domains thoroughly, and it must learn in very few samples.

Efficient Model-based Exploration in Continuous State-space Environments

Author : Ali Nouri
Publisher :
ISBN 13 :
Total Pages : 169 pages
Book Rating : 4.:/5 (76 download)

DOWNLOAD NOW!

Book Synopsis Efficient Model-based Exploration in Continuous State-space Environments by : Ali Nouri

Download or read book Efficient Model-based Exploration in Continuous State-space Environments written by Ali Nouri and published by . This book was released on 2011 with total page 169 pages. Available in PDF, EPUB and Kindle. Book excerpt: The impetus for exploration in reinforcement learning (RL) is decreasing uncertainty about the environment for the purpose of better decision making. As such, exploration plays a crucial role in the efficiency of RL algorithms. In this dissertation, I consider continuous state control problems and introduce a new methodology for representing uncertainty that engenders more efficient algorithms. I argue that the new notion of uncertainty allows for more efficient use of function approximation, which is essential for learning in continuous spaces. In particular, I focus on a class of algorithms referred to as model-based methods and develop several such algorithms that are much more efficient than the current state-of-the-art methods. These algorithms attack the long-standing "curse of dimensionality''-- learning complexity often scales exponentially with problem dimensionality. I introduce algorithms that can exploit the dependency structure between state variables to exponentially decrease the sample complexity of learning, both in cases where the dependency structure is provided by the user a priori and cases where the algorithm has to find it on its own. I also use the new uncertainty notion to derive a multi-resolution exploration scheme, and demonstrate how this new technique achieves anytime behavior, which is very important in real-life applications. Finally, using a set of rich experiments, I show how the new exploration mechanisms affect the efficiency of learning, especially in real-life domains where acquiring samples is expensive.

Efficient Reinforcement Learning Through Variance Reduction and Trajectory Synthesis

Author : Xiaoming Zhao
Publisher :
ISBN 13 :
Total Pages : pages
Book Rating : 4.:/5 (129 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning Through Variance Reduction and Trajectory Synthesis by : Xiaoming Zhao

Download or read book Efficient Reinforcement Learning Through Variance Reduction and Trajectory Synthesis written by Xiaoming Zhao and published by . This book was released on 2019 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt:

Towards the Understanding of Sample Efficient Reinforcement Learning Algorithms

Author : Tengyu Xu
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (137 download)

DOWNLOAD NOW!

Book Synopsis Towards the Understanding of Sample Efficient Reinforcement Learning Algorithms by : Tengyu Xu

Download or read book Towards the Understanding of Sample Efficient Reinforcement Learning Algorithms written by Tengyu Xu and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Reinforcement learning (RL), which aims at designing a suitable policy for an agent via interacting with an unknown environment, has achieved remarkable success in the recent past. Despite its great potential to solve complex tasks, current RL algorithms suffer from requiring a large amount of interaction data, which could result in significant cost in real world applications. Thus, the goal of this thesis is to study the sample complexity of fundamental RL algorithms, and then, to propose new RL algorithms to solve real-world problems with provable efficiency. To achieve this goal, this thesis makes the contributions along the following three main directions: 1. For policy evaluation, we proposed a new on-policy algorithm called variance reduce TD (VRTD) and established the state-of-the-art sample complexity result for off-policy two-timescale TD learning algorithms. 2. For policy optimization, we established improved sample complexity bounds for on-policy actor-critic (AC) type algorithms and proposed the first doubly robust off-policy AC algorithm with provable efficiency guarantee. 3. We proposed three new algorithms: GenTD, CRPO and PARTED to address challenging practical problems of general value function evaluation, safe RL and trajectory-wise reward RL, respectively, with provable efficiency.

Efficient Reinforcement Learning with Value Function Generalization

Author : Zheng Wen
Publisher :
ISBN 13 :
Total Pages : pages
Book Rating : 4.:/5 (878 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning with Value Function Generalization by : Zheng Wen

Download or read book Efficient Reinforcement Learning with Value Function Generalization written by Zheng Wen and published by . This book was released on 2014 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Reinforcement learning (RL) is concerned with how an agent should learn to make decisions over time while interacting with an environment. A growing body of work has produced RL algorithms with sample and computational efficiency guarantees. However, most of this work focuses on "tabula rasa" learning; i.e. algorithms aim to learn with little or no prior knowledge about the environment. Such algorithms exhibit sample complexities that grow at least linearly in the number of states, and they are of limited practical import since state spaces in most relevant contexts are enormous. There is a need for algorithms that generalize in order to learn how to make effective decisions at states beyond the scope of past experience. This dissertation focuses on the open issue of developing efficient RL algorithms that leverage value function generalization (VFG). It consists of two parts. In the first part, we present sample complexity results for two classes of RL problems -- deterministic systems with general forms of VFG and Markov decision processes (MDPs) with a finite hypothesis class. The results provide upper bounds that are independent of state and action space cardinalities and polynomial in other problem parameters. In the second part, building on insights from our sample complexity analyses, we propose randomized least-square value iteration (RLSVI), a RL algorithm for MDPs with VFG via linear hypothesis classes. The algorithm is based on a new notion of randomized value function exploration. We compare through computational studies the performance of RLSVI against least-square value iterations (LSVI) with Boltzmann exploration or epsilon-greedy exploration, which are widely used in RL with VFG. Results demonstrate that RLSVI is orders of magnitude more efficient.

Explorations in Efficient Reinforcement Learning

Author : Marco Wiering
Publisher :
ISBN 13 :
Total Pages : 198 pages
Book Rating : 4.:/5 (247 download)

DOWNLOAD NOW!

Book Synopsis Explorations in Efficient Reinforcement Learning by : Marco Wiering

Download or read book Explorations in Efficient Reinforcement Learning written by Marco Wiering and published by . This book was released on 1999 with total page 198 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Bayesian Reinforcement Learning

Author : Mohammad Ghavamzadeh
Publisher :
ISBN 13 : 9781680830880
Total Pages : 146 pages
Book Rating : 4.8/5 (38 download)

DOWNLOAD NOW!

Book Synopsis Bayesian Reinforcement Learning by : Mohammad Ghavamzadeh

Download or read book Bayesian Reinforcement Learning written by Mohammad Ghavamzadeh and published by . This book was released on 2015-11-18 with total page 146 pages. Available in PDF, EPUB and Kindle. Book excerpt: Bayesian methods for machine learning have been widely investigated, yielding principled methods for incorporating prior information into inference algorithms. This monograph provides the reader with an in-depth review of the role of Bayesian methods for the reinforcement learning (RL) paradigm. The major incentives for incorporating Bayesian reasoning in RL are that it provides an elegant approach to action-selection (exploration/exploitation) as a function of the uncertainty in learning, and it provides a machinery to incorporate prior knowledge into the algorithms. Bayesian Reinforcement Learning: A Survey first discusses models and methods for Bayesian inference in the simple single-step Bandit model. It then reviews the extensive recent literature on Bayesian methods for model-based RL, where prior information can be expressed on the parameters of the Markov model. It also presents Bayesian methods for model-free RL, where priors are expressed over the value function or policy class. Bayesian Reinforcement Learning: A Survey is a comprehensive reference for students and researchers with an interest in Bayesian RL algorithms and their theoretical and empirical properties.

TensorFlow Reinforcement Learning Quick Start Guide

Author : Kaushik Balakrishnan
Publisher : Packt Publishing Ltd
ISBN 13 : 1789533449
Total Pages : 175 pages
Book Rating : 4.7/5 (895 download)

DOWNLOAD NOW!

Book Synopsis TensorFlow Reinforcement Learning Quick Start Guide by : Kaushik Balakrishnan

Download or read book TensorFlow Reinforcement Learning Quick Start Guide written by Kaushik Balakrishnan and published by Packt Publishing Ltd. This book was released on 2019-03-30 with total page 175 pages. Available in PDF, EPUB and Kindle. Book excerpt: Leverage the power of Tensorflow to Create powerful software agents that can self-learn to perform real-world tasks Key FeaturesExplore efficient Reinforcement Learning algorithms and code them using TensorFlow and PythonTrain Reinforcement Learning agents for problems, ranging from computer games to autonomous driving.Formulate and devise selective algorithms and techniques in your applications in no time.Book Description Advances in reinforcement learning algorithms have made it possible to use them for optimal control in several different industrial applications. With this book, you will apply Reinforcement Learning to a range of problems, from computer games to autonomous driving. The book starts by introducing you to essential Reinforcement Learning concepts such as agents, environments, rewards, and advantage functions. You will also master the distinctions between on-policy and off-policy algorithms, as well as model-free and model-based algorithms. You will also learn about several Reinforcement Learning algorithms, such as SARSA, Deep Q-Networks (DQN), Deep Deterministic Policy Gradients (DDPG), Asynchronous Advantage Actor-Critic (A3C), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO). The book will also show you how to code these algorithms in TensorFlow and Python and apply them to solve computer games from OpenAI Gym. Finally, you will also learn how to train a car to drive autonomously in the Torcs racing car simulator. By the end of the book, you will be able to design, build, train, and evaluate feed-forward neural networks and convolutional neural networks. You will also have mastered coding state-of-the-art algorithms and also training agents for various control problems. What you will learnUnderstand the theory and concepts behind modern Reinforcement Learning algorithmsCode state-of-the-art Reinforcement Learning algorithms with discrete or continuous actionsDevelop Reinforcement Learning algorithms and apply them to training agents to play computer gamesExplore DQN, DDQN, and Dueling architectures to play Atari's Breakout using TensorFlowUse A3C to play CartPole and LunarLanderTrain an agent to drive a car autonomously in a simulatorWho this book is for Data scientists and AI developers who wish to quickly get started with training effective reinforcement learning models in TensorFlow will find this book very useful. Prior knowledge of machine learning and deep learning concepts (as well as exposure to Python programming) will be useful.

Efficient Reinforcement Learning in Continuous Environments

Author : Mohammad A. Al-Ansari
Publisher :
ISBN 13 :
Total Pages : 160 pages
Book Rating : 4.:/5 (59 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning in Continuous Environments by : Mohammad A. Al-Ansari

Download or read book Efficient Reinforcement Learning in Continuous Environments written by Mohammad A. Al-Ansari and published by . This book was released on 2001 with total page 160 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Decision Making Under Uncertainty and Reinforcement Learning

Author : Christos Dimitrakakis
Publisher : Springer Nature
ISBN 13 : 3031076141
Total Pages : 251 pages
Book Rating : 4.0/5 (31 download)

DOWNLOAD NOW!

Book Synopsis Decision Making Under Uncertainty and Reinforcement Learning by : Christos Dimitrakakis

Download or read book Decision Making Under Uncertainty and Reinforcement Learning written by Christos Dimitrakakis and published by Springer Nature. This book was released on 2022-12-02 with total page 251 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents recent research in decision making under uncertainty, in particular reinforcement learning and learning with expert advice. The core elements of decision theory, Markov decision processes and reinforcement learning have not been previously collected in a concise volume. Our aim with this book was to provide a solid theoretical foundation with elementary proofs of the most important theorems in the field, all collected in one place, and not typically found in introductory textbooks. This book is addressed to graduate students that are interested in statistical decision making under uncertainty and the foundations of reinforcement learning.

Data Efficient Reinforcement Learning with Off-policy and Simulated Data

Author : Josiah Paul Hanna
Publisher :
ISBN 13 :
Total Pages : 486 pages
Book Rating : 4.:/5 (115 download)

DOWNLOAD NOW!

Book Synopsis Data Efficient Reinforcement Learning with Off-policy and Simulated Data by : Josiah Paul Hanna

Download or read book Data Efficient Reinforcement Learning with Off-policy and Simulated Data written by Josiah Paul Hanna and published by . This book was released on 2019 with total page 486 pages. Available in PDF, EPUB and Kindle. Book excerpt: Learning from interaction with the environment -- trying untested actions, observing successes and failures, and tying effects back to causes -- is one of the first capabilities we think of when considering autonomous agents. Reinforcement learning (RL) is the area of artificial intelligence research that has the goal of allowing autonomous agents to learn in this way. Despite much recent success, many modern reinforcement learning algorithms are still limited by the requirement of large amounts of experience before useful skills are learned. Two possible approaches to improving data efficiency are to allow algorithms to make better use of past experience collected with past behaviors (known as off-policy data) and to allow algorithms to make better use of simulated data sources. This dissertation investigates the use of such auxiliary data by answering the question, "How can a reinforcement learning agent leverage off-policy and simulated data to evaluate and improve upon the expected performance of a policy?" This dissertation first considers how to directly use off-policy data in reinforcement learning through importance sampling. When used in reinforcement learning, importance sampling is limited by high variance that leads to inaccurate estimates. This dissertation addresses this limitation in two ways. First, this dissertation introduces the behavior policy gradient algorithm that adapts the data collection policy towards a policy that generates data that leads to low variance importance sampling evaluation of a fixed policy. Second, this dissertation introduces the family of regression importance sampling estimators which improve the weighting of already collected off-policy data so as to lower the variance of importance sampling evaluation of a fixed policy. In addition to evaluation of a fixed policy, we apply the behavior policy gradient algorithm and regression importance sampling to batch policy gradient policy improvement. In the case of regression importance sampling, this application leads to the introduction of the sampling error corrected policy gradient estimator that improves the data efficiency of batch policy gradient algorithms. Towards the goal of learning from simulated experience, this dissertation introduces an algorithm -- the grounded action transformation algorithm -- that takes small amounts of real world data and modifies the simulator such that skills learned in simulation are more likely to carry over to the real world. Key to this approach is the idea of local simulator modification -- the simulator is automatically altered to better model the real world for actions the data collection policy would take in states the data collection policy would visit. Local modification necessitates an iterative approach: the simulator is modified, the policy improved, and then more data is collected for further modification. Finally, in addition to examining them each independently, this dissertation also considers the possibility of combining the use of simulated data with importance sampled off-policy data. We combine these sources of auxiliary data by control variate techniques that use simulated data to lower the variance of off-policy policy value estimation. Combining these sources of auxiliary data allows us to introduce two algorithms -- weighted doubly robust bootstrap and model-based bootstrap -- for the problem of lower-bounding the performance of an untested policy

Models for Data-efficient Reinforcement Learning on Real-world Applications

Author : Andreas Dörr
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (129 download)

DOWNLOAD NOW!

Book Synopsis Models for Data-efficient Reinforcement Learning on Real-world Applications by : Andreas Dörr

Download or read book Models for Data-efficient Reinforcement Learning on Real-world Applications written by Andreas Dörr and published by . This book was released on 2021 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Efficient Reinforcement Learning with Agent States

Author : Shi Dong (Researcher of reinforcement learning)
Publisher :
ISBN 13 :
Total Pages : 0 pages
Book Rating : 4.:/5 (135 download)

DOWNLOAD NOW!

Book Synopsis Efficient Reinforcement Learning with Agent States by : Shi Dong (Researcher of reinforcement learning)

Download or read book Efficient Reinforcement Learning with Agent States written by Shi Dong (Researcher of reinforcement learning) and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: In a wide range of decision problems, much focus of academic research has been put on stylized models, whose capacities are usually limited by problem-specific assumptions. In the previous decade, approaches based on reinforcement learning (RL) have received growing attention. With these approaches, a unified method can be applied to a broad class of problems, circumventing the need for stylized solutions. Moreover, when it comes to real-life applications, such RL-based approaches, unfettered from the constraining models, can potentially leverage the growing amount of data and computational resources. As such, continuing innovations might empower RL to tackle problems in the complex physical world. So far, empirical accomplishments of RL have largely been limited to artificial environments, such as games. One reason is that the success of RL often hinges on the availability of a simulator that is able to mass-produce samples. Meanwhile, real environments, such as medical facilities, fulfillment centers, and the World Wide Web, exhibit complex dynamics that can hardly be captured by hard-coded simulators. To bring the achievement of RL into practice, it would be useful to think in terms of how the interactions between the agent and the real world ought to be modeled. Recent works on RL theory tend to focus on restrictive classes of environments that fail to capture certain aspects of the real world. For example, many of such works model the environment as a Markov Decision Process (MDP), which requires that the agent always observe a summary statistic of its situation. In practice, this means that the agent designer has to identify a set of "environmental states, " where each state incorporates all information about the environment relevant to decision-making. Moreover, to ensure that the agent learns from its trajectories, MDP models presume that some environmental states are visited infinitely often. This could be a significant simplification of the real world, as the gifted Argentine poet Jorge Luis Borges once said, "Every day, perhaps every hour, is different." To generate insights on agent design in authentic applications, in this dissertation we consider a more general framework of RL that relaxes such restrictions. Specifically, we demonstrate a simple RL agent that implements an optimistic version of Q-learning and establish through regret analysis that this agent can operate with some level of competence in any environment. While we leverage concepts from the literature on provably efficient RL, we consider a general agent-environment interface and provide a novel agent design and analysis that further develop the concept of agent state, which is defined as the collection of information that the agent maintains in order to make decisions. This level of generality positions our results to inform the design of future agents for operation in complex real environments. We establish that, as time progresses, our agent performs competitively relative to policies that require longer times to evaluate. The time it takes to approach asymptotic performance is polynomial in the complexity of the agent's state representation and the time required to evaluate the best policy that the agent can represent. Notably, there is no dependence on the complexity of the environment. The ultimate per-period performance loss of the agent is bounded by a constant multiple of a measure of distortion introduced by the agent's state representation. Our work is the first to establish that an algorithm approaches this asymptotic condition within a tractable time frame, and the results presented in this dissertation resolve multiple open issues in approximate dynamic programming.