Recent Advances in Reinforcement Learning

Marcus Hutter received his masters in computer sciences in 1992 at the Technical University in Munich, Germany. After his PhD in theoretical particle physics he developed algorithms in a medical software company for 5 years. For four years he has been working as a researcher at the AI institute IDSIA in Lugano, Switzerland. His current interests are centered around reinforcement learning, algorithmic information theory and statistics, universal induction schemes, adaptive control theory, and related areas.

This book constitutes revised and selected papers of the 9th European Workshop on Reinforcement Learning, EWRL 2011, which took place in Athens, Greece in September 2011. The papers presented were carefully reviewed and selected from 40 submissions. The papers are organized in topical sections online reinforcement learning, learning and exploring MDPs, function approximation methods for reinforcement learning, macro-actions in reinforcement learning, policy search and bounds, multi-task and transfer reinforcement learning, multi-agent reinforcement learning, apprenticeship and inverse reinforcement learning and real-world reinforcement learning.

Fast-track conference proceedings

Invited Talk Abstracts.-Invited Talk: UCRL and Autonomous Exploration.-Invited Talk: Increasing Representational Power and Scaling Inference in Reinforcement Learning.-Invited Talk: PRISM - Practical RL: Representation, Interaction, Synthesis, and Mortality.-Invited Talk: Towards Robust Reinforcement Learning Algorithms.-Online Reinforcement Learning Automatic Discovery of Ranking Formulas for Playing with Multi-armed Bandits.-Goal-Directed Online Learning of Predictive Models.-Gradient Based Algorithms with Loss Functions and Kernels for Improved On-Policy Control.-Learning and Exploring MDPs -Active Learning of MDP Models.-Handling Ambiguous Effects in Action Learning.-Feature Reinforcement Learning in Practice.-Function Approximation Methods for Reinforcement Learning Reinforcement Learning with a Bilinear Q Function.-1-Penalized Projected Bellman Residual.-Regularized Least Squares Temporal Difference Learning with Nested 2 and 1 Penalization.-Recursive Least-Squares Learning with Eligibility Traces.-Value Function Approximation through Sparse Bayesian Modeling.-Automatic Construction of Temporally Extended Actions for MDPs Using Bisimulation Metrics.-Unified Inter and Intra Options Learning Using Policy Gradient Methods.-Options with Exceptions.-Policy Search and Bounds.-Robust Bayesian Reinforcement Learning through Tight Lower Bounds.-Optimized Look-ahead Tree Search Policies.-A Framework for Computing Bounds for the Return of a Policy.-Multi-Task and Transfer Reinforcement Learning.-Transferring Evolved Reservoir Features in Reinforcement Learning Task.-Transfer Learning via Multiple Inter-task Mappings.-Multi-Task Reinforcement Learning: Shaping and Feature Selection.-Multi-Agent Reinforcement Learning.-Transfer Learning in Multi-Agent Reinforcement Learning Domains.-An Extension of a Hierarchical Reinforcement Learning Algorithm for Multiagent Settings.-Apprenticeship and Inverse Reinforcement Learning Bayesian Multitask Inverse ReinforcementLearning.-Batch, Off-Policy and Model-Free Apprenticeship Learning.-Real-World Reinforcement Learning Introduction of Fixed Mode States into Online Profit Sharing and Its Application to Waist Trajectory Generation of Biped Robot.-MapReduce for Parallel Reinforcement Learning.-Compound Reinforcement Learning: Theory and an Application to Finance.-Proposal and Evaluation of the Active Course Classification Support System with Exploitation-Oriented Learning.