Markov Decision Processes in Artificial Intelligence

Table of Contents

Preface

List of Authors

PART 1. MDPS: MODELS AND METHODS

Chapter 1. Markov Decision Processes

1.1. Introduction

1.2. Markov decision problems

1.3. Value functions

1.4. Markov policies

1.5. Characterization of optimal policies

1.6. Optimization algorithms for MDPs

1.7. Conclusion and outlook

1.8. Bibliography

Chapter 2. Reinforcement Learning

2.1. Introduction

2.2. Reinforcement learning: a global view

2.3. Monte Carlo methods

2.4. From Monte Carlo to temporal difference methods

2.5. Temporal difference methods

2.6. Model-based methods: learning a model

2.7. Conclusion

2.8. Bibliography

Chapter 3. Approximate Dynamic Programming

3.1. Introduction

3.2. Approximate value iteration (AVI)

3.3. Approximate policy iteration (API)

3.4. Direct minimization of the Bellman residual

3.5. Towards an analysis of dynamic programming in Lp-norm

3.6. Conclusions

3.7. Bibliography

Chapter 4. Factored Markov Decision Processes

4.1. Introduction

4.2. Modeling a problem with an FMDP

4.3. Planning with FMDPs

4.4. Perspectives and conclusion

4.5. Bibliography

Chapter 5. Policy-Gradient Algorithms

5.1. Reminder about the notion of gradient

5.2. Optimizing a parameterized policy with a gradient algorithm

5.3. Actor-critic methods

5.4. Complements

5.5. Conclusion

5.6. Bibliography

Chapter 6. Online Resolution Techniques

6.1. Introduction

6.2. Online algorithms for solving an MDP

6.3. Controlling the search

6.4. Conclusion

6.5. Bibliography

PART 2. BEYOND MDPs

Chapter 7. Partially Observable ...