学会学习更多：元强化学习-FlyAI

学会学习更多：元强化学习

深度学习机器学习

最后更新 2020-11-26 14:57 阅读 5070

最后更新 2020-11-26 14:57

阅读 5070

深度学习机器学习

##FlyAI文献翻译
**英文原文：**[：Learning to Learn More: Meta Reinforcement Learning](https://towardsdatascience.com/learning-to-learn-more-meta-reinforcement-learning-f0cc92c178c1)

> **标签** ：强化学习

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602835696810710.png)

Photo by Photos Hobby on Unsplash

>图片由Unsplash的Photos Hobby提供

The [ELI5](https://www.dictionary.com/e/slang/eli5/ "ELI5") definition for Reinforcement Learning would be training a model to perform better by iteratively learning from its previous mistakes. Reinforcement learning provides a framework for agents to solve problems in case of real-world scenarios. They are able to learn rules (or policies) to solve specific problems, but one of the major limitations of these agents are that they are unable to generalize the learned policy to newer problems. A previously learned rule would cater to a specific problem only, and would often be useless for other (even similar) cases.

A good meta-learning model on the other hand, is expected to generalize to new tasks or environments that have not been encountered by the model in training. The process of adaption to this new environment can be termed a mini learning session and happens with testing with limited exposure to newer configuration. In the absence of explicitly fine-tuning models, it is observed that meta-learning is able to autonomously adjust internal states to generalize to newer environments.

>ELI5对强化学习的定义是, 通过从以前的错误中反复学习来训练一个模型来使其表现更好。强化学习为智能代理(agent)提供了一个框架，以便在解决现实场景下的问题。它们能够学习规则（或策略）来解决特定的问题，但这些代理的一个主要限制是它们无法将学习到的策略推广到更新的问题上。之前习得的规则只能解决某个特定问题，对于其他（甚至类似的）的场景往往无用.

>而一个好的元学习模型则有望泛化到模型在训练中从未遇见过的新任务或环境。适应这种新环境的过程可以被称为迷你学习环节(mini learning session)，并且在测试中有限的暴露给更新的环境配置。在没有刻意精调过模型的时候，可以观察到元学习能够自主调整内部状态以泛化到更新的环境中.

Meta-Reinforcement Learning is just Meta-Learning applied to Reinforcement Learning

>元强化学习只是用于强化学习的元学习

Furthermore, [Wang et al](https://arxiv.org/abs/1611.05763 "Wang et al"). described meta-RL as “the special category of meta-learning that use recurrent models, applied to RL”, which seems like a much more comprehensive definition than the one above.

>此外, Wang等人将元RL描述为 "使用循环模型的一类用于RL的特殊元学习"，这个定义似乎比以上定义更全面.

##Let’s get started

In reinforcement learning, an agent receives observations at each step (such as the position of a character in a video game) and based on those observations, it outputs actions like ‘move forward’ or ‘turn right’. Based on the results of these actions, the agent receives rewards or penalties which guide it further in training, helping it make more meaningful observations for later steps. The aim of the model is to maximize rewards and minimize penalties.

In meta-reinforcement Learning, the training and testing tasks are different, but are drawn from the same family of problems. A good example would be mazes with different layouts, or different probabilities of a multi-armed bandit problem (explained below).

>###让我们开始吧

>在强化学习中，代理在每一步都会接收到观察结果（如视频游戏中角色的位置），并根据这些观察结果，输出 "向前移动 "或 "向右转 "等动作。根据这些动作的结果，代理会得到奖励或惩罚，这些奖励或惩罚会指导它进一步的训练，帮助它为以后的步骤做出更有意义的观察。该模型的目标是最大化奖励且最小化惩罚。

>在元强化学习中，训练和测试任务虽不同但却来自同一族问题。一个好例子是布局不同的迷宫，或者不同概率的多臂赌博机（后文解释）。

A simple experiment

The[ multi-armed bandit problem ](https://lilianweng.github.io/lil-log/2018/01/23/the-multi-armed-bandit-problem-and-its-solutions.html " multi-armed bandit problem ")is a classic problem which demonstrates the exploration vs exploitation dilemma in an excellent way. You can put yourself in the situation by imagining yourself in a room with two levers and nothing else.

>###一个简单的实验

>多臂赌博机问题是一个经典的问题，它很好地展示了探索-利用的困境。你可以想象自己在一个什么都没有的房间里，只有两个拉杆。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602835689508204.png)

The multi-armed bandit problem; Image by Author

>多臂赌博机问题；图片由作者提供。

1. Pulling the left lever A1 would give you a probability p to receive reward r.

3. Pulling the right lever A2 would give you a probability of (1-p) to receive reward r.

>1. 拉动左边的拉杆A1，你获得奖励r的概率为p。

>2. 拉动右边的杠杆A2，得到奖励r的概率为(1-p)。

To answer this question logically, you would have to know the value of the probability p. A higher value of p would guarantee a higher chance of reward from the lever A1, while a lower value of p would guarantee a higher chance of reward from the lever A2. And this is exactly why meta-RL is so interesting. If you throw enough values at your model, and it will get increasingly better at choosing the correct lever after interacting with the environment and learning from it. A traditional RL-based approach would not be able to handle the changing probabilities and would usually fail with different values of p.

[Practically](https://hackernoon.com/learning-policies-for-learning-policies-meta-reinforcement-learning-rl%C2%B2-in-tensorflow-b15b592a2ddf "Practically"), the meta-RL agent trained on the two-lever problem with different probabilities is able to choose the correct lever out of the two which leads to the highest reward using a very small number of data points. It uses the couplet (action, reward) for computing the ‘risk vs reward’ factor for each of the levers.

Here is an example of an untrained agent on the left (with p=0.92) and a trained meta-RL agent on the right (with p=0.76)

>要解这个问题我们必须知道概率p的值. p的值越大，从拉杆A1获得奖励的机会越大，而p的值越小，那么从拉杆A2获得奖励的可能就越大。而这正是元RL如此有趣的原因。如果你抛给模型足够多的值，而它在与环境互动并从中学习后，会越来越善于选择正确的杠杆。传统的RL方法将无法处理不断变化的概率，通常会在p值不同的时候发散。

>实际上，在不同概率的双拉杆问题上训练的元RL代理能够使以极少的数据样本从两个拉杆中选择出正确的拉杆，从而获得最高的奖励。它使用（动作，奖励）对来计算每个拉杆的 "风险vs奖励 "系数。

>下面是一个例子，左边是一个未经训练的代理（p=0.92），右边是一个经过训练的元RL代理（p=0.76）。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602835689428110.png)

>一个未经训练和训练好的元RL代理; 图源见[链接](https://hackernoon.com/learning-policies-for-learning-policies-meta-reinforcement-learning-rl%C2%B2-in-tensorflow-b15b592a2ddf "链接")

###Key Components

There are three key components involved in meta-RL. They are described in detail below.

A model with memory: Without memory, a meta-RL model would be useless. It needs memory to acquire and store knowledge about the current task from the immediate environment, which would help it to update its hidden state. A recurrent neural network maintains the hidden state of the meta-RL model.

The dynamics to creating a good RNN would be too broad for the scope of this article. However, meta-RL and meta-RL², both used an LSTM to manage their hidden states.

The meta-learning algorithm: A meta-learning algorithm would define how we update the weights of the model based on what it learnt. The main objective of the algorithm is to help optimize the model to solve an unseen task in the minimum amount of time, applying what it learnt from previous tasks. Previous research usually used ordinary gradient descent update of the LSTM cell.

MAML and Reptile, are proven methods that are able to update model parameters in order to achieve a good generalization performance on new and unseen tasks.

>###关键组件

>元强化学习包含如下所述三个关键组件：

>内存模块：内存是元强化学习的核心，没有内存，元RL模型将一无是处。有了内存，模型能够即时从环境中获取并存储当前任务相关知识，以此帮助模型隐藏状态的更新。使用RNN来维护元RL模型的隐藏状态。

>创建良好的RNN对于模型至关重要。meta-RL和meta-RL²都使用LSTM来管理其隐藏状态。

>元学习算法：元学习算法通过所学知识来更新模型权重。该算法的主要目标是应用从先前任务中学到的知识优化模型，以在最短时间内使模型能够处理全新的任务。先前的研究一般是通过LSTM单元的梯度下降来更新权重。

>[MAML](https://hackernoon.com/learning-policies-for-learning-policies-meta-reinforcement-learning-rl%C2%B2-in-tensorflow-b15b592a2ddf "MAML")和[Reptile](https://arxiv.org/abs/1803.02999 "Reptile")是典型的元学习算法，通过该算法更新模型参数，使得模型在未知的新任务中具有良好的泛化性能。

A proper distribution of MDPs: A Markov Decision Process (MDP) refers to the entire process of the agent observing the environment output, consisting of a reward and a next state, and then making further decisions based on that. Since the agent is exposed to a number of different types of environments and tasks during its training, it needs to be able to quickly adapt to changing conditions and different MDPs.

Among the three components, this is the least studied and probably the most specific component in regards to meta-RL. As each task is a MDP, we can build a distribution of MDPs by either modifying the reward configuration or the environment.

[Evolutionary algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm "Evolutionary algorithms") are a great way to ensure the generation of a good environment. They are usually heuristic-based and inspired by the process of natural selection. A population of randomly generated solutions go through a loop of evaluation, selection, mutation (if we throw genetic algorithms in the mix) and reproduction; out of which the good solutions make it till the end. POET, by Wang et al. is a good example of a framework based on an evolutionary algorithm.

The Pair Open-Ended Trailblazer [(POET)](https://eng.uber.com/poet-open-ended-deep-learning/ "(POET)"), demonstrated below, initially begins with a trivial environment and a randomly initialized agent. It then grows and maintains a population of one-to-one paired environments and agents. According to the authors, POET aims to achieve two goals, to evolve the population of environments with regards to diversity and complexity; and optimizing the agents to solve their parallel environments.

>MDP的正确分布：马尔可夫决策过程（MDP）是指智能体观察环境输出（包括奖励和新的环境状态），以此做出进一步决策的整个过程。由于在训练过程中，智能体面临多种不同类型的环境和任务，因此需要能够快速适应不断变化的条件和不同MDP。

>在上述介绍的三个组件中，虽然关于MDP分布的研究最少，但该组件却也是最具体的。由于将每个任务都视为MDP，因此我们可以通过修改奖励或环境来构建MDP分布。

>受自然选择过程的启发，启发式的进化算法可以确保生成良好的环境。具体而言，通过评估、选择、突变（引入遗传算法）和繁殖的循环，随机生成大量方案；一直选择好的解决方案即可。Wang等人提出的POET就是基于进化算法的成功示例。

>如下所示，Pair Open-Ended Trailblazer（POET）首先随机初始化一个环境和代理，然后逐渐扩展形成若干一对一的环境和智能体的集合。据作者所说，POET旨在实现两个目标，一是确保环境的多样性和复杂性，二是在并行环境中优化相应智能体。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602835690913654.png)

POET: A framework based on an evolutionary algorithm; source

>基于进化算法的POET框架；[来源](https://eng.uber.com/poet-open-ended-deep-learning/ "来源")

An MDP without a reward function is known as a Controlled Markov Process (CMP) Given a predefined CMP, we can learn knowledge about a variety of tasks by generating a collection of reward functions R, that encourage the training of an effective meta-learning policy.

[Gupta et al](https://arxiv.org/abs/1806.04640 "Gupta et al"). proposed two unsupervised approaches for growing the task distribution in the context of CMP. Assuming that there is an underlying latent variable associated with every task, it parameterizes the reward function as a function of the latent variable along with a discriminator function (which is used to extract the latent variable from the state). — [Source](https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html "Source")

>没有奖励函数的MDP称为受控马尔可夫过程（CMP）。给定预定义的CMP，我们可以通过生成奖励函数R的集合来学习不同任务的知识，从而训练有效的元学习策略。

>Gupta等人提出了两种在CMP环境下增强任务分布的无监督方法。假设每个任务都有一个潜在变量，那么可以将奖励函数参数化为一个潜在变量以及带有判别功能（用于从状态中提取潜在变量）的潜在变量函数。

The research paper described two main ways for constructing the discriminator function:

>论文提出两种判别函数的构造方法：

1. Sampling random weights of the discriminator

3. Learning a discriminator function to encourage a diversity-driven exploration. If you are looking for a more comprehensive analysis on the subject, you can refer to their other paper, [‘DIAYN’](https://arxiv.org/abs/1802.06070 "‘DIAYN’") (Diversity is all you need).

>1. 随机采样判别器权重

>3. 通过多样性探索来学习判别函数。阅读论文‘DIAYN’了解更多信息（Diversity is all you need）

The intricacies of growing the task distribution in the context of CMP is beyond the scope of the article, and I would highly recommend anyone interested to delve into [this](https://arxiv.org/abs/1806.04640 "this") paper for a more in-depth view.

>在CMP环境下增加任务分布复杂性是一个广阔的话题，本文不再做过多介绍，建议感兴趣的读者阅读相关文章做更深入的了解。

###Comparison with Reinforcement Learning

A meta-RL system is pretty much similar to that of an ordinary RL algorithm, except for the fact that the last reward as well as the last action are also included into the policy observation, along with the current state. The purpose of this change is to feed and keep track of the history of all tasks and observations, so that the model can internally update the dynamics between the states, actions and rewards based on the current MDP, and adjust its strategy for other MDPs accordingly.

Both meta-RL and meta-RL² implemented an LSTM policy, where the LSTM’s hidden state served as the memory to track the change in characteristics. The policy being recurrent in nature, the need to explicitly input the final values is not seen.

>###与强化学习的比较

>meta-RL系统与普通RL算法非常相似，不同之处在于meta-RL在策略观察过程中不仅获取当前状态，还关注最终奖励和最后的动作。通过这样的改变，系统得到所有任务和观察的历史记录，这样模型不仅可以根据当前MDP动态更新状态、动作和奖励，还可以根据其他MDP调整其策略。

>LSTM的隐藏状态可用于表征特性变化，meta-RL和meta-RL²都采用了LSTM策略。由于该策略本质上是周期性的，因此最终值的输入是不可知的。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602835689107333.png)

The different actor-critic architectures used in the meta-RL paper (all of them are recurrent models); [source](https://arxiv.org/abs/1611.05763 "source")

The training procedure works as follows:

>meta-RL使用的不同actor-critic架构（都是循环模型）；图源

>训练流程如下：

1. Sample a new MDP

3. Reset the hidden state of the model

5. Collect multiple trajectories and update the weights of the model

7. Repeat from step 1

>1. 采样一个新的MDP

>2. 重置模型的隐藏状态

>3. 收集多个轨迹并更新模型权重

>4. 重复1-3

###Beyond

Training RL algorithms can be difficult sometimes. If a meta-learning agent could become so smart that the distribution of tasks that it could solve from the knowledge it inferred while being trained on a particular task, becomes extremely broad, we would be well on our way towards generalized intelligence (or as the new buzzword stands — Artificial General Intelligence {AGI}) — essentially building a brain which would be able to solve all types of RL problems.

As a side-note, I would also like to point out the unsurprising resemblance between meta-RL and [AI-GAs](https://arxiv.org/abs/1905.10985AI-GAs) (by J Clune), which proposed that the efficient way towards AGI is by making learning autonomous. It stood on three pillars: meta-learning architectures, meta-learning algorithms and automatically generated algorithms for effective learning.

>###总结

>RL的训练有时会非常麻烦。如果元学习智能体能够根据任务分布从掌握的知识推理到其他特定任务，那么元学习智能体会具有广泛的适用性。通过这样方式，可以构造出一个解决所有RL问题的“机器大脑”，通用人工智能（也就是人们常说的AGI）将变得可能。

>另外值得注意的时，meta-RL和AI-GA（J Clune）存在相似之处，其中AI-GA的关键是通过自主性学习实现AGI，其也包括三个关键组分：元学习结构，元学习算法和自主学习生成算法。

###References and Further Reading

This work would not have been possible without the efforts of [Arthur Juliani](https://medium.com/awjuliani "Arthur Juliani"). You can check out his excellent implementation of the meta-RL algorithms[ here](https://github.com/awjuliani/Meta-RL " here"). This [research paper ](https://arxiv.org/abs/1611.05763 "research paper ")by Wang et al. also gave me a great insight into some of the core concepts of meta-RL, and I would highly recommend reading it if you want a more comprehensive view. Lastly, this [post](https://eng.uber.com/poet-open-ended-deep-learning/ "post") on the Uber Engineering blog by Clune, Stanley, Lehman and Wang also helped me understand the open-endedness of fields like these, and a way to overcome prohibitively difficult challenges. This[ blog](https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html " blog") by Lilian Weng is also a great resource for further reading.

>###参考

>Arthur Juliani的工作为本文提供了巨大支撑，读者可以点此查阅他提供的meta-RL算法实现。另外，Wang等人的论文使我对meta-RL的一些核心概念有了更深刻的了解，如果你也也想对此有个全面了解，强烈建议阅读该论文。最后，Clune，Stanley，Lehman和Wang在Uber工程博客上发表的这篇文章还帮助我了解了更管关于该领域的开放性、重大挑战的解决发发等内容，丰富了我的知识。Lilian Weng的博客也值得阅读。

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/768

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除