By Ashvin Nair and Abhishek Gupta. Originally published by the Berkeley Artificial Intelligence Research Blog on September 10, 2020.
Restoration note, September 3, 2026: This article preserves the experiments and comparisons reported in the 2020 BAIR post and the contemporaneous arXiv v3 preprint. It is a historical technical account, not a current state-of-the-art assessment. The paper was later revised on April 24, 2021, adding real-world robotics experiments; those later results are not silently folded into the 2020 body. The original figures and animations remain available in the canonical post and on the project website.
Our method learns complex behaviors by training offline from prior datasets—expert demonstrations, data from previous experiments, or random exploration data—and then fine-tuning with online interaction.
Robots trained with reinforcement learning (RL) have the potential to address a wide variety of challenging real-world problems. To apply RL to a new problem, one typically sets up the environment, defines a reward function, and trains the robot by allowing it to explore. Although this may eventually work, online RL methods can require substantial interaction, and repeating the process from scratch for every task makes real-world application difficult.
What if, instead, we could reuse data across problems and experiments? Prior demonstrations, unsuccessful trials, random exploration, and data collected by earlier policies may all contain useful information. Reusing them could reduce the amount of new interaction required for each problem.
The first step toward this data-driven paradigm is offline, or batch, RL: learning a policy from previously collected off-policy data without additional exploration during offline training. The data may come from other policies, other robots, or teleoperation. Depending on its quality and the task, however, offline training may need to be followed by targeted online improvement. This offline-to-online transition presents challenges of its own.
Challenges in offline RL with online fine-tuning
To analyze these challenges, we used the HalfCheetah-v2 locomotion benchmark. The prior dataset in the reported experiment contained 15 trajectories from an expert policy and 100 suboptimal trajectories sampled from a behavioral clone of those demonstrations.
1. Data efficiency
A simple way to use demonstrations is to pretrain a policy with imitation learning and then fine-tune it with an on-policy RL algorithm such as advantage-weighted regression (AWR) or demonstration-augmented policy gradient (DAPG). This has two drawbacks. First, the prior data may not be optimal, making direct imitation undesirable. Second, on-policy fine-tuning does not continue to reuse the prior data during the RL stage.
For real-world robotics, data efficiency is vital. Off-policy algorithms that perform dynamic programming can use a value function or model to combine useful transitions from different trajectories—often described intuitively as “stitching” partial trajectories together. In the reported HalfCheetah comparison, the on-policy and Monte Carlo return methods learned about an order of magnitude more slowly than the off-policy actor-critic methods under the tested conditions.
2. Bootstrapping error
Actor-critic methods can, in principle, learn efficiently from off-policy data by estimating a value function V(s) or action-value function Q(s, a) through Bellman bootstrapping. Yet when we applied a standard off-policy actor-critic method—soft actor-critic (SAC)—it performed poorly during offline pretraining. In the reported comparison, SAC with the prior dataset initially performed similarly to SAC trained from scratch. Even when its policy was pretrained with behavioral cloning, performance dipped at the start of online learning.
This behavior can be attributed to the accumulation of off-policy bootstrapping error. Q estimates are imperfect, particularly for actions not represented in the dataset. A policy update may exploit an overestimated value for an out-of-distribution action. That action then enters a Bellman target, propagating the error and potentially producing a poor policy update.
3. Non-stationary behavior models
Offline RL algorithms such as BCQ, BEAR, and BRAC address this problem by preventing the learned policy π from moving too far from the behavior distribution πβ represented by the replay buffer. Keeping the actor near observed actions reduces its opportunity to exploit erroneous Q values far outside the data distribution.
The behavior distribution is usually unknown and must be estimated. Methods including BEAR, BCQ, and ABM fit a parametric behavior model from replay-buffer samples. They then enforce the constraint through a penalty on the policy update or through an architecture that samples candidate actions from the learned model.
In the experiments reported in 2020, these constrained methods could perform well during offline training but improved slowly after online interaction began. BEAR, for example, had much stronger initial offline performance than ordinary SAC in the HalfCheetah analysis, yet its subsequent online improvement was limited.
The proposed explanation was the difficulty of tracking a changing behavior distribution. In a purely offline setting, a behavior model can be fitted to a fixed dataset. During online fine-tuning, it must continually adapt as new samples are mixed with data from previous policies. The resulting distribution may be complex and multimodal. In the reported diagnostic, the likelihood assigned by the learned behavior model deteriorated as online data entered the buffer, making the policy constraint overly conservative.
We therefore wanted an off-policy RL algorithm that constrained the actor enough to prevent offline error accumulation, but not so strongly that imperfect behavior modeling blocked online improvement. AWAC uses an implicit constraint and does not require a separately fitted behavior-policy model.
Advantage Weighted Actor Critic
Advantage Weighted Actor Critic (AWAC) combines an off-policy critic trained by temporal-difference learning with a supervised-style actor update. The critic provides data efficiency through Bellman bootstrapping. The actor is updated toward actions present in the replay buffer, with each action weighted according to its estimated advantage.
At iteration k, the policy-improvement problem can be written as follows:
Constrained policy improvement: πk+1 = arg max over π of Ea∼π(·|s)[Aπk(s,a)], subject to DKL(π(·|s) || πβ(·|s)) ≤ ε.
The nonparametric solution is proportional to the behavior distribution multiplied by an exponentiated advantage:
Nonparametric solution: π*(a|s) = (1/Z(s)) πβ(a|s) exp(Aπk(s,a)/λ).
Projecting this solution onto a parameterized policy with forward KL divergence produces the actor update used by AWAC:
Actor update: θk+1 = arg max over θ of Es,a∼β[log πθ(a|s) (1/Z(s)) exp(Aπk(s,a)/λ)].
This resembles weighted behavior cloning. State-action pairs are sampled directly from the replay buffer, avoiding an explicit density model for πβ. High-advantage actions receive large weights, while low-advantage actions receive small weights. If the critic provides no useful distinction between the sampled actions, the update approaches ordinary behavior cloning of the buffer; with an informative critic, it emphasizes the better actions.
Operationally, AWAC initializes its replay buffer with the offline dataset, repeatedly updates the critic using Bellman targets, and updates the actor with advantage-weighted regression. After the offline-training phase, it collects trajectories from the current policy, adds them to the same buffer, and continues both updates. The paper provides the full derivation, pseudocode, and implementation details.
Experiments
In the experiments presented with the 2020 post, we evaluated AWAC on high-dimensional, sparse-reward dexterous-manipulation tasks using demonstrations and additional off-policy data. We also evaluated it with suboptimal data generated by a random process. The contemporaneous paper includes further analysis on the HalfCheetah, Walker2d, and Ant MuJoCo benchmarks.
Dexterous manipulation
We wanted tasks representative of the difficulties encountered in robot learning, where offline pretraining and online fine-tuning are especially relevant. We used the dexterous-manipulation suite introduced by Rajeswaran and colleagues. These simulated tasks control a 28-degree-of-freedom, five-fingered hand: rotating a pen in hand, opening a door by unlatching its handle, and picking up a sphere and moving it to a target location.
The environments combine high-dimensional actions, intermittent contacts, complex simulated physics, randomized initial conditions, and sparse task-completion rewards. The supplied dataset contained 25 human demonstrations for each task. These demonstrations solved the tasks but were not assumed to be optimal. Because that dataset was small, we trained a behavioral-cloning policy and sampled another 500 interaction trajectories from it.
We compared AWAC with methods for off-policy learning, offline learning, and bootstrapping from demonstrations. Under the reported experimental conditions, AWAC used the prior data to attain useful initial performance and fine-tuned more quickly than DAPG. On the pen task, the paper reports reaching its solution criterion after 120,000 online timesteps, corresponding to approximately 20 minutes of equivalent interaction at the experiment’s control rate.
Within the evaluation horizon, several baselines and ablations made progress on the pen task, but the alternative off-policy and offline methods tested were largely unable to solve the door and relocation tasks. The reported ablations attributed AWAC’s advantage over AWR to off-policy critic estimation and its advantage over ABM to avoiding an explicit behavior model. These comparisons apply to the specified 2020 implementations, tasks, data, and training budgets.
Fine-tuning from random-policy data
Off-policy RL can also incorporate suboptimal experience rather than demonstrations alone. We evaluated this setting in a simulated tabletop-pushing environment with a Sawyer robot.
The offline dataset contained 500 trajectories generated by a random process. The task was to push an object to a target position in a 40-centimeter by 20-centimeter goal region. In the reported learning curves, many methods began with similar performance, but AWAC improved fastest online and made effective use of the offline dataset; some of the tested alternatives did not learn successfully within the evaluation window.
Future directions
Being able to reuse prior data and then fine-tune on new problems opens several research directions. We were particularly interested in moving from single-task RL toward multitask learning, with data sharing and generalization across tasks. Achieving broad generalization in robotics will require algorithms that can use large bodies of prior experience while still collecting targeted data online.
Robotics differs from supervised domains in an important way: collecting high-quality task data may be nearly as difficult as solving the task itself. Active data collection therefore remains an essential part of the problem rather than merely a source of labels.
This work focused primarily on mismatch between the learned policy’s action distribution π and the behavior data πβ. Off-policy learning also involves mismatch between their marginal state distributions. Consider a problem with two solutions, A and B, where B has higher return but the offline data mostly follows A. Even if online exploration discovers B, updates sampled uniformly from the combined buffer may still concentrate on states along A. This state-distribution problem has been studied in prior work; accounting for both action and state mismatch may lead to better algorithms.
Finally, AWAC suggested a practical way to retain the otherwise disposable data produced during research. Before solving a task with RL, we might teleoperate the robot to confirm that the task is feasible, test a hard-coded controller, run behavioral-cloning experiments, or sweep the hyperparameters of another RL algorithm. Instead of discarding those interactions, we can save them as prior data for subsequent offline pretraining and online fine-tuning.
A contemporaneous version of the work is available as arXiv v3. The latest paper revision is v6, dated April 24, 2021. AWAC example code is included in the Berkeley RLkit repository, and videos, environments, and project materials are available from the project website.
Source: The Berkeley Artificial Intelligence Research Blog. This archive entry originated as an automated feed import into A.I. Kung Fu.
Responses