ReBRAC-v2: The Return of the King

The question

How far can careful modernization take a conventional offline RL algorithm?

ReBRAC-v2 is not built around a single replacement. It integrates complementary advances while preserving the behavior-regularized actor-critic structure that made ReBRAC simple and strong.

Abstract

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value-guidance mechanisms. We ask whether comparable progress can instead come from systematically modernizing a conventional behavior-regularized actor-critic while preserving its algorithmic simplicity. We introduce ReBRAC-v2, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection. Rather than tuning this recipe separately for every task, we develop one shared configuration through roughly 600 Bayesian proposals on six challenging OGBench tasks, freeze all structural and optimization choices, and adapt only two behavior-regularization coefficients over a 16-point grid. Across ten common state-based OGBench categories, ReBRAC-v2 averages 74.8 and ranks first in eight categories. The same recipe, without structural changes, obtains the strongest averages in our matched comparisons on D4RL AntMaze and Adroit. These results show that disciplined, transferable engineering can achieve strong aggregate performance without abandoning a minimalist offline RL foundation.

The recipe

One actor-critic loop, four coordinated upgrades

The flow is the policy itself: the same parameters receive exact-likelihood, pointwise cloning, and critic gradients. There is no iterative sampler, proxy behavior model, or distilled one-step actor.

01

Exact-likelihood flow actor

Fourteen affine-coupling blocks model multimodal actions with one-pass sampling and tractable density. Mixed NLL, MSE, and MAE behavior cloning keeps policy improvement close to the offline data.

NLL + MSE + MAE − λQ_min

02

Categorical residual critics

Four deep residual critics predict 201-bin HL-Gauss values, share a persistent next-state objective, and use a mean Bellman target during training.

4 critics · 201 bins · mean target

03

Staged optimization

Behavior cloning first produces a useful policy, critic warm-up then learns its value function, and only afterward does joint actor-critic training begin.

100K BC → 200K critic → 700K joint

04

Test-time action search

The frozen EMA actor samples 32 candidates. Every action receives two normalized critic-gradient steps, and the minimum ensemble score selects the action to execute.

sample 32 → refine 2× → arg max Q_min

Shared development ≈600 proposals across six hard tasks

Freeze architecture, training, and inference

Adapt two BC coefficients on a 4 × 4 grid

Matched evaluation

Broad gains across OGBench and D4RL

ReBRAC-v2 leads the aggregate comparison across score thresholds, robust summary statistics, and most pairwise probabilities of improvement. The advantage is not driven by one task family.

On OGBench, ReBRAC-v2 improves the ten-category average by 22.5 normalized-score points over the next-best aggregate result and ranks first in eight categories.

AntMaze Giant 75.0 HumanoidMaze Large 72.2 Scene 95.4 Puzzle 3×3 99.6

Hover for exact values and confidence intervals. Drag to zoom; double-click to reset.

Loading interactive plot…

Performance profile. ReBRAC-v2 stays ahead across most score thresholds.

Median, interquartile mean, and mean normalized scores with ReBRAC-v2 highest on all three aggregate metrics — **Robust aggregates.** ReBRAC-v2 leads in median, IQM, and mean.

Probability-of-improvement plot comparing ReBRAC-v2 against ReBRAC, FQL, GFP, FAC, and DriftQL — **Pairwise evidence.** Estimated probability that ReBRAC-v2 improves on each baseline.

Controlled analysis

Sampling helps most; sufficient flow capacity matters

These are fixed-recipe substitutions: all surrounding hyperparameters, including both cloning coefficients, remain unchanged. They measure recipe sensitivity rather than performance after retuning.

Hover over a cell or point to inspect the exact score, uncertainty, and experimental setting.

Heatmap of average OGBench score over numbers of policy samples and critic-gradient refinement steps — **Inference-time compute.** Moving from one to eight samples adds 14.1%; refinement is smaller and task-dependent.

Line chart showing average and task-level normalized scores as normalizing-flow depth changes from two to eighteen layers — **Flow depth.** Shallow actors degrade sharply; performance largely plateaus from 8 to 18 layers.

88.4%

Mixed cloning is central

Removing the auxiliary MSE+MAE term causes the largest objective-level decrease under fixed coefficients.

13.9%

Warm-ups are useful

Removing both behavior-cloning and critic warm-ups weakens the selected recipe across development tasks.

8.3

Cube Double remains open

The main failure case may involve a task-specific critic mismatch, but the current evidence is not conclusive.

Reproduce and extend

Code, configurations, and raw results

The release contains the JAX implementation, final OGBench and D4RL configurations, an exact environment, and the CSV sources used to construct the result tables.

Repository

git clone https://github.com/DT6A/ReBRAC-v2.git

BibTeX

@article{tarasov2026rebracv2,
  title={ReBRAC-v2: The Return of the King},
  author={Tarasov, Denis and Katzschmann, Robert K.},
  journal={arXiv preprint arXiv:2608.01205},
  year={2026}
}

ReBRAC-v2 The Return of the King