ResidualAnchor

Fast local sampling in continuous latent space

Continuous latents
Gaussian mixtures

Flow matching generates an image by consulting the same neural network several times along a path from noise. ResidualAnchor tests how much of that repeated inference can be avoided: it starts from a stored representation the decoder already understands, makes one small geometry-aware move, and decodes once. That representation is a long list of numbers called a latent. PCA splits it into a smaller map that is easier to move through and a remainder that preserves detail worth keeping. A class-conditional Gaussian mixture model, or GMM, learns the typical local shapes within each class, describing which directions and scales vary around an anchor.

01
Anchor
Choose an image from class $y$ and use its saved latent $z_{e, a}$ as the starting point. The encoding is prepared before sampling.
$z_{e,a} = E(x_a), \quad x_a \in \mathcal{D}_y$
02
Split
Project the anchor into the smaller coordinate $u_{a}$ that is easy to move through, then preserve the fine detail removed by compression as $r_{a}$ .
$u_a = \operatorname{PCA}(z_{e,a}), \quad r_a = z_{e,a} - \operatorname{PCA}^{-1}(u_a)$
03
Move
Assign the anchor to its most likely GMM component $k$ . Its covariance $Σ_{y, k}$ shapes a zero-mean local change $ε$ , while $τ$ controls how far $u_{a}$ moves.
$\varepsilon \sim \mathcal{N}(0, \Sigma_{y,k}), \quad u' = u_a + \tau \varepsilon$
04
Decode
Map the moved coordinate back to the full latent, restore $r_{a}$ , and decode once to produce the final image $x^{'}$ .
$z_e' = \operatorname{PCA}^{-1}(u') + r_a, \quad x' = D(z_e')$

That is the entire sampling path. At $τ$ = 0, it simply decodes the stored anchor without changing it. Positive values make a new local move; larger values move farther and generally exchange fidelity for more variation. We test this idea on CIFAR-10 using RF++ as the pretrained flow-matching baseline. RF++ starts from noise and asks a neural network which direction the image should move, repeating that process five times. ResidualAnchor tests what happens when those repeated flow-model calls are replaced with the single local move above.

RF++ baseline5 flow-model calls

Noise

Field1Field2Field3Field4Field5

Sample

ResidualAnchor0 flow calls · 1 decode

Stored latent

Local moveDecode ×1

Sample

Across every tested positive value of $τ$ , ResidualAnchor recorded roughly 2,306 to 3,032 images per second with zero flow-model calls and one decoder pass. The chart below compares that recorded image throughput with distance from real CIFAR-10. Orange circles show ResidualAnchor at different move sizes, called $τ$ . The green square shows the continuous autoencoder reconstruction reference, and the white diamond shows the five-step RF++ baseline. Higher means faster; farther left means closer to real CIFAR-10 in this measurement. The ideal direction is toward the top-left.

ResidualAnchor at different values of $τ$
Continuous autoencoder reconstruction
RF++ flow-matching baseline

Continuous reconstruction: 5,197/s7.10 distance
$τ$ = .01: 2,598/s8.95 distance
$τ$ = .025: 2,306/s9.10 distance
$τ$ = .05: 2,986/s9.35 distance
$τ$ = .075: 2,987/s10.02 distance
$τ$ = .10: 3,032/s10.68 distance
RF++ · 5 flow calls: 170/s4.45 distance

Exact recorded generation speed and feature-space distance values
Generating method	Feature-space distance	Images / second	Flow-model calls	Timing source
Continuous reconstruction	7.0956	5196.668	0	In-process reconstruction
τ = .01	8.9543	2597.507	0	In-process evaluation
τ = .025	9.0977	2305.650	0	In-process evaluation
τ = .05	9.3519	2986.035	0	In-process evaluation
τ = .075	10.0224	2986.976	0	In-process evaluation
τ = .10	10.6810	3032.190	0	In-process evaluation
RF++ · 5 flow calls	4.4545	170.018	5	Saved run metadata

A pretrained ResNet18 turns the real and evaluated images into feature vectors, then this distance compares the average and spread of those two sets. Lower means closer according to that network. This score is a distribution-level measurement, not a grade for each picture, and it is not standard Inception FID. Each group below shows all 16 published samples in a four-by-four grid. Groups were sampled independently, so positions do not follow one anchor as $τ$ changes.

Existing images rebuilt

Continuous reconstruction Autoencoder reference

Samples / second: ~5,197
Flow evaluations: 0
Feature distance ↓: 7.10

Stored-anchor replay

ResidualAnchor $τ$ = 0

Samples / second: ~1,494
Flow evaluations: 0
Feature distance ↓: 9.17

Small move

ResidualAnchor $τ$ = 0.025

Samples / second: ~2,306
Flow evaluations: 0
Feature distance ↓: 9.10

Medium move

ResidualAnchor $τ$ = 0.05

Samples / second: ~2,986
Flow evaluations: 0
Feature distance ↓: 9.35

Larger move

ResidualAnchor $τ$ = 0.10

Samples / second: ~3,032
Flow evaluations: 0
Feature distance ↓: 10.68

Flow baseline

RF++ baseline 5 flow steps

Samples / second: ~170
Flow evaluations: 5
Feature distance ↓: 4.45

Reconstruction rebuilds existing test images and $τ$ = 0 decodes stored anchors, so neither is independent generation. The chart also includes $τ$ = 0.01 and $τ$ = 0.075, which were measured quantitatively but are not shown in these image grids.

These measurements show a tradeoff, not a winner. ResidualAnchor occupies the faster region of the recorded results; RF++ occupies the lower-distance region. There is no universally best value of $τ$ . Smaller moves stayed closer to their anchors; larger moves generally increased variation and distribution distance. To see whether the shortcut could inherit more of the flow model’s geometry, we used a frozen RF++ model as a teacher and adapted the GMM’s local movement. The generation path itself did not change: sampling still uses no RF++ calls. The adapted sampler followed the teacher more closely, while its measured distance from real images remained essentially unchanged. The teacher transferred local geometry, not a measurable image-quality gain.

VLA, world-model, video-model, and flow-based action policies often place an iterative generative sampler between the model backbone and the result the system uses. If an analogous local sampler preserves task-level behavior while replacing several field evaluations with one learned local move, it could shorten that stage, increase the control rate, produce more candidate actions per cycle, or free compute for perception and planning.

ObservationModel backboneAction, state, or video samplingUse the result

Sampling speed alone is not enough. Image-distribution measures such as FID - and the related ResNet feature-distance score used above - describe visual similarity, not whether a predicted action is safe, dynamically feasible, or effective. Many robot action spaces are smaller and more structured than image space, so an image-level fidelity change does not translate directly into the same loss of action quality. The useful threshold is the fastest setting that preserves trajectory accuracy, constraint satisfaction, task success, and closed-loop stability, as well as temporal or rollout consistency where relevant.