I am Vassillen Chizhov, currently in my final year as a PhD student at the Mathematical
Image Analysis Group at Saarland University, Germany.
I will be presenting our work "Perceptual Error Optimization for Monte Carlo Rendering".
This is joint work with Iliyan Georgiev from Autodesk, United Kingdom, Karol Myszkowski
and Gurprit Singh from the Max-Planck Institute for Informatics, Saarland, Germany.
First, I will motivate our work using some practical applications.
Then I will go over relevant previous work and its relation to our approach.
We will also need some basic ideas from halftoning, and I will describe how our work transfers
those to rendering.
I will conclude with a discussion of our algorithms, results, and extensions.
In practice we want to synthesize renderings that have a lower perceptual error at the
same number of samples.
We achieve this by optimizing the screen-space samples' distribution resulting in the high-quality
image on the right.
In contrast, the noisy image on the left was synthesized using a standard approach.
Our optimization is performed with respect to a specific blurring kernel.
If this kernel is used not only for optimization, but also for reconstruction, then we can get
a very smooth image, as shown on the right.
Georgiev and Fajardo's work introduces the first practical algorithm for optimizing the
screen-space samples' distribution in rendering.
They perform an offline optimization with the goal of achieving a blue noise error distribution.
Heitz and colleagues also employ an offline optimization, but it is adapted to a Sobol
sampler, and has better scaling with the number of samples.
We mainly build upon the work of Heitz and Belcour, where they use an online optimization
and offset its cost over several frames in an animation setting.
Finally, Ahmed and Wonka proposed a Sobol sampler with similar performance to the work
of Heitz and colleagues, with lower storage requirements.
Our work provides a unifying framework that can be used to analyze any of the above methods.
It also provides insights into their advantages, disadvantages, assumptions, limitations, and
it lays out guidelines for future work.
In order to understand our approach, we need to introduce some basic ideas from halftoning.
Given a reference image "I", and a set of quantization levels, the main goal of halftoning
is to produce a quantized image "Q" similar to the reference.
More precisely, we want the quantized image to look similar to the reference from a specific
distance.
As an example, consider a quantization set consisting of 3 gray values: black, gray,
and white.
We can generate a quantized image using these 3 colors.
If the quantized image is the result of an optimization taking into account the human
visual system,
then the kernel used to model the human visual system can also be used for the reconstruction,
as shown on the right.
To understand how the optimization may be derived we can look at the error image.
In the top left we have the error for an "optimal" quantization, while on the bottom right we
have the error resulting from random dithering.
The key is to study how those evolve under convolution with kernels modeling the human
visual system.
In the example above, we consider a Gaussian kernel (Pappas and Neuhoff 1999), and it becomes
clear that the optimized error decays much faster under convolution.
This is even more apparent for a Gaussian with a larger standard deviation, which corresponds
to a larger viewing distance.
This means that we want to minimize the error perceived from a certain distance, which we
can model through the following minimization problem:
This minimization problem lies at the heart of halftoning.
Methods such as dispersed-dot dithering and error diffusion aim to quickly approximate
the optimum for a specific kernel.
Our main contribution lies in transferring those halftoning ideas to rendering.
However, there are some non-trivial challenges.
The first difficulty is that what used to be the quantization image "Q" is now an image
formed by Monte Carlo estimators for the measurement equation (Veach 1997).
The final image is thus a function of samples.
A bigger issue is that we do not have the reference "I" in a rendering context.
However, we will see that it may be replaced by a suitable smooth approximation, which
we term a surrogate or guidance image.
We first tackle the issue of the image being a function of samples.
Consider the setting where we have generated a vertical stack of images at one sample per
pixel.
The vertical set of estimates within a single pixel can then be interpreted as the already
familiar quantization levels' set from halftoning.
We term the above search space "vertical", and then the optimization can be done as in
halftoning.
In fact the left image from the motivation section was produced by random dithering on
the stack of estimates,
while the image on the right minimizes the described halftoning energy.
The "vertical" search space corresponds to the search space in the "histogram" approach
of Heitz and Belcour,
however they do not formulate an explicit minimization problem.
Additionally their halftoning method is an order statistics match, which can be seen
as a special case of dispersed-dot dithering,
and their algorithm also implicitly relies on a sub-optimal guidance image.
The horizontal search space consists of all possible permutations of the per pixel sample
sets.
That is, we allow the optimization to swap the samples used to estimate two pixels.
In practice this would incur a heavy cost, as each swap requires a re-rendering.
To alleviate this we choose to use pixel swaps instead of sample swaps for the optimization.
At the end, we use the resulting permutation on the samples, and then re-render.
The procedure is illustrated in the following images.
The top row corresponds to the results where the pixels have been swapped.
The bottom row corresponds to the results where the samples have been swapped and the
image has been re-rendered.
We can see that Heitz and Belcour's approach results in a much noisier image, especially
near edges.
This is because they use disjoint tile regions for the allowed swaps, and they use order
statistics matching for halftoning.
Our approach uses overlapping disk regions for allowed swaps, and an iterative minimization,
which results in a much better fit.
We illustrate the effect of replacing the true solution with a guidance image.
On the left we have a piece-wise constant averaged image, and while it is much smoother
than the noisy estimate on the right,
it is also biased and exhibits block artifacts.
Our optimization essentially projects this biased guidance image on the space of unbiased
estimates, reducing the bias,
while still being much smoother than the noisy image on the right.
This can also be used to reduce hallucinated artifacts, visible on the left, resulting
from a denoiser.
Applying the same optimization approach, the artifacts are minimized, while the smoothness
is preserved to a large extent.
The question arises how to minimize the energy in practice.
Different methods can be categorized in three main classes.
The first option is to apply a greedy iterative minimization procedure such as best-first
search.
It is the most flexible of all of our methods, but it is also the slowest.
Error diffusion approaches, such as Floyd-Steinberg, are much faster but aren't as flexible.
Finally, dispersed-dot dithering is the fastest, but it also has the lowest fitting power.
Heitz and Belcour's "histogram" and "permutation" approaches fall under this last category.
Both dispersed-dot dithering and error diffusion can be seen as fast approximations to the
solution
produced from the iterative minimization approach.
In the following, we illustrate the robustness of iterative minimization methods to different
kernels, using a few instructive experiments.
We start with a white noise distribution and a horizontal/permutation search space.
If we optimize the energy with respect to a low-pass filter, then the result is a blue
noise distribution.
The power spectrum of the kernel is shown on the bottom left, while the power spectrum
of the optimized image is shown on the bottom right.
Note that ideally the power spectra should be inverses of each other, as is the case
here.
Using a band-stop filter results in a green noise distribution as expected, and using
a high-pass filter, results in red noise.
Similarly using a band-pass filter results in violet noise.
The optimization can also handle anisotropic kernels and spatially varying kernels.
The latter ought to be helpful for future work aiming to incorporate denoisers in the
optimization, since denoisers can produce both anisotropic and spatially varying kernels.
Thus we have seen, that the iterative minimization algorithms that we consider, are extremely
robust and flexible, which explains their high fitting power and good performance.
Next we provide visual comparisons of our best methods to the "histogram" and "permutation"
algorithms of Heitz and Belcour.
In all cases the perceptual error, that is the formulated halftoning energy, is also
always much lower for our approaches.
Note the higher noise for horizontal methods near edges, such as the ceiling and at the
base of the windows.
This is less apparent for the bathroom scene, but it still occurs near the edges of the
sink.
In the living room scene, this effect is invisible for our horizontal method,
however for Heitz and Belcour's permutation it is still very prominent near the picture
frame and on the wall near the floor.
We emphasize that the above experiments are chosen to be an ideal case for their approach,
as the scene remains static and
horizontal approaches are given 16 "animation" frames to converge.
We use a state-of-the-art visual quality metric: HDR-VDP-2 by Mantiuk and colleagues, to evaluate
our results.
The error map from HDR-VDP-2 is visualized on the right halves of the images.
The ranking from the perceptual energy seems to match well this advanced metric.
That is, the methods with lowest fitting power are Heitz and Belcour's approaches and our
horizontal optimization,
while the ones with the highest fitting power are our vertical methods.
We note that the worse the guidance image is, the lower the fitting power used should
be.
This means that there are cases where any of the methods outperform the others, depending
on the quality of the guidance image.
The discussed iterative energy minimization allows for straightforward extensions through
small changes in the energy.
For example, we can take into account viewing distance by modifying the blurring kernel
used in the optimization.
On the left, the image is optimized for near viewing, while on the right the optimization
aims at far viewing distances.
Note the enhancement of the edges in the right image.
Another important feature is the handling of colors.
Heitz and Belcour convert the image to grayscale in order to match the order statistics.
We have illustrated such a result using our methods in the top right.
It exhibits objectionable color noise.
A straightforward improvement consists of modifying the norm in the energy to work on
RGB vectors, see the image in the top left.
The best quality is achieved by applying the optimization to each channel separately, as
illustrated in the image in the bottom right corner.
However, note that the latter is applicable only for vertical search spaces.
If the rendered image is tonemapped, then it also makes sense to include the tonemapping
in the optimization,
as otherwise sub-optimal results are achieved, see for example the lower left corner.
We constructed a framework by transferring key ideas from halftoning to the rendering
setting.
This naturally lead to a collection of algorithms mirroring their halftoning counterparts.
Additionally, we proposed several extensions to handle standard rendering features.
A promising avenue for future research is replacing the convolution in the energy with
a nonlinear reconstruction.
This can be used to couple a denoiser to the optimization.
Another important use of our theory is the construction of new offline methods.
Some preliminary experiments regarding this are available in our supplemental document.
Thank you for your attention.