High Dynamic Range Imaging

High Dynamic Range Imaging Summer Internship Project Report

by

V Lakshana VI semester Information Technology Department College of Engineering and Tehnology, Bhubaneswar

under the guidance of

Prof. Pankaj Kumar Sa Lecturer Computer Science and Engineering Department National Institute of Technology Rourkela

Department of Computer Science and Engineering National Institute of Technology Rourkela Rourkela-769 008, Orissa, India.

Certificate This is to certify that the work in this project report entitled High Dynamic Range Imaging by V. Lakshana, student of Information Technology, College of Engineering and Technology, Bhubaneswar, has been carried out under my supervision during Summer Internship Programme from 8th May 2009 to 15th July 2009 in Department of Computer Science & Engineering, National Institute of Technology Rourkela.

NIT Rourkela 15 July 2009

Pankaj Kumar Sa Lecturer CSE department of NIT Rourkela

Acknowledgements While undergoing this project I was helped by many people. I express my sincere gratitude to my project guide, Prof. Pankaj Ku Sa, Dept of Computer Science and Engineering,NIT Rkl for his kind and able guidance for the completion of the thesis work. His consistent support and intellectual guidance made me energize and innovate new ideas. I would also like to thank Miss Hunny Mehrotra and Mr. Ratnakar Dash for their generous help in various ways for the completion of this project.

V. Lakshana 4th year, Information Technology College of Engineering & Technology, Bhubaneswar

Abstract While real scenes produce a wide range of brightness variations, vision systems use low dynamic range image detectors that typically provide 8 bits of brightness data at each pixel. The resulting low quality images greatly limit what vision can accomplish today. In image processing, computer graphics and photography, High Dynamic Range Imaging (HDRI) is a set of techniques that allows a greater dynamic range of exposures (the range of values between light and dark areas) than normal digital imaging techniques. The intention of HDRI is to accurately represent the wide range of intensity levels found in real scenes ranging from direct sunlight to shadows. This project attempts improving the effective dynamic range of cameras by using multiple photographs of the same scene taken with different exposure times. Using this method enables the photographer to accurately capture scenes that contain a high dynamic range, i.e., scenes that have both very bright and very dark regions. The approach requires an initial calibration, where the camera response function is determined. Once the response function for a camera is known, high dynamic range images can be computed easily. The end result is an imaging system that can measure a very wide range of scene radiances and produce a substantially larger number of brightness levels, with a slight reduction in spatial resolution. One problem with HDR has always been in viewing the images. Typical computer monitors (CRTs, LCDs), prints, and other methods of displaying images only have a limited dynamic range. Various methods of converting HDR images into a viewable format have been developed, generally called Tone Mapping. In this project, it is considered that nearly any camera can actually capture a vast dynamic range– just not in a single photo. By varying the shutter speed alone, most digital cameras can change how much light they let in by a factor of 50,000 or more. This project attempts to combine these captured images into a single High Dynamic Range Image by utilizing the different exposures characterized by these images composed of multiple exposures. The difficulties that accompany HDRI are Halo effects and artifacts. Care has been taken to avoid these pitfalls. The complete project has been simulated using Matlab 7.6 software. The simulation results are shown as figures. Results have shown very high quality rendering of images by this technique.

Contents Certificate

i

Acknowledgements

ii

Abstract

iii

List of Figures

v

1 Introduction

1

1.1

Historical Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2

Traditional Techniques Used . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.3

Recent Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

1.4

Tone Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

2 Problem Statement

4

3 Sample Example

5

4 Background Research

8

4.1

Recovering High Dynamic Range Radiance Maps from Photographs . . . 4.1.1

4.2

The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8 10

A Simple Spatial Tone Mapping Operator for High Dynamic Range Images 15 4.2.1

The Need for Tone Mapping . . . . . . . . . . . . . . . . . . . . .

15

4.2.2

The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

5 Simulation

18

5.1

Implementation Results— Chair image . . . . . . . . . . . . . . . . . . .

18

5.2

Implementation Results— Sunset image

18

. . . . . . . . . . . . . . . . . .

6 Applications of HDRI

21

7 Conclusions

24

References

25

iv

List of Figures 1

Limitations of dynamic range of the capturing equipment. . . . . . . . .

2

Here the dynamic range of the HDR image is demonstrated by showing

1

how the overexposed image(left) and the underexposed image(middle) are combined to form it(right). . . . . . . . . . . . . . . . . . . . . . . . . . .

4

3

Sample input set of images with different exposure time. . . . . . . . . .

5

4

Linear map of an hdr picture into range [0,1] . . . . . . . . . . . . . . . .

6

5

Clamping hdr picture at 1 . . . . . . . . . . . . . . . . . . . . . . . . . .

6

6

Logarithmic mapping of the hdr picture into range [0,1] . . . . . . . . . .

6

7

hdr picture mapped with reinhard global tonemapping . . . . . . . . . .

7

8

hdr picture mapped with reinhard local tonemapping . . . . . . . . . . .

7

9

Image Acquisition Pipeline: shows how scene radiance becomes pixel values for both film and digital cameras. Unknown nonlinear mappings can occur during exposure, development, scanning, digitization, and remapping. . .

10

9

In the figure, the symbols represent samples of the g curve derived from the digital values at one pixel for 5 different known exposures using Equation(4). The + and o symbols show samples of g curve segments derived by consideration of two other pixels. . . . . . . . . . . . . . . . . . . . . . . . .

11

13

The unknown log irradiance ln Ei has been arbitrarily assumed to be 0. Its worth noting that the shape of the g curve is correct, though its position on the vertical scale is arbitrary corresponding to the unknown ln Ei ; again the vertical position of each segment is arbitrary. Essentially, what we want to achieve in the optimization process is to slide the 3 sampled curve segments up and down (by adjusting their ln Ei s) until they line up into a single smooth, monotonic curve, as shown in the figure. The vertical position of the composite curve will remain arbitrary.

. . . . . . . . . .

14

12

Hdr image tone mapped with reinhard local operator on image A . . . .

19

13

Hdr image tone mapped with reinhard global operator on image A . . . .

19

14

Hdr image tone mapped with reinhard local operator on image B . . . .

20

15

Hdr image tone mapped with reinhard global operator on image B . . . .

20

v

16

The input of the HDRI program. Inputimage-¿chair Note the variations in the details of each image. The exposure times are in decreasing order. i.e. the darkest image has the least exposure time while the brightest has the maximum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

21

The HDR radiance map. x − axis Represents log Exposure(X) while y − axis represents pixelvalueZ. The three curves represent the Red, Green, and Blue components of the images. . . . . . . . . . . . . . . . . . . . . .

21

18

The HDR image obtained from the radiance map. . . . . . . . . . . . . .

22

19

The final tone mapped HDR image (LDR). . . . . . . . . . . . . . . . . .

22

20

Two of the input images of Sunset. (a) Over exposed image, (b) Under exposed image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

23

The HDR radiance map. x − axis Represents log Exposure(X) while y − axis represents pixelvalueZ. The three curves represent the Red, Green, and Blue components of the images. . . . . . . . . . . . . . . . . . . . . .

22

23

The output images of Sunset. (a)HDR image obtained from the radiance map, (b)The final tone mapped HDR image(LDR). . . . . . . . . . . . .

vi

24

1 Introduction

1

Introduction

The need for HDR-rendering arises when comparing standard display devices, which have a dynamic range of luminance of about 100:1, with human vision which has a dynamic range of about 1,00,00,00,000:1, from bright sunlight down to starlight. High Dynamic Range Imaging (HDRI) is a set of techniques that allows a greater dynamic range of luminance between light and dark areas of a scene than normal digital imaging techniques. High dynamic range (HDR) images enable photographers to record a greater range of tonal detail than a given camera could capture in a single photo. This opens up a whole new set of lighting possibilities which one might have previously avoided. The intention of HDRI is to accurately represent the wide range of intensity levels found in real scenes ranging from direct sunlight to shadows. The problem caused due to the dynamic range of the capturing equipment is illustrated in figure 1. In the first image the shelf, the table and the sofa set are the subject, whereas the window is the surrounding. Not much is seen outside the window, but the details on shelf and the table are seen pretty clearly. In the second image the surrounding is clear enough but nothing is visible inside the room. The limitations of dynamic range of camera donot allow to accommodate both the features in one image.

(a)

(b)

Figure 1: Limitations of dynamic range of the capturing equipment.

1.1

Historical Perspective

In the past, photographers dealt with the limitation of dynamic range by experimenting with media. Ansel Adams was perhaps the first to systematically measure the sensitivity range of all of the equipment he used to precisely indicate what the photograph would display depending on the length of time the shutter was open [1]. Before HDR was put 1

1.2 Traditional Techniques Used

into practice, photographers and film makers needed to find ways around the limitations of modern digital imaging which fail to capture the total perceptual display that the human eye is capable of seeing. The problem is that real-world scenes contain light ranges that exceed a 50,000:1 dynamic range while digital equipment is limited to around 300:1 dynamic range1 . The idea of using several exposures to fix a too-extreme range of luminance was pioneered as early as the 1850s by Gustave Le Gray to render seascapes showing both the sky and the sea. Such rendering was impossible at the time using standard techniques, the luminosity range being too extreme. Le Gray used one negative for the sky, and another one with a longer In 1997 the technique of combining several differently exposed images to produce a single HDR image was presented to the public by Paul Debevec and his research has since contributed significantly to the popularization of HDRI.exposure for the sea, and combined the two in a single picture in positive.As technology advanced so did the digital imaging market, thus creating HDR. Probably the first practical application of HDRI was by the movie industry in late 1980s and, in 1985, Gregory Ward created the Radiance RGBE image file format which was the first (and still the most commonly used) High Dynamic Range Imaging file format. Steve Mann developed the method for producing digital images having extended dynamic range. Images tend to result in the subject seen and the surrounding blanked out.

1.2

Traditional Techniques Used

Most photographic systems struggle on high contrasts scenes. With the entire modern technology one would be wrong to think a digital camera would resolve the issues related to dynamic range. The reason for this is simple, current film and image sensors are developed to record enough light that can be reproduced on paper or computer screen and neither are currently capable of displaying a dynamic range that we can see with our eyes. Until recently, to overcome this, digital photographers have either been using the old approach of ND graduated filters at the taking stage or combining the best bits of 1

The dynamic range is the ratio between the maximum and minimum values of a physical measurement. Its definition depends on what the dynamic range refers to.

2

1.3 Recent Technology two different exposures on the computer, much like the dodge 2 & burn technique used by experienced darkroom users.

1.3

Recent Technology

New technology is on the way to record and display at higher dynamic ranges, but for now there are a range of software solutions. The job of HDR processing is to allow photographers to produce images with a much larger dynamic range. The photographer takes a series of different exposures of the same scene and then uses HDR software to automatically merge the resulting images into one image, referred to as exposure blending. This produces a file with the entire desired dynamic range, known as a radiance map. Ideally the fewer images used the better, because it’s less memory intensive and there’s less chance of subject been out of alignment on each shot as a result of camera or subject movement. HDR uses 16 to 32 bits values. This jump in resolution allows for a much greater range of colours, greater than the monitor can display at once. As a result, to view these images, the computer breaks the images into levels of brightness and darkness. The brightest part of the image are viewed with the brightest colours the monitor can display the darkest are viewed with the darkest the screen can display. This allows for each image to display details, bright highlights and dark shadows.

1.4

Tone Mapping

The above mentioned merged photographs are saved as 32-bit. This means there’s very little one can do to edit the photos, they cannot be saved as JPG without reducing the bit-depth3 and one cannot view the superb dynamic range on a conventional monitor. In such cases, the whole dynamic range of the scene (e.g. 100,000:1 or higher) has to be reproduced on a medium with less than 100:1 contrast ratio. Tone mapping deals with the issue of reproducing the dynamic range captured. This is why it becomes necessary to use techniques that scale the dynamic range down while preserving the appearance 2

Dodging and Burning are terms used in photography for a technique used during the printing process, to manipulate the exposure of a selected area(s) on a photographic print, deviating from the rest of the image’s exposure. Dodging decreases the exposure for areas of the print that the photographer wishes to be lighter, while burning increases the exposure to areas of the print that should be darker. 3 Bit depth of a capturing or displaying device gives an indication of its dynamic range capacity, i.e. the highest dynamic range that the device would be capable of reproducing.

3

2 Problem Statement

Figure 2: Here the dynamic range of the HDR image is demonstrated by showing how the overexposed image(left) and the underexposed image(middle) are combined to form it(right). of the original image captured. One way around this is to apply tone mapping which converts the HDR image into a balanced exposure. Tone mapping operators are divided into two broad categories, Global and Local. Global operators are simple and fast. They map each pixel based on its intensity and global image characteristics, regardless of the pixel’s spatial location. Global operators are OK for mapping 12-bit sensor data but usually don’t work Ill with HDR images. An example of a global type of tone mapping is a tonal curve. Local operators take into account the pixel’s surroundings for mapping it. This means that a pixel of a given intensity will be mapped to a different value depending on whether it is located in a dark or bright area. This makes local operators slower but tends to produce more pleasing results, given that our eyes react locally to contrast.

2

Problem Statement

Here is the problem in a nutshell: • Real-world scenes contain light ranges that exceed a 50,000:1 dynamic range. • Media has been limited to around a 300:1 dynamic range. 4

3 Sample Example • So we have a mapping issue: how do we represent light values in a scene using a much more limited set of light values for a particular media? • What if the set of images with different exposures have unknown noise in it? Can we get the HDR image sans the noise (noise gets added, so it has to be removed.).

3

Sample Example

A sample example is shown from figure 3 to figure 8.

Figure 3: Sample input set of images with different exposure time. Since it is impossible to display a hdr radiance map, three simple compressed representations are given— 1. Linear mapping of hdr valuesto the displayable range [0,1]. Since the largest range of brightness values are used for the highlights,only those are visible in the mapped image. 2. Clamping the hdr radiance map at 1. Since most values in the hdr radiance map are much bigger than 1, this representation looks very bright. 5

3 Sample Example

Figure 4: Linear map of an hdr picture into range [0,1]

Figure 5: Clamping hdr picture at 1 3. Logarithmic mapping of the hdr radiance map,then doing the linear mapping into the range [0,1].

Figure 6: Logarithmic mapping of the hdr picture into range [0,1]

6

3 Sample Example

Figure 7: hdr picture mapped with reinhard global tonemapping

Figure 8: hdr picture mapped with reinhard local tonemapping

7

4 Background Research

4

Background Research

The method of recovering high dynamic range radiance maps from photographs taken with conventional imaging equipments [3] is presented in this project work. The proposed algorithm by Debevec & Malik [3] uses these differently exposed photographs to recover the response function of the imaging process, up to factor of scale, using the assumption of reciprocity. With the known response function, the algorithm can fuse the multiple photographs into a single, high dynamic range radiance map whose pixel values are proportional to the true radiance values in the scene. The algorithm and its implementations are detailed in section 4.1. Besides this, the project also implements (section 4.2) a simple and effective tone mapping operator proposed by Biswas & Pattanaik [2], that preserves visibility and contrast impression of high dynamic range images. It use a s-function type operator which takes into account both the global average of the image, as well as local luminance in the immediate neighbourhood of each pixel. The local luminance is computed using a median filter. It is seen that the resulting low dynamic range image preserves fine details, and avoids common artefacts such as halos, gradient reversals or loss of local contrast.

4.1

Recovering High Dynamic Range Radiance Maps from Photographs

Digitized photographs are becoming increasingly important in computer graphics. When one photographs a scene, either with film or an electronic imaging array, and digitizes the photograph to obtain a two dimensional array of brightness4 values, these values are rarely true measurements of relative radiance in the scene. For example, if one pixel has twice the value of another, it is unlikely that it observed twice the radiance5 . Instead, there is usually an unknown, nonlinear mapping that determines how radiance in the scene becomes pixel values in the image. This nonlinear mapping is hard to know beforehand because it is actually the composition of several nonlinear mappings that occur in the photographic process. In a conventional camera (see figure 9), the film is first exposed to light to form a latent image. The film is then developed to change this latent image into variations in 4 5

Brightness values represent the gray levels of the image. Radiance is the irradiance/luminance value of the object at the site of the scene.

8

4.1 Recovering High Dynamic Range Radiance Maps from Photographs

transparency, or density, on the film. The film can then be digitized using a film scanner, which projects light through the film onto an electronic light-sensitive array, converting the image to electrical voltages. These voltages are digitized, and then manipulated before finally being written to the storage medium. If prints of the film are scanned rather than the film itself, then the printing process can also introduce nonlinear mappings.

Figure 9: Image Acquisition Pipeline: shows how scene radiance becomes pixel values for both film and digital cameras. Unknown nonlinear mappings can occur during exposure, development, scanning, digitization, and remapping. In the first stage of the process, the film response to variations in exposure X 6 is a non-linear function, called the characteristic curve of the film. The development, scanning and digitization processes usually introduce their own nonlinearities which compose to give the aggregate nonlinear relationship between the image pixel exposures X and their values Z 7 . As with film, the most the most significant nonlinearity in the response curve is at its saturation point, where any pixel with radiance above a certain level is mapped to the same maximum image value. The most obvious difficulty is that of limited dynamic rangeone has to choose the range of radiance values that are of interest and determine the exposure time suitably. Sunlit scenes, and scenes with shiny materials and artificial light sources, often have extreme differences in radiance values that are impossible to capture without either under-exposing or saturating the film. To cover the full dynamic range in such a scene, one can take a series of photographs with different exposures. This then poses a problem: how can we combine these separate images into a composite radiance map? Here the fact that the mapping from scene radiance to pixel values is unknown and nonlinearity begins to haunt us. A simple technique for recovering this response function, up to a scale factor, using nothing more than a set of photographs taken with varying, known exposure durations 6 7

X = EΔt ; i.e. X is the product of the irradiance E the film receives and the exposure time Δt. Z is the final digital value after processing. Refer figure 9.

9

4.1 Recovering High Dynamic Range Radiance Maps from Photographs

has been the proposed work by Debevec & Malik [3] and is simulated in this project. With this mapping, the pixel values from all the available photographs are used to construct an accurate map of the radiance in the scene, up to a factor of scale. This radiance map will cover the entire dynamic range captured by the original photographs. 4.1.1

The Algorithm

This section presents the algorithm cited in [3], for recovering the film response function, and then presents the method of reconstructing the high dynamic range radiance image from the multiple photographs. The algorithm assumes gray scale images, but can be extended to colour images with proper conversions8 . Film Response Recovery This algorithm is based on exploiting a physical property of imaging systems, both photochemical and electronic, known as reciprocity. Let us consider photographic film first. The response of a film to variations in exposure is summarized by the characteristic curve (or Hurter-Driffield curve) [3]. This is a graph of the optical density D of the processed film against the logarithm of the exposure X to which it has been subjected. The exposure X is defined as the product of the irradiance E at the film and exposure time, Δt, so that its units are Jm− 2 . Key to the very concept of the characteristic curve is the assumption that only the product EΔtis important, and that halving E and doubling Δt will not change the resulting optical density D Under extreme conditions (very large or very low Δt ), the reciprocity assumption can break down, a situation described as reciprocity failure. In typical print films, reciprocity holds to within 1/3 stop9 for exposure times of 10 seconds to 1/10000 of a second. After the development, scanning and digitization processes, we obtain a digital number Z, which is a nonlinear function of the original exposure X at the pixel. Let us call this function f , which is the composition of the characteristic curve of the film as well as all the nonlinearities introduced by the later processing steps. The first goal is to recover this function f Once we have that, we can compute the exposure X at each pixel, as X = f −1 (Z). A reasonable assumption made is that the function f is monotonically increasing [3], so its inverse f −1 is well defined. Knowing the exposure X and the exposure 8

We can convert RGB images to YCbCr images and consider only the Y (luminance) component or else we can consider each component individually and plot the mapping. 9 1 stop is a photographic term for a factor of two ; 1/3 stop is thus 21/3 .

10

4.1 Recovering High Dynamic Range Radiance Maps from Photographs time Δt, the irradiance E is recovered as E=X/Δt, which will be taken to be proportional to the radiance L in the scene. The input to the algorithm is a number of digitized photographs taken from the same vantage point with different known exposure durations Δtj 10 It is assumed that the scene is static and that this process is completed quickly enough that lighting changes can be safely ignored. It can then be assumed that the film irradiance values Ei for each pixel i are constant. Pixel values are denoted by Zij where i is a spatial index over pixels and j indexes over exposure times Δtj . Thus the film reciprocity equation is: Zij = f (Ei Δtj )

(1)

Since it is assumed f is monotonic, it is invertible, and so we can rewrite (1) as: f −1 (Zij ) = Ei Δtj

(2)

Taking the natural logarithm of both sides of (2), we have: ln f −1 (Zij ) = ln Ei + ln Δtj

(3)

To simplify notation, let us define a function, g = ln f −1 We then have the set of equations: g(Zij ) = ln Ei + ln Δtj

(4)

where i ranges over pixels and j ranges over exposure durations. In this set of equations, the Zij are known, along with their corresponding Δtj The unknowns are the irradiances Ei as well as the function g, although it is assumed that g is smooth and monotonic. Our aim now is to recover the function g and the irradiances Ei that best satisfy the set of equations arising from Equation (4) in a least-squared error sense. It can be verified that recovering g only requires recovering the finite number of values that g(Z) can take since the domain of Z, pixel brightness values, is finite. Let Zmin and Zmax be the least and greatest pixel values (integers), N be the number of pixel locations, and P be the number of photographs. 10

The actual exposure times are varied by powers of two between stops (1/64, 1/32, 1/16, 1/8, 1/4, 1/2, 1, 2, 4, 8, 16, 32), rather than the rounded numbers displayed on the camera readout (1/60, 1/30, 1/15, 1/8, 1/4, 1/2, 1, 2, 4, 8, 15, 30).

11

4.1 Recovering High Dynamic Range Radiance Maps from Photographs Thus the existing problem can be formulated as one of finding the (Zm in − Zm ax + 1) values of g(Z) and the N values of ln Ei that minimize the following quadratic objective function: o=

N  P 

[[g(Zij ) − ln Ei − ln Δtj ]]2 + λ

i=1 j=1

z=Z max −1

g  (z)2

(5)

z=Zmin +1

The first term ensures that the solution satisfies the set of equations arising from Equation (4) in a least squares sense. The second term is a smoothness term on the sum of squared values of the second derivative of g to ensure that the function g is smooth. In this discrete setting g  (z) = g(z − 1) − 2g(z) + g(z + 1) is used. This smoothness term is essential to the formulation in that it provides coupling between the values g(z) in the minimization. The scalar λ weights the smoothness term relative to the data fitting term, and should be chosen appropriately for the amount of noise expected in the Zij measurements. Because it is quadratic in the Ei ’s and g(z)’s, minimizing o is a straightforward linear least squares problem. The over-determined system of linear equations is robustly solved using the singular value decomposition (SVD) method. Along with this equation some addition points need to be considered. First, the solution for the g(z) and Ei values can only be up to a single factor α [3]. If each log irradiance value ln Ei were replaced by ln Ei + α, and the function g replaced by g + α, the system of equation (4) and also the objective function o would remain unchanged. To establish a scale factor we [3] introduced the additional constraint (Zmid ) = 0, where Zmid = 1/2(Zmin + Zmax ), simply by adding this as an equation in the linear system. The meaning of this constraint is that a pixel with value midway between Zmid and Zmid will be assumed to have unit exposure. Second, the solution can be made to have a much better fit by anticipating the basic shape of the response function. Since g(z) will typically have a steep slope near Zmin and Zmax we should expect that g(z) will be less smooth and will fit the data more poorly near the extremes [3]. To recognise this, we need to introduce a weight function w(z) to emphasize the smoothness and fitting terms towards the middle of the curve. A sensible choice of w is a simple hat function: ⎧ ⎪ ⎨

z − Zmin

f or

z ≤ 12 (Zmin + Zmax )

⎩

Zmax − z

f or

z ≥ 12 (Zmin + Zmax )

w(z) = ⎪

12

(6)

4.1 Recovering High Dynamic Range Radiance Maps from Photographs

Thus equation (5) now becomes: o=

N  P 

[w(Zij )[g(Zij ) − ln Ei − ln Δtj ]]2 + λ

i=1 j=1

z=Z max −1

[w(z)g  (z)]2

(7)

z=Zmin +1

Finally, we need not use every available pixel site in this solution procedure. Given measurements of N pixels in P photographs, we have to solve for N values of ln Ei and (Zmax − Zmin ) samples of g. To ensure a sufficiently over-determined system, we want N(P − 1) > (Zmax − Zmin ). For the pixel value range (Zmax − Zmin ) = 255,P = 11 photographs, a choice of N on the order of 50 pixels is more than adequate. Since the size of the system of linear equations arising from Equation (5) is on the order of N × P + (Zmax − Zmin ), computational complexity considerations make it impractical to use every pixel location in this algorithm. The pixels are best sampled from regions of the image with low intensity variance so that radiance can be assumed to be constant across the area of the pixel, and the effect of optical blur of the imaging system is minimized. We can represent this idea in a graphical manner as below:

Figure 10: In the figure, the symbols represent samples of the g curve derived from the digital values at one pixel for 5 different known exposures using Equation(4). The + and o symbols show samples of g curve segments derived by consideration of two other pixels.

Construction of High Dynamic Range Radiance Map Once the response curve g is recovered, it can be used to quickly convert pixel values to relative radiance values, assuming the exposure Δtj is known.

13

4.1 Recovering High Dynamic Range Radiance Maps from Photographs

Figure 11: The unknown log irradiance ln Ei has been arbitrarily assumed to be 0. Its worth noting that the shape of the g curve is correct, though its position on the vertical scale is arbitrary corresponding to the unknown ln Ei ; again the vertical position of each segment is arbitrary. Essentially, what we want to achieve in the optimization process is to slide the 3 sampled curve segments up and down (by adjusting their ln Ei s) until they line up into a single smooth, monotonic curve, as shown in the figure. The vertical position of the composite curve will remain arbitrary. From equation (4), we obtain: ln Ei = g(Zij ) − ln Δtj

(8)

For robustness, and to recover high dynamic range radiance values, we should use all the available exposures for a particular pixel to compute its radiance. For this, we reuse the weighting function in Equation (6) to give higher weight to exposures in which the pixels value is closer to the middle of the response function: p 

ln Ei =

j=1

w(Zij )(g(Zij ) − ln Δtj ) p  j=1

w(Zij )

(9)

Combining the multiple exposures has the effect of reducing noise in the recovered radiance values. It also reduces the effects of imaging artefacts such as film grain. Since

14

4.2 A Simple Spatial Tone Mapping Operator for High Dynamic Range Images the weighting function ignores saturated pixel values, blooming11 artefacts have little impact on the reconstructed radiance values. Implementation The heart of the algorithm is the optimization routine that needs to be implemented effectively. The implementation is given in [3].

4.2

A Simple Spatial Tone Mapping Operator for High Dynamic Range Images

After obtaining the HDR scene values, the next task was to tone map these values onto a LDR matrix for being able to see it on the screen. This project presents a simple and effective tone mapping operator, which preserves visibility and contrast impression of high dynamic range images. The method is conceptually simple, and easy to use. It uses a s − f unction type operator which takes into account both the global average of the image, as well as local luminance in the immediate neighbourhood of each pixel. The local luminance is computed using a median filter. It is seen that the resulting low dynamic range image preserves fine details, and avoids common artefacts such as halos, gradient reversals or loss of local contrast. 4.2.1

The Need for Tone Mapping

The real world scenes often have a very high range of luminance values. While digital imaging technology now enables us to capture full dynamic range of the real world scene, still we are limited by the low dynamic range displays. Thus the scene can be visualized on a display monitor only after the captured high dynamic range is compressed to available range of the display device. This has been referred to as the tone mapping problem in the literature and a great deal of work has been done in this area by using a mapping that varies spatially depending on the neighbourhood of a pixel, often at multiple scales [8];[5];[9]. In this paper [2] the authors have proposed a simple tone mapping operator which allows us to preserve the visual content of the real-world scene without the user having to manually set a number of parameters. We show that by using a log of relative luminance at a pixel with respect to its local luminance in a small neighbourhood, the standard s − f unction can be modified to yield visually pleasing results. This project 11

Blooming occurs when charge or light at highly saturated sites on the imaging surface spills over and affects values at neighboring sites.

15

4.2 A Simple Spatial Tone Mapping Operator for High Dynamic Range Images

computes the local luminance using a median filter, which provides a stronger central indicator than the mean filter. 4.2.2

The Algorithm

The global contrast helps us to differentiate between various regions of the HDR image, which can be loosely classified as dark, dim, lighted,, bright etc. Within each region objects become distinguishable due to local contrast against the background- either the object is darker than the background or it is brighter than the background. If the HDR image consisted of only regions of uniform illuminations, the following s − f unction would compress the range of illumination across the image, to displayable luminance Y D in range 0–1. Y D(x, y) = Y (x, y)/[Y (x, y) + GC]

(10)

where GC is the global contrast factor computed through GC = cY A

(11)

where Y A is the average luminance value of the whole image and c is a multiplying factor. This would have the effect of bringing the high luminance value closer to 1, while the low luminance values would be a fraction closer to zero. The choice of factor c has an effect on bridging the gap between the two ends of the luminance value. While equation (10) is able to provide a global contrast reduction, it does not attempt to preserve local luminance variations within a region, causing many details to be lost. A detail in the image can result from two possible reasons- either the pixel is brighter than its surrounding pixels or the pixel is darker than its surrounding pixels. If luminance Y at a pixel is more than Y L, the local luminance in its immediate neighbourhood, we would like to increase the displayable value Y D at that point, so that it appears brighter against its neighbourhood. On the other hand, if the luminance Y at a point is less than the local luminance Y L in its immediate neighborhood, we would like to reduce the displayable value Y D, so that it appears darker than its neighbourhood. This has been achieve by modifying the s − f unction as follows: Y D(x, y) = Y (x, y)/[Y (x, y) + CL(x, y)]

16

(12)

4.2 A Simple Spatial Tone Mapping Operator for High Dynamic Range Images where CL is the contrast luminance at a point (x, y) obtained by modifying the global contrast factor GC with a term involving the logarithm of ratio of local low-pass filtered value Y L to original luminance value, CL = Y L[log(δ + Y L/y)] + GC

(13)

where δ is a very small value and is included to avoid singularity while doing the log operation. Also Y L is computed using a median filter in a [3 × 3] neighbourhood rather than a mean filter. The median is a stronger central indicator than the average. Cases of Interest: Case 1: Pixel (x, y) belongs to a uniform region, i.e. the Luminance Y and the local luminance Y L have same magnitude. In this case the log value is going to be close to zero and thus the contribution of the first term is negligible. CL will be very close to GC, and the new Y D value using equation (12), is almost same as computed by equation (10). Case 2: Pixel (x, y) is darker than its immediate surrounding neighbourhood, i.e. the magnitude of pixel luminance Y is less than the local luminance Y L. In this case, the first term in equation (13) is positive. The contribution from Y L is moderated by a factor which depends on how relatively high Y L is with respect to Y itself at that pixel. If the difference is large, the first term in equation (13), is significantly large. In any case a positive term adds on to GC, the global contrast value, resulting in an overall reduction in Y D using equation (12).(Compared to Case 1). Case 3: Pixel (x, y) is brighter than its immediate neighborhood, i.e. the magnitude of Y is more than the local luminance Y L Since the denominator is higher in log(Y L/Y ), the first term is going to provide a negative contribution, reducing the value of CL and consequently resulting in a higher value of Y D(Compared to Case 1). For most HDR images, the computed YD can be displayed directly. Implementation The luminance values are obtained from the RGB inputs with the formula: Y = 0.299 × R + 0.587 × G + 0.114 × B 17

(14)

5 Simulation

After computing the YD, the displayable luminance, the new R, G, B values are computed using the formula [4]:

RD = (R/Y )γ ; GD = (G/Y )γ ; BD = (B/Y )γ ; where γ controls the display colour on the monitor. γ = 0.4 is used in this implementation. Some examples of hdr images tonemapped with local and global operators on two different images (image A and B) are shown in figures 12 to 15.

5

Simulation

5.1

Implementation Results— Chair image

The proposed scheme was simulated on the chair image. A series of 5 different exposures of the image was taken as input which are shown in Fig.16.The exposures of the five images were in decreasing order. The darkest image has the least exposure time while the brightest has the maximum. Then the radiance map was found which is the relation that maps the original or natural radiance values of the image to digital radiance values seen on the input images. the graph was plotted between the pixel values of the input image and the log of exposures of the input images as shown in Fig.17. Then the HDR image was generated from the inverse of the found function whish is shown in Fig.18.Then the HDR image was tonemapped (see Fig.19. It was observed that the proposed scheme gave satisfactory results.

5.2

Implementation Results— Sunset image

The proposed scheme was again simulated on the sunset image. Two images one of which was over exposed and the other under exposed, was taken as input which are shown in Fig.20. Then the radiance map was found which is the relation that maps the original or natural radiance values of the image to digital radiance values seen on the input images. the graph was plotted between the pixel values of the input image and the log of exposures 18

5.2 Implementation Results— Sunset image

Figure 12: Hdr image tone mapped with reinhard local operator on image A

Figure 13: Hdr image tone mapped with reinhard global operator on image A

19

5.2 Implementation Results— Sunset image

Figure 14: Hdr image tone mapped with reinhard local operator on image B

Figure 15: Hdr image tone mapped with reinhard global operator on image B

20

6 Applications of HDRI

Figure 16: The input of the HDRI program. Inputimage-¿chair Note the variations in the details of each image. The exposure times are in decreasing order. i.e. the darkest image has the least exposure time while the brightest has the maximum. 3

2

1

Pixel Value Z

0

−1

−2

−3

−4

−5 0

Red Green Blue 50

100

150 log Exposure X

200

250

300

Figure 17: The HDR radiance map. x − axis Represents log Exposure(X) while y − axis represents pixelvalueZ. The three curves represent the Red, Green, and Blue components of the images. of the input images as shown in Fig.21. Then the HDR image was generated from the inverse of the found function and the HDR image was then tonemapped as shown in Fig.22.

6

Applications of HDRI

Today, the main users of HDR imaging devices are specialized professionals working in the film, animation and VR industries. Some applications are listed below. Film - Tools such as HDRShop enables one to convert a series of photographs into a

21

6 Applications of HDRI

Figure 18: The HDR image obtained from the radiance map.

Figure 19: The final tone mapped HDR image (LDR). light probe - a special image that represents the lighting environment in a room. One can then use the light probe to light virtual objects, so that the virtual objects actually appear to be lit by the light from the room. This technique is especially useful for compositing computer graphic objects into images of real scenes. Hollywood films use light maps extensively to blend CGI into a scene. Panoramas - Another use for HDR is in panoramic images. Panoramas often have a wide dynamic range, e.g. one part of the panorama may contain the sun, and another part may be in deep shadow. Online web panoramas constructed from HDR images look much better than non-HDR equivalents. Games - A third use for HDR is in computer games. Recent computer graphics cards

22

6 Applications of HDRI

(a)

(b)

Figure 20: Two of the input images of Sunset. (a) Over exposed image, (b) Under exposed image. 2

1.5

1

Pixel Value Z

0.5

0

−0.5

−1

Red Green Blue

−1.5

−2

−2.5 0

50

100

150 log Exposure X

200

250

300

Figure 21: The HDR radiance map. x − axis Represents log Exposure(X) while y − axis represents pixelvalueZ. The three curves represent the Red, Green, and Blue components of the images. support HDR texture maps. With HDR texture maps, you can render objects using light probes, in real time, yielding much more dynamic and interesting lighting effects. ”High Dynamic Range Lighting Effects” are used in many new high-end games. By determining the response functions of the imaging device, the method presented here allows one to correctly fuse pixel data from photographs taken at different exposure settings. As a result, one can properly photograph outdoor areas with short exposures, and indoor areas with longer exposures, without creating inconsistencies in the data set. Furthermore, knowing the response functions can be helpful in merging photographs taken with different imaging systems, such as video cameras, digital cameras, and film cameras with various film stocks and digitization processes.

23

7 Conclusions

hdr image

tone mapped hdr image

(a)

(b)

Figure 22: The output images of Sunset. (a)HDR image obtained from the radiance map, (b)The final tone mapped HDR image(LDR). Most image processing operations, such as blurring, edge detection, colour correction, and image correspondence, expect pixel values to be proportional to the scene radiance. Because of nonlinear image response, especially at the point of saturation, these operations can produce incorrect results for conventional images. In computer graphics, one common image processing operation is the application of synthetic motion blur to images. It can be shown that by using true radiance maps one can produce significantly more realistic motion blur effects for high dynamic range scenes.

7

Conclusions

HDR-rendering as a whole is a wide subject and there are many approaches to the problem of generating and visualizing HDR images. Some may just approach the problem by trying what looks good, others try the old methods known from the 150 years of history of photography, and yet others try to go deeper in the workings of the human visual system (HVS) in order to find out how the ”biological tone reproduction” works. Regardless of the approach, HDR images are bringing more realistic computer images to the display devices in the future. For example, HDR rendering allows the graphic designer to truly set some part of the image/texture ”very bright”, without the need of manually fixing all of the environment so that the resulting bright portion would truly look bright. With the freshly released new commodity graphics hardware, and the upcoming models in the following couple of years, new levels of real-time graphics may be achieved.

24

References

References [1] High Dynamic Range Imaging. http://en.wikipedia.org/wiki/High Dynamic Range Imaging. [2] K. K. Biswas and S. Pattanaik. A simple spatial tone mapping operator for high dynamic range images. In Thirteenth Color Imaging Conference, New York, NY, USA, 2005. [3] Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 369–378, New York, NY, USA, 1997. ACM Press/Addison-Wesley Publishing Co. [4] Jiang Duan and Guoping Qiu. Fast tone mapping for high dynamic range images. Pattern Recognition, International Conference on, 2:847–850, 2004. [5] Raanan Fattal, Dani Lischinski, and Michael Werman. Gradient domain high dynamic range compression. In SIGGRAPH ’02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 249–256, New York, NY, USA, 2002. ACM. [6] Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing. Pearson Education, second edition, 1992. [7] Rafael C. Gonzalez, Richard E. Woods, and Steven L. Eddins. Digital Image Processing Using MATLAB. Pearson Education, fourth edition, 2008. [8] Sumanta N. Pattanaik, James A. Ferwerda, Mark D. Fairchild, and Donald P. Greenberg. A multiscale model of adaptation and spatial vision for realistic image display. In SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 287–298, New York, NY, USA, 1998. ACM. [9] Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. In SIGGRAPH ’02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 267–276, New York, NY, USA, 2002. ACM.

25