High Dynamic Range Imaging

HIGH DYNAMIC RANGE IMAGING A PROJECT REPORT By

Pratyush Pandab1 Under The Guidance Of

Prof. B. Majhi Professor and Head of Dept. Computer Science and Engineering, NIT Rourkela

Department of Computer Science Engineering NATIONAL INSTITUTE OF TECHNOLOGY ROURKELA ROURKELA-769008 (ORISSA)

1

Pratyush Pandab is a 4 th year B. Tech student from Dept. of Computer Science and Engineering at CET, BBSR.. 1

National Institute of Technology Rourkela Certificate This is to certify that the work in this Project entitled “High Dynamic Range Imaging” by Pratyush Pandab has been carried out under my supervision during the Summer Internship 2009 at NIT RKL in the Department of Computer Science and Engineering.

Place: Rourkela Date: July 15, 2009

Prof. Banshidhar Majhi Professor and Head of Dept. of CSE NIT, Rourkela.

2

Declaration by the Candidate I hereby declare that the project report entitled “High Dynamic Range Imaging” is a record of bonafide project work carried out by me under the guidance of Prof. B. Majhi of NIT- Rourkela, Orissa.

Pratyush Pandab Dept. of CSE, CET Bhubaneswar.

3

Table of Contents Abstract ......................................................................................... 1. Introduction to HDRI ........................................................ 1.1. A Brief History of HDRI 1.2. Traditional Techniques used 1.3. Recent Technology 1.4. Tone Mapping

5 6 6 7 7 8

2. Problem Statement .............................................................. 9 3. Sample Example .................................................................. 10 4. Background Research .......................................................... 4.1. Recovering High Dynamic Range Radiance Map from Photographs 4.1.1. The Algorithm 4.1.1.1. Film Response Recovery 4.1.1.2. Construction of High Dynamic Range Radiance Map. 4.1.2. Implementation 4.2. A Simple Spatial Tone Mapping Operator for High Dynamic Range Images. 4.2.1. The Need for Tone Mapping. 4.2.2. The Algorithm 4.2.2.1. Cases of Interest 4.2.3. Implementation

11 11-12 12 13-16 17 17-18 19 19 19-20 20-21 21-22

5. Implementation Results ...................................................... 23-26 6. Application of HDRI .......................................................... 27 7. Conclusions .......................................................................... 28 8. References ............................................................................ 28.

4

Abstract In image processing, computer graphics, and photography, High Dynamic Range Imaging (HDRI) is a set of techniques that allows a greater dynamic range of luminance between light and dark areas of a scene than normal digital imaging techniques. The intention of HDRI is to accurately represent the wide range of intensity levels found in real scenes ranging from direct sunlight to shadows.

Information stored in high dynamic range images usually corresponds to the physical values of luminance or radiance that can be observed in the real world. This is different from traditional digital images, which represent colors that should appear on a monitor or a paper print.

One problem with HDR has always been in viewing the images. Typical computer monitors (CRTs, LCDs), prints, and other methods of displaying images only have a limited dynamic range. Various methods of converting HDR images into a viewable format have been developed, generally called Tone Mapping.

Recent methods have attempted to compress the dynamic range into one reproducible by the intended display device. The more complex methods tap into research on how the human eye and visual cortex perceive a scene, trying to show the whole dynamic range while retaining realistic color and contrast.

Figure 1: Here the dynamic range of the image is demonstrated by adjusting the "exposure" when tone mapping the HDR image into an LDR one for display.

In this project, it is considered that nearly any camera can actually capture a vast dynamic range-- just not in a single photo. By varying the shutter speed alone, most digital cameras can change how much light they let in by a factor of 50,000 or more. This project attempts to combine these captured images into a single High Dynamic Range Image by utilizing the different exposures characterized by these images composed of multiple exposures. The difficulties that accompany HDRI are Halo effects and artifacts. Care has been taken to avoid these pitfalls. The complete project was simulated using Matlab 7.6 software. The simulation results are shown as figures. Results have shown very high quality rendering of images by this technique.

5

1. Introduction to HDRI High Dynamic

Range Imaging (HDRI) is a set of techniques that allows a greater dynamic range of luminance between light and dark areas of a scene than normal digital imaging techniques. High dynamic range (HDR) images enable photographers to record a greater range of tonal detail than a given camera could capture in a single photo. This opens up a whole new set of lighting possibilities which one might have previously avoided. The intention of HDRI is to accurately represent the wide range of intensity levels found in real scenes ranging from direct sunlight to shadows.

1.1. A Brief History of HDRI In the

past, photographers dealt with the limitation of dynamic range by experimenting with media. Ansel Adams was perhaps the first to systematically measure the sensitivity range of all of the equipment he used to precisely indicate what the photograph would display depending on the length of time the shutter was open. Before HDR was put into practice, photographers and film makers needed to find ways around the limitations of modern digital imaging which fail to capture the total perceptual display that the human eye is capable of seeing. The problem is that real-world scenes contain light ranges that exceed a 50,000:1 dynamic range2 while digital equipment is limited to around 300:1 dynamic range.

Also,

through Adams' work it was discovered that one way to get really great dynamic range with colour photography is to use black-and-white film and colour filters. Although this was a practical solution, as technology advanced so did the digital imaging market, thus creating HDR.

Images tend to result in the subject seen and the surrounding blanked out.

Figure 2: In the first image the shelf, the table and the sofa set are the subject, whereas the window is the surrounding. Not much is seen outside the window, but the details on shelf and the table are seen pretty clearly. In the second im age the surrounding is clear enough but nothing is visible inside the room. This is the problem caused due to the dynamic r ange of the capturing equipment. 2

Dynam ic Range: The dynami c range is the ra tio between the ma ximum and mi nimum values of a physi cal measurement. Its definition depends on wha t the dynami c range refers to.

6

1. 2 Traditional Techniques Used Most

photographic systems struggle on high contrasts scenes. With the entire modern technology one would be wrong to think a digital camera would resolve the issues related to dynamic range. The reason for this is simple, current film and image sensors are developed to record enough light that can be reproduced on paper or compute r screen and neither are currently capable of displaying a dynamic range that we can see with our eyes. Until recently, to overcome this, digital photographers have either been using the old approach of ND graduated filters at the taking stage or combining the best bits of two different exposures on the computer, much like the dodge & burn 3 technique used by experienced darkroom users.

1. 3 Recent Technology New technology is on the way to record and display at higher dynamic ranges, but for now there are a range of software solutions.

The job of HDR processing is to allow photographers to produce images with a much larger dynamic range. The photographer takes a series of different exposures of the same scene and then uses HDR software to automatically merge the resulting images into one image, referred to as exposure blending. This produces a file with the entire desired dynamic range, known as a radiance map. Ideally the fewer images used the better, because it's less memory intensive and there's less chance of subject been out of alignment on each shot as a result of camera or subject movement.

HDR uses 16 to 32 bits values. This jump in resolution allows for a much greater range of colours, greater than the monitor can display at once. As a result, to view these images, the computer breaks the images into levels of brightness and darkness. The brightest part of the image are viewed with the brightest colours the monitor can display the darkest are viewed with the darkest the screen can display. This allows for each image to display details, bright highlights and dark shadows.

3

Dodging and Burning are terms used in photography for a technique used during the printing process , to manipulate the exposure of a selected area(s) on a photographic print, deviating from the rest of the image's exposure. Dodging decreases the exposure for areas of the print that the photographer wishes to be lighter, while burning increases the exposure to areas of the print that should be darker.

7

1. 4 Tone Mapping The above

mentioned merged photographs are saved as 32-bit. This means there's very little one can do to edit the photos, they cannot be saved as JPG without reducing the bitdepth 4 and one cannot view the superb dynamic range on a conventional monitor. In such cases, the whole dynamic range of the scene (e.g. 100,000:1 or higher) has to be reproduced on a medium with less than 100:1 contrast ratio. Tone mapping deals with the issue of reproducing the dynamic range captured. This is why it becomes necessary to use techniques that scale the dynamic range down while preserving the appearance of the original image captured. One way around this is to apply tone mapping which converts the HDR image into a balanced exposure.

Tone mapping operators are divided into two broad categories, Global and Local Global operators are simple and fast. They map each pixel based on its intensity and global image characteristics, regardless of the pixel's spatial location. Global operators are OK for mapping 12-bit sensor data but usually don't work well with HDR images. An example of a global type of tone mapping is a tonal curve.

Local operators take into account the pixel's surroundings for mapping it. This means that a pixel of a given intensity will be mapped to a different value depending on whether it is located in a dark or bright area. This makes local operators slower but tends to produce more pleasing results, given that our eyes react locally to contrast.

4

Bit depth of a ca pturing or displa ying devi ce gi ves an indi ca tion of i ts dynami c range capa city, i .e. the hi ghes t dyna mi c range tha t the devi ce would be capable of reproducing.

8

2. Problem Statement

Here is the problem in a nutshell: a. Real-world scenes contain light ranges that exceed a 50,000:1 dynamic range. b. Media has been limited to around a 300:1 dynamic range. c.

So we have a mapping issue: how do we represent light values in a scene using a

much more limited set of light values for a particular media? d. What if the set of images with different exposures have unknown noise in it? Can we get the HDR image sans the noise (noise gets added, so it has to be removed.).

9

3. Sample Example

Figure 3 : Sample input set of im ages with different exposure time.

Figure 4: Output; tone mapped HDR im age. (Note the details present in this single image.)

10

4. Background Research The method of recovering high dynamic range radiance maps from photographs taken with conventional imaging equipments [𝐼] is presented in this project work. The proposed algorithm by Debevec & Malik [𝐼] uses these differently exposed photographs to recover the response function of the imaging process, up to factor of scale, using the assumption of reciprocity. With the known response function, the algorithm can fuse the multiple photographs into a single, high dynamic range radiance map whose pixel values are proportional to the true radiance values in the scene.

Besides

this, the project also implements a simple and effective tone mapping operator proposed by Biswas & Pattanaik [𝐼𝐼], that preserves visibility and contrast impression of high dynamic range images. It use a s-function type operator which takes into account both the global average of the image, as well as local luminance in the immediate neighbourhood of each pixel. The local luminance is computed using a median filter. It is seen that the resulting low dynamic range image preserves fine details, and avoids common artefacts such as halos, gradient reversals or loss of local contrast.

4.1. Recovering High Dynamic Range Radiance Maps from Photographs [𝐼] Digitized photographs

are becoming increasingly important in computer graphics. When one photographs a scene, either with film or an electronic imaging array, and digitizes the photograph to obtain a two dimensional array of brightness 5 values, these values are rarely true measurements of relative radiance in the scene. For example, if one pixel has twice the value of another, it is unlikely that it observed twice the radiance 6. Instead, there is usually an unknown, nonlinear mapping that determines how radiance in the scene becomes pixel values in the image. This nonlinear mapping is hard to know beforehand because it is actually the composition of several nonlinear mappings that occur in the photographic process.

In a conventional camera (see Fig.5), the film is first exposed to light to form a latent image. The film is then developed to change this latent image into variations in transparency, or density, on the film. The film can then be digitized using a film scanner, which projects light through the film onto an electronic light-sensitive array, converting the image to electrical voltages. These voltages are digitized, and then manipulated before finally being written to the storage medium. If prints of the film are scanned rather than the film itself, then the printing process can also introduce nonlinear mappings.

5 6

Brightness values represent the gra y levels of the ima ge. Radiance is the i rradiance/l uminance value of the object a t the si te of the s cene.

11

Lens

Scene radiance

Shutter

Film

𝜠

𝑿

Sensor irradiance

Sensor exposure

Development

CCD

ADC

Remapping

𝜡 Latent image

Film density

Digital camera

Analog voltage

Digital values

Final digital values

Figure 5 : Image Acquisition Pipeline: shows how scene radiance becomes pixel values for both film and digital cameras. Unknown nonline ar mappings can occur during exposure, de velopment, scanning, digitization, and remapping.

In the first stage of the process, the film response to variations

in exposure 𝑿7 is a nonlinear function, called the “characteristic curve” of the film. The development, scanning and digitization processes usually introduce their own nonlinearities which compose to give the aggregate nonlinear relationship between the image pixel exposures 𝑿 and their values 𝒁8.

As

with film, the most the most significant nonlinearity in the response curve is at its saturation point, where any pixel with radiance above a certain level is mapped to the same maximum image value. The most obvious difficulty is that of limited dynamic range—one has to choose the range of radiance values that are of interest and determine the exposure time suitably. Sunlit scenes, and scenes with shiny materials and artificial light sources, often have extreme differences in radiance values that are impossible to capture without either under-exposing or saturating the film. To cover the full dynamic range in such a scene, one can take a series of photographs with different exposures. This then poses a problem: how can we combine these separate images into a composite radiance map? Here the fact that the mapping from scene radiance to pixel values is unknown and nonlinearity begins to haunt us.

A

simple technique for recovering this response function, up to a scale factor, using nothing more than a set of photographs taken with varying, known exposure durations has been the proposed work by Debevec & Malik [𝐼] and is simulated in this project. With this mapping, the pixel values from all the available photographs are used to construct an accurate map of the radiance in the scene, up to a factor of scale. This radiance map will cover the entire dynamic range captured by the original photographs.

4.1.1. The Algorithm This section presents

the algorithm cited in [𝐼], for recovering the film response function, and then presents the method of reconstructing the high dynamic range radiance image from the multiple photographs. The algorithm assumes gray scale images, but can be extended to colour images with proper conversions 9. 7 8

𝑿 = 𝑬𝜟𝒕 ; i.e. 𝑿 is the product of the i rra diance 𝑬 the film recei ves and the exposure time 𝜟𝒕. 𝒁 is the final di gi tal value after processing. Refer fig. 5.

9

We ca n convert RGB images to YCbCr i mages and consider onl y the Y (luminance) component or else we can consider ea ch component indi vidually and plot the mapping.

12

4.1.1.1. Film Response Recovery This

algorithm is based on exploiting a physical property of imaging systems, both photochemical and electronic, known as reciprocity.

Let us consider photographic film first. The response of a film

to variations in exposure is summarized by the characteristic curve (or Hurter-Driffield curve) [𝐼]. This is a graph of the optical density 𝑫 of the processed film against the logarithm of the exposure 𝑿 to which it has been subjected. The exposure 𝑿 is defined as the product of the irradiance 𝑬 at the film and exposure time, 𝜟𝒕, so that its units are 𝑱𝒎−2 . Key to the very concept of the characteristic curve is the assumption that only the product 𝑬𝜟𝒕 is important, and that halving 𝑬 and doubling 𝜟𝒕 will not change the resulting optical density 𝑫. Under extreme conditions (very large or very low 𝜟𝒕 ), the reciprocity assumption can break down, a 1 situation described as reciprocity failure. In typical print films, reciprocity holds to within 3 1

stop10 for exposure times of 10 seconds to 10000 of a second.

After the development, scanning and digitization processes, we obtain a digital number 𝒁, which is a nonlinear function of the original exposure 𝑿 at the pixel. Let us call this function 𝒇, which is the composition of the characteristic curve of the film as well as all the nonlinearities introduced by the later processing steps. The first goal is to recover this function 𝒇. Once we have that, we can compute the exposure 𝑿 at each pixel, as 𝑿 = 𝒇−𝟏 (𝒁). A reasonable assumption made is that the function 𝒇 is monotonically increasing [𝐼], so its inverse 𝒇−𝟏 is well defined. Knowing the exposure 𝑿 and the exposure time 𝜟𝒕, the irradiance 𝑬 is recovered as 𝑬 = 𝑿/𝜟𝒕, which will be taken to be proportional to the radiance 𝑳 in the scene.

The

input to the algorithm is a number of digitized photographs taken from the same vantage point with different known exposure durations 𝜟𝒕𝒋 .11 It is assumed that the scene is static and that this process is completed quickly enough that lighting changes can be safely ignored. It can then be assumed that the film irradiance values 𝑬 𝒊 for each pixel 𝒊 are constant. Pixel values are denoted by 𝒁𝒊𝒋 where 𝒊 is a spatial index over pixels and 𝒋 indexes over exposure times 𝜟𝒕𝒋 . Thus the film reciprocity equation is:

𝒁𝒊𝒋 = 𝒇(𝑬𝒊 𝜟𝒕𝒋 )

(𝟏)

Since it is assumed 𝒇 is monotonic, it is invertible, and so we can rewrite (𝟏) as: 𝒇−𝟏 (𝒁𝒊𝒋 ) = 𝑬𝒊 𝜟𝒕𝒋

(𝟐)

Taking the natural logarithm of both sides of (𝟐) , we have: 𝐥𝐧 𝒇−𝟏 (𝒁𝒊𝒋 ) = 𝐥𝐧 𝑬𝒊 + 𝐥𝐧 𝜟𝒕𝒋 10 11

1

(𝟑)

1

1 stop is a photographi c term for a fa ctor of two ; 3 s top is thus 23 .

1

1

1

1 1 1

The a ctual exposure ti mes a re va ried by powers of two between s tops (64 , 32 , 16 , 8 , 4 , 2, 1, 2, 4, 8, 16, 32), ra ther

than the rounded numbers displa yed on the camera

1 1 1 1 1 1 readout (60 , 30 , 15 , 8 , 4 , 2, 1,

2, 4, 8, 15, 30).

13

simplify notation, let us define a function, 𝒈 = 𝐥𝐧 𝒇−𝟏 . We then have the set of equations:

To

𝒈(𝒁𝒊𝒋 ) = 𝐥𝐧 𝑬𝒊 + 𝐥𝐧 𝜟𝒕𝒋

(𝟒)

where 𝒊 ranges over pixels and 𝒋 ranges over exposure durations. In this set of equations, the 𝒁𝒊𝒋 are known, along with their corresponding 𝜟𝒕𝒋 . The unknowns are the irradiances 𝑬 𝒊 as well as the function 𝒈, although it is assumed that 𝒈 is smooth and monotonic.

Our aim now is to recover the function 𝒈 and the irradiances

𝑬 𝒊 that best satisfy the set of equations arising from Equation 𝟒 in a least-squared error sense. It can be verified that recovering 𝒈 only requires recovering the finite number of values that 𝒈(𝒛) can take since the domain of 𝒁, pixel brightness values, is finite. Let 𝒁𝒎𝒊𝒏 and 𝒁𝒎𝒂𝒙 be the least and greatest pixel values (integers), 𝑵 be the number of pixel locations, and 𝑷 be the number of photographs.

Thus the existing problem can be formulated as one of finding the (𝒁𝒎𝒊𝒏 −

𝒁𝒎𝒂𝒙 + 𝟏) values of 𝒈(𝒁) and the 𝑵 values of 𝐥𝐧 𝑬 𝒊 that minimize the following quadratic objective function: 𝑵

𝑷

𝓞=

𝒈 𝒁𝒊𝒋 − 𝐥𝐧 𝑬𝒊 − 𝐥𝐧 𝜟𝒕𝒋 𝒊=𝟏 𝒋=𝟏

𝟐

𝒛=𝒁𝒎𝒂𝒙 −𝟏

𝒈′′ 𝒛

+𝝀

𝟐

𝒛=𝒁𝒎𝒊𝒏 +𝟏

(𝟓)

The first term ensures that the solution satisfies the set of equations

arising from Equation (𝟒) in a least squares sense. The second term is a smoothness term on the sum of squared values of the second derivative of 𝒈 to ensure that the function 𝒈 is smooth. In this discrete setting 𝒈′′ 𝒛 = 𝒈 𝒛 − 𝟏 − 𝟐𝒈 𝒛 + 𝒈(𝒛 + 𝟏) is used. This smoothness term is essential to the formulation in that it provides coupling between the values 𝒈(𝒛) in the minimization. The scalar 𝝀 weights the smoothness term relative to the data fitting term, and should be chosen appropriately for the amount of noise expected in the 𝒁𝒊𝒋 measurements. 𝒈(𝒛)’s, minimizing 𝓞 is a straightforward linear least squares problem. The over-determined system of linear equations is robustly solved using the singular value decomposition (SVD) method.

Because it is quadratic in the 𝑬𝒊’s and

Along with this equation some addition points need to be considered. First, the solution for the 𝒈(𝒛) and 𝑬𝒊 values can only be up to a single factor 𝜶 ;[𝐼]. If each log irradiance value 𝐥𝐧 𝑬 𝒊 were replaced by 𝐥𝐧 𝑬 𝒊 + 𝜶, and the function 𝒈 replaced by

𝒈 + 𝜶, the system of equation (4) and also the objective function 𝓞 would remain unchanged. To establish a scale factor we [𝐼] introduced the additional constraint 𝟏 𝒁𝒎𝒊𝒅 = 𝟎, where 𝒁𝒎𝒊𝒅 = 𝟐 (𝒁𝒎𝒊𝒏 + 𝒁𝒎𝒂𝒙 ), simply by adding this as an equation in the 14

linear system. The meaning of this constraint is that a pixel with value midway between 𝒁𝒎𝒊𝒅 and 𝒁𝒎𝒊𝒅 will be assumed to have unit exposure.

Second, the solution can be made to have a much better fit by anticipating the basic shape of the response function. Since 𝒈(𝒛) will typically have a steep slope near 𝒁𝒎𝒊𝒏 𝒂𝒏𝒅 𝒁𝒎𝒂𝒙 we should expect that 𝒈(𝒛) will be less smooth and will fit the data more poorly near the extremes;[𝐼]. To recognise this, we need to introduce a weight function 𝒘(𝒛) to emphasize the smoothness and fitting terms towards the middle of the curve. A sensible choice of 𝒘 is a simple hat function:

𝒘 𝒛 =

𝒛 − 𝒁𝒎𝒊𝒏 𝒁𝒎𝒂𝒙 − 𝒛

𝟏 (𝒁 + 𝒁𝒎𝒂𝒙 ) 𝟐 𝒎𝒊𝒏 𝟏 𝑓𝑜𝑟 𝒛 > (𝒁𝒎𝒊𝒏 + 𝒁𝒎𝒂𝒙 ) 𝟐 𝑓𝑜𝑟 𝒛 ≤

(𝟔)

Thus equation (𝟓) now becomes: 𝑵

𝑷

𝓞=

𝒘(𝒁𝒊𝒋) 𝒈 𝒁𝒊𝒋 − 𝐥𝐧 𝑬𝒊 − 𝐥𝐧 𝜟𝒕𝒋

𝟐

𝒛=𝒁𝒎𝒂𝒙 −𝟏

𝒘 𝒛 𝒈′′ (𝒛)

+𝝀

𝒊=𝟏 𝒋=𝟏

𝟐

𝒛=𝒁𝒎𝒊𝒏 +𝟏

Finally,

we need not use every available pixel site in this solution procedure. Given measurements of 𝑵 pixels in 𝑷 photographs, we have to solve for 𝑵 values of 𝐥𝐧 𝑬 𝒊 and (𝒁𝒎𝒂𝒙 − 𝒁𝒎𝒊𝒏 ) samples of 𝒈. To ensure a sufficiently over-determined system, we want 𝑵(𝑷 − 𝟏) > (𝒁𝒎𝒂𝒙 − 𝒁𝒎𝒊𝒏 ). For the pixel value range (𝒁𝒎𝒂𝒙 − 𝒁𝒎𝒊𝒏 ) = 𝟐𝟓𝟓, 𝑷 = 𝟏𝟏 photographs, a choice of 𝑵 on the order of 50 pixels is more than adequate. Since the size of the system of linear equations arising from Equation (𝟓) is on the order of 𝑵 × 𝑷 + (𝒁𝒎𝒂𝒙 − 𝒁𝒎𝒊𝒏 ), computational complexity considerations make it impractical to use every pixel location in this algorithm. The pixels are best sampled from regions of the image with low intensity variance so that radiance can be assumed to be constant across the area of the pixel, and the effect of optical blur of the imaging system is minimized.

A Matlab routine is given to show the simplicity of the evaluation of the equation (𝟒). We can represent this idea in a graphical manner as below:

15

Figure 6: In the figure on the 𝑡𝑜𝑝 𝑙𝑒𝑓𝑡, the × symbols represent samples of the

𝒈 curve derived from the digital values at one pixel for 5 different known exposures using Equation(𝟒). The + and ο symbols show samples of 𝒈 curve segments derived by consideration of two other pixels. (𝑏𝑜𝑡𝑡𝑜𝑚 𝑓𝑖𝑔𝑢𝑟𝑒)The unknown log irradiance 𝒍𝒏𝑬𝒊 has been arbitrarily assumed to be 0. It’s worth noting that the shape of the 𝒈 curve is correct, though its position on the vertical scale is arbitrary corresponding to the unk nown 𝒍𝒏𝑬𝒊.; again the vertical position of each segment is arbitrary. Essentially, what we want to achieve in the optimization process is to slide the 3 sampled curve segments up and down (by adjusting their 𝒍𝒏𝑬𝒊’s) until they “line up” into a single smooth, monotonic curve, as shown in the 𝑏𝑜𝑡𝑡𝑜𝑚 𝑓𝑖𝑔𝑢𝑟𝑒. The vertical position of the composite curve will remain arbitrary.

16

4.1.1.2. Construction of High Dynamic Range Radiance Map Once the response curve 𝒈 is

recovered, it can be used to quickly convert pixel values to relative radiance values, assuming the exposure 𝜟𝒕𝒋 is known.

From equation (𝟒), we obtain: 𝐥𝐧 𝑬𝒊 = 𝒈 𝒁𝒊𝒋 − 𝐥𝐧 𝜟𝒕𝒋

(𝟕)

For robustness, and to recover high dynamic range radiance values, we should use all the available exposures for a particular pixel to compute its radiance. For this, we reuse the weighting function in Equation (𝟔) to give higher weight to exposures in which the pixel’s value is closer to the middle of the response function:

𝐥𝐧 𝑬𝒊 =

𝑷 𝒋=𝟏 𝒘

𝒁𝒊𝒋 (𝒈 𝒁𝒊𝒋 − 𝐥𝐧 𝜟𝒕𝒋 ) 𝑷 𝒋=𝟏 𝒘

𝒁𝒊𝒋

(𝟖)

Combining

the multiple exposures has the effect of reducing noise in the recovered radiance values. It also reduces the effects of imaging artefacts such as film grain. Since the weighting function ignores saturated pixel values, blooming 12 artefacts have little impact on the reconstructed radiance values.

4.1.2. Implementation The

heart of the algorithm is the optimization routine that needs to be implemented effectively. The implementation is given in [𝐼]. The Matlab routine is give below:

12

Blooming occurs when charge or light a t highl y satura ted si tes on the imaging surface spills over a nd affects values at nei ghboring si tes.

17

18

4.2. A Simple Spatial Tone Mapping Operator for High Dynamic Range Images [II] After obtaining the HDR scene values, the next task was to tone map these values onto a LDR matrix for being able to see it on the screen. This project presents a simple and effective tone mapping operator, which preserves visibility and contrast impression of high dynamic range images. The method is conceptually simple, and easy to use. It uses a 𝑠 − 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 type operator which takes into account both the global average of the image, as well as local luminance in the immediate neighbourhood of each pixel. The local luminance is computed using a median filter. It is seen that the resulting low dynamic range image preserves fine details, and avoids common artefacts such as halos, gradient reversals or loss of local contrast.

4.2.1. The Need for Tone Mapping The

real world scenes often have a very high range of luminance values. While digital imaging technology now enables us to capture full dynamic range of the real world scene, still we are limited by the low dynamic range displays. Thus the scene can be visualized on a display monitor only after the captured high dynamic range is compressed to available range of the display device. This has been referred to as the tone mapping problem in the literature and a great deal of work has been done in this area by using a mapping that varies spatially depending on the neighbourhood of a pixel, often at multiple scales [𝐼𝐼𝐼];[𝐼𝑉]; 𝑉 . In this paper[𝐼𝐼] the authors have proposed a simple tone mapping operator which allows us to preserve the visual content of the real-world scene without the user having to manually set a number of parameters. We show that by using a log of relative luminance at a pixel with respect to its local luminance in a small neighbourhood, the standard 𝑠 − 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 can be modified to yield visually pleasing results. This project computes the local luminance using a median filter, which provides a stronger central indicator than the mean filter.

4.2.2. The Algorithm The global contrast helps

us to differentiate between various regions of the HDR image, which can be loosely classified as dark, dim, lighted,, bright etc. Within each region objects become distinguishable due to local contrast against the background − either the object is darker than the background or it is brighter than the background.

If

the HDR image consisted of only regions of uniform illuminations, the following 𝑠 − 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 would compress the range of illumination across the image, to displayable luminance 𝒀𝑫 in range 0 − 1. 19

𝒀𝑫(𝒙, 𝒚) = 𝒀(𝒙, 𝒚)/[𝒀(𝒙, 𝒚) + 𝑮𝑪]

(𝟗)

where 𝑮𝑪 is the global contrast factor computed through

𝑮𝑪 = 𝒄 𝒀𝑨

(𝟏𝟎)

where 𝒀𝑨 is the average luminance value of the whole image and 𝒄 is a multiplying factor. This would have the effect of bringing the high luminance value closer to 1, while the low luminance values would be a fraction closer to zero. The choice of factor 𝒄 has an effect on bridging the gap between the two ends of the luminance value.

While equation (𝟗) is able to provide a global contrast reduction, it does

not attempt to preserve local luminance variations within a region, causing many details to be lost.

A detail in the image can result from two possible reasons− either the pixel is brighter than its surrounding pixels or the pixel is darker than its surrounding pixels. If luminance 𝒀 at a pixel is more than 𝒀𝑳, the local luminance in its immediate neighbourhood, we would like to increase the displayable value 𝒀𝑫 at that point, so that it appears brighter against its neighbourhood. On the other hand, if the luminance 𝒀 at a point is less than the local luminance 𝒀𝑳 in its immediate neighborhood, we would like to reduce the displayable value 𝒀𝑫, so that it appears darker than its neighbourhood. This has been achieve by modifying the 𝑠 − 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 as follows:

𝒀𝑫(𝒙, 𝒚) = 𝒀(𝒙, 𝒚)/[𝒀(𝒙, 𝒚) + 𝑪𝑳(𝒙, 𝒚)]

(𝟏𝟏)

where 𝑪𝑳 is the contrast luminance at a point (𝑥, 𝑦) obtained by modifying the global contrast factor 𝑮𝑪 with a term involving the logarithm of ratio of local low-pass filtered value 𝒀𝑳 to original luminance value,

𝑪𝑳 = 𝒀𝑳[𝒍𝒐𝒈(𝜹 + 𝒀𝑳/𝒀)] + 𝑮𝑪

(𝟏𝟐)

where 𝜹 is a very small value and is included to avoid singularity while doing the 𝑙𝑜𝑔 operation.

Also 𝒀𝑳 is computed using a median filter in a [3 × 3] neighbourhood rather than a mean filter. The median is a stronger central indicator than the average.

4.2.2.1. Cases of Interest 𝑪𝒂𝒔𝒆 𝟏: Pixel (𝑥, 𝑦) belongs to a uniform region,

i.e. the Luminance 𝒀 and the local luminance 𝒀𝑳 have same magnitude. In this case the log value is going to be close to zero and thus the contribution of the first term is negligible. 𝑪𝑳 will be very close to 𝑮𝑪, and the new 𝒀𝑫 value using equation (𝟏𝟏), is almost same as computed by equation (𝟗).

20

𝑪𝒂𝒔𝒆 𝟐: Pixel (𝑥, 𝑦) is darker than its immediate surrounding neighbourhood, i.e. the magnitude of pixel luminance 𝒀 is less than the local luminance 𝒀𝑳. In this case, the first term in equation (𝟏𝟐) is positive. The contribution from 𝒀𝑳 is moderated by a factor which depends on how relatively high 𝒀𝑳 is with respect to 𝒀 itself at that pixel. If the difference is large, the first term in equation (𝟏𝟐), is significantly large. In any case a positive term adds on to 𝑮𝑪, the global contrast value, resulting in an overall reduction in 𝒀𝑫 using equation (𝟏𝟏). (𝑐𝑜𝑚𝑝𝑎𝑟𝑒𝑑 𝑡𝑜 𝑪𝒂𝒔𝒆 𝟏).

𝑪𝒂𝒔𝒆 𝟑: Pixel (𝑥, 𝑦) is

brighter than its immediate neighborhood, i.e. the magnitude of 𝒀 is more than the local luminance 𝒀𝑳. Since the denominator is higher in 𝒍𝒐𝒈(𝒀𝑳/𝒀), the first term is going to provide a negative contribution, reducing the value of 𝑪𝑳 and consequently resulting in a higher value of 𝒀𝑫 (𝑐𝑜𝑚𝑝𝑎𝑟𝑒𝑑 𝑡𝑜 𝑪𝒂𝒔𝒆 𝟏).

For most HDR images, the computed YD can be displayed directly. 4.2.3. Implementation The luminance values are obtained from the RGB inputs with the formula: 𝒀 = 𝟎. 𝟐𝟗𝟗 × 𝑹 + 𝟎. 𝟓𝟖𝟕 × 𝑮 + 𝟎. 𝟏𝟏𝟒 × 𝑩

After computing the YD, the displayable luminance, the new R, G, B values are computed using the formula [𝑉𝐼]:

𝑹𝑫 = (𝑹/𝒀)𝜸 × 𝒀𝑫; 𝑮𝑫 = (𝑮/𝒀)𝜸 × 𝒀𝑫; 𝑩𝑫 = (𝑩/𝒀)𝜸 × 𝒀𝑫; where 𝜸controls the display colour on the monitor. 𝜸 = 𝟎. 𝟒 is used in this implementation.

The implemented Matlab code is given below.

21

22

5. 1. Implementation Results

Figure 7: The input of the HDRI program. 𝑰𝒏𝒑𝒖𝒕 𝒊𝒎𝒂𝒈𝒆 −> 𝑐𝑕𝑎𝑖𝑟. Note the variations in the details of each image. The exposure times are in decreasing order. i.e. the darkest image has the least exposure time while the brightest has the maximum.

Figure 8: The HDR radiance map. 𝒙 − 𝒂𝒙𝒊𝒔 Represents “𝒍𝒐𝒈 𝑬𝒙𝒑𝒐𝒔𝒖𝒓𝒆 (𝑿)” while 𝒚 − 𝒂𝒙𝒊𝒔 ”𝒑𝒊𝒙𝒆𝒍 𝒗𝒂𝒍𝒖𝒆 𝒁”. The three curves represent the Red, Green, and Blue components of the images.

represents

23

Figure 9: The HDR image obtained from the radiance map.

Figure 10: The final tone mapped HDR image (LDR).

24

5. 2. Implementation Results

Figure 11: The input of the HDRI program. 𝑰𝒏𝒑𝒖𝒕 𝒊𝒎𝒂𝒈𝒆 −> 𝑠𝑢𝑛𝑠𝑒𝑡. It is obvious from the images that the first image is overexposed (high exposure time) while the second one is underexposed (less exposure time).

Figure 12: The HDR radiance map. 𝒙 − 𝒂𝒙𝒊𝒔 Represents “𝒍𝒐𝒈 𝑬𝒙𝒑𝒐𝒔𝒖𝒓𝒆 (𝑿)” while 𝒚 − 𝒂𝒙𝒊𝒔 ”𝒑𝒊𝒙𝒆𝒍 𝒗𝒂𝒍𝒖𝒆 𝒁”. The three curves represent the Red, Green, and Blue components of the images.

represents

25

Figure 13: (𝒍𝒆𝒇𝒕 𝒊𝒎𝒂𝒈𝒆) HDR image obtained from the radiance map. (𝒓𝒊𝒈𝒉𝒕 𝒊𝒎𝒂𝒈𝒆) The final tone mapped HDR image(LDR).

26

5. Applications of HDRI Today, the main users of HDR imaging devices are specialized professionals working in the film, animation and VR industries. Some applications are listed below.

Film - Tools such as HDRShop enables one to convert a series of photographs into a light probe - a special image that represents the lighting environment in a room. One can then use the light probe to light virtual objects, so that the virtual objects actually appear to be lit by the light from the room. This technique is especially useful for compositing computer graphic objects into images of real scenes. Hollywood films use light maps extensively to blend CGI into a scene.

Panoramas

- Another use for HDR is in panoramic images. Panoramas often have a wide dynamic range, e.g. one part of the panorama may contain the sun, and another part may be in deep shadow. Online web panoramas constructed from HDR images look much better than non-HDR equivalents.

Games - A third use for HDR is in computer games. Recent computer graphics cards support HDR texture maps. With HDR texture maps, you can render objects using light probes, in real time, yielding much more dynamic and interesting lighting effects. "High Dynamic Range Lighting Effects" are used in many new high-end games.

By determining the response functions of the imaging device, the method presented here allows one to correctly fuse pixel data from photographs taken at different exposure settings. As a result, one can properly photograph outdoor areas with short exposures, and indoor areas with longer exposures, without creating inconsistencies in the data set. Furthermore, knowing the response functions can be helpful in merging photographs taken with different imaging systems, such as video cameras, digital cameras, and film cameras with various film stocks and digitization processes.

Most image processing operations, such as blurring, edge detection, colour correction, and image correspondence, expect pixel values to be proportional to the scene radiance. Because of nonlinear image response, especially at the point of saturation, these operations can produce incorrect results for conventional images. In computer graphics, one common image processing operation is the application of synthetic motion blur to images . It can be shown that by using true radiance maps one can produce significantly more realistic motion blur effects for high dynamic range scenes.

27

6. Conclusion In this

project a simple implementation of High Dynamic Range Imaging along with tone mapping has been shown. The results achieved are pleasing to the eye and the images are comparable to any of the standard HDRI toolbox available in the market.

7. References [𝐼] Debevec, P. E., & Malik, J. (n.d.). Recovering High Dynamic Range Radiance Maps from Photographs. [𝐼𝐼] Biswas, K. K., & Pattanaik, S. (n.d.). A Simple Spatial Tone Mapping Operator for High Dynamic. [𝐼𝐼𝐼] Pattanaik, S. N., Ferwerda, J. A., Fairchild, M. D., & Greenberg, D. P. (1998). A multiscale model of adaptation and spatial vision for realistic images display. SIGGRAPH,Annual Conference Series,ACM , 267-276. [𝐼𝑉] Fattal, R., Lischinski, D., & Werman, M. (2002). Gradient Domain High Dynamic range compression. SIGGRAPH,Annual Conference series,ACM , 246-249. [𝑉] Reinhard, E., Stark, M., Shirley, P., & Ferwerda, J. (2002). Photographic Tone reproduction for digital images. SIGGRAPH,Annual Conf. Series,ACM , 267-276. [𝑉𝐼] Duan, J., & Qiu, G. (2004). Fast Tone Mapping for high dynamic range images. International Conference on Pattern Recognition, ICPR , 847-850.

28