Fuzzy Image Retrieval

Content Based Image Retrieval using a Neuro-Fuzzy Technique S. Kulkarni1, B. Verma1,2, P. Sharma3 and H. Selvaraj4 1

School of Information Technology Griffith University-Gold Coast Campus PMB 50, Gold Coast Mail Centre, Gold Coast Australia, QLD 9726 Telephone: +61 7 5594 8738 Fax: +61 7 5594 8066 E-mail: {s.kulkarni, b.verma}@gu.edu.au 2

Department of Computer Engineering and Computer Science University of Missouri-Columbia Columbia, MO 65211 E-mail: [email protected]

3

CISCO SYSTEMS 250 Apollo Drive, Chelmsford, MA Telephone: +1 978 2448395 E-mail: [email protected] 4

Department of Electrical and Computer Engineering University of Nevada, Las Vegas 4505 Maryland Parkway, Box 454026 Las Vegas, Nevada 89154-4026 Telephone: +1 702 895 4184 Fax +1 702 895 4075 E-mail: [email protected]

Abstract

1. Introduction

In this paper, we propose a neuro-fuzzy technique for content based image retrieval. The technique is based on fuzzy interpretation of natural language, neural network learning and searching algorithms. Firstly, fuzzy logic is developed to interpret natural expressions such as mostly, many and few. Secondly, a neural network is designed to learn the meaning of mostly red, many red and few red. The neural network is independent to the database used, which avoids re-training of the neural network. Finally, a binary search algorithm is used to match and display neural network’s output and images from database. The proposed technique is very unique and the originality of this research is not only based on hybrid approach to content based image retrieval but also on the new idea of training neural networks on queries. One of the most unique aspects of this research is that neural network is designed to learn queries and not databases. The technique can be used for any real-world online database. The technique has been implemented using CGI scripts and C programming language. Experimental results demonstrate the success of the new approach.

Large collection of images is growing rapidly due to the advent of cheaper storage devices, fast computers & communication technologies, internet access of databases, etc. Retrieving images from such large collections efficiently and effectively, based on their content, has become an important research issue for database, image processing and computer vision communities. Several systems [1, 2] have been developed so far. These systems model image data using features such as color, texture and shape. Such features are usually extracted from images and stored into databases [3]. Color is one of the most straightforward features utilised by people for visual recognition and discrimination. However, people show natural ability of using different levels of color specificity in different contexts [4, 5]. The inherent features of the content-based image retrieval systems are “imprecision”, “partial information” and “user preferences” [6, 7]. To retrieve images that have green lawn in them from a database, the user has to select a green color pallet (available in the system) and ask the system to retrieve images with the matching color. Other features of the image can be used similarly. This is the partial and

imprecise query. From the user’s point of view the given color pallet represents part of the user’s information need (partial information). From the database’s point of view the query is imprecise. Existing content based retrieval systems deal with “imprecision” providing partial information through user interfaces such as “retrieve all images that have some specific color” [8]. For example, the user can pose a query to retrieve images having green color (by selecting the green color pallet) with preference 0.8 and round object (by selecting the shape example) with preference 0.2. Such querying has been widely supported in text retrieval. The color extraction algorithm identifies arbitrarily shaped regions within images that contain colors participating in specific color sets derived from a color palette. By the way of a search over the color sets for each image, localised color features are extracted and the index for the database is built directly [9]. Feature values are either represented in the complex form such as matrices or have a huge number of possible values. If a user is asked to retrieve images by posing queries based on color information, the user always refers to a small set of colors. It has been experimentally found that the user can only recognise a small set of feature values in general. Hence taking human perceptual range into account, our prototype system distinguishes between nine colors only [10]. We propose a neuro-fuzzy model for content based image retrieval system, considering inherent nature of the problem. The prototype system is efficient and user friendly. The system is built using CGI script in C language. The rest of the paper is organised as follows. In section 2, we explain the proposed neuro-fuzzy content based image retrieval system, implementation is discussed in section 3, experimental results are presented in section 4, and conclusion are given in section 5.

Figure 1: Query Interpretation Stage 2: Figure 2 gives the basic idea of our neuro-fuzzy content based image retrieval system. Different images are downloaded from World Wide Web (WWW) and stored as image database. Features of images such as color, texture and shape are extracted from the image and stored into the database. Let FS denote the set of all features used to represent an image; for example Fs = {color, texture, shape}. But we use only the color feature in our initial prototype implementation, Fs = {color}. Experiments with shape and texture will be considered at a later stage. We use Fu to denote the set of feature values that an user can recognise. Our feature representation set is motivated by the research results of Carson and Ogle [7] who identify nine colors that fall within the range of human perception. The color feature is extracted from the images using the program developed by the Berkeley Digital Library Project These (http://elib.cs.berkely.edu/src/cypress/meets.c). colors are stored in a separate database as the feature database.

2. Proposed Neuro-Fuzzy Content Based Image Retrieval System This section explains the technique of the proposed neurofuzzy content based image retrieval system in two stages. Stage 1: Referring to Figure 1, the query to retrieve the images from database is prepared in terms of natural language such as mostly content, many content and few content of some specific color. Fuzzy logic is used to define the query. We define nine colors that fall within the range of human perception. The feature representation set of colors is: rep (color)= {red, green, blue, white, black, yellow, orange, pink, purple}. These nine colors are used as input to the neural network and the content type as output. Mostly, many and few indicate the output.

Figure 2: Block Diagram of Neuro-Fuzzy System Users like to provide the queries in terms of natural language such as mostly, many and few. So we assume the particular color content for each image to be “mostly”, “many” and “few”. In our model, the interpretation domain is a fuzzy set [0, 1]. The ranges of the values used are [0.9, 1] for “mostly”, [0.4, 0.5] for “many” and [0.15, 0.25] for few. Also, the numeric weights such as 0.9 and 0.92 are so close to each other that they both indicate the particular feature is mostly present in the desired images. We define

Fu as the set of nine colors. Np represents the number of pixels in each image. For each color value (red, green, blue, etc.) we record the number of the pixels that belong to the value, denoted by Nf where f ∈ Fu. The output obtained after training the neural network is used to calculate the confidential factor for each image for the specific query.

3. Implementation The training file of neural network was prepared by considering nine inputs as colors and three outputs as content type. Table 1 enlists all the parameters and their values used to train the neural network. The values of these parameters provide optimum results.

The number of inputs indicates all nine colors that we have selected for the experiments. The number of outputs is three and it indicates the content type such as mostly, many and few. The number of hidden units is taken as five for our experiment. The training pairs are decided for each content type. There are ninety-nine training pairs for each content, so the total training pairs obtained is 297. Learning rate (η) and momentum (α) are kept as 0.7 and 0.2 respectively for our initial experiments. We performed the experiments by taking 100 iterations initially. The neural network is trained only once for the queries. As the neural network is not trained on the database of images, it is not necessary to re-train it, if there is change in the database of the images. This technique has been implemented using CGI script in C language.

Table 1. Neural Network Parameters for Experiments Inputs

Outputs

9

3

Hidden Units 5

4. Experimental Results To test the effectiveness of our system, the preliminary implementation of our prototype is kept simple. We used a collection of images from World Wide Web (WWW) as our database. We loaded these images into our system. If the user wants to retrieve images such as “Green, Mostly”, the queries are selected from the tabular form from color and type. The queries are given in natural language such as few, many and mostly rather than giving as 50% or 90% content of a particular color. Figure 3 shows the result of the color image retrieval query along with the color and the content type specified for the query. Thus we see that it is easy to pose the required queries in natural way and get

Training Pairs 297

η

α

Iterations

0.7

0.2

100

the required response. The images are obtained along with their image number from the database and the confidential factor. The best image for suitable query appears first with the highest confidential factor. In figure 3, image #77 contains the maximum green color having the confidential factor 0.992769. It contains the picture of a horse on a lawn. This image has the maximum percentage of green color pixels compared to any other image in the database. Other images appear in sequence with the confidential factor in descending order. Similarly the queries can be formed by different combinations such as “Red, Many” or “White, Few” etc.

Figure 3: Result of the Query: “Mostly Green”

The results obtained by submitting the query “Many Red” are shown in figure 4. Six images are obtained for this query. The images contain red color flowers. These images contain red color to a certain extent. The term certain has been defined using fuzzy logic as many. The number of red pixels is more than the number of pixels for any other color in these images. First image (image # 1) has the greatest confidential factor, 0.997681. More images are

found according to their confidential factor in descending order. In a similar way, we have performed experiments for combination of all colors and all types and received satisfactory results. If there are no images in the image database for some specific query given by the user, then the user gets the message “There are no images for the specific query.”

Figure 4: Result of the Query: “Mostly Red”

5. Conclusion Single color query is an extremely useful content-based query tool for users of image databases. We have proposed a method for automatically extracting the single and multiple color regions within images. The single color approach allows the user to specify the particular color and content type. The preliminary results support our basic ideas, and the approach appears promising. Only one neural network is used to train the query, for color and content type. Our future work will incorporate other feature of image such as texture (eg. sky, snow, grass etc.) and shape (eg. triangular, circular, rectangular etc.).

[3] Smith, J. R. and Chang, S. F., (1995), Single Color Extraction and Image Query, International Conference on Image Processing, Washington D.C. [4] Flickner, M., Sawhney, H. and Niblack, W., (1995), Query by Image and Video Content: The QBIC System. Computer, Volume 5, pages 249-274. [5] Nepal, S., Ramakrishna, M. V., and Thom, J. A., (1998), A Fuzzy System for Content Based Image Retrieval, IEEE International Conference on Intelligent Processing Systems, Gold Coast, pages 335-339.

References

[6] Smith, J. R., and Chang, S. F., (1996), Automated Image Retrieval Using a Color and Texture, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Special Issue on Digital Libraries: Representation and Retrieval.

[1] Pentland, A., Picard, R.W. and Sclaroff, S., (1994), Photobook: Content Based Manipulation of Image Databases, IEEE Multimedia, pages 73-75.

[7] Carson, C. and Ogle, V. E., (1996), Storage and Retrieval of feature Data for a Very large Online Image Collection, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, Volume 19, pages 19-25.

[2] Ma, W. Y. and Manjunath, B. S., (1997), Netra: A Toolbox for Navigating Large Image Databases, Proceedings of the IEEE International conference on Image Processing.

[8] Santani, S. and Jain., R., (1996), Similarity Queries in Image Databases, Proceedings of CVPR 96, IEEE International Conference on Computer Vision and Pattern Recognition, San Francisco.

[9] Fukushima, S. and Ralescu, A. L., (1996), Improved Retrieval in a Fuzzy Database from Adjusted user input. Journal of Intelligent Information Systems, Volume 5, pages 249-274.

[12] Smith, J. R., and Chang, S. F., (1995), Extracting multidimensional signal features for content-based visual query. In Symposium on Visual Communications and Signal Processing, SPIE.

[10] Fagin, R. and Wimmers E. D., (1997), Incorporating user Preferences in Multimedia Queries. International Conference on Database Theory. URL: http://www.almaden.ibm.com/cs/people/fagin.

[13] Malik, J., Forsyth, D., Fleck, M., Greenspan, T., Leung, T., Carson, C., Belongie, S. and Bregler, C., (1996), Finding the objects in image databases by grouping. International Conference on Image Processing, Special Session on Images in Digital Libraries.

[11] Funt, B. V. and Finalyson G. D., (1995), Color constant color indexing. IEEE Transaction on Pattern Analysis and machine Intelligence Volume 17, pages 122-129.