Semantic content based image retrieval using object-process diagrams

The increase in accessability to on-line visual data has promoted the interest in browsing and retrieval of images from image Databases. Current approaches assume either a text based key-word oriented approach or a visual feature based approach. Tile keyword approach usually provides neither layout information nor object relevence and significance in the scene. The visual feature based approach relies on low level features such as color, texture and orientation as image descriptors. These are non-intuitive and unnatural for human observers. This paper presents a new approach to image retrieval in which image content, based on the “visual scene”, is the basis for both retrieval and user interface. We propose to model image content using Object-Process Diagrams. Our hierarchical approach incorporates both the low-level image features and textual key sentencess as descriptors of the image. These descriptions involve the objects in the scene and their inter-and intra-relationships. This allows for abstract, high-level representation of the layout of the scene, as well as a distinction between the dominant core of the scene and its background. Querying is is performed by representing the sought image with an Object-Process Diagram and finding the images in tile database whose Object-Process Diagrams best match the query.