|
(11) | EP 3 422 217 A1 |
| (12) | EUROPEAN PATENT APPLICATION |
|
|
|
|
|||||||||||||||||||||||
|
||||||||||||||||||||||||
| (54) | RETRIEVING SYSTEM, RETRIEVING METHOD, AND SECURITY INSPECTION DEVICE BASED ON CONTENTS OF FLUOROSCOPIC IMAGES |
| (57) Disclosed are a retrieving system and a retrieving method based on content of fluoroscopic
images, the retrieving system (100) comprising: a pre-classifying module (101), configured
to pre-classify fluoroscopic images and classify the fluoroscopic images into texture
and non-texture images; an image content feature extracting module (102), configured
to perform feature extraction for contents of the fluoroscopic images; an image representing
module (103), configured to combine feature description vectors of the content of
the fluoroscopic images, and construct an image representation vector; and a retrieving
module (104), configured to retrieve, from a fluoroscopic image representation database
based on a retrieval model, a plurality of images having a higher similarity to the
image representation vectors under retrieval, and thereby construct a result of preliminary
candidates.
|
TECHNICAL FIELD
[0001] The present invention relates to the crossing fields of radiation imaging, image understanding, machine learning, auxiliary examination and inspection, and the like, and in particular, to a retrieving system based on contents of fluoroscopic images picked up via radiation during security inspection.
BACKGROUND
[0002] Radiation fluoroscopic imaging technology is a major means for noninvasive inspection on articles, luggage, or the like, and has been currently widely used at air ports, customs, bus stations, large gatherings, and the like sites. The physical mechanism of radiation imaging causes the image feature presented to be notably different from a natural image. The difference directly causes the security inspection personnel not to perceptually understand image content. What is worse, the security inspection personnel's sensitivity on hazardous articles is extremely reduced. Therefore, the technical solution of globally or partially retrieving inspection fluoroscopic images to find similar images from a database, and giving annotation information such as similarity, target attribute (atomic number range, and weight and thickness ranges), and hazardous extent, is urgently desired. At present, in the field of security inspection, retrieval of the fluoroscopic images is mainly based on file name and time, without introducing any content information of the fluoroscopic images. With popularity of the radiation security inspection devices, a large amount of history image data is generated. If no convenient retrieval and comparison means is available, the large amount of history image data exerts no positive effects.
[0003] Currently, in the field of retrieval based on image contents, natural images and medical fluoroscopic images are mainly concentrated. Content retrieval of the natural images has gained rapid development in recent years, and a plurality of practical products have been developed, for example, Google image retrieving system, Baidu image retrieving system, and the like. Content retrieval of the medical fluoroscopic images mainly concerns accurate and precise search from the database for other patients having the same examination area, such that the doctor conveniently determines differences of pathological changes in the same examination area of different patients. However, such systems are still in the trial stage currently. Since the medical image data content is single, the data cannot be directly applied to the more widely used radiation fluoroscopic images.
[0004] In recent years, with rapid development of machine learning, mode recognition, and the like fields, such auxiliary means of quickly searching for fluoroscopic images having similar contents and providing these fluoroscopic images to the security inspection personnel for comparison and analysis has become a focus in the industry. However, due to complexity of the article categories and particularity of the fluoroscopic images, the field never sees fast development and promotion, and no relevant publication has been found.
SUMMARY
[0005] In view of practical circumstances of fluoroscopic images picked up during security inspection-a variety categories of articles to be scanned, superimposition of article structural information, high natural expression information loss and low image resolution, and actual needs of helping security inspection personnel to better understand and inspect fluoroscopic images, the present invention provides a system capable of retrieving fluoroscopic images based on contents of the fluoroscopic images and outputting a diversified (similar contents but different article categories) retrieval result and relevant annotation information.
[0006] The retrieving system mainly implements the following functions: (1) fuzzy retrieval of image contents, providing a diversified (similar contents but different article categories) retrieval result; (2) parallel computation, achieving real-time return of the retrieval result; (3) providing annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like; (4) automatically regulating the retrieval model according to user's feedback information. The retrieving system may be mainly used to train security inspection personnel for better understanding of X-ray images, help the security inspection personnel to analyze image contents, and improve the inspection efficiency.
[0007] According to an embodiment of the present invention, a retrieving system based on contents of fluoroscopic images picked up during security inspection is provided. The retrieving system includes:
a pre-classifying module, configured to pre-classify fluoroscopic images, and classify the fluoroscopic images into texture images and non-texture images;
an image content feature extracting module, configured to perform feature extraction for contents of the fluoroscopic images;
an image representing module, configured to combine feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector;
a retrieving module, configured to retrieve, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
a diversified filtering module, configured to filter the result of preliminary candidates, select an image subset capable of covering a plurality of article categories, and thereby construct a diversified retrieval result;
a correlation feedback regulating module, configured to receive information feedback on the retrieval result from a user, and update the retrieval model; and
an interacting module, configured to display the retrieval result, and collect feedback on user's satisfaction of the retrieval result.
[0008] The retrieving system implements fuzzy retrieval of image contents, and provides a diversified (similar contents but different article categories) retrieval result; by means of parallel computation, achieves real-time return of the retrieval result; and provides annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like. According to user's feedback information, the retrieving system is also capable of automatically regulating the retrieval model.
[0009] Preferably, the image content feature extracting module is configured to: with respect to a texture image, establish an image pyramid, perform super pixel classification layer by layer, and establish a texture description vector using a super pixel as a unit; and with respect to a non-texture image, establish an image pyramid, perform feature point detection pixel by pixel and layer by layer, and establish a feature description vector for a detected feature point.
[0010] Preferably, the image representing module is configured to: with respect to a texture image, by using a codebook policy, collect statistics for all the texture description vectors established using a super pixel as a unit, establish a histogram, and use the histogram as a representation vector of the image; and with respect to a non-texture image, aggregate all feature description vectors established on feature points, thereby construct a feature vector set, and use the feature vector set as a representation vector of the image.
[0011] Preferably, the retrieving module is configured to: with respect to a texture image, represent the image using a feature vector; and with respect to a non-texture image, represent the image using a feature vector set.
[0012] Preferably, the interacting module employs a two-stage tree-like display solution: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering module, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulating module.
[0013] According to another embodiment of the present invention, a retrieving method based on contents of fluoroscopic images is further provided. The retrieving method includes:
a pre-classification step: pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
an image content feature extraction step: performing feature extraction for contents of the fluoroscopic images;
an image representation step: combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector;
a retrieval step: retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
a diversified filtering step: filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result;
a correlation feedback regulation step: receiving information feedback on the retrieval result from a user, and updating the retrieval model; and
an interaction step: displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result.
[0014] In a particular embodiment of the retrieving method, the image content feature extraction step comprises:
with respect to a texture image, establishing an image pyramid, building super-pixels layer by layer, and establishing a texture description vector for every super-pixel; and
with respect to a non-texture image, establishing an image pyramid, performing feature point detection pixel by pixel and layer by layer, and establishing a feature description vector for a detected feature point.
[0015] In a particular embodiment of the retrieving method, the image representation step comprises:
with respect to a texture image, by using a codebook policy, collecting statistics for all the texture description vectors established using a super pixel as a unit, establishing a histogram, and using the histogram as a representation vector of the image; and
with respect to a non-texture image, aggregating all feature description vectors established on feature points, thereby constructing a feature vector set, and using the feature vector set as a representation vector of the image.
[0016] In a particular embodiment of the retrieving method, the retrieval step comprises:
with respect to a texture image, representing the image using a feature vector; and
with respect to a non-texture image, representing the image using a feature vector set.
[0017] In a particular embodiment of the retrieving method, in the interaction step, a two-stage tree-like display solution is employed: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering step, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulation step.
[0018] In a particular embodiment, the retrieving method further comprises establishment and update of the fluoroscopic image representation database:
subjecting all fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; performing the image content feature extraction step to obtain feature vectors describing the contents of the fluoroscopic images; and performing the image representation step to construct image representations, and storing the image representations in the fluoroscopic image representation database.
[0019] In a particular embodiment, the retrieving method further comprises update of the retrieval model based on user information feedback:
via the interaction step, evaluating, by the user, satisfaction on a retrieval result returned by a retrieval engine, and feeding back an evaluation result to a retrieving system; and regulating, by the retrieving system, model parameters according to the evaluation result feedback, and triggering an update process of the fluoroscopic image representation database.
[0020] In a particular embodiment the retrieving method further comprises update of a fluoroscopic image database:
submitting, by the user, a to-be-retrieved image to the fluoroscopic image database, to automatically trigger a building process of the fluoroscopic image representation database, thereby achieving synchronization between the fluoroscopic image database and the fluoroscopic image representation database.
[0021] The invention also relates to a security inspection device, comprising a retrieving system based on contents of fluoroscopic images as mentioned above.
[0022] According to still another embodiment of the present invention, a computer program is further provided. The computer program performs any one of the above methods when being executed on a processor of a device.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] With reference to the embodiments described hereinafter, various details and aspects of the present invention would be clearly illustrated.
[0024] In the drawings:
FIG. 1 illustrates a retrieving system based on contents of fluoroscopic images picked up via radiation during security inspection according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of display of a tree-like retrieval result according to the present invention; and
FIG. 3 illustrates a flowchart of a retrieving method based on contents of fluoroscopic images according to an embodiment of the present invention.
DETAILED DESCRIPTION
[0025] To make the objectives, structures, and advantages of the present invention clearer, the present invention is further described in detail with reference to the attached drawings. For brevity of description, only one of multiple possible configurations is illustrated in the drawings and the specification. A person skilled in the art would understand that without departing from the spirit of the present invention, various modifications and replacements may be made to the following embodiments of the present invention.
[0026] FIG. 1 illustrates a retrieving system 100 based on contents of fluoroscopic images picked up via radiation during security inspection according to an embodiment of the present invention. The retrieving system 100 includes: a pre-classifying module 101, an image content feature extracting module 102, an image representing module 103, a retrieving module 104, a diversified filtering module 105, a correlation feedback regulating module 106, and an interacting module 107.
Pre-classifying module
[0027] The pre-classifying module 101 pre-classifies fluoroscopic images, and classifies the fluoroscopic images into texture images and non-texture images. In the subsequent processing, the retrieving system separately processes the texture images and the non-texture images.
Image content feature extracting module
[0028] The image content feature extracting module 102 performs feature extraction for contents of the fluoroscopic images. Based on a pre-classification result, the texture images and the non-texture images are separately processed.
[0029] With respect to a texture image, the image content feature extracting module 102 establishes an image pyramid, performs super pixel classification layer by layer, and establishes a texture description vector using a super pixel as a unit.
[0030] With respect to a non-texture image, the image content feature extracting module 102 establishes an image pyramid, performs feature point detection pixel by pixel and layer by layer, and establishes a feature description vector for a detected feature point.
Image representing module
[0031] The image representing module 103 combines feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector. The texture images and the non-texture images are separately processed.
[0032] With respect to a texture image, the image representing module 103, by using a codebook policy, collects statistics for all the texture description vectors established using a super pixel as a unit, establishes a histogram, and uses the histogram as a representation vector of the image.
[0033] With respect to a non-texture image, the image representing module 103 aggregates all feature description vectors established on feature points, thereby constructs a feature vector set, and uses the feature vector set as a representation vector of the image.
Retrieving module
[0034] The retrieving module 104 retrieves, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructs a result of preliminary candidates. By using the retrieval model, the result of preliminary candidates may be quickly obtained.
Diversified filtering module
[0035] The diversified filtering module 105 filters the result of preliminary candidates, selects an image subset capable of covering a plurality of article categories, and thereby constructs a diversified retrieval result.
Correlation feedback regulating module
[0036] The correlation feedback regulating module 106 receives information feedback on the retrieval result from a user, and updates the retrieval model.
Interacting module
[0037] The interacting module 107 displays the retrieval result, and collects feedback on user's satisfaction of the retrieval result.
[0038] According to another embodiment of the present invention, a retrieving method based on contents of fluoroscopic images picked up during security inspection is further provided. FIG. 3 illustrates a flowchart of the retrieving method. The retrieving method includes: a pre-classification step, an image content feature extraction step, an image representation step, a retrieval step, a diversified filtering step, a correlation feedback regulation step, and an interaction step.
Pre-classification step
[0039] In the pre-classification step, fluoroscopic images are pre-classified, and the fluoroscopic images are classified into texture images and non-texture images. In the subsequent processing, the retrieving system separately processes the texture images and the non-texture images.
Image content feature extraction step
[0040] In the image content feature extraction step: feature extraction is performed for contents of the fluoroscopic images. Based on a pre-classification result, the texture images and the non-texture images are separately processed.
[0041] With respect to a texture image, an image pyramid is established, super pixel classification is performed layer by layer, and a texture description vector is established using a super pixel as a unit.
[0042] With respect to a non-texture image, an image pyramid is established, feature point detection is performed pixel by pixel and layer by layer, and a feature description vector is established for a detected feature point.
Image representation step
[0043] In the image representation step, feature description vectors of the contents of the fluoroscopic images are combined, and an image representation vector is constructed. The texture images and the non-texture images are separately processed.
[0044] With respect to a texture image, by using a codebook policy, statistics for all the texture description vectors established using a super pixel as a unit are collected, a histogram is established, and the histogram is used as a representation vector of the image.
[0045] With respect to a non-texture image, all feature description vectors established on feature points are aggregated, thereby a feature vector set is constructed, and the feature vector set is used as a representation vector of the image.
Retrieval step
[0046] In the retrieval step, a plurality of images having a higher similarity to the image representation vectors under retrieval are retrieved from a fluoroscopic image representation database based on a retrieval model, and thereby a result of preliminary candidates is constructed. By using the retrieval model, the result of preliminary candidates may be quickly obtained.
Diversified filtering step
[0047] In the diversified filtering step, the result of preliminary candidates is filtered, an image subset capable of covering a plurality of article categories is selected, and thereby a diversified retrieval result is constructed.
Correlation feedback regulation step
[0048] In the correlation feedback regulation step, information feedback on the retrieval result is received from a user, and the retrieval model is updated.
Interaction step
[0049] In the interaction step, the retrieval result is displayed, and feedback on user's satisfaction of the retrieval result is collected.
[0050] The retrieving system and method based on contents of fluoroscopic images picked up via radiation during security inspection according to the present invention implement fuzzy retrieval of image contents, and provide a diversified (similar contents but different article categories) retrieval result; by means of parallel computation, achieve real-time return of the retrieval result; and provide annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like. According to user's feedback information, the retrieving system is also capable of automatically regulating the retrieval model. The retrieving system and method according to the present invention may be mainly used to train security inspection personnel for better understanding of X-ray images, help the security inspection personnel to analyze image contents, and improve the inspection efficiency.
[0051] Running of the whole retrieving system/method may also be partitioned into four parts encircled by different dotted lines as illustrated in FIG. 1. The first part illustrates establishment and update of the fluoroscopic image representation database (indicated by 108); the second part illustrates representation computation, retrieval, and diversified filtering of the to-be-retrieved images, and display of the retrieval result (indicated by 109); the third part illustrates acquisition of user information feedback and regulation of the retrieval model (indicated by 110); and the fourth part illustrates update of the fluoroscopic image database (indicated by 111) and synchronization between the fluoroscopic image database and the fluoroscopic image representation database.
[0052] Running mechanism of the above-described parts will be described hereinafter:
- (1) Establishment and update of the fluoroscopic image representation database
All fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed are firstly subjected to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; then the image content feature extraction step is performed to obtain feature vectors describing the contents of the fluoroscopic images; and finally the image representation step is performed to construct image representations, and the image representations are stored in the fluoroscopic image representation database. - (2) Retrieval of the fluoroscopic images and display of the retrieval result
The user submits to-be-retrieved images. Firstly, these images are subjected to the pre-classification step to obtain a preliminary judgment on image contents (texture images or non-texture images). Subsequently, the image content feature extraction step is performed to obtain feature vectors describing the contents of the fluoroscopic images. Finally, the image representation step is performed to construct image representation. After the image representations are obtained, the retrieval step is performed to obtain a set of fluoroscopic image representations having a higher similarity to each other from the fluoroscopic image representation database according to the retrieval model, and thereby obtain a result of preliminary candidates. Subsequently, the diversified filtering step is performed to select an image subset capable of covering a plurality of article categories from the result of preliminary candidates. A corresponding fluoroscopic image is selected and output from the fluoroscopic image database according to the index of the subset. - (3) Update of the retrieval model based on user information feedback
Via the interaction step, the user evaluates satisfaction on the retrieval result returned by a retrieval engine, and feeds back an evaluation result to the retrieving system. The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database. - (4) Update of the fluoroscopic image database
The user submits a to-be-retrieved image to the fluoroscopic image database, to automatically trigger an establish process of the fluoroscopic image representation database, thereby achieving synchronization between the fluoroscopic image database and the fluoroscopic image representation database.
[0053] Without loss of generality, an embodiment of a specific algorithm is illustrated herein. However, it should be understood that under teachings of the present invention, a person skilled in the art may make variations or replacements to the specific algorithm in this embodiment without departing from the spirit of the present invention.
Pre-classifying module
[0054] The pre-classifying module randomly samples 50 x 50 image blocks on the to-be-retrieved image, and the coverage area of all the sampled image blocks needs to be over 60% of the total area of the image. The image blocks are subjected to a Histogram of Oriented Gradient (HOG) feature description, and are classified by a Support Vector Machine (SVM) classifier.
[0055] The HOG feature description is employed herein and an SVM classification model is employed for classification. However, the present invention is not limited to such combination. Any texture feature and classifier may be applied to the present invention.
Image content feature extracting module
[0056] Image content feature extraction is performed on the texture images and the non-texture images using different methods.
With respect to a texture image:
<1> feature description using a super pixel as a unit
[0057] Super pixel segmentation is performed for the images using the Statistical Region Merge (SRM) algorithm to establish a super pixel label graph. The images are sampled at an equal spacing, and an HOG feature description is established in a 20 x 20 region at the sampling site. Based on the super pixel label graph, the HOG feature descriptions at the sampling site of each of the super pixels are combined to construct the HOG feature descriptions of the super pixels.
<2> Establishment of a super pixel feature dictionary on a training image set
[0058] Feature descriptions are established to all the super pixels in each of the training images. HOG feature description vectors of all the super pixels are aggregated to 2000 categories by using the KMeans algorithm. In this way, a super pixel feature dictionary having a capacity of 2000 is constructed. The dictionary would be used in the image representing module.
[0059] Herein, the SRM algorithm is used to perform super pixel segmentation, and HOG texture feature description is employed. However, the present invention is not limited to such combination. Any image segmentation algorithm and texture feature description algorithm may be applied to the present invention.
With respect to a non-texture image:
[0060] Feature point detection and feature description are performed on the image by using the Scale-Invariant Feature Transform (SIFT) algorithm, and thereby a feature description vector set is constructed.
[0061] Herein, the SIFT algorithm is used to perform feature detection and feature description. However, the present invention is not limited to such combination. Any feature point detection algorithm and feature description algorithm may be applied to the present invention.
Image representing module
[0062] The texture images and the non-texture images are represented using different image representation methods.
With respect to a texture image:
[0063] Based on the super pixel feature dictionary, feature descriptions of all the super pixels in the image are quantified, and thereby an image representation vector is constructed. The dimensionality of the representation vector is equal to the capacity of the super pixel feature dictionary, and is extremely sparse.
With respect to a non-texture image:
[0064] The set of the feature vectors of all the feature points is used as the image representation.
Retrieving module
With respect to a texture image:
[0066] The image representation is indicated by a feature vector. Items relatively similar to the representation of the to-be-retrieved image are searched from the fluoroscopic image representation database using the K Nearest Neighbor (KNN) algorithm, and fluoroscopic image data is obtained from the fluoroscopic image database according to indexes of the items, and the obtained fluoroscopic image data is used as the result of preliminary candidates. A result of preliminary candidates having 200 elements is constructed.
With respect to a non-texture image:
[0067] The image representation is indicated by a feature vector set. With respect to each of the feature vectors in the feature vector set, items similar to the representation of the to-be-retrieved image are searched from the fluoroscopic image representation database using the KNN algorithm, and fluoroscopic image data is obtained from the fluoroscopic image database according to indexes of the items. If at least three feature vectors in the feature vector set point at the same fluoroscopic image, then the fluoroscopic image is added to the result of preliminary candidates. A result of preliminary candidates having 200 elements is constructed.
[0068] The similarity comparison herein employs the Euclidean distance. However, the present invention is not limited thereto. Any method for similarity measurement may be applied to the present invention, for example, Manifold Learning, and the like.
Diversified filtering module
[0069] Three images capable of maximally covering different article categories are selected from a preliminary search result using the structural SVM model.
Correlation feedback regulating module
[0070] User's information feedback on the retrieval result is received, and the retrieval model is updated.
Interacting module
[0071] A two-stage tree-like display solution is employed, as illustrated in FIG. 2. A retrieval region is selected on a scanning image via a mouse. Upon selection of the retrieval region, two-stage tree-like retrieval results are displayed in real time around the retrieval region. A first stage is displaying an output result from the diversified filtering module, and a second stage is displaying other similar images that are in the same article category as a previous stage. When the mouse slides over a node at either of the two stages, a selection label for user's satisfaction on the retrieval results is displayed. When feedback information is acquired, the feedback information is sent to the correlation feedback regulating module.
[0072] According to a further embodiment of the present invention, a security inspection device is disclosed. The security inspection device includes the retrieving system based on contents of fluoroscopic images as described above.
[0073] Although the present invention has been illustrated and described in detail with reference to the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary but not restrictive. The present invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments may be understood and effected by a person skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims.
[0074] In the claims and specification, the word "comprising" or "includes" does not exclude other elements or steps, and the infinite article "a" or "an" does not exclude a plurality. A single element or another unit may fulfill functions of several features recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
[0076] A retrieving system based on contents of fluoroscopic images, comprises: a pre-classifying module, configured to pre-classify fluoroscopic images, and classify the fluoroscopic images into texture images and non-texture images; an image content feature extracting module, configured to perform feature extraction for contents of the fluoroscopic images; an image representing module, configured to combine feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector; a retrieving module, configured to retrieve, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby construct a result of preliminary candidates; a diversified filtering module, configured to filter the result of preliminary candidates, select an image subset capable of covering a plurality of article categories, and thereby construct a diversified retrieval result; a correlation feedback regulating module, configured to receive information feedback on the retrieval result from a user, and update the retrieval model; and an interacting module, configured to display the retrieval result, and collect feedback on user's satisfaction of the retrieval result.
[0077] In the retrieving system described in paragraph [00139] the image content feature extracting module may be configured to: with respect to a texture image, establish an image pyramid, build super-pixel layer by layer, and establish a texture description vector for every super-pixel; and with respect to a non-texture image, establish an image pyramid, perform feature point detection pixel by pixel and layer by layer, and establish a feature description vector for a detected feature point.
[0078] In the retrieving system described in paragraph [00139] or in paragraph [00140], the image representing module may be configured to: with respect to a texture image, by using a codebook policy, collect statistics for all the texture description vectors of super-pixels, establish a histogram, and use the histogram as a representation vector of the image; and with respect to a non-texture image, aggregate all feature description vectors established on feature points, thereby construct a feature vector set, and use the feature vector set as a representation vector of the image.
[0079] In the retrieving system described in any one of paragraphs [00139] to [00141], the retrieving module may be configured to: with respect to a texture image, represent the image using a feature vector; and with respect to a non-texture image, represent the image using a feature vector set.
[0080] In the retrieving system described in any one of paragraphs [00139] to [00142], the interacting module may employ a two-stage tree-like display solution: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering module, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulating module.
[0081] A retrieving method based on contents of fluoroscopic images comprises:
a pre-classification step: pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
an image content feature extraction step: performing feature extraction for contents of the fluoroscopic images;
an image representation step: combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector;
a retrieval step: retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
a diversified filtering step: filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result;
a correlation feedback regulation step: receiving information feedback on the retrieval result from a user, and updating the retrieval model; and
an interaction step: displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result.
[0082] In the retrieving method described in paragraph [00144] the image content feature extraction step may comprise: with respect to a texture image, establishing an image pyramid, building super-pixels layer by layer, and establishing a texture description vector for every super-pixel; and with respect to a non-texture image, establishing an image pyramid, performing feature point detection pixel by pixel and layer by layer, and establishing a feature description vector for a detected feature point.
[0083] In the retrieving method described in paragraph [00144] or paragraph [00145], the image representation step may comprise: with respect to a texture image, by using a codebook policy, collecting statistics for all the texture description vectors established using a super pixel as a unit, establishing a histogram, and using the histogram as a representation vector of the image; and with respect to a non-texture image, aggregating all feature description vectors established on feature points, thereby constructing a feature vector set, and using the feature vector set as a representation vector of the image.
[0084] In the retrieving method described in any one of paragraphs [00144] to [00146], wherein the retrieval step may comprise: with respect to a texture image, representing the image using a feature vector; and with respect to a non-texture image, representing the image using a feature vector set.
[0085] In the retrieving method described in any one of paragraphs [00144] to [00147], in the interaction step, a two-stage tree-like display solution may be employed: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering step, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulation step.
[0086] The retrieving method described in any one of paragraphs [00144] to [00148] may further comprise establishment and update of the fluoroscopic image representation database: subjecting all fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; performing the image content feature extraction step to obtain feature vectors describing the contents of the fluoroscopic images; and performing the image representation step to construct image representations, and storing the image representations in the fluoroscopic image representation database.
[0087] The retrieving method described in any one of paragraphs [00144] to [00149] may further comprise update of the retrieval model based on user information feedback: via the interaction step, evaluating, by the user, satisfaction on a retrieval result returned by a retrieval engine, and feeding back an evaluation result to a retrieving system; and regulating, by the retrieving system, model parameters according to the evaluation result feedback, and triggering an update process of the fluoroscopic image representation database.
[0088] The retrieving method described in any one of paragraphs [00144] to [00150] may further comprise update of a fluoroscopic image database: submitting, by the user, a to-be-retrieved image to the fluoroscopic image database, to automatically trigger a building process of the fluoroscopic image representation database, thereby achieving synchronization between the fluoroscopic image database and the fluoroscopic image representation database.
1. A retrieving system (100) based on contents of fluoroscopic images, comprising:
a pre-classifying module (101), configured to pre-classify fluoroscopic images, and classify the fluoroscopic images into texture images and non-texture images;
an image content feature extracting module (102), configured to perform feature extraction for contents of the fluoroscopic images;
an image representing module (103), configured to combine feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector; and
a retrieving module (104), configured to retrieve, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby construct a result of preliminary candidates.
2. The retrieving system based on contents of fluoroscopic images according to claim
1, further comprising
a diversified filtering module (105), configured to filter the result of preliminary candidates, select an image subset capable of covering a plurality of article categories, and thereby construct a diversified retrieval result.
a diversified filtering module (105), configured to filter the result of preliminary candidates, select an image subset capable of covering a plurality of article categories, and thereby construct a diversified retrieval result.
3. The retrieving system based on contents of fluoroscopic images according to claim
1 or 2, further comprising
a correlation feedback regulating module (106), configured to receive information feedback on the retrieval result from a user, and update the retrieval model.
a correlation feedback regulating module (106), configured to receive information feedback on the retrieval result from a user, and update the retrieval model.
4. The retrieving system based on contents of fluoroscopic images according to claim
3, further comprising
an interacting module (107), configured to display the retrieval result, and collect feedback on user's satisfaction of the retrieval result;
wherein the interacting module (107) employs a two-stage tree-like display solution: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering module, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulating module.
an interacting module (107), configured to display the retrieval result, and collect feedback on user's satisfaction of the retrieval result;
wherein the interacting module (107) employs a two-stage tree-like display solution: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering module, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulating module.
5. The retrieving system based on contents of fluoroscopic images according to claim
4, wherein updating the retrieval model by the correlation feedback regulating module
(106) comprises:
regulating, by the retrieving system, model parameters according to the evaluation result; and
triggering, by the retrieving system, an update process of the fluoroscopic image representation database.
6. The retrieving system based on contents of fluoroscopic images according to any one
of claims 1 to 5, wherein the pre-classifying module (101) pre-classifies fluoroscopic
images by a Histogram of Oriented Gradient (HOG) feature description and a Support
Vector Machine (SVM) classifier.
7. The retrieving system based on contents of fluoroscopic images according to any one
of claims 1 to 6, wherein the image content feature extracting module (102) is configured
to:
with respect to a texture image, establish an image pyramid, build super-pixel layer by layer, and establish a texture description vector for every super-pixel; and
with respect to a non-texture image, establish an image pyramid, perform feature point detection pixel by pixel and layer by layer, and establish a feature description vector for a detected feature point.
8. The retrieving system based on contents of fluoroscopic images according to any one
of claims 1 to 7, wherein the image representing module (103) is configured to:
with respect to a texture image, by using a codebook policy, collect statistics for all the texture description vectors of super-pixels, establish a histogram, and use the histogram as a representation vector of the image; and
with respect to a non-texture image, aggregate all feature description vectors established on feature points, thereby construct a feature vector set, and use the feature vector set as a representation vector of the image.
9. The retrieving system based on contents of fluoroscopic images according to any one
of claims 1 to 8, wherein the retrieving module (104) is configured to:
with respect to a texture image, represent the image using a feature vector; and
with respect to a non-texture image, represent the image using a feature vector set.
10. A retrieving method based on contents of fluoroscopic images, comprising:
a pre-classification step: pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
an image content feature extraction step: performing feature extraction for contents of the fluoroscopic images;
an image representation step: combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector; and
a retrieval step: retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates.
11. The retrieving method based on contents of fluoroscopic images according to claim
10, further comprising
a diversified filtering step: filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result.
a diversified filtering step: filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result.
12. The retrieving method based on contents of fluoroscopic images according to claim
10 or 11, further comprising
a correlation feedback regulation step: receiving information feedback on the retrieval result from a user, and updating the retrieval model.
a correlation feedback regulation step: receiving information feedback on the retrieval result from a user, and updating the retrieval model.
13. The retrieving method based on contents of fluoroscopic images according to claim
12, further comprising
an interaction step: displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result;
wherein in the interaction step, a two-stage tree-like display solution is employed: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering step, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulation step.
an interaction step: displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result;
wherein in the interaction step, a two-stage tree-like display solution is employed: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering step, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulation step.
14. The retrieving method based on contents of fluoroscopic images according to claim
13, further comprising update of the retrieval model based on user information feedback:
via the interaction step, evaluating, by the user, satisfaction on a retrieval result returned by a retrieval engine, and feeding back an evaluation result to a retrieving system; and regulating, by the retrieving system, model parameters according to the evaluation result feedback, and triggering an update process of the fluoroscopic image representation database.
via the interaction step, evaluating, by the user, satisfaction on a retrieval result returned by a retrieval engine, and feeding back an evaluation result to a retrieving system; and regulating, by the retrieving system, model parameters according to the evaluation result feedback, and triggering an update process of the fluoroscopic image representation database.
15. The retrieving method based on contents of fluoroscopic images according to any one
of claims 10 to 14, further comprising establishment and update of the fluoroscopic
image representation database:
subjecting all fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; performing the image content feature extraction step to obtain feature vectors describing the contents of the fluoroscopic images; and performing the image representation step to construct image representations, and storing the image representations in the fluoroscopic image representation database.
subjecting all fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; performing the image content feature extraction step to obtain feature vectors describing the contents of the fluoroscopic images; and performing the image representation step to construct image representations, and storing the image representations in the fluoroscopic image representation database.
16. The retrieving method based on contents of fluoroscopic images according to any one
of claims 10 to 15, wherein the pre-classification step conducts pre-classification
by a Histogram of Oriented Gradient (HOG) feature description and a Support Vector
Machine (SVM) classifier.
17. A security inspection device, comprising a retrieving system based on contents of
fluoroscopic images according to any one of claims 1 to 9.