SCALE ROBUST FEATURE-BASED IDENTIFIERS FOR IMAGE IDENTIFICATION

Description

BACKGROUND TO THE INVENTION

[0001] This invention relates to the problem of robust image identification and discloses a new method that offers advantages over prior art methods.

DESCRIPTION OF THE RELATED ART

[0002] The aim of robust identification of multimedia content is to extract information that is preserved under typical modifications that may be performed on it. That modification may be for innocent or malicious purposes. In the case of images, a robust identifier should be substantially preserved when subjected to compression, greyscale transformation, rotation, noise, colour alteration, cropping and translation amongst others. Additionally, an identifier should provide discrimination powers such that independent content has sufficiently different identifiers.

[0003] The techniques disclosed in European patent application No. EP 06255239.3, and UK patent application Nos. GB 0700468.2, GB 0712388.8 and GB 0719833.6 are similar in that they all extract an identifier based on global information, that is information about the whole image. This family of related identifiers provide robustness close to 100% at very low false alarm rates (which means they provide high discrimination ability), with typical rates being less than 0.1 parts per million.

[0004] Large numbers of images exist in various image databases, embedded in web-pages, presentation and documents. A survey of prominent web sites shows that Flickr has over 2 billion images, Photobucket has over 4 Billion and Facebook has 1.7 Billion. Consumers often have thousands of photographs in their personal collections and professionals will have significantly more. To be of practical use given the large size of datasets, identifiers must also allow for very fast searching. The techniques disclosed in patent applications EP 06255239.3, GB 0700468.2, GB 0712388.8 and GB 0719833.6 may allow for search speeds of over 10 million images per second.

[0005] Despite high levels of robustness to many common image modifications the prior art identifiers disclosed in patent applications EP 06255239.3, GB 0700468.2, GB 0712388.8 and GB 0719833.6 may not provide the required level of robustness to a class of image modifications, where a significant part of image content is lost or replaced, such as translation and cropping. The technique disclosed in GB 0800364.2 provides significantly improved robustness under this class of image modifications. It is a process that involves i) feature detection and selection, ii) feature region and identifier extraction and iii) matching. The present invention proposes an improvement on the robustness of the prior art image identifiers.

[0006] Prior art feature point methods have been developed in the areas of computer vision and object recognition [references 2, 3 infra]. The approaches rely on a very large number of points:

"An important aspect of this approach is that it generates large numbers of features that densely cover the image... A typical image of size 500x500 pixels will give rise to about 2000 features."[reference 2 infra]

[0007] The result of extracting large numbers of features is that searching and matching calculations become very slow; in the naive case all features from a query image should be compared with all features from all images in the database. In order to reduce the search times prior art methods use various forms of data-structures, however these structures come at the cost of additional database initialisation times, additional storage and memory requirements and often some approximations are made which trades improved search speed for lower detection rates.

[0008] The present invention is concerned with detecting images and their modified versions, which differs from the area of application of much of the prior art. In particular, image duplicate detection is concerned with finding modified versions of an image e.g. cropped, compressed etc. In contrast, most prior art methods are concerned with detecting specific objects or a class of objects in images, e.g. a Ford Focus car or cars. Whilst the distinction between these problems may appear trivial or subtle it is important. Consider, for example, two images of St Paul's Cathedral in London, the images are taken from two different view points. Now under the object recognition task the similarity between these images should be detected but under the image duplicate detection the similarity should not be detected. With this in mind different techniques are needed to deal with the two different problems.

[0009] The method disclosed in GB 0800364.2 develops a binary feature point descriptor based on the Trace transform. The advantage of using a binary descriptor is that the Hamming distance is used for feature matching, which makes the searching process very fast. Furthermore, a selection process limits the number of features to a small set that proves to be surprisingly robust to difficult image modifications such as cropping and translation. Whilst the robustness is good it is desirable to provide further improvements.

[0010] LINDEBERG T: "Feature Detection with Automatic Scale Selection" Int. J. Of Computer Vision, [Online] vol. 30, no. 2, 1998, pages 79-116, relates to feature detection with automatic scale selection, and a framework for generating hypotheses about interesting scale levels in image data, based on a general principle that local extrema over scales of different combinations of γ-normalised derivatives are likely candidates to correspond to interesting structures. It is shown how this idea can be used as a mechanism in algorithms for automatic scale selection, which adapt the local scales of processing to the local image structure.

[0011] COPELAND A C ET AL: "Localized Radon Transform-Based Detection Of Ship Wakes In Sar Images" IEEE Transactions On Geoscience And Remote Sensing, Ieee Service Center, Piscataway, NJ, US, vol. 33, no. 1, 1 January 1995 (1995-01-01), pages 35-45, relates to linear feature detection using a localisation of the Radon transform whereby intensity integration is performed over short line segments rather than across the entire image.

SUMMARY OF THE INVENTION

[0012] In accordance with to a first aspect, the present invention provides a method for deriving a representation of an image, i.e. an image descriptor, by processing signals corresponding to the image, as set out in claim 1. The method comprises deriving a scale-space representation of the image. The scale-space representation is processed to detect a plurality of feature points in the scale-space representation having amplitude extrema (i.e. maxima or minima values). For one or more of the detected plurality of feature points, the method determines a scale-dependent image region associated with the corresponding feature point. A representation for each determined, scale-dependent image region is then derived.

[0013] Thus, an image descriptor may be derived, based on the features of the image having the highest feature strength.

[0014] In an embodiment, the plurality of feature points are detected by applying a first operator based on second order image gradients, to the scale-space representation of the image, and identifying those feature points in the scale-space representation with the highest resulting values. A second operator may then be applied to the detected plurality of feature points, the second operator based on first order image gradients, and features points with resulting values below a threshold removed. A predetermined number of features points may be selected from the plurality of features points detected by applying the first and second operators.

[0015] In other embodiments, the plurality of feature points may be detected using the operators in a different order, or using different techniques, according to the desired application.

[0016] Preferably, for selected one or more detected features points, a scale-dependent image region is determined by determining a spatial region of the image associated with the spatial position of the feature point, wherein the size of the spatial region is based on the scale of the feature point.

[0017] A representation for each determined, scale-dependent image region, may be derived based on line projections across the image region, such as by using the Trace transform, or equivalent.

[0018] In one example, for each of the one or more determined scale-dependent image regions, a function of at least part of the image region is derived, where the function of a translated, scaled or rotated version of an image region is a translated or scaled version of the function of the image region. The step of deriving a function of at least part of the image region may comprise applying a first functional along lines over the image region. The method may further comprise applying a second functional to the derived function to obtain an intermediate representation of the at least part of the image region.

[0019] A representation for each determined, scale-dependent image region may comprise a binary image descriptor.

[0020] A representation of the image may be derived from two or more of the derived representations of the scale-dependent image regions.

[0021] In accordance with a second aspect, the present invention provides a method for comparing images using descriptors of images derived using a method according to the first aspect of the present invention. The method comprises comparing representations of one or more scale-dependent image regions associated with the descriptor of a first image, with representations of one or more scale-dependent image regions associated with the descriptor of a second image.

[0022] Further aspects of the present invention include: use, including transmission or reception, of a representation of an image derived using a method according to the first aspect of the present invention; an apparatus for executing a method according to the first aspect or the second aspect of the present invention, or both, and a computer-readable storage medium comprising instructions that, when executed by a computed, perform a method according to the first aspect or the second aspect of the present invention, or both.

[0023] Other features and advantages of the invention will be apparent from the following description and accompanying claims.

[0024] This invention includes two main aspects that improve the robustness as required, i) region-scale is incorporated into the process of feature selection and extraction and ii) a matching process of up to four stages is developed that maintains robustness whilst minimising computational cost.

[0025] A feature detection process is used to select a plurality of features, having a spatial and scale location. The feature selection uses a novel process to choose features appropriate to the task and that have been shown experimentally to work well with Trace transform based feature descriptors.

[0026] For image matching a trade off is needed between detection performance, independence, computational time and memory requirements. In the disclosed embodiment, a four step matching process is introduced which aims to maximise detection rates and independence, whilst minimising both computational time and memory requirements. The first stage quickly determines hypotheses (i.e. candidates) with high probability and eliminates low probability matches, the second stage tests these hypotheses and removes a significant number of false hypothesis. The third stage eliminates further false hypotheses. Each stage involves an increased level of complexity over the previous stages, so complexity is managed and minimised by eliminating false hypotheses as early as possible in the process.

[0027] As well as the aforementioned aspects, other aspects of the invention include deriving a representation of an image based on the selected and extracted features of the image, and an apparatus and a computer program for implementing processes according to one or more aspects of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Embodiments of the invention will now be described with reference to the accompanying drawings, of which:

Fig. 1a shows an image;

Fig. 1b shows a reduced version of the image of Fig. 1a;

Fig. 1c shows a rotated version of the image of Fig. 1a;

Fig. 1d shows a blurred version of the image of Fig. 1a;

Fig. 1e shows a flipped (left-right) version of the image of Fig. 1a;

Fig. 1f shows a heavily compressed version of the image of Fig. 1a;

Fig. 1g shows a cropped version of the image of Fig. 1a;

Fig. 2 shows an image and a bit string representation of the image according to the prior art;

Fig. 3 shows an example set of interest points of an image detected by a feature detector;

Fig. 4 shows an image from which a final set of features is chosen, the scale of the feature is indicated by the size of the circle and the direction of the feature is indicated by the line;

Fig. 5 shows interest points found to match using the Hamming distance from (a) an original image and (b) a rotated version of (a), in which the line lengths between the interest points are used to apply geometric constraints;

Fig. 6 is a block diagram of an apparatus according to an embodiment of the present invention, and

Fig. 7 is a diagram showing the structure of a feature identifier (a) and an image identifier (b) according to and embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Various embodiments for deriving a representation of an image, specifically an image identifier, and for using such a representation/identifier for the purposes of, for example, identification, matching or validation of an image or images, will be described below. The present invention is especially useful for, but is not restricted to, identifying an image. In the described embodiments, an "image identifier" (also referred to as "identifier", "descriptor", "image descriptor", "signature" or "image signature") is an example of a representation of an image and the term is used merely to denote a representation of an image, or descriptor.

[0030] The skilled person will appreciate that the specific design details of an image identification apparatus and method, according to an embodiment of the invention, and the derivation of an image identifier for use in image identification, is determined by the requirements related to the type of image modifications it should be robust to, the size of the identifier, extraction and matching complexity, target false-alarm rate, etc.

[0031] The following example illustrates a generic design that results in an identifier that is robust to the following modifications to an image (this is not an exhaustive list):

• Colour reduction,
• Blurring,
• Brightness Change,
• Flip (left-right & top-bottom),
• Greyscale Conversion,
• Histogram Equalisation,
• JPEG Compression,
• Noise,
• Rotation,
• Cropping,
• Scaling,
• Translation,
• Skewing and
• Perspective change.

[0032] It has been found that the proposed design may typically achieve a low false-alarm rate of less than 1 parts per million (ppm) on a broad class of images and typical detection rates above 98.0%.

[0033] Fig. 1 shows an example of an image and modified versions of the image. More specifically, Fig. 1a is an original image, Fig. 1b is a reduced (scaled) version of the image of Fig. 1a, Fig. 1c is a rotated version of the image of Fig. 1a, Fig. 1d is a blurred version of the image of Fig. 1a, Fig. 1e is a flipped version of the image of Fig. 1a, Fig. 1f is a compressed version of the image of Fig. 1a and Fig. 1g is a cropped version of the image of Fig. 1a.

[0034] An embodiment of the invention derives a representation of an image, and more specifically, an image identifier, by processing signals and/or image data corresponding to the image.

Feature Point Detection and Selection

[0035] Consistent with EP 06255239.3, GB 0700468.2, GB 0712388.8, GB 0719833.6 and GB 0800364.2, in the initial stage of extraction of the image identifier the image is optionally pre-processed by resizing and filtering. The resizing step is used to normalise the images before processing. The filtering step can comprise of filtering to remove effects such as aliasing it can also include region selection and tapering. In one embodiment the image is resized to a resolution of 256xN or Nx256, where N ≥256 and preserving the aspect ratio. In another embodiment the image is resized to a square of 256x256. The image is then low pass filtered with a 3x3 Gaussian kernel. The pre-processing steps are optional and can include any combination of the above.

[0036] In the next stage feature points are detected in the image. One embodiment uses a feature detector to detect potential features in an image and their strength based on measuring image gradients. One suitable image feature detector based on first order image gradients is the Harris as used in GB 0800364.2


where I_x(x,y) is the first order derivative of the image intensity I(x,y) at location (x,y) and α is a parameter that controls feature detection.

[0037] An alternative second order detector is the Laplacian (which corresponds to the trace of the second order partial derivative Hessian matrix)


where I_xx(x, y) is the second order derivative of the image at location (xy).

[0038] In GB 0800364.2 the features are selected at just one scale, that is the normalised version of the original image. Furthermore the area around the feature, which forms an image region from which a feature signature is extracted, is fixed. However, under the cropping modification, part of the image is lost. When the image is geometrically normalised (as in the aforementioned pre-processing or by a user) a region becomes larger relative to the same region in the original image. Under heavy cropping the Trace transform based descriptors of these two regions becomes unacceptably different; the distance may be above the threshold for declaring similarity.

[0039] In order to address this problem, in accordance with an embodiment of the present invention, "scale estimation" may be used. Scale estimation attempts to select an appropriate scale for analysis of each feature. It should be noted that each feature will be associated with its own scale.

[0040] First, a scale-space representation of an image is derived. In embodiments, a scale-space representation of an image may be constructed in one of two ways (although other techniques are possible and contemplated). The first is to successively smooth and sample the original image, the second is to repeatedly smooth the image with increasing filter size. Features are then detected by finding maxima in the location-scale space. It has been shown that an optimal choice for the smoothing filter is a Gaussian (filter), an image smoothed by a Gaussian (filter) is denoted


where * is the convolution operator and G(σ_D) is a Gaussian filter and σ_D is the standard deviation of the smoothing filter that determines the scales.

[0041] In one embodiment, two detectors are used to provide good detection of the location and scale of features in the scale-space representation, one detector uses first order gradients, the other uses second order gradients.

[0042] Firstly, the modified scale corrected Laplacian of Gaussian operator


is used to detect a plurality of feature locations and scales (i.e. feature points in the scale-space representation). Where G(σ_I) is the Gaussian filter used for integration that averages the derivatives, σ_I is the integration scale and y is a scale normalisation parameter. The introduction of integration smoothing to the scale adapted Laplacian helps to improve the performance of the feature detector.

[0043] The second detector is the scale-adapted Harris operator




where

[0044] Accordingly, the Laplacian equation (4) applied in scale space, and the locations and scales with the highest value are then chosen as potential features. Then those features with a non-significant (i.e. below a threshold) Harris detector response are removed from the set of potential features. A selection procedure as described below is applied to select a representative set of features from the set of potential features resulting from the use of the two detectors. In an embodiment, up to a predetermined number of features are chosen.

[0045] The strongest feature, according to the Laplacian detector equation (4), from the allowable features amongst the detected features is selected first. A feature is rejected if it is closer to the edge of the image than the radius r computed from the feature scale, see equation (7). The subsequent features are selected by order of strength with the restriction that a feature is not selected if it is within an exclusion zone around any of the previously selected features. In one embodiment, a circular region with a radius of 10 pixels is used as the exclusion zone. The selection process terminates once a predetermined number of representative features (e.g. 32 features) has been selected or no more features remain.

[0046] A set of features of the image of Fig. 2, selected from those detected, as shown in Fig. 3, is shown in Fig. 4, the scale is represented by the size of the circle and the feature direction is represented by the line. It will be appreciated that the detected features and their surrounding image regions are representative of, or significant to, the image as a whole.

[0047] Now that the location of the feature (or interest) points has been selected, their position is preferably refined at the selected scale to sub-pixel resolution using for example a method such as described in OpenCV (FindComerSubPix) 03/01/2008., which can be obtained on the Internet at http://opencylibrary.sourceforge.net/CvReference, reference [1] infra).

Feature Region & Identifier Extraction

[0048] In GB 0719833.6, a method for extracting a binary image identifier or image representation is disclosed. In an embodiment of the present invention, the method of GB 0719833.6 is applied to extract a binary "feature identifier" from regions around feature points. Thus, each feature identifier is a binary representation of an image region associated with a detected feature. The image region may have a predetermined size. For example, in one embodiment a circular image region, having a radius of 32 pixels, centred on a feature point, is selected and the binary signature is computed for the image region. The circular region is preferably extracted to sub-pixel level accuracy, using cubic interpolation, for example.

[0049] Preferably, instead of using a fixed feature region radius, the scale at which the feature is detected is used to adapt the region size appropriately. The radius of the region to be selected at scale s is given by:


where r₀ is the base radius, σ₀ is the initial scale and κ is a constant that is a design parameter. It is also worth noting that a good choice for the integration scale is

[0050] In a preferred embodiment κ = 0.6, σ₀ = 1.14, r₀ = 20 s ∈ (0,1,...,S) and S = 14. The region determined by the feature location (x,y) and radius r is extracted to sub-pixel accuracy, using cubic interpolation. The region is then normalised to a predefined size, given by the normalised radius r̂ which is 32 in the preferred embodiment. The extracted region is then processed to derive a corresponding image identifier, using the method according to GB 0719833.6. A brief summary of this method is provided below.

[0051] A Trace transform T(d,θ) of the image region is performed by projecting all possible lines, parameterised by d, θ over the image and applying a first functional T over these lines. The result of the Trace transform may be decomposed to reduce its resolution in either or both of its dimensions d, θ in accordance with the method disclosed in GB 0700468.2. A second functional P may be applied to the columns of the Trace transform to give a vector of real numbers (i.e. a one dimensional function), as in the methods of EP 06255239.3 and GB 0700468.2. This second functional P is known as the diametrical functional and the resulting vector is known as the circus function.

[0052] A third functional, the circus functional, may be applied to the circus function to give a single number, although this step is not used in the preferred embodiment.

[0053] Alternatively, in accordance with the method of GB 0719833.6, a so-called "band-circus" function may be obtained, as an alternative to the circus function discussed above, which restricts the Trace transform to a subset of lines of the image, by selecting and processing values from only a part of the Trace transform.

[0054] The properties of the result can be controlled by appropriate choices of the first, second and/or third functional (Trace, diametrical and circus).

[0055] A binary identifier for the image region is extracted from the circus function (or the band-circus function) via a frequency representation. For example, a function c(ω) may be defined on the magnitude coefficients of the Fourier transform. One illustration of this function is taking the difference of neighbouring coefficients

[0056] A binary string can be extracted by applying a threshold to the resulting vector, such that

[0057] Suitable choices for S include S = 0 and S = mean(c). The binary representation of the image region is then made up of these binary string values B = {b₀,...,b_n}.

[0058] In an embodiment of the present invention, since local image regions are used (as opposed to entire images as in GB 0719833.6) modifications need to be made to the signature extraction technique described therein. In particular, two identifiers are extracted from the circus functions of the full Trace transform using the functionals:


and


in accordance with the method disclosed in EP 06255239.3.

[0059] A further six, "multi resolution" identifiers may be extracted from circus functions obtained by decomposing (or sub-sampling) the distance (d) parameter of the Trace transform by a factor of 8, 16 and 32 and applying both equations (10) and (11), in accordance with the method disclosed in GB 0700468.2.

[0060] A further three functions are selected from bands in the Trace transform, in accordance with the method disclosed in GB 0712388.8, and two functions are extracted from the trace-annulus functions and one function is extracted from the trace-circular function, in accordance with the method disclosed in GB 0719833.6.

[0061] Each of these 14 component identifiers is 8 bits, giving a total of 112 bits for each feature identifier.

[0062] Additionally, a scale adapted feature direction is also calculated that integrates over a neighbourhood for each feature

[0063] The direction θ is quantised to preferably 16 levels, which can be represented by 4 bits of data.

[0064] The image identifier or descriptor is formed from all or some of the feature signatures. An image identifier for the complete set of feature points may be represented in a number of ways. In a preferred embodiment the image descriptor is a bit stream containing a number of fields of information, the first field is the number of features. Then each feature is represented in the bitstream as the normalised feature point coordinates (x and y), the quantised feature direction, followed by the binary feature signature. Optionally, rather than store the number of features, padding could be used with a flag set to represent an unused field. The binary feature signature includes all 112 bits or optionally a subset of the 112 bits may be selected, derived or stored. The structure of feature identifiers is shown in Fig. 7(a) and the image signature in Fig. 7(b).

Image Matching using Feature Identifiers

[0065] Image matching may be performed, by comparing or matching feature signatures, which form part of the image identifiers. As mentioned previously, for efficiency the full matching procedure between a pair of images, by comparing feature signatures, is made up of four stages. Hypotheses are formed in stages one and three. A series of geometric tests are performed in stages two and four, these tests must be passed in order for a hypothesis to progress to the next stage. The stages become increasingly computationally complex so each stage aims to minimise the number of hypotheses that are accepted for subsequent processing.

[0066] A first step involves forming hypotheses for potential matching features by comparing the first M out of N feature signatures from one image with the first M out of N feature signatures from the second image. The advantage of using a subset of M is that it speeds up whilst maintaining robustness. In a preferred embodiment N is 32 and M is 24.

[0067] To perform matching between two signatures B₁ and B₂, the Hamming distance is taken:


where ⊗ is the exclusive OR (XOR) operator.

[0068] In a preferred embodiment, a set of hypotheses (candidate pairs of potentially matching features) is generated with all combinations of three pairs of matching features. A hypothesis is made up a set of three pairs of feature points (a₁,a₂,a₃,b₁,b₂,b₃). A set of three pairs of features is declared a match if the distance of each feature pair is below a constant threshold T_A and the cumulative distance is below some second constant threshold T_B. In a preferred hypothesis T_A = 0.06 and T_B = 0.2. The set of hypotheses is ordered by their cumulative distance, with the lowest distance (i.e. highest probability of being a match) first. In order to minimise complexity the number of hypotheses may be limited to the K hypotheses corresponding to the lowest distance. In a preferred embodiment K is 18.

[0069] A second step applies geometric constraints to each generated from stage one. Two geometric constraints are applied to reduce false matches, i) feature direction and ii) the ratio of line lengths.

[0070] As described above, a direction is associated with each feature, the direction for feature a_n is denoted θ_an. The difference between angles corresponding to features from the same image is taken to ensure that the angle distance measure is unaffected by a rotation. The distance measure must address the issue of left-right and top-bottom flip modification as is done here using these two measures of angle distance

[0071] Thresholds are applied to the two angle distance measures Ω₁, Λ₁ to test whether they are in allowable intervals. Allowable values for the angle distances are dependant on the number of levels in the angle quantisation Q and are given by


where q is a constant that can be used to vary the sensitivity to changes of angle. In a preferred embodiment q is set to 1 and Q is 16.

[0072] Using the locations of features selected in each hypothesis a ratio of line lengths between the features is used as a constraint, as shown in Fig. 5. By |a₁a₂| we denote the length of the line connecting the location of feature a₁ to the location of feature a₂. Three feature points |a₁a₂|, |a₂a₃| and |a₃a₁| are used to find the ratio of lines lengths. Three line length ratios are computed

[0073] Measures of similarity (G_1, G₂, G₃) between the line length ratios for two images A and B are defined as:

[0074] Advantageously, the ratio of line lengths between defined points in an image are invariant to transformations such as rotation, translation and scaling, such that the above geometric constraint is suitable for matching an image that has undergone such geometric transformations. However, other suitable geometric constraints may be used, consistent with design requirements.

[0075] In an embodiment two thresholds are applied to the distance measures, one to constrain the magnitude of the individual measures and the other to constrain the sum of the distances.

[0076] Any hypothesis that fails one or more of the tests described above is removed from the set of valid hypotheses. If after testing all hypotheses the set is empty the images are declared non-matching. If the hypothesis has passed all tests then it progresses to the next stage to generate four feature hypotheses.

[0077] The third stage of matching compares all N features from the first image with all N features from the second image. Now, there is already a hypothesis for three feature pairs so the aim is to find a set of hypotheses with four pairs, hypotheses that add one more feature to the current three feature pair hypothesis. Potential features are found based on the signature distance as in stage one. The set of hypotheses are ordered by the distance of the fourth feature.

[0078] In stage four geometric analysis is carried out on the sets of four feature hypotheses to further reduce false acceptance rates. The first geometric constraint is based on the feature direction equations (14) and (15) as for three features. The ratio of lines used previously is invariant to similarity transforms. With four features the invariance can be relaxed to the more general affine transformation by using the ratio of areas as described below. This allows tighter thresholds to be set which accept true matches whilst rejecting false matches.

[0079] From a set of four features (a₁,a₂,a₃,a₄), four triangles may be constructed (a₁a₂a₃,a₂a₃a₄,a₃a₄a₁,a₄a₁a₂), the areas of these triangles can be found from


where s = (|a₁a₂| + |a₂a₃| + |a₃a₁|)/2. Area ratios can then be calculated as


and the distances between an area ratio of image A and an area ratio for image B can then be found by

[0080] Two thresholds T_c and T_D are applied to the set of four area ratio distances between the feature sets. The first threshold rejects hypothesis which have any of the area ratios distance above a first threshold T_c. The second rejects hypothesis for which the sum of distances is larger than a threshold T_D. In a preferred embodiment T_c =0.15 and T_D =0.4

[0081] The image matching technique of the above described embodiments of the present invention may be combined with other image matching techniques, such as the method disclosed in GB 0719833.6. A descriptor is generated for each image that contains all of the individual identifiers. In a preferred embodiment the identifier bitstream contains two identifiers based on GB 0719833.6, and a descriptor according to the present invention. Of the two identifiers based on GB 0719833.6 the first is generated by pre-processing the image to maintain the image aspect ratio and the other where it is processed to a square.

[0082] Preferably the matching is performed first with the faster algorithm of GB 0719833.6 at a very low false alarm threshold. Any image pairs below the threshold are considered to be a match, and pairs above the threshold (that is not considered a match by GB 0719833.6) are then processed in accordance with an embodiment of the present invention.

[0083] An apparatus for implementing the present invention, according to an embodiment, for carrying the above methods is shown in Fig. 6. The implementation involves building a database of identifiers 240 for images stored in an image database 230. The two databases 230 and 240 may be the same or separated databases. The apparatus allows searching for an identifier 260 extracted from a query image 250 to find matches in the identifier database 140. A, possibly ordered, list of images is returned to the user 290 or a query application.

[0084] As the skilled person will appreciate, many variations and modification may be made to the described embodiments. For example, whilst the described embodiments identify image regions, from which the image signature is derived, based on feature detection, other techniques may be used to identify image regions that are representative of, or significant within, the image. Moreover, whilst the described embodiment uses a feature detector based on an image gradient method, many other suitable methods may be used for the detection of interest points or regions in an image. Similarly, other techniques for selecting the best or strongest detected feature or interest points are possible.

[0085] In addition, it is possible to use one or more of a variety of geometric constraints other than the distance measures relating to the ratio of lines lengths associated with a set of three feature points, used in the described embodiment. For example, any number of features points can be used in conjunction with distance measures. Alternatively, other geometric constraints may be based on angles between feature points. Moreover, an affine model may be used to define the geometric constraint.

Alternative implementations

[0086] Whilst the description above uses brute force searching to generate hypotheses in some situations it may be beneficial to use a structure that allows approximate Hamming distance calculations. This would provide increased search speed at the cost of some level of detection.

[0087] The feature matching approach presented is amenable to casting into a probabilistic model along the lines presented in [reference 4 infra].

REFERENCES

[0088] 

1. Open CV Reference Manual, Chapter 1: Image Processing Gradients, Edge and Comers - "FindComerSubPix": http://opencvlibrary.sourceforge.net/CvReference

2. D.G. Lowe, "Distinctive Image Features from Scale Invariant Keypoints", International Journal of Computer Vision, 60, 2, 2004. pp. 91-110.

3. D.G. Lowe, "Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image", US 6,711,293, Mar. 2000.

4. C. Schmid, "A Structured Probabilistic Model for Recognition", Proc. Computer Vision and Pattern Recognition, 2, 485-490, Jun. 1999.

Claims

1. A method for deriving a representation of an image by processing signals corresponding to the image, the method comprising:

deriving a scale-space representation of the image;

processing the scale-space representation to detect a plurality of feature points having amplitude extrema;

for one or more of the detected plurality of feature points, determining a scale-dependent image region associated with the corresponding feature point, and

deriving a representation for each determined, scale-dependent image region, the method comprising selecting a predetermined number of feature points from the detected plurality of feature points, and

determining a scale-dependent image region associated with each of the selected feature points,

and characterised in that

the spatial positions of the selected feature points are substantially distributed across the image, and the spatial position of each selected feature point is separated from the other selected feature points by at least a predetermined distance in the image, by selecting feature points by order of strength with the restriction that a feature point is not selected if it is within an exclusion zone around any of the previously selected features.

2. A method as claimed in claim 1, wherein the plurality of feature points are detected by applying an operator based on second order image gradients, to the scale-space representation of the image, and identifying those feature points in the scale-space representation with the highest resulting values.

3. A method as claimed in claim 1 or claim 2, where the step of selecting comprises removing from the detected plurality of feature points those feature points having the lowest amplitude values in the scale-space representation, or amplitude values in the scale-space representation below a threshold.

4. A method as claimed in any preceding claim, wherein the step of selecting comprises removing, from the detected plurality of feature points, feature points according to their spatial position in the image.

5. A method as claimed in any preceding claim, wherein the step of selecting comprises applying an operator to the detected plurality of feature points, the operator based on first order image gradients, and removing the feature points with resulting values below a threshold, for example, wherein the operator is a scale adapted Harris operator.

6. A method as claimed in any preceding claim, wherein the step of determining a scale-dependent image region for a feature point in the scale-space representation comprises determining a spatial region of the image associated with the spatial position of the feature point, wherein the size of the spatial region is based on the scale of the feature point.

7. A method as claimed in any preceding claim, further comprising:

before determining a scale-dependent image region, refining the spatial positions of the one or more feature points to sub-pixel resolution, preferably using cubic interpolation.

8. A method as claimed in any preceding claim, wherein the step of deriving a representation for each determined, scale-dependent image region, comprises deriving a representation based on line projections across the image region, preferably using the Trace transform, or equivalent.

9. A method as claimed in any preceding claim, wherein the step of deriving a representation for each determined, scale-dependent image region, comprises:

deriving a function of at least part of the image region; and

using the function to obtain an intermediate representation of the at least part of the image region.

10. A method as claimed in claim 9, comprising:

for each of the one or more determined scale-dependent image regions;

deriving a function of at least part of the image region, where the function of a translated, scaled or rotated version of an image region is a translated or scaled version of the function of the image region.

11. A method as claimed in claim 9 or claim 10, wherein:

the step of deriving a function of at least part of the image region comprises applying a first functional along lines over the image region, the method preferably further comprising,

applying a second functional to the derived function to obtain an intermediate representation of the at least part of the image region.

12. A method as claimed in any preceding claims, further comprising:

for the one or more of the plurality of feature points, determining a feature direction for the feature point, preferably by integrating over a neighbourhood of the feature point, for example,
wherein the feature direction is determined using the equation:


where: θ is the feature direction of the feature point at spatial position (x, y).

13. A method as claimed in any preceding claim, wherein the step of deriving a representation for each determined, scale-dependent image region comprises deriving a binary image descriptor for the scale-dependent image region.

14. A method for comparing images using descriptors of images derived using a method as claimed in any preceding claim, the method comprising:

comparing representations of one or more scale-dependent image regions associated with the descriptor of a first image, with representations of one or more scale-dependent image regions associated with the descriptor of a second image.

15. Apparatus comprising means adapted to execute the method of any one of claims 1 to 14.

Ansprüche

1. Verfahren zum Ableiten einer Darstellung eines Bildes durch die Verarbeitung von Signalen, die dem Bild entsprechen, wobei das Verfahren umfasst:

Ableiten einer Skalenraumdarstellung des Bildes;

Verarbeiten der Skalenraumdarstellung zum Ermitteln mehrerer Eigenschaftspunkte, die Amplitudenextremwerte aufweisen;

für einen oder mehrere der ermittelten mehreren Eigenschaftspunkte, Bestimmen eines skalenabhängigen Bildbereichs, der mit dem entsprechenden Eigenschaftspunkt verknüpft ist, und

Ableiten einer Darstellung für jeden bestimmten, skalenabhängigen Bildbereich, wobei das Verfahren das Auswählen einer vorbestimmten Anzahl an Eigenschaftspunkten aus den ermittelten mehreren Eigenschaftspunkten umfasst, und

Bestimmen eines skalenabhängigen Bildbereichs, der mit jedem der ausgewählten Eigenschaftspunkte verknüpft ist,

und dadurch gekennzeichnet ist, dass die räumlichen Positionen der ausgewählten Eigenschaftspunkte im Wesentlichen über das Bild verteilt sind, und die räumliche Position von jedem der ausgewählten Eigenschaftspunkte von den anderen ausgewählten Eigenschaftspunkten durch mindestens einen vorbestimmten Abstand im Bild getrennt ist, indem Eigenschaftspunkte in der Reihenfolge der Stärke mit der Einschränkung ausgewählt werden, dass ein Eigenschaftspunkt nicht ausgewählt wird, wenn er sich innerhalb eines Ausnahmebereichs um eine der zuvor ausgewählten Eigenschaften herum befindet.

2. Verfahren nach Anspruch 1, wobei die mehreren Eigenschaftspunkte durch Anwenden eines Operators basierend auf Bildverläufen zweiter Ordnung auf die Skalenraumdarstellung des Bildes ermittelt werden, und Identifizieren dieser Eigenschaftspunkte in der Skalenraumdarstellung mit den höchsten resultierenden Werten.

3. Verfahren nach Anspruch 1 oder Anspruch 2, wobei der Schritt des Auswählens das Entfernen dieser Eigenschaftspunkte, die die niedrigsten Amplitudenwerte in der Skalenraumdarstellung oder Amplitudenwerte in der Skalenraumdarstellung unterhalb eines Schwellenwerts aufweisen, von den ermittelten mehreren Eigenschaftspunkten umfasst.

4. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Auswählens das Entfernen von Eigenschaftspunkten von den ermittelten mehreren Eigenschaftspunkten gemäß ihrer räumlichen Position im Bild umfasst.

5. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Auswählens das Anwenden eines Operators auf die ermittelten mehreren Eigenschaftspunkte, wobei der Operator auf Bildverläufen erster Ordnung basiert, und das Entfernen der Eigenschaftspunkte mit resultierenden Werten unterhalb eines Schwellenwerts umfasst, beispielsweise wobei der Operator ein skalenadaptierter Harris-Operator ist.

6. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Bestimmens eines skalenabhängigen Bildbereichs für einen Eigenschaftspunkt in der Skalenraumdarstellung das Bestimmen eines Raumbereichs des Bildes, der mit der räumlichen Position des Eigenschaftspunkts verknüpft ist, umfasst, wobei die Größe des Raumbereichs auf der Skala des Eigenschaftspunkts basiert.

7. Verfahren nach einem der vorhergehenden Ansprüche ferner umfassend:

vor dem Bestimmen eines skalenabhängigen Bildbereichs, das Verfeinern der räumlichen Positionen des einen oder der mehreren Eigenschaftspunkte auf Subpixel-Auflösung vorzugsweise mithilfe von kubischer Interpolation.

8. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Ableitens einer Darstellung für jeden bestimmten, skalenabhängigen Bildbereich das Ableiten einer Darstellung basierend auf Linienprojektionen über den Bildbereich hinweg vorzugsweise mithilfe der Trace-Transformation oder einer Entsprechung umfasst.

9. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Ableitens einer Darstellung für jeden bestimmten, skalenabhängigen Bildbereich Folgendes umfasst:

Ableiten einer Funktion von mindestens einem Teil des Bildbereichs; und

Verwenden der Funktion zum Erhalten einer intermediären Darstellung von mindestens einem Teil des Bildbereichs.

10. Verfahren nach Anspruch 9 umfassend:

für jeden des einen oder der mehreren bestimmten, skalenabhängigen Bildbereiche;

Ableiten einer Funktion von mindestens einem Teil des Bildbereichs, wobei die Funktion von einer translatorisch bewegten, skalierten oder gedrehten Version eines Bildbereichs eine translatorisch bewegte oder skalierte Version der Funktion des Bildbereichs ist.

11. Verfahren nach Anspruch 9 oder Anspruch 10, wobei:

der Schritt des Ableitens einer Funktion von mindestens einem Teil des Bildbereichs das Anwenden einer ersten Funktion entlang den Linien über dem Bildbereich umfasst, wobei das Verfahren vorzugsweise ferner umfasst:

das Anwenden einer zweiten Funktion auf die abgeleitete Funktion zum Erhalten einer intermediären Darstellung von dem mindestens einen Teil des Bildbereichs.

12. Verfahren nach einem der vorhergehenden Ansprüche ferner umfassend:

für den einen oder mehrere der mehreren Eigenschaftspunkte, das Bestimmen einer Eigenschaftsrichtung für den Eigenschaftspunkt, vorzugsweise durch das Integrieren über einer Nachbarschaft des Eigenschaftspunkts, zum Beispiel,
wobei die Eigenschaftsrichtung mithilfe folgender Gleichung bestimmt wird:


wobei: θ die Eigenschaftsrichtung des Eigenschaftspunkts in einer räumlichen Position (x, y) ist.

13. Verfahren nach einem der vorhergehenden Ansprüche, wobei der Schritt des Ableitens einer Darstellung für jeden bestimmten, skalenabhängigen Bildbereich das Ableiten eines binären Bilddeskriptors für den skalenabhängigen Bildbereich umfasst.

14. Verfahren zum Vergleichen von Bildern mithilfe von Deskriptoren für Bilder, die mithilfe eines Verfahrens nach einem der vorhergehenden Ansprüche abgeleitet werden, wobei das Verfahren Folgendes umfasst:

das Vergleichen von Darstellungen von einem oder mehreren skalenabhängigen Bildbereichen, die mit dem Deskriptor eines ersten Bilds verknüpft sind, mit Darstellungen von einem oder mehreren skalenabhängigen Bildbereichen, die mit dem Deskriptor eines zweiten Bildes verknüpft sind.

15. Vorrichtung, die Mittel umfasst, die zum Ausführen des Verfahrens nach einem der Ansprüche 1 bis 14 angepasst sind.

Revendications

1. Procédé destiné à obtenir par dérivation une représentation d'une image en traitant des signaux qui correspondent à l'image, le procédé comprenant les étapes consistant à :

obtenir par dérivation une représentation d'espace échelle de l'image ;

traiter la représentation d'espace échelle de façon à détecter une pluralité de points caractéristiques qui présentent des extrema d'amplitude ;

pour l'un ou plusieurs de la pluralité de points caractéristiques détectés, déterminer une région d'image dépendante de l'échelle associée au point caractéristique correspondant ; et

obtenir par dérivation une représentation de chaque région d'image déterminée, dépendante de l'échelle, le procédé comprenant une étape consistant à sélectionner un nombre prédéterminé de points caractéristiques à partir de la pluralité de points caractéristiques détectés ; et

déterminer une région d'image dépendante de l'échelle, associée à chacun des points caractéristiques sélectionnés ;
et caractérisé en ce que
les positions spatiales des points caractéristiques sélectionnés sont sensiblement réparties à travers l'image, et la position spatiale de chaque point caractéristique sélectionné est séparée de celle des autres points caractéristiques sélectionnés par au moins une distance prédéterminée dans l'image, en sélectionnant des points caractéristiques par ordre de robustesse, sachant qu'un point de caractéristique n'est pas sélectionné s'il se trouve à l'intérieur d'une zone d'exclusion située autour de n'importe laquelle des caractéristiques sélectionnées précédemment.

2. Procédé selon la revendication 1, dans lequel la pluralité de points caractéristiques sont détectés en appliquant un opérateur basé sur des gradients d'image du second ordre à la représentation d'espace échelle de l'image, et en identifiant les points caractéristiques dans la représentation d'espace échelle qui présentent les valeurs les plus élevées qui en résultent.

3. Procédé selon la revendication 1 ou la revendication 2, dans lequel l'étape de sélection comprend une étape consistant à retirer de la pluralité de points caractéristiques détectés, les points caractéristiques qui présentent les valeurs d'amplitude les plus faibles dans la représentation d'espace échelle, ou les valeurs d'amplitude dans la représentation d'espace échelle qui se situent en dessous d'un seuil.

4. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape de sélection comprend une étape consistant à retirer de la pluralité de points caractéristiques détectés, des points caractéristiques selon leur position spatiale dans l'image.

5. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape de sélection comprend une étape consistant à appliquer un opérateur à la pluralité de points caractéristiques détectés, l'opérateur étant basé sur des gradients d'image du premier ordre, et à retirer les points caractéristiques qui présentent des valeurs en résultant qui se situent en dessous d'un seuil, par exemple, dans lequel l'opérateur est un opérateur de Harris adapté à l'échelle.

6. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape consistant à déterminer une région d'image dépendante de l'échelle d'un point caractéristique dans la représentation d'espace échelle, comprend une étape consistant à déterminer une région spatiale de l'image associée à la position spatiale du point caractéristique, dans lequel la taille de la région spatiale est basée sur l'échelle du point caractéristique.

7. Procédé selon l'une quelconque des revendications précédentes, comprenant en outre une étape consistant à :

avant l'étape consistant à déterminer une région d'image dépendante de l'échelle, affiner les positions spatiales du ou des point(s) caractéristique(s) à un résolution de pixel secondaire, de préférence en utilisant une interpolation cubique.

8. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape consistant à obtenir par dérivation une représentation de chaque région d'image déterminée, dépendante de l'échelle, comprend une étape consistant à obtenir par dérivation une représentation basée sur des projections de ligne à travers la région d'image, de préférence en utilisant la transformée trace, ou similaire.

9. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape consistant à obtenir par dérivation une représentation de chaque région d'image déterminée, dépendante de l'échelle, comprend les étapes consistant à :

obtenir par dérivation une fonction d'une partie au moins de la région d'image ; et

utiliser la fonction de façon à obtenir par dérivation une représentation intermédiaire de la ou des partie(s) de la région d'image.

10. Procédé selon la revendication 9, comprenant une étape consistant à :

pour chacune de la ou des région(s) d'image déterminée(s), dépendante(s) de l'échelle;

obtenir par dérivation une fonction d'une partie au moins de la région d'image, où la fonction d'une version translatée, mise à l'échelle ou tournée d'une région d'image est une version translatée ou mise à l'échelle de la fonction de la région d'image.

11. Procédé selon la revendication 9 ou la revendication 10, dans lequel :

l'étape consistant à obtenir par dérivation une fonction d'une partie au moins de la région d'image comprend une étape consistant à appliquer un premier fonctionnel le long de lignes au-dessus de la région d'image, le procédé comprenant en outre de préférence une étape consistant à ;

appliquer un second fonctionnel à la fonction obtenue par dérivation de façon à obtenir une représentation intermédiaire de la ou des partie(s) de la région d'image.

12. Procédé selon l'une quelconque des revendications précédentes, comprenant en outre une étape consistant à :

pour le ou les point(s) caractéristique(s) de la pluralité de points caractéristiques, déterminer une direction de caractéristique du point caractéristique, en intégrant de préférence un voisinage du point caractéristique, par exemple ;

dans lequel la direction de caractéristique est déterminée en utilisant l'équation :


dans laquelle : est la direction de caractéristique du point de caractéristique à la position spatiale (x, y).

13. Procédé selon l'une quelconque des revendications précédentes, dans lequel l'étape consistant à obtenir par dérivation une représentation de chaque région d'image déterminée, dépendante de l'échelle, comprend une étape consistant à obtenir par dérivation un descripteur d'image binaire de la région d'image dépendante de l'échelle.

14. Procédé destiné à comparer des images en utilisant des descripteurs d'images obtenus par dérivation en utilisant un procédé selon l'une quelconque des revendications précédentes, le procédé comprenant une étape consistant à :

comparer des représentations d'une ou de plusieurs régions d'image dépendantes de l'échelle, associées au descripteur d'une première image, à des représentations d'une ou de plusieurs régions d'image dépendantes de l'échelle associées au descripteur d'une deuxième image.

15. Appareil comprenant des moyens conçus de façon à exécuter le procédé selon l'une quelconque des revendications 1 à 14.

Drawing