The invention relates to image processing method and apparatus for automatically discriminating the direction of a document of an image read by a photoelectric converting device such as a CCD or the like.
Hitherto, in an information processing apparatus, characters are generally recognized from original information (image data) read by an optical reader such as a scanner or the like. However, the character recognition is performed on the assumption that the characters were certainly inputted so as to face the proper direction (namely, the characters are arranged in the proper direction for the scanner), so that the character recognition result exactly differs from the input original.
For example, in case of characters 1001 shown in Figs. 10A to 10D, when an image is inputted so that the character faces the proper direction as shown in Fig. 10A, a recognition result as a character 1011 is correctly obtained. However, in the case where an image is rotated by 270° and is inputted as shown in Fig. 10B, it is erroneously recognized as a character 1012. In the case where image data is rotated by 180° and the image is inputted as shown in Fig. 10C, it is erroneously recognized as a character 1013. In the case where image data is rotated by 90° and the image is inputted as shown in Fig. 10D, it cannot be recognized. Those recognition results are shown for convenience of explanation and the actual results don't always become the results as mentioned above.
In order to avoid the problem such that when the input image is character recognized as it is as mentioned above, so long as the direction of a document is not the proper direction, the correct result cannot be obtained, there is also the following method. Namely, the image inputted by the scanner is displayed and the user judges whether the image has been inputted in the proper direction or not. When the inputting direction is not the proper direction, the rotation of the image in an image memory is instructed so as to manually rotate the direction of the image by 90°, 180°, and 270° by the user, thereby rotating the image, or the setting direction of the original to the scanner is changed to the proper direction and the image is again inputted or the like, thereby correcting the image to the proper direction, and after that, the start of the character recognition is instructed.
However, owing to the improvement of a processing speed of the scanner, an automatic document feeding function called an auto feeder is annexed to the apparatus and a plurality of originals can be automatically read and inputted. The number of originals as targets to be processed can be set to a large quantity of papers nowadays. In such a situation, it is very difficult to perform operations for manually judging the proper direction one by one for a plurality of originals and for correcting the direction of the original image.
The contents of the document of the original is expressed by various patterns such as vertical writing, lateral writing, and the like. That is, for example, there are various kinds of patterns such as document of the A4 size which is vertically placed and is written in the lateral direction (see Fig. llA: for example, this pattern is often used in a lateral writing Japanese document, English document, or the like), document of the A4 size which is laterally placed and is written in the lateral direction (see Fig. 11B: for example, such a pattern is frequently used in a document having a long sentence, a document for the OHP, a document in case of reducing and copying a document of the A3 size, B4 size, or the like which is laterally placed, or the like), a document of the A4 size which is laterally placed and is laterally written and in which the column setting is switched at the center (see Fig. 11C: for example, such a pattern is often used in case of continuously reducing and copying two sheets of documents of the A4 size), a document of the A4 size which is vertically placed and is vertically written (see Fig. 11D), and the like.
On the other hand, the direction of an image which is displayed on a display screen is determined by the scanning direction (image reading direction) of the scanner when inputting the image. Therefore, in case of reading the original at a position such that the scanning direction of the scanner coincides with the direction of a character train of the document, the document is displayed at the inherent position.
However, in case of the scanner of the A4 size, since the direction of the original is unconditionally determined, the original cannot be read at the position such that the scanning direction of the scanner is made coincide with the direction of the character train of the document and, for example, as shown in Fig. 12B, the document is displayed in a state in which it is rotated by 90° from the inherent position, so that there is a case where it is very difficult to read the image.
Hitherto, there is a method of automatically discriminating the direction of the inputted original so as to reduce the operation to correct the direction of the original by the operator as mentioned above. Namely, as shown in Fig. 13A, there is a method whereby lines of a table, a separator (a), and the like are extracted from an inputted original image and it is judged that the direction of the lines is the lateral direction. As shown in Fig. 13B, there is a method whereby projections (histograms) in the vertical and lateral directions of a document are detected and the direction is judged by checking a degree of delimiter, or the like. According to the above methods, the direction is fairly arbitrarily decided. According to such methods, it is impossible to cope with documents of many kinds of formats as shown in Figs. 11A to 11D. For example, a condition such that the separator in the lateral direction exists in the document, the document is laterally written, or the like is needed. As for the document which doesn't satisfy such conditions, the direction of the original cannot be automatically discriminated.
It is known from US-A-5077811 to determine the orientation of character images by comparing each one of a set of character images with each one of a set of reference characters and accumulating matching data as the result of these comparisons. This process is repeated for each of four document directions and a determination as to the correct direction made on the basis of the accumulating matching data.
It is also known from JP-A-6103410 to provide detection of character direction by executing character recognition in four different character directions and comparing the number of characters recognised for each direction.
It is also known from US-A-5235651 to provide trial optical character recognition and to determine incorrect orientation if thirty percent of characters are unrecognisable.
According to the present invention there is disclosed, an image processing apparatus and method as defined in the independent claims.
By using such a method, the direction of the document can be automatically discriminated at a high precision irrespective of the format of the document.
An advantage of the invention is that, in the case where the document is not placed in the proper direction, by correcting the direction of the document by rotating the image information in accordance with the information in the proper direction which was automatically judged, a burden of the operation of the operator is eliminated and the image inputting operations up to at least the character recognition can be automated.
A further advantage is that, by automatically judging the direction of the original and performing the character recognition in the proper direction, a high recognition ratio can be obtained.
Embodiments of the present invention will now be described in detail hereinbelow with reference to the drawings.
Fig. 1 is a system constructional diagram of a data processing system to which a document direction automatic discriminating apparatus and a document direction automatic correcting apparatus according to the embodiment of the invention are applied. The system has an image input unit and an image processing unit and is realized using either one of a scanner, a copying apparatus, an intelligent input device and a sole input device which is connected to a computer through an interface to be made intelligent.
A scanner 1 is means for optically reading a document original and photoelectrically converting and inputting as digital image data. By attaching an auto feeder to the scanner 1, a plurality of originals can be continuously inputted. A CPU/memory 2 is provided for executing various kinds of controls and temporarily storing image data.
By paying attention to a point that a factor which most accurately expresses the direction of a document is a character, a character recognizer/direction discriminator 3 performs a character recognition to several kinds of character areas in the document from the directions of 0°, 90°, 180°, and 270° and sets the direction in which the precision is highest among the precisions of the character recognition in the respective directions (reliabilities of the character recognition: distances for a feature distribution of a character) to the document direction.
An area separator 4 is a block to execute processes for separating a character area, a figure area, a natural image area, a table area, and the like into rectangular areas from the document image data and adding an attribute (character area or the like) of each area as a pre-process for performing the character recognizing and direction discriminating processes by the character recognizer/direction discriminator 3.
A storage 5 is constructed by, for example, a hard disk, a magnetooptic disk, or the like and is used to preserve various processing results (image data, area separation result, character recognition result, etc.). An interface (I/F) 6 is constructed by an SCSI, an RS232C, or the like and is provided to transmit data to the outside. A computer 7 obtains and uses information through the I/F 6 or data from a movable storage such as a magnetooptic disk or the like. For the image data inputted from the scanner 1, a printer 8 prints and outputs image data or the like which was subjected to various kinds of processes on the basis of area separation information and character recognition information.
An outline of the processes for the document direction automatic discrimination and correction and the character recognition in the embodiment will now be described in accordance with a flowchart of Fig. 2.
The image data (binary image or multi-level image) which was inputted by the scanner 1 and stored in the memory 2 is first separated into rectangular areas by the area separator 4 for every attribute such as character area, figure area, natural image area, table area, etc. (steps S1, S2). In this instance, area information surrounded by a rectangle is actually formed. The area information is stored into the memory 2 by making coordinate data indicative of the area and attribute (character, figure, natural image, table, etc.) information of the area correspond to each other.
Subsequently, the area information in which the attribute information indicative of the character area among the attributes is made correspond is extracted from the memory 2 (step S3). The character area relates to a sentence area, a title area, a character in a table, a caption area of a diagram, or the like and is an image area including at least one character pattern. For example, in case of documents shown in Figs. 3A and 3C, rectangle information of character areas as shown in Figs. 3B and 3D is discriminated in step S2 and is extracted in step S3. The document direction is judged by using a few blocks in those rectangle information (step S4). Thus, when the document direction is the proper direction, the character recognizing process is also subsequently executed to the other character areas in the images stored in the memory 2 (step S7).
On the other hand, when the document direction is the improper direction, a correction to rotate the image data in the correct direction is executed (step S5). The area separating process similar to step S2 is executed to the rotated image and a process such that the area separation information which was extracted from the image before the direction is corrected and which has already been stored in the memory 2 is corrected to the area separation information extracted from the image whose direction was corrected is executed (step S6). In this process, a difference between the area separation information in association with the image rotation is corrected. As such a correcting method, there is a method of again executing the area separating process to the rotated image data obtained in step S5. There is also another method of performing an address conversion to the area separation information obtained in step S2 and stored in the memory 2. Since the area separating process generally assumes that an image as a target to be processed is placed in the proper direction, in many cases, the result of the area separating process which was performed at the initial stage and the result of the area separating process which was executed for the rotated image data are different. It is, accordingly, desirable to use the former method.
The processing routine advances to step S7 and a character area block in the rotated image data is subjected to the character recognition by the character recognition processing system. Thus, the area separation information and character recognition information are finally obtained with respect to both cases where there is no rotation and where there is a rotation (step S8).
The processing result is transmitted to the computer 7 through the I/F 6. A text of the character recognition result is displayed by a display unit of the computer and is used for an application software or the like of the filing on the computer 7. In the system to transfer the processing result to the storage 5, the result is used for a method which is used like a batch processing such that the image information is continuously inputted and the information is subsequently read out in a lump. Further, in the system to transfer the processing result to the printer 8, so long as the printer 8 has a function to interpret a page describing language, the processing result is used when a document is reconstructed or a fair copy is made by an inverse PDL (method of forming a page describing language by the image data) by the character recognizing process and area separating process.
In the case where a plurality of images are inputted in a lump, the document direction is discriminated with respect to each document, the document direction is stored into the memory 2 in correspondence to the page ID information of the image, and the image of each document can be also subjected to the character recognition in accordance with the page ID information in step S8.
A method of discriminating the document direction by using the character recognizing process will now be described.
The area separating process which is executed by the area separator 4 in step S2 will now be described.
Black pixels of the document image data stored in the memory 2 are detected, the area in which the black pixels exist is extracted by an outline tracing or labelling method, and area information as a group of images is formed. Subsequently, a black pixel density in the rectangular area, the presence or absence of the adjacent rectangular block, a ratio between the vertical and lateral dimensions of the rectangle, or the like is used as a judgment reference, thereby discriminating the attribute of the image such as character area (title, body, caption, etc.), figure area, natural image area, table area, or the like with respect to each image area.
The character recognizing process which is executed by the character recognizer 3 for the document direction discriminating process (S4) and the conversion of the image into the text (S7) will now be described.
As a method for the character recognizing process, there is a feature vector extracting and comparing method. For example, as shown in Fig. 4A, it is now assumed that a character area 41 including a character 40 was discriminated by the area separating process and was specified as a target of the character recognizing process. As a first stage, a character extracting process is executed with respect to the character area 41 (refer to Fig. 4B). The character extracting process is a process to extract a rectangle of one character and the rectangle is obtained by detecting a state of continuity of the black pixels or by getting a histogram. According to the process at the first stage, total six character patterns of characters 42 to 47 are extracted from the character area 41. As a second stage, one character is divided into (m × n) [for example, 64 × 64] pixel blocks and those blocks are extracted (refer to Fig. 4C). The distribution direction of the black pixels is extracted (direction vector information: refer to Fig. 4D) by using a window of (3 × 3) pixels among them.
Fig. 4D shows an example of a part of the direction vector information. By shifting the window of (3 × 3) pixels, tens of direction vector information are obtained. The feature vector information becomes a feature of the input character. Vector information similar to it has been also stored as standard data in a character recognition dictionary. The feature vector extracted from the input character and the contents of the character recognition dictionary which has previously been stored are compared. Candidate characters are sequentially extracted from the dictionary in accordance with the order from the character whose feature is closest to the feature vector extracted from the input character. In this case, the first candidate, second candidate, third candidate, ... are extracted in accordance with the order from the characters whose features are close to the feature vector. A similarity indicative of a closeness of the feature to the feature vector, namely, closeness of the distance to the character becomes a numerical value called a reliability (precision) of the result of the character recognition.
In this way, the reliability of the result of each character recognition is obtained. Each character direction discriminating process based on the reliability will now be described with respect to an example in case of performing the character direction discriminating process as for the character area 41 shown in Fig. 5A.
First, explanation will now be made with respect to processes such that the character 40 as a first character in the character area 41 is recognized in four directions and the direction of the character pattern 40 of the first character is discriminated from the reliability of the character recognition result derived in each direction.
Fig. 5A shows a sentence in the proper direction. Fig. 5B shows a sentence rotated by 270°. Attention is now paid to the character pattern 40 of the first character which is first extracted from the character area. In case of discriminating the character direction, as shown in Fig. 5C, the character recognition is executed with respect to one character 40 from four directions of 0°, 90°, 180°, and 270°, respectively. It is sufficient to set each rotational angle by changing a reading method of the image of the character rectangle from the memory 2 when the character recognition is performed. There is no need to particularly rotate the original.
As shown in Fig. 5C, the character recognition results at the respective rotational angles are different from each other. The temporary character recognition results and reliabilities for explanation are shown in Fig. 5C and the actual results and reliabilities do not always coincide them.
In Fig. 5C, when the character recognition is executed from 0°, the image is correctly recognized as a character 50 and the reliability is also equal to a high value of 0.90. In case of performing the character recognition from the direction rotated by 90°, the image is erroneously recognized as a character 51 and the reliability also decreases to 0.40. The reason why the erroneous recognition occurs and the reliability also deteriorates is because the character recognition is performed on the basis of the feature vector in case of seeing from the rotated direction. Similarly, when the character recognition is executed from the rotated directions of 180° and 270° as well, the erroneous recognition occurs and the reliability deteriorates. As the character is complicated, the difference between the reliabilities for the directions of the character recognition typically appears.
As for the result of Fig. 5C, since the reliability of 0° is the highest, it is judged that a possibility such that the document faces in the proper direction is high. To further improve the precision of the discrimination about the character direction, the character recognition is similarly executed from four directions with respect to a plurality of characters in the same block. For example, in case of the character area 41 shown in Fig. 4A, the character recognition is performed from four directions with respect to each character of not only the character 42 but also the characters 43 to 47. Further, in the case where the character direction is judged in only one block, since there is a possibility such that the character direction is erroneously judged with respect to a special character train, a similar character recognition can be also executed with respect to a plurality of blocks.
It is also possible to construct in a manner such that with regard to each block, the average values of the reliabilities for every four directions of each of the recognition target characters in the relevant block are obtained, the direction in which the highest average value among them is derived is judged as a character direction of the block, and such a direction is decided as a document direction of the input image. Further, the character pattern of the character direction discrimination is selected from a plurality of character areas, the mean values for the average values of the reliabilities for every four directions in each block is obtained, and the direction in which the mean value is the highest among them is recognized as a character direction (document direction).
As mentioned above, by recognizing the character direction on the basis of a plurality of characters in the same block and, further, on the basis of the reliabilities of a plurality of characters in the same block without recognizing the character direction by the reliability of only one character, the character (document) direction can be discriminated at a high precision. It will be obviously understood that even if the character direction is discriminated by the reliability of only one character or is judged by the reliabilities of a plurality of characters in the same block, the character direction can be discriminated at a higher precision than the conventional one.
When the discrimination result of the character direction (document direction) indicates the direction other than the proper direction, the original image stored in the memory 2 is rotated so that the character direction is set to the proper direction. The rotation can be easily executed by the well-known technique by using the CPU/memory 2 in Fig. 1 and its description is omitted here.
By the processes as mentioned above, original image data shown in Fig. 6A, area separation data shown in Fig. 6B, and character recognition information shown in Fig. 6C can be obtained. Those information is used by an application such as electronic filing, document modification, DTP, or the like as mentioned above.
A format of the area separation data which is obtained by the area process in step S2 and is stored in the memory 2 is constructed by "header" indicative of the area separation data and identifiers "rectl" to "rect4" of the separated areas as shown in Fig. 6B. Information of each area (block) classified by those identifiers is constructed by: a number "order" of the block; an attribute (character area, figure area, etc.) "att" of the block; coordinate values "x1" and "y1" at the left upper position of the block; a width "w" of the block; a height "h" of the block; "direction" indicating that the document is written in the vertical or lateral direction; "SelfID" as an ID of the relevant block; "upperID" as an ID of a master block including the relevant block; an attribute "upperAtt" of the master block; and a reserved area "reserve".
As shown in Fig. 6C, the character recognition information has "header" indicative of the character recognition information and is constructed by combination information of character recognition information "OCR1" or the like regarding a single character such as "character 40" or the like and "blk header" corresponding to the above identifier "rect1" or the like indicative of the block including the relevant character.
Each character recognition information such as "OCR1" or the like is constructed by: "type" indicative of a character or a blank; first to fifth candidate characters "characterl" to "character5" according to the reliability of the character recognition mentioned above; extracting positions "x1" and "y1" of the character; a width "w" of the character; a height "h" of the character; and a reserved area "reserve".
According to the embodiment, in the case where the document doesn't face the proper direction as a result of the discrimination of the document direction and the image has to be rotated, the whole original image data is rotated and the character recognition is executed. However, the character recognition can be also performed without rotating the original image data.
In this case, when the rectangle information of the character area is obtained, as shown in Fig. 7, it is sufficient that only the character rectangle data is rotated and copied into another memory and the character recognition is executed for the rectangle image data.
As shown in Fig. 8, it is also possible that a character extraction is executed for the character rectangular area without rotating the image data and when each character is read out, it is read out in a state in which the reading direction is rotated, and the character recognition is performed (refer to step S9 in Fig. 9). Since the other steps in a flowchart of Fig. 9 are almost similar to those mentioned in Fig. 1, they are designated by the same step numbers and their descriptions are omitted here.
In the flowchart of Fig. 2, all of the processes from the input of the image to the process to form the text of the image data by the character recognition have been described as a series of processes. However, the image data in which it is judged that the document direction is the proper direction as a result of the discrimination of the document direction in S4 and the image data as a result in which the image was rotated (S5) and corrected to the proper direction are outputted by display means (not shown) or the printer 8. Thus, the user can automatically obtain the image data in the proper direction and the subsequent working efficiency can be raised.
The invention is not limited to each of the above embodiments but can be also applied to, for example, documents written by other languages instead of limiting to only Japanese. Particularly, in case of the document written in English or the like, since there is no document written vertically, the document direction can be discriminated at a higher precision.
The document direction discriminating process in the case where the original as a target of the judgment about the document direction is written in English will now be described with reference to Figs. 14A, 14B, and 15A to 15C.
An image 1400 inputted in S1 is stored into the memory 2 and four image areas 1401 to 1404 are extracted by the area separator 4 (S2). It is judged that three areas 1401 to 1403 among the extracted image areas are character areas and one area 1404 is a figure area from the feature of the image.
In the detection of the character area in S3, the area information of the three image areas 1401 to 1403 which were judged as character areas in S2 is read out from the memory 2 and the process to discriminate the document direction from the image data of each area is executed (S3).
For example, when the character patterns are extracted from the image data in the character area 1401, five characters 1411 to 1415 are extracted as shown in Fig. 14B. Among them, the results obtained by recognizing the first character "T" from four directions are as shown in Fig. 15C. Correctly, when the character 1411 of "T" is recognized from 0°, the result such that the recognized character (first candidate character) is "T" and the reliability is equal to 0.95 is obtained. When the character 1411 is recognized from 90°, the result such that the recognized character is "→" and the reliability is equal to 0.40 is obtained. Similarly, in case of 180°, the recognized character is "1" and the reliability is equal to 0.50. In case of 270°, the recognized character is "←" and the reliability is equal to 0.40.
From the above results, the recognized character "T" with the highest reliability corresponds to the correct recognition result of the character 1411. The rotational angle 0° at which the correct recognition result is obtained is judged to be the direction of the character 1411.
It is also possible to discriminate that the direction judged here is set to the document direction of the image 1400.
Or, in order to discriminate the document direction at a higher precision, it is also possible to construct in a manner such that the character recognition is performed with respect to all of the characters 1411 to 1415 included in the character area 1401 selected from all of the character areas (1401 to 1403) extracted in step S2 and the average value of the reliabilities is obtained in each direction and the direction in which the average value is the highest among them is judged as a direction of the document. It is also possible to selectively perform the character recognition to, for example, only the first and fifth characters in accordance with a predetermined order instead of executing the character recognition with regard to all of the characters 1411 to 1415. In this case, the characters "T" and "E" are recognized.
Or, in order to discriminate the document direction at a further high precision, it is also possible to select and recognize the characters from all of the character areas extracted in S2 and to discriminate the document direction by using the recognition results. In this case, there is used a method whereby the average value of the reliabilities of all of the recognition results is obtained for each direction and the direction in which the average value is the largest among them is discriminated as a document direction. Or, there is used a method whereby the average value of the reliabilities for each direction is obtained every area and the mean value obtained of the average values of each area is obtained with respect to all of the areas and the direction in which the mean value is the highest among them is discriminated as a document direction.
The characters to be recognized can be selected at random or can be sequentially selected in accordance with a predetermined order as mentioned above. For example, now assuming that the first and fifth characters are selected in a manner similar to the above, "T" and "E" are selected from the character area 1401, "I" and "T" are selected from the character area 1402, and "D" and "M" are selected from the character area 1403. The document direction of the image 1400 is discriminated in accordance with the results df the character recognition of those six characters.
In case of English, since only the documents written in the lateral direction exist, the direction in which the image in the character area extracted by the area separating process is placed laterally assumes 0° in accordance with the shape of the extracted character area. Since the documents of 90° and 270° cannot exist, the character recognizing process is omitted. The character recognition is performed with respect to only 0° and 180° and the document direction can be judged by checking which reliability is higher.
Since the character recognizing process can be reduced by the above processing procedure, the processing time for document discrimination can be reduced.