Technical Field
[0001] The present invention relates to a medical image processing device and a medical
image processing method that are suitable for detecting a lesion candidate region
from a medical image to detect a lesion such as a polyp.
Background Art
[0002] In recent years, endoscopes have been widely used, for example, for medical examination
and diagnosis in the field of medical treatment.
[0003] In such cases, the physician inserts the insertion portion of the endoscope inside
a body cavity such as the colon of the patient to pick up images inside the body cavity
using image pickup means provided at the distal end portion of the insertion portion.
The physician performs endoscopy such as an examination or diagnosis of a lesion such
as a polyp by observing the endoscopic images that are displayed on a monitor. In
this case, since it is desirable that the physician causes little pain or distress
to the patient and also performs endoscopy smoothly, the burden on the physician increases.
[0004] Therefore, for example, Japanese Patent Application Laid-Open Publication No.
2004-180932 as a first example of the prior art discloses an arrangement in which lesion candidates
for the same region of interest are detected by a first image diagnosis apparatus
comprising an X-ray CT apparatus or the like and a second image diagnosis apparatus
such as an X-ray TV apparatus, respectively. A detection result in which these two
detection results are compared and combined is then shown to the physician. It is
thereby possible to prevent an oversight by the physician.
[0005] Further, as a second example of the prior art, Japanese Patent Application Laid-Open
Publication No.
2005-192880 discloses an image processing method that detects a lesion candidate region from
an endoscopic image based on color tone information.
[0006] However, the above described first example of the prior art is difficult to apply
in the case of diagnosing or screening for a lesion such as a polyp from endoscopic
images picking up inside a body cavity such as the colon. Further more, with images
captured by X-ray CT, detection is difficult in a case in which there is no distinctive
hemispherical protuberant finding.
[0007] In images in which the inside of a body cavity is optically observed, such as in
endoscopic images, since color information can also be obtained, it is desirable to
also utilize the color information for lesion detection and the like.
[0008] The second example of the prior art is arranged to utilize color information and
contour information to detect hemorrhaging and reddening as well as elevations and
concavities and the like. However, this second example of the prior art does not effectively
utilize information regarding a region that incidentally presents attributes of color
tone changes of reddening or discoloration arising in a polyp or a region that incidentally
presents attributes of a region with an abnormal finding that arises in mucous membrane
on the periphery of a polyp. Therefore, it is desirable to effectively utilize information
of these regions to enable detection of lesions with greater accuracy and high reliability.
[0009] It is also desirable to generate a three-dimensional shape image from a two-dimensional
image to enable lesion detection with improved reliability.
[0010] The present invention has been accomplished in consideration of the above described
points, and an object of the invention is to provide a medical image processing device
and a medical image processing method that can accurately detect a lesion such as
a polyp from a medical image having color information such as an endoscopic image.
[0011] US Patent Publication No. 2003/0048931 discloses a method for developing an automated feature identification and quantification
involving a cascading sequence of segmentation. Within a given tissue-containing image,
necrotic areas are segmented into objects based on a histogram of a saturation band
and a prescribed parameter, such as a percentage of pixels of an image that possible
necrosis is found. The resulting objects are binarized, filled, and a close operation
is performed. The boundaries of the resulting objects are measured and those objects
larger than a prescribed cutoff parameter are classified as necrotic areas.
[0012] A further object of the present invention is to provide a medical image processing
device and a medical image processing method that can carry out lesion detection with
improved reliability by utilizing a three-dimensional shape image.
Disclosure of the Invention
Means for Solving the Problem
[0013] A medical image processing device according to the present invention comprises the
features as defined in claim 1.
[0014] Further, a medical image processing method according to the present invention includes
the steps as defined in claim 4.
Brief Description of the Drawings
[0015]
Fig. 1 is a block diagram that illustrates the configuration of an endoscope system
according to Embodiment 1 of the present invention;
Fig. 2 is a view that illustrates a state in which an endoscope is inserted into a
tubular region such as a colon to pick up images according to Embodiment 1 of the
present invention;
Fig. 3A is a view that illustrates an example of an endoscopic image that is picked
up by the image pickup apparatus provided in the endoscope shown in Fig. 2;
Fig. 3B is a view that illustrates an example of an endoscopic image that is picked
up by the image pickup apparatus provided in the endoscope shown in Fig. 2;
Fig. 4 is a block diagram that illustrates image processing functions performed by
a CPU according to Embodiment 1 of the present invention;
Fig. 5 is a flowchart that illustrates a process to detect a polyp as an elevated
lesion by image processing according to Embodiment 1 of the present invention;
Fig. 6 is a flowchart that illustrates a process to calculate three-dimensional shape
information at step S4 in Fig. 5;
Fig. 7 is a view that illustrates curved shapes according to values of a shape index
SI that are calculated according to step S12 in Fig. 6;
Fig. 8 is a flowchart that illustrates the processing contents at step S6 in Fig.
5;
Fig. 9A is a view that illustrates an example of an endoscopic image that is picked
up by an image pickup apparatus provided in an endoscope according to Embodiment 2
of the present invention;
Fig. 9B is a view that illustrates an example of an endoscopic image that is picked
up by an image pickup apparatus provided in an endoscope according to Embodiment 2
of the present invention;
Fig. 10 is a block diagram that illustrates image processing functions performed by
a CPU according to Embodiment 2 of the present invention;
Fig. 11 is a flowchart that illustrates a process that detects a polyp as an elevated
lesion by image processing according to Embodiment 2 of the present invention;
Fig. 12 is a flowchart that illustrates the processing contents of a mucosal attribute
information judgment at S51 in Fig. 11; and
Fig. 13 is a flowchart that illustrates the processing contents of an abnormal vascular
hyperplasia judgment at step S51 in Fig. 11.
Best Mode for Carrying Out the Invention
[0016] Hereunder, embodiments of the present invention are described referring to the drawings.
(Embodiment 1)
[0017] Fig. 1 to Fig. 8 relate to Embodiment 1 of the present invention. Fig. 1 is a block
diagram that illustrates the configuration of an endoscope system. Fig. 2 is a view
that illustrates a state in which an endoscope is inserted into a tubular region such
as a colon to pick up images. Fig. 3A is a view that illustrates an example of an
endoscopic image that is picked up by the image pickup apparatus provided in the endoscope
shown in Fig. 2. Fig. 3B is a view that illustrates an example of an endoscopic image
that is picked up by the image pickup apparatus provided in the endoscope shown in
Fig. 2. Fig. 4 is a block diagram that illustrates image processing functions performed
by a CPU. Fig. 5 is a flowchart that illustrates a process to detect a polyp as an
elevated lesion by image processing. Fig. 6 is a flowchart that illustrates a process
to calculate three-dimensional shape information at step S4 in Fig. 5. Fig. 7 is a
view that illustrates curved shapes according to values of a shape index SI that are
calculated according to step S12 in Fig. 6. Fig. 8 is a flowchart that illustrates
the processing contents at step S6 in Fig. 5.
[0018] An endoscope system 1 shown in Fig. 1 includes an endoscopic observation apparatus
2, an endoscopic image processing device (hereunder, referred to simply as "image
processing device") 3 such as a personal computer that performs image processing with
respect to an endoscopic image obtained by the endoscopic observation apparatus 2,
and a display monitor 4 that displays an image that is subjected to image processing
by the image processing device 3.
[0019] The endoscopic observation apparatus 2 includes an endoscope 6 that is inserted inside
a body cavity, a light source 7 that supplies an illumination light to the endoscope
6, a camera control unit (abbreviated as "CCU") 8 that performs signal processing
with respect to image pickup means of the endoscope 6, and a monitor 9 that displays
an endoscopic image that is picked up with the image pickup device by input of a video
signal that is outputted from the CCU 8.
[0020] The endoscope 6 includes an insertion portion 11 that is inserted into a body cavity,
and an operation portion 12 that is provided at the rear end of the insertion portion
11. A light guide 13 that transmits an illumination light is passed through the inside
of the insertion portion 11.
[0021] A light source 7 is connected to the rear end of the light guide 13. An illumination
light that is supplied from the light source 7 is transmitted by the light guide 13.
The transmitted illumination light is emitted from a distal end face of an illumination
window provided in a distal end portion 14 of the insertion portion 11 and illuminated
onto an object such as a diseased part.
[0022] An image pickup apparatus 17 comprises an observation lens 15 that is mounted in
an observation window adjoining the illumination window and, for example, a charge
coupled device (abbreviated as "CCD") 16 as a solid state image pickup device that
is disposed at an image formation position of the observation lens 15. An optical
image that is formed on the image pickup surface of the CCD 16 is subjected to photoelectric
conversion by the CCD 16.
[0023] The CCD 16 is connected to the CCU 8 via a signal wire. When a CCD driving signal
is applied from the CCU 8, the CCD 16 outputs an image signal that has undergone photoelectric
conversion. The image signal is subjected to signal processing by a video processing
circuit inside the CCU 8 and converted into a video signal. The video signal is outputted
to the monitor 9 to display an endoscopic image on the screen of the monitor 9. The
video signal is also inputted to the image processing device 3.
[0024] The image processing device 3 includes an image inputted section 21 into which is
input a video signal corresponding to an endoscopic image that is inputted from the
endoscopic observation apparatus 2, a CPU 22 as a central processing unit that performs
image processing corresponding to image data that is input from the image input section
21, and a processing program storage section 23 that stores a processing program (control
program) that causes the CPU 22 to execute image processing.
[0025] The image processing device 3 also includes an image storage section 24 that stores
image data and the like that is inputted from the image inputted section 21, an information
storage section 25 that stores information and the like that is processed by the CPU
22, a hard disk 27 as a storage device that stores image data and information and
the like that is processed by the CPU 22 via a storage device interface 26, a display
processing section 28 that performs display processing for displaying image data and
the like that is processed by the CPU 22, and an input operation section 29 comprising
a keyboard or the like through which a user inputs data such as image processing parameters
or an instruction operation.
[0026] A video signal that is generated by the display processing section 28 is displayed
on the display monitor 4 to thereby display a processed image that has undergone image
processing on the screen of the display monitor 4. In this connection, the image input
section 21, the CPU 22, the processing program storage section 23, the image storage
section 24, the information storage section 25, the storage device interface 26, the
display processing section 28, and the input operation section 29 are mutually connected
via a data bus 30.
[0027] According to the present embodiment, as shown in Fig. 2, an insertion portion 11
of a forward-viewing endoscope 6 is inserted into, for example, a tubular region (tubular
organ) such as a colon 31 and images are picked up by the image pickup apparatus 17.
[0028] Fig. 3A and Fig. 3B illustrate examples of a two-dimensional endoscopic image having
a polyp as an elevated lesion that is picked up by the endoscope 6.
[0029] Fig. 3A is a view that shows an endoscopic image having a polyp accompanied by a
reddened color tone portion 32. The top part of the polyp is a discolored portion
33 that is a white color.
[0030] Fig. 3B is a view showing an endoscopic image having a polyp accompanied by a discolored
portion 33 that is a white color in a wider area than in the example shown in Fig.
3A.
[0031] With respect to a polyp as an elevated lesion of this kind, cases often occur in
which a color tone change such as a reddening tone or a white tone (discoloration)
occurs as an incidental attribute. According to the present embodiment, the existence
or non-existence of a color tone change region that presents this kind of color tone
change as an incidental region is detected or judged. Further, a detection standard
to be utilized when performing polyp detection or judgment thereafter is changed (or
controlled) in accordance with the detection result for the color tone change region.
More specifically, the threshold value of the detection standard is changed.
[0032] Further, according to the present embodiment, to improve the accuracy of the above
described polyp detection, a three-dimensional shape of an examination target region
(for example, an inner surface having a luminal shape) is estimated based on the two-dimensional
endoscopic image that is picked up. Using the information for the estimated three-dimensional
shape, a region with an elevation change is detected as a polyp candidate.
[0033] With respect to a portion corresponding to a detected region with an elevation change
(including the periphery thereof), the threshold value of the detection standard for
performing polyp detection is changed in accordance with the result of detection of
an incidental region that is accompanied by the above described color tone change.
[0034] More specifically, when accompanied by a color tone change, as the detection standard
to be applied when performing the polyp detection according to an elevation change,
a detection standard is applied for which the detection conditions are more relaxed
than in a case that is not accompanied by a color tone change.
[0035] As described later, in a case in which polyp detection is performed using the method
illustrated in Fig. 5, polyp detection (judgment) is performed according to an elevation
change by employing a detection standard 1 as a threshold value in a case that is
not accompanied by a color tone change, and employing a detection standard 2, for
which the detection conditions are more relaxed compared to the detection standard
1, as a threshold value in a case that is accompanied by a color tone change.
[0036] By performing polyp detection with image processing in this manner, a polyp detection
result can be obtained accurately (or with high reliability).
[0037] Fig. 4 is a view that illustrates image processing functions that are performed by
the CPU 22 inside the image processing device 3 of the present embodiment. The CPU
22 has a region with elevation change detection function 22a as lesion candidate region
detecting means that detects a region with an elevation change as an elevated lesion
candidate region based on contrast information (or luminance information) from endoscopic
image data that is picked up, a region with color tone change detection function 22b
as incidental region detecting means that detects a color tone change region from
a plurality of color signals of an endoscopic image, a polyp detection function 22d
that detects a polyp as an elevated lesion with respect to a region with an elevation
change that is detected, and a detection standard setting function 22c as detection
standard changing means that changes a polyp detection standard depending on whether
or not there is an accompanying color tone change region when performing the polyp
detection.
[0038] The CPU 22 also has a three-dimensional shape information generating function 22e
that generates three-dimensional shape information by using luminance information
of a two-dimensional endoscopic image as shape information with which to estimate
the shape thereof in a case where the CPU 22 detects a region with an elevation change
using the region with elevation change detection function 22a.
[0039] Next, the operations that are carried out until polyp detection is performed based
on an endoscopic image according to the present embodiment are described.
[0040] In the present embodiment, the functions shown in Fig. 4 are implemented by software.
More specifically, the CPU 22 reads out a processing program that is stored (held)
in the processing program storage section 23, and by performing processing according
to this processing program the CPU 22 executes image processing for polyp detection
shown in Fig. 5.
[0041] When the power of the endoscopic observation apparatus 2 and the image processing
device 3 shown in Fig. 1 is turned on, the image pickup apparatus 17 of the endoscope
6 picks up an image, and (a video signal of) an endoscopic image that underwent signal
processing at the CCU 8 is displayed on the monitor 9 and is also inputted to the
image processing device 3 via the image input section 21.
[0042] When a physician operates an image processing start key or the like of an unshown
keyboard or the like, that instruction signal is sent to the CPU 22. Thereupon, the
CPU 22 starts image processing to perform the polyp detection illustrated in Fig.
5.
[0043] First, at step S1, the CPU 22 sets a parameter "i" that indicates the number of an
endoscopic image I to the initial value i=1. Next, at step S2, the CPU 22, for example,
loads an endoscopic image Ii comprising an i
th RGB signal that is sequentially stored in the hard disk 27 or the image storage section
24 shown in Fig. 1.
[0044] The series of processes according to the present embodiment are applied to an endoscopic
image Ii of respective frames that are inputted consecutively as a moving image. However,
the present invention is not limited thereto, and for example, a configuration may
be employed in which the image processing shown in Fig. 5 is performed on, for example,
endoscopic images at intervals of every several frames. Further, a configuration may
also be employed in which the image processing shown in Fig. 5 is performed on endoscopic
images that are filed in the hard disk 27 or the like.
[0045] After step S2, at step S3 the CPU 22 extracts an R-signal component (abbreviated
as "R image") Ri in the endoscopic image Ii. Next, at step S4, the CPU 22 calculates
(generates) three-dimensional shape information of the R image Ri based on a change
in contrast (change in luminance information). Although according to the present embodiment
the CPU 22 calculates the three-dimensional shape information using the R image Ri,
additionally, a configuration may be adopted in which the CPU 22 calculates the three-dimensional
shape information by using, for example, an image of a luminance signal component.
[0046] A processing method that calculates three-dimensional shape information based on
the two-dimensional R image Ri is illustrated in Fig. 6.
[0047] To calculate three-dimensional shape information on the basis of a two-dimensional
image, the CPU 22 estimates (generates) a three-dimensional shape as shown at step
S21 in Fig. 6.
[0049] Next, at step S22, the CPU 22 calculates a shape index SI and a curvedness CV as
feature quantities that represent a curved shape at each curved face of the calculated
three-dimensional shape. The shape index SI and curvedness CV are calculated as described
below.
[0050] With respect to a calculated three-dimensional shape, the CPU 22 calculates first
order partial differential coefficients fx, fy, and fz and second order partial differential
coefficients fxx, fyy, fzz, fxy, fyz, and fxz for an R pixel value f in a local region
(curved face) including the three-dimensional position (x, y, z) of interest.
[0052] The principal curvatures k1 and k2 (k1 ≥ k2) of the curved face are expressed using
the Gauss curvature K and mean curvature H as

[0053] Further, the shape index SI and curvedness CV as a feature value that represents
a curved shape in this case are, respectively,

[0054] Thus, the CPU 22 calculates the shape index SI and the curvedness CV for each three-dimensional
curved face as three-dimensional shape information.
[0055] The shape index SI is expressed by an index having values from 0 to 1 for curved
shapes as shown in Fig. 7. In this case, when the shape index SI is 0, the curved
shape is concave, and when the shape index SI is 1 the curved shape is convex. That
is, the closer the value of the shape index SI is to 1, the more features of a convex
elevated shape the region in question has.
[0056] Accordingly, by setting a detection standard as a threshold value having the shape
index SI value is close to 1, and detecting a region having a shape index SI value
larger than this threshold value, it is possible to objectively detect a polyp as
an elevated lesion in image processing. In this connection, according to detection
standard 1 used at step S 10 of Fig. 5 that is described later, (SI =) 0.9 is set
as the shape index SI threshold value, and according to detection standard 2 used
at step S11, (SI =) 0.8 is set as the shape index SI threshold value.
[0057] The curvedness CV represents the inverse number of the curvature radius, and is utilized
when limiting the size of a convex shaped region of the target curved face. The processing
at step S4 in Fig. 5 is performed in this manner.
[0058] According to the present embodiment, when performing image processing for detecting
a polyp as an elevated lesion as described below, the CPU 22 uses detection standard
1 and detection standard 2.
[0059] At step S5 in Fig. 5, the CPU 22 extracts an R image Ri in the endoscopic image Ii
and a G-signal component (abbreviated as G image Gi) in the endoscopic image Ii. Next,
at step S6, the CPU 22 performs detection of a color tone change region to detect
a region showing a color tone change involving reddening or discoloration.
[0060] The processing to detect a region having a color tone change at step S6 is, for example,
performed as illustrated in Fig. 8. At the initial step S31, the CPU 22 performs processing
to exclude unsuitable pixels such as dark portions, halation or residue from the R
image Ri and the G image Gi. To exclude dark portions and halation, exclusion can
be simply performed using a threshold value that is set in correspondence to the dark
portion and halation. For the residue, it is possible to remove the residue by combining
residual color tones with a shape judgment for that portion.
[0061] Next, at step S32, the CPU 22 performs processing to divide the region in question
into, for example, 8x8 sub-regions.
[0062] At step S33, the CPU 22 sets a parameter m that represents the number of a sub-region
and a parameter j that represents the number of a pixel inside the sub-region to an
initial value 1. In this connection, when numbering the sub-regions, when the proportion
of pixels that are excluded by the processing of step S31 that are inside the sub-region
exceeds, for example, 50%, that sub-region is excluded from the processing objects
(an m number is not allocated thereto). More specifically, unsuitable sub-regions
are also excluded from the processing objects.
[0063] At step S34, the CPU 22 calculates the chromaticity gj/rj of the j
th pixel. In this case, gj represents the luminance level of the j
th pixel inside a (m
th) sub-region of the G image Gi, and rj represents the luminance level of the j
th pixel inside a (m
th) sub-region of the R image Ri. Next, at step S35, the CPU 22 determines whether or
not j is the final pixel number jend inside the sub-region. When j does not correspond
to jend, at step S36 the CPU 22 increments j by 1 and returns to step S34 to repeat
the same processing.
[0064] After calculating the chromaticity gj/rj as far as the final pixel number jend in
the m
th sub-region in this manner, the CPU 22 advances to step S37.
[0065] At step S37, the CPU 22 calculates a chromaticity mean value µgrm of the m
th sub-region. Next, at step S38, the CPU 22 determines whether or not the number m
of the sub-region is the final number mend. When the number m does not correspond
to the final number mend, at step S39 the CPU 22 increments m by 1 and returns to
step S34 to repeat the processing of step S34 to S39.
[0066] When m matches the final number mend, the CPU 22 proceeds to step S40. At step S40,
the CPU 22 calculates the chromaticity mean value µgr for all the sub-regions and
the standard deviation σgr.
[0067] Next, at step S41, the CPU 22 uses the chromaticity mean value µgr in the case of
all sub-regions and the standard deviation σgr to determine whether or not the chromaticity
mean value µgrm of the m
th sub-region calculated at step S37 indicates that the m
th sub-region is a sub-region showing a color tone change involving reddening and discoloration.
[0068] More specifically, as shown at step S41, the CPU 22 considers that the chromaticity
mean value µgrm of the sub-region is normally distributed, and determines whether
or not that chromaticity mean value µgrm is within a range of ± (1.5 x σgr) from the
distribution position of the chromaticity mean value µgr for all the sub-regions.
[0069] When the sub-region in question satisfies the determining condition at step S41,
as shown at step S42, the CPU 22 detects (judges) that the sub-region is not a sub-region
that shows a color tone change involving reddening and discoloration.
[0070] In contrast, when the sub-region does not satisfy the condition at step S41, as shown
at step S43, the CPU 22 detects the sub-region as a sub-region showing a color tone
change of reddening or discoloration.
[0071] More specifically, when the chromaticity mean value µgrm deviates to outside by -1.5
x σgr from the distribution position of the chromaticity mean value µgr, the CPU 22
detects (judges) the sub-region as a sub-region showing a color tone change of reddening,
and when the chromaticity mean value µgrm deviates to outside by +1.5 x σgr from the
distribution position of the chromaticity mean value µgr, the CPU 22 detects (judges)
the sub-region as a sub-region showing a color tone change of discoloration.
[0072] The detection (judgment) result is stored in the information storage section 25 (see
Fig. 1) or the like together with the sub-region number m or two-dimensional positional
information of the sub-region. That detection result information is utilized for a
judgment at step S9 in Fig. 5 that is described below.
[0073] In this connection, the processing from steps S41 to S43 in Fig. 8 is performed from
1 to mend by changing the number m of the sub-region. After performing the processing
of step S42 or S43 for all sub-regions of the processing object in this manner, the
CPU 22 proceeds to the processing of step S7 or step S8 in Fig. 5.
[0074] Although an example is described above which uses the chromaticity gj/rj, an arrangement
may be used in which bj/gj is added as the chromaticity. In that case, it is possible
to support detection of yellow-colored mucous membrane.
[0075] Further, in the processing shown in Fig. 8, detection may be performed based on a
ratio µrate = µgrm/µgr as the ratio between the chromaticity mean values µgrm and
µgr so as, for example, to detect a color tone change as reddening if µrate < 0.5
and detect a color tone change as discoloration if µrate > 1.5.
[0076] Next, at step S7 in Fig. 5, based on three-dimensional shape information calculated
at step S4, the CPU 22 performs processing to detect a region with an elevation change.
For example, the CPU 22 detects a sub-region section having the aforementioned shape
index SI value is 0.8 or more.
[0077] Further, at step S8, by associating the three-dimensional shape information obtained
at step S4 with information regarding detection of a region with a color tone change
that is obtained at step S6, the CPU 22 identifies each pixel or that sub-region included
in the region with a color tone change that is detected at step S6, which position
corresponds to the position on the three-dimensional shape.
[0078] At step S9, the CPU 22 performs judgment processing based on the results obtained
at steps S7 and S8. More specifically, the CPU 22 judges whether or not the region
with an elevation change on the three-dimensional shape that is detected at step S7
corresponds with the region with a color tone change that is identified at step S8.
[0079] If the judgment result at step S9 is No, i.e. if the elevation change is not accompanied
by a color tone change, at step S10 the CPU 22 applies polyp detection processing
according to detection standard 1.
[0080] In contrast, if the judgment result at step S9 is Yes, i.e. if the elevation change
is accompanied by a color tone change, at step S11 the CPU 22 applies polyp detection
processing according to detection standard 2.
[0081] According to the present embodiment, as illustrated in steps S10 and S 11, the assessment
standard or detection standard for performing polyp detection is changed according
to the judgment result at step S9. In this case, since there is a high possibility
that the region in question and the periphery thereof is a polyp when a color tone
change is detected, by detecting a section having an elevation change feature using
detection conditions that are more relaxed than in a case in which a color tone change
is not detected, a polyp or a polyp candidate is accurately detected.
[0082] Further, although various standards can be applied for polyp detection processing,
according to the present embodiment a detection standard according to the aforementioned
shape index SI is used.
[0083] As the threshold value of the shape index SI for detecting a convex shape (or cup
shape) indicating a polyp as an elevated lesion, detection standard 1 is set to (SI
=) 0.9 and detection standard 2 is set to (SI =) 0.8 (in this connection, as described
above, the nearer the value is to 1, the closer the shape is to a convex shape). At
step S10 or S11, the CPU 22 performs a comparison with the 0.9 or 0.8 shape index
SI that is the detection standard, and if the value is greater than that value the
sub-region is detected (judged) as being a polyp.
[0084] In this case, for example, for the curvedness CV, (CV=) 0.2 is taken as the threshold
value for both detection standards 1 and 2, and when the value for the relevant sub-region
is greater than this value the sub-region is detected (judged) as being a polyp.
[0085] Although in the above description the value of the shape index SI as a detection
standard changes for a case which is accompanied by a color tone change and a case
which is not accompanied by a color tone change, a configuration may also be adopted
in which the curvedness CV value is changed to change the detection standard as described
below.
[0086] More specifically, a configuration may be adopted in which the shape index SI threshold
value is set to 0.9 and the curvedness CV threshold value is set to 0.20 as detection
standard 1, and the shape index SI threshold value is set to 0.9 and the curvedness
CV threshold value is set to 0.15 as detection standard 2.
[0087] Further, a change may also be made to lower the shape index SI threshold value and
the curvedness CV threshold value of detection standard 2 with respect to detection
standard 1.
[0088] For example, a configuration may be adopted in which the shape index SI threshold
value is set to 0.9 and the curvedness CV threshold value is set to 0.20 as detection
standard 1, and the shape index SI threshold value is set to 0.85 and the curvedness
CV threshold value is set to 0.15 as detection standard 2.
[0089] Thus, at step S10 or step S 11 the CPU 22 sets detection standard 1 or 2 as a threshold
value and performs polyp detection processing according to the shape index SI value
for the region with an elevation change.
[0090] The CPU 22 then stores the detection result in association with the endoscopic image
Ii of the detection object in, for example, the hard disk 27 shown in Fig. 1, and,
for example, displays the detection result side by side with the endoscopic image
Ii of the detection object on the display monitor 4 via the display processing section
28.
[0091] Subsequently, at step S12 in Fig. 5, the CPU 22 increments i by 1 and performs the
same processing for the next endoscopic image Ii.
[0092] According to the present embodiment that performs image processing in this manner,
it is possible to change (control) a detection standard value or a condition value
by referring to color tone change information corresponding to the feature value of
a color tone change that occurs incidentally in the case of a polyp, and thereby improve
the accuracy (or reliability) of a detection with respect to whether or not a detection
object is a polyp. That is, in comparison to the case of image processing that performs
polyp detection using only one type of detection, polyp detection can be performed
with greater accuracy by combining both kinds of detection.
[0093] Although a case is described above in which polyp detection processing is performed
at step S10 or step S 11 using the threshold value of a shape index SI and a curvedness
CV as feature quantities that represent a curved shape, an arrangement may also be
adopted in which detection is performed based on information regarding a height from
a reference surface in the region with an elevation change.
[0094] In this case, when the threshold value for a height in the case of detection standard
1 is taken as H1, it is sufficient to change the detection standard to a threshold
value H2 that is less than H1. Thus, when accompanied by a color tone change, highly
accurate polyp detection can be performed by changing (controlling) the detection
standard so as to relax the detection conditions for the polyp detection.
[0095] Although according to the present embodiment a case is described of detecting a polyp
as an elevated lesion as a method or means of lesion detection, the present embodiment
is not limited thereto, and naturally the present embodiment can also be applied to
a depressed lesion.
[0096] In this case, for example, in Fig. 5, it is sufficient to change the processing so
that at step S7, instead of detecting a region with an elevation change, a depressed
lesion region is detected, and to similarly change the processing at step S9.
[0097] Further, the detection standards at step S10 and step S11 are also changed to values
that are close to the feature value of a depressed lesion. By adopting such changes,
the present embodiment can also be applied to a depressed lesion.
(Embodiment 2)
[0098] Next, Embodiment 2 of the present invention is described referring to Fig. 9 to Fig.
13.
[0099] In the present embodiment also, a polyp as an elevated lesion is detected by detecting
an elevated change region from three-dimensional information (shape information) that
is calculated based on a change in contrast.
[0100] There are cases in which a polyp is accompanied by an abnormal finding in peripheral
mucous membrane. Conversely, when an abnormal finding is observed in mucous membrane,
the possibility that a polyp has developed also increases. Examples of abnormal findings
in mucous membrane include white spots as illustrated in Fig. 9A and abnormal vascular
hyperplasia as illustrated in Fig. 9B.
[0101] According to the present embodiment, attention is focused on such abnormal findings,
and when an abnormal finding is present on a mucous membrane surface the accuracy
of polyp detection can be improved by changing the detection standard (parameter or
threshold value or the like) for polyp detection using shape information.
[0102] More specifically, for a region with an elevation change that is accompanied by an
abnormal finding in peripheral mucous membrane, detection standards are used in which
the detection conditions are more relaxed than in a case in which a region with an
elevation change is not accompanied by an abnormal finding.
[0103] The configuration of the image processing device 3 according to the present embodiment
is the same as in Fig. 1, although the processing contents are different. The CPU
22 in the image processing device 3 according to the present embodiment has the processing
functions shown in Fig. 10. More specifically, the CPU 22 according to the present
embodiment has a region with abnormal finding detection function 22f instead of the
region with color tone change detection function 22b shown in Fig. 4 according to
Embodiment 1. In this connection, the abnormal finding region is detected (judged)
by processing for a mucosal attribute finding judgment.
[0104] Next, referring to Fig. 11, polyp detection operations according to the image processing
of the present embodiment are described. Since step S 1 to step S5 in the processing
of the flowchart shown in Fig. 11 are the same as step S1 to step S5 in the processing
of the flowchart shown in Fig. 5, a description thereof is omitted here.
[0105] At step S5, the CPU 22 extracts an R image Ri in the endoscopic image Ii and a G
image Gi in the endoscopic image Ii. Next, at step S51, the CPU 22 executes processing
for a mucosal attribute finding judgment (to detect a region with an abnormal finding).
[0106] In the processing for a mucosal attribute finding judgment, the CPU 22 determines
whether or not a region presenting a characteristic mucosal attribute finding in the
endoscopic image Ii is a region presenting an abnormal finding. According to the present
embodiment, by using the R image Ri and G image Gi and applying a series of processes
that are described later in step S51 as described above, detection of a region with
an abnormal finding such as white spots or abnormal vascular hyperplasia is performed.
[0107] Subsequently, similarly to step S7 in Fig. 5, detection of a region with an elevation
change is performed based on three-dimensional shape information that is calculated
at step S4.
[0108] Next, at step S52, the CPU 22 executes judgment processing based on the detection
results obtained at steps S51 and S7. If the result of the judgment processing at
step S52 indicates no abnormal finding, the CPU 22 advances to step S10. If there
is an abnormal finding, the CPU 22 advances to step S11.
[0109] Similarly to Embodiment 1, at step S10 the CPU 22 applies polyp detection processing
according to detection standard 1. At step S11, the CPU 22 applies polyp detection
processing according to detection standard 2. After the processing at steps S10 and
S11, the CPU 22 returns to step S2 via step S12.
[0110] Various standards can be applied for polyp detection processing. For example, similarly
to Embodiment 1, detection may be performed according to the shape index SI or using
a curvedness CV threshold value.
[0111] The processing for a mucosal attribute finding judgment at step S51 in Fig. 11 according
to the present embodiment will now be described referring to the flowcharts shown
in Fig. 12 and Fig. 13.
[0112] As mucosal attribute finding judgment processing, according to the present embodiment,
processing to detect an abnormal finding of white spots that is illustrated in Fig.
12 and processing to detect an abnormal finding of abnormal vascular hyperplasia that
is illustrated in Fig. 13 are executed. First, the processing to detect an abnormal
finding of white spots illustrated in Fig. 12 is described.
[0113] Initially, at step S61, the CPU 22 performs processing to exclude unsuitable pixels
such as dark portions, halation, and residue. Next, at step S62, the CPU 22 calculates
the chromaticity gj/rj of each pixel. Here, the suffix "j" represents the pixel number.
[0114] At step S63, the CPU 22 executes binarization processing using threshold value processing.
In this case, for example, a (chromaticity) threshold value Vth is set to 0.8 to perform
binarization using this threshold value Vth. In the binarization, 1 is set if gj/rj
> 0.8, and 0 is set if otherwise, i.e. if gj/rj ≤ 0.8. In this manner, a binarization
image is generated.
[0115] Subsequently, at step S64, the CPU 22 performs labeling processing that tracks pixels
for which a binarization value is 1 in the binarization image to label the region
and generate a labeling image from the binarization image.
[0116] Next, at step S65, the CPU 22 performs processing to detect small clusters of white
spots with respect to the labeling image that is generated by the labeling processing.
That is, the CPU 22 performs detection of small clusters of white spots to detect
whether or not clusters corresponding to small clusters caused by white spots occur
in the labeling image.
[0117] When performing the detection of small clusters of white spots, for example, the
CPU 22 uses the degree of elongation (= area/thinning length). The CPU 22 then detects
a degree of elongation value E for the labeling image and performs a comparison to
determine whether or not the value E is less than the (judgment standard) threshold
value Vth (for example, Vth = 10). The CPU 22 then calculates a number of small clusters
N such that E < Vth (=10).
[0118] In this connection, a configuration may be adopted in which the CPU 22 calculates
the number of small clusters N using the roundness or moment feature quantity or the
like as a detection standard instead of the degree of elongation.
[0119] Next, at step S66, the CPU 22 judges whether or not the number of small clusters
N detected at step S65 is larger than a threshold value Nth (for example, Hth = 20)
of the detection standard for white spot detection. When the condition N > Nth is
fulfilled, as shown at step S67, the CPU 22 judges that white spots are detected in
the labeling image (G image Gi and R image Ri). Conversely, when the condition N >
Nth is not fulfilled, at step S68 the CPU 22 judges that white spots are not detected.
[0120] The detection result at step S67 and step S68 is utilized for the judgment at step
S52. That is, the detection result is utilized to indicate the existence or non-existence
of an abnormal finding according to the existence or non-existence of a white spot
detection.
[0121] Further, as the mucosal attribute finding judgment processing at step S51, processing
to detect an abnormal finding due to abnormal vascular hyperplasia as illustrated
in Fig. 13 that is described next is performed. In order to detect abnormal vascular
hyperplasia, processing is performed that detects localized thick blood vessels and
minute complex branching from the endoscopic image Ii.
[0122] From the initial step S71 to step S74, the CPU 22 performs processing that is substantially
the same as that shown in Fig. 12. More specifically, at step S71, the CPU 22 performs
processing to exclude unsuitable pixels such as dark portions, halation, and residue,
at step S72 the CPU 22 calculates the chromaticity gj/rj of each pixel, and at step
S73 the CPU 22 performs binarization processing according to threshold value processing.
In this case, the CPU 22 performs binarization using a threshold value Vth that is
different from the case illustrated in Fig. 12.
[0123] For example, the CPU 22 sets the threshold value Vth to 0.2, and uses this threshold
value Vth to perform binarization in which 1 is set if gj/rj < 0.2, and 0 is set if
otherwise, i.e. if gj/rj ≥ 0.2. In this manner, a binarization image is generated.
[0124] The CPU 22 performs labeling processing at the subsequent step S74. In this case,
the CPU 22 creates a thin line image as a labeling image.
[0125] Next, at step S75, the CPU 22 performs processing to detect a feature value corresponding
to abnormal vascular hyperplasia. For the feature value detection processing, the
CPU 22 detects the number of branches/intersecting points with respect to the thin
line image. Next, at step S76, the CPU 22 determines the existence or non-existence
of an abnormal vascular hyperplasia detection by using a threshold value for the number
of branches/intersecting points or a discriminant function.
[0126] In this connection, as the feature value detection processing, instead of performing
the processing using the number of branches/intersecting points, a configuration may
be adopted in which, by using the degree of elongation or the like, the CPU 22 detects
the number of times the degree of elongation exceeds the threshold value and performs
detection (judgment) of abnormal vascular hyperplasia according to whether or not
that number exceeds a threshold value for abnormal vascular hyperplasia detection.
[0127] The detection result obtained by the processing shown in Fig. 13 is utilized for
the judgment at step S52. At step S52, the CPU 22 judges whether or not a region with
an elevation change that is detected at step S7 is one accompanied by, as a mucosal
attribute finding thereof, the detection of white spots according to the processing
shown in Fig. 12 or accompanied by the detection of abnormal vascular hyperplasia
according to the processing shown in Fig. 13.
[0128] The processing performed by the CPU 22 after the result of judgment processing obtained
at step S52 is the same as the processing in the case of Embodiment 1 that is shown
in Fig. 5.
[0129] More specifically, when the judgment processing at step S52 indicates that the region
with an elevation change is not accompanied by an abnormal finding, at step S10 the
CPU 22 performs polyp detection by applying detection standard 1. In contrast, when
the judgment processing indicates the region with an elevation change is accompanied
by an abnormal finding, at step S 11 the CPU 22 performs polyp detection by applying
detection standard 2. After the processing at step S 10 and step S11, the CPU 22 increments
the image parameter i by 1, and performs the same processing for the next endoscopic
image Ii.
[0130] For the detection standard 1 and the detection standard 2, detection is performed
using the shape index SI or using a curvedness CV threshold value as described above.
[0131] In this connection, for the foregoing description, a configuration may also be adopted
in which a detection filter for detecting white spots or abnormal vascular hyperplasia
is applied with respect to the G image Gi, to thereby detect white spots or abnormal
vascular hyperplasia.
[0132] Further, when a region with an abnormal finding such as white spots or abnormal vascular
hyperplasia is detected over a plurality of frames, the detection standard can be
changed (controlled) so as to perform polyp detection that applies detection standard
2 with respect to the other frames also.
[0133] Thus, according to the present embodiment, it is possible to perform highly accurate
detection of a polyp by performing polyp detection as elevated lesion detection by
changing the detection standard with respect to a region with an elevation change
according to a case in which an abnormal finding is detected and a case in which an
abnormal finding is not detected by processing that judges a mucosal attribute finding.
[0134] The order of the image processing in Fig. 11 may be changed. For example, the processing
of step S7 may be performed prior to the processing of step S51 Further, in order
to reduce the amount of calculation, a configuration may be adopted in which the processing
that judges a mucosal attribute finding at step S51 is performed for a region corresponding
to a detection region of a non-elevation change portion that does not show a three-dimensional
elevation at step S7. That is, since a region with an abnormal finding in the present
embodiment incidentally arises at the periphery of a polyp as an elevated lesion,
a configuration may be adopted that detects an abnormal finding for a peripheral mucous
membrane or a flat part on the periphery of the region with an elevated lesion change.
[0135] The detection standards 1 and 2 are not limited to a case applying detection according
to the shape index SI or a curvedness CV threshold value as described above, and as
described in Embodiment 1, a configuration may be adopted that detects an elevated
lesion based on the value of a height from a reference surface or the like. In this
case also, when a polyp is accompanied by white spots or abnormal vascular hyperplasia
on a mucous membrane surface, control is performed so as to increase the sensitivity
for polyp detection, thereby enabling highly accurate polyp detection.
[0136] The lesion detecting means and lesion detection method according to the present embodiment
are not limited to the case of an elevated lesion, and can be similarly applied to
a depressed lesion as described in Embodiment 1.
[0137] In the above described Embodiment 1 and Embodiment 2, a case is described in which,
when detecting a lesion by image processing, the detection standard when detecting
a lesion candidate such as an elevated lesion region based on luminance information
is changed in accordance with the detection result when a region (i.e. an incidental
region) with a color tone change or an abnormal finding is detected. However, a configuration
may also be adopted so as to detect lesions in a comprehensive manner based on the
two detection results.
[0138] Further, as another detection method or detection means for these embodiments, a
configuration may be adopted so as to detect whether or not a lesion candidate is
a lesion in accordance with whether or not a region exists in which there is a correlation
between the two detection results. For example, to respond to demands to detect only
regions for which it is considered there is a sufficiently high possibility that a
candidate is a lesion, the accuracy of lesion section detection can be increased by
detecting a region as a lesion candidate in a case where a correlation exists between
the two results that detect the region in question as a lesion candidate.
[0139] Although according to Embodiment 1 and Embodiment 2 a technique that is based on
estimating a three-dimensional shape is described as a lesion detecting technique,
the present invention is not limited thereto. For example, an arrangement can also
be considered that applies a technique that, after extracting an edge by band pass
filtering with respect to a two-dimensional R image, performs binarization using threshold
value processing, and then detects an edge that shows a circular arc shape presenting
a polyp-like lesion by known Hough transformation.
[0140] In this case also, it is possible to change the detection standard based on an abnormal
finding at the periphery or the lesion by employing the length, area or curvature
of an arc or the like of the edge that is detected as a lesion candidate as a detection
standard.
[0141] Although a technique that is based on threshold value processing with respect to
the curvedness CV and the shape index SI is described in the aforementioned embodiments
as a lesion detection technique, the present invention is not limited thereto. For
example, without performing threshold value processing, it is possible to prepare
image groups having findings of respective shapes such as a convex shape or a cup
shape as training data, and utilize these for a detection technique employing a discrimination
circuit that uses each feature value that is calculated. For example, in a case where
an abnormal finding is present when using a discriminator that uses a known linear
discriminant function, control that facilitates detection of a convex shape or the
like can be performed by controlling the weighting for values obtained by application
of a linear discriminant function for each shape.
[0142] It is to be understood that a modified example of the above described embodiments
in which the embodiments are partially combined or in which the image processing order
is changed is also included in the scope of the present invention.
[0143] According to the present invention as described above, a candidate region of a lesion
such as a polyp is detected based on luminance information in a medical image having
color information such as an endoscopic image, and an incidental region that is accompanied
by a color tone change such as reddening or a discoloration portion that arises because
of incidental attributes that accompany a lesion is also detected. Thus, highly accurate
lesion detection can be performed by changing the detection standard for detecting
a lesion from the lesion candidate region according to the detection result for the
incidental region.
[0144] Further, the present invention has the features described in the following addenda.
(Addendum 1)
[0145] A medical image processing device, comprising lesion candidate region detecting means
that detects a lesion candidate region based on at least one color signal in a medical
image including a plurality of color signals, incidental region detecting means that
detects an incidental region that arises because of incidental attributes accompanying
a lesion from the medical image, and detection standard changing means that changes
a detection standard when detecting a lesion from the lesion candidate region in accordance
with a detection result of the incidental region.
(Addendum 2)
[0146] The medical image processing device according to addendum 1, wherein the lesion candidate
region detecting means has three-dimensional shape information generation means that
generates three-dimensional shape information based on the at least one color signal
from the medical image, and detects a lesion candidate region utilizing three-dimensional
shape information that is generated.
(Addendum 3)
[0147] The medical image processing device according to addendum 1, wherein the incidental
region detecting means detects a region with a color tone change that presents attributes
of a color tone change such as reddening or discoloration as the incidental region.
(Addendum 4)
[0148] The medical image processing device according to addendum 1, wherein the incidental
region detecting means detects a region with an abnormal finding that presents attributes
of an abnormal finding of white spots or abnormal blood vessels as the incidental
region.
(Addendum 5)
[0149] The medical image processing device according to addendum 3, wherein the detection
standard changing means changes, as the detection standard, a threshold value of at
least one of a shape index and a curvedness of a feature value that represents a curved
shape in the three-dimensional shape information that is generated.
(Addendum 6)
[0150] A medical image processing device, comprising lesion candidate region detecting means
that detects a lesion candidate region based on at least one color signal in a medical
image including a plurality of color signals, and incidental region detecting means
that detects an incidental region that arises because of incidental attributes accompanying
a lesion from the medical image, wherein the medical image processing device performs
lesion detection based on existence or non-existence of a region in which a lesion
candidate region that is detected by the lesion candidate region detecting means and
an incidental region that is detected by the incidental region detecting means correlate.
(Addendum 7)
[0151] A medical image processing method, comprising a lesion candidate region detection
step that detects a lesion candidate region based on at least one color signal in
a medical image including a plurality of color signals, an incidental region detection
step that detects existence or non-existence of an incidental region that arises because
of incidental attributes accompanying a lesion from the medical image, and a detection
standard changing step that changes a detection standard when detecting a lesion from
the lesion candidate region according to a detection result regarding existence or
non-existence of the incidental region.
(Addendum 8)
[0152] The medical image processing method according to addendum 7, wherein the lesion candidate
region detection step has a three-dimensional shape information generation step that
generates three-dimensional shape information based on the at least one color signal
from the medical image, and detects a lesion candidate region using three-dimensional
information that is generated.
(Addendum 9)
[0153] The medical image processing method according to addendum 7, wherein the incidental
region detection step detects a region with a color tone change that presents attributes
of a color tone change of reddening or discoloration as the incidental region.
(Addendum 10)
[0154] The medical image processing method according to addendum 7, wherein the incidental
region detection step detects a region with an abnormal finding that presents attributes
of white spots or abnormal blood vessels as the incidental region.
(Addendum 11)
[0155] The medical image processing method according to addendum 8, wherein the detection
standard changing step changes, as the detection standard, a threshold value of at
least one of a shape index and a curvedness of a feature value that represents a curved
shape in the three-dimensional shape information that is generated.
(Addendum 12)
[0156] The medical image processing device according to addendum 3, wherein the devices
changes, as the detection standard, a threshold value of a height from a reference
surface or at least one of a shape index and a curvedness of a feature value that
represents a curved shape in the three-dimensional shape information that is generated.