(19)
(11)EP 2 261 853 B1

(12)EUROPEAN PATENT SPECIFICATION

(45)Mention of the grant of the patent:
02.12.2015 Bulletin 2015/49

(21)Application number: 10165286.5

(22)Date of filing:  08.06.2010
(51)International Patent Classification (IPC): 
G06T 5/00(2006.01)
H04N 5/21(2006.01)

(54)

Image processing apparatus, medium, and method

Bildverarbeitungsvorrichtung, -medium und -verfahren

Appareil de traitement d'images, support et procédé


(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

(30)Priority: 09.06.2009 KR 20090050877

(43)Date of publication of application:
15.12.2010 Bulletin 2010/50

(73)Proprietor: Samsung Electronics Co., Ltd.
Suwon-si, Gyeonggi-do, 443-742 (KR)

(72)Inventors:
  • Lim, Hwa Sup
    445-728 Gyeonggi-do (KR)
  • Kang, Byong Min
    448-160 Gyeonggi-do (KR)
  • Kim, Seong Jin
    139-956 Seoul (KR)

(74)Representative: Grootscholten, Johannes A.M. et al
Arnold & Siedsma Bezuidenhoutseweg 57
2594 AC The Hague
2594 AC The Hague (NL)


(56)References cited: : 
  
  • ERIC P BENNETT ET AL: "Video Enhancement Using Per-Pixel Virtual Exposures" ACM TRANSACTIONS ON GRAPHICS, ACM, US, 1 January 2005 (2005-01-01), pages 845-852, XP002477673 ISSN: 0730-0301
  • BUADES, A., COLL, B., & MOREL, J. M.: "Denoising image sequences does not require motion estimation" PREPRINT OF THE CMLA, [Online] N2005-18, May 2005 (2005-05), XP002598966 PREPRINT OF THE CMLA Retrieved from the Internet: URL:http://www.cmla.ens-cachan.fr/fileadmi n/Documentation/Prepublications/2005/CMLA2 005-18.pdf> [retrieved on 2010-09-01]
  • ANTONIOG DOPICO ET AL: "Distributed Computation of Optical Flow" 13 May 2004 (2004-05-13), COMPUTATIONAL SCIENCE - ICCS 2004; [LECTURE NOTES IN COMPUTER SCIENCE;;LNCS], SPRINGER-VERLAG, BERLIN/HEIDELBERG, PAGE(S) 380 - 387 , XP019006471 ISBN: 978-3-540-22115-9 * section 2.2 *
  • GOKTURK S B ET AL: "A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions", 20040627; 20040627 - 20040602, 27 June 2004 (2004-06-27), pages 35-35, XP010761980,
  • SCHUON S ET AL: "High-quality scanning using time-of-flight depth superresolution", COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, 2008. CVPR WORKSHOPS 2008. IEEE COMPUTER SOCIETY CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 23 June 2008 (2008-06-23), pages 1-7, XP031285727, ISBN: 978-1-4244-2339-2
  
Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention).


Description

Background


1. Field



[0001] One or more embodiments relate to an image processing apparatus and method that may remove noise of a depth camera used for obtaining a depth image, and provide the depth image with the decreased noise.

2. Description of the Related Art



[0002] Admittedly, the publication by Eric P. Bennet et al: "Video enhancement using per-pixel virtual exposures" ACM TRANSACTIONS ON GRAPHICS, ACM, US, January 1, 2005, pages 845 - 852, XP-002477673, ISSN: 0730-0301, relates to denoising algorithms for use with or on images in general.

[0003] Currently, information about a three-dimensional (3D) image is widely used in a variety of applications. Generally, a 3D image includes geometry information and color information. The color information may be obtained using a camera that employs a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor. The geometry information may be obtained using a depth camera.

[0004] The depth camera may generate a depth image by emitting light such as an infrared (IR) light to an object, sensing a reflection ray using a sensor, and measuring a Time of Flight (TOF) that is the time taken for an emitted ray to be reflected from the object.

[0005] A depth sensor used for the depth camera of the TOF may be easily affected by electrical and/or thermal noise of the depth sensor and by a characteristic of a surrounding light source and material. Therefore, depth values of the depth image generated by the depth camera may be affected by the noise. The noise may deteriorate a quality of 3D modeling.

[0006] The noise may be removed using various types of schemes. According to an existing scheme, the noise may be removed by calculating the average of depth images that are obtained by performing a plurality of measurements for the same object.

[0007] The existing scheme may have a relatively excellent characteristic for a static object, whereas the existing scheme may cause a distortion, for example, motion blurring, for a dynamic object

Summary



[0008] According to an aspect of one or more embodiments, there may be provided an image processing apparatus and/or an image processing method in accordance with a relevant one of the appended independent claims.

[0009] Preferred steps may be provided in the method corresponding to the features as set out above with respect to the dependent apparatus claims.

[0010] Additional aspects, features, and/or advantages of embodiments will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.

Brief description of the drawings



[0011] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0012] These and/or other aspects and advantages will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 illustrates a configuration of an image processing apparatus according to an embodiment;

FIG. 2 illustrates depth images of frames that are input into an image processing apparatus according to an embodiment;

FIG. 3 illustrates a graph plotting a measured depth value of a particular pixel of FIG. 2 that is input into an image processing apparatus according to an embodiment;

FIG. 4 illustrates a graph plotting a result of processing a depth value corresponding to the graph of FIG. 3 using a conventional temporal average filter;

FIG. 5 illustrates a graph plotting a change of a temporal weight with respect to an obtainment time difference between frame depth values according to an embodiment;

FIG. 6 illustrates a graph plotting a change of a range weight with respect to a difference between frame depth values according to an embodiment;

FIG. 7 illustrates a graph plotting a result of processing a depth value corresponding to the graph of FIG. 3 according to an embodiment; and

FIG. 8 illustrates a flowchart of an image processing method according to an embodiment.


Detailed description



[0013] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Embodiments are described below to explain the present disclosure by referring to the figures.

[0014] FIG. 1 illustrates a configuration of an image processing apparatus 100 according to an embodiment.

[0015] The image processing apparatus 100 may receive a plurality of frame depth images or a plurality of frame depth values with respect to a particular pixel that is provided from a depth camera 101, and may store the received plurality of frame depth images or frame depth values in a buffer memory 110.

[0016] The image processing apparatus 100 may include a calculation apparatus 120 including a temporal weight calculator 121 to calculate a temporal weight, a range weight calculator 122 to calculate a range weight, and a corrected depth value calculator 123 to calculate a corrected depth value.

[0017] To correct a first frame depth value of a first pixel, the temporal weight calculator 121 may calculate the temporal weight to be applied to each of at least one second frame depth values of the first pixel. The temporal weight may be applied based on an obtainment time difference between frames.

[0018] When the first frame depth value of the first pixel is d(i, j, t) and the second frame depth value is d(i, j, t-k), the temporal weight calculator 121 may determine a weight to be applied to a second frame as a smaller value as k becomes greater. Here, k denotes a real number and the obtainment time difference between the frames.

[0019] According to an embodiment, as k increases, the temporal weight may exponentially decrease. In particular, the temporal weight may decrease along a Gaussian curve distribution. An operation of the temporal weight calculator 121 will be described in detail later with reference to FIG. 5.

[0020] To correct the first frame depth value of the first pixel, the range weight calculator 122 may calculate a range weight to be applied to each of at least one second frame depth value of the first pixel. The range value may be applied based on a depth value difference.

[0021] According to an embodiment, as a difference between d(i, j, t) and d(i, j, t-k) becomes greater, a range weight ratio may exponentially decrease and may also decrease along the Gaussian curve distribution. A process of calculating, by the range weight calculator 122, the range weight will be further described in detail with reference to FIG. 6.

[0022] The corrected depth value calculator 123 may calculate a corrected first frame depth value of the first pixel using a linear sum of the first frame depth value of the first pixel and the at least one second frame depth value of the first pixel by applying the calculated temporal weight and/or range weight.

[0023] An operation of the corrected depth value calculator 123 will be described in detail with reference to FIGS. 6 and 7.

[0024] FIG. 2 illustrates depth images 210, 220, and 230 of frames that are input into an image processing apparatus according to an embodiment.

[0025] Here, it is assumed that the depth image 210 is obtained from an object space corresponding to a color image 201 at t=20, the depth image 220 is obtained from the object space corresponding to a color image 202 at t=30, and the depth image 230 is obtained from the object space corresponding to a color image 203 at t=40.

[0026] The depth images 210, 220, and 230 correspond to a plurality of depth image frames that are obtained in different points in times with respect to the same object space.

[0027] When comparing an area 211 of the depth image 210, an area 221 of the depth image 220, and an area 231 of the depth image 230, it can be seen that pixels included in the depth images 210, 220, and 230 are affected by noise.

[0028] A particular pixel of any one depth image among the plurality of depth images 210, 220, and 230 may be selected. For example, a particular pixel of the depth image 230 may be selected.

[0029] Here, it is assumed that a first pixel 232 marked by "X" within the depth image 230 obtained at t=40 is selected. In this case, the first pixel 232 may correspond to a pixel 212 marked by "X" within the depth image 210 obtained at t=20. Also, the first pixel 232 may correspond to a pixel 222 marked by "X" within the depth image 220 obtained at t=30.

[0030] An exemplary graph plotting a depth value of the first pixel 232, measured over time, may be shown in FIG. 3.

[0031] FIG. 3 illustrates a graph plotting a measured depth value of the first pixel 232 of FIG. 2 that is input into an image processing apparatus according to an embodiment

[0032] In FIG. 3, a horizontal axis denotes a time when a corresponding depth value is measured, and a vertical axis denotes a measured depth value. Accordingly, when observing a pattern of depth values, it is possible to verify a noise affect.

[0033] It can be seen from the graph that the depth value significantly increases at around t=25. There is a large difference between depth values corresponding to t<25 and depth values corresponding to t>25. The large difference may be determined to occur due to a motion of an object.

[0034] Referring again to FIG. 2, a depth value of the pixel 212 at t=20 corresponds to a hand of a human being that is the object. Since the hand is moved, the pixel 222 at t=30 or the first pixel 232 at t=40 corresponds to a background. Accordingly, in the graph of FIG. 3, the depth value significantly increases at around t=25.

[0035] In a conventional art, there is provided a corrected or noise-removed depth image value by simply calculating the average of a plurality of frame depth image values to remove noise in a depth image.

[0036] For example, let a current depth value be d(i, j, t). Here, i and j denote a row value and a column value to identify a particular pixel within a depth image. Also, t denotes a current time.

[0037] A conventional average temporal filtering scheme may correct a depth value of a current frame using depth values of a plurality of frames, for example, two frames excluding the current frame.

[0038] When depth values of previous frames are expressed by d(i, j, t-k1) and d(i, j, t-k2), the average temporal filtering scheme may calculate a corrected depth value d'(i, j, t) by applying the same weight to the current frame depth value d(i, j, t), and the previous frame depth values d(i, j, t-k1) and d(i, j, t-k2).

[0039] Specifically, such a conventional average temporal filtering scheme would be performed in accordance with the following equation: d'(i, j, t) = 1/3 x (d(i,j, t) + d(i, j, t-k1) + d(i, j, t-k2)).

[0040] When the object is in a motionless state, that is, in a static state, the average temporal filtering scheme may relatively excellently remove noise.

[0041] However, when the object is in a dynamic state as shown in FIG. 2, blurring may occur between the depth values, whereby the depth values may be distorted. This phenomenon may be referred to as motion blurring.

[0042] FIG. 4 illustrates a graph plotting a result of processing a depth value corresponding to the graph of FIG. 3 using a conventional temporal average filter.

[0043] It can be seen that motion blurring occurs in a portion 410 of the graph. The temporal average filtering scheme is proposed so that depth values of previous frames are correlated with a depth value of a current frame. However, in FIG. 2, due to a motion of the object at around t=25, there is a significant change in the depth value of the first pixel 232.

[0044] The motion blurring phenomenon may frequently occur since the same weight, for example, 1/3 in the above example, is applied to the depth values of the previous frames to be used in a calculation for a noise removal.

[0045] According to an embodiment, a different weight may be applied to depth values of previous frames to be used in a calculation for a noise removal.

[0046] For example, as a point in time of when a depth value of a previous frame is obtained is temporally further away from a point in time of when the depth value of the current frame is obtained, a smaller weight may be applied. Specifically, it is assumed that, as the obtainment time difference between the frames becomes greater, a correlation is smaller. Hereinafter, the aforementioned scheme is referred to as an adaptive temporal weight or a temporal weight.

[0047] The embodiment will be further described in detail with reference to FIG. 5.

[0048] Also, as the difference between the depth value of the previous frame and the depth value of the current frame becomes greater, the smaller weight may be applied separately or together with the temporal weight. Specifically, it is assumed that, as the depth value difference becomes greater, the depth value may change due to a change in an actual state of the object, as opposed to a change due to noise. Since the noise is known to have a white Gaussian characteristic, the above assumption may be reasonable.

[0049] The embodiment will be further described in detail with reference to FIG. 6.

[0050] FIG. 5 illustrates a graph plotting a change of a temporal weight with respect to an obtainment time difference between frame depth values according to an embodiment.

[0051] Here, it is assumed that a current frame depth value of a first pixel is d(i, j, k) and a previous frame depth value of the first pixel is d(i, j, t-k). Here, k denotes a real number and an obtainment time difference between frames.

[0052] A smaller weight may be assigned for a previous frame with a greater k. Accordingly, a depth value of a temporally closer frame may be further reflected in the depth value of the first pixel.

[0053] In FIG. 5, a horizontal axis denotes the obtainment time difference k, and a vertical axis denotes a temporal weight ratio of a weight of the previous frame depth value to a weight of the current depth value, that is, the temporal weight ratio of a second weight to a first weight.

[0054] According to an embodiment, as k increases, the temporal weight ratio may exponentially decrease, and may also decrease along a Gaussian curve distribution. In this case, a weight F(t, t-k) to be applied to the previous frame depth value d(i, j, t-k) may be calculated according to the following Equation 1:



[0055] Here, σt denotes an attenuation coefficient that is a positive real number.

[0056] Also in this equation, Σ exp(-σt x kn2) may normalize F(t, t-k) with respect to n frames including the current frame. Accordingly, it is possible to satisfy an equation of Σ F(t, t-k) = 1.

[0057] When comparing the above equation 1 with the graph of FIG. 5, the horizontal axis denotes the obtainment time difference k and the vertical axis denotes F(t, t-k).

[0058] FIG. 6 illustrates a graph plotting a change of a range weight with respect to a difference between frame depth values according to an embodiment.

[0059] As in FIG. 5, it is assumed that a current frame depth value of a first pixel is d(i, j, k) and a previous frame depth value of the first pixel is d(i, j, t-k). Here, k denotes a real number and an obtainment time difference between frames.

[0060] According to an embodiment, as a difference between d(i, j, t) and d(i, j, t-k) becomes greater, a range weight ratio may exponentially decrease, and may also decrease along a Gaussian curve distribution. In this case, a weight G(d(i, j, t), d(i,j, t-k)) to be applied to d(i, j, t-k) may be calculated according to the following Equation 2:



[0061] Here, σR denotes an attenuation coefficient that is a positive real number. Also here, a denominator Σexp(-σR x (d(i, j, t) - d(i, j, t-kn))2) may normalize the weight G(d(i, j, t), d(i, j, t-kn)) with respect to n frames including the current frame. Accordingly, it is possible to satisfy an equation of Σ G(d(i, j, t), d(i, j, t-k)) = 1.

[0062] When comparing the above equation with the graph of FIG. 6, the horizontal axis denotes a difference of an obtainment time depth value d(i, j, t) - d(i, j, t-k1), and the vertical axis denotes G(d(i, j, t), d(i, j, t-k)).

[0063] According to an embodiment, a final weight may be obtained by multiplying the range weight and the temporal weight. Also, the first pixel value may be calculated by applying either the range weight or the temporal weight.

[0064] A corrected depth value of the first pixel may be calculated by applying both the range weight and the temporal weight, as given by the following Equation 3:



[0065] Here, d(i, j, t) denotes a before-correction depth value of a first pixel of a first frame that is a current frame, and d'(i, j, t) denotes an after-correction depth value of the first pixel.

[0066] Also, d(i, j, t-k) denotes a depth value of a second frame that has an obtainment time difference k with respect to the first frame, F(t, t-k) denotes the temporal weight to be applied to the depth value of the second frame, G(d(i, j, t), d(i, j, t-k)) denotes the range weight to be applied to the depth value of the second frame. The range weight and the temporal weight are described above with reference to the above Equation 1 and Equation 2, and thus further description related thereto will be omitted.

[0067] Referring to the above Equation 3, when a new depth value d(i, j, t) is calculated by correcting the first frame depth value d(i, j, t) of the first pixel, it is possible to calculate the corrected depth value based on a linear sum of weighted depth values using another frame depth value, for example, at least one second frame depth value d(i, j, t-k) that is obtained at a different point in time from the first frame depth value.

[0068] As the obtainment time difference k between the first frame depth value and the second frame depth value becomes greater, the temporal weight F(t, t-k) to be multiplied by d(i, j, t-k) may be smaller. For example, as in FIG. 5, as k increases, the temporal weight may decrease along the Gaussian curve distribution.

[0069] As the first frame depth value d(i, j, t) and the second frame depth value (i, j, t-k) becomes greater, the range weight G(d(i, j, t), d(i, j, t-k)) to be multiplied by d(i, j, t-k) may be smaller. For example, as in FIG. 6, as k increases, the range weight may decrease along the Gaussian curve distribution.

[0070] FIG. 7 illustrates a graph plotting a result of processing a depth value corresponding to the graph of FIG. 3 using the above Equation 3 according to an embodiment.

[0071] It can be seen from the graph that a corrected depth value of a first pixel shows a smooth characteristic due to decreased noise. It also can be seen that motion blurring significantly decreases at around t=25, that is, in a portion 710 of where a motion of an object changes.

[0072] FIG. 8 illustrates a flowchart of an image processing method according to an embodiment.

[0073] In operation S810, a plurality of frame depth values of a first pixel extracted from a plurality of frame depth images obtained from a depth camera may be input into an image processing apparatus.

[0074] In operation S820, a temporal weight to be applied to the plurality of frame depth values of the first pixel may be calculated.

[0075] The temporal weight may be calculated using the above Equation 1. Also, a first frame depth value of the first pixel may be corrected by applying only the temporal weight.

[0076] Also, the first frame depth value of the first pixel may be corrected by applying a range weight instead of the temporal weight, or by applying both the temporal weight and the range weight.

[0077] In operation S830, the range weight to be applied to the plurality of frame depth values of the first pixel may be calculated.

[0078] The range weight may be calculated using the above Equation 2.

[0079] In operation S840, a new depth value that is a corrected first frame depth value of the first pixel may be calculated by applying the calculated temporal weight and/or the range weight. The process of calculating the corrected first frame depth value of the first pixel is described above with reference to the above Equation 3.

[0080] According to an embodiment, operations S810 through S840 may be iteratively performed with respect to other pixels within the first frame after operations S810 through S840 are performed with respect to the first pixel.

[0081] Even though the above example describes that a noise removal process may be sequentially performed with respect to a plurality of pixels, the noise removal process may be performed in parallel. In this case, a matrix calculation with respect to a depth value may be employed.

[0082] The image processing method according to the above-described embodiments may also be implemented through computer readable code/instructions in/on a medium, e.g., a computer readable medium, to control at least one processing element to implement any above described embodiment. The medium can correspond to any medium/media permitting the storing and/or transmission of the computer readable code.

[0083] The computer readable code can be recorded/transferred on a medium in a variety of ways, with examples of the medium including recording media, such as magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROMs, or DVDs), and transmission media such as media carrying or including carrier waves, as well as elements of the Internet, for example. Thus, the medium may be such a defined and measurable structure including or carrying a signal or information, such as a device carrying a bitstream, for example, according to embodiments of the present invention. The media may also be a distributed network, so that the computer readable code is stored/transferred and executed in a distributed fashion. Still further, as only an example, the processing element could include a processor or a computer processor, and processing elements may be distributed and/or included in a single device.

[0084] Although a few embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles of the disclosure, the scope of which is defined by the claims and their equivalents.


Claims

1. An image processing apparatus (100), comprising:

a first calculator (120; 121; 122) to calculate weights of frame values of pixels in images; and

a second calculator (120; 123) to calculate corrected frame values of pixels using linear sums calculated by applying calculated weights to the frame values of the pixels,

wherein the images comprise depth images and the frame values comprise frame depth values,

wherein the first calculator (120; 121; 122) is arranged to calculate a first weight of a first frame depth value of a first pixel of a first depth image at a first obtainment time and at least a second weight of a second frame depth value of the first pixel of a second depth image at a second obtainment time, based on at least one of

- a difference between the first frame depth value of the first pixel at the first obtainment time and the second frame depth value of the first pixel at the second obtainment time, and

- a difference in time between obtainment times between frames,

wherein the second calculator (120; 123) is arranged to determine a corrected first frame depth value of the first pixel using a linear sum calculated by applying the first weight to the first frame depth value of the first pixel, and by applying the second weight to the second frame depth value of the first pixel, and

wherein the second obtainement time occurs before the first obtainement time, and the corrected first frame depth value is determined without using any frame depth value acquired after the first obtainment time.


 
2. The image processing apparatus of claim 1, wherein, as the difference between the first frame depth value of the first pixel at the first obtainment time and the second frame depth value of the first pixel at the second obtainment time becomes greater, the first calculator (120; 121; 122) adjusts a difference between the first weight and the second weight to be greater.
 
3. The image processing apparatus of claim 1 or 2, wherein, as the difference between the first frame depth value of the first pixel at the first obtainment time and the second frame depth value of the first pixel at the second obtainment time becomes greater, the first calculator (120; 121; 122) adjusts a ratio of the second weight to the first weight to be smaller.
 
4. The image processing apparatus of claim 3, wherein, as the difference between the first frame depth value of the first pixel at the first obtainment time and the second frame depth value of the first pixel at the second obtainment time becomes greater, the first calculator (120; 121; 122) adjusts the ratio of the second weight to the first weight to be smaller along a Gaussian curve distribution.
 
5. The image processing apparatus of any of the preceding claims, wherein the first weight and the at least one second weight are normalized with respect to a number of first and second weights.
 
6. The image processing apparatus of any one of the preceding claims, wherein, as the difference between obtainment times between frames becomes greater, the first calculator (120; 121; 122) adjusts a difference between the first weight and the second weight to be greater.
 
7. The image processing apparatus of any one of the preceding claims, wherein, as the difference between obtainment times between frames becomes greater, the first calculator (120; 121; 122) adjusts a ratio of the second weight to the first weight to be smaller.
 
8. The image processing apparatus of any one of the preceding claims, wherein, as the difference between obtainment times between frames becomes greater, the first calculator (120; 121; 122) adjusts the ratio of the second weight to the first weight to be smaller along a Gaussian curve distribution.
 
9. An image processing method, comprising:

first, calculating weights of frame values of pixels in images; and

second, calculating corrected frame values of pixels using linear sums calculated by applying calculated weights to the frame values of the pixels,

wherein the images comprise depth images and the frame values comprise frame depth values,

the first calculation comprises calculating a first weight of a first frame depth value of a first pixel of a first depth image at a first obtainment time and at least a second weight of a second frame depth value of the first pixel of a second depth image at a second obtainment time, based on at least one of:

- a difference between the first frame depth value of the first pixel at the first obtainment time and the second frame depth value of the first pixel at the second obtainment time, and

- a difference in time between obtainment times between frames,

the second calculation comprises determining a corrected first frame depth value of the first pixel using a linear sum calculated by applying the first weight to the first frame depth value of the first pixel, and by applying the second weight to the second frame depth value of the first pixel, and

wherein the second obtainement time occurs before the first obtainement time, and the corrected first frame depth value is determined without using any frame depth value acquired after the first obtainment time.


 


Ansprüche

1. Bildverarbeitungsvorrichtung (100), umfassend:

einen ersten Rechner (120; 121; 122) zur Berechnung von Gewichten von Rahmenwerten von Pixeln in Bildern; und

einen zweiten Rechner (120; 123) zur Berechnung von korrigierten Rahmenwerten von Pixeln mit Hilfe von linearen Summen, die durch Anwenden der berechneten Gewichte auf die Rahmenwerte der Pixel berechnet werden,

wobei die Bilder Tiefenbilder umfassen und die Rahmenwerte Rahmentiefewerte umfassen,

wobei der erste Rechner (120; 121; 122) dazu ausgerichtet ist, ein erstes Gewicht eines ersten Rahmentiefewerts eines ersten Pixels eines ersten Tiefenbildes an einem ersten Einholungszeitpunkt zu berechnen und wenigstens ein zweites Gewicht eines zweiten Rahmentiefewerts des ersten Pixels eines zweiten Tiefenbildes an einem zweiten Einholungszeitpunkt zu berechnen, basierend auf:

- einem Unterschied zwischen dem ersten Rahmentiefewert des ersten Pixels an einem ersten Einholungszeitpunkt und dem zweiten Rahmentiefewert des ersten Pixels an dem zweiten Einholungszeitpunkt, und/oder

- einem Zeitunterschied zwischen Einholungszeitpunkten zwischen Rahmen,

wobei der zweite Rechner (120; 123) dazu ausgerichtet ist, einen korrigierten ersten Rahmentiefewert des ersten Pixels mit Hilfe einer linearen Summe, die durch Anwenden des ersten Gewichts auf den ersten Rahmentiefewert des ersten Pixels und durch Anwenden des zweiten Gewichts auf den zweiten Rahmentiefewert des ersten Pixels berechnet wird, zu bestimmen, und
wobei der zweite Einholungszeitpunkt vor dem ersten Einholungszeitpunkt liegt und der korrigierte erste Rahmentiefewert ohne Verwendung eines Rahmentiefewerts, der nach dem ersten Einholungszeitpunkt erlangt wurde, bestimmt wird.
 
2. Bildverarbeitungsvorrichtung gemäß Anspruch 1, wobei, während der Unterschied zwischen dem ersten Rahmentiefewert des ersten Pixels an dem ersten Einholungszeitpunkt und dem zweiten Rahmentiefewert des ersten Pixels an dem zweiten Einholungszeitpunkt größer wird, der erste Rechner (120; 121; 122) einen Unterschied zwischen dem ersten Gewicht und dem zweiten Gewicht so einstellt, dass dieser größer ist.
 
3. Bildverarbeitungsvorrichtung gemäß Anspruch 1 oder 2, wobei, während der Unterschied zwischen dem ersten Rahmentiefewert des ersten Pixels an dem ersten Einholungszeitpunkt und dem zweiten Rahmentiefewert des ersten Pixels an dem zweiten Einholungszeitpunkt größer wird, der erste Rechner (120; 121; 122) das Verhältnis des zweiten Gewichts zu dem ersten Gewicht so einstellt, dass dieses kleiner ist.
 
4. Bildverarbeitungsvorrichtung gemäß Anspruch 3, wobei, während der Unterschied zwischen dem ersten Rahmentiefewert des ersten Pixels an dem ersten Einholungszeitpunkt und dem zweiten Rahmentiefewert des ersten Pixels an dem zweiten Einholungszeitpunkt größer wird, der erste Rechner (120; 121; 122) das Verhältnis des zweiten Gewichts zu dem ersten Gewicht so einstellt, dass dieses kleiner ist entlang einer Gauß'schen Verteilungskurve.
 
5. Bildverarbeitungsvorrichtung gemäß einem der vorangegangenen Ansprüche, wobei das erste Gewicht und das wenigstens eine zweite Gewicht im Bezug auf eine Zahl von ersten und zweiten Gewichten normalisiert werden.
 
6. Bildverarbeitungsvorrichtung gemäß einem der vorangegangenen Ansprüche, wobei, während der Unterschied zwischen den Einholungszeitpunkten zwischen den Rahmen größer wird, der erste Rechner (120; 121; 122) einen Unterschied zwischen dem ersten Gewicht und dem zweiten Gewicht so einstellt, dass dieser größer ist.
 
7. Bildverarbeitungsvorrichtung gemäß einem der vorangegangenen Ansprüche, wobei, während der Unterschied zwischen den Einholungszeitpunkten zwischen den Rahmen größer wird, der erste Rechner (120; 121; 122) das Verhältnis des zweiten Gewichts zu dem ersten Gewicht so einstellt, dass dieses kleiner ist.
 
8. Bildverarbeitungsvorrichtung gemäß einem der vorangegangenen Ansprüche, wobei, während der Unterschied zwischen den Einholungszeitpunkten zwischen den Rahmen größer wird, der erste Rechner (120; 121; 122) das Verhältnis des zweiten Gewichts zu dem ersten Gewicht so einstellt, dass dieses kleiner ist entlang einer Gauß'schen Verteilungskurve.
 
9. Bildverarbeitungsverfahren, umfassend:

erstens das Berechnen von Gewichten von Rahmenwerten von Pixeln in Bildern; und zweitens das Berechnen von korrigierten Rahmenwerten von Pixeln mit Hilfe von linearen Summen, die durch Anwenden der berechneten Gewichte auf die Rahmenwerten der Pixel berechnet werden,

wobei die Bilder Tiefenbilder umfassen und die Rahmenwerte Rahmentiefewerte umfassen,

wobei die erste Berechnung das Berechnen eines ersten Gewichts eines ersten Rahmentiefewerts eines ersten Pixels eines ersten Tiefenbildes an einem ersten Einholungszeitpunkt und wenigstens eines zweiten Gewichts eines zweiten Rahmentiefewerts des ersten Pixels an einem zweiten Einholungszeitpunkt umfasst, basierend auf:

- einem Unterschied zwischen dem ersten Rahmentiefewert des ersten Pixels an dem ersten Einholungszeitpunkt und dem zweiten Rahmentiefewert des ersten Pixels an dem zweiten Einholungszeitpunkt, und/oder

- einen Zeitunterschied zwischen Einholungszeitpunkten zwischen Rahmen,

wobei die zweite Berechnung das Bestimmen eines korrigierten ersten Rahmentiefewerts des ersten Pixels mit Hilfe einer linearen Summe, die durch Anwenden des ersten Gewichts auf den ersten Rahmentiefewert des ersten Pixels und durch Anwenden des zweiten Gewichts auf den zweiten Rahmentiefewert des ersten Pixels berechnet wird, umfasst, und
wobei der zweite Einholungszeitpunkt vor dem ersten Einholungszeitpunkt liegt, und der korrigierte erste Rahmentiefewert ohne Verwendung eines Rahmentiefewerts, der nach dem ersten Einholungszeitpunkt erlangt wurde, bestimmt wird.
 


Revendications

1. Appareil de traitement d'image (100), comprenant:

un premier calculateur (120 ; 121 ; 122) destiné à calculer des pondérations de valeurs de trame des pixels d'images; et

un deuxième calculateur (120 ; 123) destiné à calculer des valeurs de trame de pixels corrigées au moyen de sommes linéaires calculées en appliquant les pondérations calculées aux valeurs de trame des pixels,

dans lequel les images comprennent des images en profondeur et les valeurs de trame comprennent des valeurs de profondeur de trame,

dans lequel le premier calculateur (120 ; 121 ; 122) est agencé pour calculer une première pondération d'une première valeur de profondeur de trame d'un premier pixel d'une première image en profondeur à un premier moment d'obtention et au moins une deuxième pondération d'une deuxième valeur de profondeur de trame du premier pixel d'une deuxième image en profondeur à un deuxième moment d'obtention, en fonction de:

- une différence entre la première valeur de profondeur de trame du premier pixel au premier moment d'obtention et la deuxième valeur de profondeur de trame du premier pixel au deuxième moment d'obtention, et/ou de

- une différence de temps entre les moments d'obtention entre les trames,

dans lequel le deuxième calculateur (120 ; 123) est agencé pour déterminer une première valeur corrigée de profondeur de trame du premier pixel au moyen d'une somme linéaire calculée en appliquant la première pondération à la première valeur de profondeur de trame du premier pixel, et en appliquant la deuxième pondération à la deuxième valeur de profondeur de trame du premier pixel, et
dans lequel le deuxième moment d'obtention se produit avant le premier moment d'obtention, et la première valeur corrigée de profondeur de trame est déterminée sans employer de valeur de profondeur de trame acquise après le premier moment d'obtention.
 
2. Appareil de traitement d'image selon la revendication 1, dans lequel, tandis que s'accroît la différence entre la première valeur de profondeur de trame du premier pixel au premier moment d'obtention et la deuxième valeur de profondeur de trame du premier pixel au deuxième moment d'obtention, le premier calculateur (120; 121; 122) règle la différence entre la première pondération et la deuxième pondération de manière qu'elle augmente.
 
3. Appareil de traitement d'image selon la revendication 1 ou 2, dans lequel, tandis que s'accroît la différence entre la première valeur de profondeur de trame du premier pixel au premier moment d'obtention et la deuxième valeur de profondeur de trame du premier pixel au deuxième moment d'obtention, le premier calculateur (120 ; 121 ; 122) règle le rapport entre la deuxième pondération et la première pondération de manière qu'il diminue.
 
4. Appareil de traitement d'image selon la revendication 3, dans lequel, tandis que s'accroît la différence entre la première valeur de profondeur de trame du premier pixel au premier moment d'obtention et la deuxième valeur de profondeur de trame du premier pixel au deuxième moment d'obtention, le premier calculateur (120 ; 121 ; 122) règle le rapport entre la deuxième pondération et la première pondération de manière qu'il diminue selon une distribution gaussienne.
 
5. Appareil de traitement d'image selon l'une quelconque des revendications précédentes, dans lequel la première pondération et l'au moins une deuxième pondération sont normalisées par rapport à un nombre des première et deuxième pondérations.
 
6. Appareil de traitement d'image selon l'une quelconque des revendications précédentes, dans lequel, tandis que s'accroît la différence entre les moments d'obtention entre les trames, le premier calculateur (120 ; 121 ; 122) règle la différence entre la première pondération et la deuxième pondération de manière qu'elle augmente.
 
7. Appareil de traitement d'image selon l'une quelconque des revendications précédentes, dans lequel, tandis que s'accroît la différence entre les moments d'obtention entre les trames, le premier calculateur (120 ; 121 ; 122) règle le rapport entre la deuxième pondération et la première pondération de manière qu'il diminue.
 
8. Appareil de traitement d'image selon l'une quelconque des revendications précédentes, dans lequel, tandis que s'accroît la différence entre les moments d'obtention entre les trames, le premier calculateur (120 ; 121 ; 122) règle le rapport entre la deuxième pondération et la première pondération de manière qu'il diminue selon une distribution gaussienne.
 
9. Procédé de traitement d'image, comprenant:

tout d'abord, le calcul de pondérations de valeurs de trame de pixels d'images ; et

ensuite, le calcul de valeurs de trame de pixels corrigées au moyen de sommes linéaires calculées en appliquant les pondérations calculées aux valeurs de trame des pixels,

dans lequel les images comprennent des images en profondeur et les valeurs de trame comprennent des valeurs de profondeur de trame,
le premier calcul comprend le calcul d'une première pondération d'une première valeur de profondeur de trame d'un premier pixel d'une première image en profondeur à un premier moment d'obtention et d'au moins une deuxième pondération d'une deuxième valeur de profondeur de trame du premier pixel d'une deuxième image en profondeur à un deuxième moment d'obtention, en fonction de:

- une différence entre la première valeur de profondeur de trame du premier pixel au premier moment d'obtention et la deuxième valeur de profondeur de trame du premier pixel au deuxième moment d'obtention, et/ou de

- une différence de temps entre les moments d'obtention entre trames,

le deuxième calcul comprend la détermination d'une première valeur corrigée de profondeur de trame du premier pixel au moyen d'une somme linéaire calculée en appliquant la première pondération à la première valeur de profondeur de trame du premier pixel, et en appliquant la deuxième pondération à la deuxième valeur de profondeur de trame du premier pixel, et

dans lequel le deuxième moment d'obtention se produit avant le premier moment d'obtention, et la première valeur corrigée de profondeur de trame est déterminée sans employer de valeur de profondeur de trame acquise après le premier moment d'obtention.


 




Drawing


























Cited references

REFERENCES CITED IN THE DESCRIPTION



This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.

Non-patent literature cited in the description