| (11) | EP 2 767 889 B1 |
| (12) | EUROPEAN PATENT SPECIFICATION |
|
|
| (54) | APPARATUS AND METHOD FOR RECOGNIZING MOTION BY USING EVENT-BASED VISION SENSOR VORRICHTUNG UND VERFAHREN ZUR BEWEGUNGSERKENNUNG MITTELS EREIGNISBASIERTER SICHTSENSOREN APPAREIL ET PROCÉDÉ PERMETTANT DE RECONNAÎTRE UN MOUVEMENT AU MOYEN D'UN CAPTEUR DE VISION BASÉ SUR UN ÉVÉNEMENT |
|
| |||||||||||||||||||||||||||||||
| Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention). |
[0001] Methods and apparatuses according to the exemplary embodiments relate to an apparatus and method for recognizing motion that tracks a movement trajectory of a finger or a movement direction of a hand using a vision sensor for sensing a movement-occurring part, and controls motion of a device.
[0002] A user interface (UI) for interaction between humans and electronic devices may include, for example, a keyboard, a mouse and a touch panel. For example, based on touch technology used in the touch panel, a user may adjust a UI through direct contact with a screen. As a number of touch instances increases, damage may occur on the screen of the panel, and the user may experience an unhygienic effect due to repeated direct contact. Accordingly, there is a need for providing an intuitive interfacing technology which reinforces a natural sense of interaction between humans and electronic devices.
[0003] Here, the article "Dynamic vision sensor camera based bare hand gesture recognition" by Jun Haeng Lee et al (XP032333574) is acknowledged as prior art, relative to which, in the appended independent claims, at least features are novel with respect to distinguishing between smaller movements of a portion of an object and larger movements of a whole object, to distinctly track a movement trajectory of the portion of the object or determine a movement direction for the whole of the object, and employing the determined movement trajectory or movement direction for the motion controller. US-5454043 merely relates to gesture input to control a user interface.
[0004] According to an aspect of the exemplary embodiments, there is provided an apparatus having features of the appended independent apparatus claim.
[0005] The apparatus is distinguished over the above acknowledged prior art, which merely disclosed determination of a direction of movement for relatively large movements, by making a distinction between small movement and large movement, whereby a motion controller is provided with appropriate input for proper selection of a control instruction to send to the device.
[0006] According to an aspect of the exemplary embodiments, there is provided a method having features of the appended independent method claim.
[0007] According to exemplary embodiments, there is provided an apparatus and method for recognizing motion using a vision sensor that receives an input of events corresponding to a movement-occurring part from a vision sensor and determines a type of movement based on a frequency of occurrence of events, determines a motion pattern based on a movement trajectory in response to a result of the movement type determination indicating a small movement, determines a movement direction in which an object moves in response to a result of the movement type determination indicating a large movement, and controls a device based on the determined motion pattern or the determined movement direction, thereby relatively rapidly recognizing a movement of an object at a speed of 1 millisecond (ms), has a low power consumption, and is insusceptible to the brightness of an environment.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008]
FIG. 1 is a diagram which illustrates an example of a configuration of an apparatus for recognizing motion according to an exemplary embodiment.
FIG. 2 is a graph which illustrates a method of determining a type of movement using an occurrence frequency of events according to an exemplary embodiment.
FIG. 3 is a diagram which illustrates an example of a configuration of a first motion determination unit according to an exemplary embodiment.
FIG. 4 is a diagram which illustrates an example of an array of sensing units configuring a vision sensor and receptive fields according to an exemplary embodiment.
FIG. 5 is a diagram which illustrates a spatial correlation amongst a first to n-th spatiotemporal correlators according to an exemplary embodiment.
FIG. 6 is a diagram which illustrates an example of generating first to third clusters by grouping first to n-th spatiotemporal correlators and setting clustering areas according to an exemplary embodiment.
FIG. 7 is a diagram which illustrates an example of tracking a movement trajectory of a movement-occurring part based on generated clustering areas according to an exemplary embodiment.
FIG. 8 is a diagram which illustrates an example of a configuration of a second motion determination unit according to an exemplary embodiment.
FIG. 9 is a diagram which illustrates an example of a winner takes all circuit according to an exemplary embodiment.
FIG. 10 is a diagram which illustrates another example of a winner takes all circuit according to an exemplary embodiment.
FIG. 11 is a diagram which illustrates a form of a receptive field for detecting a movement direction according to an exemplary embodiment.
FIG. 12 is a diagram which illustrates detected directions in response to a winner takes all circuit detects eight directions according to an exemplary embodiment.
FIG. 13 is a diagram which illustrates an example of weights applied to a plurality of neurons of a winner takes all circuit included in a first direction detection filter according to an exemplary embodiment.
FIG. 14 is a diagram which illustrates an example of a configuration of a winner takes all circuit included in a second direction detection filter according to an exemplary embodiment.
FIG. 15 is a diagram which illustrates an example of weights applied to a plurality of neurons of a winner takes all circuit included in a second direction detection filter according to an exemplary embodiment.
FIG. 16 is a diagram which illustrates an example of a configuration of a winner takes all circuit based on a type of events received via a vision sensor according to an exemplary embodiment.
FIG. 17 is a flowchart which illustrates an operation of an apparatus for recognizing motion according to an exemplary embodiment.
FIG. 18 is a flowchart which illustrates an operation of recognizing motion of a finger according to an exemplary embodiment.
FIG. 19 is a flowchart which illustrates an operation of detecting a movement direction of a hand according to an exemplary embodiment.
DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS
[0009] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Exemplary embodiments are described below in order to explain the exemplary embodiments by referring to the figures.
[0010] FIG. 1 is a diagram which illustrates an example of a configuration of an apparatus 100 for recognizing motion according to an exemplary embodiment.
[0011] Referring to FIG. 1, the apparatus 100 for recognizing the motion that recognizes a motion pattern based on a small movement such as a finger, or recognizes a movement direction of a large movement such as a palm of a hand includes a vision sensor 110, a movement type determination unit 120, e.g., a motion type determiner, a first motion determination unit 130, e.g. a first motion determiner 130 a second motion determination unit 140, e.g., a second motion determiner 140, a third motion determination unit 145, e.g., a third motion determiner 145, and a motion control unit 150, e.g., a motion controller.
[0012] The vision sensor 110 senses a movement-occurring part within an object and outputs sensed events. The vision sensor 110 includes an array of a plurality of sensing units, e.g. sensors, as illustrated in FIG. 4. The sensing units may be provided in a pixel unit of an image. For example, in response to a touch screen for an interface outputting an image of 60 × 60, sensing units may be provided in a form of 60 × 60.
[0013] The sensing units refers to light receiving elements. In response to an object or a portion of the object moving, the intensity of light sensed by sensing units of a first image sensor 111 may vary. In an instance of the first image sensor 111, target sensing units, e.g. target sensors, from which a change in light intensity is sensed from among the sensing units output event signals. Target sensing units which correspond to a movement-occurring part from among the sensing units output event signals. The target sensing units may refer to sensing units that output events.
[0014] An event may include information, for example, an event-occurring time, a location of a sensing unit outputting an event, and a polarity. The polarity may be "ON" in response to an event caused by an increase of intensity of light received from a sensing unit occurring, and may be "OFF" when an event caused by a decrease of intensity of light received occurs. The sensing unit may output an event in response to a level of change in light intensity being greater than or equal to a predetermined reference value.
[0015] The movement type determination unit 120 determines a type of movement using an occurrence frequency, e.g., a frequency of occurrence of events outputted through the vision sensor 110. The type of movement includes a small movement and a large movement. As an example, the small movement may be a forward/backward movement of an object, for example, a hand, or a movement of a finger, and the large movement may be a movement of an entire hand, for example, a palm of a hand.
[0016] The movement type determination unit 120 may detect one of the forward/backward movement of the hand and the movement of the finger, or distinguish between the forward/backward movement of the hand and the movement of the finger based on more detailed criteria, in response to a request of a predetermined user when a small movement is detected.
[0017] In various examples described herein, the small movement is referred to as a finger movement, and the large movement is referred to as a hand movement, for conciseness and ease of description.
[0018] The movement type determination unit 120 determines the type of movement, as shown in an example of FIG. 2.
[0019] FIG. 2 is a graph which illustrates a method of determining a type of movement using an occurrence frequency of events according to an exemplary embodiment.
[0020] Referring to FIG. 2, an occurrence frequency of events of the vision sensor 110 is low when a finger moves, and an occurrence frequency of events of the vision sensor 110 is high when a hand moves. Accordingly, the movement type determination unit 120 determines whether a finger moves or an entire hand moves, based on a predetermined threshold value.
[0021] The first motion determination unit 130 tracks a movement trajectory of a movement-occurring part, and determines a motion pattern based on a movement trajectory based on a spatiotemporal correlation of events outputted from the vision sensor 110.
[0022] FIG. 3 is a diagram which illustrates an example of a configuration of a first motion determination unit 130 according to an exemplary embodiment.
[0023] Referring to FIG. 3, the first motion determination unit 130 includes first to n-th spatiotemporal correlators 312, 314, and 316, a movement trajectory tracking unit 320, e.g., a movement trajectory tracker 320, a pattern storage unit 330, e.g., a pattern storage, and a motion pattern determination unit 340, e.g. a motion pattern determiner 330.
[0024] The spatiotemporal correlators 312, 314, and 316 may calculate a plurality of spatiotemporal correlations amongst target sensing units based on events inputted from the target sensing units. The spatiotemporal correlators 312, 314, and 316 may be implemented by spiking neurons.
[0025] Hereinafter, for descriptions of a spatiotemporal correlation, a receptive field will be discussed with reference to FIG. 4, and an example of a first spatiotemporal correlator 121-1 will be used.
[0026] FIG. 4 is a diagram which illustrates an example of an array of sensing units configuring a vision sensor and receptive fields according to an exemplary embodiment.
[0027] Referring to FIG. 4, the first spatiotemporal correlator 312 may receive an input of an event outputted from sensing units forming a predetermined area of the vision sensor 110. The predetermined area may refer to a receptive field. An area electrically connected with the first spatiotemporal correlator 312 and occupied by sensing units that output events to the first spatiotemporal correlator 312 may refer to a receptive field. The receptive field may have a size of m × m, "m" being a positive number. Accordingly, the first spatiotemporal correlator 312 may receive an input of events from at least one of the sensing units of the receptive fields. The second to n-th spatiotemporal correlators 314 through 316 may also receive an input of events by being connected to sensing units of a corresponding receptive field.
[0028] Referring to FIG. 3, the first to n-th spatiotemporal correlators 312, 314, and 316 may have a plurality of internal state values which respectively indicate current spatiotemporal correlations. The plurality of internal state values may be identical to or differ from one another, and an example may include a voltage value. The internal state value may be determined by a current internal state value and a new input of an event. In response to an event being input, an internal state value may increase, and in an absence of an event, an internal state value may decrease based on a predetermined period of elapsed time. The decrease in the internal state value may minimize a bandwidth load of a memory storing the internal state value.
[0029] The first to n-th spatiotemporal correlators 312, 314, and 316 may increase the internal state value in response to events being input from the corresponding vision sensor 110 to compare the increased internal state value to a predetermined threshold value, in order to determine a fluctuation of a spatiotemporal correlation. The spatiotemporal correlation may indicate a temporal correlation and a spatial correlation of events respectively inputted to the first to n-th spatiotemporal correlators 312, 314, and 316.
[0030] Equation 1 is an equation for calculating an output value outputted from a spatiotemporal correlator in response to an input of an event.
[0031] In Equation 1, Qn(t) denotes an internal state value of an n-th spatiotemporal correlator at a point of time "t", Qn(tprev) denotes an internal state value of an n-th spatiotemporal correlator at a previous point of time; for example, a current internal state value, and "tprev" denotes a point of time at which a most recent event set occurs from among previously inputted event sets. e(t) denotes an event set inputted at a point of time "t", outn(t) denotes an output value of an n-th spatiotemporal correlator, and φ denotes a threshold value. An event set may refer to basic information related to events occurring at a point of time "t". For example, in response to three events occurring at a point of time "t = 0", e(0) = {e_1, e_2, e_3}. "e_n", where "n" = 1, 2, 3, ..., denotes an event of an n-th target sensing unit.
[0032] The n-th spatiotemporal correlator may increase a previous internal state value Qn(tprev) in response to an event being inputted from a single target sensing unit from among sensing units of a corresponding receptive field at a point of time "t". A degree of the increase may be affected by a weight set for the target sensing unit from which the event occurs. Accordingly, in response to a plurality of events being simultaneously input, a speed of the increase may correspondingly accelerate. Differing weights may be set among a plurality of target sensing units.
[0033] The first to n-th spatiotemporal correlators 312, 314, and 316 may output output values differing based on a fluctuation of spatiotemporal correlations to the movement trajectory tracking unit 320. For example, the n-th spatiotemporal correlator may determine that a spatiotemporal correlation amongst target sensing units outputting events to the n-th spatiotemporal correlator is high and output "1" in response to an internal state value Qn(t) exceeding a predetermined threshold value , the n-th spatiotemporal correlator may determine that a spatiotemporal correlation is low and output zero when the internal state value Qn(t) is less than or equal to the predetermined threshold value . "1" may be outputted in a form of "high" or "true", and zero may be outputted in a form of "low" or "false". Alternatively, in response to an output of the n-th spatiotemporal correlator being "1", the n-th spatiotemporal correlator may reduce an internal state value of the n-th spatiotemporal correlator 121-n by a predetermined value.
[0034] The internal state value Qn(t) calculated by the n-th spatiotemporal correlator may indicate a spatial correlation and a temporal correlation of events inputted to the n-th spatiotemporal correlator from target sensing units. For example, in response to a series of events being successfully inputted to the n-th spatiotemporal correlator from a single target sensing unit, an internal state value Qn(t) may indicate a temporal correlation amongst the inputted events. In response to two events being simultaneously inputted to the n-th spatiotemporal correlator from two target sensing units, for example, in response to two target sensing units being adjacent to each other and connected to an identical n-th spatiotemporal correlator, the two events may have a high level of spatial correlation. The two events may also have a temporal correlation because the two events are simultaneously inputted from the two target sensing units.
[0035] The first to n-th spatiotemporal correlators 312, 314, and 316 may have a spatial correlation differing from one another.
[0036] FIG. 5 is a diagram illustrating a spatial correlation amongst first to n-th spatiotemporal correlators according to an exemplary embodiment.
[0037] Referring to FIG. 5, the vision sensor 110 may be logically divided into a plurality of receptive fields. The plurality of divided receptive fields may overlap at least one neighboring area. The first to n-th spatiotemporal correlators 312, 314, and 316 may be mapped to the plurality of divided receptive fields of the vision sensor 110. Sensing units disposed in the plurality of divided receptive fields may output an event to a corresponding spatiotemporal correlator from among the first to n-th spatiotemporal correlators 312, 314, and 316.
[0038] In FIG. 5, C(i, j) denotes center coordinates of a receptive field disposed at a center of the vision sensor 110. C(i-1, j) denotes center coordinates of a receptive field that moves towards an "x" axis direction, for example, a horizontal direction, by a degree of "1". In FIG. 4, "1" may indicate spacing by a degree of three pixels, and the degree of spacing may be subject to change.
[0039] As shown in FIG. 5, an overlapping portion of the receptive fields may indicate that an identical event may be simultaneously outputted to at least two spatiotemporal correlators. A spatial correlation may be given among the first to n-th correlators 312, 314, and 316 or a spatial correlation may be given among the receptive fields due to the overlapping portion of the receptive fields. The spatiotemporal correlation among the first to n-th spatiotemporal correlators 312, 314, and 316 may affect tracking a movement trajectory.
[0040] Referring to FIG. 3, the movement trajectory tracking unit 320 may track a movement trajectory of a movement-occurring part based on output values of spatiotemporal correlations calculated from the first to n-th spatiotemporal correlators 312, 314, and 316, respectively, and internal state values of the first to n-th spatiotemporal correlators.
[0041] The movement trajectory tracking unit 320 may generate a single cluster by grouping spatiotemporal correlators having a high spatiotemporal correlation from among the first to n-th spatiotemporal correlators 312, 314, and 316. The movement trajectory tracking unit 320 may group spatiotemporal correlators that output "1" from among the first to n-th spatiotemporal correlators 312, 314, and 316 based on overlapping of receptive fields. The movement trajectory tracking unit 320 may generate a single cluster by grouping overlapping spatiotemporal correlators. Outputting "1" may indicate having a high spatiotemporal correlation.
[0042] The movement trajectory tracking unit 320 may generate a cluster by grouping spatiotemporal correlators of which an output is "1", and set a predetermined area including the cluster to be a clustering area.
[0043] FIG. 6 is a diagram which illustrates an example of generating first to third clusters by grouping first to n-th spatiotemporal correlators and setting clustering areas according to an exemplary embodiment.
[0044] Referring to FIG. 6, the first cluster, hereinafter also referred to as "cluster 1", may be generated by grouping two receptive fields 601 and 602 overlapping each other. The second cluster, hereinafter also referred to as a "cluster 2", may be generated by grouping ten overlapping receptive fields. The third cluster, hereinafter also referred to as a "cluster 3", may be generated by grouping seven overlapping receptive fields.
[0045] A first clustering area 610 may refer to a predetermined area including the cluster 1. A second clustering area 620 may refer to a predetermined area including the cluster 2. A third clustering area 630 may refer to a predetermined area including the cluster 3. A clustering area may be generated based on a center of a cluster.
[0046] The movement trajectory tracking unit 320 may calculate a center of a clustering area to be a location of an object. The movement trajectory tracking unit 320 may calculate the center of the clustering area based on internal state values of spatiotemporal correlators within the clustering area.
[0047] The movement trajectory tracking unit 320 may re-calculate a center of a previous clustering area based on internal state values of spatiotemporal correlators within the previous clustering area in an absence of spatiotemporal correlators of which an output is "1", and calculate the re-calculated center of the previous clustering area to be a location of an object.
[0048] The movement trajectory tracking unit 320 may track a movement trajectory in conjunction with the calculated location of the object and the re-calculated location of the object. The cluster may have a form which corresponds to a movement-occurring part.
[0049] The movement trajectory tracking unit 320 may multiply a location of a plurality of spatiotemporal correlators included in a clustering area and a constant, and determine an average value of the multiplication to be a center of a clustering area. The location of the spatiotemporal correlators may be a location representative of an area covered by the spatiotemporal correlator. The constant may include various values, for example, an internal state value Q(t) calculated from the spatiotemporal correlators and a number of events inputted to the spatiotemporal correlators during a predetermined period of time.
[0050] For example, a clustering area is generated by grouping the first and second spatiotemporal correlators 312 and 314, and an internal state value of the first spatiotemporal correlator 312 is Q1, and an internal state value of the second spatiotemporal correlator 314 is Q2. A location representative of the first spatiotemporal correlator 312 corresponds to (x1, y1), and a location representative of the second spatiotemporal correlator 314 corresponds to (x2, y2). The movement trajectory tracking unit 320 may calculate a center of the clustering area based on Equation 2.
In Equation 2, "x"' denotes an "x" coordinate of a center of a clustering area, and "y‴ denotes a "y" coordinate of a center of a clustering area.
[0051] FIG. 7 is a diagram which illustrates an example of tracking a movement trajectory of a movement-occurring part based on generated clustering areas according to an exemplary embodiment.
[0052] The movement trajectory tracking unit 320 may generate a clustering area, or calculate a center of the clustering area at predetermined intervals. Referring to FIG. 7, the movement trajectory tracking unit 320 may generate a clustering area at sequential points of time "t1", "t2", and "t3", and calculate a center of the clustering area. The movement trajectory tracking unit 320 may track a movement trajectory in conjunction with the calculated center. A movement trajectory of a movement-occurring part may be tracked by calculating locations of grouped spatiotemporal correlators; for example, grouped receptive fields.
[0053] Referring to FIG. 3, the motion pattern determination unit 330 may determine a motion pattern of the movement-occurring part from the movement trajectory tracked by the movement trajectory tracking unit 320. The motion pattern determination unit 330 may obtain feature components for representing a motion pattern from the tracked movement trajectory and compare the feature components to motion patterns stored in the pattern storage unit 340, e.g. the pattern storage, to determine the motion pattern of the movement-occurring part. The feature components may include various parameters, for example, a location of a cluster, a direction in which the cluster moves, and a movement angle of the cluster.
[0054] The pattern storage unit 340 may store values of a plurality of feature components and motion patterns which correspond to the plurality of feature components.
[0055] Referring to FIG. 1, the second motion determination unit 140 detects a movement direction in which an object moves based on event signals outputted from the vision sensor 110.
[0056] FIG. 8 is a diagram which illustrates an example of a configuration of the second motion determination unit 140 according to an exemplary embodiment.
[0057] Referring to FIG. 8, the second motion determination unit 140 includes a first direction detection filter 810 and a second direction detection filter 820.
[0058] The first direction detection filter 810 may detect a movement direction in which an object moves for a plurality of receptive fields based on events outputted from the vision sensor 110.
[0059] The first direction detection filter 810 includes winner takes all circuits 812, 814, and 816 of which a number, at a minimum, corresponds to a number of receptive fields. The winner takes all circuits 812, 814, and 816 may receive an event from a corresponding receptive field and detect a movement direction in which an object moves within the corresponding receptive field.
[0060] The second direction detection filter 820 may detect a final movement direction of the object based on the movement direction detected for each of the receptive fields via the first direction detection filter 810.
[0061] The second direction detection filter 820 includes at least one final winner takes all circuit 822. The winner takes all circuit 822 may receive movement direction information related to an object from a corresponding receptive field detected for each of the winner takes all circuits 812, 814, and 816, and determine the final movement direction based on the vision sensor 110.
[0062] The second direction detection filter 820 may be implemented absent a winner takes all circuit.
[0063] The second direction detection filter 820 may calculate a vector sum of the movement direction information of the corresponding receptive field detected for each of the winner takes all circuits 812, 814, and 816 inputted from the first direction detection filter 810, and calculate the final movement direction based on the vision sensor 110.
[0064] In FIG. 8, the second motion determination unit 140 is implemented by two direction detection filters having a hierarchical structure, however, the second motion determination unit 140 may be implemented by a greater number of direction detection filters having a hierarchical structure.
[0065] The second motion determination unit 140 may detect a movement direction using a direction detection filter having a hierarchical structure. However, various changes and modifications of the filter may be made to also detect a movement, for example, using a filter for detecting a predetermined shape.
[0066] A method of configuring the winner takes all circuits 812, 814, and 816 and the final winner takes all circuit 822 will be discussed with reference to FIG. 9 or FIG. 10.
[0067] FIG. 9 is a diagram which illustrates an example of a winner takes all circuit according to an exemplary embodiment.
[0068] Referring to FIG. 9, the winner takes all circuit includes four neurons 910, 920, 930, and 940 for detecting a direction and an inhibitory neuron 950.
[0069] Each of the neurons 910, 920, 930, and 940 may receive an event inputted from a corresponding receptive field, apply a greater weight to an event in a direction detected by a corresponding neuron, and calculate an internal state value by calculating a sum of events to which the greater weight is applied.
[0070] Each of the neurons 910, 920, 930, and 940 may output a signal which indicates that a movement is detected in a corresponding direction in response to the internal state value exceeding a predetermined threshold value.
[0071] In response to an output being generated from among the neurons 910, 920, 930, and 940, the inhibitory neuron 950 may inhibit neurons from generating an output, aside from a neuron generating an initial output.
[0072] Each of the neurons 910, 920, 930, and 940 may detect four directions by detecting a single direction in an example of FIG. 9.
[0073] FIG. 10 is a diagram which illustrates another example of a winner takes all circuit according to an exemplary embodiment.
[0074] Referring to FIG. 10, the winner takes all circuit includes four neurons 1010, 1020, 1030, and 1040 for detecting a direction.
[0075] Each of the neurons 1010, 1020, 1030, and 1040 may receive an input of an event from a corresponding receptive field, apply a greater weight to an event in a direction allocated to a corresponding neuron, and calculate an internal state value by calculating a sum of events to which the greater weight is applied.
[0076] Each of the neurons 1010, 1020, 1030, and 1040 may output a signal which indicates that a movement is detected in a corresponding direction in response to the internal state value exceeding a predetermined threshold value.
[0077] In response to a neuron outputting the signal which indicates that the movement is detected in the corresponding direction occurring from among the neurons 1010, 1020, 1030, and 1040, the neuron outputting the signal may receive an output and generation of a subsequent output may be inhibited.
[0078] The neurons may inhibit one another from generating an output, aside from a neuron generating an initial output.
[0079] Each of the neurons 1010, 1020, 1030, and 1040 may detect four directions by detecting a single direction in an example of FIG. 10.
[0080] Referring to FIG. 3, a receptive field, for example, a unit providing an event to each of the winner takes all circuits 812, 814, and 816, may be implemented in various forms. However, an example of FIG. 11 will be provided herein.
[0081] FIG. 11 is a diagram which illustrates a form of a receptive field for detecting a movement direction according to an exemplary embodiment.
[0082] Referring to FIG. 11, the form of the receptive field is provided in a circular form because directivity towards a predetermined direction may occur due to the form of the receptive field. Accordingly, receptive fields of events inputted to the winner takes all circuits 812, 814, and 816 may be set to be circular receptive fields.
[0083] The winner takes all circuits 812, 814, and 816 and the final winner takes all circuit 822 may detect movements in eight directions as shown in FIG. 12 through use of eight neurons detecting differing directions. However, directions of the winner takes all circuits 812, 814, and 816 and the final winner takes all circuit 822 may be predetermined.
[0084] FIG. 12 is a diagram illustrating detected directions when a winner takes all circuit detects eight directions according to an exemplary embodiment.
[0085] Referring to FIG. 12, the winner takes all circuits 812, 814, and 816 and the final winner takes all circuit 822 may detect movements towards eight directions represented in indices of "0" through "7".
[0086] In FIG. 12, "0" indicates a movement towards a left, "1" indicates a movement towards a downward left, "2" indicates a downward movement, "3" indicates a movement towards a downward right, "4" indicates a movement towards a right, "5" indicates a movement toward an upward right, "6" indicates an upward movement, and "7" indicates a movement towards an upward left.
[0087] In response to the winner takes all circuits 812, 814, and 816 detecting eight directions as shown in FIG. 12, a weight to be applied to a receptive field connected to each of neurons detecting the directions may be set, as illustrated in FIG. 13.
[0088] FIG. 13 is a diagram which illustrates an example of weights applied to a plurality of neurons of a winner takes all circuit included in a first direction detection filter according to an exemplary embodiment.
[0089] Referring to FIG. 13, shading may indicate a degree of weight, and a weight to be applied to a receptive field connected to each neuron in a direction which corresponds to the index of FIG. 12 may have a gradient value based on the direction.
[0090] In response to the second motion determination unit 140 determining a movement in eight directions, a configuration of the final winner takes all circuit 822 may be illustrated, as in FIG. 14.
[0091] FIG. 14 is a diagram which illustrates an example of a configuration of a winner takes all circuit included in a second direction detection filter according to an exemplary embodiment.
[0092] Referring to FIG. 14, the final winner takes all circuit 822 includes nine neurons 1410, 1420 and 1430.
[0093] Each of the neurons 1410, 1420, and 1430 may receive movement direction information detected for each of the winner takes all circuits 812, 814, and 816, apply a greater weight to a direction allocated to a corresponding neuron, and calculate an internal state value of the neurons 1410, 1420, and 1430.
[0094] Each of the neurons 1410, 1420, and 1430 may output a signal indicating that a movement is detected in a corresponding direction in response to the internal state value exceeding a predetermined threshold value.
[0095] The neurons 1410, 1420, and 1430 may inhibit one another from generating an output, aside from a neuron generating an initial output.
[0096] The neuron (0) 1410 to the neuron (7) 1420 may refer to a neuron detecting a movement towards a direction corresponding to the index represented in FIG. 12.
[0097] The neuron (g) 1430 may refer to a neuron detecting an instance in which an occurrence frequency of an event is greater than a reference value, or a movement is not in a predetermined direction.
[0098] In response to the occurrence frequency of the event being greater than the reference value, or the movement not being in the predetermined direction, the movement may be a movement of a hand moving back and forth based on the vision sensor 110, or a circling movement.
[0099] In response to the final winner takes all circuit 822 detecting eight directions as shown in FIG. 12, a weight to be applied to movement direction information detected for each of the winner takes all circuits 812, 814, and 186 inputted to neurons detecting the directions may be set, as shown in FIG. 15.
[0100] FIG. 15 is a diagram which illustrates an example of weights applied to a plurality of neurons of a winner takes all circuit included in a second direction detection filter, according to an exemplary embodiment.
[0101] Referring to FIG. 15, the more similar to a corresponding direction, the higher the weight to be applied to the movement direction information detected for each of the winner takes all circuits 812, 814, and 816 inputted to the neurons 1410, 1420, and 1430 included in the final winner takes all circuit 822, and the more opposite to the corresponding direction, the lower the weight to be applied.
[0102] However, the neuron (g) 1430 may apply an identical weight to all directions because a direction to be detected is non-existent.
[0103] The winner takes all circuits 812, 814, and 816 included in the second motion determination unit 140 may be classified among a plurality of types of events outputted from the vision sensor 110.
[0104] For example, in response to a type of event outputted from the vision sensor 110 including an ON event and an OFF event, the second motion determination unit 140 may include a configuration of detecting a movement direction based on the ON event separate from a configuration of detecting a movement direction based on the OFF event, and may determine a final movement direction by integrating the movement direction detected based on the ON event and the movement direction detected based on the OFF event.
[0105] FIG. 16 is a diagram which illustrates an example of a configuration of a winner takes all circuit based on a type of events received via a vision sensor, according to an exemplary embodiment.
[0106] Referring to FIG. 16, in response to types of events outputted from the vision sensor 110 including an ON event and an OFF event, an ON event of a receptive field may be inputted to an ON event winner takes all circuit 1610, and an OFF event of a receptive field may be inputted to an OFF event winner takes all circuit 1620.
[0107] Referring to FIG. 1, the third motion determination unit 145 may calculate a rate of an OFF event to an ON event in response to a type of movement being a small movement and may compare the rate to at least one predetermined reference value to determine a forward movement or a backward movement of an object.
[0108] The third motion determination unit 145 may determine the type of movement to be a forward movement in response to the ON event occurrence frequency/OFF event occurrence frequency being greater than a first reference value, and may determine the type of movement to be a backward movement in response to the ON event occurrence frequency/OFF event occurrence frequency being less than a second reference value in a state of increasing light. The first reference value is greater than "1", and the second reference value is less than "1".
[0109] The third motion determination unit 145 may determine the type of movement to be a backward movement in response to the ON event occurrence frequency/OFF event occurrence frequency being greater than the first reference value, and may determine the type of movement to be a forward movement in response to the ON event occurrence frequency/OFF event occurrence frequency being less than the second reference value in a state of diminishing light. The first reference value is greater than "1", and the second reference value is less than "1".
[0110] The first motion determination unit 130 and the third motion determination unit 145 may both use a small movement, and one of the first motion determination unit 130 and the third motion determination unit 145 may operate based on a user setting.
[0111] The motion control unit 150 outputs a control instruction for controlling a device (not shown) to the device based on a motion pattern determined by the first motion determination unit 130 or a movement direction determined by the second motion determination unit 140. The device may be disposed at a location at which wired/wireless communication with the apparatus 100 for recognizing the motion is available, or the apparatus 100 for recognizing the motion may be disposed inside the device.
[0112] Hereinafter, a method of recognizing motion using an event-based vision sensor according to the aforementioned configuration will be described with reference to the drawings.
[0113] FIG. 17 is a flowchart illustrating an operation of an apparatus 100 for recognizing motion according to an exemplary embodiment.
[0114] Referring to FIG. 17, in operation 1710, the apparatus 100 for recognizing the motion may output an event which corresponds to a movement-occurring part from a vision sensor.
[0115] In operation 1712, the apparatus 100 for recognizing the motion may verify an occurrence frequency of an event.
[0116] In operation 1714, the apparatus 100 for recognizing the motion compares the occurrence frequency of the event to a predetermined threshold value, and verify whether a type of movement is a small movement, such as, a finger movement. A verification method includes determining the type of movement to be a finger movement in response to the occurrence frequency of the event being less than the predetermined threshold value, and determining the type of movement to be a whole hand movement in response to the occurrence frequency being greater than the predetermined threshold value.
[0117] In response to a result of operation 1714 is verified to be the finger movement, the apparatus 100 for recognizing the motion tracks a position of the finger, and calculates a movement trajectory of the finger, in operation 1716.
[0118] In operation 1718, the apparatus 100 for recognizing the motion determines a motion pattern of the finger of a movement-occurring part from the movement trajectory of the finger.
[0119] In response to a result of operation 1714 being verified to be the whole hand movement, the apparatus 100 for recognizing the motion tracks a movement direction of the hand, in operation 1720.
[0120] Subsequent to operation 1718 or operation 1720, the apparatus 100 for recognizing the motion outputs a control instruction to control a device based on the determined motion pattern or the determined movement direction in operation 1722.
[0121] FIG. 18 is a flowchart illustrating an operation of recognizing motion of a finger according to an exemplary embodiment.
[0122] Referring to FIG. 18, in operation 1810, the apparatus 100 for recognizing the motion may receive an event from the vision sensor 110. In operation 1812, the apparatus 100 for recognizing the motion may calculate a spatiotemporal correlation of spatiotemporal correlators based on the received event.
[0123] In operation 1814, the apparatus 100 for recognizing the motion may verify a presence of a spatiotemporal correlator of which an output is "1".
[0124] When the spatiotemporal correlator of which the output is "1" is verified to be present as a result of operation 1814, the apparatus 100 for recognizing the motion may generate a cluster by grouping spatiotemporal correlators of which an output is "1" in operation 1816.
[0125] In operation 1818, the apparatus 100 for recognizing the motion may set a clustering area from a location of a cluster. The clustering area may correspond to a predetermined area including the cluster, and may be a predetermined area expanding from a center.
[0126] In operation 1820, the apparatus 100 for recognizing the motion may calculate the center of the clustering area based on internal state values of spatiotemporal correlators within the clustering area, and calculate the center of the clustering area to be a location of an object.
[0127] In response to the spatiotemporal correlator of which the output is "1" being verified to be absent as a result of operation 1814, the apparatus 100 for recognizing the motion may re-calculate the center of the clustering area based on the internal state values of the spatiotemporal correlators within the previous clustering area, and calculate the re-calculated center of the clustering area to be the location of the object in operation 1820. As used herein, the previous clustering area may refer to a most recently set clustering area.
[0128] Subsequent to operation 1820 or operation 1822, the apparatus 100 for recognizing the motion may track a movement trajectory in conjunction with the previously calculated location of the object in operation 1824.
[0129] In operation 1826, the apparatus 100 for recognizing the motion may extract feature components for representing a motion pattern from the tracked movement trajectory. The feature components may include various parameters, for example, a location of a cluster, a movement direction of a cluster, and a movement angle of a cluster.
[0130] In operation 1828, the apparatus 100 for recognizing the motion may compare feature components to stored motion patterns, determine a motion pattern of a movement-occurring part, and output a control instruction for controlling a device based on the determined motion pattern.
[0131] FIG. 19 is a flowchart which illustrates an operation of detecting a movement direction of a hand according to an exemplary embodiment.
[0132] Referring to FIG. 19, in operation 1910, the apparatus 100 for recognizing the motion may receive an event from the vision sensor 110. In operation 1912, the apparatus 100 for recognizing the motion may detect a movement direction of an object for a plurality of receptive fields based on events outputted from the vision sensor 110. The receptive fields may be provided in a circular form in order to avoid predetermined directivity based on the form of the receptive fields.
[0133] In operation 1914, the apparatus 100 for recognizing the motion may detect a final movement direction, for example, a movement direction of an object based on the vision sensor 110, based on direction information detected for the plurality of receptive fields.
[0134] Exemplary embodiments include computer-readable storage media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, tables, and the like. The media and program instructions may be those specially designed and constructed for the purposes of the exemplary embodiments, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM discs; magneto-optical media such as floptical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and random access memory (RAM). Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by a computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described exemplary embodiments, or vice versa.
[0135] Although a few exemplary embodiments have been shown and described, the exemplary embodiments are not limited to the described embodiments. Instead, it would be appreciated by those skilled in the art that changes may be made to these exemplary embodiments without departing from the principles of the disclosure, the scope of which is defined by the claims and their equivalents.
a vision sensor (110) comprising a plurality of light receiving elements configured to sense changes in light intensity in response to a hand or a portion of the hand moving, and output event signals corresponding to the changes in light intensity;
a movement type determiner (120) configured to distinguish a small movement of a portion of the hand and a large movement of the hand using an occurrence frequency of the event signals output through the vision sensor (110), by calculating the occurrence frequency of the event signals output through the vision sensor (110) and comparing the occurrence frequency to a predetermined threshold value to determine the occurrence frequency to be a small movement in which a portion of the hand moves in response to the occurrence frequency being less than the threshold value, and determine the occurrence frequency to be a large movement in which the hand moves in response to the occurrence frequency being greater than or equal to the threshold value;
a first motion determiner (130) configured to track a movement trajectory of the portion of the hand and determine a motion pattern based on the tracked movement trajectory in response to a result of the movement type determiner indicating the small movement;
a second motion determiner (140) configured to determine a direction of movement in which the hand moves based on the output event signals in response to a result of the movement type determiner indicating the large movement; and
a motion controller (150) configured to output a control instruction to control a device based on the motion pattern determined by the first motion determiner, in case of a small movement, or the movement direction determined by the second motion determiner, in case of a large movement.
spatiotemporal correlators (312, 314, 316), which correspond to a plurality of predetermined receptive fields of the vision sensor (110), receive inputs of event signals output from the predetermined receptive fields and calculate a spatiotemporal correlation for each of the predetermined receptive fields;
a movement trajectory tracker (320) configured to track a movement trajectory of the portion of the hand based on a fluctuation in a spatiotemporal correlation of the predetermined receptive fields; and
a motion pattern determiner (330) configured to determine a motion pattern of the portion of the hand from the tracked movement trajectory.
a first direction detection filter (810) configured to receive an input of event signals outputted from a plurality of predetermined receptive fields of the vision sensor (110) and detect a movement direction in which the hand moves for the plurality of predetermined receptive fields; and
a second direction detection filter (820) configured to determine a final movement direction of the hand based on the movement direction detected for the plurality of receptive fields.
at least one winner takes all circuit (812) which corresponds to each of the predetermined receptive fields,
wherein the winner takes all circuit is configured to output the movement direction of the hand from a corresponding receptive field, using a number of neurons which correspond to a number of directions determined by the second motion determiner (140).
a winner takes all circuit (822) configured to determine the final movement direction of the hand based on the movement direction detected for the plurality of receptive fields, wherein the winner takes all circuit (822) is configured to output the final movement direction of the hand based on the movement direction detected for the plurality of predetermined receptive fields using a number of neurons which correspond to the number of directions determined by the second motion determiner (140); and/or
wherein the second direction detection filter (820) is configured to determine the final direction of movement of the hand by calculating a vector sum of the movement direction detected for the plurality of predetermined receptive fields.
receiving an input of a plurality of event signals which correspond to a movement of a hand or a portion of the hand from an event-based vision sensor;
determining a type of movement based on a frequency of occurrence of event signals by calculating an occurrence frequency of the event signals output through the vision sensor and comparing the occurrence frequency to a predetermined threshold value to determine the occurrence frequency to be a small movement in which a portion of the hand moves in response to the occurrence frequency being less than the threshold value, and determine the occurrence frequency to be a large movement in which the hand moves in response to the occurrence frequency being greater than or equal to the threshold value;
tracking a movement trajectory of the portion of the hand and determining a motion pattern based on the movement trajectory in response to a result of determining the type of movement indicating the small movement;
determining a movement direction in which the hand moves based on the event signals received from the vision sensor in response to a result of the determining the type of movement indicating the large movement; and
controlling a device based on the determined motion pattern or the determined direction of movement.
receiving an input of event signals output from a plurality of predetermined receptive fields of the vision sensor, and calculating a spatiotemporal correlation of spatiotemporal correlators which correspond to each of the predetermined receptive fields;
tracking a movement trajectory of the portion of the hand based on a fluctuation in a spatiotemporal correlation for each of the predetermined receptive fields; and
determining the motion pattern of the portion of the hand from the tracked movement trajectory.
detecting the movement direction in which the hand moves for a plurality of predetermined receptive fields based on the event signals output from the predetermined receptive fields of the vision sensor; and
determining a final movement direction of the hand based on the movement direction detected for the plurality of predetermined receptive fields.
einen Bildverarbeitungssensor (110), der eine Vielzahl von Lichtempfangselementen umfasst, die dazu konfiguriert sind, Änderungen der Lichtintensität als Reaktion auf die Bewegung einer Hand oder eines Teils einer Hand zu erfassen und Ereignissignale auszugeben, die den Änderungen der Lichtintensität entsprechen;
eine Bewegungsartbestimmungseinrichtung (120), die dazu konfiguriert ist, eine kleine Bewegung eines Teils der Hand und eine große Bewegung der Hand unter Verwendung einer Auftrittshäufigkeit der durch den Bildsensor (110) ausgegebenen Ereignissignale zu unterscheiden, und zwar durch Berechnen der Auftrittshäufigkeit der durch den Bildverarbeitungssensor (110) ausgegebenen Ereignissignale und Vergleichen der Auftrittshäufigkeit mit einem vorbestimmten Schwellenwert, so dass die Auftrittshäufigkeit als kleine Bewegung bestimmt wird, bei der sich ein Teil der Hand bewegt, als Reaktion darauf, dass die Auftrittshäufigkeit kleiner als der Schwellenwert ist, und die Auftrittshäufigkeit als große Bewegung bestimmt wird, bei der sich die Hand bewegt, als Reaktion darauf, dass die Auftrittshäufigkeit größer oder gleich dem Schwellenwert ist, und
eine erste Bewegungsbestimmungseinrichtung (130), die dazu konfiguriert ist, eine Bewegungsbahn des Teils der Hand zu verfolgen und als Reaktion darauf, dass ein Ergebnis der Bewegungsbestimmungseinrichtung die kleine Bewegung anzeigt, basierend auf der verfolgten Bewegungsbahn ein Bewegungsmuster zu bestimmen;
eine zweite Bewegungsbestimmungseinrichtung (140), die dazu konfiguriert ist, basierend auf den ausgegebenen Ereignissignalen eine Bewegungsrichtung zu verfolgen, in der die Hand sich bewegt, als Reaktion darauf, dass ein Ergebnis der Bewegungsartbestimmungseinrichtung die große Bewegung anzeigt; und
eine Bewegungssteuerung (150), die dazu konfiguriert ist, eine Steueranweisung auszugeben zum Steuern eines Geräts basierend auf dem durch die erste Bewegungsbestimmungseinrichtung bestimmten Bewegungsmuster im Falle einer kleinen Bewegung oder basierend auf der von der zweiten Bewegungsbestimmungseinrichtung bestimmten Bewegungsrichtung im Falle einer großen Bewegung.
räumlich-zeitliche Korrelatoren (312, 314, 316), die einer Vielzahl von vorbestimmten Empfangsfeldern des Bildverarbeitungssensors (110) entsprechen, Eingaben von Ereignissignalen empfangen, welche von den vorbestimmten Empfangsfeldern ausgegeben werden, und eine räumlich-zeitliche Korrelation für jedes der vorbestimmten Empfangsfelder berechnen;
eine Bewegungsbahn-Verfolgungseinrichtung (320), die dazu konfiguriert ist, eine Bewegungsbahn des Teils der Hand basierend auf einer Fluktuation einer räumlich-zeitlichen Korrelation der vorbestimmten Empfangsfelder zu verfolgen; und
eine Bewegungsmusterbestimmungseinrichtung (330), die dazu konfiguriert ist, aus der verfolgten Bewegungsbahn ein Bewegungsmuster des Teils der Hand zu bestimmen.
einen ersten Richtungserfassungsfilter (810), der dazu konfiguriert ist, eine Eingabe von Ereignissignalen zu empfangen, die aus einer Vielzahl von vorbestimmten Empfangsfeldern des Bildverarbeitungssensors (110) ausgegeben werden, und eine Bewegungsrichtung der Hand zu erfassen, und zwar für die Vielzahl von vorbestimmten Empfangsfeldern; und
einen zweiten Richtungserfassungsfilter (820), der dazu konfiguriert ist, eine endgültige Bewegungsrichtung der Hand basierend auf der Bewegungsrichtung zu bestimmen, die für die Vielzahl von Empfangsfeldern erfasst wurde.
mindestens eine "Winner Takes All"-Schaltung (812), die jedem der vorbestimmten Empfangsfelder entspricht, und
wobei die "Winner Takes All"-Schaltung dazu konfiguriert ist, die Bewegungsrichtung der Hand von einem entsprechenden Empfangsfeld auszugeben, und zwar unter Verwendung einer Anzahl von Neuronen, die einer Anzahl von Richtungen entsprechen, die durch die zweite Bewegungsbestimmungseinrichtung (140) bestimmt werden.
eine "Winner Takes All"-Schaltung (822), die dazu konfiguriert ist, die endgültige Bewegungsrichtung der Hand basierend auf der Bewegungsrichtung zu bestimmen, die für die Vielzahl von Empfangsfeldern erfasst wurde, wobei die "Winner Takes All"-Schaltung (822) dazu konfiguriert ist, die endgültige Bewegungsrichtung der Hand basierend auf der Bewegungsrichtung zu bestimmen, die für die Vielzahl von Empfangsfeldern erfasst wurde, und zwar unter Verwendung einer Anzahl von Neuronen, die der Anzahl von Richtungen entsprechen, die durch die zweite Bewegungsbestimmungseinrichtung (140) bestimmt werden, und/oder
wobei der zweite Richtungserfassungsfilter (820), dazu konfiguriert ist, die endgültige Bewegungsrichtung der Hand durch Berechnen einer Vektorsumme der Bewegungsrichtung zu bestimmen, die für die Vielzahl von vorbestimmten Empfangsfeldern erfasst wurde.
Empfangen einer Eingabe einer Vielzahl von Ereignissignalen, die einer Bewegung einer Hand oder eines Teils der Hand entsprechen, aus einem ereignisbasierten Bildverarbeitungssensor;
Bestimmen einer Bewegungsart basierend auf einer Häufigkeit des Auftretens von Ereignissignalen durch Berechnen einer Auftrittshäufigkeit der durch den Bildverarbeitungssensor ausgegebenen Ereignissignale und Vergleichen der Auftrittshäufigkeit mit einem vorbestimmten Schwellenwert, so dass die Auftrittshäufigkeit als kleine Bewegung bestimmt wird, bei der sich ein Teil der Hand bewegt, als Reaktion darauf, dass die Auftrittshäufigkeit kleiner als der Schwellenwert ist, und die Auftrittshäufigkeit als große Bewegung bestimmt wird, bei der sich die Hand bewegt, als Reaktion darauf, dass die Auftrittshäufigkeit größer oder gleich dem Schwellenwert ist, und
Verfolgen einer Bewegungsbahn des Teils der Hand und Bestimmen eines Bewegungsmusters basierend auf der Bewegungsbahn als Reaktion darauf, dass ein Ergebnis der Bewegungsartbestimmung die kleine Bewegung anzeigt;
Bestimmen einer Bewegungsrichtung der Hand basierend auf den aus dem Bildverarbeitungssensor empfangenen Ereignissignalen als Reaktion darauf, dass ein Ergebnis des Bewegungsartbestimmung die große Bewegung anzeigt; und
Steuern eines Geräts basierend auf dem bestimmten Bewegungsmuster oder der bestimmten Bewegungsrichtung.
Empfangen einer Eingabe von Ereignissignalen, die von einer Vielzahl von vorbestimmten Empfangsfeldern des Bildverarbeitungssensors ausgegeben werden, und Berechnen einer räumlich-zeitlichen Korrelation von räumlich-zeitlichen Korrelatoren, die jedem der vorbestimmten Empfangsfelder entsprechen;
Verfolgen einer Bewegungsbahn des Teils der Hand basierend auf einer Fluktuation einer räumlich-zeitlichen Korrelation für jedes der vorbestimmten Empfangsfelder; und
Bestimmen eines Bewegungsmusters des Teils der Hand aus der verfolgten Bewegungsbahn.
Erfassen der Bewegungsrichtung der Hand für eine Vielzahl von vorbestimmten Empfangsfeldern basierend auf den Ereignissignalen, die von den vorbestimmten Empfangsfeldern des Bildverarbeitungssensors ausgegeben werden; und
Bestimmen einer endgültigen Bewegungsrichtung der Hand basierend auf der Bewegungsrichtung, die für die Vielzahl von vorbestimmten Empfangsfeldern erfasst wurde.
un capteur de vision (110) comprenant une pluralité d'éléments récepteurs de lumière conçus pour détecter des changements d'intensité lumineuse en réaction au mouvement d'une main ou d'une partie de la main, et pour émettre des signaux d'événement correspondant aux changements d'intensité lumineuse ;
un organe de détermination de type de mouvement (120) conçu pour différencier un petit mouvement d'une partie de la main d'un grand mouvement de la main à l'aide de la fréquence de survenue des signaux d'événement émis par le capteur de vision (110), grâce au calcul de la fréquence de survenue des signaux d'événement émis par le capteur de vision (110) et à la comparaison de la fréquence de survenue à une valeur seuil prédéterminée afin de déterminer, en réaction au fait que la fréquence de survenue est inférieure à la valeur seuil, que la fréquence de survenue est un petit mouvement selon lequel une partie de la main a bougé, et pour déterminer, en réaction au fait que la fréquence de survenue est supérieure ou égale à la valeur seuil, que la fréquence de survenue est un grand mouvement selon lequel la main a bougé ;
un premier organe de détermination de mouvement (130) conçu pour suivre une trajectoire de mouvement de la partie de la main et pour déterminer une configuration de mouvement compte tenu de la trajectoire de mouvement suivie, en réaction au fait que le résultat de la détermination du type de mouvement indique un petit mouvement ;
un deuxième organe de détermination de mouvement (140) conçu pour déterminer la direction du mouvement de la main compte tenu des signaux d'événement émis, en réaction au fait que le résultat de la détermination du type de mouvement indique un grand mouvement ; et
un organe de commande de mouvement (150) conçu pour émettre une instruction de commande permettant de commander un appareil en fonction de la configuration de mouvement déterminée par le premier organe de détermination de mouvement, dans le cas d'un petit mouvement, ou en fonction de la direction du mouvement déterminée par le deuxième organe de détermination de mouvement, dans le cas d'un grand mouvement.
des corrélateurs spatio-temporels (312, 314, 316) qui correspondent à une pluralité de champs réceptifs prédéterminés du capteur de vision (110), reçoivent en entrée des signaux d'événement émis par les champs réceptifs prédéterminés et calculent une corrélation spatio-temporelle pour chacun des champs réceptifs prédéterminés,
un organe de suivi de trajectoire de mouvement (320) conçu pour suivre une trajectoire de mouvement de la partie de la main compte tenu d'une fluctuation de corrélation spatio-temporelle des champs réceptifs prédéterminés, et
un organe de détermination de configuration de mouvement (330) conçu pour déterminer une configuration de mouvement de la partie de la main à partir de la trajectoire de mouvement suivie.
un premier filtre de détection de direction (810) conçu pour recevoir en entrée des signaux d'événement émis par une pluralité de champs réceptifs prédéterminés du capteur de vision (110) et pour détecter la direction du mouvement de la main relativement à la pluralité de champs réceptifs prédéterminés, et
un deuxième filtre de détection de direction (820) conçu pour déterminer une direction finale du mouvement de la main compte tenu de la direction du mouvement détectée relativement à la pluralité de champs réceptifs.
au moins un circuit « winner takes all » (812) qui correspond à chacun des champs réceptifs prédéterminés,
ledit circuit « winner takes all » étant conçu pour communiquer la direction du mouvement de la main provenant d'un champ réceptif correspondant, à l'aide d'un certain nombre de neurones correspondant au nombre de directions déterminées par le deuxième organe de détermination de mouvement (140).
un circuit « winner takes all » (822) conçu pour déterminer la direction finale du mouvement de la main compte tenu de la direction du mouvement détectée relativement à la pluralité de champs réceptifs,
ledit circuit « winner takes all » (822) étant conçu pour communiquer la direction finale du mouvement de la main compte tenu de la direction du mouvement détectée relativement à la pluralité de champs réceptifs prédéterminés, à l'aide d'un certain nombre de neurones correspondant au nombre de directions déterminées par le deuxième organe de détermination de mouvement (140) ; et/ou
dans lequel le deuxième filtre de détection de direction (820) est conçu pour déterminer la direction finale du mouvement de la main en calculant une somme vectorielle de la direction du mouvement détectée relativement à la pluralité de champs réceptifs prédéterminés.
la réception en entrée d'une pluralité de signaux d'événement correspondant au mouvement d'une main ou d'une partie de la main, en provenance d'un capteur de vision basé sur un événement ;
la détermination du type de mouvement compte tenu de la fréquence de survenue de signaux d'événement grâce au calcul de la fréquence de survenue des signaux d'événement émis par le capteur de vision et à la comparaison de la fréquence de survenue à une valeur seuil prédéterminée afin de déterminer, en réaction au fait que la fréquence de survenue est inférieure à la valeur seuil, que la fréquence de survenue est un petit mouvement selon lequel une partie de la main a bougé, et pour déterminer, en réaction au fait que la fréquence de survenue est supérieure ou égale à la valeur seuil, que la fréquence de survenue est un grand mouvement selon lequel la main a bougé ;
le suivi d'une trajectoire de mouvement de la partie de la main et la détermination d'une configuration de mouvement compte tenu de la trajectoire de mouvement en réaction au fait que le résultat de la détermination du type de mouvement indique un petit mouvement ;
la détermination de la direction du mouvement de la main compte tenu des signaux d'événement reçus en provenance du capteur de vision, en réaction au fait que le résultat de la détermination du type de mouvement indique un grand mouvement ; et
la commande d'un appareil en fonction de la configuration de mouvement déterminée ou de la direction de mouvement déterminée.
la réception en entrée de signaux d'événement émis par une pluralité de champs réceptifs prédéterminés du capteur de vision, et le calcul d'une corrélation spatio-temporelle de corrélateurs spatio-temporels correspondant à chacun des champs réceptifs prédéterminés,
le suivi d'une trajectoire de mouvement de la partie de la main compte tenu d'une fluctuation de corrélation spatio-temporelle relativement à chacun des champs réceptifs prédéterminés, et
la détermination de la configuration de mouvement de la partie de la main à partir de la trajectoire de mouvement suivie.
la détection de la direction du mouvement selon laquelle la main bouge relativement à une pluralité de champs réceptifs prédéterminés, compte tenu des signaux d'événement émis par les champs réceptifs prédéterminés du capteur de vision, et
la détermination de la direction finale du mouvement de la main compte tenu de la direction du mouvement détectée relativement à la pluralité de champs réceptifs prédéterminés.
REFERENCES CITED IN THE DESCRIPTION
This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.
Patent documents cited in the description
Non-patent literature cited in the description
- JUN HAENG LEE et al.Dynamic vision sensor camera based bare hand gesture recognition, [0003]