|
(11) | EP 3 282 421 B1 |
| (12) | EUROPEAN PATENT SPECIFICATION |
|
|
| (54) |
SYSTEMS AND METHODS FOR IDENTIFYING BODY JOINT LOCATIONS BASED ON SENSOR DATA ANALYSIS SYSTEME UND VERFAHREN ZUR IDENTIFIZIERUNG VON KÖRPERGELENKSPOSITIONEN AUF BASIS VON DATENANALYSE SYSTÈMES ET PROCÉDÉS POUR IDENTIFIER DES EMPLACEMENTS D'ARTICULATIONS DU CORPS SUR LA BASE D'UNE ANALYSE DE DONNÉES DE CAPTEUR |
|
|
|||||||||||||||||||||||||||||||
| Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention). |
CROSS-REFERENCE TO RELATED APPLICATIONS AND PRIORITY
[0001] This patent application claims priority to India Patent Application 201621027388, filed on August 10, 2016.
TECHNICAL FIELD
[0002] The disclosure herein generally relate to sensor data analysis, and, more particularly, to systems and methods for identifying body joint locations based on sensor data analysis.
BACKGROUND
[0003] Joint motion analysis is an important aspect of health monitoring / rehabilitation of patients suffering from neurological disorders, post stroke patients, and elderly subjects. Joint movement data capture is traditionally done using clinical devices such as Vicon, which is marker-based and extremely expensive and ill-suited for prolonged rehabilitation therapy. Kinect® is a marker-less motion capture device that is being used for its low cost, portability and ease of cost when compared to its expensive counterpart. The accuracy of Kinect® as an alternative to Marker-based systems has been widely studied and there are extensive reports on its validity and reliability in comparison to Vicon. However, the reports indicate a lower accuracy of the human body joint information provided by Kinect® skeletal data. The skeleton data provided by Kinect® requires further optimization to improve its accuracy and allow effective and reliable assessment.
[0004] Ho, E. et al, "Improving posture classification accuracy for depth sensor-based human activity monitoring in smart environments", Computer Vision and Image Understanding, Vol. 148, pp. 97-110, 2016 discloses the improvement of the position of Kinect joint locations to discard wrongly tracked joints for posture analysis in e.g. rehabilitation monitoring.
[0005] From Metcalf, C. et al, "Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation", IEEE Trans. on Biomedical Engineering, Vol. 60, No. 8, pp. 2184-2192, 2013) is known the identification of had grip modes from (markerless) Kinect RGB-D data based on joint angles of a finger and the length of phalangeal segments.
SUMMARY
[0006] Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in an aspect, a hardware processor implemented method according to claim 1 for identifying body joint locations is provided. The method comprises obtaining, by one or more hardware processors, input data comprising skeletal data, depth data, and red, green, and blue (RGB) data pertaining to at least one user at one or more time stamps, wherein the one or more input data comprises three dimensional (3D) coordinates pertaining to one or more skeleton joints. The method further comprises estimating, using the one or more input data, by the one or more hardware processors, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain (i) an estimate of one or more body joint locations and (ii) an estimate of one or more body segment lengths. In an embodiment, the one or more body segment lengths are based on a Euclidean distance between adjacent skeleton joints. One or more probable correct body joint locations are identified in a bounded neighborhood around the one or more body joint locations that are previously obtained in a previous frame, wherein the one or more probable correct body joint locations are iteratively identified based on the depth data and the RGB data. In an example embodiment, a search region is defined for identifying the one or more probable correct body joint locations using a radius equivalent to a displacement of a corresponding skeleton joint in a single frame. The method further comprises performing, by the one or more hardware processors, a comparison of (i) a body segment length associated with each of the one or more probable correct body joint locations and (ii) a reference length. In an embodiment, the reference length is derived based on the estimate of the one or more body segment lengths. A subset of the one or more probable correct body joint locations are identified as one or more candidate body joint locations based on the comparison. In an embodiment, the subset of the one or more probable correct body joint locations are identified as one or more candidate body joint locations based on the comparison resulting in a minimal body segment length variation. A physical orientation of each body segment pertaining to each of the one or more candidate body joint locations is determined by segmenting one or more 3D coordinates of each body segment based on the depth data and performing an analysis on each segmented 3D coordinate. A (corrected) body joint location is identified from the one or more candidate body joint locations based on a minimal deviation in direction from the physical orientation of a corresponding body segment along with a feature descriptor of the RGB data and the depth data, wherein the minimal deviation is based on the depth data, and, wherein the minimal deviation is based on one or more actions performed by the user.
[0007] In another aspect, a system according to claim 6 for identifying a body joint location is provided. The system comprises a memory storing instructions, one or more communication interfaces, and one or more hardware processors coupled to the memory using the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to: obtain one or more input data comprising skeletal data, depth data and red, green, and blue (RGB) data pertaining to at least one user at one or more time stamps, wherein the one or more input data comprising three dimensional (3D) coordinates pertaining to one or more skeleton joints. The skeletal data, depth data and red, green, and blue (RGB) data are captured by a sensor (e.g., Kinect® sensor). The one or more hardware processors are further configured by the instructions to estimate, using the one or more input data, one or more body joint locations and a body segment length of one or more body segments connecting to corresponding skeleton joints to obtain (i) an estimate of one or more body joint locations and (ii) an estimate of one or more body segment lengths. In an embodiment, one or more body segment lengths are based on a Euclidean distance between adjacent skeleton joints. In an embodiment, the system is further configured to identify one or more probable correct body joint locations in a bounded neighborhood around the one or more body joint locations that are previously obtained from a previous frame, wherein the one or more probable correct body joint locations are iteratively identified based on the depth data and the RGB data. In an embodiment, a search region is defined to identify the one or more probable correct body joint locations based on a radius equivalent to a displacement of a corresponding skeleton joint in a single frame.
[0008] The system further performs a comparison of (i) a body segment length associated with each of the one or more probable correct body joint locations and (ii) a reference length. In an embodiment, the reference length is derived based on the estimate of the one or more body segment lengths. The system further identifies a subset of the one or more probable correct body joint locations as one or more candidate body joint locations based on the comparison. In an embodiment, the subset of the one or more probable correct body joint locations are identified as one or more candidate body joint locations based on the comparison resulting in a minimal body segment length variation. In other words, probable correct body joint locations having a minimal body segment length variation with respect to the reference length are identified as one or more candidate body joint locations.
[0009] The system is further configured to determine a physical orientation of each body segment pertaining to each of the one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on the depth data and performing an analysis on each segmented 3D coordinate. The system is further configured to identify, from the one or more candidate body joint locations, a (corrected) body joint location based on a minimal deviation in direction from the physical orientation of a corresponding body segment along with a feature descriptor of the RGB data and the depth data, wherein the minimal deviation is based on the depth data, and wherein the minimal deviation is based on one or more actions performed by the user. In an embodiment, a body joint location having a minimal deviation in direction from the physical orientation of a corresponding body segment is identified as a corrected joint body location along with a feature descriptor of the RGB data and the depth data.
[0010] In yet another aspect, one or more non-transitory machine readable information storage mediums according to claim 11 comprising one or more instructions is provided. The one or more instructions which when executed by one or more hardware processors causes obtaining input data comprising skeletal data, depth data and red, green, and blue (RGB) data pertaining to at least one user at one or more time stamps, wherein the input data comprises three dimensional (3D) coordinates pertaining to one or more skeleton joints. The one or more instructions which when executed by one or more hardware processors further causes estimating, using the one or more input data, one or more body joint locations and a body segment length of one or more body segments connecting to corresponding skeleton joints to obtain (i) an estimate of one or more body joint locations and (ii) an estimate of one or more body segment lengths. In an embodiment, the one or more body segment lengths are based on a Euclidean distance between adjacent skeleton joints. The one or more instructions which when executed by one or more hardware processors causes identifying one or more probable correct body joint locations in a bounded neighborhood around the one or more body joint locations that are previously obtained, wherein the one or more probable correct body joint locations are iteratively identified based on the depth data and the RGB data. In an example embodiment, a search region is defined for identifying the one or more probable correct body joint locations using a radius equivalent to a displacement of a corresponding skeleton joint. The one or more instructions which when executed by one or more hardware processors further causes performing a comparison of (i) a body segment length associated with each of the one or more probable correct body joint locations and (ii) a reference length. In an embodiment, the reference length is derived based on the estimate of the one or more estimated body segment lengths. The one or more instructions which when executed by one or more hardware processors further causes identifying a subset of the one or more probable correct body joint locations as one or more candidate body joint locations based on the comparison. In an embodiment, identifying a subset of the one or more probable correct body joint locations as one or more candidate body joint locations is based on the comparison resulting in a minimal body segment length variation. The one or more instructions which when executed by one or more hardware processors further causes determining a physical orientation of each body segment pertaining to each of the one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on the depth data and performing an analysis on each segmented 3D coordinate. The one or more instructions which when executed by one or more hardware processors further causes identifying, from the one or more candidate body joint locations, a (corrected) body joint location based on a minimal deviation in direction from the physical orientation of a corresponding body segment along with a feature descriptor of the RGB data and the depth data, wherein said minimal deviation is based on the depth data, and, wherein the minimal deviation is based on one or more actions performed by the user.
[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles:
FIG. 1 illustrates an exemplary block diagram of a system for identifying a body joint location of a user in accordance with an embodiment of the present disclosure.
FIG. 2 illustrates an exemplary flow diagram of a method for identifying a body joint location of a user using the system of FIG. 1 in accordance with an embodiment of the present disclosure.
FIG. 3 illustrates an exemplary table indicating body segment length statistics for ROM exercises for 10 subjects, according to an embodiment of the present disclosure.
FIG. 4 illustrates a graphical representation of variation in length of forearm joints in accordance with an embodiment of the present disclosure.
FIG. 5 illustrates a graphical representation of variation in length of arm in accordance with an embodiment of the present disclosure.
FIG. 6 illustrates a graphical representation of variation in forearm length for elbow flexion in accordance with an embodiment of the present disclosure.
FIG. 7 depicts Body segment orientation correction by the system of FIG. 1 in accordance with an embodiment of the present disclosure.
DETAILED DESCRIPTION OF EMBODIMENTS
[0013] Exemplary embodiments are described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with the true scope and spirit being indicated by the following claims.
[0014] Body joint movement analysis is extremely essential for health monitoring and treatment of patients with neurological disorders and stroke. Chronic hemiparesis of the upper extremity following a stroke causes major hand movement limitations. There is possibility of permanent reduction in muscle co-activation and corresponding joint torque patterns due to stroke. Several studies suggest that abnormal coupling of shoulder abductors with elbow flexors, and shoulder adductors with elbow extensors often leads to some stereotypical movement characteristics exhibited by severe stroke patients. Therefore continuous and effective rehabilitation therapy is absolutely essential to monitor and control such abnormalities. There is a substantial need for home-based rehabilitation post-clinical therapy.
[0015] Marker-based systems for human body motion capture and analysis, (such as Vicon) are popular in upper extremity rehabilitation for their clinically approved levels of accuracy. But marker less systems for motion capture, such as Microsoft Kinect®, are feasible for home-based rehabilitation due to their affordability and portability. The accuracy, validity and test-retest reliability measures of Kinect have been studied for range of motion, postural control and gait. The results reported indicate a disparity between body joint locations observed by Kinect and that obtained by a clinical gold standard stereophotogrammetry system such as Vicon. Body segment length variations of the order of 6-8 centimeters (cms) for arm, and 2-5 cms for forearm were reported while using Kinect® as it provides a non-anthropometric skeleton model. Moreover the error was found to be lower for upper body than lower body joints. The performance of Kinect® has also been studied for non-healthy subjects and elderly population Experimental results showed that the accuracy of Kinect® for measuring gross spatial movement, such as shoulder and elbow movement, was higher than that for fine movements (such as hands). The disparity in body semantics such as body segment length and orientation increases during movement. It is therefore essential to improve accuracy of Kinect® in measurements related to clinical assessment and biomechanical modeling. For example, accurate measurement of arm length is important for assessing performance of reaching task for patients with impairments in the paretic arm. Prior work has been done on categorizing patients from movement pattern using only bone length derived from Kinect® skeleton data. Some had explicitly mentioned the need of further optimization of joint positions and body segment length using additional depth information from Kinect®.
[0016] The embodiments of the present disclosure aims at improving accuracy of Kinect® in rehabilitation for upper extremities such as shoulder and elbow joints by improving joint center localization using additional Depth and RGB (RGBD) data, with constraints on body segment length and orientation. More particularly, the embodiments of the present disclosure implement systems and methods for sensor data analysis based identification of a body joint location that enables minimizing temporal variation in body segment length based on RGB and depth information obtained from Kinect® sensor, aligns connected joint pairs in the direction of physical body segment orientation using Depth-based segmentation and analysis (e.g., Principal Component Analysis (PCA)) on segmented 3D coordinates, and provides reliable range of motion analysis system based on corrected joint information.
[0017] Referring now to the drawings, and more particularly to FIGS. 1 through 7, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
[0018] FIG. 1 illustrates an exemplary block diagram of a system 100 for identifying a body joint location of a user in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more processors 104, communication interface device(s) or input/output (I/O) interface(s) 106, and one or more data storage devices or memory 102 operatively coupled to the one or more processors 104. The one or more processors 104 that are hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) is configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like.
[0019] The I/O interface device(s) 106 can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.
[0020] The memory 102 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, one or more modules (not shown) of the system 100 can be stored in the memory 102.
[0021] FIG. 2 illustrates an exemplary flow diagram of a method for identifying a body joint location of a user using the system 100 of FIG. 1 in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 comprises one or more data storage devices or the memory 102 operatively coupled to the one or more hardware processors 104 and is configured to store instructions for execution of steps of the method by the one or more processors 104. The steps of the method of the present disclosure will now be explained with reference to the components of the system 100 as depicted in FIG. 1, and the flow diagram. In an embodiment of the present disclosure, at step 202, the one or more processors 104 obtain input data comprising skeletal data, depth data and red, green, and blue (RGB) data pertaining to at least one user. In an example embodiment, the input data may be captured at one or more time stamps. In an example embodiment, the one or more input data comprises three dimensional (3D) coordinates pertaining to one or more skeleton joints. In an example embodiment, the input data comprising skeletal data, depth data and red, green, and blue (RGB) data is captured by a sensor, for example, a Kinect® sensor. In an example embodiment, the skeletal data, depth data and red, green, and blue (RGB) data is received from the Kinect® sensor at a frame rate of approximately 25-30 frames per second The Kinect® sensor may be integrated within the system 100, in one example embodiment. The Kinect® sensor may be an external component that is connected to the system 100 via one or more communication interfaces, in another example embodiment.
[0022] In an embodiment of the present disclosure, at step 204, the one or more processors 104 estimate (or compute) one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain (i) an estimate of one or more estimated body joint locations and (ii) an estimate of one or more estimated body segment lengths. In an embodiment, the one or more body segment lengths are based on a Euclidean distance between adjacent skeleton joints in the skeleton data input. In an example embodiment, the estimate of the body joint locations and the body segment lengths are obtained using the input data.
[0023] In an embodiment of the present disclosure, at step 206, the one or more processors 104 iteratively identify one or more probable correct body joint locations in a bounded neighborhood around the one or more body joint locations that are previously obtained from a previous frame. In one embodiment of the present disclosure, the one or more probable correct body joint locations are iteratively identified based on the depth data and the RGB data. In an example embodiment of the present disclosure, a search region is defined for identifying the one or more probable correct body joint locations using a radius equivalent to a displacement of a corresponding skeleton joint in a single frame. In an embodiment of the present disclosure, at step 208, the one or more processors 104 perform a comparison of (i) a body segment length associated with each of the one or more probable correct body joint locations and (ii) a reference length. In an example embodiment, the reference length is computed or derived based on the estimate of the one or more body segment lengths (e.g., true body segment lengths). In other words, the reference length is computed or derived based on Euclidean distance between body joint locations during system initialization. The initial estimates of the body joint locations (e.g., true body joint locations) and the reference length of body segment are computed by taking the average of input skeleton joint location and Euclidean distance between adjacent input joint locations respectively, after removing outliers, if any.
[0024] In an embodiment of the present disclosure, at step 210, the one or more processors 104 identify at least a subset of the one or more probable correct body joint locations as one or more candidate body joint locations based on the comparison. In an example of the present disclosure, the subset of the one or more probable correct body joint locations are identified as one or more candidate body joint locations based on the comparison resulting in a minimal body segment length variation. In other words, probable correct body joint locations having a minimal body segment length variation with respect to the reference length are identified as the one or more candidate body joint locations.
[0025] In an embodiment of the present disclosure, at step 212, the one or more processors 104 determine a physical orientation of each body segment pertaining to each of the one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on the depth data and performing an analysis on each segmented 3D coordinate. In one embodiment of the present disclosure, the system 100 performs a principle component analysis on each segmented 3D coordinate, based on which the physical orientation is determined. In an embodiment of the present disclosure, at step 214, the one or more processors 104 identify a corrected body joint location from the one or more candidate body joint locations based on a deviation in direction from the physical orientation of a corresponding body segment along with a feature descriptor of the RGB data and the depth data, wherein the minimal deviation is based on the depth data. In an example embodiment of the present disclosure, the system 100 uses the RGB-D based feature descriptor to identify a corrected body joint location having (or that has) a minimal deviation in direction from the physical orientation of a corresponding body segment (as depicted in FIG. 7). In an embodiment, the minimal deviation is based on, or subject to, one or more actions performed by the user.
Experimental setup
[0026] Ten healthy subjects (age: 21-65 years, weight: 45kg - 120kg and height: 1:42m - 1:96m) with no pre-existing symptoms of neurological diseases, major orthopedic lesions or vestibular disorders, were chosen for the experiments. The Kinect® v1 device has been used to capture Depth, RGB and Skeleton data (at 25fps approximately) along with timestamp information. Participants/subjects stood at a distance of 2:1 meters - 2:4 meters from the Kinect® device (Kinect® sensor), which was placed 1 meter above the ground. Each participant performs the following (active) Range of motion (ROM) exercises - shoulder abduction and adduction, shoulder flexion and extension, elbow flexion and extension. In all experiments for an initial 1-2 seconds the subjects were required to stand completely stationary for initialization of the proposed methodology.
[0027] Joint movement analysis using Kinect® sensor is based on spatiotemporal variation of three-dimensional coordinates of 20 skeleton joints given by:
[0028] Where
denotes the 3D coordinates of the jth joint provided by the Kinect® sensor at time instance t (corresponding to frame ft). Let Bi,j represent the body segment connecting joints i and j. The joint coordinates are subject to noise due to room lighting (or any other ambient affecting conditions), infrared (IR) interferences, subject's distance from Kinect® sensor, quantization errors introduced during computations etc. The errors in joint center locations
and
causes variation in length of Bi,j, as well as its orientation in 3D space.
[0029] Hence, in order to reliably measure parameters such as joint range of motion, correct body joint locations are obtained that result in an accurate body segment length and orientation. With (direct) utilization of the depth sensor values as well as RGB information, it is possible to obtain the correct estimate of the joint's location that satisfies body segment length and orientation constraints. The intuition is that the correct location
of the jth joint must lie in the vicinity of the 3-D coordinates
reported by the Kinect® sensor. A hierarchical searching technique is carried out, as described in the steps of FIG. 2, for the body joint location that satisfies - (1) maximum similarity in time-varying characteristics of depth and RGB, represented by a proposed feature descriptor, (2) maximum alignment of Bi,j towards true physical body segment orientation, (which is estimated from depth segmentation and analysis such as principle component analysis), and (3) minimum deviation in ∥Bi,j∥ from reference length estimated during the initialization phase.
[0030] During the initialization phase when the subject(s) is made to remain stationary for 30 - 50 frames, an initial estimate of the body joint locations and the body segment length is computed based on quartile measures of observations. For each joint, candidate locations are searched in the vicinity of the Kinect® skeleton coordinates, which satisfy the constraint of minimal variation in length of body segment. Then the search is further refined by minimizing temporal variations in RGB-D based feature descriptor weighted by the deviation in direction of Bi,j from the physical orientation of the body segment.
Coordinate transformation:
[0031] In order to find depth and RGB value for a particular joint j, the real world coordinates
are projected to two-dimensional depth map coordinates
using Kinect® IR camera intrinsic properties as shown in below expression (1) by way of example. Further an affine transformation is used to find correspondence between depth coordinates
and RGB image coordinates
as shown in below expression (2) by way of example:
Search for body joint location:
[0032] The corrected body joint location
at time t is searched in a bounded neighborhood around
in projected depth and RGB space (coordinate transformation as described above). The search region S for jth joint at the time t is defined using a radius equivalent to the joint's displacement in a single frame.
Body segment length constraint:
[0033] The search is subject to the constraint that body segment length (the Euclidean distance between 3D coordinates of two physically connected joints) should remain invariant during movement of the corresponding joints. The search region S is refined to S1 by selecting candidate locations
that satisfy the length constraint, as shown in below expression by way of example:
[0034] Where
is the corrected 3D location of joint i,
is the Euclidean distance between
and
is the physical length of the body segment joining joints i and j, which is estimated during initialization. In one example embodiment, the search regions S and S1 may be defined by the system 100. In another example embodiment, one or more inputs may be obtained (e.g., from a user) by the system to define the search regions S and S1.
Estimation of body segment orientation:
[0035] At each time instance t, the vector Bi,j is selected so that it exhibits maximum alignment towards the true body segment orientation. It is possible to segment the human body from the background in Kinect® Depth space. A bounded region around the body joint location of joints i and j is used to separate the coordinates of the body segment or limb from the rest of the human (user) form. An analysis (e.g., but not limited to, principle component analysis) is performed over segmented coordinates to get an Eigen vector Ei,j whose direction represents the direction of maximum variation of coordinates representing the body segment. Ei,j provides an estimate of the physical orientation of the body orientation in each instance of time.
Feature descriptor
[0036] The search region (or space) S1 consists of candidate locations among which the joint's actual location lies. In order to select the true body joint location (example, an exact or actual body joint location amongst the one or more candidate body joint locations), a set of features based on RGB and depth characteristics is used to uniquely identify the joint as it makes it's trajectory over time.
[0037] During any ROM exercise, in-spite of variation of depth values for any joint, the relative depth variation between a depth-pixel and its neighbors ought to remain unchanged for any two consecutive frames. The RGB values in a pixel neighborhood demonstrate similar properties. For a joint centre
feature descriptor consisting of elements related to depth differences and RGB values is defined as λ = {λD, λR}. λD is a (2w + 1) × (2w + 1) matrix where w ∈ I+ centered at depth pixel P = (px,py)T and is expressed as
Hereafter the notation λD | Q̃ is used to denote λD centered at any arbitrary location Q.
Where
Dx,y = (depth(px,py) - depth(x,y)) ∗ g(x, y)
depth(x,y) = depth value at coordinates (x,y)
g(x,y) represents a Gaussian window centered at P with variance σ2.
[0039] Finally the corrected location
for joint j is obtained by the following equations:
where α, γ ∈ (0,1) are constants, whose values may be experimentally determined.
[0040] FIG. 3, with respect to FIGS. 1-2, illustrates an exemplary table indicating body segment length statistics for ROM exercises for 10 subjects, according to an embodiment of the present disclosure. More particularly, FIG. 3 depicts correction and/or percentage (%) improvement in body segment length each ROM exercise with respect to mean (in meter), standard deviation (in meter), range (in meter), co-efficient of variation for input data obtained from a Kinect® sensor. As depicted in FIG. 3, for example, for Shoulder ROM exercise (Abduction Adduction), the percentage (%) improvement in body segment length is 73.8 and 74.3 respectively. Similarly, for Shoulder ROM exercise (Flexion and Extension), the percentage (%) improvement in body segment length is 72.9 and 80.7 respectively. Likewise, for Elbow ROM exercise (Flexion and Extension) the percentage (%) improvement in body segment length is 71.6 and 58.7 respectively.
[0041] FIG. 4, with reference to FIGS. 1 through 3, illustrates a graphical representation of variation in length of forearm joints in accordance with an embodiment of the present disclosure. More specifically, FIG. 4 depicts variation in length between elbow left and wrist left joints. FIG. 5, with reference to FIGS. 1 through 4, illustrates a graphical representation of variation in length of arm in accordance with an embodiment of the present disclosure. More specifically, FIG. 5 depicts variation in length between shoulder left and elbow left joints. The performance is evaluated by the system 100 both for stationary and dynamic joints. FIGS. 4 and 5 indicate a significant reduction in temporal variation of arm and forearm length during shoulder abduction, with the proposed methodology.
[0042] FIG. 6, with reference to FIGS. 1 through 5, illustrates a graphical representation of variation in forearm length for elbow flexion in accordance with an embodiment of the present disclosure. FIG. 6 clearly shows similar trend for elbow flexion and extension. Performance comparison of length variation is carried out for all subjects using the following metrics - mean, standard deviation, range and coefficient of variation (CV) as depicted in the table of FIG. 3. The results reported in the table of FIG. 3 indicate a clear reduction in standard deviation of body segment length from the order of 1-2 centimeters for Kinect skeleton data to a few millimeters for corrected skeleton data. There is an average 72% improvement in CV over all ROM exercises and all subjects.
Body segment orientation correction:
[0043] FIG. 7, with reference to FIGS. 1 through 6, depicts Body segment orientation correction by the system 100 of FIG. 1 in accordance with an embodiment of the present disclosure. More specifically, FIG. 7 depicts body segment orientation wherein dotted line denotes Kinect® output 702, solid line denotes correction 704 by the system 100. The corrected joint information not only helps achieve higher accuracy of measurements for joint Range of Motion analysis, but also aids in improving reliability of other assessment (e.g., clinical) of posture, gait, balance etc. from Kinect® skeleton information (or Kinect® Skeletal data). The body segment orientation obtained from raw Kinect® coordinates often suffers from inaccuracy during motion, an example of which is shown in FIG. 7 where the 2D projection of the elbow and wrist skeletal coordinates is shown to exceed physical boundaries of the hand while the hand is in motion. This situation has been frequently observed during Range of Motion activities of multiple subjects. By correcting the orientation and length of body segment it is evident that the accuracy of dynamic measurement of joint angles is much higher. During range of motion exercise the corrected length and orientation of the limb may reflect the actual trajectory of the physical limb. It has been validated by the ROM measurement with a clinical Goniometer. For shoulder abduction the deviation of angle computed from Kinect® skeleton data is in the order of 6.39 degrees ± 5.06 degrees which eventually corresponds to the fact reported by conventional systems and methods, whereas the proposed method is able to reduce the deviation by 1.88 degrees ± 1.1 degrees. The proposed body segment length and orientation correction also influences joint angle and range of motion analysis.
[0044] The systems and methods of the present disclosure provide techniques to improve accuracy of Kinect® skeletal joint coordinates for various applications (e.g., but are not limited to, rehabilitation). The Kinect® skeleton being a non-anthropometric model, there are body segment length and orientation variations over time. The proposed method implemented by the system 100 achieves and improves accuracy in estimation of body segment length as well as aligns the coordinates in the direction of the physical body segment that helps reconstruct the dynamic trajectory and angle of the body segment motion as accurately as possible. Although embodiments of the present disclosure describe body joint location identification in a 3D space, the embodiments/method may be implemented by the system 100 to identify body joint location in 2D space (or N-dimensions) as well.
[0045] The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims.
[0046] It is to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
[0047] The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
[0048] The illustrated steps are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments. Also, the words "comprising," "having," "containing," and "including," and other similar forms are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.
[0049] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.
obtaining, by one or more hardware processors (104), one or more input data comprising skeletal data, depth data and red, green, and blue, RGB, data pertaining to at least one user at one or more time stamps, wherein said one or more input data comprises three dimensional, 3D, coordinates pertaining to one or more skeleton joints;
estimating, using said one or more input data, by said one or more hardware processors (104), in a first set of frames, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain i) an estimate of one or more body joint locations and ii) an estimate of one or more body segment lengths;
iteratively identifying, by said one or more hardware processors, one or more probable correct body joint locations in a bounded neighborhood around said estimate of one or more body joint locations based on said depth data and said RGB data;
performing, by said one or more hardware processors, a comparison of i) each body segment length from the estimate of one or more body segment length associated with said one or more probable correct body joint locations and ii) a reference length;
identifying, by said one or more hardware processors, at least a subset of said one or more probable correct body joint locations as one or more candidate body joint locations based on said comparison;
determining a physical orientation of each body segment pertaining to each of said one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on said depth data and performing an analysis on each segmented 3D coordinate; and
identifying, from said one or more candidate body joint locations, a body joint location based on a minimal deviation in direction from said physical orientation of a corresponding body segment along with a feature descriptor of said RGB data and said depth data, wherein said minimal deviation is based on said depth data, and wherein said minimal deviation is based on range of motions performed by said user.
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors (104) coupled to said memory using said one or more communication interfaces, wherein said one or more hardware processors (104) are configured by said instructions to:
obtain one or more input data comprising skeletal data, depth data and red, green, and blue RGB data pertaining to at least one user at one or more time stamps, wherein said one or more input data comprising three dimensional, 3D, coordinates pertaining to one or more skeleton joints,
estimate, using said one or more input data, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain i) an estimate of one or more body joint locations and ii) an estimate of one or more body segment lengths,
iteratively identify one or more probable correct body joint locations in a bounded neighborhood around said estimate of one or more body joint locations based on said depth data and said RGB data,
perform a comparison of i) each body segment length from the estimate of one or more body segment length associated with said one or more probable correct body joint locations and ii) a reference length;
identify at least a subset of said one or more probable correct body joint locations as one or more candidate body joint locations based on said comparison,
determine a physical orientation of each body segment pertaining to each of said one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on said depth data and performing an analysis on each segmented 3D coordinate, and
identify, from said one or more candidate body joint locations, a body joint location based on a minimal deviation in direction from said physical orientation of a corresponding body segment along with a feature descriptor of said RGB data and said depth data, wherein said minimal deviation is based on said depth data and wherein said minimal deviation is based on range of motions performed by said user.
obtaining, by one or more hardware processors, one or more input data comprising skeletal data, depth data and red, green, and blue, RGB, data pertaining to at least one user at one or more time stamps, wherein said one or more input data comprises three dimensional, 3D, coordinates pertaining to one or more skeleton joints;
estimating, using said one or more input data, by said one or more hardware processors (104), in a first set of frames, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain i) an estimate of one or more body joint locations and ii) an estimate of one or more body segment lengths;
iteratively identifying, by said one or more hardware processors (104), one or more probable correct body joint locations in a bounded neighborhood around said estimate one or more body joint locations based on said depth data and said RGB data;
performing, by said one or more hardware processors (104), a comparison of i) each body segment length from the estimate of one or more body segment length associated with said one or more probable correct body joint locations and ii) a reference length;
identifying, by said one or more hardware processors (104), at least a subset of said one or more probable correct body joint locations as one or more candidate body joint locations based on said comparison;
determining a physical orientation of each body segment pertaining to each of said one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on said depth data and performing an analysis on each segmented 3D coordinate; and
identifying, from said one or more candidate body joint locations, a body joint location based on a minimal deviation in direction from said physical orientation of a corresponding body segment along with a feature descriptor of said RGB data and said depth data, wherein said minimal deviation is based on said depth data, and wherein said minimal deviation is based on range of motions performed by said user.
Erlangen, durch einen oder mehrere Hardware-Prozessoren (104), eines oder mehrerer Eingabedaten, die Skelettdaten, Tiefendaten und Rot-, Grün- und Blau-Daten, RGB-Daten, umfassen, die sich auf mindestens einen Benutzer bei einem oder mehreren Zeitstempeln beziehen, wobei das oder die Eingabedaten dreidimensionale Koordinaten, 3D-Koordinaten, umfassen, die sich auf eine oder mehrere Gelenke eines Skeletts beziehen;
Abschätzen in einem ersten Satz von Rahmen, unter Verwendung des oder der Eingabedaten, einer oder mehrerer Körpergelenkstellen und einer Körpersegmentlänge eines oder mehrerer Körpersegmente, die mit entsprechenden Gelenken eines Skeletts verbunden sind, durch den oder die Hardware-Prozessoren (104), um i) eine Abschätzung einer oder mehrerer Körpergelenkstellen und ii) eine Abschätzung einer oder mehrerer Körpersegmentlängen zu erlangen;
iteratives Identifizieren, durch den oder die Hardware-Prozessoren, einer oder mehrerer wahrscheinlicher korrekter Körpergelenkstellen in einer begrenzten Nachbarschaft um die Abschätzung einer oder mehrerer Körpergelenkstellen auf der Grundlage der Tiefendaten und der RGB-Daten;
Durchführung eines Vergleichs von i) jeder Körpersegmentlänge aus der Abschätzung einer oder mehrerer Körpersegmentlängen, die der oder den wahrscheinlichen korrekten Körpergelenkstellen zugeordnet sind, und ii) einer Referenzlänge durch den oder die Hardware-Prozessoren;
Identifizieren, durch den oder die Hardware-Prozessoren, mindestens einer Teilmenge der wahrscheinlichen korrekten Körpergelenkstelle oder -stellen als ein oder mehrere Kandidaten für Körpergelenkstellen auf der Grundlage des Vergleichs;
Bestimmen einer physischen Orientierung jedes Körpersegments, zu jeder der besagten Kandidaten für eine Körpergelenkstelle, indem eine oder mehrere 3D-Koordinaten jedes Körpersegments auf der Grundlage der Tiefendaten segmentiert werden und eine Analyse jeder segmentierten 3D-Koordinate durchgeführt wird; und
Identifizieren einer Körpergelenkstelle aus der oder den Kandidaten für eine Körpergelenkstelle auf der Grundlage einer minimalen Richtungsabweichung von der physischen Orientierung eines entsprechenden Körpersegments sowie eines Merkmalsdeskriptors der RGB-Daten und der Tiefendaten, wobei die minimale Abweichung auf den besagten Tiefendaten basiert und wobei die besagte minimale Abweichung auf dem Bereich der von dem Benutzer ausgeführten Bewegungen basiert.
einen Speicher, der Anweisungen speichert;
eine oder mehrere Kommunikationsschnittstellen; und
einen oder mehrere Hardware-Prozessoren (104), die unter Verwendung der einen oder mehreren Kommunikationsschnittstellen mit dem Speicher gekoppelt sind, wobei der oder die Hardware-Prozessoren (104) durch die Anweisungen konfiguriert werden, um
ein oder mehrere Eingabedaten zu erlangen, die Skelettdaten, Tiefendaten und Rot-, Grün- und Blau-Daten, RGB-Daten, umfassen, die sich auf mindestens einen Benutzer bei einem oder mehreren Zeitstempeln beziehen, wobei das oder die besagten Eingabedaten dreidimensionale Koordinaten, 3D-Koordinaten, umfassen, die sich auf eine oder mehrere Gelenke eines Skeletts beziehen,
unter Verwendung des oder der besagten Eingabedaten, ein oder mehrere Körpergelenkstellen und eine Körpersegmentlänge eines oder mehrerer Körpersegmente abzuschätzen, die mit entsprechenden Gelenken eines Skeletts verbunden sind, um i) eine Abschätzung einer oder mehrerer Körpergelenkstellen und ii) eine Abschätzung einer oder mehrerer Körpersegmentlängen zu erlangen,
iterativ eine oder mehrere wahrscheinliche korrekte Körpergelenkstellen in einer begrenzten Nachbarschaft um die besagte Abschätzung einer oder mehrerer Körpergelenkstellen auf der Grundlage der Tiefendaten und der RGB-Daten zu identifizieren,
einen Vergleich i) jeder Körpersegmentlänge aus der Abschätzung einer oder mehrerer Körpersegmentlängen, die der oder den wahrscheinlichen korrekten Körpergelenkstellen zugeordnet sind, und ii) einer Referenzlänge durchzuführen;
auf der Grundlage dieses Vergleichs mindestens eine Teilmenge der besagten wahrscheinlichen korrekten Körpergelenkstelle oder -stellen als eine oder mehrere Kandidaten für Körpergelenkstellen zu identifizieren,
eine physische Orientierung jedes Körpersegments zu jeder der besagten Kandidaten für eine Körpergelenkstelle zu bestimmen, indem eine oder mehrere 3D-Koordinaten jedes Körpersegments auf der Grundlage der Tiefendaten segmentiert werden und eine Analyse jeder segmentierten 3D-Koordinate durchgeführt wird, und
aus der oder den Kandidaten für eine Körpergelenkstelle eine Körpergelenkstelle auf der Grundlage einer minimalen Richtungsabweichung von der physischen Orientierung eines entsprechenden Körpersegments sowie eines Merkmalsdeskriptors der RGB-Daten und der Tiefendaten zu identifizieren, wobei die minimale Abweichung auf den Tiefendaten basiert und wobei die minimale Abweichung auf dem Bereich der von dem Benutzer ausgeführten Bewegungen basiert.
Erlangen, durch einen oder mehrere Hardware-Prozessoren, eines oder mehrerer Eingabedaten, die Skelettdaten, Tiefendaten und Rot-, Grün- und Blau-Daten, RGB-Daten, umfassen, die sich auf mindestens einen Benutzer bei einem oder mehreren Zeitstempeln beziehen, wobei das oder die Eingabedaten dreidimensionale Koordinaten, 3D-Koordinaten, umfassen, die sich auf eine oder mehrere Gelenke eines Skeletts beziehen;
Abschätzen in einem ersten Satz von Rahmen, unter Verwendung des oder der Eingabedaten, einer oder mehrerer Körpergelenkstellen und einer Körpersegmentlänge eines oder mehrerer Körpersegmente durch den oder die Hardware-Prozessoren, die mit entsprechenden Gelenken eines Skeletts verbunden sind, um i) eine Abschätzung einer oder mehrerer Körpergelenkstellen und ii) eine Abschätzung einer oder mehrerer Körpersegmentlängen zu erlangen;
iteratives Identifizieren, durch den oder die Hardware-Prozessoren, einer oder mehrerer wahrscheinlicher korrekter Körpergelenkstellen in einer begrenzten Nachbarschaft um die Abschätzung einer oder mehrerer Körpergelenkstellen auf der Grundlage der Tiefendaten und der RGB-Daten;
Durchführen eines Vergleichs i) jeder Körpersegmentlänge aus der Abschätzung einer oder mehrerer Körpersegmentlängen, die einer oder mehreren wahrscheinlichen korrekten Körpergelenkstellen zugeordnet sind, und ii) einer Referenzlänge durch den oder die Hardware-Prozessoren (104);
Identifizieren, durch den oder die Hardware-Prozessoren (104), mindestens einer Teilmenge der wahrscheinlichen korrekten Körpergelenkstelle oder -stellen als ein oder mehrere Kandidaten für Körpergelenkstellen auf der Grundlage des Vergleichs;
Bestimmen einer physischen Orientierung jedes Körpersegments zu jeder der besagten einen oder mehreren Kandidaten für eine Körpergelenkstelle, indem eine oder mehrere 3D-Koordinaten jedes Körpersegments auf der Grundlage der Tiefendaten segmentiert werden und eine Analyse an jeder segmentierten 3D-Koordinate durchgeführt wird; und
Identifizieren einer Körpergelenkstelle aus der oder den Kandidaten für eine Körpergelenkstelle auf der Grundlage einer minimalen Richtungsabweichung von der physischen Orientierung eines entsprechenden Körpersegments sowie eines Merkmalsdeskriptors der RGB-Daten und der Tiefendaten, wobei die minimale Abweichung auf den Tiefendaten basiert und wobei die minimale Abweichung auf dem Bereich der von dem Benutzer ausgeführten Bewegungen basiert.
obtenir, par le biais d'un ou plusieurs processeurs matériels (104), une ou plusieurs données d'entrée comprenant des données de squelette, des données de profondeur et des données rouges, vertes et bleues, RGB, se rapportant à au moins un utilisateur, à une ou plusieurs estampilles temporelles, dans lequel ladite une ou lesdites plusieurs données d'entrée comprennent des coordonnées tridimensionnelles, 3D, se rapportant à une ou plusieurs articulations de squelette ;
estimer, en utilisant ladite une ou lesdites plusieurs données d'entrée, par le biais dudit un ou desdits plusieurs processeurs matériels (104), dans un premier ensemble de trames, un ou plusieurs emplacements d'articulations de corps et une longueur de segment de corps d'un ou plusieurs segments de corps connectés à des articulations de squelette correspondantes, en vue d'obtenir i) une estimation d'un ou plusieurs emplacements d'articulations de corps et ii) une estimation d'une ou plusieurs longueurs de segments de corps ;
identifier de manière itérative, par le biais dudit un ou desdits plusieurs processeurs matériels, un ou plusieurs emplacements d'articulations de corps corrects probables dans un voisinage délimité autour de ladite estimation d'un ou plusieurs emplacements d'articulations de corps, sur la base desdites données de profondeur et desdites données RGB ;
mettre en œuvre, par le biais dudit un ou desdits plusieurs processeurs matériels, une comparaison entre i) chaque longueur de segment de corps provenant de l'estimation d'une ou plusieurs longueurs de segments de corps associées audit un ou auxdits plusieurs emplacements d'articulations de corps corrects probables et ii) une longueur de référence ;
identifier, par le biais dudit un ou desdits plusieurs processeurs matériels, au moins un sous-ensemble dudit un ou desdits plusieurs emplacements d'articulations de corps corrects probables, en tant qu'un ou plusieurs emplacements d'articulations de corps candidats, sur la base de ladite comparaison ;
déterminer une orientation physique de chaque segment de corps se rapportant à chaque emplacement parmi ledit un ou lesdits plusieurs emplacements d'articulations de corps candidats, en segmentant une ou plusieurs coordonnées 3D de chaque segment de corps sur la base desdites données de profondeur, et en mettant en œuvre une analyse sur chaque coordonnée 3D segmentée ; et
identifier, à partir dudit un ou desdits plusieurs emplacements d'articulations de corps candidats, un emplacement d'articulation de corps, sur la base d'un écart minimal de direction par rapport à ladite orientation physique d'un segment de corps correspondant, avec un descripteur de caractéristiques desdites données RGB et desdites données de profondeur, dans lequel ledit écart minimal est basé sur lesdites données de profondeur, et dans lequel ledit écart minimal est basé sur une série de mouvements réalisés par ledit utilisateur.
une mémoire stockant des instructions ;
une ou plusieurs interfaces de communication ; et
un ou plusieurs processeurs matériels (104), couplés à ladite mémoire, en utilisant ladite une ou lesdites plusieurs interfaces de communication, dans lequel ledit un ou lesdits plusieurs processeurs matériels (104) sont configurés par lesdites instructions de manière à :
obtenir une ou plusieurs données d'entrée comprenant des données de squelette, des données de profondeur et des données rouges, vertes et bleues, RGB, se rapportant à au moins un utilisateur, à une ou plusieurs estampilles temporelles, dans lequel ladite une ou lesdites plusieurs données d'entrée comprennent des coordonnées tridimensionnelles, 3D, se rapportant à une ou plusieurs articulations de squelette ;
estimer, en utilisant ladite une ou lesdites plusieurs données d'entrée, un ou plusieurs emplacements d'articulations de corps et une longueur de segment de corps d'un ou plusieurs segments de corps connectés à des articulations de squelette correspondantes, en vue d'obtenir i) une estimation d'un ou plusieurs emplacements d'articulations de corps et ii) une estimation d'une ou plusieurs longueurs de segments de corps ;
identifier de manière itérative un ou plusieurs emplacements d'articulations de corps corrects probables dans un voisinage délimité autour de ladite estimation d'un ou plusieurs emplacements d'articulations de corps, sur la base desdites données de profondeur et desdites données RGB ;
mettre en œuvre une comparaison entre i) chaque longueur de segment de corps provenant de l'estimation d'une ou plusieurs longueurs de segments de corps associées audit un ou auxdits plusieurs emplacements d'articulations de corps corrects probables et ii) une longueur de référence ;
identifier au moins un sous-ensemble dudit un ou desdits plusieurs emplacements d'articulations de corps corrects probables, en tant qu'un ou plusieurs emplacements d'articulations de corps candidats, sur la base de ladite comparaison ;
déterminer une orientation physique de chaque segment de corps se rapportant à chaque emplacement parmi ledit un ou lesdits plusieurs emplacements d'articulations de corps candidats, en segmentant une ou plusieurs coordonnées 3D de chaque segment de corps sur la base desdites données de profondeur, et en mettant en œuvre une analyse sur chaque coordonnée 3D segmentée ; et
identifier, à partir dudit un ou desdits plusieurs emplacements d'articulations de corps candidats, un emplacement d'articulation de corps, sur la base d'un écart minimal de direction par rapport à ladite orientation physique d'un segment de corps correspondant, avec un descripteur de caractéristiques desdites données RGB et desdites données de profondeur, dans lequel ledit écart minimal est basé sur lesdites données de profondeur, et dans lequel ledit écart minimal est basé sur une série de mouvements réalisés par ledit utilisateur.
obtenir, par le biais d'un ou plusieurs processeurs matériels, une ou plusieurs données d'entrée comprenant des données de squelette, des données de profondeur et des données rouges, vertes et bleues, RGB, se rapportant à au moins un utilisateur, à une ou plusieurs estampilles temporelles, dans lequel ladite une ou lesdites plusieurs données d'entrée comprennent des coordonnées tridimensionnelles, 3D, se rapportant à une ou plusieurs articulations de squelette ;
estimer, en utilisant ladite une ou lesdites plusieurs données d'entrée, par le biais dudit un ou desdits plusieurs processeurs matériels (104), dans un premier ensemble de trames, un ou plusieurs emplacements d'articulations de corps et une longueur de segment de corps d'un ou plusieurs segments de corps connectés à des articulations de squelette correspondantes, en vue d'obtenir i) une estimation d'un ou plusieurs emplacements d'articulations de corps et ii) une estimation d'une ou plusieurs longueurs de segments de corps ;
identifier de manière itérative, par le biais dudit un ou desdits plusieurs processeurs matériels (104), un ou plusieurs emplacements d'articulations de corps corrects probables dans un voisinage délimité autour de ladite estimation d'un ou plusieurs emplacements d'articulations de corps, sur la base desdites données de profondeur et desdites données RGB ;
mettre en œuvre, par le biais dudit un ou desdits plusieurs processeurs matériels (104), une comparaison entre i) chaque longueur de segment de corps provenant de l'estimation d'une ou plusieurs longueurs de segments de corps associées audit un ou auxdits plusieurs emplacements d'articulations de corps corrects probables et ii) une longueur de référence ;
identifier, par le biais dudit un ou desdits plusieurs processeurs matériels (104), au moins un sous-ensemble dudit un ou desdits plusieurs emplacements d'articulations de corps corrects probables, en tant qu'un ou plusieurs emplacements d'articulations de corps candidats, sur la base de ladite comparaison ;
déterminer une orientation physique de chaque segment de corps se rapportant à chaque emplacement parmi ledit un ou lesdits plusieurs emplacements d'articulations de corps candidats, en segmentant une ou plusieurs coordonnées 3D de chaque segment de corps sur la base desdites données de profondeur, et en mettant en œuvre une analyse sur chaque coordonnée 3D segmentée ; et
identifier, à partir dudit un ou desdits plusieurs emplacements d'articulations de corps candidats, un emplacement d'articulation de corps, sur la base d'un écart minimal de direction par rapport à ladite orientation physique d'un segment de corps correspondant, avec un descripteur de caractéristiques desdites données RGB et desdites données de profondeur, dans lequel ledit écart minimal est basé sur lesdites données de profondeur, et dans lequel ledit écart minimal est basé sur une série de mouvements réalisés par ledit utilisateur.
REFERENCES CITED IN THE DESCRIPTION
This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.
Patent documents cited in the description
Non-patent literature cited in the description
- HO, E. et al.Improving posture classification accuracy for depth sensor-based human activity monitoring in smart environmentsComputer Vision and Image Understanding, 2016, vol. 148, 97-110 [0004]
- METCALF, C. et al.Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb RehabilitationIEEE Trans. on Biomedical Engineering, 2013, vol. 60, 82184-2192 [0005]