SYSTEM AND METHOD FOR AUTOMATED APPLICATION OF A COSMETIC PRODUCT TO A USER

Abstract

 This patent of invention belongs to the technical field of portable devices or articles for cosmetic treatments with application of fluent materials to the lip surface, including the identification and processing of image data, more specifically it relates to a system and method for automatic application of cosmetic product to user with artificial intelligence, which recognizes the lip through an image and automatically applies lipstick to the user's lips with precision, essentially by means of a robotic applicator device (2) operating on a Z axis by a CNC system; computer vision camera (3) for image capture and corresponding depth map aimed at obtaining the 3D mapping of the lip and conversion to the 3D perspective of said robotic applicator device (2); and multisensory devices for user interaction for control and actuation of said robotic applicator device (2).

Description

[0001] This patent of invention belongs to the technical field of portable devices or articles for cosmetic treatments with application of fluent materials on the lip surface, including identification and processing of image data, more specifically it relates to a system and method for automatic application of cosmetic product to user with artificial intelligence, which recognizes the lip through an image and automatically applies lipstick to the user's lips with precision.

STATE OF THE ART

[0002] Different devices and methods for automatically applying cosmetic product to user are known in the state of the art. For example, patent documents BR102020022162-0 and BR102022004560-7, both owned by the applicant of the present invention, respectively disclosed a method for detecting and segmenting the lip region, and a device and method for automatically applying cosmetic product to user.

[0003] Among other patent literature documents, publication number IN202111050964 describes an automatic makeup application device, including an imaging unit that captures an image of the user's lips, providing selection suggestions regarding the lip pigment color and the designs stored in the database,

[0004] Patent document CN112643691 presents an automatic intelligent robot-type makeup and skin care device comprising a touch screen, a storage module, an automatic robot module, a high-definition camera, a three-dimensional scanner, a light sensor, and an infrared scanner. Said publication further describes the presence of a calculation module, a communication module, a storage module, a control module, a voice recognition module, and a sound system integrated into the touch screen.

[0005] Lastly, patent document US11568675 presents systems and methods for automated makeup application that allow a user to select and apply desired makeup styles to the user's face. The systems and methods include a computer application with a graphical user interface that allows selection of an appearance from a plurality of preconfigured appearances. A camera attached to a robotic arm records a facial map and color coding and sends this data to be stored in a virtual server database.

TECHNICAL PROBLEM

[0006] Thus, although the state of the art provides methods for image processing of the lip region, as well as a device for movement and automatic application of lipstick to user by means of CNC-oriented mechanisms, there is a need for a more advanced solution that uses a more precise automatic cosmetic product application system for a user, essentially comprising a lipstick applicator with resilience characteristics, a continuous liquid lipstick dispensing mechanism adjacent to said applicator, whose pressure applied to the user's lip is measured and controlled, in addition to other multisensory interaction components.

TECHNICAL SOLUTION

[0007] The present invention provides an automatic system for applying cosmetic product to user, ensuring more precise application, with facilitated lip contouring, as well as a surprising and pleasant user experience.

[0008] The present invention provides an improvement to a cosmetic product applicator with greater precision on the Z axis through computer vision, as well as a dispenser that ensures a constant flow of raw material at the applicator tip.

[0009] The present invention provides an improvement to a cosmetic product applicator by means of a resilient applicator tip and wherein the applicator device has a controlled pressure for applying lipstick to a user's lip.

[0010] The present invention provides an improvement in 3D mapping of the lip with AI by combining 2D lip segmentation with a depth map captured by the computer vision camera.

[0011] Additionally, the present invention provides greater possibility of inclusion for people with special needs due to the multisensory feedback made possible by visual (LEDs), sensory (raised buttons), and auditory components present in the automatic lipstick application device.

BRIEF DESCRIPTION OF THE FIGURES

[0012] The description that follows is not limited to the drawings or components cited, with reference to the following illustrations referenced below.

Figures 1A and 1B show views of the automatic cosmetic product application system to user, according to the present invention.

Figure 2 presents the automatic application system for cosmetic product to user, according to the present invention, with emphasis on the computer vision camera.

Figure 3 presents images of a three-dimensional representation of the lips, composed of a 3D point cloud of the lips obtained in one of the application tests of the present invention.

Figure 4 presents a diagram of the lip touch detection system, as specifically applied to mannequin.

Figure 5 presents test images of a setup for calculating the spring constant.

Figure 6 presents the applicator device and the lipstick wear area on the applicator tip.

Figure 7A presents an image of some exemplified samples under test for selecting FSR sensors.

Figure 7B presents an image of the sample exemplified in the test, including 3D cases encapsulating the test FSR sensors.

Figure 8 presents a simplified image of a test setup for verifying the sensitivity of the application FSR sensor.

Figure 9 presents a setup for conducting tests with a barometric pressure sensor.

Figure 10 presents the automatic application system for cosmetic product to user, highlighting an applicator tip present in the robotic applicator device, and a lipstick dispensing device connected to said applicator tip, for the purpose of automatically and continuously moistening said lipstick applicator tip.

Figure 11 presents an image of an extruder component present in the automatic cosmetic product application system to user.

Figure 12 presents the automatic cosmetic product application system to user, highlighting the presence of a touch sensor component located adjacent to the front surface of the case of the housing structure for the components of said automatic cosmetic product application system to user.

DETAILED DESCRIPTION

[0013] As seen in Figures 1A and 1B, the present invention presents an automatic cosmetic product application system to user (1), comprising a new robotic applicator device (2), application score, multisensory user interaction devices, and computer vision by means of a computer vision camera (3) that uses 3D mapping with image capture of the lips by AI.

[0014] More specifically, by means of the present invention it is possible to apply lipstick automatically by determining the positioning of the lip relative to the robotic applicator device (2).

[0015] Moreover, the present invention uses the computer vision camera (3), as shown in Figure 2, for capturing a color image and a corresponding depth map in order to obtain the 3D mapping of the lip, wherein, using Artificial Intelligence (AI), the 2D segmentation of the lip is combined, wherein it is detected in the color image, with a depth map captured by the computer vision camera (3), and, once the position of the lip is obtained, a conversion to the 3D perspective of the robotic applicator device is performed.

[0016] For determining the depth map, initially the computer vision camera (3), for example, the Intel RealSense D405 model, is capable of capturing both the color image and the depth map. For this purpose, for example, a library such as pyrealsense2 is used to perform essential operations.

[0017] First, the color image of the lip is captured, and the depth map is aligned to this image. Subsequently, spatial and temporal filters are applied in order to smooth and stabilize the depth map, so as to improve the quality of the obtained data.

[0018] In turn, the 3D mapping of the lip is obtained from the combination of the 2D segmentation with the depth map. Initially, the X and Y pixel coordinates of the lip are obtained from this color image. Next, for each of these pixels, the corresponding Z coordinate is extracted in millimeters from the depth map.

[0019] The conversion of X and Y coordinates in pixels to millimeters is performed using the intrinsic depth parameters of the computer vision camera, for example, the Intel RealSense D405, these parameters being obtained using the library, for example, pyrealsense2. This allows us to obtain the X, Y, and Z coordinates in millimeters, relative to the depth camera coordinate system.

[0020] This approach provides a three-dimensional representation of the lips, as illustrated in the images of Figure 3. In this figure, the 3D point cloud of the lips obtained in one of the application tests is presented, demonstrating the effectiveness of the method used.

[0021] Additionally and in a surprising manner, the present invention further presents a lip touch detection system, such as specifically applied to mannequin, shown in the diagram of Figure 4, essentially comprising a fixed mannequin (4), a computer vision camera (3) for mapping the X, Y, and Z axes, and capturing 3D images, a robotic applicator device (2) by convolutional neural network (CNC), a Z-axis sensor (5) that reads application parameters, and a mannequin with applied lipstick (6).

[0022] The essential electromechanical structure and operation of the robotized applicator device (2) by convolutional neural network (CNC) for carrying out the present invention may be understood, for example, from the understandings available to a person skilled in the art, provided by reading the teachings disclosed in patent document BR102022004560-7, owned by the applicant of the present invention.

[0023] To achieve the surprising effect of the present invention, it was necessary to choose the type of sensor to be used and the range of values within which they should operate in order to make it possible to have a lipstick application responsive to the pressure applied to the user's lip, the variable of interest being the pressure on the lip when applying a lipstick.

[0024] For determination of the variable of interest, the pressure on the lip when applying lipstick, several scientific articles were analyzed to define the necessary operating range, involving parameters of Lip pressure, Lipstick application, Lip cosmetics, Lipstick pressure, Lip force, Lip make-up, Lipstick technique, Lipstick deposition, Lipstick wear, and Lipstick adherence.

[0025] For example, the article "A pressure-distribution sensor (PDS) for evaluation of lip functions, 1996" focuses on the creation and application of a pressure sensor to analyze the forces and pressure patterns applied to the lips. In the experiments described in this article, the pressures used were between 1 and 2 g/mm², and with this result we have that pressures in this range are not inconvenient for the user.

[0026] In turn, the study presented in "Lip pressure at rest and during function in 2 groups of patients with different occlusions, 2011" analyzed the influence of the perioral musculature on tooth position, specifically the forces exerted by the soft tissues at rest and during swallowing. Lip pressure was measured using a force-sensitive device placed at the midline between the upper incisors. The measurements revealed that the average lip pressure at rest was 24.59 g/cm² and during swallowing was 24.87 g/cm².

[0027] The development of technologies for measuring lip pressure during the performance of wind instruments, such as trumpet and cornet, is addressed in the article "Sensor and Software Technologies for Lip Pressure Measurements in Trumpet and Cornet Playing - from Lab to Classroom, 2015". The calibration method used for the sensor system demonstrated measurement accuracy with a deviation of less than 5% and a measurement range from 0.6 N to a maximum load of 70 N.

[0028] A study aimed at testing a new sensor to evaluate lip function is presented in the article "Development of a novel composite sensor for evaluating lip function, 2018". The sensor is called a composite sensor (CS) and was used to measure the electromyographic activity (EMG) of the upper orbicularis oris muscle (OOM), the lip closure pressure (LP), and the intraoral barometric pressure (IP) in 20 healthy individuals (10 women and 10 men). The participants performed lip closure, blowing, and sucking tasks while EMG signals from the upper and lower OOM were recorded with conventional electrodes to evaluate the accuracy of the CS. The average lip closure pressures during maximum lip closure, blowing, and sucking ranged between 2 and 6 kPa for women and between 5 and 7 kPa for men. The corresponding intraoral pressures were evaluated as 0.0 ± 0.5, 3.2 ± 1.4, and -4.4 ± 2.6 kPa for women, and -0.5 ± 1.4, 4.9 ± 1.8, and -5.6 ± 2.8 kPa for men.

[0029] Furthermore, in the article "Lip pressure and resistance in subjects with and without occlusal alteration, 2019," participants were classified according to malocclusion into Class I, Class II, and Class III. A general decrease in lip pressure values was observed in all groups, including the group with normal occlusion, when compared with the reference values from the Iowa Oral Performance Instrument (IOPI) manual, presenting the following mean lip pressure values: Class I Group (women): Average of 20.75 Kpa, Minimum of 10 Kpa, and Maximum of 29 Kpa; Class I Group (men): Average of 29.50 Kpa; Minimum of 16 Kpa, and Maximum of 48 Kpa; Class II Group (women): Average of 16.80 Kpa, Minimum of 2 Kpa, and Maximum of 28 Kpa; Class III Group (women): Average of 15.55 Kpa, Minimum of 4 Kpa, and Maximum of 36 Kpa; and Class III Group (men): Average of 16.29 kPa, Minimum of 2 kPa, and Maximum of 51 kPa.

[0030] Thus, based on the results of this study, it is possible to identify that the sensor's operating range should operate at up to at least 50 kPa, which corresponds to lip pressure that causes injuries in the experiments.

[0031] There is a wide variety of sensors available on the market today, which can be broadly categorized according to the type of measurement (physical quantity), the detection principle employed, and the output signal.

[0032] However, in this invention, technology is provided that can extract the pressure value of a system composed of a device with an applicator tip pressed against a human lip.

[0033] In the present invention, two methods are used to measure the physical quantity: direct measurement and indirect measurement. Direct measurement is performed when the sensing element is sensitive to the quantity of interest, and indirect measurement is when the sensing element is sensitive to another quantity, but due to some specific arrangement this other quantity and the quantity of interest are correlated in a known manner.

[0034] In the direct methodology, sensors of three types of measurements are applied:

• Absolute: measured relative to perfect vacuum, that is, it is the difference between the pressure at a given measurement point and the vacuum pressure (absolute zero);
• Gaussian: measured relative to ambient pressure, that is, relative to atmospheric pressure, that is, the difference between the absolute pressure measured at any point and the atmospheric pressure; and
• Differential: pressure difference measured between two points, in which when any point other than vacuum or atmosphere is taken as a reference, it is said to measure differential pressure.

[0035] As an indirect measurement, a mass-spring system is envisioned, and for this arrangement a force sensor is used that performs:

• the measurement of the force that the mass exerts against the spring; and
• the measurement of the force that the spring exerts against the wall.

[0036] Still for this arrangement, it is possible to use a displacement sensor and measure the compression and extension of the spring.

[0037] For the system and method for automatic application of cosmetic product to user according to the present invention, a direct and absolute pressure sensor was chosen. This choice is due to the fact that the pressure of interest refers to the difference between the moment when an applicator tip is in contact with the lips and the moment when the applicator tip is not in contact with the lips. Additionally, and to assist, an indirect force sensor was also selected.

[0038] After understanding the technologies available on the market, it is necessary to estimate the sensitivity required for the automatic application of cosmetic product to user, and for this purpose, various testing efforts were carried out in laboratory to achieve the desired technical effect.

[0039] In this sense, it is necessary to calculate the maximum and minimum pressure of applying a lipstick for the selected sensor, whose formula for calculating pressure is given by Equation 1, in which P = pressure, F = Force, and A = Area:

[0040] With reference to the estimation of the spring elastic constant, for the force calculation, it was necessary to find the elastic constant (k) of the spring coupled to the smart lipstick applicator. For this purpose, masses (m) of known values were placed, and the displacement (x) of the spring was verified in each of the situations as shown in the images of Figure 5, which presents a setup for calculating the spring constant.

[0041] Thus, according to the elastic force formula Fe = k·x, with m representing the known masses m.g = k·x, g the acceleration of gravity of 9.8 m/s², 3 elastic constants were calculated, and, from an average, the estimated value was k = 21.3 Kg/s².

[0042] With the elastic constant calculated, the next step is to calculate the minimum and maximum force when applying the lipstick. For this purpose, for example, 4 people applied the lipstick to the mannequin with the lipstick coupled to a spring, and through the maximum and minimum compression of the spring it was possible to calculate the respective forces (Fe = k·x):

and

[0043] Finally, from such obtained force values, where P = F/A, it is possible to finally calculate the maximum and minimum pressure that the sensor must measure. For this purpose, the contact area of the lipstick is used when applying it on the mannequin, as shown in Figure 6, which presents the applicator device (2) and the lipstick wear area (7) at the applicator tip (9):

and

[0044] In addition to the sensor range and sensitivity criteria, a third selection criterion was added to the research: the nature of the output signal. The priority of the choices is for sensors with digital signal output to facilitate implementation and because it is a signal that will be used as feedback for the system, which requires a high response speed. However, sensors with an analog output signal are not completely ruled out and are also considered for the tests.

Table 1 - Selected Sensors
Name	Type	Scope	Sensitivity	Digital Output
WSEN-PDUS	Absolute Piezoresistive Pressure Sensor	0-1500kPa	2.7mV/KPa	I2C
ELVH-B002A-HRNJ-C-N2A4	Relative Piezoresistive Pressure Sensor	0 ~ 200kPa		I2C
MS563702BA03-50	Relative Piezoresistive Pressure Sensor	300 mbar - 1.2 bar	2 mbar	I2C

[0045] With reference to the tests with the selected sensors, the FSR sensor (Force-Sensing Resistor) is a device used to measure the force or pressure applied to a surface. It consists of a pressure-sensitive material that varies its electrical resistance according to the force exerted on it. The tests are initiated using this FSR sensor, as illustrated by some samples in Figure 7A.

[0046] Additionally, with the objective of achieving a uniform reading of the pressure on the sensors, 3D cases were designed, which are designed and manufactured to encapsulate the test FSR sensors, as shown in Figure 7B.

[0047] However, this pressure measurement method may be inefficient, since the area of the cap, by occupying the entire surface of the pressure sensor, does not provide sufficient signal outputs to be adequately interpreted, and only under extremely high manual pressures can the output values be read satisfactorily.

[0048] Additionally, it was found that the larger-sized sensor model, represented by Table 2, does not fit into the Z extrusion applicator arm, and consequently, this geometry is discarded.

Table 2 (Larger size model)
sensor reading	calculated pressure (Pa)	Mass
230.00	435.50	10g
400.00	871.10	20g
550.00	1,742.20	40g
750.00	4,355.00	100.00
800.00	6,533.00	150.00

[0049] After analyzing the behavior of FSR sensors of different sizes, including the medium-sized model (represented by Table 3) and the smaller-sized model (represented by Table 4), the medium-sized FSR sensor model was selected for use in the automatic lipstick application system, according to the present invention.

Table 3 (Medium-sized model)
sensor reading	calculated pressure (Pa)	Mass
80.00	1,248.40	10g
230.00	2,496.80	20g
450.00	4,993.60	40g
650.00	12,484.10	100.00
700.00	18,726.11	150.00

Table 4 (Smaller size model)
sensor reading	calculated pressure (Pa)	Mass
-	3,467.80	10g
-	6,935.60	20g
60.00	13,871.20	40g
100.00	34,677.99	100.00
140.00	52,016.99	150.00

[0050] In this way, a test setup was built to verify the sensitivity of the FSR sensor for the intended application. Figure 8 presents the planned setup, which comprises the FSR sensor (10), a hot glue piece point (11), a movable plunger (8), a flexible elastic component (12) (such as a spring), preceding the applicator device (2) with applicator tip (9).

[0051] To make the testing process feasible, it was necessary to make modifications to the Z-axis system in order to allow integration of the FSR sensor. Additionally, it was also necessary to implement a plunger that exerts pressure on the FSR sensor, allowing measurement of the applied force.

[0052] Initially, it is observed that the plunger applies very low pressure, insufficient to be detected by the sensor. To solve this problem, the strategy adopted was to reduce the contact area between the plunger and the sensor, using the hot glue piece (11), for example, with an approximate area of 25 mm².

[0053] The test was conducted as follows: the Z axis of the CNC system is coupled and integrated with the functionality of a repetier host. Through this control, the applicator tip (9) is gradually brought closer to the test mannequin's lips.

[0054] During the test, a reference was established for the "No pressure" state, when there is no contact with the lips. When the applicator tip makes a gentle contact with the mannequin's lips, "Light touch" has been defined. From this point onward, when the applied pressure is increased, the definition "Soft touch" is used. Values that should be avoided were categorized as "Moderate touch". The pressure values found are presented in Table 5.

Table 5
Category	Pressure
Without pressure	<520
Light touch	<550
Soft touch	<600
Moderate touch	600+

[0055] A second sensor option is considered for the present invention comprising: a barometric pressure sensor, such as with I2C communication, with the ELVH-B002A-HRNJ-C-N2A4 and WSEN-PDUS sensors being acquired, whose technical characteristics are presented in Table 6.

Table 6
Sensor: ELVH-B002A-HRNJ-C-N2A4	Sensor: WSEN-PDUS
ELV - Digital and Analog Series	Range from 0 to 1500 kPa 5,722 * 10-2 kPa/digit
B002A - 0 bar to 2 bar Absolute	Precision 0,01 %FSS 15-bit ADC
HRNJ - Package	Response time 2.2 ms
C - 0 to 50°C	5V supply voltage
N - No Parylene
2 - I2C, Address 0x28
A - 214 digital counter
4 - 3V3 Power Supply
Response time 0.46 ms
Startup 3ms
Clock 100kHz to 400kHz
500-ohm pull-up resistor

[0056] For conducting the tests, a test setup (13) was built with a barometric pressure sensor of greater complexity than the FSR sensor, as shown in Figure 9, which comprises a pressure sensor (14), hose (15), syringe (16) preceding the applicator device (2).

[0057] For conducting the tests, the following methodology is adopted, in which the characterization of the two sensors is performed, whose test setup comprises:

• connecting the system to the syringe with the plunger in the 2.5 mL position and recording the value read by the sensor;
• pressing the plunger to move it to the 2.0 mL position and recording the value read by the sensor;
• pressing the plunger to move to the 1.0 ml position and recording the value read by the sensor;
• pressing the plunger to move to the 0.5 mL position and recording the value read by the sensor;
• relaxing the plunger so that it returns to the original 2.5 ml position; in this case it is necessary to pull it all the way so that it releases some of the internal pressure, since in the 0.5 ml position part of the pressure escapes, doing this until the relaxed system is at 2.5 ml;
• pulling the plunger to move to the 3.0 ml position and noting the value read by the sensor;
• pulling the plunger to move to the 4.0 mL position and recording the value read by the sensor; and
• pulling the plunger to move to the 5.0 ml position and recording the value read by the sensor.

[0058] Based on the results obtained in the tests, it is concluded that the setup created can operate within a range of 0.6 bar to 1.8 bar, and values exceeding these thresholds cause seal failure, and the system tends to lose pressure when it reaches a value above 1.8 bar, or gain pressure when it reaches a value below 0.6 bar. With the syringe used and the system starting from 1 atm, a 1.5 cm excursion is obtained in the syringe plunger.

[0059] According to the present invention, and as shown in Figure 10, the automatic cosmetic product application system to user (1), which, in addition to enabling improved Z-axis precision through the computer vision provided by the computer vision camera (3), further comprises a resistive sensor capable of measuring and enabling control over the degree of pressure exerted by the applicator tip (9) during the application of lipstick to the lip, performed by the robotic applicator device (2), which is responsive to touch on the lip.

[0060] The surprising robotic applicator device (2) presented by the present invention further comprises the applicator tip (9) as a resilient component, and a spring structure that allows resilient movement of said applicator tip (9), while signaling to the control module of the automatic cosmetic product application system to user (1) the variation in application pressure occurring upon contact of the applicator tip (9) with a user's lip, and the need or not for correction of this application pressure value.

[0061] In order for the automatic cosmetic product application system to user (1) to provide a more precise application of lipstick to the user's lip, the applicator tip (9) may further comprise a flocked cushion tip.

[0062] In addition to the applicator tip (9) of the automatic cosmetic product application system to user (1), a lipstick dispensing device (17) is also provided, connected to said applicator tip (9), for the purpose of automatically and continuously moistening said lipstick applicator tip (9).

[0063] Thus, as seen from the various and exhaustive tests performed for the scope of the automatic cosmetic product application system to user (1) of the present invention, it resulted from meticulous and well-structured methodologies, integrating several stages of research, development, and evaluation, beginning with comprehensive preliminary studies to explore the variety of lipstick applicators available on the market, identifying trends, innovative features, and points for improvement to be addressed by the present invention.

[0064] Based on the insights preliminarily acquired during the development of the present invention, prototypes were developed and structured within the applicator mechanical arm, containing the applicator tip, compliant Z, spring, plunger, slot for the pressure sensor (FSR), and the extrusion duct for the liquid lipstick.

[0065] Next, the complete structure of the object of the present invention underwent rigorous performance and usability tests, including a plurality of use simulations with the participation of real users, so as to ensure the selection of the best solution to meet consumer expectations, guaranteeing precise application, facilitated lip contouring, and a surprising and pleasant user experience.

[0066] To determine the ideal application thresholds for the new lipstick applicator, a systematic procedure was carried out involving the participation of human users. This threshold acquisition process was divided into two stages, the first with the objective of delimiting the safe pressure zone for the user.

[0067] In a first step, the applicator mechanical arm was removed from the housing case of the accommodation structure of the automatic cosmetic product application system to user, being manually applied to the users' lips. These data were collected, statistically processed, and the safe pressure threshold for users' lips was defined.

[0068] In a second stage, these thresholds were refined through tests with various users. Users applied the lipstick following a specific protocol under controlled conditions, and the quantitative data collected made it possible to establish consistent application patterns. These thresholds are fundamental to ensure that the new applicator provides proper lipstick application, offering comfort, safety, and uniformity in application.

[0069] According to the present invention, for the automatic cosmetic product application system to user (1), a lipstick extruder component (18) was created and developed, as shown in the image of Figure 11, and initially, the use of a peristaltic pump was conceived to pressurize the liquid lipstick in the ducts up to the applicator tip.

[0070] The peristaltic pump is an efficient device for this purpose, as it uses a system of rollers that compress the hose and propel the liquid precisely. However, during the development of the present invention, it was found that this option occupies considerable space and makes it difficult to integrate the extruder into a compact format.

[0071] Thus, for the extrusion of the liquid lipstick through the applicator tip (9), a custom pump was created and developed that pressurizes a syringe (19), for example of 10 ml, using a stepper motor (20), both positioned on a support structure (21) of said lipstick extruder component (18), which actuates the plunger of said syringe (19). The support structure (21) of said lipstick extruder (18), shown in the image of Figure 11, for example, can be 3D printed (FDM) with PLA material. This structural arrangement of the lipstick extruder component (18) converts the rotational movement of the stepper motor (20) into linear movement, pushing the plunger of the syringe (19) filled with liquid lipstick, and once the liquid lipstick exits the syringe, it travels through a hose (22), for example made of silicone, until it reaches the mechanical arm of the robotic applicator device (2).

[0072] The connection between the hose (22) and the mechanical arm of the robotic applicator device (2) is made using small plastic tubes with a thickness greater than the thickness of the hose (22), so as to ensure a good seal. Inside the mechanical arm of the robotic applicator device (2), the liquid lipstick travels by capillarity until it reaches its end at the applicator tip (9), to be applied to the user's lips.

[0073] To ensure safety and proper sealing, several tests were conducted. These tests consisted of pressurizing the liquid lipstick with pressures greater than those normally used during a typical application. The objective was to prevent any lipstick spillage inside the structure of the automatic cosmetic product application system on a user (1) of the present invention, and to ensure a controlled and safe extrusion process.

[0074] In addition, experiments were conducted to evaluate the performance of the lipstick extruder component (18). In this phase, the amounts of lipstick released per unit of time, the uniformity of lipstick output, and the consistency of the dosages during use were measured, such data being essential to optimize the operation of the automatic cosmetic product application system to user (1) of the present invention, and to ensure precise and homogeneous application of the lipstick to the lips.

[0075] In this way, the change to a custom pump using a stepper motor proved advantageous in terms of efficiency and size, making the lipstick extruder component (18) more practical and suitable for the purpose of the present invention.

[0076] Still, with the objective of ensuring that all users could interact efficiently with the solution proposed by the automatic cosmetic product application system to user (1), regardless of their disabilities, multisensory feedback components were created and developed to integrate into the hardware system of said automatic cosmetic product application system to user (1), comprising components involving sound aspects (such as speakers and/or microphones, indicative of information and/or operation commands), visual aspects (such as LEDs indicative of operation information) and tactile aspects (such as buttons or touch sensors and/or in relief), aiming at the greatest possible inclusion for the user.

[0077] For example, as seen in Figure 12, the automatic cosmetic product application system to user (1) is shown, highlighting the presence of a touch sensor component (23) located adjacent to the front surface (24) of the case of the component housing structure of said automatic cosmetic product application system to user (1).

[0078] Although the present invention illustrates various multisensory feedback components, such as auditory, visual, and tactile, obviously, other types of embodiments of the present solution may be implemented in the present automatic cosmetic product application system to user (1), without departing from the scope claimed herein, for example:

• it is possible that the solution should adapt to the possible heights of the user, so that the user can move to either side and lose the idea of a totem or rigid, non-adjustable structure;
• it is important that the solution presents a way to ensure that the user can know exactly where to position their head comfortably and safely; and
• it is possible for the user to choose the lipstick colors, so that such chosen lipstick color option adapts to the lipstick extruder component.

[0079] In particular, for defining the activation button of the present automatic cosmetic product application system for a user, options available on the market were surveyed that meet the needs of the target audience, for example, a person with a physical disability and a person with a visual disability. Based on this survey, the options were evaluated according to their compliance with the prototype's hardware and software requirements, the necessary development effort, acquisition cost, and alignment with the user profile.

[0080] Considering that the system proposed by the present invention aims to provide a product that promotes the inclusion of all people, without discriminating against any disability, favoring the usability and non-adaptability of the product. It is important that, for people with the target audience profile, solutions that require physical actuation must have a size, color, and texture that helps identify and locate the button, and must provide sensory feedback to the user. In addition, considering people with physical disabilities, this button should be easy to actuate, such as a joystick or a tactile surface, which do not require grasping movement or specific gestures. Thus, it was concluded that the options for activation component forms by push button, joystick, or touch sensor are capable of being implemented in execution modes of the present invention and may satisfactorily serve the user.

[0081] In order to provide the various functionalities achieved by the present invention, the automatic cosmetic product application system to user further comprises control circuits and a user interface interconnected with the sound, visual, and tactile components; and a feedback sensor circuit.

[0082] With reference to the user interface circuit, for audio feedback, for example, a loudspeaker is used, and for visual feedback, for example, an LED strip is used. In other possible embodiments of the present invention, an RGB LED strip may be implemented without departing from the scope claimed herein.

[0083] With reference to the control circuit, the activation of the LED strip, speaker, pressure sensor reading, and, above all, processing of the computer vision model is carried out by a single multiplatform board, for example, such as a Raspberry Pi. The control circuit may also comprise, for example, an Arduino Mega, to perform control of the motors present in the automatic cosmetic product application system to user according to the present invention.

[0084] Finally, with reference to the pressure sensor circuit, this comprises a circuit, for example, for the analog FSR sensor. As described above, two sensors are considered in the tests of the present invention. In one embodiment of the present invention, the pressure sensor circuit may also comprise a circuit for an I2C digital sensor, if its use is required.

Claims

1. SYSTEM FOR AUTOMATIC APPLICATION OF COSMETIC PRODUCT TO USER, characterized by said system for automatic application of cosmetic product to user (1), comprising:

a robotic applicator device (2) acting along a Z axis by a CNC system; a computer vision camera (3) for image capture and corresponding depth map acquisition for obtaining the 3D mapping of the lip and conversion to the 3D perspective of said robotic applicator device (2); and multisensory devices for user interaction for control and actuation of said robotic applicator device (2);

wherein the multisensory user interaction devices of said automatic cosmetic product application system to user (1) comprise components involving sound aspects, such as speakers and/or microphones, indicating information and/or operating commands; components involving visual aspects, such as LEDs indicating operating information; and components involving tactile aspects, such as buttons or touch sensors and/or in relief;

wherein said robotic applicator device (2) comprises an applicator tip (9) being a resilient component with a flocked cushion tip; a resistive FSR sensor (10) for measuring the pressure exerted by the applicator tip (9) during lipstick application to the lip; a flexible elastic component (12), such as a spring, for resilient movement of said applicator tip (9), positioned adjacent to the applicator device (2) with said applicator tip (9);

a control module of the automatic cosmetic product application system to user (1), for controlling and actuating the variation in application pressure occurring upon contact of the applicator tip (9) with a user's lip;

a lipstick dispensing device (17) connected to said applicator tip (9) for automatic and continuous moistening of lipstick on said applicator tip (9) of lipstick;

a lipstick extruder component (18) that operates as a peristaltic pump for extruding the liquid lipstick through the applicator tip (9), comprising a syringe (19) with a plunger and a stepper motor (20), both positioned on a support structure (21) of said lipstick extruder component (18); and

a hose (22) and a mechanical arm of the robotic applicator device (2) connected by a conduit tube for liquid lipstick by capillarity to one end of the applicator tip (9).

2. METHOD FOR AUTOMATIC APPLICATION OF COSMETIC PRODUCT TO USER, characterized by:

the computer vision camera (3) performing the capture of a color image and the capture of a depth map of the lip aligned with that image; performing the combination of the 2D segmentation of the lip, detecting the color image with a depth map; applying spatial and temporal filters in order to smooth and stabilize the depth map; obtaining the 3D mapping of the lip from the combination of the 2D segmentation with the depth map; wherein initially, the X and Y coordinates in pixels of the lip are obtained from this color image; next, for each of these pixels, the corresponding Z coordinate in millimeters is extracted from the depth map; performing the conversion of the X and Y coordinates in pixels to millimeters relative to the depth parameters of the computer vision camera; obtaining the X, Y and Z coordinates in millimeters, referring to the coordinate system of the depth camera; and, with the lip position obtained, performing a conversion to the 3D perspective of the robotic applicator device (2) acting on a Z axis by means of a CNC system;

a resistive FSR sensor (10), acting within an operating range, measuring and providing control over the degree of pressure exerted by the applicator tip (9) during lipstick application to the lip, performed by the robotic applicator device (2), which is responsive to touch on the lip and signals to the control module of the automatic cosmetic product application system to user (1), the variation in application pressure occurring when the applicator tip (9) touches the lip of a user, and the need, or not, to correct this pressure application value;

a lipstick dispensing device (17) connected to said applicator tip (9), automatically and continuously moistening said applicator tip (9) with lipstick;

a lipstick extruder component (18), acting as a peristaltic pump, pressurizing a syringe (19), using a stepper motor (20), both positioned on a support structure (21) of said lipstick extruder component (18); converting the rotary motion of the stepper motor (20) into linear motion, pushing the plunger of the syringe (19) filled with liquid lipstick, and once the liquid lipstick leaves the syringe (19), it travels through a hose (22) until it reaches the mechanical arm of the robotic applicator device (2); and within the mechanical arm of the robotic applicator device (2), the liquid lipstick travels by capillarity until it reaches its end at the applicator tip (9), to be applied to the user's lips; and

multisensory devices comprising components involving sound aspects, components involving visual aspects, and components involving tactile aspects, interacting with the user through control circuits and user interface and feedback sensor circuit.

3. METHOD FOR AUTOMATIC APPLICATION OF COSMETIC PRODUCT TO USER, according to claim 2, and characterized by the resistive FSR sensor (10) for measuring the pressure exerted by the applicator tip (9) during lipstick application on the lip, operating under the minimum measured pressure of 1.33 kPa and the maximum measured pressure of 6.65 kPa.

Drawing