The present invention relates to a method for sorting objects travelling on a conveyor belt, where image data is captured by at least one imaging sensor for an image comprising at least one object travelling on the conveyor belt and where imaging sensor provides color image data.

In many recycling centers that receive recyclable materials, sortation of materials may be done by hand or by machines. For example, a stream of materials may be carried by a conveyor belt, and the operator of the recycling center may need to direct a certain fraction of the material into a bin or otherwise off the current conveyer. These conventional sorting systems are large in size and lack flexibility due to their large size. Moreover, they lack the ability to be used in recycling facilities that handle various types of items such as plastic bottles, aluminum cans, cardboard cartons, and other recyclable items, or to be readily updated to handle new or different materials. It is also known to use automated solutions using sensors or cameras to identify materials carried on a conveyor belt, which via a controller may activate a sorting mechanism. However, these new solution does not always function perfect.

The conventional plastic sorting solutions are based on near-infrared / short-wave-infrared (NIR/SWIR) spectrometry, where e.g. a NIR/SWIR reflection spectrum is collected for each plastic object and the spectrum identifies the material type of the plastic object - which determines the sorting.
The NIR/SWIR-spectrometric sorting systems are unable to handle dark and black plastics as all dark and black plastics return the same flat spectrum in the NIR/SWIR-range regardless of the material type. Moreover, NIR/SWIR-systems also cannot discriminate properly between white and transparent plastics, which is important for proper recycling. Another drawback of the spectrometric systems is that the system cannot sort waste by application - e.g. they cannot sort food from non-food plastics.

Finally spectrometric systems are also challenged by composite plastic objects, e.g. a bottle with a bottle cap and a foil covering the bottle - the spectrometric system might sort the object based on the foil.

An object of the present invention is to provide a method for identifying and sorting waste material in a more precise manner.

A further object is to provide a cost-effective and effective method of identifying and sorting waste material, in particular waste material comprising plastic

Normally, when waste and garbage is collected and initial sorting into different material categories is performed. The categories may e.g. be glass, metal, plastic, cardboard, paper and biological waste. Thus, when the waste reaches the recycling center each material fraction is normally sorted into even finer fractions. The metal fraction may sorted into aluminium and iron fractions and plastic into fractions based on different plastic types such as PE, PP or fractions with soft and hard plastic.

The present invention relates to a method for sorting objects travelling on a conveyor belt,
the method comprising:

• receiving image data captured by at least one imaging sensor for an image comprising at least one object travelling on the conveyor belt said imaging sensor provides color image data with a spatial resolution of at least 0.4 px/mm;
• executing a product detection and recognition module on a processor, the product detection and recognition module being configured to detect characteristics of the at least one object travelling on the conveyor belt by processing the image data;
• determining an expected time when the at least one object will be located within a sorting area of at least one sorting device; and
• selectively generating a robot control signal to operate the at least one sorting device on whether the at least one object comprises a target object.

In this context the term "sorting device" should includes a robot, mechanical actuators, actuators based on a solenoid, air jet nozzles etc.

The terms "object", "item" and "product" and their plural form are used interchangeable in this text.

The imaging sensor is preferably a camera, which are able to provide color images in environment with low light intensity, e.g. light intensities around 500 lumen. Preferably, the camera operates at light intensities around 1000 lumen or more, such as 1500 lumen or more.

In an embodiment the target object is guided to a collection device in the sorting area by means of the sorting device. The sorting robot may control e.g. a pusher device or air jet nozzles which are suitable for guiding the target object to a collection device.

In an embodiment of the method according to the invention, the characteristics of the at least one object travelling on the conveyor belt is the physical appearance or shape of the object. Thus, the method is capable of identifying objects based on their design features.

In an embodiment of the method according the invention, the characteristics of the at least one object travelling on the conveyor belt is the color and/or transparency of the object. Thus, the method is also suitable for detecting objects based on their color or transparency.

In an embodiment the characteristics of the at least one object travelling on the conveyor belt is selected from vendor names, brand names, product names, trademarks, logos, symbols, slogans or a combination of two or more of the characteristics. The product detection and recognition module may interact with one or more databases comprising information about vendor names, brand names, product names, trademarks, and slogans and retract information from these database to identify objects.

In respect of the three above mentioned embodiments it is clear that the features of these embodiments, may be combined in any desireable manner.

For the purpose of obtaining a more precise identification the product detection and recognition module may apply two or more characteristics in the product detection and recognition process.

In an embodiment the imaging sensor has a spatial resolution is at least 2 px/mm (pixel/mm). With such a spatial resolution the imaging sensor is able to provide very detailed images.

In an embodiment the spatial resolution is at least 4 px/mm. When the spatial resolution is about 4 px/mm or more, the imaging sensor is able to detect very small scale details, such as logos with an extent of about 5 mm or less.

In an embodiment the method is adapted for detecting and recognizing objects used as packaging or container for food items, such a bottles and trays. The objects may e.g. be bottles for juice and soft drinks made from plastic, such as transparent plastic. The object may also be a tray used for e.g. meat or biscuits. The trays may e.g. be made from plastic material in any desired colors. The trays may be marked with a "fork and knife" logo indicating the tray is for use with foodstuff.

In an embodiment the method is adapted for detecting and recognizing black objects. Black objects are difficult to detect due to the low reflection from the material, however, the method according to the invention has proven to be surprisingly efficient in detecting and recognizing black objects. The black object may e.g. be made from plastic which it is desirable to sort properly. Preferably the black object is tray for food, such as a plastic tray for meat.

In one aspect of the method the detection and recognition of object are based on the detection and recognition modules interaction with one or more databases, such as databases comprising information about e.g. specific product (such as materials used in the product), vendor names, brand names, product names, trademarks, and slogans.

The method may also apply a convolutional neural network.

Thus, in an embodiment of the method according to the invention, the product detection and recognition involves a convolutional neural network.

For the convolutional neural network to be used for identification of items/objects learned during training operations, the method proceeds with an inference process where during operation the neural network parameters are loaded into a computer processor (such as the processor mentioned above) in a neural network program that implements the convolutional neural network. During operation, the processor may then receive images from the imaging sensor, and pass that image through the convolutional neural network program. The convolutional neural network then outputs a decision, indicating, for example, the type of object present in the image with highest likelihood.

In a training operation, the labeled data is used by a training algorithm (which may be performed by a training processor) to optimize the convolutional neural network to identify the object in the captured images with the greatest feasible accuracy. As would be readily appreciate by one of ordinary skill in the art, a number of algorithms may be utilized to perform this optimization, such as Stochastic Gradient Descent, Nesterov's Accelerated Gradient Method, the Adam optimization algorithm, or other well-known methods. In Stochastic Gradient Descent, a random collection of the labeled images is fed through the network. The error of the output neurons is used to construct an error gradient for all the neuron parameters in the network. The parameters are then adjusted using this gradient, by subtracting the gradient multiplied by a small constant called the "learning rate". These new parameters may then be used for the next step of Stochastic Gradient Descent, and the process repeated.

The result of the optimization includes a set of convolutional neural network parameters (which are stored in a memory) that allow the convolutional neural network to determine the presence of an object in an image. During operation, the neural network parameters may be stored on digital media. In an example of implementation, the training process may be performed by creating a collection of images of items, with each image labeled with the category of the items appearing in the image. Each of the categories can be associated with a number, for instance the conveyor belt might be 0, a carton 1, a transparent plastic bottle 2, etc. The convolutional neural network would then comprise a series of output neurons, with each neuron associated with one of the categories. Thus, neuron 0 is the neuron representing the presence of conveyor belt, neuron 1 represents the presence of a carton, neuron 2 represents the presence of a transparent plastic bottle, and so forth for other categories.

The method may be designed to detect and recognize waste objects using very specific categories, product-specific categories, i.e. to classify each waste object as belonging to a specific vendor, brand, product and/or application (food, cosmetics, other). This may be enabled by e.g., using an application/shape/color hierarchical ordering:

• Food
  • ∘ Bottle
    • ▪ Transparent
    • ▪ White
    • ▪ Black
    • ▪ Blue
    • ▪ Green
    • ▪ Red
    • ▪ Other
  • ∘ Tray
    • ▪ Transparent
    • ▪ White
    • ▪ Black
    • ▪ Blue
    • ▪ Green
    • ▪ Red
    • ▪ Other
  • ∘ Other
    • ▪ Transparent
    • ▪ White
    • ▪ Black
    • ▪ Blue
    • ▪ Green
    • ▪ Red
    • ▪ Other
• Cosmetics
  • ∘ Bottle
    • ▪ Transparent
    • ▪ White
    • ▪ Black
    • ▪ Blue
    • ▪ Green
    • ▪ Red
    • ▪ Other
  • ∘ Other
    • ▪ Transparent
    • ▪ White
    • ▪ Black
    • ▪ Blue
    • ▪ Green
    • ▪ Red
    • ▪ Other
• Other
  • ▪ Transparent
  • ▪ White
  • ▪ Black
  • ▪ Blue
  • ▪ Green
  • ▪ Red
  • ▪ Other

For the convolutional neural network to be used for identification of items/materials learned during training operations, the method proceeds with an inference process where the neural network parameters are loaded into a computer processor (such as the processor mentioned above) in a neural network program that implements convolutional neural network. During operation, the processor may then receive images from the imaging sensor, and pass that image through the convolutional neural network program. The neural network then outputs a decision, indicating, for example, the type of item/material present in the image with highest likelihood.

In an embodiment of the method, the method further comprise interaction with a product database. The product database may contain information about an identified object, such as which material or materials the object is manufactured from. Such information is very useful in a sorting process.

In an embodiment the object is a plastic object. The object may be made from plastic material such as e.g. PE, PP, PS, PET, PVC, PVA or ABS. Large amount of plastic is used today, which generates large amount of plastic waste and the present invention provides a method for efficient sorting of plastic material.

The invention also provides a system for sorting objects, the system comprising:

• at least one imaging sensor;
• a controller comprising a processor and a memory storage, wherein the controller receives image data captured by the at least one imaging sensor; and
• at least one sorting robot coupled to the controller, wherein the at least one sorting robot is configured to receive an actuation signal from the controller;
• wherein the processor executes an object identification module configured to detect objects travelling on a conveyor belt and recognize at least one target item travelling on a conveyor belt by processing the image data and to determine an expected time when the at least one target item will be located within a diversion path of the sorting robot; and
• wherein the controller selectively generates the actuation signal based on whether a sensed object detected in the image data comprise the at least one target item.

The invention will now be described in further details with reference to drawings in which:

Figure 1:

shows an embodiment with a conveyor and a robot;

Figure 2:

shows an embodiment with just a conveyor;

Figure 3:

shows an embodiment without conveyor (nor robot);

Figure 4:

shows a detailed view of the invention;

Figure 5:

shows a method for logo/symbol detection;

Figure 6:

shows the principles of text detection and recognition;

Figure 7:

illustrates the principles of neural network object detection;

Figure 8:

illustrates the principles of two-stage neural network object detection;

Figure 9:

shows an embodiment linking high resolution with a neural network; and

Figure 10:

shows examples of symbols, which can be detected by the method.

The figures are only intended to illustrate the principles of the invention and may not be accurate in every detail. Moreover, parts which do not form part of the invention may be omitted. The same reference numbers are used for the same parts.

Figure 1 is a diagram showing the principles of the invention. Reference number 1 indicates the conveyer belt. Box 2 illustrates the "scene" on the conveyer belt 1, i.e. the conveyor belt with one or a number of items. The scene 2 reflects light, which are registered by the camera 3, and transformed into an image. The image is processed in a product detection and recognition module 4 to identify the item or items present in the scene 2. The information from the product detection and recognition module 4 is send to the sorting control 5, which may obtain further information about the identified items from the product database 6.

The sorting control 5 communicates with a robot controller 7 which control a robot 8, which is physically able to intervene in scene 2b in a sorting area on the conveyer belt 1 and sort the item or items into specific categories of waste material.

The speed of the conveyor belt 1 is monitored, and an encoder 9 sends information about the speed of the conveyer belt 1 to a synchronizer 10. The synchronizer sends signals to the camera 3 and determines how many images the camera 3 should take per second. The synchronizer also sends signals to the robot controller 7 with information about when the scene 2b reaches the sorting area. The encoder 9 may also send signals directly to the robot controller 7.

Scene 2a and scene 2b are in principle identical, and the reference numbers only indicates that the conveyor belt has moved the scene a distance from the point where scene 2a was registered by the camera 3.

Figure 2 illustrates the principles of the conveyor belt information system. The speed of the conveyor belt is monitored, and the information about the speed is transformed by the encoder 9 and send as an encoder signal to the synchronizer 10. The synchroniser 10 sends a signal to the camera 3 when an image of the scene 2a needs to be provided. Depending on the actual speed of the conveyor belt the camera may provide several images of the scene 2a per second. However, if the speed of the conveyor belt is slow the camera 3 only needs to provide a few images per minute.

The images from the camera 3 are send to the product detection and recognition module 4 to be processed and the items in the image identified. The information about the identified items are then send to the visualization and statistics module 5a for further processing to display or otherwise provide the information that can be extracted or accumulated from the detection system. The visualization and statistics module 5a is integrated with the sorting control 5.

The visualization and statistics module 5a communicates with the product database 6 to obtain more detailed information about product properties for an identified item. The information about product properties may e.g. be information about material.

Based on the information available the sorting control sends commands to the robot controller (not shown in figure 2), which will activate the robot to perform desired sorting motions and actuations, when the scene 2a reaches the sorting area (scene 2b).

Figure 3 illustrates the principles of the information system. The information system includes the camera 3, the product detection and recognition module 4, the visualization and statistics module 5a and the product database 6.

The images from the camera 3 are send to the product detection and recognition module 4 where the items on the images (appearing on the scene 2a) are identified.

The camera 3, the lightning and the conveyor speed must be adjusted to provide images which meet the requirements, e.g. images with sufficient lightning and with little motion blur.

The information about the identified items are then send to the visualization and statistics module 5a for further processing. The visualization and statistics module 5a is integrated with the sorting control 5.

The visualization and statistics module 5a communicates with the product database 6. The visualization and statistics module 5a can search the product database 6 and obtain more detailed information about product properties for an identified item. The information about product properties may e.g. be information about material.

Based on the information available, the sorting control sends commands to the robot controller, which will activate the robot to perform desired sorting motions and actuations. This will result in that the items appearing on the scene 2a on the conveyor belt will be sorted to desired fractions.

Figure 4 shows the principles of product detection and recognition. The image distributor 21 receives and image and distributes the image to a neural network object detection module 22, a logo detection module 23, and symbol detection module 24, and a text detection and text+font recognition module 25.

The information which is deduced from the neural network object detection module 22, the logo detection module 23, and the symbol detection module 24 are send to the recognition module 4a for further processing.

The information from the text detection and text+font recognition module 25 is further processed in the vendor name recognition module 26, the brand name recognition module 27, the product name recognition module28, the slogan recognition module 29, and product description recognition module 30, before the information is send to the product recognition module 4a for further processing.

The product recognition module 4a is integrated in the product detection and recognition module 4.

Figure 5 is illustrates a method for logo and symbol detection as shown in figure 4.

In the logo detection module and symbol detection module the overall detection principles are generally the same. When the modules receive an image from the image distributor, the image is first processed in a feature extraction module 40, extracting local features. The information is sent to a feature description module 41 which describes the local features and send the information to a matching module 42. The matching module 42 interacts with a feature descriptor database 44 which can provide further information about the features. From the matching module 42 matched local feature descriptors are send to a clustering module 43, before the information is provided to the product recognition module for further processing.

Figure 6 illustrates in more details the principles of text detection and recognition carried out in the text detection and text+font recognition module 25.

When the text detection and text+font recognition module receive an image from the image distributor, the image is first processed in a convolutional neural network 50 which send a compressed image representation to a text detection module 25a which again sends text boxes to a text recognition module 25b and font recognition module 25c. The text recognition module 25b and the font recognition module 25c provides information about text and font to the modules 26 - 30 in figure 4. After processing in the modules 26 - 30, text information is provided to the product recognition module.

During the processing of the image, the convolutional neural network 50, the text detection module 25a, and the text recognition module 25b interact with a images and annotations database 51. The images and annotations database 51 is a training database which supports the image the convolutional neural network 50. Neural network parameters are learned in the training phase from images and annotations. It is the learned model that is extracted from the images and annotations which is interacted with during operation/processing.

Figure 7 illustrates the general principles of neural network object detection. The image is send to the convolutional neural network 50 for processing and the convolutional neural network 50 sends compressed image representation to an object detection module 52 which detects the objects.

During the process the convolutional neural network 50 and the object detection module 52 interact with the images and annotations database 51. Neural network parameters are learned in the training phase from images and annotations. It is the learned model that is extracted from the images and annotations which is interacted with during operation/processing.

Figure 8 illustrates the general principles of two-stage neural network object detection.

An image is distributed from the image distributor module 21. The image is sent to the convolutional neural network 50 and the object recognition module 53. The convolutional neural network 50 sends compressed image representation to the object detection module 52 which detects the objects and sends the information to the object recognition module 53, which recognize the objects.

The convolutional neural network 50, the object detection module 52, and the object recognition module 53 interact with the images and annotations database 51 during the detection and recognition process. The neural network parameters are learned in the training phase from images and annotations. It is the learned model that is extracted from the images and annotations which is interacted with during operation/processing.

Figure 9 illustrates an embodiment where an image with high resolution is linked to a neural network for object detection. The architecture of the network is adapted to the high resolution in the images by neural network layers 50a, 50b and 50 c in the beginning of the network. The embodiment corresponds to the embodiment shown in figure 7, but adapted for images with high resolution.

Figure 10 illustrates examples of symbols which can be detected by the method according to the invention.