(19)
(11)EP 3 517 997 B1

(12)EUROPEAN PATENT SPECIFICATION

(45)Mention of the grant of the patent:
29.07.2020 Bulletin 2020/31

(21)Application number: 18164016.0

(22)Date of filing:  26.03.2018
(51)International Patent Classification (IPC): 
G01S 17/89(2020.01)
G01S 7/48(2006.01)
G06T 7/55(2017.01)
G01S 17/931(2020.01)
G06K 9/00(2006.01)
G01S 17/86(2020.01)

(54)

METHOD AND SYSTEM FOR DETECTING OBSTACLES BY AUTONOMOUS VEHICLES IN REAL-TIME

VERFAHREN UND SYSTEM ZUR ERKENNUNG VON HINDERNISSEN DURCH AUTONOME FAHRZEUGE IN ECHTZEIT

PROCÉDÉ ET SYSTÈME DE DÉTECTION D'OBSTACLES PAR DES VÉHICULES AUTONOMES EN TEMPS RÉEL


(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

(30)Priority: 30.01.2018 IN 201841003468

(43)Date of publication of application:
31.07.2019 Bulletin 2019/31

(73)Proprietor: Wipro Limited
560 035 Karnataka (IN)

(72)Inventors:
  • MEGANATHAN, Ranjith Rochan
    643001 The Nilgiris (IN)
  • JAGANNATH, Sujatha
    13 Bangalore (IN)

(74)Representative: Finnegan Europe LLP 
1 London Bridge
London SE1 9BG
London SE1 9BG (GB)


(56)References cited: : 
US-A1- 2017 104 980
US-A1- 2018 018 528
  
      
    Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention).


    Description

    TECHNICAL FIELD



    [0001] The present disclosure relates to autonomous vehicles. Particularly, but not exclusively, the present disclosure relates to a method and a system for detecting obstacles around an autonomous vehicle in real-time.

    BACKGROUND



    [0002] An autonomous vehicle (driverless vehicle) uses various sensors to navigate through a path. Also, various techniques are used to detect obstacles in the surroundings of the autonomous vehicle. The autonomous vehicle senses the environment with the help of sensors such as laser, Light Detection and Ranging (LIDAR), computer vision, camera and the like. The autonomous vehicle has a central control unit that helps the autonomous vehicle to traverse a path to reach a destination location from a source location. Also, the central control unit helps the autonomous vehicle to manoeuvre obstacles detected in the path. In a few scenarios, when the autonomous vehicle is moving at relatively high speeds, there may be difficulty in detecting the obstacles. In such scenarios the detection of obstacles and the decision of manoeuvring the obstacles has to be expedited in order to prevent accidents. Some existing autonomous vehicles use only LIDAR sensors for obstacle detection. Other existing autonomous vehicles use camera imaging for obstacle detection. In another scenario, when the obstacles are at a far distance from the autonomous vehicle, the data obtained from the LIDAR sensors may not be accurate, thereby causing difficulty in the detection of obstacles. In an existing autonomous vehicle, when using a camera to detect obstacles present around the autonomous vehicle, the control unit undergoes a rigorous training, with various training datasets. Thereby, while processing the images captured by the camera, in real-time, the computation time required to identify the obstacles increases. Thus, the existing autonomous vehicle consumes more time to provide results. The existing autonomous vehicle makes use of several frames of images captured by the camera of the autonomous vehicle. Thus, each of the several frames of images is processed to detect the obstacles, thereby increasing the computation time. Thus, existing autonomous vehicles do not provide an efficient solution to address the above-mentioned problems.

    [0003] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

    SUMMARY



    [0004] In accordance with the aspects of the invention, there are provided: a method of detecting obstacles by a vehicle in real-time, an obstacle detection system for detecting obstacles by a vehicle in real-time, and a non-transitory computer readable medium for detecting obstacles by a vehicle in real-time, as set forth in claims 1, 5, and 9, respectively.

    [0005] In an embodiment, the present disclosure relates to a method for detecting obstacles by a vehicle in real-time. The method comprises receiving, by an Electronic Control Unit (ECU) of a vehicle, one or more images of surroundings of the vehicle from an imaging unit and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging (LIDAR) unit. The method further comprise generating a disparity image by analysing the one or more images, extracting one or more regions comprising one or more obstacles, from the disparity image, determining boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image, extracting point cloud data points from the point cloud data based on the image co-ordinates of the boundary points corresponding to each of the one or more obstacles and determining a gradient for each of the point cloud data points to determine a structural orientation of each of the one or more obstacles, thereby detecting each of the one or more obstacles.

    [0006] In an embodiment, the present disclosure related to an obstacle detection system for detecting obstacles by a vehicle in real-time. The obstacle detection system comprises an Electronic Control Unit (ECU). The ECU is configured to receive a pair of images of surroundings of the vehicle from an imaging unit and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging (LIDAR) unit. The ECU is further configured to generate a disparity image by analysing the pair of images, extract one or more regions comprising one or more obstacles, from the disparity image, determine boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image, extract point cloud data points from the point cloud data based on the image co-ordinates of the boundary points corresponding to each of the one or more obstacles and determine a gradient for each of the point cloud data points to determine a structural orientation of each of the one or more obstacles, thereby detecting each of the one or more obstacles.

    [0007] In an embodiment, the present disclosure relates to a non-transitory computer readable medium including instruction stored thereon that when processed by at least one processor cause an obstacle detection system to receive an application. The instruction may cause the processor to receive one or more images of surroundings of the vehicle from an imaging unit and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging (LIDAR) unit. The instruction may cause the processor to generate a disparity image by analysing the one or more images, extract one or more regions comprising one or more obstacles, from the disparity image, determine boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image. The instruction may cause the processor to extract point cloud data points from the point cloud data based on the image co-ordinates of the boundary points corresponding to the one or more obstacles and determine a gradient for each of the point cloud data points to determine a structural orientation of the one or more obstacles, thereby detecting the one or more obstacles.

    [0008] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

    BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS



    [0009] The novel features and characteristic of the disclosure are set forth in the appended claims. The disclosure itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying figures. One or more embodiments are now described, by way of example only, with reference to the accompanying figures wherein like reference numerals represent like elements and in which:

    Figure 1 is illustrative of an exemplary block diagram of an obstacle detection system for detecting obstacles by a vehicle, in accordance with some embodiments of the present disclosure;

    Figure 2 illustrates a block diagram of internal architecture of an Electronic Control Unit (ECU) configured to detect obstacles in the surrounding of a vehicle, in accordance with some embodiments of the present disclosure;

    Figure 3 shows an exemplary flow chart illustrating method steps for detecting obstacles by a vehicle, in accordance with some embodiments of the present disclosure;

    Figure 4 shows an exemplary flow chart illustrating method steps for forming a gradient to determine a structural orientation of obstacles, in accordance with some embodiments of the present disclosure;

    Figure 5 shows an exemplary flow chart illustrating method steps for detecting obstacles using enhanced point cloud data, in accordance with some embodiments of the present disclosure;

    Figure 6A shows a disparity image, generated by the ECU of an obstacle detection system 100, in accordance with some embodiments of the present disclosure;

    Figure 6B shows a LIDAR map of surroundings of a vehicle for detecting obstacles around the vehicle, in accordance with some embodiments of the present disclosure;

    Figure 7A is indicative of an obstacle region extracted from a disparity image, in accordance with some embodiments of the present disclosure;

    Figure 7B is indicative of boundary point of obstacles present in the disparity image, in accordance with some embodiments of the present disclosure;

    Figure 8 is indicative of point cloud data points extracted using boundary points of obstacles, in accordance with some embodiments of the present disclosure;

    Figures 9A, Figure 9B, Figure 9C, Figure 9D and Figure 9E are indicative of gradient formation for point cloud data points corresponding to obstacles, in accordance with some embodiments of the present disclosure;

    Figure 10 illustrates pixel to point cloud mapping, in accordance with some embodiments of the present disclosure; and

    Figure 11 is indicative of formation of 2D virtual grids on the LIDAR map, for detecting the obstacles, in accordance with some embodiments of the present disclosure.



    [0010] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.

    DETAILED DESCRIPTION



    [0011] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

    [0012] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below.

    [0013] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus proceeded by "comprises... a" does not, without more constraints, preclude the existence of other elements or additional elements in the system or apparatus.

    [0014] Embodiments of the present disclosure relate to detection of obstacles by an autonomous vehicle in real-time. An obstacle detection system of an autonomous vehicle obtains point cloud data from single frame Light Detection and Ranging (LIDAR) data and camera data, of the surroundings of the vehicle. The point cloud data of the surroundings comprises a plurality of 3-Dimensional (3D) data points defined by corresponding cartesian co-ordinates. Further, the system processes the camera data to identify and extract regions comprising the obstacles. Further, the system extracts point cloud data corresponding to the obstacles and enhances the point cloud data, with the detailed information of the obstacles provided by the camera data. Further, the system processes the enhanced point cloud data to determine the obstacles along with the structure and orientation of obstacles. Upon determining the details of the obstacles, the system provides instructions to the vehicle to manoeuvre the obstacle. The disclosed obstacle detection system provides accurate data about a structure, an orientation and a location of the obstacles.

    [0015] Figure 1 is illustrative of an exemplary block diagram of an obstacle detection system 100 for detecting obstacles by a vehicle, in accordance with some embodiments of the present disclosure. The obstacle detection system 100 comprises an imaging unit 101, a LIDAR unit 102, and an Electronic Control Unit (ECU) 103. In an embodiment, the vehicle may be, but not limited to, an autonomous vehicle and semi-autonomous vehicle. The ECU 103 may receive an information regarding the obstacles present in the surroundings of the vehicle from the imaging unit 101 and the LIDAR unit 102. The imaging unit 101 captures one or more images of the surrounding of the vehicle and provides the captured one or more images to the ECU 103 for processing. In an embodiment, let us consider that the imaging unit 101 is a stereo camera. The stereo camera may comprise of two monocular cameras separated by a baseline distance. The baseline distance is the distance between a centre of focus of the two monocular cameras. The stereo camera captures two images, a first image and a second image of the surrounding of the vehicle from two different views. Each of the monocular camera captures an image, thereby constituting a pair of images. The two images are captured at a same time interval, such that each of the monocular camera has a common focus. The two images captured by the stereo camera may be used by the ECU 103 to produce a disparity image. The ECU 103 may map every pixel in the first image to a corresponding pixel in the second image, and compute a disparity value. The disparity value indicates a pixel difference between the two images to produce a disparity image. Each pixel in the disparity image also comprises the disparity value. The disparity image may be used for detecting the one or more obstacles.

    [0016] In an embodiment, the LIDAR unit 102 comprises a LIDAR sensor. The LIDAR unit 102 produces a LIDAR map comprising elevation of every object in the surrounding of the vehicle. The data related to every object in the surrounding of the vehicle is called as point cloud data. The point cloud data may comprise several point cloud data points with each of the point cloud data points defined by a corresponding 3D cartesian co-ordinate. The point cloud data points may indicate the elevation of every object present in the surroundings of a vehicle. The point cloud data is received by the ECU 103 from the LIDAR unit 102.

    [0017] The ECU 103 may extract one or more regions comprising the one or more obstacles from the disparity image. In an embodiment, each region of the one or more regions may comprise at least one obstacle. The ECU 103 may further determine boundary points of each of the one or more obstacles present in respective one or more regions. Further, image co-ordinates of the boundary points of each of the one or more obstacles is determined by the ECU 103. The image co-ordinates of the boundary points of each of the one or more obstacles may be converted to 3D co-ordinates. Then, the ECU 103 extracts the point cloud data points corresponding to 3D co-ordinates of the one or more obstacles from the point cloud data. The extracted point cloud data points represent the one or more obstacles present in the one or more regions in the disparity image.

    [0018] In an embodiment, when the one or more obstacles in the surroundings of the vehicle are present closer to vehicle, the point cloud data points corresponding to the one or more obstacles provide accurate depth information of the one or more obstacles. The depth information defines the 3D data of the point cloud data points corresponding to the one or more obstacles. In an embodiment, the point cloud data points may be densely populated in the LIDAR map when the one or more obstacles are closer the vehicle. When the one or more obstacles are far from the vehicle, the point cloud data points may not be accurate, thereby the one or more obstacles may not be identified easily. For the scenario mentioned above, the one or more images captured by the imaging unit 101 are used to determine the one or more obstacles. The one or more images may comprise a plurality of pixels representing every object in the surrounding. The plurality of pixels may be dense (having pixel wise information on each of the one or more obstacles). Thus, the one or more obstacles may be identified easily.

    [0019] Further, the ECU 103 determines a structural orientation of each of the one or more obstacles based on the corresponding extracted point cloud data points. Processing the extracted point cloud data points may result in detection of the one or more obstacles present in the surroundings of the vehicle with accurate location, structure and orientation of the one or more obstacles. In an embodiment, the ECU 103 may communicate with a navigation unit 104. The navigation unit 104 receives the information on the location, structure and orientation of each of the one or more obstacles, from the ECU 103 and enables the vehicle to manoeuvre each of the one or more obstacles present in the surroundings of the vehicle.

    [0020] In an embodiment, the imaging unit 101 may be, but is not limited to, a low-resolution camera, an Infrared (IR) camera, a stereo camera, a Digital Single-Lens Reflex (DSLR) camera and the like. In an embodiment, the stereo camera comprises two monocular cameras. The two monocular cameras are placed on the vehicle such that the two monocular cameras are on a same plane and are parallel to a reference plane, for example the ground. An optical axis of a first monocular camera may be parallel to an optical axis of a second monocular camera.

    [0021] In an embodiment, the ECU 103 may be integrated with variety of sensors used for sensing the environment and surroundings of the vehicle. Various techniques, employing the variety of sensors may include, but are not limited to, Radio Detection and Ranging (RADAR), odometry, computer vision and the like. The variety of sensors may include, but are not limited to, an ultrasound sensor, a proximity sensor and the like.

    [0022] Figure 2 illustrates a block diagram of internal architecture of ECU 103 configured to detect obstacles in the surrounding of a vehicle, in accordance with some embodiments of the present disclosure. The ECU 103 may include at least one Central Processing Unit ("CPU" or "processor") 203 and a memory 202 storing instructions executable by the at least one processor 203. The processor 203 may comprise at least one data processor for executing program components for executing user or system-generated requests. The memory 202 is communicatively coupled to the processor 203. The ECU 103 further comprises an Input/ Output (I/O) interface 201. The I/O interface 201 is coupled with the processor 203 through which an input signal or/and an output signal is communicated.

    [0023] In an embodiment, data 204 may be stored within the memory 202. The data 204 may include, for example, image data 205, LIDAR data 206, disparity image data 207, threshold data 208 and other data 209.

    [0024] In an embodiment, the image data 205 may include, but is not limited to, information on the one or more images captured by the imaging unit 101. The image data 205 may store each of the one or more images captured by the imaging unit 101 in the memory 202.

    [0025] In an embodiment, the LIDAR data 206 comprises the point cloud data. The point cloud data may comprise several point cloud data points with each of the point cloud data point defined by a corresponding 3D co-ordinate (x, y and z co-ordinate). The point cloud data points may indicate the elevation of every object present in the surroundings of the vehicle.

    [0026] In an embodiment, the disparity image data 207 may store the disparity image generated by the ECU 103. The disparity image data 207 may also store the image co-ordinates of the boundary points of each of the one or more obstacles extracted from the disparity image.

    [0027] In an embodiment, the threshold data 208 may include information about the different thresholds used in the process of detecting each of the one or more obstacles in the surroundings of a vehicle. The threshold data 208 may include a first threshold, a second threshold, a distance error threshold and a height threshold. In an embodiment, the first threshold indicates a maximum limit on a radius of a virtual hemisphere, projected during formation of a gradient. The second threshold indicates a maximum limit on a number of point cloud data points that may lie within the virtual hemisphere. Details on formation of gradient may be found in the description of method steps 300 and method steps 400 as illustrated in Figure 3 and Figure 4 respectively. The distance error threshold value indicates a maximum limit for a distance error between pixels of the disparity image. The height threshold indicates a maximum limit on a height difference computed between at least two enhanced point cloud data points in each of one or more 2D virtual grids.

    [0028] In an embodiment, the other data 209 may include, but is not limited to, obstacles data, traffic information, terrain information and the like.

    [0029] In an embodiment, the data 204 in the memory 202 is processed by modules 210 of the ECU 103. As used herein, the term module refers to an Application Specific Integrated Circuit (ASIC), an electronic circuit, a Field-Programmable Gate Arrays (FPGA), Programmable System-on-Chip (PSoC), a combinational logic circuit, and/or other suitable components that provide the described functionality. The modules 210 when configured with the functionality defined in the present disclosure will result in a novel hardware.

    [0030] In one implementation, the modules 210 may include, for example, a communication module 211, a disparity image generation module 212, an obstacle region extraction module 213, a point cloud extraction module 214, a gradient determination module 215, a pixel-point cloud mapping module 216, an enhanced point cloud data processing module 217 and other modules 218. It will be appreciated that such aforementioned modules 210 may be represented as a single module or a combination of different modules. Further, person of ordinary skill in the art will appreciate that in an implementation, the modules 210 may be integrated with the memory 202.

    [0031] The communication module 211 facilitates communication between the ECU 103 and other units (not shown in figures) of the vehicle. In an embodiment, the other units may comprise, but are not limited to, the imaging unit 101, the LIDAR unit 102 and the navigation unit 104. For example, the communication module 211 may receive the one or more images captured by the imaging unit 101 and may also receive the point cloud data produced by the LIDAR unit 102. In an embodiment, the communication module 211 may help in communication between the other units.

    [0032] In an embodiment, the disparity image generation module 212 generates a disparity image. The disparity image generation module 212 may receive a pair of images captured by the imaging unit 101. In an exemplary embodiment, the pair of images may be captured using the stereo camera. Each of the two monocular cameras separated by the baseline distance and having same focus, captures an image. Further, the pair of images represent a same scene captured from two different views at a same time. The pair of images may be represented as a first image and a second image. The disparity image generation module 212 may match at least one pixel in the first image to a corresponding at least one pixel in the second image, and determine co-ordinates of each pixel in each of the first and the second image. Further, the disparity image generation module 212 computes a disparity value (pixel difference) between co-ordinates of each of the at least one pixel in the first image and co-ordinates of each of the at least one pixel in the second image. Here, the distance between the pixels represent a distance between, an object in the first image and the object in the second image. Thereafter, a disparity image is generated based on the pixel differences between the first and the second image. Thus, each pixel in the disparity image represents the distance/disparity value. Further, the disparity image generation module 212 may employ any image processing techniques to generate the disparity image. The disparity image may be generated by taking blocks of similar width in each pair of images. In an embodiment, the disparity image generation module 212 may employ Sum of Squared Difference (SSD) technique for generating the disparity image.

    [0033] In an embodiment, the obstacle region extraction module 213 extracts one or more regions from the disparity image. Each of the one or more regions comprises an obstacle. The obstacle region extraction module 213 receives the disparity image from the disparity image generation module 212. Once the disparity image is generated, various image processing techniques may be employed to extract the one or more regions comprising the one or more obstacles. In an embodiment the obstacle region extraction module 213 may employ colour based segmentation to extract the one or more regions. The colour based segmentation is employed to separate each of the one or more obstacles at different depths. The colour based segmentation is based on a pixel/colour threshold value. If the pixel values are beyond the pixel/colour threshold value, the pixels are considered to indicate different contours. Thus, the disparity image is segmented into contours indicating the presence of the one or more obstacles. The segmented disparity image may be scanned horizontally from a top most pixel to a bottom most pixel to find boundaries of the contours. If the number of pixels exceeds a predefined pixel threshold, the horizontal line of pixels is considered as the boundary pixels of an identified obstacle. Hence, boundary points of the identified obstacle are extracted, and the boundary points provide a position of the obstacle in terms of image co-ordinates in the disparity image. Similarly, the boundary points are determined for the one or more obstacles.

    [0034] In an embodiment, the point cloud extraction module 214 retrieves the point cloud data from the LIDAR data 206 and further receives the boundary points of the one or more obstacles from the obstacle region extraction module 213. Further, the point cloud extraction module 214 extracts the point cloud data points from the point cloud data based on the image co-ordinates of the boundary points of each of the one or more obstacles. Further, the image coordinates of the boundary points of each of the one or more obstacles are converted into spherical co-ordinates. The 3D co-ordinates of the point cloud data points are also converted into spherical co-ordinates. In an embodiment, the image co-ordinates and the point cloud data may be converted to any common co-ordinate system. The spherical co-ordinates of the boundary points of each of the one or more obstacles may be used for extracting the point cloud data points from the point cloud data, lying within the boundaries of each of the one or more obstacles.

    [0035] In an embodiment, the placement of LIDAR unit 102 and the imaging unit 101 may be important for mapping the image co-ordinates with spherical co-ordinates. The scene viewed by the LIDAR unit 102 may be similar to the scene viewed by the imaging unit 101, therefore the placement of LIDAR unit 102 and the imaging unit 101 plays an important role. For example, the LIDAR unit 102 may be placed at a centre of the baseline in which the first monocular camera and second monocular camera are placed.

    [0036] In an embodiment, the gradient determination module 215 is used to determine gradient for each of the point cloud data points extracted within the boundary points of each of the one or more obstacles. At least one point is chosen from the point cloud data points extracted within the boundary points of each of the one or more obstacles. The gradient determination module 215 projects a first virtual hemisphere and a second virtual hemisphere outwardly towards a top and a bottom from the at least one point. The projection of the first virtual hemisphere and the second virtual hemisphere is performed until one of a radius of virtual hemispheres is equal to the first threshold value and number of point cloud data points present in the virtual hemisphere is equal to the second threshold value. The gradient is determined for the at least one point by connecting the at least one point to each of the point cloud data points that lie within the first virtual hemisphere. Further, a resultant gradient is determined in the first virtual hemisphere based on the gradient corresponding to each of the point cloud data points in the first virtual hemisphere, with respect to the at least one point. Similarly, a gradient is determined in the second virtual hemisphere and a resultant gradient is determined in the second virtual hemisphere. The gradient determination module 215 accumulates the resultant gradient in the first virtual hemisphere and the second virtual hemisphere of each of the point cloud data points, lying within the boundary points of each of the one or more obstacles, for determining the structural orientation of each of the one or more obstacles.

    [0037] In an embodiment, the pixel- point cloud mapping module 216 maps each pixel within the boundary points of the identified obstacle in the disparity image to the gradient formed for each of the point cloud data points, lying within the boundary points of the identified obstacle. In an embodiment the image coordinate of each pixel is converted into the spherical co-ordinates. The spherical co-ordinates of each pixel is mapped to the spherical co-ordinates of the gradient formed for each of the point cloud data points. The gradient having the spherical co-ordinates same as the spherical co-ordinates of a pixel in the disparity image is chosen. The point cloud data corresponding to the point cloud data points, lying within the boundary points of each of the one or more obstacles is enhanced with the corresponding pixel data within the boundary points of the identified obstacle in the disparity image.

    [0038] In an embodiment, the enhanced point cloud data processing module 217 receives the enhanced point cloud data from the pixel-point cloud mapping module 216. The enhanced point cloud data processing module 217 processes the enhanced point cloud data to determine a structure, an orientation and a location of each of the one or more obstacles. The LIDAR map comprising the point cloud data is divided into plurality of two Dimensional (2D) virtual grids. Each of the plurality of 2D virtual grids comprises a plurality of enhanced point cloud data points. The enhanced point cloud data processing module 217 detects one or more 2D virtual grids from the plurality of 2D virtual grids representing the one or more obstacles. Detection of the one or more obstacles may be based on a height difference between at least two enhanced point cloud data points in each of the one or more 2D virtual grids, where the height difference exceeds a height threshold.

    [0039] In an embodiment, the other modules 218 may include, but is not limited to, an indication module to indicate the one or more obstacles detected by the ECU 103, etc. The indication module may display the one or more obstacles to a user. The indication module may be one of, but not limited to, a monitor, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display and/or any other module present which is capable of displaying an output. In an embodiment the indication module may be present in a user device.

    [0040] Figure 3 shows an exemplary flow chart illustrating method 300 steps for detecting obstacles by a vehicle, in accordance with some embodiments of the present disclosure.

    [0041] As illustrated in Figure 3, Figure 4 and Figure 5 the methods 300, method 400 and method 500 respectively may comprise one or more steps for detecting obstacles by an autonomous vehicle, in accordance with some embodiments of the present disclosure. The methods 300, 400 and 500 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform particular functions or implement particular abstract data types.

    [0042] The order in which the method 300, method 400 and method 500 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

    [0043] At step 301, the communication module 211 receives one or more images of surroundings of the vehicle from the imaging unit 101. The ECU 103 further receives the point cloud data of the surroundings of the vehicle from the LIDAR unit 102.

    [0044] At step 302, the disparity image generation module 212 receives one or more images of surroundings of the vehicle from the communication module 211. The disparity image generation module 212 may generate a disparity image based on the pair of images of the one or more images. In an embodiment, the pair of images are the images captured by the imaging unit 101 comprising two monocular cameras separated by the baseline distance. Further, the pair of images are of a same scene captured from two different views at a same time. The pair of images may be represented as a first image and a second image.

    [0045] The disparity image generation module 212 may map every pixel in the first image, to a corresponding pixel in the second image and compute a disparity value. The disparity value indicates a pixel difference between the two images to produce a disparity image. Each pixel in the disparity image also comprises the disparity value. The disparity image may be used for detecting the one or more obstacles. Further, the disparity image generation module 212 may employ any image processing technique to generate the disparity image. The disparity image may be generated by taking blocks of similar width in each of the pair of images. In an embodiment, the disparity image generation module 212 may employ Sum of Squared Difference (SSD) technique for generation of disparity image.

    [0046] Considering an example for disparity image generation. Let (a, b) be the co-ordinate of a pixel in image A, and the co-ordinate of a corresponding pixel in image B be (c, d). The disparity value is calculated as (a-c) as the value of (b-d) may be zero or a negligible value. and a similar loop is carried out for all the co-ordinates in the pair of images. To compare image A with image B, the co-ordinate of corresponding pixels in image A and image B are compared. The two images in comparison are of the same size, pixel-wise. By generating the disparity image, objects present at different depths in the pair of images may be separated.

    [0047] Figure 6A shows a disparity image 601, generated by the ECU 103 of an obstacle detection system 100, in accordance with some embodiments of the present disclosure. The disparity image 601 as illustrated in Figure 6A may be generated by the disparity image generation module 212. The disparity image 601 is generated based on the pair of images of surroundings of the vehicle, captured by the imaging unit 101. The surroundings of the vehicle may comprise one or more obstacles.

    [0048] Figure 6B shows a LIDAR map indicative of a point cloud data 602 of the surroundings for detecting obstacles around the vehicle, in accordance with some embodiments of the present disclosure. The surroundings illustrated in Figure 6B is same as the surroundings shown in Figure 6A. The point cloud data of the surroundings comprises several point cloud data points 603a, 603b, 603c......., 603n, indicating one or more objects in the surroundings as illustrated in Figure 6B.

    [0049] Referring back to Figure 3, at step 303, the obstacle region extraction module 213 may extract one or more regions in the disparity image. Each of the one or more regions comprises an obstacle. The obstacle region extraction module 213 receives the disparity image from the disparity image generation module 212, stored as the disparity image data 207. Once the disparity map is generated, various image processing techniques are employed to extract the one or more regions comprising the one or more obstacles.

    [0050] In an embodiment the obstacle region extraction module 213 may employ colour based segmentation to extract the one or more regions from the disparity image. The colour based segmentation is employed to separate each of the one or more obstacles at different depths. The colour based segmentation is based on a pixel/colour threshold value. If the pixel values are beyond the pixel/colour threshold value, the pixels are considered to indicate different contours. Thus, the disparity image is segmented into contours indicating the presence of the one or more obstacles. The segmented disparity image may be scanned horizontally from a top most pixel to a bottom most pixel to find boundaries of the contours. If the number of pixels exceeds a predefined pixel threshold, the horizontal line of pixels is considered as the boundary pixels of an identified obstacle. Hence, boundary points of the identified obstacle are extracted, and the boundary points provides a position of the obstacle in terms of image co-ordinates in the disparity image. Similarly, the boundary points are determined for the one or more obstacles.

    [0051] Figure 7A is indicative of an obstacle region extracted from the disparity image 601, in accordance with some embodiments of the present disclosure. Figure 7A comprises an extracted region having an obstacle 701, illustrated in bold dotted line. The extracted region having the obstacle 701 is present in the disparity image 601. Various image processing techniques may be employed by the obstacle region extraction module 213 for determining the extracted region having the obstacle 701.

    [0052] Referring back to Figure 3, at step 304, boundary points of the one or more obstacles are determined. The boundary points of the one or more obstacles provides the position of the corresponding obstacle in terms of image co-ordinates in the disparity image.

    [0053] Figure 7B is indicative of boundary point of obstacles present in the disparity image 601, in accordance with some embodiments of the present disclosure. As illustrated in Figure 7B, boundary points along the corners of the obstacle 701 are determined based on the extracted region (as illustrated in Figure 6A). The boundary points of the obstacle 701 are indicated in bold dots along the corners of the obstacle 701, in Figure 7B. The boundary points of the obstacle 701 are identified by the image co-ordinates in the disparity image 601.

    [0054] Referring back to Figure 3, at step 305, the point cloud extraction module 214 may retrieve the point cloud data from the LIDAR data 206. Further, the point cloud extraction module 214 extracts the point cloud data points from the point cloud data based on the image co-ordinates of the boundary points of each of the one or more obstacles. The point cloud extraction module 214 receives the point cloud data from the LIDAR unit 102. The image coordinates of the boundary points of each of the one or more obstacles are converted into spherical co-ordinates. The conversion of image co-ordinates into spherical co-ordinates (θi and ϕi) may be computed using equations 1-4 mentioned below.







    where,

    PW is pixel per degree of width,

    PH is pixel per degree of height,

    θi is elevation angle,

    ϕi is azimuth angle,

    W is width of the disparity image in pixels,

    H is height of the disparity image in pixels,

    FOV (w) is Field of View along the width of disparity image,

    FOV (h) =Field of View along the width of disparity image,

    Cw= center pixel with respect to width of disparity image,

    Ch= center pixel with respect to height of disparity image.

    X, Y are image coordinates.



    [0055] Further, the 3D co-ordinates (x, y, z) of the point cloud data points are also converted into spherical co-ordinates (r, θp, ϕp) using equations 5-7 mentioned below.





    where,

    r is radial distance,

    θp is the elevation angle,

    ϕp is the azimuth angle.



    [0056] θi and ϕi define the elevation angle and azimuth angle of each of the boundary points (determined form the disparity image) of each of the one or more obstacles, respectively. r, θp and ϕp define the radial distance, elevation angle and azimuth angle of each of the point cloud data points. The point cloud data points that lie at the same elevation and azimuth angle as that of the boundary points of each of the one or more obstacles are extracted. Further, using the extracted point cloud data points all the point cloud data points that lie within the boundary points are extracted. Thus, each of the one or more obstacles are now defined by the point cloud data points that lie within the boundary points of the corresponding obstacle, extracted from the point cloud data.

    [0057] The spherical co-ordinates of the boundary points of each of the one or more obstacles may be used for extracting the point cloud data points from the point cloud data, lying within the boundary points of each of the one or more obstacles.

    [0058] In an embodiment, the image co-ordinates of the boundary points of each of the one or more obstacle and the 3D co-ordinates of the point cloud data points may be converted to a common co-ordinate system and compared, for the extraction of point cloud data points.

    [0059] In an embodiment, the placement of LIDAR unit 102 and the imaging unit 101 may be important for mapping the image co-ordinates with spherical co-ordinates. The scene viewed by the LIDAR unit 102 may be similar to the scene viewed by the imaging unit 101, therefore the placement of LIDAR unit 102 and the imaging unit 101 plays an important role. For example, the LIDAR unit 102 may be placed at a centre of the baseline in which the first monocular camera and second monocular camera are placed.

    [0060] Figure 8 is indicative of point cloud data points extracted using boundary points of obstacles, in accordance with some embodiments of the present disclosure. The point cloud data points 800 pertaining to the obstacle 701 are extracted from the point cloud data 602 (illustrated in Figure 6B) of the surroundings of the vehicle. The extracted point cloud data points are illustrated in Figure 8. The point cloud data points 800 pertaining to the obstacle 701 are extracted based on the image co-ordinates of the boundary points of the obstacle 701. The extraction of point cloud data points is as per the steps defined in the method step 305.

    [0061] Referring back to Figure 3, at step 306, the gradient determination module 215 may determine a gradient for each of the point cloud data points extracted within the boundary points of the one or more obstacles. Thus, the structural orientation of each of the one or more obstacles is determined, thereby detecting each of the one or more obstacles.

    [0062] Figure 4 shows an exemplary flow chart illustrating method steps 400 for forming a gradient to determine a structural orientation of obstacles, in accordance with some embodiments of the present disclosure.

    [0063] At step 401, the gradient determination module 215 is used to determine gradient for each of the point cloud data points extracted within the boundary points of the one or more obstacles. At least one point is chosen from the point cloud data points extracted within the boundary points of the one or more obstacles.

    [0064] Figures 9A, Figure 9B, Figure 9C, Figure 9D and Figure 9E are indicative of gradient formation for point cloud data points corresponding to obstacles, in accordance with some embodiments of the present disclosure. As illustrated in Figure 9A at least one point cloud data point 900 is chosen from the point cloud data points 800 pertaining to the obstacle 701.

    [0065] Referring back to Figure 4, at step 402, the gradient determination module 215 projects a first virtual hemisphere and a second virtual hemisphere outwardly towards the top and bottom of the chosen at least one point. The first virtual hemisphere and the second virtual hemisphere is projected with the chosen at least one point as a centre of the first virtual hemisphere and the second virtual hemisphere respectively. The first virtual hemisphere may be projected outwardly towards a top direction and the second virtual hemisphere may be projected outwardly towards a bottom direction of the chosen at least one point. The projection of the first virtual hemisphere and the second virtual hemisphere is performed until one of a radius of virtual hemispheres is equal to the first threshold value and number of point cloud data points present in the virtual hemisphere is equal to the second threshold value.

    [0066] As illustrated in Figure 9B, a first virtual hemisphere 901 is projected outwardly towards a top direction of the at least one point cloud data point 900 as a centre. Radius of the first virtual hemisphere 901 is within the first threshold value and also none of the point cloud data points lie within the first virtual hemisphere as illustrated by the Figure 9B. Thus, the first virtual hemisphere 901 is further projected in the top direction as illustrated in Figure 9C. The radius of the first virtual hemisphere 901 may now be equal to the first threshold value or the number of point cloud data points that lie within the first virtual hemisphere 901 may be equal to the second threshold value. Thus, the projection of the first virtual hemisphere 901 may be stopped. Similarly, a second virtual hemisphere (not shown) may be projected towards a bottom direction of the at least one point cloud data point 900 as a centre.

    [0067] Referring back to Figure 4, at step 403 and step 404, the gradient determination module 215 determines gradient for the chosen at least one point by connecting the at least one point to each of the point cloud data points that lie within the first virtual hemisphere, for forming a gradient corresponding to each of the point cloud data points in the first virtual hemisphere. Further, at step 404 the gradient determination module 215 determines gradient for the at least one point by connecting the at least one point to each of the point cloud data points that lie within the second virtual hemisphere, for forming a gradient corresponding to each of the point cloud data points in the second virtual hemisphere.

    [0068] As illustrated in Figure 9D, the at least one point cloud data point 900 is connected to three point cloud data points that lie within the first virtual hemisphere 901, via a first gradient, a second gradient and a third gradient. Similarly, gradients may be formed in the second virtual hemisphere (not shown) by connecting the at least one point cloud data point 900 to the point cloud data points that lie within the second virtual hemisphere.

    [0069] Referring back to Figure 4, at step 405, the gradient determination module 215 may determine a resultant gradient in the first virtual hemisphere and the second virtual hemisphere based on the gradient corresponding to each of the point cloud data points in the first virtual hemisphere and the second virtual hemisphere, with respect to the at least one point. The resultant gradient determined for a selected point defines the orientation of the selected point.

    [0070] As illustrated in Figure 9E, a resultant gradient 902 is determined for the first gradient, the second gradient and the third gradient present in the first virtual hemisphere 901. The resultant gradient 902 provides the orientation of the point cloud data point 800. Similarly, a resultant gradient is determined in the second virtual hemisphere (not shown).

    [0071] Referring back to Figure 4, at step 406, the resultant gradient in the first virtual hemisphere and the second virtual hemisphere, of each of the point cloud data points, are accumulated. The gradient determination module 215 accumulates the resultant gradient in the first virtual hemisphere and the second virtual hemisphere of each of the point cloud data points, lying within the boundary points of the one or more obstacles for determining the structural orientation of the one or more obstacles. The resultant gradient determined for a selected point provides the orientation of the selected point. Thus, by determining a resultant gradient for each of the point cloud data points that lie within the one or more obstacles, the structural orientation of the one or more obstacles may be determined.

    [0072] Figure 5 shows an exemplary flow chart illustrating method steps 500 for detecting obstacles using enhanced point cloud data, in accordance with some embodiments of the present disclosure.

    [0073] At step 501, the pixel- point cloud mapping module 216 maps each pixel within the boundary points of the identified obstacle in the disparity image to gradient formed for each of the point cloud data points, lying within the boundary points of the identified obstacle. In an embodiment the image coordinates of each pixel are converted into the spherical co-ordinates (θi and ϕi). The spherical co-ordinates of each pixel are mapped to the spherical co-ordinates of the gradient formed for each of the point cloud data points. The gradient whose direction is along the spherical co-ordinates of a given pixel is chosen. The disparity image may not have accurate data about the distance of the one or more obstacles. While mapping each pixel in the disparity image to a corresponding gradient, an error is computed at which the disparity map perceives the object. The distance error is computed for each pixel using equation as mentioned below.



    [0074] If the distance error for a given pixel is within the distance error threshold (when mapped with a selected gradient), the given pixel is mapped to the selected gradient. Similarly, each pixel is mapped to a gradient. If the distance error for a given pixel is greater than the distance error threshold (when mapped with a selected gradient), an implication may be provided that that there may be a projection in the obstacle which is not detected by the LIDAR unit 102. Such pixels are mapped with the help of a stored error value rather than gradients. Thus, the point cloud data corresponding to the point cloud data points, lying within the boundary points of each of the one or more obstacles is enhanced with the corresponding pixel data within the boundary points of the identified obstacle in the disparity image. Thus, producing enhanced point cloud data points for each of the one or more obstacles.

    [0075] Figure 10 illustrates the pixel to point cloud mapping, in accordance with some embodiments of the present disclosure. As illustrated in Figure 9, each pixel present within the boundary point (as illustrated in Figure 6B) of the obstacle 701 are mapped to a corresponding resultant gradient formed for each of the point cloud data points 800 pertaining to the obstacle 701. Thus, the mapping results in enhancing the point cloud data points 800.

    [0076] Referring back to Figure 5 at step 502, the enhanced point cloud data processing module 217 divides the LIDAR map of the surroundings into plurality of two Dimensional (2D) virtual grids, wherein each of the 2D virtual grid comprises a plurality of enhanced point cloud data points.

    [0077] Figure 11 is indicative of formation of 2D virtual grids on the LIDAR map, for detecting the obstacles, in accordance with some embodiments of the present disclosure. The LIDAR map comprises the point cloud data 602. The point cloud data 602 of the surroundings with enhanced point cloud data points 800, corresponding to the obstacle 701 is divided into plurality of 2 dimensional (2D) virtual grids. The plurality of 2D virtual grids are present in a virtual rectangular bar defined by the limits x (limit 1) and x (limit 2) along x-axis and y (limit 1) and y (limit 2) along the y-axis. The limits along the x-axis and the y-axis are decided based on the range of the LIDAR unit 102. The enhanced point cloud data points 800 are present within at least one of the plurality of virtual grids. Each of the plurality of the virtual grids are defined by a grid index. The enhanced point cloud data points that lie within a virtual grid are saved into the corresponding virtual grid bearing a grid index.

    [0078] Referring back to Figure 5, at step 503, a first point and a second point are determined in each of the 2D virtual grids. The first point may be a maximum height valued point and the second point may be a minimum height valued point. The enhanced point cloud data processing module 217 may compute the height difference between the first point and the second point. The grid where the height difference exceeds the height threshold is marked as an obstacle grid. The above step is repeated for each of the plurality of 2D virtual grids. Thus, height of each of the one or more obstacles is determined. Each of the 2D virtual grids marked as an obstacle grid are grouped together to locate the obstacle with accurate location.

    [0079] In an embodiment, the gradient and the virtual grid may be used together to determine structure, orientation and height of the one or more obstacles.

    [0080] In an embodiment, the method steps 300, method steps 400 and method steps 500 are performed in real-time.

    [0081] In an embodiment, the present disclosure discloses a method and an obstacle detection system for detecting the one or more obstacles present in the surroundings of an autonomous vehicle. The disclosed obstacle detection system provides accurate location, structure and orientation of the obstacles present around the autonomous vehicle thereby allowing the autonomous vehicle to manoeuvre obstacles easily even when the vehicle is moving at greater speeds.

    [0082] In an embodiment, the present disclosure discloses a method and an obstacle detection system which does not need prior frame data from the LIDAR unit and the imaging unit, thereby making the system independent and efficient.

    [0083] In an embodiment, the present disclosure discloses a method and an obstacle detection system which detects the obstacles present at a far distance from the autonomous vehicle with accuracy.

    [0084] The terms "an embodiment", "embodiment", "embodiments", "the embodiment", "the embodiments", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the invention(s)" unless expressly specified otherwise.

    [0085] The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise.

    [0086] The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms "a", "an" and "the" mean "one or more", unless expressly specified otherwise.

    [0087] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the invention.

    [0088] When a single device or article is described herein, it will be readily apparent that more than one device/article (whether or not they cooperate) may be used in place of a single device/article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device/article may be used in place of the more than one device or article or a different number of devices/articles may be used instead of the shown number of devices or programs. The functionality and/or the features of a device may be alternatively embodied by one or more other devices which are not explicitly described as having such functionality/features. Thus, other embodiments of the invention need not include the device itself.

    [0089] The illustrated operations of Figure 3, Figure 4 and Figure 5 show certain events occurring in a certain order. In alternative embodiments, certain operations may be performed in a different order, modified or removed. Moreover, steps may be added to the above described logic and still conform to the described embodiments. Further, operations described herein may occur sequentially or certain operations may be processed in parallel. Yet further, operations may be performed by a single processing unit or by distributed processing units.


    Claims

    1. A method for obstacle detection by a vehicle in real-time, comprising:

    receiving, by an Electronic Control Unit, ECU (103), of a vehicle, one or more images of surroundings of the vehicle from an imaging unit (101) and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging unit (102);

    generating, by the ECU (103), a disparity image (601) from the one or more images by analysing the one or more images, wherein the one or more images comprise a first image and a second image;

    extracting, by the ECU (103), one or more regions comprising one or more obstacles, from the disparity image (601);

    determining, by the ECU (103), boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image (601);

    extracting, by the ECU (103), point cloud data points (800) from the point cloud data based on the image co-ordinates of the boundary points corresponding to the one or more obstacles; and

    determining, by the ECU (103), a gradient for each of the point cloud data points (800) to determine a structural orientation of the one or more obstacles, thereby detecting the one or more obstacles, wherein determining the structural orientation of the one or more obstacles comprises:

    identifying by the ECU (103), at least one point from the point cloud data points (800);

    projecting, by the ECU (103), a first virtual hemisphere (901) and a second virtual hemisphere with the identified at least one point as a centre of the first virtual hemisphere (901) and the second virtual hemisphere respectively, wherein the first virtual hemisphere (901)projects outwardly towards a top direction and the second virtual hemisphere projects outwardly towards a bottom direction of the identified at least one point, wherein the projection of the first virtual hemisphere (901) and the second virtual hemisphere is performed until one of a radius of virtual hemispheres is equal to a first threshold value and number of point cloud data points (800) present in the virtual hemisphere is equal to a second threshold value;

    connecting, by the ECU (103), the identified at least one point to each of the point cloud data points (800) that lie within the first virtual hemisphere (901), for forming a gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901);

    connecting, by the ECU (103), the identified at least one point to each of the point cloud data points (800) that lie within the second virtual hemisphere, for forming a gradient corresponding to each of the point cloud data points (800) in the second virtual hemisphere;

    determining, by the ECU (103), a resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere based on the gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901) and the second virtual hemisphere, with respect to the at least one point; and

    accumulating, by the ECU (103), the resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere, of each of the point cloud data points (800), for determining the structural orientation of the one or more obstacles.


     
    2. The method as claimed in claim 1, wherein detecting each of the one or more obstacles comprises:

    mapping, by the ECU (103), each pixel within the boundary points of each of the one or more obstacles to the gradient formed for each point of the point cloud data points (800) of the corresponding one or more obstacles, based on a distance error, to generate an enhanced point cloud data corresponding to the one or more obstacles;

    dividing, by the ECU (103), the point cloud data of the surroundings into a plurality of two-dimensional virtual grids, wherein each of the two-dimensional virtual grids comprises a plurality of enhanced point cloud data points (800); and

    detecting, by the ECU (103), one or more two-dimensional virtual grids representing each of the one or more obstacles based on a height difference between at least two enhanced point cloud data points (800) in each of the one or more two-dimensional virtual grids, wherein the height difference exceeds a height threshold.


     
    3. The method as claimed in any of the preceding claims, wherein the point cloud data of the surroundings comprises a plurality of 3-dimensional data points defined by corresponding cartesian co-ordinates.
     
    4. The method as claimed in any of the preceding claims, wherein generating the disparity image (601) comprises:

    matching each pixel in the first image to a corresponding pixel in the second image;

    computing a disparity value between each pixel in the first image to a corresponding pixel in the second image; and

    generating a disparity image (601) based on the disparity value, wherein each pixel in the disparity image (601) comprises the disparity value.


     
    5. An obstacle detection system for detecting obstacles by a vehicle in real-time, the obstacle detection system comprising;
    an Electronic Control Unit, ECU (103), configured to:

    receive one or more images of surroundings of the vehicle from an imaging unit (101) and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging unit (102);

    generate a disparity image (601) from the one or more images by analysing the one or more images, wherein the one or more images comprise a first image and a second image;

    extract one or more regions comprising one or more obstacles from the disparity image (601);

    determine boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image (601);

    extract point cloud data points (800) from the point cloud data based on the image co-ordinates of the boundary points corresponding to the one or more obstacles; and

    determine a gradient for each of the point cloud data points (800) to determine a structural orientation of the one or more obstacles, thereby detecting each of the one or more obstacles, wherein during determining the structural orientation of the one or more obstacles, the ECU (103) is configured to:

    identify at least one point from the point cloud data points (800);

    project a first virtual hemisphere (901) and a second virtual hemisphere with the identified at least one point as a centre of the first virtual hemisphere (901) and the second virtual hemisphere respectively, wherein the first virtual hemisphere (901) projects outwardly towards a top direction and the second virtual hemisphere projects outwardly towards a bottom direction of the identified at least one point, wherein the projection of the first virtual hemisphere (901) and the second virtual hemisphere is performed until one of a radius of virtual hemispheres is equal to a first threshold value and number of point cloud data points (800) present in the virtual hemisphere is equal to a second threshold value;

    connect the identified at least one point to each of the point cloud data points (800) that lie within the first virtual hemisphere (901), for forming a gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901);

    connect the identified at least one point to each of the point cloud data points (800) that lie within the second virtual hemisphere, for forming a gradient corresponding to each of the point cloud data points (800) in the second virtual hemisphere;

    determine a resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere based on the gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901) and the second virtual hemisphere, with respect to the at least one point; and

    accumulate the resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere, of each of the point cloud data points (800), for determining the structural orientation of the one or more obstacles.


     
    6. The obstacle detection system as claimed in claim 5, wherein for detecting one or more obstacles in the surrounding of the vehicle the ECU (103) is configured to:

    map each pixel within the boundary points of each of the one or more obstacles to the gradient formed for each point of the point cloud data points (800) of the corresponding one or more obstacles, based on a distance error, to generate an enhanced point cloud data corresponding to the one or more obstacles;

    divide the point cloud data of the surroundings into a plurality of two-dimensional virtual grids, wherein each of the two-dimensional virtual grids comprises a plurality of enhanced point cloud data points (800); and

    detect one or more two-dimensional virtual grids representing each of the one or more obstacles based on a height difference between at least two enhanced point cloud data points (800) in each of the one or more two-dimensional virtual grids, wherein the height difference exceeds a height threshold.


     
    7. The obstacle detection system as claimed in any of claims 5 to 6, wherein the point cloud data of the surroundings comprises a plurality of 3-dimensional data points defined by corresponding cartesian co-ordinates.
     
    8. The obstacle detection system as claimed in any of claims 5 to 7, wherein generating the disparity image (601) comprises:

    matching each pixel in the first image to a corresponding pixel in the second image;

    computing a disparity value between each pixel in the first image to a corresponding pixel in the second image; and

    generating a disparity image (601) based on the disparity value, wherein each pixel in the disparity image (601) comprises the disparity value.


     
    9. A computer readable medium including instructions stored thereon that when processed by at least one processor cause an obstacle detection system to perform operations comprising:

    receiving one or more images of surroundings of the vehicle from an imaging unit (101) and point cloud data of the surroundings of the vehicle from a Light Detection and Ranging unit (102);

    generating a disparity image (601) from the one or more images by analysing the one or more images, wherein the one or more images comprise a first image and a second image;

    extracting one or more regions comprising one or more obstacles, from the disparity image (601);

    determining boundary points of each of the one or more obstacles, for determining a position of each of the one or more obstacles in terms of image co-ordinates in the disparity image (601);

    extracting point cloud data points (800) from the point cloud data based on the image co-ordinates of the boundary points corresponding to the one or more obstacles; and

    determining a gradient for each of the point cloud data points (800) to determine a structural orientation of the one or more obstacles, thereby detecting the one or more obstacles, wherein determining the structural orientation of the one or more obstacles comprises:

    identifying at least one point from the point cloud data points (800);

    projecting a first virtual hemisphere (901) and a second virtual hemisphere with the identified at least one point as a centre of the first virtual hemisphere (901) and the second virtual hemisphere respectively, wherein the first virtual hemisphere (901) projects outwardly towards a top direction and the second virtual hemisphere projects outwardly towards a bottom direction of the identified at least one point, wherein the projection of the first virtual hemisphere (901) and the second virtual hemisphere is performed until one of a radius of virtual hemispheres is equal to a first threshold value and number of point cloud data points (800) present in the virtual hemisphere is equal to a second threshold value;

    connecting the identified at least one point to each of the point cloud data points (800) that lie within the first virtual hemisphere (901), for forming a gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901);

    connecting the identified at least one point to each of the point cloud data points (800) that lie within the second virtual hemisphere, for forming a gradient corresponding to each of the point cloud data points (800) in the second virtual hemisphere;

    determining a resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere based on the gradient corresponding to each of the point cloud data points (800) in the first virtual hemisphere (901) and the second virtual hemisphere, with respect to the at least one point; and

    accumulating the resultant gradient (902) in the first virtual hemisphere (901) and the second virtual hemisphere, of each of the point cloud data points (800), for determining the structural orientation of the one or more obstacles.


     
    10. The medium as claimed in claim 9, wherein detecting each of the one or more obstacles comprises:

    mapping each pixel within the boundary points of each of the one or more obstacles to the gradient formed for each point of the point cloud data points (800) of the corresponding one or more obstacles, based on a distance error, to generate an enhanced point cloud data corresponding to the one or more obstacles;

    dividing the point cloud data of the surroundings into a plurality of two-dimensional virtual grids, wherein each of the two-dimensional virtual grids comprises a plurality of enhanced point cloud data points (800); and

    detecting one or more two-dimensional virtual grids representing each of the one or more obstacles based on a height difference between at least two enhanced point cloud data points (800) in each of the one or more two-dimensional virtual grids, wherein the height difference exceeds a height threshold.


     
    11. The medium as claimed in any of claims 9 to 10, wherein the point cloud data of the surroundings comprises a plurality of 3-dimensional data points defined by corresponding cartesian co-ordinates.
     
    12. The medium as claimed in any of claims 10 to 11, wherein the disparity image (601) is generated from the one or more images, the one or more images comprising a first image and a second image, wherein generating of the disparity image (601) comprises:

    matching each pixel in the first image to a corresponding pixel in the second image;

    computing a disparity value between each pixel in the first image to a corresponding pixel in the second image; and

    generating a disparity image (601) based on the disparity value, wherein each pixel in the disparity image (601) comprises the disparity value.


     


    Ansprüche

    1. Verfahren zur Hinderniserkennung durch ein Fahrzeug in Echtzeit, umfassend:

    Empfangen, durch eine elektronische Steuereinheit ECU (103) eines Fahrzeugs, eines oder mehrerer Bilder der Umgebung des Fahrzeugs von einem Bildgebungsgerät (101) sowie Punktwolkendaten der Umgebung des Fahrzeugs von einem Lichterkennungs- und Entfernungsmessgerät (102),

    Erzeugen, durch die ECU (103), eines Disparitätsbildes (601) aus dem einen oder den mehreren Bildern durch Analysieren des einen oder der mehreren Bilder, wobei das eine oder die mehreren Bilder ein erstes Bild und ein zweites Bild umfassen,

    Extrahieren, durch die ECU (103), einer oder mehrerer Regionen, die ein oder mehrere Hindernisse umfassen, aus dem Disparitätsbild (601),

    Bestimmen, durch die ECU (103), der Grenzpunkte jedes des einen oder der mehreren Hindernisse, um eine Position jedes des einen oder der mehreren Hindernisse in Form von Bildkoordinaten im Disparitätsbild (601) zu bestimmen,

    Extrahieren, durch die ECU (103), von Punktwolken-Datenpunkten (800) aus den Punktwolkendaten auf der Grundlage der Bildkoordinaten der Grenzpunkte, die dem einen oder mehreren Hindernissen entsprechen, und

    Bestimmen, durch die ECU (103), eines Gradienten für jeden der Punktwolken-Datenpunkte (800), um eine strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen, wodurch das eine oder die mehreren Hindernisse erkannt werden, wobei das Bestimmen der strukturellen Ausrichtung des einen oder der mehreren Hindernisse Folgendes umfasst:

    Identifizieren, durch die ECU (103), mindestens eines Punktes aus den Punktwolken-Datenpunkten (800),

    Projizieren, durch die ECU (103), einer ersten virtuellen Halbkugel (901) und einer zweiten virtuellen Halbkugel mit dem identifizierten mindestens einen Punkt als Zentrum der ersten virtuellen Halbkugel (901) bzw. der zweiten virtuellen Halbkugel, wobei die erste virtuelle Halbkugel (901) nach außen in eine Richtung nach oben und die zweite virtuelle Halbkugel nach außen in eine Richtung nach unten des identifizierten mindestens einen Punktes projiziert, wobei die Projektion der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel durchgeführt wird, bis einer von einem Radius virtueller Halbkugeln gleich einem ersten Schwellenwert ist und die Anzahl von in der virtuellen Halbkugel vorhandenen Punktwolken-Datenpunkten (800) gleich einem zweiten Schwellenwert ist,

    Verbinden, durch die ECU (103), des identifizierten mindestens einen Punktes mit jedem der Punktwolke-Datenpunkte (800), die innerhalb der ersten virtuellen Halbkugel (901) liegen, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) entspricht,

    Verbinden, durch die ECU (103), des identifizierten mindestens einen Punktes mit jedem der Punktwolke-Datenpunkte (800), die innerhalb der zweiten virtuellen Halbkugel liegen, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der zweiten virtuellen Halbkugel entspricht,

    Bestimmen, durch die ECU (103), eines resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel auf der Grundlage des Gradienten, der jedem der Punktwolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel entspricht, in Bezug auf den mindestens einen Punkt, und

    Akkumulieren, durch die ECU (103), des resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel von jedem der Punktwolken-Datenpunkte (800), um die strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen.


     
    2. Verfahren nach Anspruch 1, wobei das Erkennen jedes des einen oder der mehreren Hindernisse Folgendes umfasst:

    Zuordnen, durch die ECU (103), eines jeden Pixels innerhalb der Grenzpunkte jedes des einen oder der mehreren Hindernisse dem Gradienten, der für jeden Punkt der Punktwolken-Datenpunkte (800) des entsprechenden einen oder der mehreren Hindernisse gebildet wird, auf der Grundlage eines Abstandsfehlers, um verbesserte Punktwolkendaten zu erzeugen, die dem einen oder den mehreren Hindernissen entsprechen,

    Aufteilen, durch die ECU (103), der Punktwolkendaten der Umgebung in eine Vielzahl zweidimensionaler virtueller Gitter, wobei jedes der zweidimensionalen virtuellen Gitter eine Vielzahl verbesserter Punktwolken-Datenpunkte (800) umfasst, und

    Erkennen, durch die ECU (103), eines oder mehrerer zweidimensionaler virtueller Gitter die jedes des einen oder der mehreren Hindernisse auf der Grundlage einer Höhendifferenz zwischen mindestens zwei verbesserten Punktwolken-Datenpunkten (800) in jedem des einen oder der mehreren zweidimensionalen virtuellen Gitter darstellen, wobei die Höhendifferenz einen Höhenschwellenwert überschreitet.


     
    3. Verfahren nach einem der vorhergehenden Ansprüche, wobei die Punktwolkendaten der Umgebung eine Vielzahl dreidimensionaler Datenpunkte umfassen, die durch entsprechende kartesische Koordinaten definiert sind.
     
    4. Verfahren nach einem der vorhergehenden Ansprüche, wobei das Erzeugen des Disparitätsbildes (601) Folgendes umfasst:

    Zuordnen jedes Pixels im ersten Bild einem entsprechenden Pixel im zweiten Bild,

    Berechnen eines Disparitätswertes zwischen jedem Pixel im ersten Bild und einem entsprechenden Pixel im zweiten Bild, und

    Erzeugen eines Disparitätsbildes (601) auf der Grundlage des Disparitätswertes, wobei jedes Pixel in dem Disparitätsbild (601) den Disparitätswert umfasst.


     
    5. Ein Hinderniserkennungssystem zum Erkennen von Hindernissen durch ein Fahrzeug in Echtzeit, wobei das Hinderniserkennungssystem Folgendes umfasst:
    eine elektronische Steuereinheit, ECU (103), eingerichtet zum:

    Empfangen eines oder mehrerer Bilder der Umgebung des Fahrzeugs von einem Bildgebungsgerät (101) sowie Punktwolkendaten der Umgebung des Fahrzeugs von einem Lichterkennungs- und Entfernungsmessgerät (102),

    Erzeugen eines Disparitätsbildes (601) aus dem einen oder den mehreren Bildern durch Analysieren des einen oder der mehreren Bilder, wobei das eine oder die mehreren Bilder ein erstes Bild und ein zweites Bild umfassen,

    Extrahieren einer oder mehrerer Regionen, die ein oder mehrere Hindernisse umfassen, aus dem Disparitätsbild (601),

    Bestimmen der Grenzpunkte jedes des einen oder der mehreren Hindernisse, um eine Position jedes des einen oder der mehreren Hindernisse in Form von Bildkoordinaten im Disparitätsbild (601) zu bestimmen,

    Extrahieren von Punktwolken-Datenpunkten (800) aus den Punktwolkendaten auf der Grundlage der Bildkoordinaten der Grenzpunkte, die dem einen oder den mehreren Hindernissen entsprechen, und

    Bestimmen eines Gradienten für jeden der Punktwolken-Datenpunkte (800), um eine strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen, wodurch das eine oder die mehreren Hindernisse erkannt werden, wobei während des Bestimmens der strukturellen Ausrichtung des einen oder der mehreren Hindernisse die ECU (103) eingerichtet ist, um:

    mindestens einen Punkt aus den Punktwolken-Datenpunkten (800) zu identifizieren, eine erste virtuelle Halbkugel (901) und eine zweite virtuelle Halbkugel mit dem identifizierten mindestens einen Punkt als Zentrum der ersten virtuellen Halbkugel (901) bzw. der zweiten virtuellen Halbkugel zu projizieren, wobei die erste virtuelle Halbkugel (901) nach außen in eine Richtung nach oben und die zweite virtuelle Halbkugel nach außen in eine Richtung nach unten des identifizierten mindestens einen Punktes projiziert, wobei die Projektion der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel durchgeführt wird, bis einer von einem Radius virtueller Halbkugeln gleich einem ersten Schwellenwert ist und die Anzahl von in der virtuellen Halbkugel vorhandenen Punktwolken-Datenpunkten (800) gleich einem zweiten Schwellenwert ist,

    den identifizierten mindestens einen Punkt mit jedem der Punktwolken-Datenpunkte (800) zu verbinden, die innerhalb der ersten virtuellen Halbkugel (901) liegen, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) entspricht,

    den mindestens einen Punkt mit jedem der Punktwolke-Datenpunkte (800), die innerhalb der zweiten virtuellen Halbkugel liegen, zu verbinden, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der zweiten virtuellen Halbkugel entspricht,

    einen resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel auf der Grundlage des Gradienten, der jedem der Punktewolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel entspricht, in Bezug auf den mindestens einen Punkt zu bestimmen, und

    den resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel von jedem der Punktwolken-Datenpunkte (800) zu akkumulieren, um die strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen.


     
    6. Hinderniserkennungssystem nach Anspruch 5, wobei die ECU (103) für das Erkennen eines oder mehrerer Hindernisse in der Umgebung des Fahrzeugs eingerichtet ist, um:

    jedes Pixel innerhalb der Grenzpunkte jedes des einen oder der mehreren Hindernisse dem Gradienten zuzordnen, der für jeden Punkt der Punktwolken-Datenpunkte (800) des entsprechenden einen oder der mehreren Hindernisse gebildet wird, auf der Grundlage eines Abstandsfehlers, um verbesserte Punktwolkendaten zu erzeugen, die dem einen oder den mehreren Hindernissen entsprechen,

    die Punktwolkendaten der Umgebung in eine Vielzahl zweidimensionaler virtueller Gitter aufzuteilen, wobei jedes der zweidimensionalen virtuellen Gitter eine Vielzahl verbesserter Punktwolken-Datenpunkte (800) umfasst, und

    ein oder mehrere zweidimensionale virtuelle Gitter zu erkennen, die jedes des einen oder der mehreren Hindernisse auf der Grundlage einer Höhendifferenz zwischen mindestens zwei verbesserten Punktwolken-Datenpunkten (800) in jedem des einen oder der mehreren zweidimensionalen virtuellen Gitter darstellen, wobei die Höhendifferenz einen Höhenschwellenwert überschreitet.


     
    7. Hinderniserkennungssystem nach einem der Ansprüche 5 bis 6, wobei die Punktwolkendaten der Umgebung eine Vielzahl dreidimensionaler Datenpunkte umfassen, die durch entsprechende kartesische Koordinaten definiert sind.
     
    8. Hinderniserkennungssystem nach einem der Ansprüche 5 bis 7, wobei das Erzeugen des Disparitätsbildes (601) Folgendes umfasst:

    Zuordnen jedes Pixels im ersten Bild einem entsprechenden Pixel im zweiten Bild,

    Berechnen eines Disparitätswertes zwischen jedem Pixel im ersten Bild und einem entsprechenden Pixel im zweiten Bild, und

    Erzeugen eines Disparitätsbildes (601), auf der Grundlage des Disparitätswertes, wobei jedes Pixel in dem Disparitätsbild (601) den Disparitätswert umfasst.


     
    9. Ein computerlesbares Medium mit darauf gespeicherten Anweisungen, die bei Verarbeitung durch mindestens einen Prozessor bewirken, dass ein Hinderniserkennungssystem Operationen ausführt, umfassend:

    Empfangen eines oder mehrerer Bilder der Umgebung des Fahrzeugs von einem Bildgebungsgerät (101) sowie Punktwolkendaten der Umgebung des Fahrzeugs von einem Lichterkennungs- und Entfernungsmessgerät (102),

    Erzeugen eines Disparitätsbildes (601) aus dem einen oder den mehreren Bildern durch Analysieren des einen oder der mehreren Bilder, wobei das eine oder die mehreren Bilder ein erstes Bild und ein zweites Bild umfassen,

    Extrahieren einer oder mehrerer Regionen, die ein oder mehrere Hindernisse umfassen, aus dem Disparitätsbild (601),

    Bestimmen der Grenzpunkte jedes des einen oder der mehreren Hindernisse, um eine Position jedes des einen oder der mehreren Hindernisse in Bezug auf Bildkoordinaten im Disparitätsbild (601) zu bestimmen,

    Extrahieren von Punktwolken-Datenpunkten (800) aus den Punktwolkendaten auf der Grundlage der Bildkoordinaten der Grenzpunkte, die dem einen oder den mehreren Hindernissen entsprechen, und

    Bestimmen eines Gradienten für jeden der Punktwolken-Datenpunkte (800), um eine strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen, wodurch das eine oder die mehreren Hindernisse erkannt werden, wobei das Bestimmen der strukturellen Ausrichtung des einen oder der mehreren Hindernisse Folgendes umfasst:

    Identifizieren mindestens eines Punktes aus den Punktwolken-Datenpunkten (800),

    Projizieren einer ersten virtuellen Halbkugel (901) und einer zweiten virtuellen Halbkugel mit dem identifizierten mindestens einen Punkt als Zentrum der ersten virtuellen Halbkugel (901) bzw. der zweiten virtuellen Halbkugel, wobei die erste virtuelle Halbkugel (901) nach außen in eine Richtung nach oben und die zweite virtuelle Halbkugel nach außen in eine Richtung nach unten des identifizierten mindestens einen Punktes projiziert, wobei die Projektion der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel durchgeführt wird, bis einer von einem Radius virtueller Halbkugeln gleich einem ersten Schwellenwert ist und die Anzahl von in der virtuellen Halbkugel vorhandenen Punktwolken-Datenpunkten (800) gleich einem zweiten Schwellenwert ist,

    Verbinden des identifizierten mindestens einen Punktes mit jedem der Punktwolken-Datenpunkte (800), die innerhalb der ersten virtuellen Halbkugel (901) liegen, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) entspricht,

    Verbinden des identifizierten mindestens einen Punktes mit jedem der Punktwolken-Datenpunkte (800), die innerhalb der zweiten virtuellen Halbkugel liegen, um einen Gradienten zu bilden, der jedem der Punktwolken-Datenpunkte (800) in der zweiten virtuellen Halbkugel entspricht,

    Bestimmen eines resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel auf der Grundlage des Gradienten, der jedem der Punktwolken-Datenpunkte (800) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel entspricht, in Bezug auf den mindestens einen Punkt, und

    Akkumulieren des resultierenden Gradienten (902) in der ersten virtuellen Halbkugel (901) und der zweiten virtuellen Halbkugel von jedem der Punktwolken-Datenpunkte (800), um die strukturelle Ausrichtung des einen oder der mehreren Hindernisse zu bestimmen.


     
    10. Mittel nach Anspruch 9, wobei das Erkennen jedes des einen oder der mehreren Hindernisse Folgendes umfasst:

    Abbilden jedes Pixels innerhalb der Grenzpunkte jedes des einen oder der mehreren Hindernisse auf den Gradienten, der für jeden Punkt der Punktwolken-Datenpunkte (800) des entsprechenden einen oder der mehreren Hindernisse gebildet wird, auf der Grundlage eines Abstandsfehlers, um verbesserte Punktwolkendaten zu erzeugen, die dem einen oder den mehreren Hindernissen entsprechen,

    Aufteilen der Punktwolkendaten der Umgebung in eine Vielzahl zweidimensionaler virtueller Gitter, wobei jedes der zweidimensionalen virtuellen Gitter eine Vielzahl verbesserter Punktwolken-Datenpunkte (800) umfasst, und

    Erkennen eines oder mehrerer zweidimensionaler virtueller Gitter, die jedes des einen oder der mehreren Hindernisse auf der Grundlage einer Höhendifferenz zwischen mindestens zwei verbesserten Punktwolken-Datenpunkten (800) in jedem des einen oder der mehreren zweidimensionalen virtuellen Gitter darstellen, wobei die Höhendifferenz einen Höhenschwellenwert überschreitet.


     
    11. Medium nach einem der Ansprüche 9 bis 10, wobei die Punktwolkendaten der Umgebung eine Vielzahl dreidimensionaler Datenpunkte umfassen, die durch entsprechende kartesische Koordinaten definiert sind.
     
    12. Medium nach einem der Ansprüche 10 bis 11, wobei das Disparitätsbild (601) aus dem einen oder den mehreren Bildern erzeugt wird, wobei das eine oder die mehreren Bilder ein erstes Bild und ein zweites Bild umfassen, wobei das Erzeugen des Disparitätsbildes (601) Folgendes umfasst:

    Zuordnen jedes Pixels im ersten Bild einem entsprechenden Pixel im zweiten Bild,

    Berechnen eines Disparitätswertes zwischen jedem Pixel im ersten Bild und einem entsprechenden Pixel im zweiten Bild, und

    Erzeugen eines Disparitätsbildes (601) auf der Grundlage des Disparitätswertes, wobei jedes Pixel in dem Disparitätsbild (601) den Disparitätswert umfasst.


     


    Revendications

    1. Procédé de détection d'obstacles par un véhicule en temps réel, comprenant :

    la réception, par une unité de commande électronique, ECU (103), d'un véhicule, d'une ou plusieurs images de l'environnement du véhicule provenant d'une unité d'imagerie (101) et de données de nuage de points de l'environnement du véhicule provenant d'une unité de détection et de télémétrie de la lumière (102) ;

    la génération, par l'ECU (103), d'une image de disparité (601) à partir de l'une ou plusieurs images en analysant l'une ou plusieurs images, l'une ou plusieurs images comprenant une première image et une deuxième image ;

    l'extraction, par l'ECU (103), d'une ou plusieurs régions comprenant un ou plusieurs obstacles, de l'image de disparité (601) ;

    la détermination, par l'ECU (103), de points de limite de chacun de l'un ou plusieurs obstacles, pour déterminer une position de chacun de l'un ou plusieurs obstacles en termes de coordonnées d'image dans l'image de disparité (601) ;

    l'extraction, par l'ECU (103), de points de données de nuage de points (800) à partir des données de nuage de points sur la base des coordonnées d'image des points de limite correspondant à l'un ou plusieurs obstacles ; et

    la détermination, par l'ECU (103), d'un gradient pour chacun des points de données de nuage de points (800) pour déterminer une orientation structurelle de l'un ou plusieurs obstacles, et détecter ainsi l'un ou plusieurs obstacles, dans lequel la détermination de l'orientation structurelle de l'un ou plusieurs obstacles comprend :

    l'identification par l'ECU (103), d'au moins un point parmi les points de données de nuage de points (800) ;

    la projection, par l'ECU (103), d'un premier hémisphère virtuel (901) et d'un deuxième hémisphère virtuel avec l'au moins un point identifié comme centre du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel respectivement, dans lequel le premier hémisphère virtuel (901) se projette vers l'extérieur dans une direction vers le haut et le deuxième hémisphère virtuel se projette vers l'extérieur dans une direction vers le bas de l'au moins un point identifié, dans lequel la projection du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel est effectuée jusqu'à ce que l'un d'un rayon des hémisphères virtuels soit égal à une première valeur de seuil et que le nombre de points de données de nuage de points (800) présents dans l'hémisphère virtuel soit égal à une deuxième valeur de seuil ;

    la connexion, par l'ECU (103), de l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent dans le premier hémisphère virtuel (901), pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) ;

    la connexion, par l'ECU (103), de l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent à l'intérieur du deuxième hémisphère virtuel, pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le deuxième hémisphère virtuel ;

    la détermination, par l'ECU (103), d'un gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel sur la base du gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, par rapport à l'au moins un point ; et

    l'accumulation, par l'ECU (103), du gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, de chacun des points de données de nuage de points (800), pour déterminer l'orientation structurelle de l'un ou plusieurs obstacles.


     
    2. Procédé selon la revendication 1, dans lequel la détection de chacun de l'un ou plusieurs obstacles comprend :

    le mappage, par l'ECU (103), de chaque pixel à l'intérieur des points de limite de chacun de l'un ou plusieurs obstacles au gradient formé pour chaque point des points de données de nuage de points (800) de l'un ou plusieurs obstacles correspondants, sur la base d'une erreur de distance, pour générer une donnée de nuage de points améliorée correspondant à l'un ou plusieurs obstacles ;

    la division, par l'ECU (103), des données de nuage de points de l'environnement en une pluralité de grilles virtuelles bidimensionnelles, chacune des grilles virtuelles bidimensionnelles comprenant une pluralité de points de données de nuage de points améliorés (800) ; et

    la détection, par l'ECU (103), d'une ou plusieurs grilles virtuelles bidimensionnelles représentant chacune l'un ou plusieurs obstacles sur la base d'une différence de hauteur entre au moins deux points de données de nuage de points améliorés (800) dans chacun de l'une ou plusieurs grilles virtuelles bidimensionnelles, la différence de hauteur dépassant un seuil de hauteur.


     
    3. Procédé selon l'une quelconque des revendications précédentes, dans lequel les données de nuage de points de l'environnement comprennent une pluralité de points de données tridimensionnels définis par des coordonnées cartésiennes correspondantes.
     
    4. Procédé selon l'une quelconque des revendications précédentes, dans lequel la génération de l'image de disparité (601) comprend :

    la mise en correspondance de chaque pixel dans la première image à un pixel correspondant dans la deuxième image ;

    le calcul d'une valeur de disparité entre chaque pixel dans la première image et un pixel correspondant dans la deuxième image ; et

    la génération d'une image de disparité (601) sur la base de la valeur de disparité, chaque pixel dans l'image de disparité (601) comprenant la valeur de disparité.


     
    5. Système de détection d'obstacles pour détecter des obstacles par un véhicule en temps réel, le système de détection d'obstacles comprenant :
    une unité de commande électronique, ECU (103), configurée pour :

    recevoir une ou plusieurs images de l'environnement du véhicule à partir d'une unité d'imagerie (101) et des données de nuage de points de l'environnement du véhicule à partir d'une unité de détection et de télémétrie de lumière (102) ;

    générer une image de disparité (601) à partir de l'une ou plusieurs images en analysant l'une ou plusieurs images, l'une ou plusieurs images comprenant une première image et une deuxième image ;

    extraire une ou plusieurs régions comprenant un ou plusieurs obstacles de l'image de disparité (601) ;

    déterminer des points de limite de chacun de l'un ou plusieurs obstacles, pour déterminer une position de chacun de l'un ou plusieurs obstacles en termes de coordonnées d'image dans l'image de disparité (601) ;

    extraire des points de données de nuage de points (800) des données de nuage de points sur la base des coordonnées d'image des points de limite correspondant à l'un ou plusieurs obstacles ; et

    déterminer un gradient pour chacun des points de données de nuage de points (800) pour déterminer une orientation structurelle de l'un ou plusieurs obstacles, et détecter ainsi chacun de l'un ou plusieurs obstacles, dans lequel lors de la détermination de l'orientation structurelle de l'un ou plusieurs obstacles, l'ECU (103) est configurée pour :

    identifier au moins un point parmi les points de données de nuage de points (800) ; projeter un premier hémisphère virtuel (901) et un deuxième hémisphère virtuel avec l'au moins un point identifié comme centre du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel respectivement, dans lequel le premier hémisphère virtuel (901) se projette vers l'extérieur dans une direction vers le haut et le deuxième hémisphère virtuel se projette vers l'extérieur dans une direction vers le bas de l'au moins un point identifié, dans lequel la projection du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel sont effectuées jusqu'à ce que l'un d'un rayon d'hémisphères virtuels soit égal à une première valeur de seuil et qu'un nombre de points de données de nuage de points (800) présents dans l'hémisphère virtuel soit égal à une deuxième valeur de seuil ;

    connecter l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent à l'intérieur du premier hémisphère virtuel (901), pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) ;

    connecter l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent à l'intérieur du deuxième hémisphère virtuel, pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le deuxième hémisphère virtuel ;

    déterminer un gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel sur la base du gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, par rapport à l'au moins un point ; et

    accumuler le gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, de chacun des points de données de nuage de points (800), pour déterminer l'orientation structurelle de l'un ou plusieurs obstacles.


     
    6. Système de détection d'obstacles selon la revendication 5, dans lequel pour détecter l'un ou plusieurs obstacles à proximité du véhicule, l'ECU (103) est configurée pour :

    mapper chaque pixel dans les points limites de chacun de l'un ou plusieurs obstacles au gradient formé pour chaque point des points de données de nuage de points (800) de l'un ou plusieurs obstacles correspondants, sur la base d'une erreur de distance, pour générer une donnée en nuage de points améliorée correspondant à l'un ou plusieurs obstacles ;

    diviser les données de nuage de points de l'environnement en une pluralité de grilles virtuelles bidimensionnelles, chacune des grilles virtuelles bidimensionnelles comprenant une pluralité de points de données de nuage de points améliorés (800) ; et

    détecter une ou plusieurs grilles virtuelles bidimensionnelles représentant chacun de l'un ou plusieurs obstacles sur la base d'une différence de hauteur entre au moins deux points de données de nuage de points améliorés (800) dans chacune de l'une ou plusieurs grilles virtuelles bidimensionnelles, la hauteur la différence dépassant un seuil de hauteur.


     
    7. Système de détection d'obstacles selon l'une quelconque des revendications 5 à 6, dans lequel les données de nuage de points de l'environnement comprennent une pluralité de points de données tridimensionnelles définis par des coordonnées cartésiennes correspondantes.
     
    8. Système de détection d'obstacles selon l'une quelconque des revendications 5 à 7, dans lequel la génération de l'image de disparité (601) comprend :

    la mise en correspondance de chaque pixel dans la première image à un pixel correspondant dans la deuxième image ;

    le calcul d'une valeur de disparité entre chaque pixel dans la première image et un pixel correspondant dans la deuxième image ; et

    la génération d'une image de disparité (601) sur la base de la valeur de disparité, chaque pixel dans l'image de disparité (601) comprenant la valeur de disparité.


     
    9. Support lisible par ordinateur comprenant des instructions stockées sur celui-ci qui, lorsqu'elles sont traitées par au moins un processeur, amènent un système de détection d'obstacles à effectuer des opérations comprenant :

    la réception d'une ou plusieurs images de l'environnement du véhicule à partir d'une unité d'imagerie (101) et des données de nuage de points de l'environnement du véhicule à partir d'une unité de détection et de télémétrie de lumière (102) ;

    la génération d'une image de disparité (601) à partir de l'une ou plusieurs images en analysant l'une ou plusieurs images, l'une ou plusieurs images comprenant une première image et une deuxième image ;

    l'extraction d'une ou plusieurs régions comprenant un ou plusieurs obstacles, de l'image de disparité (601) ;

    la détermination de points de limite de chacun de l'un ou plusieurs obstacles, pour déterminer une position de chacun de l'un ou plusieurs obstacles en termes de coordonnées d'image dans l'image de disparité (601) ;

    l'extraction de points de données de nuage de points (800) à partir des données de nuage de points sur la base des coordonnées d'image des points de limite correspondant à l'un ou plusieurs obstacles ; et

    la détermination d'un gradient pour chacun des points de données de nuage de points (800) pour déterminer une orientation structurelle de l'un ou plusieurs obstacles, et détecter ainsi l'un ou plusieurs obstacles, dans lequel la détermination de l'orientation structurelle de l'un ou plusieurs obstacles comprend :

    l'identification d'au moins un point parmi les points de données de nuage de points (800) ;

    la projection d'un premier hémisphère virtuel (901) et d'un deuxième hémisphère virtuel avec l'au moins un point identifié comme centre du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel respectivement, dans lequel le premier hémisphère virtuel (901) se projette vers l'extérieur dans une direction vers le haut et le deuxième hémisphère virtuel se projette vers l'extérieur dans une direction vers le bas de l'au moins un point identifié, dans lequel la projection du premier hémisphère virtuel (901) et du deuxième hémisphère virtuel est effectuée jusqu'à ce que l'un d'un rayon des hémisphères virtuels soit égal à une première valeur de seuil et qu'un nombre de points de données de nuage de points (800) présents dans l'hémisphère virtuel soit égal à une deuxième valeur de seuil ;

    la connexion de l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent à l'intérieur du premier hémisphère virtuel (901), pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) ;

    la connexion de l'au moins un point identifié à chacun des points de données de nuage de points (800) qui se trouvent à l'intérieur du deuxième hémisphère virtuel, pour former un gradient correspondant à chacun des points de données de nuage de points (800) dans le deuxième hémisphère virtuel ;

    la détermination d'un gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel sur la base du gradient correspondant à chacun des points de données de nuage de points (800) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, par rapport à l'au moins un point ; et

    l'accumulation du gradient résultant (902) dans le premier hémisphère virtuel (901) et le deuxième hémisphère virtuel, de chacun des points de données de nuage de points (800), pour déterminer l'orientation structurelle de l'un ou plusieurs obstacles.


     
    10. Support selon la revendication 9, dans lequel la détection de chacun de l'un ou plusieurs obstacles comprend :

    le mappage de chaque pixel à l'intérieur des points de limite de chacun de l'un ou plusieurs obstacles au gradient formé pour chaque point des points de données de nuage de points (800) de l'un ou plusieurs obstacles correspondants, sur la base d'une erreur de distance, pour générer une donnée en nuage de points améliorée correspondant à l'un ou plusieurs obstacles ;

    la division des données de nuage de points de l'environnement en une pluralité de grilles virtuelles bidimensionnelles, chacune des grilles virtuelles bidimensionnelles comprenant une pluralité de points de données de nuage de points améliorés (800) ; et

    la détection d'une ou plusieurs grilles virtuelles bidimensionnelles représentant chacun de l'un ou plusieurs obstacles sur la base d'une différence de hauteur entre au moins deux points de données de nuage de points améliorés (800) dans chacune de l'une ou plusieurs grilles virtuelles bidimensionnelles, la différence de hauteur dépassant un seuil de hauteur.


     
    11. Support selon l'une quelconque des revendications 9 à 10, dans lequel les données de nuage de points de l'environnement comprennent une pluralité de points de données tridimensionnelles définis par des coordonnées cartésiennes correspondantes.
     
    12. Support selon l'une quelconque des revendications 10 à 11, dans lequel l'image de disparité (601) est générée à partir d'une ou plusieurs images, l'une ou plusieurs images comprenant une première image et une deuxième image, dans lequel la génération de l'image de disparité (601) comprend :

    la mise en correspondance de chaque pixel dans la première image à un pixel correspondant dans la deuxième image ;

    le calcul d'une valeur de disparité entre chaque pixel dans la première image et un pixel correspondant dans la deuxième image ; et

    la génération d'une image de disparité (601) sur la base de la valeur de disparité, chaque pixel dans l'image de disparité (601) comprenant la valeur de disparité.


     




    Drawing