The present invention concerns a method for providing information assisting a driver of a vehicle which is equipped with a navigation system. It also concerns a corresponding system for use in a vehicle.

There is a need to improve current navigation systems. It is quite frustrating that still today these systems are not able to deal properly with traffic congestions in urban areas. There is quite some potential to reduce traffic jams and to reduce at the same time the pollution caused by cars which are waiting in queues, e.g. in front of traffic lights, tunnel portals, parking decks, shopping centers, and so forth.

There is a vehicle-to-vehicle communication system known as "TrafficView" which enables the spreading of traffic related information from car to car using short-range wireless communication. Details can be found in "TrafficView: A Driver Assistant Device for Traffic Monitoring based on Car-to-Car Communication", S. Dashtinezhad et al., cf.
http://www.cs.rutgers.edu/∼iftode/vtcsp04.pdf.

There is the long felt desire to provide real-time information or even images while driving inside a car. It is thus not surprising to see that a number of approaches and schemes have been developed which employ road-side cameras and which link the cameras with a display inside the car.

The following presentations

• "Embedded Systems", WSITC FORUM, AN INTERNATIONAL CES 2007 PERSPECTIVE, which was held by Robert Mitchell, AEEMA, 20 February 2007,
• "What's Hot in ICT?", WSITC FORUM, AN INTERNATIONAL CES 2007 PERSPECTIVE, which was held by Angus M Robinson, AEEMA, 20 February 2007

For details see also:

• http://interdependent.com.au/wsitc/documents/WSITC_Forum_20Feb07_Progra m_PPTs.pdf

The use of road-side cameras near intersections is proposed and discussed in "Visual Assistance for Drivers by Mixed Reality", Y. Kameda et al., 14th World Congress at of ITS, 2007, Beijing. The road-side cameras are used in order to provide visual assistance and a better overview for a driver who has a receiver in his car. For details see: http://www.kamedalab.org/research/publication/2007/200710_ITSWC/ITSWC2007-3210-kameda.pdf

Yet another approach, called AHS (Automated Highway System), is discussed in the following paper: "NaviView: Bird's-Eye View for Highway Drivers Using Roadside Cameras," Eitaro Ichihara, Hiroyuki Takao, Yuichi Ohta, icmcs, vol. 2, pp.559, 1999 IEEE International Conference on Multimedia Computing and Systems (ICMCS'99) - Volume 2, 1999. The AHS is believed to increase the traffic potential of highways because it enables drivers to shorten the distance between cars. AHS may make the drivers uncomfortable because it presents them with a reduced field of vision of the car ahead and may induce anxiety over the reliability of the control system. In order to improve the comfort of drivers under the control of AHS, it is necessary to recover the driver's field of vision. In this paper, a NaviView system for the purpose of recovery is proposed. In AHS, the entire traffic load on the highway is visually monitored by many roadside video cameras. NaviView utilizes these video images for the recovery of the driver's visual field. It enables the driver to observe the behavior of other cars by means of a virtual view from a point just above his car. The virtual view presented to the driver is generated by combining the video images of the fixed roadside cameras.

Another approach is presented in the following publication: "Car navigation system with image recognition," Kohei Ito, Naohiko Ichihara, Hiroto Inoue, Ryujiro Fujita, Mitsuo Yasushi, icce, pp.1-2, 2009 Digest of Technical Papers International Conference on Consumer Electronics, 2009. The authors state that many cars now would have on-board cameras, and that many kinds of driver support systems that use image recognition are being developed. The authors themselves claim to have developed a car navigation system with image recognition that enhances drivers' safety, convenience, and entertainment.

There are GPS (global positioning system) based navigation systems with backup camera display. The camera and transmitter are powered by the cables that power the vehicle's reverse lights. When the vehicle is put into reverse gear, the reversing lights turn on and power is sent to the camera and the transmitter, sending out a video signal. The GPS unit then automatically switches over to the camera view. When the vehicle is taken out of reverse gear, the reverse lights go out and the camera stops sending an image. At the same time the unit will stop showing the camera's image and return to the previous function. See for instance: http://www.navsgo.com/GO740RV.html

Another navigation system with integration of live images is disclosed in WO 01/82261 A1. The respective system employs an approach with a two-way communications between the navigation system in a car and an image server. This approach causes quite some load on existing communication systems in particular in urban areas where many cars and many cameras will be put to use.

Another navigation system is known from the published patent application WO 2007/057696 A1. This system is dsigned to show symbols on a display which represent cameras. The driver can pick a camera by touching the respective symbole on the display. This selection causes a play back of audo or visual information which was recorded by a camera before. This application also addresses bandwidth-related issues. In order to overcome bandwidth limitations, it is proposed to use a compression scheme.

All these systems mentioned above confirm that there is a desire for establishing a connection between cameras and cars. It is a disadvantage of the known approaches that they are just providing overview information which is not linked at all to the capabilities of a navigation system. These systems have the serious disadvantage that their use while driving might distract the driver. It is yet another disadvantage of known systems, that they require quite some bandwidth for the transmission of image information. Any system which would be offered to all users of navigation systems in an urban area would lead to a communication or capacity overload.

It is an objective of the present invention to provide a camera-assisted or camera-based navigation system which offers real time or close-to-real time information.

It is another objective to provide a system which is well suited for large scale operation e.g. in urban area.

The method, according to the present invention, provides information assisting a driver of a vehicle equipped with a navigation system in accordance with claim 1.

The inventive system is designed for use in a vehicle. The system comprises the features of claim 10.

The features of advantageous embodiments are presented in the dependent method and apparatus claims. The respective advantages are addressed or become apparent from the following detailed description.

For a more complete description of the present invention and for further objects and advantages thereof, reference is made to the following description, taken in conjunction with the accompanying drawings:

Fig. 1

shows a schematic structural diagram of a system in accordance with the present invention;

Fig. 2

shows a schematic diagram of a display of a system in accordance with the present invention;

Fig. 3

shows a schematic diagram of a display of a system in accordance with the present invention;

Fig. 4

shows a schematic diagram of a display of a system in accordance with the present invention;

Fig. 5A

shows a schematic diagram of a real traffic situation;

Fig. 5B

shows a schematic diagram of a display of a system in accordance with the present invention where an image of the real traffic situation of Fig. 5A is shown;

Fig. 6

shows a schematic flow-chart in accordance with the present invention;

Fig. 7

shows a schematic flow-chart in accordance with the present invention;

Fig. 8

shows a schematic flow-chart in accordance with the present invention;

Fig. 9

shows a schematic map with two objects representing two road-side cameras in accordance with the present invention;

Fig. 10A

shows a schematic diagram of a real traffic situation;

Fig. 10B

shows a schematic diagram of the relevant static image content of Fig. 10A;

Fig. 10C

shows a schematic diagram of the relevant dynamic image content of Fig. 10A;

Fig. 10D

shows a schematic diagram of the "reconstructed" image which is based on or derived from the image of Fig. 10A;

Fig. 11

shows several different objects which could be used to show on a display the position of a road-side camera;

Fig. 12

shows a schematic illustration of an image obtained by a road-side camera to which a filter is attached.

Fig. 1 shows a schematic structural diagram of a system 200 in accordance with the present invention. The system 200 comprises a navigation system 2 inside a vehicle 1 (e.g. a car), a remote system 100 and at least one road-side camera C1. The system 200 furthermore comprises communication means or channels, as indicated in Fig. 1 by arrows. The image information, which is transmitted from the road-side camera C1 to the remote system 100, is represented by the reference sign I1. The image information 12 is the information which is directly or indirectly transmitted from the remote system 100 to the navigation system 2. According to the invention, the data size of 12 is smaller than the data size of I1 or than the data size of the image recorded by the road-side camera C1. If the compression is carried out inside the road-side camera C1, the data size of 12 might be about the same as the data size of I1.

A "navigation system" 2, as herein used, is an electronic processors-based system which for instance enables or supports the driver of a vehicle 1 to find a certain route or to find a location. The navigation system 2 typically has a GPS or other positioning feature which is included or connected. The navigation system 2 may be integrated into the vehicle 1 (pre-installed system), or it may be a portable system (add-on system) which can be used, as needed, in different vehicles. Any portable computer-based system (e.g. a smart phone) in which a navigation application is implemented, or which is able to connect to a server-based or cloud-based navigation system is also considered to be a navigation system for the purposes of the present invention. A web-based navigation system is also considered to be a navigation system for the purposes of the present invention. In other words, a navigation system 2, as herein used, may be entirely onboard the vehicle 1, or it may be located elsewhere and communicate via radio or other signals with the vehicle, or it may use a combination of these methods.

The navigation system 2 can be a system presenting 2-dimensional or 3-dimensional maps. It could also be a system 2 where real images are used to provide a more realistic look. The invention can be used in connection with any of these systems 2.

A "remote system 100" is a single server (i.e. a physical computer dedicated to running a special service) or host, a set-up where two or more servers (e.g. a series or network of computers) are connected, or a cloud-computing environment (e.g. an arrangement where shared resources, software, and information are provided to computers and other devices, such as the navigation system 2). The remote system 100 is a stationary system. At least a part of the navigation system 2 (for instance the receiver and the display 2) is mobile.

The remote system 100 is configured and programmed to provide essential inventive services across a communication link or network to at least one vehicle 1 comprising a navigation system 2.

A "road-side camera C1" is either a camera at a fixed location, e.g. attached to a pole, lamp, tree or building, or a vehicle-based camera which keeps a pre-defined position for a certain period of time, e.g. during rush hours. A camera C1 is herein considered to be a device that records images. The camera C1 preferably comprises or is connected to communication means for establishing a direct or indirect communication link to the remote system 100. The road-side camera C1 preferably provides digital image information.

Figures 6, 7, and 8 show exemplary schematic flow-charts of the various aspects of the inventive method.

The invention concerns a method which provides information assisting a driver of a vehicle 1 which is equipped with a navigation system 2. The method comprises the following steps:

• Providing real-time image information I1 to the remote system 100, the real-time image information I1 having been obtained from at least one road-side camera C1 (as illustrated in Fig. 1, for instance).
• Displaying an object 3 (see for instance Figures 2, 3, 4, 5B, 9) on a display 4 of the navigation system 2 which represents the at least one road-side camera C1.
• Selecting a corresponding road-side camera C1 using a user interface of the navigation system 2 (e.g. by touching the object 3 on a touch sensitive display 4 or by using a keyboard or by using voice control). This step enables the user of the system 2 to select a particular (corresponding) road-side camera C1 (cf. step S1 in Fig. 6).
• Transmitting image information 12 which is based on the real-time image information I1 provided by the corresponding road-side camera C1 which has been selected (cf. step S3 in Fig. 6). The transmission takes place between the remote system 100 or a relay unit attached or connected to the remote system 100 and the navigation system 2 (as illustrated in Fig. 1, for instance).
• Displaying an image 13 based on the image information 12 on the display 4 in order to provide visual information of a specific section or part of the route R1 (cf. step S6 in Fig. 6). In Fig. 5B a situation or embodiment is shown, where on the display 4 an image 5 in an image is shown. A split-screen solution could also be used.

The road-side camera C1 is typically selected by the driver of the vehicle 1 before a specific section or part of a route R1 is approached.

Before the image information 12 is transmitted, it is made available by the remote system 100 (cf. step S2 in Fig. 6). Before displaying the image information 13 on the navigation system's display 4, the image information 12 is directly or indirectly (e.g. via a cellular phone) received (cf. step S4 in Fig. 6) and processed (cf. step S5 in Fig. 6) by the navigation system 2.

The driver typically does not know where the road-side cameras C1 are positioned and in which direction they are pointing. The object 3, which is displayed on the display 4, thus indicates one or more of the following:

• F1: a viewing direction 102 of a corresponding road-side camera C1 (cf. Fig. 3),
• F2: a line of sight of a corresponding road-side camera C1,
• F3: field of view 101 of a corresponding road-side camera C1 (cf. Fig. 2),
• F4: a range 103 of a corresponding road-side camera C1 (cf. Fig. 4).

This feature or these features F1 - F4 is/are important since the driver should not be distracted when selecting the right camera C1. In addition, it is important that the driver is able to match, relate or connect the information 13 displayed on the display 4 to the real world. Due to the fact that the object 3 includes one or more of the features F1 - F4, it is easy to interpret the visual information provided by the road-side camera C1.

The object 3, which is displayed on the display 4, may be temporarily magnified if the driver touches or pre-selects the object 3. This feature can be used in connection with all embodiments.

Fig. 11 shows several different objects 3.1 through 3.5 which could be used to show on a display 4 the position of a road-side camera C1. Objects 3 are preferred where the object as such indicates a viewing direction. It is obvious when looking at the different objects 3.1 - 3.5 in Fig. 11 that all of them are pointing to the right hand side. These objects 3.1 - 3.5 are preferred. They can be used in connection with any of the embodiments.

All embodiments of the invention may comprise additional features, objects or means which help the driver to "read" an image 13. The following features could be used as single feature or two or more of these features can be combined:

• F5: when reaching the position of the road-side camera C1 a visual or audible cue can be provided by the navigation system 2.
• F6: when reaching the zone or area which is within the viewing field of a roadside camera C1, the viewing direction 102 and/or the line of sight and/or the field of view 101 and/or the range 103 is/are highlighted or put in the foreground on the display 4.
• F7: In a 3-D navigation system 2 or in a navigation system 2 using real images, the image information 13 (at least the dynamic content) is mapped onto the 3-D image or real image (called overlay mode).

The invention is facilitated by a number of measures which have been taken in order to control the bandwidth or to keep it low. The respective measures are listed below. The actual embodiments of the invention may comprises one or more of these measures.

The most important inventive measures M1 through M6, which help to keep bandwidth requirement low, are listed below:

• M1: Real-time image information I1 is only offered locally, that is a driver can only obtain images of cameras C1 which are along his route R1 or within a certain local area. This measure M1 is preferably combined with an active route finding process being carried out by the navigation system 2.
• M2: In addition to any of the other measures or instead of the other measures, a distinction is made between static (motionless) image content and dynamic or quasi dynamic image content (i.e. changing content). This can be done by means of software-implemented edge detection for instance, which ensures that only limited image data are transmitted for static portions or areas of an image. These data are then used in the vehicle 1 to build or reconstruct an image.
• M3: In addition to any of the other measures or instead of the other measures, software or hardware-based techniques (such as data compression) are employed which ensure that only limited image data or a reduced data volume are transmitted. These data are then used in the vehicle 1 to build or reconstruct an image;
• M4: In addition to any of the other measures or instead of the other measures, software or hardware-based techniques are employed which ensure that dynamic or quasi dynamic image content is transmitted on request only.
• M5: In addition to any of the other measures or instead of the other measures, software or hardware-based techniques are employed which ensure that real-time image information is transmitted on request only. In this case either the user of the inventive system or the system itself requests real-time image information relevant for or related to a certain route R1.
• M6: A distinction can be made between road-side cameras C2 which currently show important traffic information and road-side cameras C3 where no relevant information is available. This can be achieved either in that an automated software-based process run by the remote system 100 or an operator distinguishes relevant information from not relevant information. The automated software-based process either uses a pattern recognition scheme, or it processes additional information. The additional information can be provided by radar cameras, induction loops integrated into the road, motion sensors and other auxiliary units. The object 3 could for instance be highlighted using a special color scheme (e.g. a green object versus a red object), or the shape of the object 3 could change. A highlighted object 3 would than indicate that there is relevant image information available.

The real-time image information I1 can be sent through different communication channels from the remote system 100 to the vehicle 1. In the following a distinction is made between push approaches and pull approaches. In both cases the remote system 100 is considered to be the source and the vehicle 1, respectively the inventive system 300 in the vehicle 1 in considered to be the destination or receiver. The real-time image information I1 is not necessarily sent from the remote system 100 right to the vehicle 1. It is also conceivable that there are systems (relays, routers, switches etc.) in-between, such as a computing environment of a mobile phone company or a server of a specialized service provider.

The following embodiments are based on a series of assumptions. A typical high resolution still image provided by a road-side camera C1 provides still images which have a size of about 1 MB or more. The ideal size of a still image is for the purposes of the present invention considered to be between 4 kB and 500 kB. If one assumes that in a certain city there are 50 road-side cameras C1 - C50 and that the images are processed (e.g. by means of compression or by a separation of real-time image data or dynamic image content from static image content) so that they have 10 kB each, a total of 50 times 10 kB is to be broadcast, if all inventive systems 200, no matter where they are in the city, are to receive the images of all 50 road-side cameras C1 - C50. If pictures are obtained from the cameras C1 - C50 or remote system 100 once per minute, then in a broadcast push approach 500 kB per minute have to be transmitted to all vehicles 1 in a certain region. In this example every system 200 at least in this region receives new images of all cameras C1 - C50 once per minute. The corresponding transmission frequency can be higher or lower.

The road-side camera C1 should have a resolution which is better than 0,2 megapixels and preferably more than 0,5 megapixels. This rule applies to all embodiments.

In a preferred embodiment it is possible to compress, filter or process the image data at the camera side or at the remote system 100 so that the size of the images gets smaller (reduced data volume). It is, however, to be kept in mind that features of an image when displayed in the vehicle 1 have to be visibly resolved. Such a compression scheme can be applied to all embodiments.

In a preferred embodiment it is possible to process the image data at the camera side or at the remote system 100 so that license plates or the faces of people are blanked out.

Even more preferred is an embodiment where a separation of real-time image data or dynamic image content from static image content is carried out at the camera side and/or at the remote system 100 so that relevant image information is always made visible whereas less important (e.g. static) image information is less well visible on a display 4. Such a scheme can be applied to all embodiments.

Even more preferred is an embodiment where the road-side camera carries on optical filter or screen in order to block information which is not considered relevant or which for legal reasons has to be blanked out. A corresponding illustration is provided in Fig. 12. The hashed field 400 represents a filter attached to a road-side camera. This approach also helps to reduce the bandwidth requirements.

The dynamic image content is what matters the most in the context of the present invention since the view of a road crossing, for instance, which does not show any moving objects (such as pedestrians or cars) is not as interesting to the driver of the vehicle 1 as an image which shows for instance the stop-and-go of vehicles in front of a traffic light. This is one of the reasons why the measure M2 is considered advantageous.

An example of a possible implementation or embodiment of the measure M2 is schematically illustrated in the sequence of Figures 10A through 10D. A corresponding flow chart is presented in Fig. 7. Fig. 10A shows the real traffic situation similar to the situation of Fig. 5A. The corresponding image information I1, including details such as windows, doors, trees, pedestrians and the like, can be transmitted from the camera C1 to the remote system 100. An image compression, e.g. a separation of static and dynamic content (measure M2) or a regular data compression (measure M3), could also be carried out by the camera C1 or by a module attached to the camera C1. More preferred is an embodiment where the image compression is carried out by the remote system 100 or by a special hardware and/or software module of the remote system 100 (steps S1.1 and S.1.2, Fig. 7). The principle on which measure M2 is based, is schematically illustrated in Fig. 10B and 10C. Fig 10B shows the static image content which is here reduced to some very basic shapes and elements, such as road markings, outlines of buildings and landmarks, for instance. In the present example elements and features (such as plants, windows, doors, etc.) which are not considered important are removed. Fig 10C shows the dynamic image content, such as cars, traffic lights, changing traffic signs, pedestrians, etc.

The transmission of the static image content does not require much bandwidth. In a vector based system it would be sufficient to just transmit vector information for lines and edges. In one embodiment of the invention the respective static image content is actually transmitted from the remote system 100 to the navigation system 2. In another embodiment of the invention the respective static image content is stored in the navigation system 2 (e.g. using a CD ROM or another storage medium). In this case the static information is not required to be transmitted.

In both embodiments the dynamic content is transmitted to the navigation system 2. The system 2, or a special module attached thereto, maps the dynamic content onto the static content (step S1.3, Fig. 7), no matter whether the static content is locally available or transmitted from the remote system 100. Fig. 10D shows the display 4 with a "reconstructed" image where static and dynamic content has been merged.

In addition or instead of any of the other measures M1, M3 - M6, a distinction is made between static (motionless) image content and dynamic or quasi dynamic image content (i.e. changing content), as described above in connection with Figures 10A - 10D.

A flow chart is presented in Fig. 7 where the static and dynamic image content are separately handled and transmitted (steps S1.1 and S1.2, Fig. 7). The static and dynamic image content are merged or combined by the navigation system 2, or by a module attached or connected to the system 2 (step S1.3, Fig. 7).

A flow chart is presented in Fig. 8 where the static image content is not transmitted because it is available at the navigation system 2 (step S2.1, Fig. 8). The static image content could be retrieved from a local storage medium, for instance. The dynamic image content is separately handled and transmitted (steps S2.2 and S2.3, Fig. 8). The static and dynamic image content are merged or combined by the navigation system 2, or by a module attached or connected to the system 2 (step S2.4, Fig. 8).

In a preferred embodiment, instead of broadcasting all images to all systems 300 in a general push approach, one could divide the whole city area into smaller cells (herein called cellular or smart push approach). In this case only images of road-side cameras (e.g. the cameras C1 - C10) within a particular cell are transmitted to systems 300 inside the cell or close to this cell. This helps to reduce the overall bandwidth requirement drastically.

In another preferred embodiment only images of road-side cameras (e.g. the cameras C5 - C10) are transmitted to systems 300 inside a vehicle 1 where a route has been defined or programmed in the navigation system 2 (i.e. if a route finding process of the navigation system 2 is active) which is passing these cameras C5 - C10. This helps to reduce the overall bandwidth requirement drastically. This means that a driver who is driving from location A to location B using the navigation system 2 and following the route R1, as shown in Fig. 9, would only be enabled to request and receive image information 12 from the road-side camera C2. Image information 12 from the road-side camera C3 is only requestable by a driver whose route passes by the position of the road-side camera C3.

In a further preferred embodiment the cell or micro-cell structure of a cellular mobile phone network is used in order to determine whether a system 200 currently is in a certain cell and whether there are any cameras (e.g. the cameras C1 - C10) in the same or in a neighboring cell which are providing real-time image information I1 which could be useful to the driver of a certain vehicle 1. This approach could be used together with the cellular or smart broadcast push approach, or it could be used in connection with a pull approach. According to this pull approach, a system 200 inside or close to a certain cell is enabled to request real-time image information I1 from a certain camera C1 - C10 within or close to the same cell, e.g. by using the systems user interface.

The pull approach is in another embodiment used without making a pre-selection or the like using the current vehicle's position inside a cell. This approach is herein called user specific pull approach. Here the user is enabled by a software module of the system 300 to request real-time image information I1 provided by a certain camera CX, no matter where he or his vehicle 1 is located. This real-time image information I1 is directly or indirectly requested from the remote system 100 and sent to the user, vehicle 1 or system 300 using a dedicated downlink, e.g. a mobile phone connection.

With current radio broadcast methods in some countries traffic information is transmitted (e.g. using the Traffic Message Channel: TCM) to radio receivers. TMC is a specific application of the FM Radio Data System (RDS) used for broadcasting real-time traffic and weather information. The TCM information is used in order to allow a dynamic route calculation in case of traffic jams and the like. Real-time image information I1 could be transmitted together with radio signals in a broadcast fashion, but this service should be limited to local radio transmitters, because it would not make much sense for a vehicle 1 in one city to receive images from cameras in other cities.

With the deployment of digital radio, high bandwidth channels become available in particular in urban areas. Digital radio describes radio communications technologies which carry information as a digital signal. Since a digital modulation method is used for the transmission, the vehicle 1 or navigation system 2 in this case comprises a digital demodulator in order to be able to receive and process the digital signals. The digital radio service could be used to transmit real-time image information I1 in a broadcast fashion.

It is also possible to use mobile phones and stationary transmitters to transmit the real-time image information I1. For this purpose either a regular phone is programmed to receive the information I1, or a separate communications module (e.g. comprising a SIM card) is implemented inside the inventive system 300 or is connectable to the system 300. Such an approach is preferably being used for realizing the cellular or smart broadcast push approach or the user specific pull approach.

According to the invention, image data messages (real-time image information 12) are received silently and decoded by a car radio, a mobile phone, a PDA, a smart phone or a navigation system 2, and delivered (e.g. made visible) to the driver in a variety of ways.

The system 300 in all embodiments includes a display 4 or other means or indicators which can be dedicated or shared with the ones already existing in the vehicle 1. Besides visual indicators the navigation system 2 may also include audible means or other means to inform the driver.