[0001] The invention relates to a method and a system for the detection of railway signals
relevant for the rail track that is currently used by a train.
[0002] More particularly, the invention relates to a method and system for automatically
detecting, whether a signal is present and relevant for the rail track that is actually
used by a train. If needed, determination of the actual state of the signal, e.g.
a stop (typically red), a go (typically green), or another state, is possible after
the relevant signal has been detected.
[0003] For the purposes of this application the term "signal" is defined and shall be understood
as railway signal.
[0004] The overall method can be used to assist a train driver to find the relevant signal
for the respective train and to avoid human failure by misinterpretation of signals.
Such misinterpretation can occur in particular on railway networks having many signals
for different rail tracks that are visible at the same time. Alternatively, the method
can also be used to provide derived and relevant signal information for automatic
train operation.
[0005] When driving a train through a region of a railway network with multiple parallel
rail tracks there is an increased probability that the driver of the train wrongly
identifies a signal, which is relevant for the current rail track of the train. Such
misinterpretation may lead to unnecessary stops of the train, or emergency brake application
or even accidents when e.g. a stop signal is ignored.
[0006] The identification of the relative position of the signal with respect to the rail
track as well as the determination of the state of the signal is of high importance
also for automatic train operation.
[0007] For this reason, one objective of the invention is to automatically detect the signals
associated with the actual rail track currently being used by the train, in particular
in order to assist the driver or for automatic train operation.
[0008] Currently, in the field of the invention there are many different types of methods
available to provide information about rail tracks using three-dimensional images
of the scene surrounding the train. Such methods have the drawback that they require,
for instance, stereo imaging sensors; using such sensors further implies a complex
and thus error-prone installation and calibration process to be carried out during
assembly. In addition, three-dimensional sensor systems lose their benefits as the
distance of the depicted objects from the sensors increases beyond a specific limit;
however, being beyond such limits might be the case for practical application of the
invention.
[0009] For the use of the invention, however, it is sufficient to have a standard two-dimensional
camera, e.g. an area scan camera as image sensor, which produces two-dimensional images.
Such image sensors are widely available, easy to assemble and in general simpler and
more reliable than other systems for three-dimensional environmental perception, e.g.
LIDAR-sensors.
[0010] It is therefore the objective of the invention to provide a simple and robust method
for the identification of signals that are associated with the rail track currently
used by a train, in particular only based on an image generated by a monocular image
sensor.
[0011] The invention solves this problem using a method of claim 1 for detecting a signal,
which is associated with the rail track that is used by a train; this method comprises
the following steps:
- a) taking at least one two-dimensional image from a train with an image sensor being
located in or on the train generating an image of the on-coming rail tracks and signals,
- b) using a classification method for identifying image regions corresponding to rail
tracks and image regions corresponding to signals within the image,
- c) among said identified image regions corresponding to rail tracks, selecting the
image region, which corresponds to the rail track that is used by the train, in particular
by identifying the rail track
- going through a predefined region of the image, and/or
- having the maximum area or pixel size among the rail tracks identified in step b),
- d) for each of the image regions corresponding to a signal:
- estimating projected points of the signals and/or the projected path of the selected
rail track into a common projection plane, in particular a ground plane, preferably estimating the position of a ground point
of the respective signal within the image,
- e) estimating a perspective transformation to transform the image into the common
projection plane, preferably into a bird's eye view image, based on visual properties of the image and/or mounting properties of the image sensor,
- f) applying the perspective transformation
- to the selected image region or parts thereof, in particular a central path within
the rail track, and
- to the projected points, and
- g) determining one or more projected points that have minimum distance and/or a distance
lower than a predefined threshold distance and/or relative placement to the selected
image region under said perspective transformation and identifying the signals and/or
image regions corresponding to signals associated with these projected points within
the image. The criteria for determining said association may be governed by regulatory
constraints and/or constraints defined by the railway network operator.
[0012] The invention further solves this problem using a system of claim 3.
[0013] Such a system is designed to detect a railway signal being associated with the rail
track that is used by a train. The system comprises:
- an image sensor mounted or mountable in or on the train that is arranged to take at
least one two-dimensional image of the on-coming rail tracks and signals, and
- a processing unit that is connected to the image sensor that is programmed to execute
the following steps:
b) using a classification method for identifying image regions corresponding to rail
tracks and image regions corresponding to signals within the image,
c) among said identified image regions corresponding to rail tracks, selecting the
image region, which corresponds to the rail track that is used by the train, in particular
by identifying the rail track
- going through a predefined region of the image, and/or
- having the maximum area or pixel size among the rail tracks identified in step b),
d) for each of the image regions corresponding to a signal:
- estimating projected points of the signals and/or the projected path of the selected
rail track into a common projection plane, in particular a ground plane, preferably estimating the position of a ground point
of the respective signal within the image,
e) estimating a perspective transformation to transform the image into the common
projection plane, preferably into a bird's eye view image, based on visual properties of the image and/or mounting properties of the image sensor,
f) applying the perspective transformation
- to the selected image region or parts thereof, in particular a central path within the rail track, and - to the projected points, and
g) determining one or more projected points that have minimum distance and/or a distance
lower than a predefined threshold distance and/or relative placement to the selected
image region under said perspective transformation and identifying the signals and/or
image regions corresponding to signals associated with these projected points within
the image.
[0014] The criteria for determining said association may be governed by regulatory constraints
and/or constraints defined by the railway network operator.
[0015] The invention can be used when a train is on a railtrack. Signals which are assigned
to or associated with the respective rail track are typically placed at predefined
positions with respect to the rail track. These positions are usually determined by
the rules of a signal regulation and/or the operator of the railway network. For example,
these predefined positions of the signals may be on the right-handside of said rail
track. Therefore, the respective states of these signals being located on this predefined
position with respect to the rail track are valid for the respective train.
[0016] Additionally, signals, which are not placed at predefined positions and are not matching
other association criteria as defined in the governing regulations for a given rail
track, are not relevant for trains on this rail track. By using the invention human
failure is avoided or minimized. In particular, situations in which the driver ignores
a signal relevant to the train, or in which the driver incorrectly considers the signal
of another rail track relevant, can be avoided. In addition, the derived and relevant
signal information from the image region containing the signal is of particular use
for the automatic train operation.
[0017] The following steps can be used to further avoid false positive detections of signals
that are not relevant for the actual rail track. These steps comprise:
- providing a map of the railway network including the positions of rail tracks and
signals, wherein each signal is associated with a rail track, and
- localizing its position on the railway map and in particular the rail track it is
using,
- using the map for determining the position of the on-coming signal associated with
the rail track it is using, and
- only if the on-coming signal is within a predetermined region in front of the train
taking an image and carrying out steps a) to g) of claim 1.
[0018] For the same reason, the system according to the invention can be improved by a geolocation
unit connected to the processing unit, wherein
- the processing unit further comprises a memory for a map of the railway network including
the positions of rail tracks and signals, wherein each signal is associated with a
rail track, and
- the processing unit is further programmed to execute the following steps:
- finding the position of the train provided by the geolocation unit within the railway
map and in particular the rail track it is using,
- using the map for determining the position of the on-coming signal associated with
the rail track it is using, and
- only if the on-coming signal is within a predetermined region in front of the train
taking an image and executing steps b) to g) of claim 3.
[0019] The following detailed description of preferred embodiments of the invention is merely
exemplary in nature and is not intended to limit the invention or the application
and uses of the invention. The invention will be described with reference to the following
figures which are provided by way of explanation only.
Fig. 1 depicts a typical image taken from a train.
Fig. 2 schematically shows the detection of rail tracks and signals.
Fig. 3 schematically shows one example for the calculation of the position of the ground
points of the signals within the image.
Fig. 4 schematically depicts a perspectival image transformation of the image of Fig. 1
to a bird-eye's view image.
Fig. 5 shows a map as used in an advantageous second embodiment of the invention using geolocation
information of the train and the signals.
[0020] In a first step (a) an image sensor, which is located in or on the train and which
is at least partly directed in the driving direction of the train, i.e. the direction
of travel or movement of the train, takes two-dimensional images I
t of the surrounding area, in particular of the rail track and of the signals. Such
image sensors can be simply mounted in the front part of the train.
[0021] One example of such an image I
t is shown in
Fig. 1. The image contains rail track-regions T
1, ..., T
4 each of which depicts one rail track. Moreover, the image I
t contains signal-regions S
1, ..., S
4, each of which shows one signal.
[0022] The image of the image sensor is transmitted to a processing unit, which is used
to further execute the following steps (b) to (g).
[0025] In both cases a classification method is used in order to identify the regions T
t corresponding to rail tracks and the regions S
t corresponding to signals in the image I
t taken by the image sensor. However, from this result it is still not possible to
determine, which signals are relevant for the driver or for the rail track that is
actually used.
[0026] In order to determine the relevant signals, the position of the actual rail track
within the image I
t is determined. Therefore, the regions of the current rail track, i.e. the rail track
T
c that is actually used by the train, is identified.
[0027] The rail track T
c used by the train and the respective region of the rail track T
c within the image I
t can be found by searching one of the rail track regions T
t containing rail tracks already identified in step (b) that goes through or overlaps
with a predefined image area A
p as shown in
Fig. 2. This image area A
p usually depends on the orientation of the image sensor within the train. If an image
sensor is heading straight in the direction of movement, the predefined image area
A
p is in the center of the lower part of the image I
t. The image area A
p can be easily defined after mounting the image sensor in or on the train by searching
for a region of the image, which is close to the train and which shows a part of the
rail track, independently of the curvature of the rail track.
[0028] Alternatively, it is also possible to search for the regions T
t having the maximum area or containing maximum pixels among the rail tracks identified
in step (b). As the rail track currently used is close to the image sensor, it typically
covers a bigger region within the image I
t than the other rail track regions.
[0029] In this preferred embodiment of the invention the rail track identified in step (c)
is represented by a center path P
c.
[0030] In order to determine the signals relevant for the identified rail track T
c in step (d) the positions of the ground points G
1, ..., G
4 of the signals S
T are estimated with respect to the image I
t. As many signals are mounted on poles the ground point G
1, ..., G
4 is the point within the image that shows the base point of the pole. More generally,
the ground point is the point within the image I
t that depicts a surface point, which is directly underneath the respective signal
S, even if the signal is not mounted on a pole, but e.g. mounted on overhead infrastructure.
[0031] In order to determine the ground point G
1, ..., G
4, the following prerequisites are typically needed. Firstly, the size, i.e. height
and width, of the panels of the signals are known. If there are different types of
signals available, said type has to be firstly estimated by the shape and/or content
of the image region of the signal.
[0032] For instance, a typical kind of signal could have a rectangular panel of 1.4-meters
height and 0.7-meters width. Moreover, the signal could typically be mounted in a
height of 4.85-meters to 6.95-meters (see
Fig. 3).
[0033] Moreover, it is assumed that the image sensor is mounted on the train in a manner
that its y-axis is - at least approximately - aligned vertically, and its x-axis is
- at least approximately - aligned horizontally.
[0034] The height h
t and width w
t of the image region of the signal can be obtained by different methods, such as detection
of an axis-parallel bounding box, etc.
[0035] Alternatively, it is also possible to determine the pixel height h
t and width w
t of the signal's image region S
t as follows: As already mentioned, the ratio r between the height and width of the
panel of the signal are known initially, for instance in the above case 2:1. In the
following, the number of pixels A corresponding to the signal's image region S
t is counted. By using the relation A = w
t∗h
t = w
t∗w
t∗r the pixel-width w
t of the image region S
t of the signal can be determined easily. Accordingly, the pixel height h
t of said signal region S
t can be determined.
[0036] As the pixel height h
t corresponds to the known height h
t of the signal, one can easily calculate the y-distance d
t from the lower edge of an image region S
t to the respective ground point G
t. Given that the image region S
t has a pixel-height of 56 px as well as the above specifications for signals the y-distance
d
t between the lower edge of the image region S
t and the ground point G
t can be calculated as:

[0037] This means, that the ground point is at a position between 194 px and 278 px under
the lower edge of the signal's image region S
t. To simplify calculations, an average pole height, for example of 6-meters can be
used, which would lead to a y-distance d
t of 240 px.
[0038] In order to define the ground point's coordinates, one has to find the lower edge
of the signal region as well as the horizontal center of the signal region. The lower
edge position y
l,t can be defined as the highest or least y-position, depending on the orientation of
the y-axis of the image sensor. The horizontal center x
m,t of the signal region can be defined as the x-coordinate of the centroid of the signal
region. The position of the ground point G
t can be defined as:

[0039] In general, the ground point G
t can be determined by calculating an offset from an image position within the region
of the signal S
t, wherein the offset is defined by the size, in particular by a length on the region
of the signal S
t. For example, the ground point G
t can be determined by calculating width and height of a rectangular bounding box that
is parallel to the image axes.
[0040] There could be other possible ways to determine the ground point of a signal. For
example, using the known gauge-width of the rail track and the signal panel's dimensions
(i.e. real height and width), one could estimate the ground point of the signal in
the two-dimensional image by comparing the signal panel's pixel-width to the gauge's
pixel-width of the gauge of the rail track, both scaled in an appropriate manner.
In order to determine correctly scaled pixel-heights and/or pixel-widths it is advantageous
to consider the image's perspective distortion and the orientation of the rail tracks.
[0041] Even if the ground points G
t were determined within the image I
t, it is still not possible to determine the signal which is closest to the actual
rail track directly from the image, because the image as taken by the image sensor
is distorted, i.e. distances within the image are not directly related to the distances
of the objects depicted in the image.
[0042] As shown, the position of a ground point G
t underneath the signal was estimated from the image by projection of the signal within
the image I
t to the ground plane.
[0043] Instead of a projection to a ground plane, it is also possible to use any other plane
as projection plane. In this case the path P
c of the selected rail track and the position of the signals are projected into this
plane.
[0044] For this reason, in a subsequent step (e) a perspective transformation R is determined,
by which it is possible to transform the sensor image I
t into the projection plane; as this projection plane is - in this very example - parallel
to the ground plane, the transformation returns a bird's eye view image I
B. As an example, the transformation R can be a perspective transformation, that is
based on visual properties of the image and/or mounting properties of the image sensor.
[0045] One particular method to determine the transformation R is to find a trapezoidal
region R
T within the region Tc of the actual rail track. Based on the mounting properties of
the image sensor and the assumption of a sufficiently flat and smooth track bed it
is clear, that this trapezoidal region R
T typically covers a rectangular region R
R within a bird's eye view image I
B. Even if the rail track is curved, there is a region close to the train, which is
sufficiently straight, to obtain an approximate trapezoid structure. The perspective
transformation is defined in such way, that it transforms at least a trapezoidal part
of the region T
c of the actual rail track to a rectangular region within the bird's eye view image
I
B.
[0046] Even if it is not required to transform the overall image I
t into the bird's eye view image I
B, said image I
B is shown for the sake of clarity in
Fig. 4. In order to determine the signals that are assigned to the actual rail track that
is used by the train, the perspective transformation R is in a further step (f) applied
to the selected rail track, in particular to a central path P
1, ..., P
4 within the rail track, and to the ground points that were determined in step (d).
From these data it is possible to determine real-world distances, or it is at least
possible to compare distances of ground points to the actual rail track.
[0047] In order to find the relevant signals, which are associated with the used rail track,
one can detect the ground point G
t that has the least distance compared to the other ground points G
t under said perspective transformation R. Alternatively, or in addition, a ground
point G
t and its respective signal S
t should be associated with the actual rail track only if the distance from the ground
point G
t to the region, in particular the path P
c, of the selected rail track S
t under said perspective transformation R is below a predefined threshold distance.
[0048] In general, it is also possible, that a regulation sets out more complicated rules
relating to the association of signals with rail tracks. These regulations can rely
on the determined distances and/or relative placement, e.g. right or left from the
rail track, only. However, it is also possible that there are more complex rules relating
to the association of rail tracks and signals that are based on a sequence of detections
or the position of the train and the signal in the railway network. These rules may
rely on distances, relative placement and sequences of signals on the rail track.
[0049] An advanced
second embodiment of the invention uses the method as described above in order to determine images
of signals that are associated with the actual rail track. In addition, this embodiment
also relies on a geolocation-map, containing the location of the rail tracks t
A, ..., t
G as well as the position S
A, ..., S
L of the signals (
Fig. 5).
[0050] For this second embodiment a geolocation-unit is used, the geolocation-unit being
connected or integrated to the processing unit. The geolocation-unit provides the
actual geolocation of the train which is further processed by the processing unit
as follows.
[0051] When the train is on the rail track t
C, in particular moves on the rail track, its geographical positions (depicted by dots
in
Fig. 5) is determined using geolocation methods, such as Global Navigation Satellite System
(GNSS) or Simultaneous Localization And Mapping (SLAM). As a result, the actual position
of the train, which is specified by coordinates, is known at each time. Using said
coordinates as well as the map containing the location information of the railway
network, it is possible to determine the actual rail track of the train independently
from the visual method shown in the first embodiment of the invention. The map of
the railway network is preferably stored on a memory connected to the processing system.
However, in order to precisely localize the signal as well as the state of the signal
the invention of the first embodiment is used. Once the train starts its journey towards
its destination, its route or driving direction DD, depicted by an arrow in
Fig. 5, is already defined and tracked in real-time using the geolocation unit.
[0052] If the map of rail tracks, signals and the driving direction DD are known, it is
possible to determine the positions S
A, ..., S
L of signals associated with the respective rail track. Therefore, it is also possible
to determine, if - within a predefined region E
1, ... E
3 calculated from the actual position of the train e
1, ..., e
3 - a signal is expected to be seen in the image. Such a signal is also referred to
as on-coming signal.
[0053] The method for detecting image regions of signals, which are associated with the
actual rail track as shown in the first embodiment of the invention only needs to
be carried out, if such a signal is expected based on the geolocation method as explained
above. If, such as for the position e
3, there is no on-coming signal within the associated region E
3, the visual search for signals is turned off in order to reduce the likelihood for
false positive matches.
[0054] For the position e
1 the associated region E
1 contains many ground points S
F, S
G, S
I, S
J. However, none of these ground points are associated with the actual rail track t
C so that there is no on-coming signal; the method for detecting image regions of signals
is turned off.
[0055] As the position e
2 and its associated region E
2 contain a signal S
D that is associated with the actual rail track t
C, it is to be expected, that there is an on-coming signal S
D. Therefore, the method for detecting image regions of signals is carried out in order
to find a visual representation of said signal in the image I
t.
[0056] Alternatively, or in addition, the information relating to the map positions of signals
can be used in order to further avoid false positive detections of signals with respect
to both further described embodiments of the invention. A matching algorithm is used
in order to match the rail tracks t
A, ..., t
G of the map to the rail tracks as contained in bird's eye view image I
B. If a selected ground point G
t matches the geolocation of a signal of the predefined map, which is associated with
the actual path as determined by the geolocation, the overall reliability of the result
will be increased.
[0057] For all preferred embodiments of the invention it is possible to display the image
region containing an image of the identified signal to the train driver. Alternatively,
it is also possible to highlight said image region when showing the overall image
to the driver. Typically, the procedure as described above is repeated regularly so
as to provide a video in order to inform the train driver about relevant on-coming
signals.
[0058] Alternatively, or in addition, any other event can be triggered by the information
automatically retrieved from the image such as simple audible, visual, or haptic warning.
[0059] Of course, it is also possible to feed the image data of the identified image region
of the relevant signal to an image processing algorithm in order to extract the state
of the signal. This state can be used to assist the train driver, e.g. to brake the
train accordingly, when a stop signal is identified or even for automatic train operation.
1. Method for detecting a railway signal, which is associated with the rail track that
is used by a train,
a) taking at least one two-dimensional image (It) from a train with an image sensor being located in or on the train generating an
image of the on-coming rail tracks and signals,
b) using a classification method for identifying image regions (Tt) corresponding to rail tracks and image regions (St) corresponding to signals within the image (It),
c) among said identified image regions (Tt) corresponding to rail tracks, selecting the image region (TC), which corresponds to the rail track that is used by the train, in particular by
identifying the rail track
- going through a predefined region of the image, and/or
- having the maximum area or pixel size among the rail tracks identified in step b),
d) for each of the image regions (St) corresponding to a signal:
- estimating projected points of the signals and/or the projected path of the selected
rail track (TC) into a common projection plane, in particular a ground plane, preferably estimating the position of a ground point
(Gt) of the respective signal within the image (It),
e) estimating a perspective transformation (R) to transform the image (It) into the common projection plane, preferably into a bird's eye view image (IB), based on visual properties of the image and/or mounting properties of the image sensor,
f) applying the perspective transformation
- to the selected image region (TC) or parts thereof, in particular a central path within the rail track, and
- to the projected points, and
g) determining one or more projected points (Gt) that have minimum distance and/or a distance lower than a predefined threshold distance
and/or relative placement to the selected image region (TC) under said perspective transformation (R) and identifying the signals and/or image
regions (St) corresponding to signals associated with these projected points within the image
(It).
2. Method according to claim 1,
characterized by the following steps:
- providing a map of the railway network including the positions of rail tracks and
signals, wherein each signal is associated with a rail track,
- localizing its position on the railway map and in particular the rail track it is
using,
- using the map for determining the position of the on-coming signal associated with
the rail track it is using,
- only if the on-coming signal is within a predetermined region in front of the train
taking an image and carrying out the method of claim 1.
3. System for detecting a railway signal, which is associated with the rail track that
is used by a train, comprising
- an image sensor mounted or mountable in or on the train that is arranged to take
at least one two-dimensional image (It) of the on-coming rail tracks and signals, and
- a processing unit that is connected to the image sensor that is programmed to execute
the following steps:
b) using a classification method for identifying image regions (Tt) corresponding to rail tracks and image regions (St) corresponding to signals within the image (It),
c) among said identified image regions (Tt) corresponding to rail tracks, selecting the image region (TC), which corresponds to the rail track that is used by the train, in particular by
identifying the rail track
- going through a predefined region of the image, and/or
- having the maximum area or pixel size among the rail tracks identified in step b),
d) for each of the image regions (St) corresponding to a signal:
- estimating projected points of the signals and/or the projected path of the selected
rail track (TC) into a common projection plane, in particular a ground plane, preferably estimating the position of a ground point
(Gt) of the respective signal within the image (It),
e) estimating a perspective transformation (R) to transform the image (It) into the common projection plane, preferably into a bird's eye view image (IB), based on visual properties of the image and/or mounting properties of the image sensor,
f) applying the perspective transformation
- to the selected image region (TC) or parts thereof, in particular a central path within the rail track, and
- to the projected points, and
g) determining one or more projected points (Gt) that have minimum distance and/or a distance lower than a predefined threshold distance
and/or relative placement to the selected image region (TC) under said perspective transformation (R) and identifying the signals and/or image
regions (St) corresponding to signals associated with these projected points within the image
(It).
4. System according to claim 3, further
characterized by a geolocation unit connected to the processing unit, wherein
- the processing unit further comprises a memory for a map of the railway network
including the positions of rail tracks and signals, wherein each signal is associated
with a rail track, and
- the processing unit is further programmed to execute the following steps:
- finding the position provided by the geolocation unit within the railway map and in particular the rail track it is using,
- using the map for determining the position of the on-coming signal associated with
the rail track it is using, and
- only if the on-coming signal is within a predetermined region in front of the train
taking an image and executing steps b) to g) of claim 3.