|
(11) | EP 2 498 246 A1 |
| (12) | EUROPEAN PATENT APPLICATION |
|
|
|
|
||||||||||||||||||||
| (54) | A display system and method for controlling a display |
| (57) A processing unit 24 is configured to receive a polychromic image input 16 and determine
at least one primary range of intensities for representing the primary image and at
least one secondary range of intensities for representing symbology. The primary range(s)
do not overlap with the secondary range(s) so that a viewer can view the primary image
in the primary range and the symbology in the secondary range. The processing unit
24 outputs to the display 12 a monochromatic image output 26 in which the primary
image is represented by intensities in said at least one primary range and the symbology
is represented by intensities in said at least one secondary range.
|
[0001] The present invention relates to a display system and method for controlling a display. In particular, the invention relates to the display of a monochromatic image generated from a source, or primary, image containing colour.
[0002] Displays are used in a multitude of different technical fields and applications, for example, military, medical and general scientific usage. There are various types of display, or graphical user interfaces, including CRT, LED, LCD and plasma. More recently, head or helmet mounted displays, or head-up displays, have found increasingly utility. These latter displays are generally referred to as augmented reality (AR) displays, in which a user can view a real world scene through the display combined with additional information or images superimposed on the display. AR displays typically comprise one or more waveguides at least one of which acts as a combiner combining a real world scene with additional images or information.
[0003] In many uses of a display, it can be necessary or desirable to over-lay, or superimpose symbology in order to highlight specific areas in real world space or convey current and historic data such as vital statistics to medical personnel.
[0004] A colour, or polychromic, display 100 is shown in Figure 1. The display may be any type of graphical user interface such as a monitor, for example, or a head up display, or head or helmet mounted display. The display may be capable of displaying colour images using for example the RGB color model which is an additive color model in which red, green, and blue light are added together in various ways to reproduce a broad array of colours. In the RGB model, a colour is given a value between 0 and 255 for each of its constituent primary colours so for instance a particular green may be R32, G128, B32 and a red may be R128, G32, B32. White is represented by R255, G255, B255 and black is R0, G0, B0. Other colour space models may also be used in the present invention for example Cyan Magenta Yellow and Key (black) referred to as CMYK, which is as subtractive colour model.
[0005] The colour display 100 is shown displaying a primary image consisting of a real world scene 102 including two trees on a hill side. The colour of the leaves on the tree will vary and may be R120, G170, B66 at one point of the image and R129, G189 , B27 at another point of the image. For certain uses of a display it may be required to superimpose graphical information, or symbology. For example, horizontal and vertical lines 104 and 106 are superimposed on the image 102 in Figure 1. In order to help a user view the lines, they may be coloured and in this example line 104 is green (R32, G128, B32) and line 106 is red (R128, G32, B32). The colour of each line is consistent and does not vary, unlike the colours of other objects in the image. The consistency of colour allows the lines to be easily seen.
[0006] In some circumstances, a monochromatic display may be preferred or required, for example, where the display is capable of displaying only monochromatic images or where a display is capable of displaying in colour but a monochromatic display is preferred. A waveguide display typically displays in monochrome, since the display of colour images by a waveguide can lead to chromatic aberration. In AR displays it can also be desirable to display in monochrome so that a viewer can easily distinguish between real world images and synthetically generated images.
[0007] It will be appreciated that a monochrome display is not well suited for the display of coloured graphical information. A monochrome display displays only one colour but is capable of displaying different intensities e.g. in grayscale for a black and white display, or green scale for example in the commonly adopted colour for AR displays. The contrast between intensities allows the user to differentiate between different aspects of the image. Figure 2 shows such a monochrome display 110 which is displaying the same primary image consisting of the real world scene and graphical information as shown in Figure 1. In Figure 2, the secondary, or monochromatic image, the lines 104, 106 lack clarity and are less easily distinguishable from the rest of the image and from each other. This is because a monochromatic display 110 may display similar colours as similar intensities or may even display different colours as similar intensities.
[0008] Whilst the present invention is not restricted to augmented reality displays, the use of head mounted displays have found recent utility in the field of medicine in which images are displayed to a surgeon, for example, during an operation. Such additional images may include X-rays, MRI and ultrasonography displayed to assist a clinician during their work.
[0009] By way of further example, a MRI scan is shown in Figure 3, in which a scan 112 of a patient's head is displayed on a polychromic display 120 with superimposed symbology in the form of datum lines 114 and 116. The colour lines 114 and 116 are easily distinguishable from the background image in the primary colour image. However, as shown in Figure 4, in a secondary image displayed on a monochrome display 122 the primary image is less easily distinguishable from the lines 114, 116.
[0010] In other known arrangements, information in the form of text or graphics may be superimposed over television images. An example of such an arrangement is where program information is displayed over visual images on a typical television broadcast or cable network. However, such arrangements are configured for the display of polychromic text or graphics over polychromic video images. Either the displayed text and graphics is difficult to read or the background image is difficult to see. The problem with these displays is that they do not enable text and graphics to be adequately viewed whilst at the same time retaining clarity of the video images.
[0011] The extant problem is that monochromatic images frequently make symbology indistinguishable from the entirety of or parts of the reproduced image. The symbology can therefore become less apparent from the original image in a reproduced monochrome image. Alternatively, the background images may become obscured for the sake of clarity of the symbology.
[0013] The present invention provides a method of controlling a display for displaying a monochromatic image from a polychromic image input, the polychromic image input comprising a primary image and symbology having a selected colour superimposed on the primary image, the method comprising: receiving said polychromic image input, determining at least one primary range of intensities for representing the primary image and at least one secondary range of intensities for representing the symbology, said at least one primary range not overlapping with said at least one secondary range, and displaying the monochromatic image by representing the primary image by intensities in said at least one primary range and by representing the symbology by intensities in said at least one secondary range.
[0014] The present invention also provides a display system comprising: a display for displaying a monochromatic image from a polychromic image input, the polychromic image input comprising a primary image and symbology having a selected colour superimposed on the primary image, and a processing unit configured to receive said polychromic image input, to determine at least one primary range of intensities for representing the primary image and at least one secondary range of intensities for representing the symbology, said at least one primary range not overlapping with said at least one secondary range, and outputting to said display a monochromatic image output in which the primary image is represented by intensities in said at least one primary range and the symbology is represented by intensities in said at least one secondary range.
[0015] Other preferred and/or optional features of the invention are defined in the accompanying claims.
[0016] In order that the invention may be well understood, some embodiments therefor, which are given by way of example only, will now be described with reference to the accompanying drawings, in which:
Figure 1 shows a known colour display;
Figure 2 shows a known monochromatic display;
Figure 3 shows another known colour display;
Figure 4 shows another known monochromatic display;
Figure 5 shows schematically a display system;
Figure 6 shows a monochrome display;
Figure 7 shows another monochrome display; and
Figure 8 shows schematically a method of displaying polychromic images on a monochrome display.
[0017] Referring to Figure 5, a display system 10 is shown which comprises a display 12 for displaying a monochromatic image 14 from a polychromic image input 16. The polychromic image input comprises a primary image 18 and symbology 20 having a selected colour superimposed on the primary image. The primary image may be any image for example of a real world scene or an image of an object, or patient, as described by way of example below. The primary image may be monochrome (e.g. grey scale or green scale) or polychrome (e.g. full or partial colour). The symbology is generated separately from the primary image and superimposed over the primary image in for example a display source 22 shown in Figure 5. The symbology generated is selected to be a particular colour, for example green R32, G128, B32 in an RGB colour model, and is generally consistent across the extent of each symbology. The consistency allows a viewer of the combined image more easily to distinguish the symbology from the primary image. There may be different symbology superimposed over the primary image and the different symbology may be generated in respective colours, for example green R32, G128, B32 and red R128, G32, B32 in the RGB model. The symbology may represent any useful information to a viewer such as datum lines, alphanumerics, scales, chronology.
[0018] The display may be any of the displays described above with reference to the prior art, for example a waveguide display or monitor. The display is configured for displaying intensities in a dynamic range which for convenience herein is referred to as 0% to 100% intensity. 0% may represent the lowest brightness and 100% may represent the highest brightness that the display can generate.
[0019] A processing unit 24 is configured to receive the polychromic image input 16 and determine at least one primary range of intensities for representing the primary image and at least one secondary range of intensities for representing the symbology. In the example described in more detail below, a single primary range and a single secondary range are generated. The primary range(s) do not overlap with the secondary range(s) so that a viewer can view the primary image in the primary range and the symbology in the secondary range. The processing unit 24 outputs to the display 12 a monochromatic image output 26 in which the primary image is represented by intensities in said at least one primary range and the symbology is represented by intensities in said at least one secondary range. The display 12 receives the monochromatic image output 26 and displays a monochromatic image having a dynamic range of intensities (image brightness). Since the primary image and symbology are in different ranges the contrast between the primary image and the symbology can be more easily distinguished than is the case with know arrangements.
[0020] The symbology intensity displayed in the or each secondary range is spaced apart, by a predefined or a dynamically generated unused intensity range, from the primary range to enhance in the monochromatic image the contrast between the displayed symbology and the primary image. In this regard, the unused intensity range is not used in the displayed image to provide improved contrast between the intensity in the primary range which is closest to the intensity of the displayed symbology.
[0021] The unused intensity range may be preselected based on the type of primary image and symbology that will be displayed in order to obtain appropriate contrast. For example, if the primary image will comprise predominantly blacks or other dark colours, it may be sufficient to provide a relatively small band of unused intensities in order to distinguish the relatively high intensity symbology from the primary image. If though the primary image will comprise whites, yellows and other bright colours a larger band of unused intensity may be required to distinguish the symbology from the primary image.
[0022] In another arrangement, the processing unit may be configured to determine the unused range by sampling the polychromic image input and dynamically generating an appropriate unused range. For example, the primary image may be sampled to determine its range of colour and brightness and dependent on said range the processing unit determines the unused bandwidth between the primary range and the symbology.
[0023] That is, a primary image may be sampled to determine unused intensity ranges that occur within the image naturally, and the symbology may be displayed at determined unused intensity. For example, a histogram of intensities of the primary image may be generated, and the symbology displayed at an intensity or intensities which is generally zero in the histogram.
[0024] A display 12 is shown in Figure 6 for displaying a monochromatic image 14 from a polychromic image input 16. The polychromic image input is the same as shown in Figure 1. The display is configured for displaying intensities from a low or zero intensity to a high intensity or full saturation. The primary range extends from a low intensity to an intermediate intensity between the low and high intensities, and the secondary range extends from the intermediate intensity to said high intensity. The consequence of this process is that, as will be appreciated from Figure 6, the primary image is displayed at a lower intensity (a shade of gray), whereas the symbology is displayed at a higher intensity (black in this example). In this way, the contrast between the symbology and the primary image is improved helping a viewer to view all aspects of the monochromatic image.
[0025] Another monochromatic image 14 is shown in Figure 7. The primary image 18 and the symbology 28, 30 are similar to that shown in Figure 3. However, in Figure 7, the primary image is relatively low in intensity consisting mostly of blacks and dark greys. Therefore, the secondary range is selected to be at relatively high intensity and the symbology is displayed as whites or light greys. This arrangement is clearly different from that shown in Figure 6 in which the symbology is displayed as blacks.
[0026] Referring to Figure 8, it will also be seen that the original symbology 28, 30 superimposed on the primary image in the display source 22 comprises two discrete colours, for example, blue and red. In the displayed monochromatic image, the symbology 28 is displayed at one intensity in the secondary range (e.g. 100% intensity) and the symbology 30 is displayed at another intensity in the secondary range (e.g. 90% intensity). The discrete symbology intensities in the secondary range at which the symbology is displayed are spaced apart one from another and from the primary range to enhance in the monochromatic image the contrast between the displayed symbology 28, 30 and between the displayed symbology 28, 30 and the primary image 18. For example as shown, the primary image intensity may be 0 to 80% and the first and second symbology intensities may be 90% and 100%. Therefore, the intensities are spaced apart by 10% of the available intensity range of the display.
[0027] In more detail, polychromic image data 16 is received from the display source by the image processor 24. The polychromic image input comprises image data corresponding to each picture element for reproducing the primary image and the symbology. The processing unit is configured to determine the colour of the picture elements and output to a symbology intensity in the secondary range for each picture element having the selected symbology colour and a primary image intensity in the primary range for each picture element not having said selected colour. Therefore, each picture element, or pixel, is sampled to determine its colour. The RGB colour model is discussed in detail herein but it will be appreciated that the invention relates to any other colour model, for example, colour may be defined simply as a wavelength in the visible spectrum. In the RGB colour model, the varying intensity from 0 to 255 of the colours red, green and blue define the displayed colour of a picture element. That is, each picture element may be represented by Rx, Gx, Bx, where x is a number from 0 to 255. The maximum intensity of each colour represents white and is defined by R255, G255, B255 and the complete absence of each colour, or zero intensity (representing black), is defined by R0, G0, B0.
[0028] A monochromatic image is represented on a scale of 0% to 100% intensity, or brightness, range where the difference in intensity between 0% and 100% constitutes the maximum contrast. According to known methods, a picture element Rx, Gx, Bx equates to an intensity z% where z is a number between 0 and 100. For example, a brown R75, G41, B33 is represented by 10% intensity and a yellow R252, G241, B46 is represented by 75% intensity.
[0029] In the present embodiment, instead of each colour in the primary image 18 be represented by an intensity from 0% to 100%, it is represented by an intensity in only part of the range, for from 0% to 80%.
[0030] Accordingly, any and all pixels from the primary image (Rx1, Gx1, Bx1) will be converted to a secondary image (Rx2, Gx2, Bx2) by scalar modification of intensity value. In Figure 8, the modification is a scalar reduction so that X2 is equal to 0.8X1. For example, where the primary image has a pixel constituted as R200, G100, B100, a scalar reduction of these intensities to 80% of their original values, will produce the secondary image pixel constituted as R160, G80, B80. Such a modification is performed for all pixels in the primary image excepting any specific and defined colour intensities of the symbology.
[0031] The modification may be made with the addition of a constant if a shift in the intensities is required over and above a reduction or an increase in the intensity range.
[0032] The polychromic image data 16 received from the display source also comprises symbology. The symbology typically occupies a plurality of adjacent picture elements and may be linear as shown in the Figures or may be alphanumeric. The colour of each symbology is constant, as opposed to the primary image 18, which will almost invariably comprise of variable colour(s). Accordingly, the image processor 24 locates those picture elements having the selected symbology colours and outputs monochromatic data 26 with those picture elements having an intensity in the secondary range. For example, symbology having a selected colour R200, G100, B100 will be assigned, in the secondary range, a constant intensity equal to 100% intensity (equivalent ordinarily to R255, G255, B255) or white. Preferably, the image processor 24 locates the target colours within a reasonable tolerance limit: for instance any pixels of the range R200±2, G100±2, B100±2 will be regarded as symbology. Alternatively, or additionally, the image processor 24 may identify symbology only if there are a plurality of picture elements in close proximity with the same colour thereby distinguishing from those isolated picture elements in the primary image which happen to have an exact match with the target colour. Still further, the image processor may store the expected shapes of the displayed symbology and determine those shapes in the image, for example, alphanumerics, circles, or lines.
[0033] If the image data 16 comprises further symbology in a second different selected colour e.g. R100, G200, B200, the image processor 24 determines the pictures elements with that colour and assigns to it a constant intensity, in the secondary range, equal to say 90% of the maximum intensity.
[0034] Further symbology having different discrete colours may be identified as required. Alternatively, all symbology may be displayed at the same constant intensity.
[0035] If it is required to display a multiplicity of symbologies having discrete colours at respective intensities in the secondary range, preferably the primary range is compressed to allow those symbology intensities to be spaced apart from each other and from the primary range to allow a viewer to readily distinguish one from another. For example, the primary range may usefully be reduced to 70% and the symbology displayed at intensities of 80%, 90% and 100% of the maximum. Generally, a spacing of about 10% intensity allows a person to distinguish the elements of the display. A spacing of 5% or possibly even lower may be acceptable however depending on the content of the displayed monochromatic image. For example, if relatively light symbology is displayed on a dark portion of the primary image then contrast may be sufficient even if the symbology is displayed at 71 % intensity and the primary range extends from 0% to 70%. The image processor 24 may be configured to determine in real time regions of dark (or light) intensity in the primary image and display the symbology at an intensity which allows it to be distinguished from the primary image.
[0036] The amount of compression of the primary image, or general reduction factor, is generally inversely rated to the number of target symbology colours, since the an increase in the number of discrete symbology colours in the secondary range requires a decrease in the primary range whilst providing optimal differentiation, given other constraints. Constraints include the clarity of the original or primary image required in the monochromatic image, since the general reduction of intensities for all pixels will serve to reduce the contrast inherent in the original image.
[0037] Where the general reduction factor = GRF, the number of target colours = TC and the optimal intensity separation between target colours in the secondary image is Isep:
[0038] In this way a viewer could clearly distinguish between not only the selected target colours (or symbology from primary image) and the background much more easily because of the enhanced contrast, but can also distinguish between multiple target colours by intensity.
[0039] The above more detailed description relates to display of symbology in a secondary range which is more intense than the primary range, as shown in Figure 7. However, referring again to Figure 6, where the original primary image is predominantly light itself, the target symbology colours are shown in darker intensities to become most apparent. In a modification of the above detailed description, the primary image is compressed and shifted by a constant (for example 20%). That is, the primary image may be compressed to a range of 0% to 80% and shifted by 20% to a range of 20% to 100%. Accordingly, if two target symbology colours are required, they can be represented at intensities 0% (R0, G0, B0) or black, and 10% (R26, G26, B26) or dark grey, and the primary image will be displayed in the intensity region 20%-100% (or RGB51-255). The constant to be added to intensity values reduced by the normal scalar process will simply be (100-GRF)% of maximum intensity.
[0040] Accordingly, the primary range may extend from a high intensity (e.g. 100%) to an intermediate intensity (e.g. 20%), and the secondary range may extend from the intermediate intensity to a low intensity (0%). Alternatively, the primary range may comprise a plurality of sub-ranges between the low intensity and the high intensity (e.g. 0% to 35% and 65% to 100%), and the secondary range may extend between the adjacent sub-ranges (e.g. 35% to 65%). This arrangement may be useful where the primary image contains picture elements of generally high and low intensity but there is relatively intensity in the intermediate range. Accordingly, the symbology can be displayed at intensities in the intermediate, generally unused, range.
[0041] The present invention has utility in a number of different fields, particular where the use of a monochromatic display is preferred or required. For example, a head-up display (or head or helmet mounted display) typically displays in monochrome. In head-up displays, a viewer must typically be able to view the primary image, the symbology and the real world scene through the display. The use of monochrome head-up displays allows the real world scene to be more easily distinguished from a single colour representing the primary image and the symbology, since the brain becomes tuned to the single colour of the display and will easily distinguish between virtual images in that colour and real images in the full visible spectrum.
[0042] The primary image may consist of one or more still images or video images reproducing a sequence of still image in succession. The displayed primary image and symbology may conform to the viewer's line of sight such that change of position or orientation of a viewer produces a commensurate change in the displayed images. Such a latter arrangement is suited for providing a viewer with additional information on head-up display on the real world scene viewed through the display. For example, the primary image may be generated by a heat sensitive camera for use by emergency services for locating people or other objects in a low visibility environment. The use of heat sensitive cameras are known however typically a viewer can either view the enhanced imagery through a viewer or the real world scene, but is not able to view one the enhanced image over the real world scene. The symbology in this example may take the form of georeferences, heading, temperature etc.
[0043] As shown in the Figures, the invention may be used in medicine for example by a surgeon executing minimally-invasive surgery, where the primary image may be historic MRI or X-ray images or live real-time feed from one or more cameras as required. The symbology may comprise real-time information from other sources, for example equipment monitoring heart rate, blood pressure or blood oxygenation levels.
1. A method of controlling a display for displaying a monochromatic image from a polychromic
image input, the polychromic image input comprising a primary image and symbology
having a selected colour superimposed on the primary image, the method comprising:
receiving said polychromic image input, determining at least one primary range of
intensities for representing the primary image and at least one secondary range of
intensities for representing the symbology, said at least one primary range not overlapping
with said at least one secondary range, and displaying the monochromatic image by
representing the primary image by intensities in said at least one primary range and
by representing the symbology by intensities in said at least one secondary range.
2. A method as claimed in claim 1, wherein the symbology intensity in said at least one
secondary range at which said symbology is displayed is spaced apart from the primary
range to enhance in the monochromatic image the contrast between the displayed symbology
and the primary image.
3. A method as claimed in claim 2, wherein the symbology intensity is spaced apart from
said primary range by a range of intensities which are not displayed in the monochromatic
image.
4. A method as claimed in any of the preceding claims, wherein said at least one primary
range extends from a low intensity to an intermediate intensity between said low and
a high intensity, and said at least one secondary range extends from said intermediate
intensity to said high intensity.
5. A method as claimed in any of the preceding claims, wherein said at least one primary
range extends from a high intensity to an intermediate intensity between said low
and a high intensity, and said at least one secondary range extends from said intermediate
intensity to said low intensity.
6. A method as claimed in any of the preceding claims, wherein said at least one primary
range comprises a plurality of sub-ranges between a low intensity and a high intensity,
and said at least one secondary range extends between the or each adjacent said sub-ranges.
7. A method as claimed in any one of the preceding claims, wherein the polychromic image
input comprises symbology having a plurality of discrete selected colours superimposed
on the primary image, and the method comprises displaying the monochromatic image
by representing the symbology in said at least one secondary range by discrete intensities
corresponding to respective said colours.
8. A method as claimed in any one of the preceding claims, wherein said polychromic image
input comprises image data corresponding to each picture element for reproducing said
primary image and said symbology, and the method comprises processing said image data
to determine the colour of each picture element, wherein if the colour of a picture
element is not generally equal to the selected colour or colours of the symbology
the intensity at which said picture element colour is displayed in said monochromatic
image is modified to correspond with an intensity in said at least one primary range,
and if the colour of a picture element is generally equal to the selected colour or
colours of the symbology the intensity at which said picture element colour is displayed
in said monochromatic image is modified to correspond with an intensity in said at
least one secondary range.
9. A method as claimed in claim 8, wherein if the colour of a picture element is not
generally equal to the selected colour or colours of the symbology the intensity at
which said picture element colour is displayed in said monochromatic image is multiplied
by a factor of less than one to correspond with an intensity in said at least one
primary range.
10. A method as claimed in claim 8 or 9, wherein if the colour of a picture element is
generally equal to the selected colour or colours of the symbology the intensity at
which said picture element colour is displayed in said monochromatic image is shifted
to one or more discrete intensities in said at least one secondary range.
11. A display system comprising: a display for displaying a monochromatic image from a
polychromic image input, the polychromic image input comprising a primary image and
symbology having a selected colour superimposed on the primary image, and a processing
unit configured to receive said polychromic image input, to determine at least one
primary range of intensities for representing the primary image and at least one secondary
range of intensities for representing the symbology, said at least one primary range
not overlapping with said at least one secondary range, and outputting to said display
a monochromatic image output in which the primary image is represented by intensities
in said at least one primary range and the symbology is represented by intensities
in said at least one secondary range.
12. A display system as claimed in claim 11, wherein said polychromic image input comprises
image data corresponding to each picture element for reproducing said primary image
and said symbology, and the processing unit is configured to determine the colour
of said picture elements and output to said display a symbology intensity in said
at least one secondary range for each picture element having said selected colour
and a primary image intensity in said at least one primary range for each picture
element not having said selected colour.
13. A display system as claimed in claim 11 or 12, wherein the symbology intensity is
spaced apart from the primary range to enhance in the monochromatic image the contrast
between the displayed symbology and the primary image.
14. A display system as claimed in any one of claims 11 to 13, wherein the polychromic
image input comprises symbology having a plurality of discrete selected colours superimposed
on the primary image, and the processing unit is configured to output a monochromatic
image output in which the symbology is represented by discrete respective intensities
corresponding to said colours in said at least one secondary range.
15. A display system as claimed in claim 14, wherein the discrete symbology intensities
in said at least one secondary range at which said symbology is displayed are spaced
apart one from another and from the primary range to enhance in the monochromatic
image the contrast between the displayed symbology and between the displayed symbology
and the primary image.