(19)
(11)EP 3 262 505 B1

(12)EUROPEAN PATENT SPECIFICATION

(45)Mention of the grant of the patent:
09.09.2020 Bulletin 2020/37

(21)Application number: 16704929.5

(22)Date of filing:  15.02.2016
(51)International Patent Classification (IPC): 
G06F 3/0481(2013.01)
G09G 3/00(2006.01)
G06F 3/01(2006.01)
G06F 9/451(2018.01)
G06T 19/00(2011.01)
(86)International application number:
PCT/GB2016/050362
(87)International publication number:
WO 2016/135446 (01.09.2016 Gazette  2016/35)

(54)

INTERACTIVE SYSTEM CONTROL APPARATUS AND METHOD

INTERAKTIVE SYSTEMSTEUERUNGSVORRICHTUNG UND VERFAHREN

APPAREIL DE COMMANDE À SYSTÈME INTERACTIF ET PROCÉDÉ


(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

(30)Priority: 25.02.2015 GB 201503116
25.02.2015 EP 15275049

(43)Date of publication of application:
03.01.2018 Bulletin 2018/01

(73)Proprietor: BAE Systems PLC
London SW1Y 5AD (GB)

(72)Inventors:
  • WHITEFORD, Christopher,James
    Preston Lancashire PR4 1AX (GB)
  • COLOSIMO, Nicholas, Giacomo, Robert
    Preston Lancashire PR4 1AX (GB)
  • WRIGHT, Julian, David
    Preston Lancashire PR4 1AX (GB)

(74)Representative: BAE SYSTEMS plc Group IP Department 

Farnborough Aerospace Centre Farnborough Hampshire GU14 6YU
Farnborough Aerospace Centre Farnborough Hampshire GU14 6YU (GB)


(56)References cited: : 
EP-A1- 2 405 349
US-A1- 2014 368 533
US-A- 6 094 625
  
      
    Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention).


    Description


    [0001] This invention relates generally to a system control apparatus and method for facilitating user control and/or operations of a system.

    [0002] Many complex systems rely on highly specialised components in order to achieve their functionality or perform a particular role, and information tends to be handled using a combinations of physical hardware and graphics represented on a two-dimensional screen. Generally, the user interacts with such information provided on-screen using a mouse or touchscreen input method, whereas specific/specialised tasks tend to be effected through the use of bespoke switch panels, dials, buttons, slide controls, and the like. The result is a complex physical control environment, which places a relatively high workload and training burden on individuals, and increases the potential for human error during interaction.

    [0003] Technological advances have addressed this issue to a limited extent, by, for example, the integration of LCDs into a single control element, thereby allowing the control element to perform multiple functions. Advances in touch screen and alternative technologies, such as voice commands and gaze recognition, have also attempted to alleviate some of the problems identified above. However, such technologies have severe limitations, particularly in relation to the types of control elements they can be used to replace or enhance, and the user is still, in any event, provided with a system with which they are required to interact in a 'conventional' manner, i.e. as if the physical controls were still present and in the same relative locations as in a system comprised entirely of physical controls. Thus, the user is still constrained, to a large extent, by conventional methods of interacting with the system, with the result that the above-mentioned disadvantages are still present.

    [0004] US6094625 discloses an apparatus which enables the use of augmented vision in survey procedures and related work. Virtual objects are presented to an operator on a see through display. Some of the objects correspond with real objects in the operators field of view at any instant and others may provide interactive functions for the operator. The operator eye positions are determined using measurements from a remote positioning system such as GPS, an operator head orientation sensing system, and knowledge of head geometry. Positions and attributes of real objects are stored in a database memory carried by the operator.Numerous functions may be provided for the operator using virtual interactive objects, depending on the work at hand.

    [0005] EP2405349 discloses an apparatus to provide augmented reality, including a relevant information acquisition unit to acquire relevant information corresponding to an object to implement augmented reality, a relevant information editing unit to edit the of acquired relevant information, and a virtual marker generating unit to generate a virtual marker based on the edited relevant information by mapping the edited relevant information to a marker element.

    [0006] US2014/368533 discloses the determination of geo-located positions of points of interest within a field of view and generates markers indicating information regarding an associated real world object which is available to the user. Markers are rendered in the display relative to the geo-located position and the field of view of the user.

    [0007] It would, therefore, be desirable to provide a method and apparatus for controlling the functions and/or operations of a system which at least addresses some of the problems outlined above.

    [0008] Virtual reality systems are known, comprising a headset which, when placed over a user's eyes, creates and displays a three-dimensional virtual environment in which a user feels immersed and with which a user can interact in a manner dependent on the application. For example, in some prior art systems, the virtual environment created may comprise a game zone, within which a user can play a game. However, in an environment where the user needs to be able to see where they are going in order to function appropriately within their physical environment, such systems are unsuitable.

    [0009] More recently, augmented and mixed reality systems have been developed, wherein an image of a real world object can be captured, rendered and placed within a 3D virtual reality environment, such that it can be viewed and manipulated within that environment in the same way as virtual objects therein. Other so-called augmented reality systems exist, comprising a headset having a transparent or translucent visor which, when placed over a user's eyes, creates a three-dimensional environment with which the user can interact, whilst still being able to view their real environment through the visor.

    [0010] However, in an augmented reality environment, whereby the user can "see" all aspects of their real world environment through the visor as well as the multiple sources of data in the virtual environment, the resultant 3D environment becomes excessively cluttered and it becomes difficult for a user to focus on the important elements thereof. Furthermore, this does not address the problem outlined above, whereby in complex systems, multiple physical controls are provided in fixed locations within the real world environment, which do not enable the user to interact with them conveniently or intuitively.

    [0011] It is therefore an object of aspects of the present invention to address at least some of these issues.

    [0012] In accordance with a first aspect of the present invention, there is provided a mixed reality control apparatus for a system comprising at least one remote data source requiring at least one physical control input, the apparatus comprising a headset for placing over a user's eyes, in use, said headset including a screen, the apparatus further comprising a processor configured to receive data from said at least one remote data source and display said data on said screen within a three-dimensional virtual environment, and image capture means for capturing images of the real world environment in the vicinity of the user, the processor being further configured to:
    • blend at least portions or objects of said images of said real world environment into said three-dimensional virtual environment to create a mixed reality environment, including said data, to be displayed on said screen; and
    • generate a virtual representation of said at least one physical control input and blend said virtual representation into said mixed reality environment at a selected location, and generate a marker representative of said selected location;
    wherein the apparatus is configured to receive a user input in association with said marker, identify said user input as an action in respect of said control input, or said virtual representation thereof, and generate a control signal to effect said action accordingly; and
    wherein said marker is associated with a user's body part, and the processor is configured to track movement of said user's body part and adjust the relative location of said virtual representation of said control input accordingly.

    [0013] The apparatus may comprise a pair of spatially separated image capture devices for capturing respective images of the real world environment in the vicinity of the user, said processor being configured to define a depth map using respective image frame pairs to produce three-dimensional image data. The image capture devices may be mounted on said headset so as to be substantially aligned with a user's eyes, in use.

    [0014] The user input may comprise a predefined bodily movement by said user in association with said marker, and said processor is configured to identify a bodily movement, determine from the relative location of said bodily movement in relation to said marker the control input to which said user input relates, determined from said bodily movement the control action required to be performed and generate a control signal to effect said action accordingly. The predefined bodily movement may be a predefined user hand gesture.

    [0015] As such, the apparatus may further comprise a database in which is stored data representative of a predefined set of bodily movements and control actions to which they relate, and said processor is configured to extract an image of a user's bodily movement from said captured images, compare said movement with said predefined set of bodily movements to determine a match and, thereby, to identify a required control action, and generate a control single operative to cause a required control action to be performed in relation to data displayed within said mixed reality environment.

    [0016] The processor may be configured to determine, from said images of said real world captured by said image capture means, the direction of a user's gaze, and affix an associated said marker thereto, such that said virtual representation of said control input remains constantly within said user's field of view.

    [0017] In another exemplary embodiment of the invention, the marker may be in a fixed relative location within said three-dimensional virtual environment displayed on said screen.

    [0018] The virtual representation of said control input may comprise an image of a corresponding physical control device, and said user input comprises a physical action representative of physical manipulation of said physical control device.

    [0019] The apparatus may be configured to selectively move said data and/or said virtual representation of said control input and associated marker in response to a user input.

    [0020] These and other aspects of the present invention will become apparent from the following specific description of embodiments of the present invention, which are described by way of examples only and with reference to the accompanying drawings, in which:

    Figure 1 is a front perspective view of a headset for use in a control apparatus according to an exemplary embodiment of the present invention;

    Figure 2 is a schematic block diagram of a control apparatus according to an exemplary embodiment of the present invention; and

    Figure 3 is a schematic diagram illustrating a system according to an exemplary embodiment of the present invention, in use.



    [0021] Referring to Figure 1 of the drawings, a system according to an exemplary embodiment of the present invention may comprise a headset comprising a visor 10 having a pair of arms 12 hingedly attached at opposing sides thereof in order to allow the visor to be secured onto a user's head, over their eyes, in use, by placing the curved ends of the arms 12 over and behind the user's ears, in a manner similar to conventional spectacles. It will be appreciated that, whilst the headset is illustrated herein in the form of a visor, it may alternatively comprise a helmet for placing over a user's head, or even a pair of contact lenses or the like, for placing within the user's eyes, and the present invention is not intended to be in any way limited in this regard. Also provided on the headset, is a pair of image capture devices 14 for capturing images of the environment, such image capture devices being mounted roughly aligned with a user's eyes in use.

    [0022] The system of the present invention further comprises a processor which is communicably connected in some way to a screen which is provided inside the visor 10. Such communicable connection may be a hard wired electrical connection, in which case the processor and associated circuitry will also be mounted on the headset. However, in an alternative exemplary embodiment, the processor may be configured to wirelessly communicate with the visor, for example, by means of Bluetooth or similar wireless communication protocol, in which case, the processor need not be mounted on the headset but can instead be located remotely from the headset, with the relative allowable distance between them being dictated and limited only by the wireless communication protocol being employed. For example, the processor could be mounted on or formed integrally with the user's clothing, or instead located remotely from the user, either as a stand-alone unit or as an integral part of a larger control unit, for example.

    [0023] Referring to Figure 2 of the drawings, a system according to an exemplary embodiment of the invention comprises, generally, a headset 100, incorporating a screen 102, a processor 104, and a pair of external digital image capture devices (only one shown) 106.

    [0024] Referring additionally to Figure 3 of the drawings, a complex control system may require the use of multiple data sources, such as maps, communication feeds, radar displays, etc, as well as a number of physical controls 201 in order to effect control functions and interact with the data. The processor 104 is configured to receive data from multiple sources and display them within the three-dimensional virtual environment 206 displayed on their screen. In the example illustrated, the data sources are displayed as "screens" 203 anchored to 'surfaces' within the three-dimensional virtual environment, but the user 204 is able to move the data sources to any convenient location within the three-dimensional environment 200 and, indeed, the manner in which the data sources themselves are displayed may take any convenient form and the present invention is not necessarily intended to be limited in this regard. Thus, the user can configure these data items within the display environment in any way they wish to, simply by means of hand gestures, for example, as will be described later.

    [0025] As illustrated in Figure 3 of the drawings, a complex control system also requires the use of physical hardware, illustrated in the form of a control panel 202, including control elements 201 such as slide switches, buttons, a keyboard and computer mouse, a joystick or lever etc. In the example shown, the physical hardware is illustrated as being located in the same general area, i.e. the control panel 202, within the user's real world environment. However, this may not be the case, and the physical hardware may alternatively be separately located in various places around the operational area.

    [0026] The user's headset 100 includes two image capture devices, as stated previously, which may be used to capture respective images of the real world environment in the vicinity of the user, and data representative thereof can be blended to produce a stereoscopic depth map which enables the processor to determine depth within the captured images without any additional infrastructure being required.

    [0027] All or selected portions of the 3D images can be blended into the virtual environment 200 being displayed on the screen. Thus, in the example shown on the screen, only the user's own body 210 is selected for inclusion in the displayed environment, with all other data and images therein being virtual images. However, in an alternative embodiment of the invention, the data displayed may actually be derived from real world captured images of the physical hardware on which it is conventionally displayed, i.e. a computer screen or the like and, thus, the 3D mixed reality environment displayed on the user's screen may, in fact, mirror much more closely their real world environment. Either way, the user is able to change the format and location of any element of the mixed reality environment, in order to "build" their virtual environment as required.

    [0028] The general concept of real time image blending for augmented and mixed reality is known, and several techniques have been proposed. The present invention is not intended to be in any way limited in this regard. However, for completeness, one exemplary method for image blending will be briefly described. Thus, once the image data for an item to be blended into the virtual environment has been generated, a threshold function may be applied in order to extract that image data from any background images. Its relative location and orientation may also be extracted and preserved by means of marker data. Next, the image and marker data is converted to a binary image, possibly by means of adaptive thresholding (although other methods are known). The marker data and binary image are then transformed into a set of coordinates which match the location within the virtual environment in which they will be blended. Such blending is usually performed using black and white image data. Thus, if necessary, colour data sampled from the source image may be backward warped, using homography, to each pixel in the resultant virtual scene. All of these computational steps require minimal processing capacity and time and can, therefore, be performed quickly and in real time. Thus, if the selected object is moving, for example, the user's own body, the corresponding image data within the virtual environment can be updated in real time.

    [0029] Also included within the mixed reality environment, are visual representations of the physical control elements required to control and manipulate various aspects of the overall system. In the example shown, a keyboard 219 and mouse 220 are represented in one part of the virtual environment, near a first screen 203, a set of buttons 221 is provided on the user's arm 210, and slide switch 222 and directional buttons 223 are provided in another part of the virtual environment, near another screen 203. In this example, the control elements 219, 220, 221, 222 and 223 are represented in the same manner as their physical counterparts on the control panel 202. Indeed, the images thereof may have been extracted from images captured by the image capture devices on the user's headset and blended into the 3D virtual environment in the manner described above, but this may not necessarily be the case, and they may be represented and/or changed in any manner convenient to the user. Equally, the user can move the virtual control elements within the displayed virtual environment and place and anchor them at any location therein that they wish to. Such rearrangement of the virtual control elements can, once again, be effected by, for example, hand gestures.

    [0030] As explained above, the image capturing module provided in the system described above can be used to capture video images of the user's hands, such that hand gestures provide a convenient method of manipulating objects and data within the 3D virtual environment. One relatively simple method of automated hand gesture recognition and control using captured digital video images involves the use of a database of images of predefined hand gestures and the command to which they relate. Thus, an auto threshold function is first performed on the image to extract the hand from the background. The wrist is then removed from the hand shape, using a so-called "blob" image superposed over the palm of the hand, to separate out the individual parts of the hand so that the edge of the blob defines the border of the image. The parts outside of the border (i.e. the wrist) are then removed from the image, following which shape recognition software can be used to extract and match the shape of the hand to the predefined hand gesture, and call the associated command accordingly.

    [0031] It will be appreciated by a person skilled in the art that other methods of manipulating objects within the 3D virtual environment may alternatively or additionally be employed, including other bodily movements, such as a user's head movement, and the present invention is not necessarily intended to be limited in this regard.

    [0032] Of course, if the user wishes to interact with one of the virtual control elements 219, 220, 221, 222, 223, it will be necessary to perform some form of direction and orientation sensing, such that it can be determined where in the virtual environment the user is performing a predefined hand gesture and, therefore, which element of the environment they wish to manipulate thereby. This can be done by any known means, for example, by image recognition within the captured images of the user's hands relative to a marker within the image, or by means of an accelerometer or similar orientation sensor mounted or otherwise provided on the user's hands. When the user configures their 3D virtual environment to include representations of the physical control elements and places or anchors them therein, a marker is generated which is representative of the respective relative location within the environment of the virtual control element. Thus, if the relative location of the virtual control element is known, the location of the user's hand is known, and the hand gesture made by the user in respect thereof is known, the system is able to determine the control action required to be effected and generate the associated control signals accordingly.

    [0033] Data and virtual controls can thus be displayed anywhere in the user's extended field of view (defined by the three-dimensional virtual environment displayed on the screen), and can be:

    o anchored (by its associated marker) to their gaze (i.e. always in view);

    o anchored to the three dimensional virtual space (or a virtual structure or objects therein, i.e. remains in the same relative position within the virtual environment displayed on the screen;

    o anchored to the user's body (for example, a screen could be anchored to the inside of a user's wrist, top of their legs, etc)

    o anchored to a symbol or marker within the virtual environment (for example, to a QR code on a virtual table or within the pages of a virtual book).



    [0034] Thus, exemplary embodiments of the present invention provide a mixed reality control apparatus particularly, but not necessarily exclusively, suited for use with a complex system comprised of multiple data sources which are traditionally manipulated using a combination of physical hardware and graphics represented on a two-dimensional screen. Using the unique functionality described above, user controls can be represented by the creation and insertion of virtual interactive objects, together with associated markers, into a mixed reality environment. The headset mounted cameras capture images of the user's own body, including their arms, hands and fingers and, using the stereo cameras described above, the position of the user's finger tips in three-dimensional space can be inferred, thereby enabling the above-mentioned interaction by the user with virtual elements within their field of view (on the screen). Data can be visualised in three dimensions and, since the design of the user controls is not constrained by physical placement, design or complexity, such virtual representations thereof can be made to appear in a much more intuitive configuration, for ease of user interaction therewith.

    [0035] Data and virtual controls can be displayed anywhere in the user's extended field of view, as stated above, and virtual data/images can be added, re-sized, relocated or removed using, for example, hand gestures. As a result, traditional 2D displays and controls can be almost entirely eliminated, in some cases, and recreated using the mixed reality control apparatus proposed by aspects of the present invention.

    [0036] It will be appreciated by a person skilled in the art, from the foregoing description, that modifications and variations can be made to the described embodiments, without departing from the scope of the invention as claimed.


    Claims

    1. A mixed reality control apparatus for a system comprising at least one remote data source requiring at least one physical control input, the apparatus comprising a headset (100) for placing over a user's eyes, in use, said headset (100) including a screen (102), the apparatus further comprising a processor (104) configured to receive data from said at least one remote data source and display said data on said screen (102) within a three-dimensional virtual environment (200), and image capture means (14) for capturing images of the real world environment (210) in the vicinity of the user, the processor (104) being further configured to:

    - blend at least portions or objects of said images of said real world environment (210) into said three-dimensional virtual environment (200) to create a mixed reality environment, including said data, to be displayed on said screen (102); and

    - generate a virtual representation (219) of said at least one physical control input (201) and blend said virtual representation into said mixed reality environment at a selected location, and generate a marker representative of said selected location;

    wherein the apparatus is configured to receive a user input in association with said marker, identify said user input as an action in respect of said control input, or said virtual representation thereof, and generate a control signal to effect said action accordingly, and
    wherein said marker is associated with a user's body part, and the processor is configured to track movement of said user's body part and adjust the relative location of said virtual representation of said control input accordingly.
     
    2. Apparatus according to claim 1, comprising a pair of spatially separated image capture devices (14) for capturing respective images of the real world environment in the vicinity of the user, said processor (104) being configured to define a depth map using respective image frame pairs to produce three-dimensional image data.
     
    3. Apparatus according to claim 2, wherein said image capture devices (14) are mounted on said headset (100) so as to be substantially aligned with a user's eyes, in use.
     
    4. Apparatus according to claim 2 or claim 3, wherein said user input comprises a predefined bodily movement by said user in association with said marker, and said processor (104) is configured to identify a bodily movement, determine from the relative location of said bodily movement in relation to said marker the control input to which said user input relates, determined from said bodily movement the control action required to be performed and generate a control signal to effect said action accordingly.
     
    5. Apparatus according to claim 4, wherein said predefined bodily movement is a predefined user hand gesture.
     
    6. Apparatus according to claim 4 or claim 5, further comprising a database in which is stored data representative of a predefined set of bodily movements and control actions to which they relate, and said processor is configured to extract an image of a user's bodily movement from said captured images, compare said movement with said predefined set of bodily movements to determine a match and, thereby, to identify a required control action, and generate a control single operative to cause a required control action to be performed in relation to data displayed within said mixed reality environment.
     
    7. Apparatus according to any of the preceding claims, wherein said processor is configured to determine, from said images of said real world captured by said image capture means, the direction of a user's gaze, and affix an associated said marker thereto, such that said virtual representation of said control input remains constantly within said user's field of view.
     
    8. Apparatus according to any of claims 1 to 6, wherein said marker is in a fixed relative location within said three-dimensional virtual environment displayed on said screen.
     
    9. Apparatus according to any of the preceding claims, wherein said virtual representation of said control input comprises an image (219) of a corresponding physical control device, and said user input comprises a physical action representative of physical manipulation of said physical control device.
     
    10. Apparatus according to any of the preceding claims, configured to selectively move said data and/or said virtual representation of said control input and associated marker in response to a user input.
     


    Ansprüche

    1. Gemischte-Realität-Steuervorrichtung für ein System, das mindestens eine ferne Datenquelle umfasst, die mindestens eine physische Steuereingabe erfordert, wobei die Vorrichtung ein Headset (100) zum Platzieren vor den Augen eines Benutzers während der Verwendung umfasst, wobei das Headset (100) einen Bildschirm (102) beinhaltet, wobei die Vorrichtung ferner einen Prozessor (104) umfasst, der dafür konfiguriert ist, Daten von der mindestens einen fernen Datenquelle zu empfangen und die Daten auf dem Bildschirm (102) innerhalb einer dreidimensionalen virtuellen Umgebung (200) anzuzeigen, und Bildaufnahmemittel (14) zum Aufnehmen von Bildern der Echte-Welt-Umgebung (210) in der Nähe des Benutzers, wobei der Prozessor (104) ferner für Folgendes konfiguriert ist:

    - Einblenden mindestens von Abschnitten oder Objekten des Bildes der Echte-Welt-Umgebung (210) in die dreidimensionale virtuelle Umgebung (200), um eine Gemischte-Realität-Umgebung zu erzeugen, welche die Daten beinhaltet und die auf dem Bildschirm (102) anzuzeigen ist, und

    - Erzeugen einer virtuellen Darstellung (219) der mindestens einen physischen Steuereingabe (201) und Einblenden der virtuellen Darstellung in die Gemischte-Realität-Umgebung an einer ausgewählten Position und Erzeugen einer Markierung, welche die ausgewählte Position darstellt,

    wobei die Vorrichtung dafür konfiguriert ist, eine Benutzereingabe in Verknüpfung mit der Markierung zu empfangen, die Benutzereingabe als eine Aktion in Hinsicht auf die Steuereingabe oder deren virtuelle Darstellung zu identifizieren und ein Steuersignal zu erzeugen, um die Aktion dementsprechend zu bewirken, und
    wobei die Markierung mit einem Körperteil eines Benutzers verknüpft ist und der Prozessor dafür konfiguriert ist, eine Bewegung des Körperteils eines Benutzers nachzuverfolgen und die relative Position der virtuellen Darstellung der Steuereingabe dementsprechend zu justieren.
     
    2. Vorrichtung nach Anspruch 1, ein Paar räumlich beabstandeter Bildaufnahmegeräte (14) zum Aufnehmen entsprechender Bilder der Reale-Welt-Umgebung in der Nähe des Benutzers umfassend, wobei der Prozessor (104) dafür konfiguriert ist, mit Hilfe entsprechender Bild-Frame-Paare eine Tiefenabbildung zu definieren, um dreidimensionale Bilddaten zu erzeugen.
     
    3. Vorrichtung nach Anspruch 2, wobei die Bilderfassungsgeräte (14) derart an das Headset (100) montiert sind, dass sie während der Verwendung im Wesentlichen an den Augen des Benutzers ausgerichtet sind.
     
    4. Vorrichtung nach Anspruch 2 oder 3, wobei die Benutzereingabe eine vordefinierte Körperbewegung des Benutzers in Verknüpfung mit der Markierung umfasst und der Prozessor (104) dafür konfiguriert ist, eine Körperbewegung zu identifizieren, aus der relativen Position der Körperbewegung im Verhältnis zu der Markierung die Steuereingabe zu bestimmen, auf welche sich die Benutzereingabe bezieht, aus der Körperbewegung die Steueraktion zu bestimmen, die ausgeführt werden muss, und ein Steuersignal zu erzeugen, um die Aktion dementsprechend zu bewirken.
     
    5. Vorrichtung nach Anspruch 4, wobei die vordefinierte Körperbewegung eine vordefinierte Handbewegung des Benutzers ist.
     
    6. Vorrichtung nach Anspruch 4 oder 5, ferner eine Datenbank umfassend, in der Daten gespeichert sind, die einen vordefinierten Satz von Körperbewegungen und Steueraktionen, auf die sie sich beziehen, darstellen, und wobei der Prozessor dafür konfiguriert ist, ein Bild einer Körperbewegung des Benutzers aus den aufgenommenen Bildern zu extrahieren, die Bewegung mit dem vordefinierten Satz von Körperbewegungen zu vergleichen, um eine Übereinstimmung zu bestimmen und dadurch eine erforderliche Steueraktion zu identifizieren und ein Steuersignal zu erzeugen, das funktionsfähig ist, die Ausführung einer erforderlichen Steueraktion in Bezug auf Daten, die in der Gemischte-Realität-Umgebung angezeigt werden, zu bewirken.
     
    7. Vorrichtung nach einem der vorhergehenden Ansprüche, wobei der Prozessor dafür konfiguriert ist, aus den Bildern der echten Welt, die von den Bildaufnahmemitteln aufgenommen werden, die Richtung eines Blicks eines Benutzers zu bestimmen und eine verknüpfte Markierung daran anzuheften, so dass die virtuelle Darstellung der Steuereingabe konstant im Blickfeld des Benutzers bleibt.
     
    8. Vorrichtung nach einem der Ansprüche 1 bis 6, wobei sich die Markierung an einer festen relativen Position in der auf dem Bildschirm angezeigten dreidimensionalen virtuellen Umgebung befindet.
     
    9. Vorrichtung nach einem der vorhergehenden Ansprüche, wobei die virtuelle Darstellung der Steuereingabe ein Bild (219) eines entsprechenden physischen Steuergeräts umfasst und die Benutzereingabe eine physische Aktion umfasst, welche die physische Handhabung des physischen Steuergeräts darstellt.
     
    10. Vorrichtung nach einem der vorhergehenden Ansprüche, dafür konfiguriert, die Daten und/oder die virtuelle Darstellung der Steuereingabe und der verknüpften Markierung in Reaktion auf eine Benutzereingabe wahlweise zu bewegen.
     


    Revendications

    1. Appareil de contrôle de réalité mixte destiné à un système comprenant au moins une source de données à distance qui nécessite au moins une entrée de contrôle physique, l'appareil comprenant un casque (100) destiné à être placé sur les yeux d'un utilisateur, pendant l'utilisation, ledit casque (100) comprenant un écran (102), l'appareil comprenant en outre un processeur (104) configuré pour recevoir des données de la part de ladite au moins une source de données à distance et pour afficher lesdites données sur ledit écran (102) dans un environnement virtuel en trois dimensions (200), et un moyen de capture d'images (14) destiné à capturer des images de l'environnement réel (210) à proximité de l'utilisateur, le processeur (104) étant en outre configuré pour :

    - mélanger au moins des parties ou objets desdites images dudit environnement réel (210) dans ledit environnement virtuel en trois dimensions (200) afin de créer un environnement de réalité mixte, comprenant lesdites données, à afficher sur ledit écran (102) ; et

    - générer une représentation virtuelle (219) de ladite au moins une entrée de contrôle physique (201) et mélanger ladite représentation virtuelle dans ledit environnement de réalité mixte à un emplacement choisi, et générer un marqueur représentatif dudit emplacement choisi ;

    dans lequel l'appareil est configuré pour recevoir une entrée utilisateur en association avec ledit marqueur, pour identifier ladite entrée utilisateur comme une action par rapport à ladite entrée de contrôle, ou ladite représentation virtuelle de celle-ci, et pour générer un signal de contrôle de façon à exécuter ladite action en conséquence, et
    dans lequel ledit marqueur est associé à une partie du corps de l'utilisateur, et le processeur est configuré pour suivre le mouvement de ladite partie du corps de l'utilisateur et pour ajuster l'emplacement relatif de ladite représentation virtuelle de ladite entrée de contrôle en conséquence.
     
    2. Appareil selon la revendication 1, comprenant une paire de dispositifs de capture d'images spatialement séparés (14) et destinés à capturer des images respectives de l'environnement réel à proximité de l'utilisateur, ledit processeur (104) étant configuré pour définir une carte de profondeur à l'aide de paires de cadres d'images respectifs de façon à produire des données d'images en trois dimensions.
     
    3. Appareil selon la revendication 2, dans lequel lesdits dispositifs de capture d'images (14) sont montés sur ledit casque (100) de façon à être sensiblement alignés avec les yeux d'un utilisateur, pendant l'utilisation.
     
    4. Appareil selon la revendication 2 ou 3, dans lequel ladite entrée utilisateur comprend un mouvement corporel prédéfini par ledit utilisateur en association avec ledit marqueur, et ledit processeur (104) est configuré pour identifier un mouvement corporel, pour déterminer, à partir de l'emplacement relatif dudit mouvement corporel par rapport audit marqueur, l'entrée de contrôle à laquelle ladite entrée utilisateur est liée, pour déterminer, à partir dudit mouvement corporel, l'action de contrôle qui doit être exécutée, et pour générer un signal de contrôle destiné à exécuter ladite action en conséquence.
     
    5. Appareil selon la revendication 4, dans lequel ledit mouvement corporel prédéfini est un geste de la main de l'utilisateur prédéfini.
     
    6. Appareil selon la revendication 4 ou 5, comprenant en outre une base de données dans laquelle sont stockées des données représentatives d'un groupe prédéfini de mouvements corporels et d'actions de contrôle auxquelles ils sont liés, et ledit processeur est configuré pour extraire une image du mouvement corporel d'un utilisateur à partir desdites images capturées, pour comparer ledit mouvement avec ledit groupe prédéfini de mouvements corporels de façon à déterminer une correspondance et, ainsi, pour identifier une action de contrôle nécessaire, et pour générer un signal de contrôle destiné à provoquer l'exécution d'une action de contrôle nécessaire en lien avec les données affichées dans ledit environnement de réalité mixte.
     
    7. Appareil selon l'une quelconque des revendications précédentes, dans lequel ledit processeur est configuré pour déterminer, à partir desdites images dudit monde réel capturées par ledit moyen de capture d'images, la direction du regard d'un utilisateur, et pour associer ledit marqueur à celle-ci, de sorte que ladite représentation virtuelle de ladite entrée de contrôle reste constamment dans le champ de vision dudit utilisateur.
     
    8. Appareil selon l'une quelconque des revendications 1 à 6, dans lequel ledit marqueur se trouve à un emplacement relatif fixe dans ledit environnement virtuel en trois dimensions affiché sur ledit écran.
     
    9. Appareil selon l'une quelconque des revendications précédentes, dans lequel ladite représentation virtuelle de ladite entrée de contrôle comprend une image (219) d'un dispositif de contrôle physique correspondant, et ladite entrée utilisateur comprend une action physique représentative d'une manipulation physique dudit dispositif de contrôle physique.
     
    10. Appareil selon l'une quelconque des revendications précédentes, configuré pour déplacer sélectivement lesdites données et/ou ladite représentation virtuelle de ladite entrée de contrôle et ledit marqueur associé en réponse à une entrée utilisateur.
     




    Drawing











    Cited references

    REFERENCES CITED IN THE DESCRIPTION



    This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.

    Patent documents cited in the description