[0001] The present invention relates to a speech recognition apparatus employing speech
recognition method performed in accordance with the viewpoint (point where the user's
eyes are focused on the display screen) of a user and an information processing apparatus
using the speech recognition apparatus.
[0002] A method for improving precision of speech recognition by controlling the recognition
information contained in communicative information has been proposed. The "recognition
information" is managed by a recognition apparatus and includes data consisting of
words, sentences, grammar, and frequency of use thereof. For example, in a calendar
application, when a speech "i∂r" (phonetic symbol) is inputted if higher weight is
put to a word "year" than a word "ear", the word "year" is more likely chosen. In
likewise manner, the recognition information can be controlled. Further, a method
for improving recognition ability of a recognition apparatus by using image information
in speech recognition has been also proposed.
[0003] However, in the aforesaid speech recognition method, neither a user's visual direction
nor application is taken into consideration. Thus, the recognition information is
controlled in accordance with a communicative situation at best. Therefore, precision
of speech recognition greatly depends on precision of control of the communicative
situation. In a case where the recognition information is controlled depending upon
communicative situations, it is difficult to quickly correspond to different users'
manners (communicative situation) which differs from person to person, resulting in
that adequate speech recognition level cannot be achieved and recognition speed cannot
be improved.
[0004] The present invention has been made in consideration of the above situation, and
aims in one aspect to provide a speech recognition apparatus which has improved speech
recognition level and/or recognition speed by properly taking user's visual direction
and application into consideration by detecting movement of a viewpoint of the user
and controlling recognition information on the basis of the user's visual direction
and application.
[0005] Embodiments of the present invention provide an information processing apparatus
having a speech recognition apparatus which achieves aforesaid speech recognition
as well as executes applications on the basis of recognition results of inputted speech.
[0006] An alternative aspect of the present invention is to improve speech recognition level
by enabling the change of different classes of recognition information for speech
recognition processing on the basis of a viewpoint of a user.
[0007] Yet another alternative aspect of the present invention is to improve speech recognition
level by putting different weights on different classes of recognition information
for speech recognition processing on the basis of a viewpoint of a user and performing
the speech recognition by using the weighted recognition information.
[0008] Yet another alternative aspect of the present invention is to enable to control recognition
processing on the basis of relationship between a viewpoint and positions of displayed
images, such as icons of applications, on a display screen.
[0009] Yet another alternative aspect of the present invention is to enable execution of
a predetermined process by selecting an application in accordance with a class of
recognition information which is used to obtain recognition result in speech recognition
processing performed on the basis of a viewpoint of a user, and by supplying the recognition
result to the selected application.
[0010] Other features and advantages of the present invention will be apparent from the
following description taken in conjunction with the accompanying drawings, in which
like reference characters designate the same or similar parts throughout the figures
thereof.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The accompanying drawings, which are incorporated in and constitute a part of the
specification, illustrate embodiments of the invention and, together with the description,
serve to explain the principles of the invention.
Fig. 1 is a block diagram showing a brief configuration of an information processing
apparatus according to a first embodiment;
Fig. 2 is a flowchart explaining an operation of speech recognition processing according
to the first embodiment;
Fig. 3 is a graphical representation of an example of an operation according to the
first embodiment;
Fig. 4 is a graphical representation showing contents of a speech input control dictionary;
Fig. 5 is a graphical representation showing contents of a dictionary for an application
A;
Fig. 6 is a graphical representation showing contents of a dictionary for an application
B;
Fig. 7 shows a picture on a display screen in an example of an operation according
to the first embodiment;
Fig. 8 shows a picture on a display screen in an example of an operation according
to the first embodiment;
Fig. 9 shows a picture on a display screen in an example of an operation according
to the first embodiment;
Fig. 10 is a table showing relationship between viewpoints and weights to be applied;
Fig. 11 is a flowchart explaining an operational sequence according to the first embodiment;
Fig. 12 is a flowchart explaining the operational sequence according to the first
embodiment;
Fig. 13 shows a picture on a display screen in an example of an operation according
to a second embodiment;
Fig. 14 is a flowchart explaining an operational sequence according to the second
embodiment;
Fig. 15 shows a picture on a display screen in an example of an operation according
to a third embodiment;
Figs. 16A and 16B are graphs showing changes of weights according to a fourth embodiment;
Figs. 17A and 17B are graphs showing changes of weights with respect to time according
to a fifth embodiment;
Fig. 18 shows a picture on a display screen in an example of an operation according
to a sixth embodiment;
Fig. 19 is a graphical representation showing correspondence between viewpoint detection
areas and weights according to a seventh embodiment;
Fig. 20 is a block diagram showing a configuration of an information processing apparatus
according to the embodiments when a control program provided by a storage medium is
loaded and executed;
Fig. 21 is a flowchart showing a sequence to execute each processing module provided
by a storage medium; and
Fig. 22 is a memory map showing a storage situation of processing modules.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Preferred embodiments of the present invention will be described in detail in accordance
with the accompanying drawings.
<First Embodiment>
[0013] Fig. 1 is a block diagram showing a brief configuration of an information processing
apparatus according to a first embodiment. In Fig. 1, reference numeral 101 denotes
a speech input unit including a microphone 101a and an A/D converter 101b; 102, a
display unit for displaying various kinds of icons and a viewpoint (or area); 103,
a viewpoint detector having a detector 103b which detects a viewpoint on the basis
of an image sensed by a camera 103a or the like; and 104, a speech recognition unit
for recognizing input speech using a selected class of recognition information. In
the first embodiment, classes of the recognition information are weighted dictionaries,
containing words, sentences, grammar, and frequency of use thereof, corresponding
to different kinds of applications.
[0014] Further, reference numeral 105 denotes a recognition information controller for controlling
recognition information (i.e., selecting one of the dictionaries) on the basis of
a detected result of a viewpoint by a viewpoint manager 106; 106, the viewpoint manager
for managing movement of a viewpoint (how long and at which point of the display unit
102 a viewpoint is fixed, and so on) detected by the viewpoint detector 103; 107,
a command processing unit for performing various kinds of processes on the basis of
a command detected in speech recognition.
[0015] In the configuration shown in Fig. 1, each of the speech recognition unit 104, the
recognition information controller 105, the viewpoint manager 106, and the command
processing unit 107 can comprise of CPU, ROM, and RAM (not shown) so as to achieve
the desired functions, or a single CPU can be used for controlling the aforesaid four
units to realize the same functions. When a single CPU is used, functions of each
unit are obtained by executing a control program stored in the ROM 111 by CPU 110
as shown in Fig. 20. The ROM 111 stores a control program for realizing a control
sequence shown in a flowchart which will be explained later. Further, RAM 112 provides
a work area to the CPU 110 when it performs various kinds of control. A control program
can be loaded to the RAM 112 from a floppy disk 114 or the like and executed by the
CPU 110. This process will be explained later.
[0016] Next, speech recognition processing by the information processing apparatus of the
first embodiment will be explained. The speech recognition processing in the first
embodiment is performed by the configuration denoted by reference numerals 101 to
106 as above. Fig. 2 is a flowchart explaining an operation of speech recognition
processing according to the first embodiment.
[0017] First, when a user powers on the information processing apparatus, a viewpoint of
the user is detected by using the viewpoint detector 103 in viewpoint detecting process
201. Next, at which position (or an area) on the display unit 102 or its vicinity
and how long the viewpoint detected in the viewpoint detecting process 201 is fixed
is obtained by the viewpoint manager 106 in viewpoint managing process 202. Then,
the obtained viewpoint and its trace are displayed on the display unit 102.
[0018] Thereafter, on the basis of the relationship between the viewpoint (area), its movement,
and recognition information (words, sentences, grammar, and frequency of use thereof),
weight P() is determined in recognition information controlling process 203. As described
above, weight P() for a class of recognition information 204 is controlled in accordance
with a viewpoint and its movement.
[0019] Meanwhile, when the user pronounces a word (or sentence), the speech is inputted
and A/D converted via the speech input unit 101 in speech input process 205. Then,
a speech recognition probability PS() is obtained in speech recognition process 206,
and a product of the weight P() and the speech recognition probability PS() is obtained
as a final recognition probability PT() of the word (or sentence).
[0020] In the first embodiment, a plurality of dictionaries are used as different classes
of the recognition information, and one of the plurality of dictionaries is selected
by putting weight "1" on a dictionary selected in accordance with the movement of
the user's viewpoint, and putting weight "0" on other dictionaries. It should be noted
that, in the first embodiment, weight is either "1" or "0", thus the speech recognition
processing works as if different classes of recognition information are switched.
[0021] As for the aforesaid operation, it will be explained more specifically using an example.
Fig. 3 is an initial state of the information processing apparatus when a user powers
on the apparatus. In Fig. 3, reference numeral 301 denotes a screen of the display
unit 102. Further, reference numeral 302 denotes a speech input control panel; and
305 and 308, icons of an application A (e.g., calendar) and an application B (e.g.,
calculator), respectively.
[0022] Further, reference numerals 303, 306, and 309 denote viewpoint detection areas for
the icons 302, 305, and 308, respectively. The viewpoint detection area 303 shows
the viewpoint detection area for the speech input control panel, and when a viewpoint
stays inside of the viewpoint detection area 303, it is determined that the user is
interested in the speech input control panel. Likewise, the viewpoint detection areas
306 and 309 are the viewpoint detection areas for the application A and the application
B, respectively. It should be noted that the viewpoint detection area demarcations
are not displayed on the screen 301. Further, reference numeral 304 denotes a speech
input control dictionary; 307, a dictionary for the application A (calendar); and
310, a dictionary for the application B (calculator).
[0023] Reference numeral 311 denotes a current viewpoint; and 312, a trace of a viewpoint.
A user's viewpoint during a predetermined past time period (e.g., for 1 sec.) is displayed.
Movement of a viewpoint is sometimes stops at positions (expressed with black points
in Fig. 3) and sometimes jumps (expressed with curves in Fig. 3). These kinds of movement
are repeated alternatively.
[0024] Figs. 4 to 6 are graphical representations showing contents of each dictionary. Fig.
4 is a graphical representation showing contents of the speech input control dictionary
304 whose recognition words are "start" and "quit". Fig. 5 is a graphical representation
showing contents of the dictionary for the application A (calendar) 307 whose recognition
words (or sentences) are, for example, "1990" (nineteen-ninety), "May, 1994" (may
nineteen-ninety-four), and "quit". Fig. 6 is a graphical representation showing contents
of the dictionary for the application B (calculator) 310 whose recognition words (or
sentences) are "3 + 6 =" (three plus six equals to) and "quit", for example.
[0025] Next, referring to Figs. 7 to 12, examples of operations will be explained. Figs.
7 to 9 show pictures on the screen in examples of operations according to the first
embodiment. Further, Fig. 10 is a table showing relationship between viewpoints and
weights to be applied, and Figs. 11 and 12 are flowcharts explaining an operational
sequence according to the first embodiment. Step numbers which will be used below
corresponds to those in the flowcharts in Figs. 11 and 12.
[0026] After the information processing apparatus is powered on, a viewpoint is detected
(at a point X1 at this time), and the detected point is displayed on the screen (Fig.
7). Under a current situation, only the speech input recognition dictionary 304 is
related to the viewpoint detection area for the speech input control panel 303 with
weight of P(O) = 0, and neither the dictionaries for the application A nor the application
B are related to their respective viewpoint detection areas (Fig. 10(1)). A detecting
process for detecting whether or not the viewpoint stays in viewpoint detection areas
is performed a plurality of times per second (e.g., 10 times/sec.).
[0027] Then, the viewpoint is moved to a position X2 inside the speech input control panel
302 (Fig. 7). If the viewpoint stays in the viewpoint detection area for the speech
input control panel 303 more than a predetermined number of times (e.g., three times)
during a predetermined time period (e.g., 1 sec.), then the viewpoint manager 106
determines that the user selected the speech input control panel 302 (at steps S10
and S11). Accordingly, the speech input recognition control dictionary 304 is related
to the viewpoint recognition area for the speech input control panel 303 with weight
of P(O) = 1 (Fig. 10(2), at step S12). Note that, in the process at step S12 at this
point, the dictionaries of other applications are not related to the corresponding
viewpoint detection areas, thus no weight is put on these dictionaries.
[0029] By selecting a word having the greatest PT() as a recognized word, "start" is selected,
thus the speech input control is to be performed. As a result, the viewpoint detection
area for the application A 306 is related to the dictionary for the application A
307 with weight of P(A) = 0, and the viewpoint detection area for the application
B 309 is related to the dictionary for the application B 310 with weight of P(B) =
0 (Fig. (4), at steps S14 to S16).
[0030] Then, the viewpoint is moved to a position X5 in the icon of the calendar (Fig. 7).
If the viewpoint stays in the viewpoint detection area for the application A 306 more
than predetermined number of times (e.g., three times) during a predetermined time
period (e.g., 1 sec.), then the viewpoint manager 106 determines that the user selected
the icon of the calendar 305 (at steps S10 and S20). Accordingly, the dictionary for
the application A 307 is related to the viewpoint detection area for the application
A 306 with weight of P(A) = 1, and words (or sentences) in the dictionary shown in
Fig. 5 can be used in the speech recognition processing. Further, weight between the
dictionary for the application B 310 and the viewpoint detection area for the application
B 309 becomes P(B) = 0, and the weight between the speech input control dictionary
304 and the viewpoint detection area for the speech input control panel 303 becomes
P(O) = 0 (Fig. 10(5), at step S21)
[0031] Then, while watching the viewpoint detection area 306 for the application A, the
user pronounces "May, 1994" (may, nineteen-ninety-four) (Fig. 10(6)). Accordingly,
a speech recognition probability PS() of each word (or sentence) in the dictionary
for the application A 307 which is related to the viewpoint detection area for the
application A 306 is determined as follow.
PS(January, 1990) = 0.1
...
PS(May, 1994) = 0.5
...
PS(quit) = 0.01
Then, word recognition probabilities PT() = P() × PS() become,
PT(January, 1990) = P(A) × PS(January, 1990) = 0.1
...
PT(May, 1994) = P(A) × PS(May, 1994) = 0.5
...
PT(quit) = P(A) × PS(quit) = 0.01
[0032] By selecting a word having the greatest PT(), "May, 1994" is selected as a recognized
word. Accordingly, a calendar window 501 of the May 1994 is opened (Fig. 8), and the
dictionary for the application A 307 is related to a viewpoint detection area for
the calendar window 501 with weight of P(A) = 1" (Fig. 10(7), at steps S22 to S24).
In this example, the size of the displayed area of the calendar window 501 and the
viewpoint detection area for the calendar window are identical, however, they do not
always have to the same size and can have different areas as for the icon of the calendar
305 and the viewpoint detection area for the calendar 306.
[0033] Next, while watching the calendar window 501, the user pronounces "December, 2001"
(december, twenty-o-one) (Fig. 10(8)). Since the calendar window is a part of the
viewpoint detection area for the application A, the processes step S20 to step S24
are repeated.
[0034] At step S23, a speech recognition probability PS() of each word in the dictionary
for the application A 307 can be determined as follow.
PS(January, 1990) = 0.1
...
PS(December, 2001) = 0.5
...
PS(quit) = 0.1
Accordingly, word recognition probabilities, PT()= P() × PS(), are,
PT(January, 1990) = P(A) × PS(January, 1990) = 0.1
...
PT(December, 2001) = P(A) × PS(December, 2001) = 0.5
...
PT(quit) = P(A) × PS(quit) = 0.1
[0035] By selecting a word having the greatest PT(), "December, 2001" is selected as a recognized
word, and the application A is executed in accordance with a command, or the recognized
word, at step S24. Thus, a calendar window changes from the one for the May, 1994
to the one for the December, 2001. Next, while watching the viewpoint detection area
for the calendar window 501, the user pronounces "quit" (Fig. 10(9)).
Accordingly, a speech recognition probability PS() for each word in the dictionary
for the application A 307 is determined as follow.
PS(January, 1990) = 0.1
...
PS(quit) = 0.6
Then, word recognition probabilities PT() = P() × PS() are,
PT(January, 1990) = P(A) × PS(January, 1990) = 0.1
...
PT(quit) = P(A) × PS(quit) = 0.6
[0036] By selecting a word having the greatest PT(), "quit" is selected as a recognized
word. Accordingly, the calendar application is closed as well as the calendar window
501 is closed. If the viewpoint is at a position X10 (Fig. 9) at this time, since
it does not belong to the viewpoint detection area for the speech input control panel
303 nor for the application A 306 nor for the application B 309, weight becomes P(O)
= P(A) = P(B) = 0 (Fig. 10(10)).
[0037] Next, the viewpoint is moved to a position X11 in the icon of the calculator 308
(Fig. 9). If the viewpoint stays in the viewpoint detection area for the application
B 309 more than predetermined number of times (e.g., three times) during a predetermined
time period (e.g., 1 sec.), then the viewpoint manager 106 determines that the user
selected the calculator (steps S10 and S30). Accordingly, the dictionary for the application
B 310 is related to the viewpoint detection area for the application B 309 with weight
of P(B) = 1, and words (or sentences or arithmetic equations) in the dictionary shown
in Fig. 6 can be used in the speech recognition processing. Further, the weight between
the dictionary for the application A 307 and the viewpoint detection area for the
application A 306 becomes P(A) = 0, and the weight between the speech input control
dictionary 304 and the viewpoint detection area for the speech input control panel
303 becomes P(O) = 0 (Fig. 10(11), at step S31). In this case, when P(B) = 1, and
the information processing apparatus of the first embodiment can recognize arithmetic
equations and a word "quit" in the speech recognition processing.
[0038] Next, while watching the viewpoint detection area for the application B 309, the
user pronounces "2 + 3 =" (two plus three equals to) (Fig. 10 (12)). Accordingly,
speech recognition probabilities PS() of the arithmetic equations and a word "quit"
in the dictionary for the application B 310 which is related to the viewpoint detection
area for the application B 309 are determined as follow.
PS(1 + 1 =) = 0.1
...
PS(2 + 3 =) = 0.5
...
PS(quit) = 0.01
Accordingly, word (arithmetic equation) recognition probabilities, PT() = P() × PS(),
become,
PT(1 + 1 =) = P(B) × PS(1 + 1 =) = 0.1
...
PT(2 + 3 =) = P(B) × PS(2 + 3 =) = 0.5
...
PT(quit) = P(B) × PS(quit) = 0.01
[0039] By selecting the word (arithmetic equation) having the greatest PT(), "2 + 3 =" is
selected as a recognized word and the calculator application is executed. Accordingly,
a calculator window is opened (not shown), and a calculated result "5" is displayed
(at steps S32 to S34). The calculator window is related to the dictionary for the
application B 310, similarly to the case for the calendar window. Then, while watching
the viewpoint detection area for the application B 309 or the calculator window, the
user pronounces "quit" (Fig. 10(13)).
[0040] Accordingly, the speech recognition probabilities PS() of the arithmetic equations
and a word "quit" in the dictionary for the application B 310 which is connected to
the viewpoint detection area for the application B 309 become as follow.
PS(1 + 1 =) = 0.1
...
PS(quit) = 0.6
Then, the recognition probabilities PT() = P() × PS() become,
PT(1 + 1 =) = P(B) × PS(1 + 1 =) = 0.1
...
PT(quit) = P(B) × PS(quit) = 0.6.
[0041] By selecting a word (or arithmetic equation) having the greatest PT(), "quit" is
selected as a recognized word. Accordingly, the calculator application is closed and
the calculator window is also closed. If the viewpoint is at a position X14 (Fig.
9) at this time, since it does not belong to the viewpoint detection area for the
speech input control panel 303 nor for the application A 306 nor for the application
B 309, weight becomes P(O) = P(A) = P(B) = 0 (Fig. 10(14), at step S40).
[0042] Then, the viewpoint is moved to a position X15 in the speech input control panel
302 (Fig. 9). If the viewpoint stays in the viewpoint detection area for the speech
input control panel 303 more than a predetermined number of times (e.g., three times)
during a predetermined time period (e.g., 1 sec.), then the viewpoint manager 106
determines that the user selected the speech input control panel 302. Accordingly,
weight for each dictionary becomes P(O) = 1, P(A) = 0, P(B) = 0 (Fig. 10(15), at step
S12).
[0044] By selecting a word having the greatest PT() as a recognized word, "quit" is selected
and executed. As a result, the relationship between the viewpoint detection area for
the application A 306 and the dictionary for the application A 307 as well as the
relationship between the viewpoint detection area 309 and the dictionary 310 both
for the application B cease (Fig. 10(17), at steps S17 and S18).
[0045] According to the first embodiment as described above, classes of the recognition
information are automatically switched. Further, it is possible to control the application
so that, if a user pronounces a word "quit", when the user is looking at the icon
of the calendar, the calendar application is terminated, and when the user is looking
at the icon of the calculator, the calculator application is terminated.
[0046] Therefore, according to the first embodiment, on the basis of a viewpoint which stayed
inside a viewpoint detection area set on the display screen and its vicinity, a dictionary
for speech recognition (recognition information) which is related to the viewpoint
detection area is selected. As a result, dictionaries used for speech recognition
are switched properly, thus improving speech recognition level and speech recognition
speed. Further, it is easy to operate the desired application by using speech commands,
thus improving operability.
<Second Embodiment>
[0047] Next, a second embodiment will be explained. In the first embodiment, a condition
for selecting an application is that a viewpoint stays in a predetermined viewpoint
detection area more than a predetermined number of times in a predetermined period
of time, and weight P() for a dictionary for the selected application is set to "1",
whereas weight P() for a dictionary for the non-selected application is set to "0".
In the second embodiment, depending upon how long the viewpoint stays inside a predetermined
area, weights P() for each class of the recognition information (dictionary) which
is related to an icon or a control button on the display screen are controlled.
[0048] Fig. 13 shows a picture on a display screen in an example of an operation according
to the second embodiment. More specifically, Fig. 13 shows a trace of a viewpoint
after a user has watched the screen 301 for 60 sec. Here, it is assumed that the viewpoint
stopped in the viewpoint detection area 303 5 times, 30 times in the viewpoint detection
area 306, 15 times in the viewpoint detection area 309, and 10 times in other area.
Fig. 14 is a flowchart explaining an operational sequence according to the second
embodiment.
[0051] By selecting a word (or sentence or arithmetic equation) having the greatest PT()
as a recognized word, "May, 1990" is selected (at step S52 and S53). Further, an application
related to the dictionary including the recognized word is specified (at step S54),
and a predetermined process (e.g., to open a calendar window of May, 1990) is performed
(at step S55)
[0052] According to the second embodiment as described above, by tracing movement of a viewpoint
for a predetermined time period, and weight for classes of the recognition information
(dictionaries related to applications) is changed on the basis of the traced result.
Therefore, in a case where the viewpoint is moving and does not stay inside a single
area, it is possible to stably determine weight for each dictionary. Accordingly,
the user does not have to stop his viewpoint so as to be in a single area, thus increasing
operability.
<Third Embodiment>
[0053] In the third embodiment, weight P() for each class of the recognition information
related to an icon or a control button is controlled depending upon distances from
the position where a viewpoint is to each viewpoint detection area of an application.
[0054] Fig. 15 shows a picture on a display screen in an example of an operation according
to the third embodiment, and reference numeral 311 denotes a current viewpoint of
a user. Distances from the viewpoint to the viewpoint detection area for the application
A 306, to the viewpoint detection area for the application B 309, and to the viewpoint
detection area for the speech input control panel 303 are denoted by d(A), d(B), and
d(O), respectively, assuming that d(A) = 5 cm, d(B) = 20 cm, and d(O) = 25 cm.
[0057] By selecting a word (or sentence or arithmetic equation) having the greatest PT(),
"May 1990" is selected as a recognized word. Further, an application related to the
dictionary including the recognized word is specified, and a predetermined process
(e.g., opening a calendar window of May, 1990) is performed. In this embodiment, weight
P() is defined as above, however, it can be defined by any functions as far as weight
decreases as the distance becomes large (e.g., P(A) = exp(-d(A))).
[0058] Note that a flowchart according to the third embodiment is similar to that in the
second embodiment, and it is omitted.
[0059] For instance, a current viewpoint is detected at step S50 in Fig. 14, and distances
from the detected viewpoint to each of the viewpoint detection areas are calculated.
Then at step S51, weight for each dictionary is calculated on the basis of the distances
from the detected viewpoint to each of the viewpoint detection areas.
[0060] According to the third embodiment as described above, by detecting distances from
a viewpoint and each of viewpoint detection areas, and weight for each class of the
recognition information (dictionaries related to applications) is controlled on the
basis of the distances. Therefore, the viewpoint does not always have to be inside
of the viewpoint detection areas. In other words, the user does not have to fix his
viewpoint inside of a viewpoint detection area, thus increasing operability.
<Fourth Embodiment>
[0061] Next, a forth embodiment will be now described. In the first embodiment, depending
upon whether or not the viewpoint stays in a viewpoint detection area, weight for
a dictionary related to the corresponding viewpoint detection area where the detected
viewpoint is in is set to "1", and weights for other dictionaries are set to "0".
In the fourth embodiment, a case where, when a viewpoint previously in one of viewpoint
detection areas is currently out of the area, weight for each class of recognition
information is not changed immediately, and keep the same weight for a fixed time
period, will be explained.
[0062] Fig. 16A is a graph showing change of weight according to a fourth embodiment.
[0063] If a time t0 is set to the last minute when a current viewpoint is inside of the
viewpoint detection area for the application A 306, then weight for the dictionary
for the application A 307 at t0 is P(A) = 1.
[0064] As shown in Fig. 16A, if the viewpoint is outside of the viewpoint detection area
306 at time t1, the weight is not changed as long as t1 - t0 < T (e.g., T = 1 sec.),
and P(A) remains 1. On the contrary, if the viewpoint is outside of the viewpoint
detection area 306 at time t2 and t2 - t0 ≧ T, then the weight P(A) is 0.
[0065] Further, when a viewpoint is moved to other position outside of viewpoint detection
areas, it is possible to gradually change weight for each class of the recognition
information. To do so, weight for the dictionary for application P(A) is changed by
following a graph shown in Fig. 16B, for example. Fig. 16B is another graph showing
change of weight according to a fourth embodiment. In Fig. 16B, predetermined values
(e.g., T1 = 1 sec., T2 = 3 sec.) are set to T1 and T2 (T2 > T1).
[0066] According to the fourth embodiment as described above, time allowance is set so as
to correspond to quick movement of a viewpoint. Thus, a user does not have to fix
his viewpoint intentionally, thereby enabling to provide an information processing
apparatus of high quality and of convenient operability.
<Fifth Embodiment>
[0067] A fifth embodiment will now be explained. In the fifth embodiment, sensitivity to
decide weight for classes of recognition information controlled in accordance with
a viewpoint is different for each class of recognition information. A time period
since a viewpoint enters an icon of an application A until the weight becomes 1 is
denoted by Tin(A), and a time period since the viewpoint exits the icon of the application
A until the weight becomes 0 is denoted by Tout(A). Similarly, Tin(B) and Tout(B)
are denoted. Figs. 17A and 17B are graphs showing changes of weight with respect to
time, where Tin(A) = 1 sec., Tout(A) = 2 sec., Tin(B) = 1 sec., and Tout(B) = 1 sec.
In this case, P(A) has higher sensitivity than P(B), thus the class of the recognition
information for the application (A) is more likely selected.
[0068] According to the fifth embodiment, time allowance is set so as to correspond to quick
movement of a viewpoint, as in the fourth embodiment.
<Sixth Embodiment>
[0069] A sixth embodiment will be explained below. In the sixth embodiment, an example in
which weight for each class of recognition information is controlled depending upon
a changed viewpoint when the viewpoint is moved during recognizing speech. Fig. 18
shows a picture on a display screen in an example of an operation according to the
sixth embodiment.
[0071] In the sixth embodiment, after speech is inputted, a viewpoint is detected in advance
to speech recognition process, and weight is determined depending upon the detected
viewpoint.
[0072] The processing sequence is the same as the flowchart in Fig. 14 with steps S50 and
S51 moved between step S52 and step S53. At step S50, a current viewpoint is detected
and, at step S51, weight corresponding to each application can be obtained on the
basis of the detected viewpoint.
[0073] According to the sixth embodiment, the user has a time allowance to move his viewpoint
to a desired icon before finishing the inputting speech.
<Seventh Embodiment>
[0074] In the first and second embodiments explained with reference to Figs. 7 to 9 and
Fig. 13, a viewpoint is detected in a fixed time and space of each viewpoint detection
area. However, the viewpoint can be determined more flexibly, such as by probability.
Fig. 19 is a graphical representation showing correspondence between viewpoint detection
areas and weight according to a seventh embodiment. In Fig. 19, three ranges of the
viewpoint detection areas corresponding to three different weight, P(A) = 0.9, 0.5,
and 0.1, are set. More specifically, the weight is set higher as the viewpoint is
closer to the icon 305. Therefore, the icon which is closest to the user's viewpoint
is considered as the one desired by the user. Accordingly, the probability that the
icon which is considered that the user is more interested in becomes higher. Thus,
there is an area allowance in detecting a viewpoint, thereby increasing operability.
[0075] Note that, in the seventh embodiment, an average of weight corresponding to areas
in which the viewpoint has stayed during a predetermined time period is used as a
final weight. The obtained weight is multiplied by speech recognition probability.
[0076] Further, in the aforesaid embodiments, the viewpoint is detected in image processes
using a camera, however, the present invention is not limited to this, and a detecting
device, such as line of sight detection glasses can be used instead of a camera.
[0077] Furthermore, in the aforesaid embodiment, recognition processing is performed on
the basis of words and their recognition probabilities for the sake of simplicity.
However, the recognition processing can be performed based on sentences, grammar,
and recognition probabilities thereof , for example.
[0078] An object of the present invention which can be achieved by the aforesaid apparatus
or the method according to the above description can also be achieved by a system
storing a program according to the invention. More specifically, by attaching a storage
medium to the aforesaid apparatus, the program loaded from the storage medium can
constitute the new function of the invention. An example of configuration of an apparatus
with the storage medium is shown in Fig. 20. CPU 110 realizes various kinds of functions
by executing a control program stored in ROM 111 or RAM 112. In this example, a floppy
disk 114 is inserted into a floppy disk unit 113, and a control program capable of
realizing functions (functions explained with reference to the aforesaid flowchart
of the speech recognition unit 104, the recognition information controller 105, and
the viewpoint manager 106 shown in Fig. 1) is loaded to the RAM 112, then the CPU
110 executes the program. Note that the control program is not only provided by the
floppy disk, but also by any kinds of storage medium, such as an ID card, a hard disk,
and CD-ROM, or can be loaded through a network.
[0079] A structural feature of the program of the present invention is as shown in Figs.
21 and 22.
[0080] Fig. 21 is a flowchart showing an operational sequence to execute each program module
provided by a storage medium. Further, Fig. 22 is a memory map showing a storage situation
of program modules.
[0081] A detecting process module 401 is for controlling the viewpoint detector 103 to detect
the viewpoint of a user. A changing process module 402 is for changing kinds of recognition
processes in accordance with the detected viewpoint. Here, to change kinds of recognition
processes, classes of recognition information (dictionary for recognition) to be used
are changed in accordance with where the viewpoint stays, for instance. Alternatively,
weight to be applied to each class of recognition information is changed in accordance
with the viewpoint in the recognition process. A recognizing process module 403 is
for recognizing speech inputted from the speech input unit 101 by following the changed
recognition processes. Further, a transferring process module 404 is for transferring
the recognized result to an application corresponding to the class of recognition
information used by the recognizing process module.
[0082] The processes by each of aforesaid modules corresponds to control represented by
flowcharts in Figs. 11, 12, and 14.
[0083] Further, the present invention can be applied to a system constituted by a plurality
of devices or to an apparatus comprising a single device.
[0084] Furthermore, the invention is applicable also to a case where the invention is embodied
by supplying a program to a system or apparatus. In this case, a storage medium, storing
a program according to the invention, constitutes the invention. The system or apparatus
installed with the program read from the storage medium realizes the functions according
to the invention.
[0085] As described above, according to the present invention, user's attention and interest
can be determined properly from his viewpoint, thereby it is possible to control recognition
information to be used in speech recognition in accordance with the detected viewpoint.
Accordingly, capabilities of the apparatus, such as speech recognition probability
and speed of recognition, are improved.
[0086] As many apparently widely different embodiments of the present invention can be made
without departing from the spirit and scope thereof, it is to be understood that the
invention is not limited to the specific embodiments thereof except as defined in
the claims.
1. A speech recognition apparatus for recognizing inputted speech characterized by comprising:
detecting means (103) for detecting a viewpoint of a user;
changing means (105) for changing recognition processes for speech recognition
in accordance with the viewpoint detected by said detecting means; and
recognizing means (104) for executing the speech recognition in accordance with
the recognition process changed by said changing means.
2. The speech recognition apparatus according to claim 1, wherein said changing means
changes classes of recognition information to be used for the speech recognition in
accordance with the viewpoint detected by said detecting means.
3. The speech recognition apparatus according to claim 2, wherein said changing means
sets a plurality of areas each of which is related to a different class of the recognition
information, selects one of the plurality of areas in accordance with the viewpoint
detected by said detecting means, and determines a class of the recognition information,
related to the selected area, to be used in the speech recognition.
4. The speech recognition apparatus according to claim 3, wherein one of the plurality
of areas is set for each respective image, shown on a display screen, representing
an application and has a predetermined size in which the image is included.
5. The speech recognition apparatus according to claim 3, wherein said changing means
detects a position where the viewpoint stays on the basis of the viewpoint detected
by said detecting means and selects one of the plurality of areas which includes said
position.
6. The speech recognition apparatus according to claim 1, wherein said changing means
controls weight for each class of the recognition information to be used in the speech
recognition in accordance with the viewpoint detected by said detecting means.
7. The speech recognition apparatus according to claim 6, wherein said changing means
sets a plurality of areas each of which is related to a different class of recognition
information and controls the weight for each class of the recognition information
in accordance with relationship between the viewpoint detected by said detecting means
and each of the plurality of areas.
8. The speech recognition apparatus according to claim 7, wherein one of the plurality
of areas is set for each respective image, shown on a display screen, representing
an application and has a predetermined size in which the image is included.
9. The speech recognition apparatus according to claim 7, wherein said changing means
counts a number of times a viewpoint has stayed in the plurality of areas during a
predetermined time period on the basis of the viewpoint detected by said detecting
means, and controls weight for each class of the recognition information to be used
in the speech recognition in accordance with the counted number.
10. The speech recognition apparatus according to claim 7, wherein said changing means
detects a position where the viewpoint stays on the basis of the viewpoint detected
by said detecting means, and controls the weight for each class of the recognition
information in accordance with relationship between the position and each of the plurality
of areas.
11. The speech recognition apparatus according to claim 7, wherein said changing means
assigns a maximum value to weight for a class of the recognition information related
to one of the plurality of areas in which the viewpoint stays, and assigns a minimum
value to weight for other classes of the recognition information.
12. The speech recognition apparatus according to claim 11, wherein, in a case where the
viewpoint exits from the area where it has stayed, said changing means changes the
weight for the class of the recognition information related to the area in which the
viewpoint had stayed to the minimum value after a predetermined time period has elapsed.
13. The speech recognition apparatus according to claim 11, wherein, in a case where the
viewpoint exits from the area where it has stayed, said changing means gradually decrease
the weight for the class of the recognition information related to the area in which
the viewpoint had stayed toward the minimum value as time elapses.
14. The speech recognition apparatus according to claim 13, wherein rate of weight changes
differ from each other depending upon different classes of the recognition information.
15. The speech recognition apparatus according to claim 11, wherein, in a case where the
viewpoint enters one of the plurality of areas, said changing means gradually increases
the weight for a class of the recognition information related to the area which the
viewpoint entered toward the maximum value as time elapses.
16. The speech recognition apparatus according to claim 15, wherein rate of weight changes
differ from each other depending upon classes of the recognition information each
of which is related to one of the plurality of areas.
17. The speech recognition apparatus according to claim 1, wherein said changing means
changes recognition processes on the basis of a viewpoint detected by said detecting
means just before the speech recognition is performed by said recognizing means.
18. The speech recognition apparatus according to claim 7, wherein each of the plurality
of areas is divided into a plurality of fields, and said changing means controls weight
for each class of the recognition information in accordance with the field in which
the viewpoint detected by said detecting means is.
19. A speech recognition method for recognizing inputted speech characterized by comprising:
a detecting step (201) of detecting a viewpoint of a user;
a changing step (203) of changing recognition processes for speech recognition
in accordance with the viewpoint detected in said detecting step; and
a recognizing step (206) of executing the speech recognition in accordance with
the recognition process changed in said changing step.
20. The speech recognition method according to claim 19, wherein, in said changing step,
classes of recognition information to be used for the speech recognition are changed
in accordance with the viewpoint detected in said detecting step.
21. The speech recognition method according to claim 20, wherein, in said changing step,
a plurality of areas each of which is related to a different class of the recognition
information are set, and one of the plurality of areas is selected in accordance with
the viewpoint detected in said detecting step, and a class of the recognition information,
related to the selected area, to be used in the speech recognition is determined.
22. The speech recognition method according to claim 21, wherein one of the plurality
of areas is set for each respective image, shown on a display screen, representing
an application and has a predetermined size in which the image is included.
23. The speech recognition method according to claim 21, wherein, in said changing step,
a position where the viewpoint stays is detected on the basis of the viewpoint detected
in said detecting step and one of the plurality of areas which includes said position
is selected.
24. The speech recognition method according to claim 19, wherein, in said changing step,
weight for each class of recognition information to be used in the recognition process
is controlled in accordance with the viewpoint detected in said detecting step.
25. The speech recognition method according to claim 24, wherein, in said changing step,
a plurality of areas each of which is related to a different class of recognition
information are set and the weight for each class of the recognition information is
controlled in accordance with relationship between the viewpoint detected in said
detecting step and each of the plurality of areas.
26. The speech recognition method according to claim 25, wherein one of the plurality
of areas is set for each respective image, shown on a display screen, representing
an application and has a predetermined size in which the image is included.
27. The speech recognition method according to claim 25, wherein, in said changing step,
a number of times a viewpoint has stayed in the plurality of areas is counted during
a predetermined time period on the basis of the viewpoint detected in said detecting
step, and weight for each class of the recognition information to be used in the speech
recognition is controlled in accordance with the counted number.
28. The speech recognition method according to claim 25, wherein, in said changing step,
a position where the viewpoint stays is detected on the basis of the viewpoint detected
in said detecting step, and the weight for each class of the recognition information
is controlled in accordance with relationship between the position and each of the
plurality of areas.
29. The speech recognition method according to claim 25, wherein, in said changing step,
a maximum value is assigned to weight for a class of the recognition information related
to one of the plurality of areas in which the viewpoint stays, and a minimum value
is assigned to weight for other classes of the recognition information.
30. The speech recognition method according to claim 29, wherein, in a case where the
viewpoint exits from the area where it has stayed, the weight for the class of the
recognition information related to the area in which the viewpoint had stayed is changed
to the minimum value, after a predetermined time period has elapsed, in said changing
step.
31. The speech recognition method according to claim 29, wherein, in a case where the
viewpoint exits from the area where it has stayed, the weight for the class of the
recognition information related to the area in which the viewpoint had stayed is gradually
decreased, toward the minimum value as time elapses, in said changing step.
32. The speech recognition method according to claim 31, wherein rate of weight changes
differ from each other depending upon different classes of the recognition information.
33. The speech recognition method according to claim 29, wherein, in a case where the
viewpoint enters one of the plurality of areas, the weight for a class of the recognition
information related to the area which the viewpoint entered is gradually increased,
toward the maximum value as time elapses, in said changing step.
34. The speech recognition method according to claim 33, wherein rate of weight changes
differ from each other depending upon classes of the recognition information each
of which is related to one of the plurality of areas.
35. The speech recognition method according to claim 19, wherein, in said changing step,
recognition processes on the basis of a viewpoint detected in said detecting step
are changed just before the speech recognition is performed in said recognizing step.
36. The speech recognition method according to claim 25, wherein each of the plurality
of areas is divided into a plurality of fields, and, in said changing step, weight
for each class of the recognition information is controlled in accordance with the
field in which the viewpoint detected in said detecting step is.
37. An information processing apparatus for executing an application characterized by
comprising:
detecting means (103) for detecting a viewpoint of a user;
changing means (105) for changing recognition processes for speech recognition
in accordance with the viewpoint detected by said detecting means;
recognizing means (104) for executing the speech recognition in accordance with
the recognition process changed by said changing means;
selecting means for selecting an application to be executed; and
executing means for executing the application on the basis of a speech recognition
result by said recognizing means.
38. The speech recognition apparatus according to claim 37, wherein said changing means
selects a class of recognition information to be used for the speech recognition in
accordance with the viewpoint detected by said detecting means, and said selecting
means selects an application in accordance with the class of the recognition information
selected by said changing means.
39. The speech recognition apparatus according to claim 37, wherein said changing means
controls weight for each class of the recognition information to be used in the speech
recognition in accordance with the viewpoint detected by said detecting means, and
said selecting means selects an application in accordance with the class of the recognition
information to be used for obtaining the recognition result by said recognizing means.
40. The information processing apparatus according to claim 37, further comprising display
means for displaying images representing each of a plurality of applications,
wherein said changing means sets a plurality of areas each of which includes each
of the plurality of images and changes recognition processes in accordance with relationship
between the viewpoint detected by said detecting means and each of the plurality of
areas.
41. A computer program product comprising a computer usable medium having computer readable
program code means embodied in said medium for speech recognition processing, said
product including:
first computer readable program code means (401) for a detecting step of detecting
a viewpoint of a user;
second computer readable program code means (402) for a changing step of changing
recognition processes for speech recognition in accordance with the viewpoint detected
in said detecting step; and
third computer readable program code means (403) for a recognizing step of executing
the speech recognition in accordance with the recognition process changed in said
changing step.
42. A computer program product according to claim 41, wherein said second computer readable
program code means changes classes of recognition information to be used for the speech
recognition in accordance with the viewpoint detected in the detecting step.
43. A computer program product according to claim 41, wherein said second computer readable
program code means controls weight of each class of the recognition information to
be used in the speech recognition in accordance with the viewpoint detected in the
detecting step.
44. A computer program product according to claim 41 further comprising fourth computer
readable program code means for selecting one of the plurality of applications in
accordance with a class of the recognition information to be used in the recognising
step and transmitting a recognised result in said recognition process to the selected
application.
45. An apparatus according to any of claims 1 to 18, wherein said detecting means (103),
said changing means (105) and said recognising means (104) are implemented by program
control of a computer apparatus.
46. A recording medium having the program for control according to claim 45 recorded thereon.
47. Use of view direction measurements to determine contact in speech recognition.
48. A method of speech recognition using a system having a display, characterised by changing
the recognition process in dependence upon where in the display the user is looking.