| (11) | EP 2 519 935 B1 |
| (12) | EUROPEAN PATENT SPECIFICATION |
|
|
| (54) | AUDITING VIDEO ANALYTICS TESTS FÜR VIDEOANALYSEN VÉRIFICATION D'ANALYSE VIDÉO |
|
| |||||||||||||||||||||||||||||||
| Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention). |
FIELD OF THE INVENTION
[0001] The present invention relates to video analytics, and more particularly to auditing video analytics data to distinguish errors.
BACKGROUND
[0002] Video analytics comprise a variety of technologies that use computers and other programmable devices and associated systems to analyze video input for specific data, behavior, objects, etc. within a video input. Video analytics may be implemented in software applications running on processors inside a computer or device, including on embedded programmable device platforms in video cameras, recording devices, or specialized video processing units. Video analytics may be useful in evaluating the contents of video to determine information with respect to the actions and movements of people within the video. Video analytics are methods, processes and systems used to evaluate the contents of video images to determine specified information from the video image. The information may be readily obtained from an image or picture, for example, a review of a number of vehicles on a section of roadway within an image may indicate traffic congestion. Multiple images, perhaps from multiple cameras, may also be considered to determine specified information, for example, considering images from different times, or from multiple camera feeds, may enable determination of speed of traffic flow or predict slowdowns due to heavy flow on the same roadway. Changing weather patterns may be observed, future weather predicted, as well as animal movements and migrations. Physical phenomena, such as structural changes and failures (for example, cracks in structural members, bending and sagging, rust, peeling or chipping paint, erosion, surface wearing and cracking, stream bed changes, silt build-up, etc.) may also be observed or detected, for example, through comparison of elements in images to the same elements in earlier-in-time images. Thus, a variety of specified information relative to the actions and movements of natural phenomena, objects, flora and fauna may thus be recognized or determined through analyzing image content through video analytics.
[0003] Video analytics applications may be further useful in determining the meaning or import of aspects of the video images, for example, to determine the motivations, implications or meaning of an action or inaction reflected by a video input, in one aspect to thereby determine or predict what a person's actions within a video show or imply with respect to expected or specified behavior. Thus, video analytics may be used to determine whether a person shown in a given area is trespassing or whether their presence is authorized, for example, by analyzing clothing (uniform vs. non-uniform), time of day (during business hours vs. after business hours), means of entry (correlated to a previous movement through an admission gate?), etc. Accordingly, video analytics are useful in a wide range of applications, and some illustrative but not exhaustive examples include property protection, safety and security contexts.
[0004] Automated video analytics systems and methods performed by computers and other programmable devices may be useful in enhancing efficiencies in auditing video data by enabling programmable devices to perform analysis on video feeds and generate alerts to human auditors of events or items of interest occurring within the video feeds for further review and analysis by human auditors. For example, alerts may be generated by applying computer vision and pattern recognition (CVPR) algorithms to a video feed input, such as by algorithms taught by "Recognition of Repetitive Sequential Human Activity" by Quanfu Fan et al., IBM T. J. Watson Research Center, Hawthorne, NY, USA, presented at CVPR 2009; or also by "GRAPH BASED EVENT DETECTION FROM REALISTIC VIDEOS USING WEAK FEATURE CORRESPONDENCE", by Lei Ding et al., IBM T. J. Watson Research Center, Hawthorne, NY, USA, presented at ICASSP 2010; and still other appropriate examples for use with present invention will be apparent to one skilled in the art. Alerts to possible abandoned bags within an areas under video surveillance may also be generated by application of abandoned baggage detection algorithms to detect possible abandoned bags from video stream, such as the algorithms taught by "Location Based Baggage Detection for Transit Vehicles" by Guillaume Milcent and Yang Cai, CMU-CyLab-05-008 December 9, 2005, Carnegie Mellon University, Pittsburgh, PA, USA, and still other examples appropriate for use with the present invention will be known to one skilled in the art.
[0005] However, the efficacy of prior art systems incorporating automated video analytics systems is limited. Human auditing is generally required to apply human analysis and discretion in adjudicating and parsing genuine events of interest from false alerts, which requires human review of video feeds used by the automated video analytics systems to trigger alerts, and thus such systems and methods still require the streaming of video data to a human auditor, resulting in large bandwidth and/or memory storage requirements to provide real-time and/or archival video stream data to an auditor. As automated video analytics systems generally generate a large number of false alerts, a substantial amount of video streaming may still be required, reducing the bandwidth, storage and times savings achieved over human analysis of the raw video streams.
[0006] Furthermore, wading through the automated video analytics systems results may be an extremely labor-intensive process. Human auditing implementation problems and costs may also rise in proportion to the numbers of cameras or video feeds utilized, with efficiencies diminishing with large numbers of cameras or video feeds. Auditing systems may also suffer from limited network bandwidths if data being audited is accessed through network systems, for example, remotely through browsing applications.
[0007] WO2007/145623 A1 discloses a method for verifying an alarm system event comprising transmitting image data corresponding to a plurality of images associated with the event. The image data is processed to create one or more processed images so as to allow an operator to visually observe changes in the plurality of images. The alarm system event is triggered by one or more sensors, such as wired or wireless motion sensors, heat sensors, infra-red sensors, glass break sensors, microwave sensors, acoustic sensors, ultrasonic sensors, sonic sound sensors, photoelectric sensors, pressure mats/sensors or magnetic sensors.
BRIEF SUMMARY OF THE INVENTION
[0008] The invention is defined by the appended claims. In a first aspect the present invention provides a method according to appended claim 1. The set of alert images generated from the visual image stream constitutes a "visual essence" of the alert. That is to say that the essence of the alert is encapsulated as an image contemporaneous to the alert and at least one other image before or after the alert, so as to encapsulate the visual context of the alert in the set of alert images as a visual essence of the alert. Human reviewers can make sense of what happens quickly from only a few images of a situation. Thus, embodiments of the present invention extract a "visual essence" of an alert as a set of images as a brief visual summary of the alert integrating representative images and potentially other cues available. The visual essence allows a reviewer to determine very quickly whether or not further investigation is needed without reviewing the visual image stream (such as a video stream). The reviewer need only review the visual image stream for the time of the alert when further evidence or context is required. Thus, the set of alert images as a visual essence provides minimal and sufficient information for the reviewer to quickly filter out false alerts, thus the productivity of the limited human resources is maximized. The set of alert images can be compact and condensed so they require much less bandwidth than the visual image stream, such as video, for visualization in an auditing system. More conveniently, the set of alert images can be downloaded to and stored in local machines for fast system response. The present invention is particularly suited to, for example, retail fraud detection and abandoned object detection.
[0009] Optionally the method further comprises: archiving the visual image stream in a repository; and retrieving the additional images provided from the visual image stream.
[0010] Optionally, generating the set of alert images further comprises: selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
[0011] Optionally the method further comprises analyzing the visual image stream via computer vision and pattern recognition to determine an occurrence of the triggering alert.
[0013] Optionally the method further comprises: archiving the visual image stream in a repository; and retrieving the additional images provided from the visual image stream.
[0014] Optionally, generating the set of alert images further comprises: selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
[0015] In a third aspect, the present invention provides a system according to appended claim 7. Optionally the system further comprises archiving means for archiving the visual image stream in a repository; and retrieving means for retrieving the additional images provided from the visual image stream.
[0016] Optionally the generating means further comprises selecting means for selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
[0017] Optionally the system further comprises analysing means for analysing the visual image stream via computer vision and pattern recognition to determine an occurrence of the triggering alert.
[0019] Optionally the system further comprises archiving means for archiving the visual image stream in a repository; and retrieving means for retrieving the additional images provided from the visual image stream.
[0020] Optionally the system further comprises text incorporating means for incorporating text data associated with the contemporaneous image and the at least one other image into the set of alert images such that the text data is available to the reviewer.
[0021] Optionally the generating means further comprises selecting means for selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert. According to a fifth aspect, the present invention accordingly provides a computer program element comprising computer program code to, when loaded into a computer system and executed thereon, cause the computer to perform the steps of the method described above.
BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings in which:
Figure 1 is a diagrammatic illustration of an embodiment of a system for human auditing of video analytics data according to a preferred embodiment of the present invention;
Figure 2 is a diagrammatic flow diagram illustration of a system according to a preferred embodiment of the present invention;
Figure 3 is an overhead perspective state diagram of a physical location appropriate for analysis through visual essences according to a preferred embodiment of the present invention; and
Figure 4 is a diagrammatic illustration of a computerized implementation of an embodiment of the present invention.
[0023] The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific parameters of the invention. The drawings are intended to depict only typical embodiments of the invention, and therefore should not be considered as limiting the scope of the invention. In the drawings, like numbering represents like elements.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
[0025] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
[0026] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in a baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
[0027] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0028] Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
[0029] Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
[0030] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
[0031] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
[0032] Referring now to Figure 1, an embodiment of a method or system for human auditing of video analytics data according to the present invention is illustrated. At least one data sensor 110 provides a visual image stream as video data (which may include video, audio, text, etc.) to a data archive manager 112 and to a video analytics component 114. Although functions of the video analytics component 114 may be performed by a human auditor through real-time review of video feeds or other data, embodiments of the present invention may use an automated video analytics system for the video analytics component 114 to enable automated alert generation, which may realize rates of data throughput which are not possible through a simple human review of video or image feeds (for example, to distinguish and detect some motions from other motions in regions of interest (ROI's) in video images, or to distinguish correlations to other transaction data stream items within and without specific time thresholds, as discussed more fully below).
[0033] The video analytics component 114 analyzes the video data input and determines the presence or occurrence of one or more alerts in the video data and outputs alert data and associated metadata at 118. A set of alert images for each of the alerts is generated as a "visual essence" 116 to provide minimal but sufficient information for a human auditor to quickly review and immediately attempt to make a true-false determination with respect to the alert to thereby filter out false alerts in a more efficient manner and presentation with respect to time and data size relative to reviewing the larger originating video and data streams. The set of alert images (hereafter generally referred to as the "visual essence") is composed and displayed at a results display 120 in a manner wherein it may be quickly reviewed and the accuracy of the alert quickly determined on a faster basis relative to reviewing a larger video feed. The productivity of limited human auditor resources may thus be maximized by reducing the amount of video and other data that must be considered in order to make determinations regarding event occurrences represented by the original video stream.
[0034] More particularly, Figure 2 illustrates a system or method for auditing video information as a function of visual essence data according to a preferred embodiment of the present invention. An alert of an event of concern is provided at 202 from video analytics alert results, for example as generated from the video analytics device 114 of Figure 1. In response to the alert at 202, at 204 a visual essence is created for association with the alert by extraction of a subset of images from video stream data, and optionally incorporating text and other non-visual data. The video and other data may be extracted in real-time directly from video cameras or other data sensors (for example, bar code scanners, turnstiles, etc.), or from video and non-video data stored in one or more storage devices (for example, from the data archive manager 112 of Figure 1), and further as a function of a context of the alert as provided or indicated by the video analytics. The subset still images, and optional textual or non-visual data, provide a summary of information automatically selected as a function of image content and system context, the set of information condensed relative to the larger video and non-visual source data and generated as a visual essence associated with the alert. The information may be derived from multiple resources (multiple cameras, transaction logs, identification data and logs, etc.).
[0035] Thus, the visual essence is presented to a human auditor in association with the alert at 206 to enable a human analyst to review and analyze the presented visual essence information at 208 to immediately make a true-false determination with respect to the alert, if possible, to thereby filter out false alerts in a more efficient manner and presentation with respect to time and data size relative to reviewing the larger originating video and data streams. Generally a human can more quickly parse out errors and false positives than a computer video analytic system. Although human intelligence is much better, it is generally more expensive and inefficient with respect to large amounts of video data; embodiments of the present invention utilizes the visual essences to make the human auditor more efficient and less costly than prior art human and computerized auditing systems. If the information conveyed by the visual essence is sufficient and no further information or data is needed, then at 212 the alert is quickly culled as false. If instead the analysis is inconclusive or more information is otherwise needed as indicated at 210, then the auditor may request more data, for example requesting additional data or streaming video and browsing the additional video at 214.
[0036] The visual essence comprises a single image, a set of images or a compact stream subset (and optionally incorporating text and other non-visual data) from a larger, original data source, i.e. a video stream and optionally other associated data. The visual essence may have a much smaller data size relative to a video feed or portion thereof (for example, one, two or three images with text compared to a stream of many more images over a period of time), reducing bandwidth, memory and other system resource demands. Examples of the result display 120 include a browser window displayed in a computer desktop, chat and text messaging applications that display images on cellular phones and other mobile devices, and additional implementations will be apparent to one skilled in the art.
[0037] The visual essences 116 each comprise at least one image that is contemporaneous in time with the alert event triggering the creation of the visual essence (i.e. occurring at the same time or immediately proximate in time), and at least one other image that occurs before or after the contemporaneous image. The visual essences 116 are then provided to an analyst in a results browser or other presentment device at 120 for auditing for determination as to the validity of the alert, for example, to determine whether the alert associated with the generated visual essence is a false alert or a true indication of a specified or associated event occurrence.
[0038] Data may be retrieved from the data archive manager 112 for generation of the visual essences at 116, and the auditor may also communicate with the data archive manager 112 at 120 to retrieve additional data as needed. In the present example, the visual essence is provided to a human analyst through a display device 120, though in other embodiments auditing at 120 may be performed by automated video analytics systems applying one or more algorithms alone or in combination with a human auditor.
[0039] The visual essence is selected as a function of the alert subject matter context to provide for an auditor to quickly analyze and identify information within the source video data in order to efficiently make sense of what happens in the domain of the selected video data with respect to specified events, actions, behaviors, etc. of concern (alerts) from a limited sampling of the video data, for example, to determine that an alert is false or true from a review of only one single image or a small set of images rather than reviewing the much larger (in both data size and time of display) original video feed of a larger set of images. The efficacy of human auditing of the video analytics data is improved by first extracting a visual essence of an alert that provides a brief summary of the alert integrating representative images and other cues available. The visual essence allows the auditor to quickly determine (for example, in seconds) whether or not further investigation is needed without watching any source video or other non-essence video data: the auditor needs only to dive into an archive system for more evidence when necessary.
[0040] As a "false" determination may be made solely from the visual essence at 212, review time and resources saving are realized over prior art systems that require browsing of the video stream or some other additional data retrieval and review. Embodiments of the present invention enable an auditor to prune out a large portion of false alerts in a shorter amount of time relative to review of larger source video or multimedia stream inputs as required in the prior art, and to provide visual essences associated with the video analytics results to an analyst in a video investigation process that may be shortened relative to prior art auditing processes, without compromising analysis accuracy.
[0041] Embodiments of the present invention provide frameworks for intelligently managing computational and bandwidth resources to maintain an acceptable level of performance for video analytics. Thus, a framework may be provided for visually summarizing the results of an analytics system, wherein a visual summary essence is depicted in one or more (in some examples, annotated) images and text. Based on an analyst inspection of the visual summary or essence images, more detailed information can be requested for further investigation at 120, e.g. retrieving larger selections of source video from the data archive manager 112. Visual summary essence images may also be streamed ahead of time.
[0042] Visual essences are generally compact and/or condensed so that they require less bandwidth than video data for visualization in an auditing system; only frames or limited video selections need be transmitted to the auditor for review, not a much larger or entire video feed as is typically required in the prior art. Smaller visual essences (for example, discrete data files) may also be more readily downloaded to, and stored in local machines for faster system responses compared to larger video feeds. Further, as analysts need only watch summaries about an event, pruning operations are faster and their throughput rate may be higher. Human auditors generally make fewer mistakes than results obtained by auditing through filtering by machine intelligence, and thus human auditors may provide better efficiencies in applications where true events may occur infrequently, such as in retail fraud detection or abandoned baggage alerts, and still other examples will be apparent to one skilled in the art.
[0043] Embodiments of the present invention may be utilized in a variety of applications wherein visual events may be associated with discrete, time-ordered transactions. For example, visual events may be created with respect to human movements in certain areas and matched to turnstile admissions, vehicles detected in relation to a transit point, badge or card swipes from an automated door lock, ticket kiosk, etc., which indicate trespassing, theft, unauthorized access to restricted areas, etc. Approaches may focus on a variety of data outputs associated with human activities, for example business transaction logs, retail receipts, injury reports, etc., and embodiments of the present invention are not limited to only those examples provided herein.
[0044] Examples of embodiments of the present invention may be implemented to quickly distinguish fraudulent retail "sweet-hearting" events from "long-scanning" and other non-fraudulent events associated with alerts triggered through review of a retail or other scanning area. For example, Figure 3 is an overhead perspective state diagram of a plurality of possible movements and actions 162 of a cashier in a retail context (for example, of a scanning lane) with respect to possible movement vectors 152, 154. (It will be understood that as used herein "cashier" is a generic term to denote a person scanning an item, and that embodiments contemplate that the scanning person may be a clerk or cashier as well as a customer, bagger, manager or other person.) In a simple scanning motion path vector 152 an item is picked-up from a pick-up area 156, scanned by passing the item within scanning range of a scanner in a scanning area 160 and then put down in a bagging or drop area 164. Examples of the pick-up areas 156 include an intake belt, a counter, a shopping basket and a cart, and the put-down area 164 may be one or more of a distribution belt, counter, shopping basket or shopping cart, each of which may entail different motions. The vectors 154 each represent an observed motion relative to other actions 162 of a cashier and observable in a video feed, illustratively including unloading items, and getting and scanning loyalty cards, supervisor/assistant override cards (SA), coupons or other bar-code cards.
[0045] A significant portion of retail shrink may be attributed to employees and occurs around cashiers at the point of sale (POS). Sweet-hearting or "fake scanning" describes the action of a cashier in intentionally failing to scan or otherwise enter an item into a retail transaction in order to provide the merchandise free of charge for a customer presenting the items for purchase, usually by moving the item from an input (pick-up) location through the processing (scan) area and into the output (drop or bagging) area by covering up the item bar code, stacking an item on top of another to occlude the scanner from reading the code or passing the item around the scan area during a scan motion to intentionally avoid a bar code reader in the scan area with respect to a bar code of the free item.
[0046] Although video analytics, bar code transaction review and other techniques may detect possible sweet-hearting activities and generate alerts for further analysis and review of video feeds of a retail scanning area, such alerts usually fail to distinguish long-scanning events. Long-scanning events exhibit a long time gap between pick-up, scan and/or drop events in a given retail transaction of an object, and wherein an otherwise legitimate scanning event may result in an erroneous report of a missing or fake scan or other system failure to capture the scanned information, through no malicious intent by the scanning cashier. For example, the motion vectors 152, 154 may entail different motions relative to others of the vectors 152, 154, and they may also have different time signatures (for example, it may take longer to reach into a cart for a pick-up or a drop relative to a belt pick-up or drop). Movement of items from the pick-up area 156 may bypass the scanning area 160 before entering the put-down area 164 for legitimate purposes, for example, to set aside in combination with directly keying-in of an item code with a missing or damaged bar code, or in weighing an item sold by weight. Video feeds for individual scans may also differ as a result of different cashier or environmental characteristics, for example, lighting may change due to time of day (daylight versus artificial lighting at night), and the age or physical size or ability of the individual cashiers may impact the range of arm motion, motion paths selected or the times to complete similar paths relative to other cashiers, and any of these attributes may result in a long-scan falsely reportable as a sweet-hearting. For example, in some prior art systems, if a time gap is longer than an expected or predefined time interval (for example two seconds, five seconds, one minute, or another value from a range encompassing said values, etc.), the scanning system may incorrectly log an item scan, or even fail to log the transaction or item scanned, and thus a fake scan/sweet-hearting alert may be issued under an assumption that the cashier has intentionally avoided scanning an item, where in fact the cashier may be performing appropriately and no fake scan has occurred.
[0047] Accordingly, in one example of a method or system according to an embodiment of the present invention for implementation in the retail context of Figure 3, transaction bar code events reported in log data from the scanner area 160 are utilized to generate visual essence data comprising discrete images from an original video stream or larger collection of still images and as a function of a temporal alignment with the bar code events for analysis through video analytics for alert generation (e.g. at 202 of Figure 2). In one embodiment, the visual essence comprises a set of three still images subset frames of video: (1) a pick-up frame {P} showing the pick-up area 156 at a predetermined amount of time prior to a logged bar code event {b} (for example, one second prior), which is expected to show a retail clerk picking up an item from the pick-up area 156; (2) a scanning action frame {S} showing the scanning area 160 at the time of the logged bar code event {b}, which is expected to show the clerk moving the picked-up item over a scanner in the scanning area 160; and (3) a deposit or drop frame {D} acquired from the check-out or customer delivery area 164 at a predetermined amount of time after the logged bar code event {b} (for example, one second later), which is expected to show the clerk depositing the item in the customer delivery area 164. The bar code transaction should be generated or otherwise associated with each of the three basic visual events that occur in this temporally sequential order: first picking up an item, then scanning the picked-up item in, and lastly depositing the picked-up and scanned item. Thus, each barcode scan {b} should correspond to a set of one each of these three different image primitives types {PSD}.
[0048] The visual essence may also be presented with a text identifier of the item registered in the logged bar code event by the scanner, for example "1 Lb bag of Carrots, Bar Code #xyz." Thus, a human auditor may compare the three visual essence images to the text identifier and quickly determine whether an alert of fraudulent action is false (for example, each of the images show a one pound bag of carrots as expected), or if instead the event may be true (for example, one or more of the images shows a wine bottle instead), without having to review a much larger original video selection as would be required in the prior art. If needed, the auditor may optionally request more data from the data repository 112 (for example, a stored vide stream with more images of the same event, or historic video or data relative to the clerk or the item, etc.) for an additional true/false determination.
[0049] Embodiments of the present invention may also provide improved efficiencies in public transportation security applications. Thus, a data stream from data sensors may be video streams from one or more cameras wherein video analytics report alerts for possible abandoned bags, for example, generating a list of suspected abandoned bags through application of an abandoned baggage detection algorithm to the video streams. Exemplary algorithms are taught by "Location Based Baggage Detection for Transit Vehicles" by Guillaume Milcent and Yang Cai, CMU-CyLab-05-008 December 9, 2005, Carnegie Mellon University, Pittsburgh, PA, USA, and still other examples appropriate for use with embodiments of the present invention will be known to one skilled in the art. The visual essence data 116 may comprise a single image of a suspected abandoned bag; if instead of an abandoned bag, the alert is generated by a stationary person (for example, someone sleeping in a passenger terminal seat and thereby remaining stationary for a longer-than-expected timeframe, causing an automated system to mistake the person for a large luggage item), then a human auditor reviewing the single image visual essence will quickly determine that this is not a bag and that the alert is false, wherein an automated system may have to expend significant computational resources to distinguish the person from an object, or a prior art auditor would have to review a much larger video feed selection, to make the same false-alert determination. In another example, a quick comparison of a single image of a suspected bag and another image taken before and after a suspected abandonment recognition that has triggered an alert, along with text information (indicating time frames as to how long the bag has been left outside of the apparent control of any nearby person) may enable a quick determination that the alert is false as the image reveals a person associated with the bag still proximate to the bag though not immediately adjacent to the bag. Lighting changes may also cause mistakes as to object presence. For example, a shadow may result in object detection, but a quick review of a visual essence can determine that the shadow is not an object, a determination that may be very difficult for a computer.
[0050] In another aspect, a good event detection or recall rate (for example, 90% of events noted and recalled by alerts) may require a lot of false positives to generate a meaningful Receiver Operating Characteristic (ROC) curve through plotting sensitivity versus specificity of results. Lowering false positives returned may provide efficiencies in the prior art, but it will also harm the recall, for example, catching only 50% of true events. To catch a relatively small amount of true events within a large stream of events, a significant amount of alerts must be generated from false events. Embodiments of the present invention enables processing of a high level of false events by enabling the rapid culling of the false events through quick human review of the visual essence data, and thereby enables high recall rates where true event frequencies may be low.
[0051] Automated video analytics systems and methods performed by computers and other programmable devices may perform analysis on original video and other data feeds to generate alerts for use in visual essence creation and association, and some may also use such processes to identify and select or generate images used in the visual essences primitives and/or to formulate the essence groupings. For example, alerts may be generated by applying computer vision and pattern recognition (CVPR) algorithms to a video feed input, such as by algorithms taught by "Recognition of Repetitive Sequential Human Activity" by Quanfu Fan et al., IBM T. J. Watson Research Center, Hawthorne, NY, USA, presented at CVPR 2009; or also by "GRAPH BASED EVENT DETECTION FROM REALISTIC VIDEOS USING WEAK FEATURE CORRESPONDENCE", by Lei Ding et al., IBM T. J. Watson Research Center, Hawthorne, NY, USA, presented at ICASSP 2010; and still other appropriate examples for use with embodiments of the present invention will be apparent to one skilled in the art.
[0052] Visual essence images may be generated through analysis of activity detected within images from given areas, in some embodiments independent of any temporally relationship to a logged event or other data stream. For example, distinct Region of Interests (ROI) may be defined within physical spaces within images for the creation of and analysis of the video images, and motion pixels obtained by frame differencing (or differentiating) a video stream may be counted in each ROI for each frame and normalized by the area of the ROI. For example, referring again to Figure 3, the pick-up area 156, scan area 160 and drop-off area 164 may be defined for creating respective pick-up {P}, scan {S} and drop-off {D} visual essence image primitives as a function of patterns observed in motion sequences within said areas, generally in response to motion pixels associated with the movement of an item and/or a hand (or both hands) of a cashier within each region. Each area 156/160/164 may itself define an ROI, with motion pixels obtained by frame differencing a video stream counted in each ROI for each frame and normalized by the area of the respective ROI 156/160/164. Alternatively, any region 156/160/164 may encompass multiple distinct (and sometimes overlapping) ROI's to provide additional granularity or primitive creation capabilities. One example for distinguishing motion patterns observed in resulting motion sequences within ROI's appropriate for practice with embodiments of the present invention is taught by "Detecting Sweethearting in Retail Surveillance Videos" by Quanfu Fan et al., ICCASSP, 2009.
[0053] Referring now to Figure 4, an exemplary computerized implementation of an embodiment of the present invention includes computer or other programmable device 322 in communication with video or still image devices 336 (for example, a video camera or video server) that generates visual essence data for determination of human behavior according to the present invention, for example, in response to computer readable code 318 in a file residing in a memory 316 or a storage system 332 through a computer network infrastructure 326. The implementation is intended to demonstrate, among other things, that embodiments of the present invention could be implemented within a network environment (e.g., the Internet, a wide area network (WAN), a local area network (LAN) or a virtual private network (VPN), etc.) Communication throughout the network 326 can occur via any combination of various types of communication links: for example, communication links can comprise addressable connections that may utilize any combination of wired and/or wireless transmission methods. Where communications occur via the Internet, connectivity could be provided by conventional TCP/IP sockets-based protocol, and an Internet service provider could be used to establish connectivity to the Internet. Still yet, the network infrastructure 326 is intended to demonstrate that an application of an embodiment of the invention can be deployed, managed, serviced, etc. by a service provider who offers to implement, deploy, and/or perform the functions of embodiments of the present invention for others.
[0054] The computer 322 comprises various components, some of which are illustrated within the computer 322. More particularly, as shown, the computer 322 includes a processing unit (CPU) 338 in communication with the memory 316 and with one or more external I/O devices/resources 328 and storage systems 332. In general, the processing unit 338 may execute computer program code, such as the code to implement one or more of the process steps illustrated in the Figures, which may be stored in the memory 316 and/or external storage system 332 or device 328.
[0055] The network infrastructure 326 is only illustrative of various types of computer infrastructures for implementing embodiments of the invention. For example, in one embodiment, computer infrastructure 326 comprises two or more computing devices (e.g., a server cluster) that communicate over a network. Moreover, the computer 322 is only representative of various possible computer systems that can include numerous combinations of hardware. To this extent, in other embodiments, the computer 322 can comprise any specific purpose computing article of manufacture comprising hardware and/or computer program code for performing specific functions, any computing article of manufacture that comprises a combination of specific purpose and general purpose hardware/software, or the like. In each case, the program code and hardware can be created using standard programming and engineering techniques, respectively.
[0056] Moreover, the processing unit 338 may comprise a single processing unit, or be distributed across one or more processing units in one or more locations, e.g., on a client and server. Similarly, the memory 316 and/or the storage system 332 can comprise any combination of various types of data storage and/or transmission media that reside at one or more physical locations. Further, I/O interfaces 324 can comprise any system for exchanging information with one or more of the external device 328. Still further, it is understood that one or more additional components (e.g., system software, math co-processing unit, etc.), not shown, can be included in the computer 322.
[0057] One embodiment performs process steps of the invention on a subscription, advertising, and/or fee basis. That is, a service provider could offer to provide automated analysis of video data for determination of human behavior. In this case, the service provider can create, maintain, and support, etc., a computer infrastructure, such as the network computer infrastructure 326 that performs the process steps of embodiments of the invention for one or more customers. In return, the service provider can receive payment from the customer(s) under a subscription and/or fee agreement and/or the service provider can receive payment from the sale of advertising content to one or more third parties.
[0058] In still another embodiment, the invention provides a computer-implemented method for executing one or more of the processes, systems and articles for automated analysis of video data for determination of human behavior described above. In this case, a computer infrastructure, such as the computer infrastructure 326, can be provided and one or more systems for performing the process steps of the embodiment of the invention can be obtained (e.g., created, purchased, used, modified, etc.) and deployed to the computer infrastructure. To this extent, the deployment of a system can comprise one or more of: (1) installing program code on a computing device, such as the computers/devices 322/336, from a computer-readable medium; (2) adding one or more computing devices to the computer infrastructure; and (3) incorporating and/or modifying one or more existing systems of the computer infrastructure to enable the computer infrastructure to perform the process steps.
[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, it is understood that the terms "program code" and "computer program code" are synonymous and mean any expression, in any language, code or notation, of a set of instructions intended to cause a computing device having an information processing capability to perform a particular function either directly or after either or both of the following: (a) conversion to another language, code or notation; and/or (b) reproduction in a different material form. To this extent, program code can be embodied as one or more of: an application/software program, component software/a library of functions, an operating system, a basic I/O system/driver for a particular computing and/or I/O device, and the like.
[0060] Certain examples and elements described in the present specification, including in the claims and as illustrated in the Figures, may be distinguished or otherwise identified from others by unique adjectives (e.g. a "first" element distinguished from another "second" or "third" of a plurality of elements, a "primary" distinguished from a "secondary" one or "another" item, etc.) Such identifying adjectives are generally used to reduce confusion or uncertainty, and are not to be construed to limit the claims to any specific illustrated element or embodiment, or to imply any precedence, ordering or ranking of any claim elements, limitations or process steps.
[0061] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
[0062] "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, it is understood that the terms "program code" and "computer program code" are synonymous and mean any expression, in any language, code or notation, of a set of instructions intended to cause a computing device having an information processing capability to perform a particular function either directly or after either or both of the following: (a) conversion to another language, code or notation; and/or (b) reproduction in a different material form. To this extent, program code can be embodied as one or more of: an application/software program, component software/a library of functions, an operating system, a basic I/O system/driver for a particular computing and/or I/O device, and the like.
[0063] Certain examples and elements described in the present specification, including in the claims and as illustrated in the Figures, may be distinguished or otherwise identified from others by unique adjectives (e.g. a "first" element distinguished from another "second" or "third" of a plurality of elements, a "primary" distinguished from a "secondary" one or "another" item, etc.) Such identifying adjectives are generally used to reduce confusion or uncertainty, and are not to be construed to limit the claims to any specific illustrated element or embodiment, or to imply any precedence, ordering or ranking of any claim elements, limitations or process steps.
[0064] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
receiving (202) an alert triggered by a video analytics component (114) for a visual image stream;
in response to the alert, generating (204) a set of alert images (116) from the visual image stream including an image contemporaneous with the triggered alert and at least one other image occurring before or after the contemporaneous image in the visual image stream such that a reviewer can review (208) whether the alert is a true or false alert using the set of alert images; and
subsequently providing additional images from the visual image stream on request (210) by the reviewer so as to confirm a determination of whether the alert is a true or false alert such that said additional images are provided only in response to said request;
the method further comprising:
archiving the visual image stream in a repository (112);
retrieving the additional images provided from the visual image stream; and
incorporating text data associated with the contemporaneous image and the at least one other image into the set of alert images (116) such that the text data is available to the reviewer;
wherein the triggering alert indicates a suspected occurrence of an event constituting an intentional failure to scan an item, for an item scanned in an item scanning system, the triggering alert being generated when a time gap between pick-up, scanning and/or dropping of an item is longer than a predefined time interval;
wherein the visual image stream is a video stream of images of a scanning station in the scanning system;
wherein the text data is a transaction log stream temporally related to the video stream;
wherein the image contemporaneous with the triggering alert is a frame image in the video stream of a scanner area (160) of the scanning station occurring contemporaneous with a logged scan event; and
wherein the at least one other image occurring before or after the contemporaneous image comprises:
a previous-in-time frame image of a pickup area (156) of the scanning station occurring prior to the scanner area image; and
a later-in-time frame image of a drop area (164) of the scanning station occurring after the scanner area image.
selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
receiving (202) an alert triggered by a video analytics component (114) for a visual image stream;
in response to the alert, generating (204) a set of alert images (116) from the visual image stream including an image contemporaneous with the triggered alert and at least one other image occurring before or after the contemporaneous image in the visual image stream such that a reviewer can review (208) whether the alert is a true or false alert using the set of alert images; and
subsequently providing additional images from the visual image stream on request (210) by the reviewer so as to confirm a determination of whether the alert is a true or false alert such that said additional images are provided only in response to said request;
the method further comprising:
archiving the visual image stream in a repository (112);
retrieving the additional images provided from the visual image stream; and
analyzing (114) the visual image stream via computer vision and pattern recognition to determine an occurrence of the triggering alert;
wherein the triggering alert is a suspected item abandonment;
wherein the visual image stream is a video camera stream including image frames of an area including the suspected abandoned item;
wherein the analyzing the visual image stream via the computer vision and pattern recognition to determine the occurrence of the suspected item abandonment comprises using an abandoned item detection algorithm; and
wherein the image contemporaneous with the suspected item abandonment is a frame image of the video camera stream of the area including the suspected abandoned item which includes the suspected abandoned item; and
wherein the at least one other image occurring before or after the contemporaneous image comprises:
a previous-in-time frame image of the area including the suspected abandoned item occurring prior to the contemporaneous image; and
a later-in-time frame image of the area including the suspected abandoned item occurring after the contemporaneous image.
selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
receiving means for receiving (202) an alert triggered by a video analytics component (114) for a visual image stream generating means for, in response to the alert, generating (204) a set of alert images (116) from the visual image stream including an image contemporaneous with the triggered alert and at least one other image occurring before or after the contemporaneous image in the visual image stream such that a reviewer (208) can review whether the alert is a true or false alert using the set of alert images; and
additional image provision means for subsequently providing additional images from the visual image stream on request (210) by the reviewer so as to confirm a determination of whether the alert is a true or false alert such that said additional images are provided only in response to said request;
the system further comprising:
archiving means for archiving the visual image stream in a repository (112);
retrieving means for retrieving the additional images provided from the visual image stream; and
text incorporating means for incorporating text data associated with the contemporaneous image and the at least one other image into the set of alert images (116) such that the text data is available to the reviewer;
wherein the triggering alert indicates a suspected occurrence of an event constituting an intentional failure to scan an item, for an item scanned in an item scanning system, the triggering alert being generated when a time gap between pick-up, scanning and/or dropping of an item is longer than a predefined time interval;
wherein the visual image stream is a video stream of images of a scanning station in the scanning system;
wherein the text data is a transaction log stream temporally related to the video stream;
wherein the image contemporaneous with the triggering alert is a frame image in the video stream of a scanner area (160) of the scanning station occurring contemporaneous with a logged scan event; and
wherein the at least one other image occurring before or after the contemporaneous image comprises:
a previous-in-time frame image of a pickup area (156) of the scanning station occurring prior to the scanner area image; and
a later-in-time frame image of a drop area (164) of the scanning station occurring after the scanner area image.
selecting means for selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
receiving means for receiving (202) an alert triggered by a video analytics component (114) for a visual image stream generating means for, in response to the alert, generating (204) a set of alert images (116) from the visual image stream including an image contemporaneous with the triggered alert and at least one other image occurring before or after the contemporaneous image in the visual image stream such that a reviewer (208) can review whether the alert is a true or false alert using the set of alert images; and
additional image provision means for subsequently providing additional images from the visual image stream on request (210) by the reviewer so as to confirm a determination of whether the alert is a true or false alert such that said additional images are provided only in response to said request;
the system further comprising:
archiving means for archiving the visual image stream in a repository (112);
retrieving means for retrieving the additional images provided from the visual image stream; and
analyzing means for analyzing (114) the visual image stream via computer vision and pattern recognition to determine an occurrence of the triggering alert;
wherein the triggering alert is a suspected item abandonment;
wherein the visual image stream is a video camera stream including image frames of an area including the suspected abandoned item;
wherein the analyzing the visual image stream via the computer vision and pattern recognition to determine the occurrence of the suspected item abandonment comprises using an abandoned item detection algorithm; and
wherein the image contemporaneous with the suspected item abandonment is a frame image of the video camera stream of the area including the suspected abandoned item which includes the suspected abandoned item; and
wherein the at least one other image occurring before or after the contemporaneous image comprises:
a previous-in-time frame image of the area including the suspected abandoned item occurring prior to the contemporaneous image; and
a later-in-time frame image of the area including the suspected abandoned item occurring after the contemporaneous image.
text incorporating means for incorporating text data associated with the contemporaneous image and the at least one other image into the set of alert images (116) such that the text data is available to the reviewer.
selecting means for selecting each of the at least one other images as a function of a proximity in time to a time of the triggered alert.
Empfangen (202) einer durch eine Videoanalytikkomponente (114) für einen visuellen Bildstrom ausgelösten Warnmeldung;
in Reaktion auf die Warnmeldung, Erzeugen (204) einer Reihe von Warnbildern (116) aus dem visuellen Bildstrom, die ein mit der ausgelösten Warnmeldung zeitgleiches Bild und mindestens ein anderes Bild, das vor oder nach dem zeitgleichen Bild in dem visuellen Bildstrom auftritt, beinhalten, sodass ein Prüfer unter Verwendung der Reihe von Warnbildern prüfen (208) kann, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist; und
anschließend Bereitstellen von zusätzlichen Bildern aus dem visuellen Bildstrom auf Anforderung (210) durch den Prüfer, um eine Bestimmung, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist, zu bestätigen, sodass die zusätzlichen Bilder nur in Reaktion auf die Anforderung bereitgestellt werden;
das Verfahren ferner umfassend:
Archivieren des visuellen Bildstroms in einem Repository (112) ;
Abrufen der aus dem visuellen Bildstrom bereitgestellten zusätzlichen Bilder; und
Aufnehmen von dem zeitgleichen Bild und dem mindestens einen anderen Bild zugeordneten Textdaten in die Reihe von Warnbildern (116), sodass die Textdaten dem Prüfer zur Verfügung stehen;
wobei die Warnmeldungsauslösung ein vermutetes Eintreten eines Ereignisses, das ein vorsätzliches Unterlassen des Scannens eines Artikels für einen in einem Artikel-Scansystem gescannten Artikel darstellt, anzeigt, wobei die Warnmeldungsauslösung erzeugt wird, wenn eine Zeitspanne zwischen Aufnehmen, Scannen und/oder Ablegen eines Artikels länger als ein vordefiniertes Zeitintervall ist;
wobei der visuelle Bildstrom ein Videobildstrom einer Scanstation in dem Scansystem ist;
wobei die Textdaten ein Transaktionsprotokollstrom, der in zeitlicher Beziehung zu dem Videostrom steht, sind;
wobei das mit der Warnmeldungsauslösung zeitgleiche Bild ein Einzelbild in dem Videostrom eines Scannerbereichs (160) der Scanstation ist, das zeitgleich mit einem protokollierten Scanereignis auftritt; und
wobei das mindestens eine andere vor oder nach dem zeitgleichen Bild auftretende Bild Folgendes umfasst:
ein zeitlich früheres Einzelbild eines Aufnahmebereichs (156) der Scanstation, das vor dem Scannerbereichsbild auftritt; und
ein zeitlich späteres Einzelbild eines Ablagebereichs (164) der Scanstation, das nach dem Scannerbereichsbild auftritt.
Auswählen jedes einzelnen des mindestens einen anderen Bildes in Abhängigkeit einer zeitlichen Nähe zu einem Zeitpunkt der ausgelösten Warnmeldung.
Empfangen (202) einer durch eine Videoanalytikkomponente (114) für einen visuellen Bildstrom ausgelösten Warnmeldung;
in Reaktion auf die Warnmeldung, Erzeugen (204) einer Reihe von Warnbildern (116) aus dem visuellen Bildstrom, die ein mit der ausgelösten Warnmeldung zeitgleiches Bild und mindestens ein anderes Bild, das vor oder nach dem zeitgleichen Bild in dem visuellen Bildstrom auftritt, beinhalten, sodass ein Prüfer unter Verwendung der Reihe von Warnbildern prüfen (208) kann, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist; und
anschließend Bereitstellen von zusätzlichen Bildern aus dem visuellen Bildstrom auf Anforderung (210) durch den Prüfer, um eine Bestimmung, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist, zu bestätigen, sodass die zusätzlichen Bilder nur in Reaktion auf die Anforderung bereitgestellt werden;
das Verfahren ferner umfassend:
Archivieren des visuellen Bildstroms in einem Repository (112) ;
Abrufen der aus dem visuellen Bildstrom bereitgestellten zusätzlichen Bilder; und
Analysieren (114) des visuellen Bildstroms mittels maschinellen Sehens und Mustererkennung, um ein Auftreten der Warnmeldungsauslösung zu ermitteln;
wobei die Warnmeldungsauslösung ein mutmaßliches Zurücklassen eines Artikels ist;
wobei der visuelle Bildstrom ein Videokamerastrom ist, der Einzelbilder eines Bereichs beinhaltet, der den mutmaßlich zurückgelassenen Artikel beinhaltet;
wobei das Analysieren des visuellen Bildstroms mittels maschinellen Sehens und Mustererkennung zum Ermitteln des Auftretens des mutmaßlichen Zurücklassens eines Artikels die Verwendung eines Algorithmus zum Erkennen zurückgelassener Artikel umfasst; und
wobei das mit dem mutmaßlichen Zurücklassen eines Artikels zeitgleiche Bild ein Einzelbild des Videokamerastroms des den mutmaßlich zurückgelassenen Artikel beinhaltetenden Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet ist; und
wobei das mindestens eine andere vor oder nach dem zeitgleichen Bild auftretende Bild Folgendes umfasst:
ein zeitlich früheres Einzelbild des Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet, das vor dem zeitgleichen Bild auftritt; und
ein zeitlich späteres Einzelbild des Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet, das nach dem zeitgleichen Bild auftritt.
Auswählen jedes einzelnen des mindestens einen anderen Bildes in Abhängigkeit einer zeitlichen Nähe zu einem Zeitpunkt der ausgelösten Warnmeldung.
Empfangsmittel zum Empfangen (202) einer durch eine Videoanalytikkomponente (114) ausgelösten Warnmeldung für visuellen Bildstrom erzeugende Mittel zum Erzeugen (204), in Reaktion auf die Warnmeldung, einer Reihe von Warnbildern (116) aus dem visuellen Bildstrom, die ein mit der ausgelösten Warnmeldung zeitgleiches Bild und mindestens ein anderes Bild, das vor oder nach dem zeitgleichen Bild in dem visuellen Bildstrom auftritt, beinhalten, sodass ein Prüfer (208) unter Verwendung der Reihe von Warnbildern prüfen kann, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist; und
zusätzliche Bildbereitstellungsmittel zum anschließenden Bereitstellen von zusätzlichen Bildern aus dem visuellen Bildstrom auf Anforderung (210) durch den Prüfer, um eine Bestimmung, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist, zu bestätigen, sodass die zusätzlichen Bilder nur in Reaktion auf die Anforderung bereitgestellt werden;
das System ferner umfassend:
Archivierungsmittel zum Archivieren des visuellen Bildstroms in einem Repository (112); Abrufmittel zum Abrufen der aus dem visuellen Bildstrom bereitgestellten zusätzlichen Bilder; und
Textaufnahmemittel zum Aufnehmen von dem zeitgleichen Bild und dem mindestens einen anderen Bild zugeordneten Textdaten in die Reihe von Warnbildern (116), sodass die Textdaten dem Prüfer zur Verfügung stehen;
wobei die Warnmeldungsauslösung ein vermutetes Eintreten eines Ereignisses, das ein vorsätzliches Unterlassen des Scannens eines Artikels für einen in einem Artikel-Scansystem gescannten Artikel darstellt, anzeigt, wobei die Warnmeldungsauslösung erzeugt wird, wenn eine Zeitspanne zwischen Aufnehmen, Scannen und/oder Ablegen eines Artikels länger als ein vordefiniertes Zeitintervall ist;
wobei der visuelle Bildstrom ein Videobildstrom einer Scanstation in dem Scansystem ist;
wobei die Textdaten ein Transaktionsprotokollstrom, der in zeitlicher Beziehung zu dem Videostrom steht, sind;
wobei das mit der Warnmeldungsauslösung zeitgleiche Bild ein Einzelbild in dem Videostrom eines Scannerbereichs (160) der Scanstation ist, das zeitgleich mit einem protokollierten Scanereignis auftritt; und
wobei das mindestens eine andere vor oder nach dem zeitgleichen Bild auftretende Bild Folgendes umfasst:
ein zeitlich früheres Einzelbild eines Aufnahmebereichs (156) der Scanstation, das vor dem Scannerbereichsbild auftritt; und
ein zeitlich späteres Einzelbild eines Ablagebereichs (164) der Scanstation, das nach dem Scannerbereichsbild auftritt.
Auswahlmittel zum Auswählen jedes einzelnen des mindestens einen anderen Bildes in Abhängigkeit einer zeitlichen Nähe zu einem Zeitpunkt der ausgelösten Warnmeldung.
Empfangsmittel zum Empfangen (202) einer durch eine Videoanalytikkomponente (114) ausgelösten Warnmeldung für visuellen Bildstrom erzeugende Mittel zum Erzeugen (204), in Reaktion auf die Warnmeldung, einer Reihe von Warnbildern (116) aus dem visuellen Bildstrom, die ein mit der ausgelösten Warnmeldung zeitgleiches Bild und mindestens ein anderes Bild, das vor oder nach dem zeitgleichen Bild in dem visuellen Bildstrom auftritt, beinhalten, sodass ein Prüfer (208) unter Verwendung der Reihe von Warnbildern prüfen kann, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist; und
zusätzliche Bildbereitstellungsmittel zum anschließenden Bereitstellen von zusätzlichen Bildern aus dem visuellen Bildstrom auf Anforderung (210) durch den Prüfer, um eine Bestimmung, ob die Warnmeldung eine zutreffende oder falsche Warnmeldung ist, zu bestätigen, sodass die zusätzlichen Bilder nur in Reaktion auf die Anforderung bereitgestellt werden;
das System ferner umfassend:
Archivierungsmittel zum Archivieren des visuellen Bildstroms in einem Repository (112); Abrufmittel zum Abrufen der aus dem visuellen Bildstrom bereitgestellten zusätzlichen Bilder; und
Analysemittel zum Analysieren (114) des visuellen Bildstroms mittels maschinellen Sehens und Mustererkennung, um ein Auftreten der Warnmeldungsauslösung zu ermitteln;
wobei die Warnmeldungsauslösung ein mutmaßliches Zurücklassen eines Artikels ist;
wobei der visuelle Bildstrom ein Videokamerastrom ist, der Einzelbilder eines Bereichs beinhaltet, der den mutmaßlich zurückgelassenen Artikel beinhaltet;
wobei das Analysieren des visuellen Bildstroms mittels maschinellen Sehens und Mustererkennung zum Ermitteln des Auftretens des mutmaßlichen Zurücklassens eines Artikels die Verwendung eines Algorithmus zum Erkennen zurückgelassener Artikel umfasst; und
wobei das mit dem mutmaßlichen Zurücklassen eines Artikels zeitgleiche Bild ein Einzelbild des Videokamerastroms des den mutmaßlich zurückgelassenen Artikel beinhaltetenden Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet ist; und
wobei das mindestens eine andere vor oder nach dem zeitgleichen Bild auftretende Bild Folgendes umfasst:
ein zeitlich früheres Einzelbild des Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet, das vor dem zeitgleichen Bild auftritt; und
ein zeitlich späteres Einzelbild des Bereichs, der den mutmaßlich zurückgelassenen Artikel beinhaltet, das nach dem zeitgleichen Bild auftritt.
Textaufnahmemittel zum Aufnehmen von dem zeitgleichen Bild und dem mindestens einen anderen Bild zugeordneten Textdaten in die Reihe von Warnbildern (116), sodass die Textdaten dem Prüfer zur Verfügung stehen.
Auswahlmittel zum Auswählen jedes einzelnen des mindestens einen anderen Bildes in Abhängigkeit einer zeitlichen Nähe zu einem Zeitpunkt der ausgelösten Warnmeldung.
la réception (202) d'une alerte déclenchée par un composant d'analyse vidéo (114) pour un flux d'images visuelles ;
en réponse à l'alerte, la génération (204) d'un ensemble d'images d'alerte (116) à partir du flux d'images visuelles comprenant une image contemporaine avec l'alerte déclenchée et au moins une autre image se produisant avant ou après l'image contemporaine dans le flux d'images visuelles de sorte qu'un réviseur puisse vérifier (208) si l'alerte est une alerte vraie ou fausse à l'aide des images d'alerte définies ; et
la fourniture ensuite d'images supplémentaires à partir du flux d'images visuelles sur demande (210) par le réviseur afin de confirmer une détermination de si l'alerte est une alerte vraie ou fausse de sorte que lesdites images supplémentaires soient fournies uniquement en réponse à ladite demande ;
le procédé comprenant en outre :
l'archivage du flux d'images visuelles dans un référentiel (112) ;
la récupération des images supplémentaires fournies à partir du flux d'images visuelles ; et
l'incorporation de données de texte associées à l'image contemporaine et l'au moins une autre image dans l'ensemble d'images d'alerte (116) de sorte que les données de texte soient disponibles pour le réviseur ;
dans lequel l'alerte de déclenchement indique une occurrence suspectée d'un événement constituant une défaillance intentionnelle de numériser un élément, pour un élément numérisé dans un système de numérisation d'élément, l'alerte de déclenchement étant générée lorsqu'un intervalle de temps entre la collecte, la numérisation et/ ou la suppression d'un élément est plus long qu'un intervalle de temps prédéfini ;
dans lequel le flux d'images visuelles est un flux vidéo d'images d'une station de numérisation dans le système de numérisation ;
dans lequel les données de texte sont un flux de journal de transaction temporellement lié au flux vidéo ;
dans lequel l'image contemporaine de l'alerte de déclenchement est une image de trame dans le flux vidéo d'une zone de numérisation (160) de la station de numérisation se produisant simultanément avec un événement de numérisation enregistré ; et
dans lequel l'au moins une autre image se produisant avant ou après l'image contemporaine comprend :
une image de trame précédente dans le temps d'une zone de prise en charge (156) de la station de numérisation se produisant avant l'image de la zone de numérisation ; et
une image de trame ultérieure dans le temps d'une zone de dépôt (164) de la station de numérisation se produisant après l'image de la zone de numérisation.
la sélection de chacune de l'au moins une autre image comme une fonction d'une proximité temporelle avec une heure de l'alerte déclenchée.
la réception (202) d'une alerte déclenchée par un composant d'analyse vidéo (114) pour un flux d'images visuelles ;
en réponse à l'alerte, la génération (204) d'un ensemble d'images d'alerte (116) à partir du flux d'images visuelles comprenant une image contemporaine de l'alerte déclenchée et au moins une autre image se produisant avant ou après l'image contemporaine dans le flux d'images visuelles telle qu'un réviseur peut vérifier (208) si l'alerte est une alerte vraie ou fausse à l'aide de l'ensemble d'images d'alerte; et
la fourniture ensuite d'images supplémentaires à partir du flux d'images visuelles sur demande (210) par le réviseur afin de confirmer une détermination de si l'alerte est une alerte vraie ou fausse de sorte que lesdites images supplémentaires soient fournies uniquement en réponse à ladite demande ;
le procédé comprenant en outre :
l'archivage du flux d'images visuelles dans un référentiel (112) ;
la récupération des images supplémentaires fournies à partir du flux d'images visuelles ; et
l'analyse (114) du flux d'images visuelles via une vision informatique et la reconnaissance de modèles afin de déterminer une occurrence de l'alerte de déclenchement ;
dans lequel l'alerte de déclenchement est un abandon d'élément suspecté ;
dans lequel le flux d'images visuelles est un flux de caméra vidéo comprenant des trames d'images d'une zone comprenant l'élément abandonné suspecté ;
dans lequel l'analyse du flux d'images visuelles via la vision informatique et la reconnaissance des modèles pour déterminer l'occurrence de l'abandon d'élément suspecté comprend l'utilisation d'un algorithme de détection d'élément abandonné ; et
dans lequel l'image contemporaine de l'abandon d'élément suspecté est une image de trame du flux de caméra vidéo de la zone comprenant l'élément abandonné suspecté qui comprend l'élément abandonné suspecté ; et
dans lequel l'au moins une autre image se produisant avant ou après l'image contemporaine comprend :
une image de trame précédente dans le temps de la zone comprenant l'élément abandonné suspecté se produisant avant l'image contemporaine ; et
une image de trame ultérieure dans le temps de la zone comprenant l'élément abandonné suspecté se produisant après l'image contemporaine.
la sélection de chacune de l'au moins une autre image comme une fonction d'une proximité temporelle avec une heure de l'alerte déclenchée.
un moyen de réception pour la réception (202) d'une alerte déclenchée par un composant d'analyse vidéo (114) pour un moyen de génération de flux d'images visuelles pour, en réponse à l'alerte, la génération (204) d'un ensemble d'images d'alerte (116) à partir du flux d'images visuelles comprenant une image contemporaine de l'alerte déclenchée et au moins une autre image se produisant avant ou après l'image contemporaine dans le flux d'images visuelles de sorte qu'un réviseur (208) puisse vérifier si l'alerte est une alerte vraie ou fausse à l'aide de l'ensemble d'images d'alerte ; et
des moyens de fourniture d'images supplémentaires pour la fourniture ultérieure d'images supplémentaires à partir du flux d'images visuelles sur demande (210) par le réviseur afin de confirmer une détermination de si l'alerte est une alerte vraie ou fausse de telle sorte que lesdites images supplémentaires ne sont fournies qu'en réponse à ladite demande ;
le système comprenant en outre :
des moyens d'archivage pour l'archivage du flux d'images visuelles dans un référentiel (112) ; des moyens de récupération pour la récupération des images supplémentaires fournies à partir du flux d'images visuelles ; et
des moyens d'incorporation de texte pour l'incorporation de données de texte associées à l'image contemporaine et à l'au moins une autre image dans l'ensemble d'images d'alerte (116) de telle sorte que les données de texte soient disponibles pour le réviseur;
dans lequel l'alerte de déclenchement indique une occurrence suspectée d'un événement constituant une défaillance intentionnelle de numériser un élément, pour un élément numérisé dans un système de numérisation d'élément, l'alerte de déclenchement étant générée lorsqu'un intervalle de temps entre la collecte, la numérisation et/ ou la suppression d'un élément est plus long qu'un intervalle de temps prédéfini ;
dans lequel le flux d'images visuelles est un flux vidéo d'images d'une station de numérisation dans le système de numérisation ;
dans lequel les données de texte sont un flux de journal de transaction temporellement lié au flux vidéo ;
dans lequel l'image contemporaine de l'alerte de déclenchement est une image de trame dans le flux vidéo d'une zone de numérisation (160) de la station de numérisation se produisant simultanément avec un événement de numérisation enregistré ; et
dans lequel l'au moins une autre image se produisant avant ou après l'image contemporaine comprend :
une image de trame précédente dans le temps d'une zone de prise en charge (156) de la station de numérisation se produisant avant l'image de la zone de numérisation ; et
une image de trame ultérieure dans le temps d'une zone de dépôt (164) de la station de numérisation se produisant après l'image de la zone de numérisation.
des moyens de sélection pour la sélection de chacune de l'au moins une autre image comme une fonction d'une proximité temporelle avec une heure de l'alerte déclenchée.
un moyen de réception pour la réception (202) d'une alerte déclenchée par un composant d'analyse vidéo (114) pour un moyen de génération de flux d'images visuelles pour, en réponse à l'alerte, la génération (204) d'un ensemble d'images d'alerte (116) à partir du flux d'images visuelles comprenant une image contemporaine de l'alerte déclenchée et au moins une autre image se produisant avant ou après l'image contemporaine dans le flux d'images visuelles de sorte qu'un réviseur (208) puisse vérifier si l'alerte est une alerte vraie ou fausse à l'aide de l'ensemble d'images d'alerte ; et
des moyens de fourniture d'images supplémentaires pour la fourniture ultérieure d'images supplémentaires à partir du flux d'images visuelles sur demande (210) par le réviseur afin de confirmer une détermination de si l'alerte est une alerte vraie ou fausse de telle sorte que lesdites images supplémentaires ne sont fournies qu'en réponse à ladite demande ;
le système comprenant en outre :
des moyens d'archivage pour l'archivage du flux d'images visuelles dans un référentiel (112) ; des moyens de récupération pour la récupération des images supplémentaires fournies à partir du flux d'images visuelles ; et
des moyens d'analyse pour l'analyse (114) du flux d'images visuelles via une vision informatique et une reconnaissance de modèles afin de déterminer une occurrence de l'alerte de déclenchement ;
dans lequel l'alerte de déclenchement est un abandon d'élément suspecté ;
dans lequel le flux d'images visuelles est un flux de caméra vidéo comprenant des trames d'images d'une zone comprenant l'élément abandonné suspecté ;
dans lequel l'analyse du flux d'images visuelles via la vision informatique et la reconnaissance des modèles pour déterminer l'occurrence de l'abandon d'élément suspecté comprend l'utilisation d'un algorithme de détection d'élément abandonné ; et
dans lequel l'image contemporaine de l'abandon d'élément suspecté est une image de trame du flux de caméra vidéo de la zone comprenant l'élément abandonné suspecté qui comprend l'élément abandonné suspecté ; et
dans lequel l'au moins une autre image se produisant avant ou après l'image contemporaine comprend :
une image de trame précédente dans le temps de la zone comprenant l'élément abandonné suspecté se produisant avant l'image contemporaine ; et
une image de trame ultérieure dans le temps de la zone comprenant l'élément abandonné suspecté se produisant après l'image contemporaine.
des moyens d'incorporation de texte pour l'incorporation de données de texte associées à l'image contemporaine et à l'au moins une autre image dans l'ensemble d'images d'alerte (116) de telle sorte que les données de texte soient disponibles pour le réviseur.
des moyens de sélection pour la sélection de chacune de l'au moins une autre image comme une fonction d'une proximité temporelle avec une heure de l'alerte déclenchée.
REFERENCES CITED IN THE DESCRIPTION
This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.
Patent documents cited in the description
Non-patent literature cited in the description
- Recognition of Repetitive Sequential Human ActivityQUANFU FAN et al.CVPRIBM T. J. Watson Research Center20090000 [0004] [0051]
- GRAPH BASED EVENT DETECTION FROM REALISTIC VIDEOS USING WEAK FEATURE CORRESPONDENCELEI DING et al.ICASSPIBM T. J. Watson Research Center20100000 [0004] [0051]
- GUILLAUME MILCENTYANG CAILocation Based Baggage Detection for Transit VehiclesCarnegie Mellon University20051209 [0004] [0049]
- QUANFU FAN et al.Detecting Sweethearting in Retail Surveillance VideosICCASSP, 2009, [0052]