PIPELINE PROCESSING OF STILL IMAGES ADAPTED FOR REAL TIME EXECUTION OF DIGITAL VIDEO EFFECTS

Description

Background of the Invention

[0001] Technology for manipulating digital video has progressed to the point where it can be readily processed and handled on computers. For example, the Avid/l Media Composer, available from Avid Technology, Inc. of Tewksbury, Massachusetts, is a system wherein digital video can be readily captured, edited, and displayed for various purposes, such as broadcast television and film and video program post-production.

[0002] The Avid/l Media Composer uses a media pipeline to provide real-time digital video output on a computer display. This media pipeline 30 is shown in Fig. 1 in is described in more detail in U.S. Patent No. 5,045,940, issued September 3, 1991. In this media pipeline 30, a permanent storage 40 stores sequences of digital still images which represent digital video and are played back at a rate which provides the appearance of video. The sequences of digital still images do not include any frame synchronization or other type of timing information which are typically found in television signals: The still images also typically are stored in compressed form. The stored sequences are accessed and placed in a data buffer 42 from where they are provided to a compression/decompression system 44. The output of the compression/decompression system 44 is applied to a frame buffer 46 which converts the still image to a typical video signal which is then applied to an input/output unit 48.
Each of the systems 40, 42, 44, 46, and 48 in this media pipeline 30 operate bi-directionally. That is, the output process discussed above can be reversed and video signals can be input via input/output unit 48 to the frame buffer 46 where they are converted to a sequence of digital still images. The images in the sequence are compressed by compression/decompression system 44, stored in data buffer 42 and then transferred to the permanent storage 40.

[0003] In such a system, only simple effects such as cuts and fades, involving only one stream of video information, can be made. More complex, arbitrary digital video effects, however, especially three-dimensional effects, are typically provided by creating the effects using another system (described below with Fig. 2), and then storing the created effect in the permanent storage 40. A variety of systems are available for performing such three-dimensional digital video effects. Such a system is commonly called an on-line controller (OLC).

[0004] A block diagram of a typical OLC 55 is shown in Fig. 2. An OLC 55 typically uses digital video signals, which include frame synchronization and other timing information.
Therefore, they typically require frame buffers such as shown at 50,54 and 56 to store the video signals being processed. The OLC 55 also includes a digital video effects DVE unit 52 which performs the desired effects. The DVE unit 52 has first and second inputs 51 and 53 which are connected respectively to outputs 57 and 59 of frame buffers 56 and 50. The DVE unit 52 has an output 61 which is connected to an input 63 of frame buffer 54. Input video sources 65 and 67, such as video tape decks, respectively provide video signal outputs 69 and 71 which are applied to inputs 73 and 75 respectively of frame buffers 56 and 50. The output 77 of the frame buffer 54 is applied to an output video storage 79, such as a video tape deck. One problem with such a typical OLC 55 is that it is not capable of being stopped during the process of creating the effect, because it operates using full-rate video signals.

[0005] Because of the complex hardware required to deal with video signals, in particular the frame buffers 50, 56 and 54, an OLC 55 is also typically expensive. Thus, producing complicated arbitrary three-dimensional video effects has not been possible in common consumer-oriented multi-media editing products.

[0006] Also, the process of preparing and editing such effects is time-consuming and, therefore, expensive. With standard OLCs, an effect is created by preparing two video tapes as input video sources 65 and 67 to provide the input video signals. These tapes must be synchronized together. Then, an output tape is prepared as the output video storage 79 to receive the output effect. The effect is then created using the DVE unit 52 by playing the input video tapes through the DVE unit 52 and recording the result on the output tape. In order for the editor to see the resulting effect, the output tape must be played back. If the result is not what is desired, the process must be repeated after appropriate adjustments are made. The repetitive recording/playing/adjusting cycle is costly.

[0007] In "Mixing and Playback of JPEG Compressed Packet Videos," by Chen et al., in Proceedings of the IEEE Global Telecommunications Conference, 1992, a system is disclosed which transmits packets of JPEG compressed video information for different streams of video which appear in different windows of a screen. Combining the video streams into one video stream with windows is performed using the compressed data. Decompression is performed after mixing. Although Chen states that mixing can be performed in the decompressed domain, Chen indicates that enough pixel buffers to hold at least one frame of each video window is required and that operations on pixel data should be at least as fast as video rate.

[0008] The invention is set out in the claims.

[0009] The embodiment improves over the prior art by providing a media pipeline with two channels of output of sequences of digital still images which sequences are input to a DVE unit which is adapted to prevent output upon receipt of a control signal. It also provides a control signal to prevent input when it cannot receive input or when it is not providing output. The DVE unit is an improvement over the prior art by the elimination of frame buffers and the addition of memory which stores sequences of digital still images, such as a FIFO.

Brief Description of the Drawing

[0010] In the drawing,

Fig. 1 is a block diagram of a media pipeline as is used in the prior art:

Fig.2 is a block diagram of a conventional on-line controller for producing special effects from video signals:

Fig. 3 is a block diagram of a modified media pipeline in accordance with the present invention;

Fig. 4 is a more detailed block diagram of a modified compression/decompression subsystem of the media pipeline in accordance with the present invention; and

Fig. 5 is a more detailed block diagram of a modified digital video effects system in accordance with the present invention.

Detailed Description

[0011] The present invention will be more completely understood through the following detailed description which should be read in conjunction with the attached drawing in which similar reference numbers indicate similar structures. All references cited herein are hereby expressly incorporated by reference.

[0012] Real-time digital video effects are provided using a media pipeline with two channels of output of sequences of digital still images, which represent video information, along with control information. Such a system can be provided by modifying the media pipeline disclosed in U.S. Patent No. 5,045,950. The two channels of digital images are used as the inputs to a digital video effects (DVE) unit such as used in typical OLCs. Thus, the modified media pipeline reduces the costs of systems for providing these effects.

[0013] The media pipeline 35 with real-time digital video effects will now be described in connection with Fig. 3. The media pipeline 30 shown in Fig. 1 is modified to include a compression/decompression (CODEC) unit 58 which is a modification of compression/decompression system 44. The modification made is such that it provides two outputs, indicated at A and B in Fig. 3. It is not necessary to use compressed data; however, compression may be preferable to reduce storage requirements. This modification is described in more detail in British provisional specification 9307894.7, filed April 16, 1993, under U.S. foreign filing license 504287 granted April 13, 1993. These changes will also be described in more detail below in connection with Fig. 4.

[0014] The OLC 55 of the prior art is also modified to create the DVE system 59 which also receives a control signal S1 from the frame buffer, and provides a control signal S2 to the CODEC unit 58. These modifications will be described in connection with Fig. 5.

[0015] In this modification, frame buffers 50, 54 and 56 of OLC 55 (Fig. 2) are replaced with memory elements 80, 82 and 84, which are preferably first-in, first-out (FIFO) elements. These memory elements 80, 82, 84 (Fig. 5) do not need to be a large as the replaced frame buffers. For example, frame buffers typically hold several hundred thousand samples (wherein a sample represents a pixel) to represent each pixel in a frame. An image of 640 by 480 pixels thus requires a frame buffer which stores about 300K pixels. In contrast, the memory elements 80, 82 and 84 need only hold several thousand samples, e.g., about 2000.

[0016] Each memory element 80, 82, 84 has a respective status line 90, 92, and 94 connected to a control unit 88. The control unit 88 is provided to ensure that digital images are accessed from a memory unit (described below) in the CODEC unit 58 and stored in the memory elements 80 and 82.

[0017] The control signal S1 indicates that the pipeline downstream from DVE unit 59 is almost full and that no further data should be output. This mechanism is described in more detail in U.S. Patent No. 5,045,940. This signal S1 is used by the DVE unit 52 to temporarily halt processing, or to continue processing but temporarily disable output. The DVE unit 52 is controlled by a system clock to perform sequential operations on the input sequences as is known in the art. Processing is stopped in response to signal S1 by gating the system clock signal with the signal S1 using a logic gate 86. The output signal S2 to the CODEC 58 is generated by control unit 88 either when the DVE 52 is processing at capacity and cannot accept further data, or whether it has stopped processing due to receipt of a "WAIT" signal from the frame buffer 46. For example, the signal S2 is sent when the input FIFO memory elements 80 and 82 are full or almost full. The detection of such a status may be performed using the status lives 90, 92 or 94 or by monitoring read and write accesses to the memory elements with a counter. Such modifications to a digital video effects system 59 are straightforward.

[0018] The modified CODEC unit 58 described in British Provisional Application 9307894.7 will now be described in connection with Fig. 4. In this figure, a control unit 60 controls two channels of coder/decoders. The modification to the media pipeline 30 is made by assigning, in the control unit 60, different sections of the compressed data buffer 42 to each channel. A sequence is also assigned to a channel. Thus, when the sequence is read into the compressed data buffer 42, it is input to the section assigned to the channel for that sequence. Thus, reading and writing of data into the FIFO 62 and 64 for the CODECs 66 and 68 is based on the assignment of a channel to a selected sequence of still images.

[0019] Each channel has a separate CODEC, either a first CODEC 66 or a second CODEC 68. Each CODEC has a respective first-in, first-out memory (FIFO) 62 and 64. The FIFO's 62 and 64 feed respectively to the CODECs 66 and 68 of which the outputs are applied to field buffers, which are also preferably FIFOs 70 and 72. These two channels may be blended using a blender 74 which is controlled in accordance with some addressing and alpha information 76. For the purposes of this invention, however, no blending is performed. Thus, a first output sequence is provided by output A from the FIFO 70 for CODEC 66. A second output sequence is then provided on the output of the blender, as the output B. Thus, FIFO 70 and blender 74 act as first and second sources of sequences of digital still images. The outputs A and B are applied to the digital video effects system 59 as shown in Fig. 3. The CODEC 58 receives control signal S2 from the digital video effects system 59 by its control unit 60. The control unit 60 appropriately fills the pipelines for CODECs 66 and 68 and stops or starts output as may be necessary. For example, it should be understood that a read signal to FIFOs 70 and 72 causes a write signal to be applied to memory elements 80 and 82.

[0020] With the modifications described above, this system 35 can therefore be readily connected to any digital video effects system 59 based on an OLC 55 which has been appropriately modified to handle pipeline control signals and sequence of digital still images. Accordingly, this invention reduces the costs in providing real-time digital video effects as well as the cost of the digital video effects systems.

[0021] Because this system operates using a media pipeline which provides sequences of digital still images, rather than a video signal, display and generation of effects can be stopped at any time. Thus, an editor can stop and view an arbitrary pixel in the created effect. Also, an editor can step forward or backward, frame-by-frame, through the effect, by virtue of the pipelined nature of the system. There is no longer any need to create and view video tapes of desired arbitrary three-dimensional video effects.

[0022] Having now described a few embodiments of the invention, it should be apparent to those skilled in the art that the foregoing is merely illustrative and not limiting, having been presented by way of example only. Numerous modifications and other embodiments are within the scope of one of ordinary skill in the art and are contemplated as falling within the scope of the invention as defined by the appended claims and equivalents thereto.

Claims

1. A media editing computer system for generating real time video effects from first and second sequences of digital still images stored on the computer, comprising:

a first memory for receiving the first sequence of digital still images;

a second memory for receiving the second sequence of digital still images;

means for receiving data forthe first and second memories and for combining the first and second sequences of digital still images according to an effect to provide as an output in real time a third sequence of digital still images; wherein the system operates using a media pipeline;

the media pipeline further comprising a control input for stopping real time output at any time and enabling a user to step forward or backward frame by frame.

2. The computer system of Claim 1, wherein the first and second memories each have a capacity of less than one of the digital still images.

3. The computer system of any of Claims 1 or 2, wherein the video effect is a composite of the first and second sequences of digital still images.

4. The computer system of any preceding claim including means for stepping forward or backward frame-by-frame.

5. A computer system according to any preceding claim including means for handling pipeline control signals.

6. The computer system of any preceding claim further comprising a control unit for providing the control input for stopping or starting forward or backward output of the media pipeline.

7. The computer system of any preceding claim including means for receiving a control signal indicative of whether a display system receiving the third sequence of digital still images has stopped displaying the third sequence.

8. The computer system of any of Claims 1 through 6, wherein the rate of playback of the third sequence is the full rate of the third sequence.

9. The computer system of any of Claims 1 through 7, wherein the rate of playback is such that display of the third sequence is advanced one image at a time according to user requests.

10. The computer system of any preceding claim including means for receiving a control signal indicative of whether a pipelined display system receiving the third sequence of digital still images has a full pipeline.

11. A media editing method of generating real time video effects from first and second sequences of digital still images stored on a computer, comprising;
   receiving a first sequence of digital still images from a first memory;
   receiving a second sequence of digital still images from a second memory;
   combining the first and second sequences of digital still images using a media pipeline, according to an effect to provide as an output in real time a third sequence of digital still images;
   wherein the method further comprises receiving a control input for the media pipeline for stopping real time output at any time and stepping forward or backward frame by frame.

Ansprüche

1. Ein Medieneditierungs-Computersystem zum Erzeugen von Echtzeit-Videoeffekten aus ersten und zweiten Sequenzen digitaler Standbilder, die auf dem Computer gespeichert sind, Folgendes umfassend:

einen ersten Speicher zum Empfangen der ersten Sequenz von digitalen Standbildern;

einen zweiten Speicher zum Empfangen der zweiten Sequenz von digitalen Standbildern;
   ein Mittel zum Empfangen von Daten für den ersten und zweiten Speicher und zum Kombinieren der ersten und zweiten Sequenz von digitalen Standbildern, und zwar gemäß einer Wirkung bzw. mit dem Ziel, um als eine Ausgabe in Echtzeit eine dritte Sequenz von digitalen Standbildern bereitzustellen; wobei das System unter Verwendung einer Medienpipeline arbeitet;

   wobei die Medienpipeline weiter eine Steuereingabe umfasst, um eine Echtzeitausgabe zu jeder Zeit zu stoppen und um einen Benutzer in die Lage zu versetzen, Vollbild für Vollbild bzw. Frame für Frame schrittweise vorwärts oder rückwärts zu schreiten.

2. Computersystem nach Anspruch 1, bei welchem der erste Speicher und der zweite Speicher jeweils eine Kapazität von weniger als eines der digitalen Standbildern hat.

3. Computersystem nach Anspruch 1 oder 2, bei welchem der Videoeffekt eine Zusammensetzung der ersten und zweiten Sequenz der digitalen Standbilder ist.

4. Computersystem nach einem vorhergehenden Anspruch, das ein Mittel enthält, um Vollbild für Vollbild bzw. Frame für Frame vorwärts oder rückwärts zu schreiten.

5. Computersystem nach einem vorhergehenden Anspruch, das ein Mittel zum Handhaben von Pipeline-Steuersignalen enthält.

6. Computersystem nach einem vorhergehenden Anspruch, das weiter eine Steuereinheit umfasst, um die Steuereingabe zum Stoppen oder Starten einer vorwärts gerichteten oder rückwärts gerichteten Ausgabe von der Medienpipeline bereitzustellen.

7. Computersystem nach einem vorhergehenden Anspruch, das ein Mittel zum Empfangen eines Steuersignals enthält, das darauf hinweist, ob ein Anzeigensystem, das die dritte Sequenz von digitalen Standbildern empfängt, gestoppt hat, die dritte Sequenz anzuzeigen.

8. Computersystem nach einem der Ansprüche 1 bis 6, bei welchen die Geschwindigkeit bzw. Rate der Wiedergabe der dritten Sequenz die volle Geschwindigkeit bzw. Rate der dritten Sequenz ist.

9. Computersystem nach einem der Ansprüche 1 bis 7, bei welchen die Geschwindigkeit bzw. Rate der Wiedergabe derartig ist, dass die Anzeige der dritten Sequenz jeweils ein Bild gemäß Benutzeranforderungen voraus ist.

10. Computersystem nach einem vorhergehenden Anspruch, das ein Mittel enthält, um ein Steuersignal zu empfangen, das darauf hinweist, ob ein mit einer Pipeline versehenes Anzeigesystem, das die dritte Sequenz digitaler Standbilder empfängt, eine volle Pipeline hat.

11. Medieneditierverfahren bzw. Medienbearbeitungsverfahren zum Erzeugen von Echtzeit-Videoeffekten von ersten und zweiten Sequenzen von digitalen Standbildern, die auf einem Computer gespeichert sind, Folgendes umfassend:

eine erste Sequenz von digitalen Standbildern wird von einem ersten Speicher empfangen;

eine zweite Sequenz von digitalen Standbildern wird von einem zweiten Speicher empfangen;

die erste und zweite Sequenz von digitalen Standbildern werden unter Verwendung einer Medienpipeline kombiniert, und zwar gemäß einer Wirkung bzw. mit dem Ziel, um als eine Ausgabe in Echtzeit eine dritte Sequenz von digitalen Standbildern bereitzustellen;

   wobei das Verfahren weiter ein Empfangen einer Steuereingabe für die Medienpipeline umfasst, um eine Echtzeitausgabe zu jeder Zeit zu stoppen und vorwärts und rückwärts Vollbild für Vollbild bzw. Frame für Frame zu schreiten.

Revendications

1. Système d'ordinateur d'édition de contenu, permettant une génération d'effets vidéo en temps réel à partir d'une première et d'une deuxième séquences d'images numériques fixes, mémorisées au sein de l'ordinateur, comprenant :

- une première mémoire pour recevoir la première séquence d'images numériques fixes;

- une deuxième mémoire pour recevoir la deuxième séquence d'images numériques fixes;

- des moyens de réception de données dans la première et la deuxième mémoires et de combinaison de la première et de la deuxième séquences d'images numériques fixes selon un effet pour fournir en sortie une troisième séquence d'images numériques fixes en temps réel, dans lequel le système utilise un pipeline de contenu (media pipeline) ;

le pipeline de contenu comprenant en outre une entrée de commande pour arrêter à tout moment la sortie en temps réel et permettant à un utilisateur d'avancer ou de reculer trame par trame.

2. Système d'ordinateur selon la revendication 1, dans lequel les première et deuxième mémoires ont chacune une capacité inférieure à une image numérique fixe.

3. Système d'ordinateur selon la revendication 1 ou 2, dans lequel l'effet vidéo est un composé de la première et de la deuxième séquences d'images numériques fixes.

4. Système d'ordinateur de l'une quelconque des revendications précédentes, comprenant des moyens pour faire avancer ou reculer les images trame par trame.

5. Système d'ordinateur selon l'une quelconque des revendications précédentes, comprenant des moyens pour manipuler des signaux de commande du pipeline de contenu.

6. Système d'ordinateur selon l'une quelconque des revendications précédentes, comprenant une unité de commande pour indiquer à l'entrée de commande d'arrêter ou d'activer la sortie du pipeline de contenu en marche avant ou en marche arrière.

7. Système d'ordinateur selon l'une quelconque des revendications précédentes, comprenant des moyens pour recevoir un signal de commande indiquant si un système d'affichage recevant la troisième séquence d'images numériques fixes a arrêté d'afficher la troisième séquence.

8. Système d'ordinateur de l'une quelconque des revendications 1 à 6, dans lequel le débit de lecture de la troisième séquence correspond au plein débit de cette troisième séquence.

9. Système d'ordinateur de l'une quelconque des revendications 1 à 7, dans lequel le débit de lecture est tel que la troisième séquence est avancée image par image, conformément aux demandes de l'utilisateur.

10. Système d'ordinateur selon l'une quelconque des revendications précédentes, comprenant des moyens pour recevoir un signal de commande indiquant si un système d'affichage en pipeline recevant la troisième séquence d'images numériques fixes dispose d'un pipeline entier.

11. Procédé, pour une édition de contenu, de génération d'effets vidéo en temps réel à partir d'une première et d'une deuxième séquences d'images numériques fixes, mémorisées dans un ordinateur, comprenant :

- une réception d'une première séquence d'images numériques fixes à partir d'une première mémoire;

- une réception d'une deuxième séquence d'images numériques fixes à partir d'une deuxième mémoire;

- une combinaison des première et deuxième séquences d'images numériques fixes selon un effet, à l'aide d'un pipeline de contenu (media pipeline), pour fournir en sortie une troisième séquence d'images numériques fixes en temps réel ;

dans lequel le procédé comprend en outre une réception d'une entrée de commande pour le pipeline de contenu afin d'arrêter à tout moment la sortie en temps réel et d'avancer ou de reculer trame par trame.

Drawing