APPARATUS AND METHOD FOR EDITING AUDIO INFORMATION SYNCHRONIZED WITH A VISUAL REPRESENTATION

Abstract

 A visual representation 404 of audio information 402 can be presented to a user for changes via a first editor 408. The first editor 408 receives the modification to the visual representation 404 of the audio information and supplies it to a second editor 412. The second editor 412 transforms the modification to the visual representation 404 into a modification to audio information synchronized 406 with the visual representation 404 and applies the transformed modification to the audio information 402 to output modified audio information 414.

Description

FIELD OF THE INVENTION

[0001] The present principles relate to an apparatus and a method for editing audio information, and more particularly, to an apparatus and a method for editing audio information using a visual representation synchronized with the audio information.

BACKGROUND

[0002] A musical score is a form of musical notation that describes how music should be played. The more complex the musical piece, the more elaborate the notation required. A musician typically learns to read this type of musical notation in order to play music that they may not be familiar with. The reverse can also be true in that a musician may want to create a musical score in order that others can be exposed to their compositions or style of playing a piece of music. Thus, musicians often play and record versions of a musical score for various reasons. If problems or errors in the musicians performance are found, this will often be edited by professionals who directly edit an audio file that was recorded as the musician played. However, amateur musicians may not have the technical training to edit the audio files nor the means to hire a professional to perform this type of task.

SUMMARY

[0003] Musical score edits are automatically transcribed into instructions that are utilized to change a corresponding audio file that has been previously aligned to the original musical score. An alignment item can be represented as a pair of a musical symbol from the score and a set of corresponding timestamps in the audio file. The user edits the musical score by opening a digital form of the music score and enters a modification to the score. The edit is then applied to the aligned audio file and a new modified audio file is created that reflects the modified musical score.

[0004] According to an embodiment of the present principles, an apparatus for editing audio information can comprise a first editor that receives at least one modification to a visual representation of audio information and a second editor that transforms the at least one modification to the visual representation into at least one modification to audio information synchronized with the visual representation and applies the transformed modification to the audio information to output modified audio information.

[0005] According to another aspect, the apparatus can comprise a processor configured to receive audio information that has been synchronized with a visual representation of the audio information, receive at least one modification of the visual representation of the audio information, transform the at least one visual presentation modification into at least one audio information modification and apply the audio information modification to the audio information to output modified audio information.

[0006] According to another aspect, the apparatus can include means for receiving at least one modification to a visual representation of the audio information and means for transforming the at least one modification to the visual representation into at least one modification to audio information synchronized with the visual representation and applying the transformed modification to the audio information to output modified audio information. The apparatus can further comprise means for receiving the audio information and synchronizing the audio information with the visual representation of the audio information.

[0007] Other aspects of the apparatus can include audio information that is audio signal data aligned with the visual representation, the visual representation of the audio information is a musical score aligned with the audio information; the musical score has musical notation synchronized with the audio information; at least one modification that comprises at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note; a transformed modification that is applied to the audio information as a relative change; and/or the transformed modification to the audio information is applied as an absolute change.

[0008] According to an embodiment of the present principles, a method for editing audio information can comprise receiving at least one modification of a visual representation of the audio information, transforming the at least one modification of the visual presentation into at least one audio information modification, and applying the audio information modification to the audio information to create modified audio information.

[0009] Another aspect of the method can include further comprising receiving the audio information and synchronizing the audio information with the visual representation of the audio information.

[0010] Yet other aspects of the method can include audio information that is audio signal data aligned with the visual representation; a visual representation of the audio information that is a musical score aligned with the audio information; a musical score that has musical notation synchronized with the audio information; a modification that comprises at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note; and/or a visual presentation modification that is applied to the audio information by at least one of a relative change and an absolute change.

[0011] According to another aspect, the present principles relate to a non-transitory program storage device, readable by a computer.

[0012] According to an embodiment of the present disclosure, said non-transitory computer readable program product tangibly embodies a program of instructions executable by a computer to perform the method described above, in any of its embodiments.

[0013] According to another aspect, the present principles relate to a computer readable storage medium carrying a software program comprising program code instructions for performing the method of the present principles, in any of its embodiments/aspects, when said non transitory software program is executed by a computer.

[0014] A computer readable storage medium can take the form of a computer readable program product embodied in one or more computer readable medium(s) and having computer readable program code embodied thereon that is executable by a computer. A computer readable storage medium as used herein is considered a non-transitory storage medium given the inherent capability to store the information therein as well as the inherent capability to provide retrieval of the information therefrom. A computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0015] The above presents a simplified summary of the subject matter in order to provide a basic understanding of some aspects of subject matter embodiments. This summary is not an extensive overview of the subject matter. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description that is presented later.

[0016] To the accomplishment of the foregoing and related ends, certain illustrative aspects of embodiments are described herein in connection with the following description and the annexed drawings. These aspects are indicative, however, of but a few of the various ways in which the principles of the subject matter can be employed, and the subject matter is intended to include all such aspects and their equivalents. Other advantages and novel features of the subject matter can become apparent from the following detailed description when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] 

FIG. 1 depicts an example system of musical score editing in accordance with an embodiment of the present principles.

FIG. 2 depicts an example of a musical piece used to obtain a single audio track recording in accordance with an embodiment of the present principles.

FIG. 3 depicts an example of an edited musical piece used to obtain a modified single audio track file in accordance with an embodiment of the present principles.

FIG. 4 depicts an example apparatus that transforms changes to a visual representation of audio information into changes applied to audio information in accordance with an embodiment of the present principles.

FIG. 5 depicts a flow diagram of a method of transforming changes from a visual representation of audio information into changes applied to audio information in accordance with an embodiment of the present principles.

DETAILED DESCRIPTION

[0018] The subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It can be evident, however, that subject matter embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the embodiments.

[0019] A user is given the ability to edit an audio file containing audio tracks using only a corresponding musical score. Contrary to traditional approaches, it requires no prior knowledge from the user about audio file processing. The techniques also enable the user to automate the processing for a given sequence of notes. One embodiment of these techniques consists of the user importing the score and the audio file, editing the score, and exporting the new audio file.

[0020] In another embodiment of the techniques, edits to musical scores are automatically transcribed into elementary instructions that can be utilized to change a corresponding synchronized audio file. To accomplish this, an audio recording and a musical score that was played in the audio recording are aligned or synchronized together. The output is an alignment between the audio file and the musical score, consisting of an ordered sequence of alignment items. An alignment item can be represented as a pair of a musical symbol from the score and a set of corresponding timestamps in the audio file. The user then opens the digital sheet music and enters a modification to a sheet. The edit is then applied to the aligned audio file and a new modified audio file is created.

[0021] Some of the advantages of utilizing these techniques include, but are not limited to, providing an alternative way to existing methods for editing an audio file of a musical recording, providing a way of editing a recording to users who do not have any knowledge in audio signal processing, providing a faster way to edit musical recordings, providing a way to apply the exact same processing to several recordings of the same score - by interacting with the score only and providing a way to apply the exact same processing to a sequence appearing several times in the music piece.

[0022] The techniques described herein can be applied to music postprocessing such as, but not limited to, editing the mistakes of a player when the music was recorded (e.g., wrong notes in the original score or wrong notes played), changing the interpretation of the player (volume, rhythm, speed), repeating a few bars by adding begin/end repeat bar-lines or by copy-pasting selected bar sequence, composing new music by ordering the bars differently and/or changing the music key or the tonality of the full music recording or of a part of it. It can also be applied to multitrack composition such as, but not limited to, mixing together different tracks that were recorded separately (thus, possibly not synchronized), by just aligning them with the corresponding scores and combining the scores together. It can also be applied to movie postproduction such as, but not limited to, adapting the recorded music track to a video, for example by adapting the original soundtrack to a specific scene in a movie so as to better synchronize highlights in the music and in the scene.

[0023] An example of a system 100 for this process is illustrated in FIG. 1. An audio recording 102 of a musical score 104 and the musical score 104 are provided for alignment or synchronization 106. It should be noted that an 'audio recording' is typically a capture of 'audio tracks' which can include a broad spectrum of audio frequencies. These audio tracks can be digitized into an 'audio file.' These terms can be used interchangeably to represent the audio recording 102. The audio tracks can be analog as well. In general terms, whether it is digital or analog, the audio recording 102 can be referred to as 'audio information.'

[0024] In one embodiment, the alignment/synchronization is accomplished by representing a segment of the audio recording 102 with at least one musical note from the musical score 104. The segment of the audio recording can be a point in time and/or a range of time. For example, when a note of the musical score is initiated (e.g., an attack for a piano note), a link to the time in the audio recording 102 can be made. In another example, when the note is first initiated, a link to the time in the audio recording 102 can be made along with an ending time and/or a duration of time. For example, an A flat note can start at 1:02 (one minute, two seconds) and end at 1:04 (one minute, four seconds) or start at 1:02 (one minute, two seconds) and last (duration) for 2 seconds. One skilled in the art will appreciate that the alignment/synchronization between the audio recording 102 and the musical score 104 can be accomplished in many different ways.

[0025] A user 110 can interact with the musical score 104 using a musical score editor 112. The musical score editor 112 can present a musical score to the user 110 such that the user 110 can edit individual notes and/or a series or sequence of notes of the musical score 104. The musical score editor 112 accepts edits or modifications from the user 110 and processes the edits into at least one modification instruction 114 for the musical score 104. A score-based audio editor 116 implements the modification instruction(s) on the aligned audio recording and creates a modified audio recording 118 that represents the modified musical score that includes the user's edits to the original musical score 104.

[0026] Further details are included in the following use example. In this scenario, a user is a flute player. She records herself playing a particular piece of sheet music. The sheet music is of a digital form and saved on an electronic device in a musical format such as MusicXML (Music Extensible Markup Language). The user is able to read the music using a system that can represent the file graphically. The obtained recording is a single audio track where she plays the whole musical piece 202 as illustrated in the example 200 of FIG. 2. The audio recording is then aligned/synchronized to the musical piece 202. The alignment can be obtained by various techniques such as, for example, saving the alignment while a user reads and plays the music, including by following the notes that are being played and/or by tracking the eye movement on the sheet music, etc. The alignment can also be obtained by carrying out postprocessing of the audio file and the sheet music.

[0027] The output of the alignment can be in the simplified form of musical note and time alignment notations such as the following example:

(G8, 3-4, 2)

(RB, 4, 20)

(C4, 4-6, 20-22, 2)

(C4, 6-8, 22-24, 2)

(C4, 8-10, 24-26, 2)

(D4, 10-12, 26-28, 2)

(E2, 12-16, 2)

(D2, 16, 20, 2)

(RE, 16, 28)

(C1, 28-36, 2)

[0028] Where the notations are defined according to TABLE 1.

TABLE 1- Notation Example

(G8, 3-4, 2)	8th note G, from second 3 to second 4 in the recording, volume level 2
(RB, 4, 20)	Beginning of repeat line, happening second 4 and second 20
(C4, 4-6, 20-22, 2)	4th note C, from sec 4 to 6 and sec 20 to 22 , volume level 2
(C4, 6-8, 22-24, 2)	4th note C, from sec 6 to 8 and sec 22 to 24, volume level 2
(C4, 8-10, 24-26, 2)	4th note C, from sec 8 to 10 and sec 24 to 26, volume level 2
(D4, 10-12, 26-28, 2)	4th note D, from sec 10 to 12 and sec 26 to 28, volume level 2
(E2, 12-16, 2)	Half note E, from sec 12 to 16, volume level 2
(D2, 16, 20, 2)	Half note D, from sec 16 to 20, volume level 2
(RE, 12, 28)	End of repeat line, happening second 12 and second 28
(C1, 28-36, 2)	Whole note C, seconds 28 to 36, volume level 2

[0029] Once the musical piece or score has been recorded, the user listens to the recording and may not be satisfied with it. For example, she wants to modify it but she does not know anything about signal processing of an associated audio file. As she can read the sheet music, she opens the musical piece in a musical score editor that enables her to modify it. She makes several modifications to the score of the musical piece 302. The modified musical piece 302 is illustrated in the example 300 of FIG. 3. There are several changes - the first note was an eighth note and is now a quarter note, it is now twice as long as before 304, there is a "piano" symbol indicating that the first, second and last bars should be played piano 306, the third note is now a D instead of a C 308, and the repeat bar was deleted and thus the first bar should not be repeated anymore 310.

[0030] There are two ways of interpreting these changes. The first way - absolute change: apply the modifications as they are written - the first note is prolonged to the duration of 4^th notes according to the tempo followed by the player, the volume of the last 3 bars is set to piano, the third note is set to a D and the second occurrence of the first bar is deleted. The corresponding modified audio file, aligned with the sheet music, can then follow this pattern:

(G4, 2-4, 2)

(C4, 4-6,1)

(D4, 6-8,1)

(C4, 8-10,1)

(D4, 10-12,1)

(E2, 12-16, 1)

(D2, 16, 20, 1)

(C1, 16-3,1)

[0031] The first note was prolonged from 1 to 2 seconds, starting from second 2 instead of 3. The repeat line was deleted. The four following notes were kept from seconds 4 to 12 but since the repeat line was deleted, the second occurrences of these notes were also deleted. Also, the volume was set to 1 ("piano") for all other notes. The third note was changed from C to D. The last note was brought earlier, starting second 16 instead of second 28.

[0032] The second way of interpreting these changes can be a relative change - apply the modifications relatively to what was recorded: the first note is prolonged to twice its original duration, the volume of the last 3 bars is reduced, the third note is set one tone higher and the second occurrence of the first bar is deleted. These types of relative changes can be easier to process.

[0033] In one embodiment of the techniques disclosed herein, the following elementary operations can be included in an implementation of a "score-based audio editor" - changing the duration of one note, changing the tonality of one note, changing the volume of one note, adding a new note and/or changing the absolute position of one note. Typically, other modifications are combinations of these five elementary ones.

[0034] In order to modify individual notes within a polyphonic music recording, audio source separation methods can be used. The goal of audio source separation is to separate (disassociate) individual sources (e.g., vocal track, piano track, etc...., in the case of musical recordings). However, without much additional difficulties, the audio source separation techniques can be applied so as to extract individual musical notes within the corresponding musical sources. Moreover, the performance of audio source (or notes) separation can be greatly improved when the musical score is available along with the corresponding audio recording, which is the case here. Once all individual notes are demixed by source separation, editing them becomes possible. To change the duration of one note (the first operation, sarpra) without affecting its tonality, it is not possible to just stretch the audio without affecting the pitch of the note. However, this operation can be done using a "phase vocoder" technique. This is a vocoder (a system used to reproduce human speech) that can scale both the frequency and time domains of audio by using phase information.

[0035] The operation of changing the note tonality (the second operation, supra) without affecting its duration is equivalent in principle to the first operation and can be implemented using the "phase vocoder" method as well. In addition, for both the first and second operations and for harmonic instruments having "attack+harmonic" note structure (e.g., piano, guitar or xylophone), it could be also beneficial to separate transient (attack) vs. tonal (harmonic) parts and apply time or frequency modifications only on the harmonic parts. Indeed, if the total duration of a piano note is twice as long, that does not mean that the duration of the "hummer hit" (the attack) must be twice as long as well, the duration of the attack must remain the same.

[0036] The third operation, supra, (changing the volume of one note) can be performed simply by multiplying the note waveform by the corresponding positive gain. It should be also noted that the first, second and third operations can be alternatively implemented not as modifications of the notes themselves, but as replacement of these notes by the notes with desired properties from other parts of the performance, if such notes are available. Such an alternative could lead to better results in some situations. Indeed, for example changing the volume of one note (the third operation), meaning making this note so as it was played stronger, does not consist exactly in multiplying the waveform by a gain. Thus, copying a note with desired amplitude from another place can lead to a better result.

[0037] To add a new note (the fourth operation, supra), the note with the corresponding characteristics (instrument/heights/duration/volume) can be found in some other place of the same recording and copied to the desired place. In case the desired note (with exactly the characteristics as specified) does not exist in the recording, it can be obtained from some other close note (e.g., all characteristics are OK except the duration) by modifying it using the first, second and/or third operations.

[0038] As for the fifth operation, supra, its implementation is simple, the corresponding note must be removed in its current position and copied to the new position.

[0039] In FIG. 4, an example 400 of an apparatus that transforms changes to a visual representation of audio information 404 (e.g., a musical score, etc.) into changes applied to audio information 402 (e.g., audio track, audio recording, audio file, etc.) in accordance with an embodiment of the present principles is shown. The audio information 402 can be, but is not limited to, audio signal data and the like associated with the visual representation of the audio information 404. The audio information 402 can be stored data such as file data and/or it can be streamed data and the like. The audio information is typically, but not limited to, audio frequency information related to musical scores and the like. The original form of the audio information 402 can be digital or analog. Analog audio information is typically digitized before use, but one skilled in the art can appreciate that modifications can be made to analog recordings as well. The visual representation of the audio information 404 can be, but is not limited to, musical notations and the like such as those found in musical scores. One skilled in the art can appreciate that other types of notations and symbology can be used to make a visual representation of audio information.

[0040] A synchronizer 406 is used to align or synchronize the visual representation of the audio information 404 with the audio information 402. This alignment information is received by an audio information editor 412. The visual representation of the audio information 404 is received by a visual representation editor 408. The visual representation editor 408 provides an interactive interface with a user and/or automated process and the like to receive changes or modifications to the received visual representation of the audio information 404. Once changes are received, the visual representation editor 408 provides the changes/modifications to the audio information editor 412. The audio information editor 412 alters the changes from the realm of the visual representation to changes in the realm of the audio information. This is facilitated by the synchronization of the audio information with the visual representation of the audio information. Thus, for example, in one embodiment a musical note on a score (visual representation) is changed to a lower octave. The audio information editor 412 converts this change in notation to a change in frequency (audio information). It 412 then correlates the position of the notation in the musical score to the timing location in the audio information. The audio information editor 412 then applies the change to the audio information creating modified audio information 414 that now correlates (aligns with) to the modified visual representation of the audio information.

[0041] In view of the exemplary apparatus shown and described above, methodologies that can be implemented in accordance with the embodiments will be better appreciated with reference to the flow chart of FIG. 5. While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of blocks, it is to be understood and appreciated that the embodiments are not limited by the order of the blocks, as some blocks can, in accordance with an embodiment, occur in different orders and/or concurrently with other blocks from that shown and described herein. Moreover, not all illustrated blocks may be required to implement the methodologies in accordance with the embodiments.

[0042] FIG. 5 is a flow diagram of a method 500 of transforming changes from a visual representation of audio information into changes applied to audio information in accordance with an embodiment of the present principles. Audio information that has been synchronized with a visual representation of the audio information is received 502. As discussed above, synchronization facilitates in applying modifications between the two forms of information. At least one modification of the visual representation of the audio information is also received 504. The received modification can be via a user and/or some type of automated process and the like. The at least one visual representation modification is transformed into at least one audio information modification 506. Means to transform the modification from the visual representation to the audio information realms have been discussed above as well. The audio information modification is then applied to the audio information to create modified audio information 508. After the modification is applied to the audio information, it will then be synchronized with the modified visual representation of the audio information. In one embodiment, modified audio information comprising a modified audio file is generated by the process.

[0043] What has been described above includes examples of the embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the embodiments, but one of ordinary skilled in the art can recognize that many further combinations and permutations of the embodiments are possible. Accordingly, the subject matter is intended to embrace all such alterations, modifications and variations. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word.

Claims

1. An apparatus for editing audio information, comprising:

means for receiving at least one modification to a visual representation of the audio information; and

means for transforming the at least one modification to the visual representation into at least one modification to audio information synchronized with the visual representation and applying the transformed modification to the audio information to output modified audio information.

2. The apparatus of claim 1, further comprising:

means for receiving the audio information and synchronizing the audio information with the visual representation of the audio information.

3. The apparatus of claim 1 or 2, wherein the audio information is audio signal data aligned with the visual representation.

4. The apparatus according to one of claims 1 to 3, wherein the visual representation of the audio information is a musical score aligned with the audio information.

5. The apparatus of claim 4, wherein the musical score has musical notation synchronized with the audio information.

6. The apparatus according to one of claims 1 to 5, wherein the at least one modification comprising at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note.

7. The apparatus according to one of claims 1 to 6, wherein the transformed modification is applied to the audio information as a relative change.

8. The apparatus of claims 1 to 6, wherein the transformed modification to the audio information is applied as an absolute change.

9. A method for editing audio information, comprising:

receiving at least one modification of a visual representation of the audio information;

transforming the at least one modification of the visual presentation into at least one audio information modification; and

applying the audio information modification to the audio information to create modified audio information.

10. The method according to claim 9, further comprising receiving the audio information and synchronizing the audio information with the visual representation of the audio information.

11. The method according to one of claims 9 to 10, wherein the audio information is audio signal data aligned with the visual representation.

12. The method according to one of claims 9 to 11, wherein the visual representation of the audio information is a musical score aligned with the audio information.

13. The method of claim 12, wherein the musical score has musical notation synchronized with the audio information

14. The method according to one of claims 9 to 13, wherein the modification comprising at least one of changing a duration of a musical note, changing a tonality of a musical note, changing a volume of a musical note, adding a new musical note and changing an occurrence of a musical note.

15. The method according to one of claims 9 to 14, wherein the visual presentation modification is applied to the audio information by at least one of a relative change and an absolute change.

Drawing