EXHIBIT 99.2
- ------------
- --------------------------------------------------------------------------------
United States Patent 6,072,933
Green June 6, 2000
- --------------------------------------------------------------------------------
System for producing personalized video recordings
ABSTRACT
A system for the production of video signals includes a playback device for
playing back prerecorded video and audio signals from a prerecorded storage
medium, a source of user supplied video and audio signals, a video and audio
mixer for combining the prerecorded and user supplied signals to provide
combined video and audio outputs, a production monitor connected to the mixer to
display to the user the mixed signals, and a storage or reproduction device
receiving a mixed video signal output from the mixer. The prerecorded storage
medium, wherein the prerecorded storage medium stores, as well as a video
channel and at least one audio channel, at least one prompting channel, the
video signals stored on the prerecorded medium being prekeyed to indicate areas
to be overlaid in the mixer by the user supplied video signals, and the mixer
being operative to convert signals from the prompting channel into production
control signals.
- --------------------------------------------------------------------------------
Inventors: Green; David (235 Yorkland Blvd. Suite 1200, North York, Ontario, CA)
Appl. No.: 807532
Filed: February 28, 1997
Current U.S. Class: 386/46; 386/95
Intern'l Class: H04N 005/76
Field of Search: 386/1,4,38,39,45,46,52,53-64,117,95
348/578,584,585,586,587,588,589,590-592
- --------------------------------------------------------------------------------
References Cited [Referenced By]
- --------------------------------------------------------------------------------
U.S. Patent Documents
4688105 Aug., 1987 Block et al. 386/55.
5099337 Mar., 1992 Cury 386/54.
5151793 Sep., 1992 Ito et al. 386/98.
5566251 Oct., 1996 Hanna et al. 348/589.
Primary Examiner: Tran; Thai
- --------------------------------------------------------------------------------
PARENT CASE TEXT
- --------------------------------------------------------------------------------
RELATED APPLICATION
This application is a continuation-in-part of my application Ser. No.
08/399,013, filed Mar. 6, 1995, now abandoned.
- --------------------------------------------------------------------------------
CLAIMS
- --------------------------------------------------------------------------------
I claim:
1. A system for video production, comprising a source of prerecorded video and
audio signals from a prerecorded storage medium, a source of user supplied video
and audio signals, a video and audio mixer for combining the prerecorded and
user supplied signals to provide combined video and audio outputs, a production
monitor connected to the mixer to display to the user the mixed signals, and a
storage or reproduction device receiving a mixed video signal output from the
mixer, wherein the prerecorded video signals from the prerecorded storage medium
have a video signal content prekeyed with a keying signal to indicate areas
within the prerecorded video signal to be replaced by the user supplied video
signals, the mixer being operative to recognize the keying signal and substitute
the user supplied video signal for those portions of said prerecorded video
including said keying signal, and the mixer being operative to convert signals
from the prompting channel into production control signals.
2. A system according to claim 1, wherein the control signals include user
prompts displayed on the production monitor but absent from the combined video
output.
3. A recording medium carrying a prerecorded video signal, prekeyed to define
background of images defined by said video signal, which video signal, on
playback by a user of the recording medium in apparatus configured to recognize
the prekeyed background areas, will generate a signal into which may be
inserted, at least in those background areas, a local signal provided by the
user.
4. A recording medium according to claim 3, wherein the video signal prerecorded
on the medium is predistorted by enhancing the brightness of at least the
lowlights of the prerecorded signal outside said background areas while
maintaining the background areas at or below black level.
5. A recording medium according to claim 3, wherein the recording medium further
carries at least one audio channel, and at least one prompting channel including
data translatable into instructions for control of the user provided video
signal.
6. A recording medium according to claim 3, wherein the data in the prompting
channel is translatable into video data optionally overlayable on video data
from said video channel.
7. A system for generating video signals comprising prerecorded video signals
overlaid on user provided video signals, comprising a recording medium carrying
a prerecorded video signal, prekeyed to define background of images defined by
said video signal, which video signal, on playback by a user of the recording
medium in apparatus configured to recognize the prekeyed background areas, will
generate a signal into which may be inserted, at least in those background
areas, a local signal provided by the user, the video signal prerecorded on the
medium being predistorted by enhancing the brightness of at least the lowlights
of the prerecorded signal outside said background areas while maintaining the
background areas at or below black level, and a mixer receiving video signals
generated by playback of video signals from said recording medium and video
signals from a user provided source, the mixer including means for enhancing the
brightness of at least the lowlights of the user provided signal to a similar
degree as the lowlight enhancement of the prerecorded signal, and a luminance
keyer receiving said prerecorded signal and said lowlight enhanced user provided
signal to produce an overlaid video signal in which the user signal is overlaid
on the keyed portions of the prerecorded signal, and means for restoring the
lowlights of the overlaid video signal to their original levels to provide an
output signal.
- --------------------------------------------------------------------------------
DESCRIPTION
- --------------------------------------------------------------------------------
FIELD OF THE INVENTION
This invention relates to the production of video sequences, in which users own
video signal can be combined with a prerecorded video signal to provide a
composite sequence, in which prerecorded images are integrated with the user's
own material.
REVIEW OF THE ART
So called video karaoke systems are well known in which a video recording is
provided, with the video display overlaid by text indicating the words of a
song, usually with some form of marker to indicate the synchronization of the
words of the song with a musical accompaniment recorded on the sound channel or
channels of the video recording. This enables persons viewing the recording to
"sing along" with the musical accompaniment.
It is also well known to produce composite video signals by overlaying one
signal on another, utilizing one of several keying techniques of which those
known as chroma-keying and luminance keying are the most common. In
chroma-keying, essential elements of a foreground scene, typically a person or
persons, are imaged against a background having a higher level of saturation of
a particular colour than is likely to occur in the foreground objects. Typically
an intense blue background is utilized, but other colours may be used provided
that, in the particular application, they enable the foreground and background
to be reliably differentiated by signal processing circuitry. An alternative
approach is known as luminance keying, in which it is arranged that the
luminance level of the background against which the foreground objects are
imaged is consistently and detectably lower than that of the wanted foreground
objects. During the combination process, boundaries between the foreground
objects and the background are detected on the basis of the above-mentioned
difference in colour content or luminance level, so as to produce a switching
signal which switches a second video signal, synchronized with the first, into
the background areas. These techniques are well known and understood in the art,
but require good quality and hence expensive equipment to operate reliably.
In U.S. Pat. No. 5,099,337 (Cury) it is proposed to provide a selection of
background audio and video recordings which can be selected from separate
libraries and combined with foreground audio and video signals provided by a
user, so as to enable the user to provide customized video recordings in which
the users own foreground images and audio signals are superimposed upon selected
background audio and video signals. This in effect provides system in which the
user, as well as providing a foreground audio signal, also provides a foreground
video signal, and is provided with means for recording the result. A performer
performs in front of a blue screen, so that the performer's image may be
chromakeyed into the background video signal, thus providing the illusion that
the performer is performing in the selected background. During performance,
prompt information is provided to a performer through a prompt monitor from a
prerecorded prompt library.
A limitation of such systems is that, by their nature, they can only provide
background for a user's performance, and may require the user,s signal to be
keyed which, as mentioned above, may be difficult to achieve reliably with
consumer quality equipment.
SUMMARY OF THE INVENTION
In its broadest aspect, the present invention relates to a system in which a
prerecorded video signal is prekeyed to define background areas which, on
playback by a user of a recording medium carrying the keyed signal on apparatus
configured to recognize the prekeyed background areas, will generate a signal
into which may be inserted, in those background areas, a local signal provided
by the user, which need not itself be keyed.
The present invention further seeks to provide a system in which a keyed video
prerecording is used to provide a prerecorded signal which is combined with a
user provided background signal to provide a final combined signal, the
prerecording including prompt channel, which can be suppressed in the final
combined signal, to assist a user and/or a user's equipment to provide a
background signal compatible with the prerecorded signal.
According to the invention, there is provided a system for the production of
video signals, comprising a playback device for playing back prerecorded video
and audio signals from a prerecorded storage medium, a source of user supplied
video and audio signals, a video and audio mixer for combining the prerecorded
and user supplied signal to provide combined video and audio outputs, a
production monitor connected to the mixer to display to the user the mixed
signals, and a storage or reproduction device receiving a mixed video signal
output from the mixer, wherein the prerecorded storage medium stores, as well as
a video channel and at least one audio channel, at least one prompting channel,
the video signals stored on the prerecorded medium being prekeyed to indicate
areas to be overlaid in the mixer by the user supplied video signals, and the
mixer being operative to convert signals from the prompting channel into
production control signals. Typically the production control signals include
prompts displayed on the production monitor but absent from the combined video
output.
The invention extends to a recording medium providing multiple channels of
information, including a video channel, a least one audio channel, and at least
one prompting channel, the video channel being recorded with a video signal
prekeyed to indicate picture areas available for overlay by a user provided
video signal, and the prompting channel including data translatable into
instructions for control of the user provided video signal.
In order to permit production of prerecorded tapes (or discs or other media)
which will perform reliably with low cost user equipment, I employ by preference
a modified luminance keying system in producing the prerecorded tapes. In simple
terms, the brightness level of at lease the lowlights of portions of images
which are not to be overlaid by the user signal is artificially enhanced so that
the "black level" of retained image portions is well above the normal black
level of a recorded signal, thus enabling the keyed portion which is at or below
the normal black level, to be readily distinguished. The user supplied signals
are also brightness enhanced in a similar manner prior to mixing with the signal
from the prerecorded tape, and the brightness level of the mixed signal is then
returned to normal so as to restore the original black levels.
The word "video signal" as used in this specification, means a television type
video signal consisting of a sequence of frames which when reproduced in radial
succession are capable of providing a moving picture. It does not include
bit-mapped or vector digital representations of a single static image.
SHORT DESCRIPTION OF THE DRAWINGS
In the drawings:
FIG. 1 is a simplified block diagram of the system;
FIG. 2 is a simplified block diagram of a mixer unit used in the system;
FIG. 3 is a more detailed block diagram of an exemplary mixer unit; and
DESCRIPTION OF THE PREFERRED EMBODIMENT
Referring to FIG. 1, which has been simplified by omitting any consideration of
sound as opposed to video channels, a prerecorded tape or video disk,
generically referred to as a prerecording 2, is replayed to provide a foreground
video signal upon which a user signal may be overlaid using chroma or luminance
keying or any other system enabling effective identification of areas of the
recorded image available for overlay by a user signal (henceforward generically
referred to as "keying") It should be understood that the keying signal,
whatever form it takes, for example a high saturation of blue, or a very low
luminance level, is already built into the prerecording 2 so as to predefine
those areas of the prerecorded signal 4 available for overlay by user provided
video signals: the user provided signals will not usually themselves be keyed,
and if they are keyed, it will be for purposes extraneous to the present
invention. It should also be understood that overlay effects depend on
successive fields of the prerecorded video being keyed in certain areas, fields
may also be either wholly keyed or not keyed at all, thus permitting scenes
provided wholly either by the user or wholly by the prerecorded tape. The
prerecorded tape also preferably carries, as well the video signal channel, at
least one audio channel for audio signals, and at least one prompting channel 6
for prompting signals prompting of instructing the user and/or the users
equipment so as to assist in rendering the content of the user provided signal
compatible with the foreground content of the prerecording. Prompts to the user
may include text prompts, e.g. the words of a song and/or stage directions, or
position of dimension indicators to assist the user in controlling direction or
zooming of a video camera providing the user signal. It should further be
understood that the playback of the prerecording may occur at a remote point,
with the signal being distributed to users over a distribution network such as a
CATV network or the internet.
A controllable user video source, usually a camera 8 (or one of multiple
selectable cameras) has a signal output to a mixer 12 which combines the video
signals by inserting the user signal output 10 in those areas of the prerecorded
signal 4 which are identified by the keying, or by mixing the user signal with
the prerecorded signal, depending upon the effect desired. The superimposition
provided by mixing may be useful for some applications for example training
videos. The prompting signals on channel 6 are translated by the mixer 12 into
signals displayed on a user monitor 14 so that a user may control the camera 8,
or sing (or otherwise perform) along with a prerecorded artist; or the user
monitor may implement a camera control function (for example a power zoom
control) which is applied directly to the camera to control its input to the
mixer 12, which replaces the keyed portions of the signal from the prerecording
2 with the signal 10 from the camera 8. It is however preferred that zooming be
performed electronically within the mixer, because of the lack of
standardization of camera controls. The signals 4 and 10 must of course be
synchronized to a common set of scanning signals before combination. This is
preferably achieved as discussed below with reference to FIG. 3, but in an
alternative arrangement the camera 8 receives a signal via control 14 which
synchronizes it to the signal 6 from the prerecorded source. The output signal
from the mixer 12 is passed to a suitable recorder and/or display 16.
Referring now to FIG. 2, this shows schematically one embodiment of the mixer 12
in somewhat more detail. Video inputs 4 (including the signal 6) and 10 are
applied to a timebase control unit 20 which ensures synchronization between the
video signals, preferably by applying them to synchronized frame memories as
described below, since this requires no feedback control of the timing of either
input signal. A brightness enhancement circuit 21 may be provided in one
advantageous embodiment of the invention to enhance the brightness of the
lowlights of the user supplied video signal so that its black levels are well
above the pedestal level of the video signal by an amount equal to a similar
brightness boost already present in unkeyed image areas of the prerecorded video
signal in unkeyed areas, keyed areas being maintained at or slightly below
normal black level. This distortion of the prerecorded video signal enables the
keying switcher to detect keyed areas very simply and reliably since there will
be a substantial minimum difference of the level of the video signal in keyed
and unkeyed areas which can readily be detected even in the presence of some
distortion of the video signal. Typically, this enhancement may be about 20% or
more the normal difference between black and white levels of the signal. The
brightness enhancement applied to the user signal by the circuit 21 should of
course be substantially the same as that applied during manufacture of the tape
providing the signal 4. Downstream of the switcher 22, a further brightness
control circuit 23 applies an equal and opposite correction to restore proper
brightness levels in the signal. A level switching circuit 22 responds to the
keying signal extracted from the signal 4 to switch between levels of the
signals 4 and 10 and thus provide a combined video output signal 24 to the
recorder and/or display 16 (FIG. 1). For full keying the different levels of the
two signals are full/zero and zero/full respectively, but other level
combinations can be utilized with suitable digital mixing technology, in a
manner known per se. The video output signal is also applied, in this
embodiment, to a closed caption decoder 28 which extracts control signals and
which are encoded into an available channel in the prerecording 2, in this case
that portion of the video interval between frames normally reserved for closed
captioning. The control signals are overlaid on a video signal 26 sent to the
monitor 14 (FIG. 1), so as to provide directions (either text prompts or
instructions or positioning marks) to the user of the system so that the user
may "sing along" or otherwise perform in synchronization with the prerecorded
signal, or exercise suitable control over the camera 8. The signal 26 may also
be decoded to provide a camera control output, for example a zoom control
signal. The mixer 12 typically also incorporates an audio effects and mixer unit
30 which receives and mixes inputs from an audio channel or channels 32 from the
prerecording 2, inputs from an audio channel or channels 34 from a user
microphone or microphones, and provides an output audio channel or channels at
36 to the monitor 14 and at 38 to the recorder 16.
Referring now to FIG. 3, one version of the mixer 12 is shown in more detail. It
should be understood that, in the main, the mixer 12 shown in FIG. 3 is similar
to known digital video mixers incorporating chroma or luminance keying
facilities except that it responds additionally to an additional prompt channel
included in the prerecorded video input 4. Typically the closed caption channel,
the data for which is encoded into certain lines of the video signal in the
vertical interval outside of the normally displayed area, is used to convey
prompt messages and control data to the system, and thus a closed caption
decoder 100 is incorporated into the mixer 12 to recover this data. Such
decoders are well understood in the art and need not be described further.
The mixer operates under control of a microcomputer 102 including appropriate
working memory. An example of a suitable device is the 87C752 from Intel
Corporation, and in general it controls the mixer in a manner similar to that of
known video mixers. Accordingly, the mixer will be described primarily with a
view to explaining how it differs from conventional digital video mixers, such
as the MX-1 digital video mixer from Videonics (although it should be understood
that many of the functions provided by such mixers are not essential to the
present application and may be omitted to reduce costs), and so as to explain
its relationship to the essential functions of the invention.
The video input 4 from the prerecorded source 2 may be either composite or
S-video. In the latter case the video chroma and luminance signals are applied
to separate analog to digital converters 104, 106 under control of a clock
generator 108 which also controls a decoder 110 passing digital Y (luminance)
and U and V (chrominance) signals to a frame memory 112 under control of a video
memory write controller 114. If a composite video signal is provided, it is
applied to the converter 106 and converter 104 is not used.
Similarly, the input 10 from the camera 8 is applied to A/D converters 124 and
126, or converter 124 only if a composite signal, under control of a clock
generator 128 also controlling a decoder 130 passing signal to a memory 132
under control of controller 134.
A synchronized memory read controller 140 reads the contents of the memories 112
and 132 in synchronism under control of the microcomputer 102, the Y, U and V
signals read from the memories being selected by a multiplexer 142, under
control of a luminance or color keyer 144 itself controlled by signals read from
the memory 112. When the selected keying signal, be it luminance or chroma, is
present at a level denoting background, then the keyer causes the multiplexer
142 to pass at least part of the signals derived from input 10 and block at
least part of the signals from input 4; otherwise it passes at least part of the
signals derived from input 4 and blocks at least part of the signals derived
from input 10. By this means signals from the camera input 10 are inserted into
the keyed portions of the video input 4.
The Y (luminance) signal derived from the video input 4 is also passed to the
closed caption decoder 100, which decodes the closed caption data in
conventional manner to recover data contained therein and pass it to
microcomputer 102 which interprets the data and either forwards it to a
conventional overlay generator 146 and/or generates signals applied on a line
148 to control zooming (for example) of camera 8, and/or to the scaler 162 or
interpolator 160 discussed below, if provided.
The Y, U and V signals from multiplexer 142 are passed to output encoders 150
and 152 under control of an output clock and synchronization separator 158,
providing the video outputs 24 and 26 via amplifiers and filters 154 and 156,
the V signal to encoder 152 being overlaid by text or video regenerated by the
overlay generator 146 so that control instructions are passed to the user
monitor 14.
The microcomputer 102 may control additional optional processing circuits
between the multiplexer 142 and the encoders 150 and 152, these being of types
known in digital video mixers; in this case there is shown a zoom interpolator
160, a scaler 162 controlled by a clock generator 164, and a lower bit switch
166. The zoom interpolator and scaler provide an electronic zoom effect which is
preferred to optical zoom controlled by the line 48. The electronic zoom
interpolator and scaler will act on the combined signal, and not just the local
camera input as would an optical zoom. The lower bit switch 166 can be activated
in known manner to provide a posterization effect. The interpolater 160 and
scaler 162 may also be configured to be controlled manually by the user, since
they act conjointly on the prerecorded and user provided signals.
Audio inputs 32 from local microphones are processed by potentiometer 168,
preamplifiers 170, mixer 172, and a master potentiometer 174 before being
applied to an A/D converter 176 while audio inputs associated with the
prerecorded video signal are applied to A/D converters 278, the outputs from the
A/D converters 176 and 178 being combined and optionally processed by a digital
signal processor 180 in known manner under control of microcomputer 102. The
processed digital audio signals are then passed through digital to analog
converters 182 and preamplifiers 184 to the outputs 36 and 38.
The programming of microcomputer 102 has not been described, since except for
any processing of the signals from the closed caption decoder 100, it is similar
to that for known digital mixers. Processing by microcomputer 102 of closed
caption data merely consist of intercepting data encoded on the closed option
line 20 which provides control signals for passage by the microcomputer to the
output 148 or the processing circuits 160, 162 and 166.
In use, the mixer 12 will operate much like a conventional digital video mixer,
except that the keying function controlled by the keying signal in the input 4
is a default function, and control signals or messages in the closed caption
field of the input 4 are decoded and output either as video overlays on the
monitor 14 or as camera or mixer control signals.
It will therefore be appreciated that a users local generated video (and audio)
signals may be combined with the video input from a prerecorded tape or disc to
provide video and audio outputs in which user contributed images and sounds are
combined with those on the prerecorded tape to provide a composite output in
which prerecorded images are inserted into images provided by the user so that
for example a famous singing star or animated cartoon character may appear to be
performing together with a user in the user's own home or the user's choice of
surroundings. The control or prompting channel, for example closed captions
appearing on the monitor 14, may provide on screen prompts to a user, which are
invisible on the output passed to a recorder 16, either in the form of words, or
indicators indicating how the user should place a locally generated image on the
screen. This channel may also carry data which can be converted by the
microcomputer 102 into data output on the line 148 in the form of camera control
signals, for example to control of a zoom function of the camera 8. Unlike prior
systems, the user does not need to perform against a blue screen or other means
to generate keying of the user signal, since the keying is prerecorded into the
prerecorded foreground signal. Such prerecorded, prekeyed signals have numerous
potential applications of which those discussed above are merely exemplary.
* * * * *
PATENT ASSIGNMENT
UNITED STATES PATENTS AND PATENT APPLICATIONS
The undersigned, David Green, the full post office address of whose principal
office or place of business is 462 Spring Gale Boulevard, Thornhill, Ontario,
Canada, L4J 5E1, in consideration of the sum of Two Dollars ($2.00) and other
good and valuable consideration, the receipt of which is hereby acknowledged,
does hereby assign, transfer and set over unto High Plateau Holdings, Inc., the
full post office address of whose principal office or place of business is 462
Spring Gate Boulevard, Thornhill, Ontario, Canada, L4J 5E1, its entire right,
interest and title in the United States patents and patent applications listed
in Schedule "A" annexed hereto, and the right to obtain Letters Patent in the
Unites States and all right, title and interest in and to any patent or patents
which may issue therefor, the same to be held and enjoyed by the said High
Plateau Holdings, Inc., its successors and assigns, as fully and completely as
by this undersigned had this Assignment not been made.
IN WITNESS WHEREOF, David Green has hereunto executed this Assignment
at Toronto, Ontario this 8th day of November, 2000.
/s/ David Green
------------------------------------
NOTARIAL ACKNOWLEDGEMENT
------------------------
PROVINCE OF ONTARIO
On this 8th day of November, 2000, before me personally came David Green, to me
known, who being by me duly sworn did depose and say that he resides at 462
Spring Gate Blvd., Thornhill, Ontario; and that he signed his name to the above
instrument.
/s/ Jan Bull
- ------------------------------
Notary Public etc.
My commission does not expire.
SCHEDULE "A": PATENTS AND PATENT APPLICATIONS
---------------------------------------------
Patent
Number Title Date Issued/Filing Date Owner
- --------------------------------------------------------------------------------
6,072,933 System for Producing June 6, 2000 David Green
Personalized Video
Recordings