[0001] The present application claims the benefit of United States Provisional Patent Application
Nos.
62/757,004, filed November 7, 2018;
62/769,875, filed November 20, 2018, and
62/792,195, filed January 14, 2019, whose disclosures are hereby incorporated by reference in their entireties into
the present disclosure.
[0002] One or more aspects of the disclosed subject matter are directed to video encoding
or decoding and more specifically to video encoding or decoding with memory bandwidth
conservation when an affine motion model is used.
[0003] The VVC (Versatile Video Coding) is a new video compression standard being developed
by the joint video experts team (JVET) jointly established by ISO/IEO MPEG and ITU-T.
The VVC standard for single layer coding will be finalized by the end of 2020, with
a design goal of being at least 50% more efficient than the previous standard MPEG
HEVC/ITU-T H.265 Main-10 profile.
[0004] Among proposed coding tools to VVC under consideration, the affine motion compensation
prediction introduces a more complex motion model for better compression efficiency.
In previous standards such as HEVC, only a translational motion model is considered,
in which all the sample positions inside a PU (prediction unit) may have a same translational
motion vector for motion compensated prediction. However, in the real world, there
are many kinds of motion, e.g., zoom in/out, rotation, perspective motions and other
irregular motions. The affine motion model supports different motion vectors at different
sample positions inside a PU, which effectively captures more complex motion. Different
sample positions inside a PU, such as four corner points of the PU, may have different
motion vectors as supported by the affine mode. A PU coded in affine mode and affine
merge mode may have uni-prediction (list 0 or list 1 prediction) or bi-directional
prediction (i.e. list 0 and list 1 bi-prediction).
[0005] In the current VVC design (see JVET-P2001, "Versatile Video Coding (Draft 7)"), the
sub-block size for the affine mode is fixed to 4x4, which creates the 4x4 bi-directional
prediction for the worst-case memory bandwidth consumption of motion compensation.
In the HEVC, the worst-case memory bandwidth consumption for motion compensation is
8x8 bidirectional prediction, while 8x4 and 4x8 PUs use uni-prediction only. The increased
memory bandwidth budget can never catch up with the path of sample rate increase (e.g.,
HEVC is typically for 4K video at 60 fps, while VVC will be used for 8K video at 60
fps, another factor of 4 increase in terms of sample processing rate).
[0006] According to an aspect, a device for video encoding or decoding is provided, comprising:
circuitry configured to
input digital video;
perform an encoding or decoding of a digital video which has been input; and output
the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.
[0007] Advantageously, the circuitry is further configured to calculate the reference block
bounding box size by determining coordinates of an upper-left and bottom-right corner
of the reference block bounding box of a plurality of consecutive sub-block vectors
of the PU, and calculating a width and height of the reference block bounding box
based on the coordinates.
[0008] Advantageously, the reference block bounding box size is calculated based on a prediction
type being unidirectional prediction or bidirectional prediction.
[0009] Advantageously, the at least one predetermined threshold is set based on a prediction
type being unidirectional prediction or bidirectional prediction.
[0010] Advantageously, the circuitry is further configured to
calculate the reference block bounding box size to determine whether the reference
block bounding size exceeds the at least one predetermined threshold, and
perform the first motion compensation operation or the second motion compensation
operation in response to a determination separately for list0 and list1 prediction
of the PU.
[0011] Advantageously, in the first motion compensation operation, all sub-block vectors
of the PU are set to a same vector, the same vector being an affine motion vector
for a single point in the PU.
[0012] Advantageously, in the second motion compensation operation, an affine sub-block
motion vector field for affine motion compensation is generated based on a sub-block
size.
[0013] Advantageously, in the first motion compensation operation, an affine sub-block motion
vector field for motion compensation is generated by using a larger sub-block size.
[0014] Advantageously, the circuitry is further configured to
select the control point motion vectors for the PU so that the resulting reference
block bounding box size does not exceed one or more predetermined thresholds.
[0015] According to an aspect, a method of video encoding or decoding is provided, comprising:
inputting digital video;
performing an encoding or decoding of the digital video which has been input; and
outputting the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.
[0016] Advantageously, the method further comprises:
calculating the reference block bounding box size by determining coordinates of an
upper-left and bottom-right corner of the reference block bounding box of a plurality
of consecutive sub-block vectors of the PU, and calculating a width and height of
the reference block bounding box based on the coordinates.
[0017] Advantageously, the method further comprises:
calculating the reference block bounding block size based on a prediction type being
unidirectional prediction or bidirectional prediction.
[0018] Advantageously, the method further comprises:
setting the at least one predetermined threshold based on a prediction type being
unidirectional prediction or bidirectional prediction.
[0019] Advantageously, the method further comprises:
calculating the reference block bounding box to
determine whether the reference block bounding size exceeds the at least one predetermined
threshold; and
perform the first motion compensation operation or the second motion compensation
operation in response to a determination separately for list0 and fist1 prediction
of the PU.
[0020] Advantageously, in the first motion compensation operation, all sub-block vectors
of the PU are set to a same vector, the same vector being an affine motion vector
for a single point in the PU.
[0021] Advantageously, in the second motion compensation operation, an affine sub-block
motion vector field for affine motion compensation is generated based on a sub-block
size.
[0022] Advantageously, in the first motion compensation operation, an affine sub-block motion
vector field for motion compensation is generated by using a larger sub-block size.
[0023] Advantageously, the method further comprises:
selecting the control point motion vectors for the PU so that the resulting reference
block bounding box size does not exceed one or more predetermined thresholds.
[0024] According to an aspect, a non-transitory, computer-readable storage medium is provided
storing instructions that, when executed on one or more processors, control the one
or more processors to perform a method of video encoding or decoding, comprising:
inputting digital video;
performing an encoding or decoding of the digital video which has been input; and
outputting the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.
[0025] Advantageously, the non-transitory, computer-readable storage medium further comprises:
calculating the reference block bounding box size by determining coordinates of an
upper-left and bottom-right corner of the reference block bounding box of a plurality
of consecutive sub-block vectors of the PU, and calculating a width and height of
the reference block bounding box based on the coordinates.
BRIEF DESCRIPTION OF THE DRAWINGS
[0026] One or more aspects of the disclosed subject matter will be set forth in detail with
reference to the drawings, in which:
[0027] The present application claims the benefit of United States Provisional Patent Application
Nos.
62/757,004, filed November 7, 2018;
62/769,875, filed November 20, 2018, and
62/792,195, filed January 14, 2019, whose disclosures are hereby incorporated by reference in their entireties into
the present disclosure.
Fig. 1A is a flow chart showing an overview of one or more aspects of the disclosed
subject matter;
Fig. 1B illustrates an affine motion model according to one or more aspects of the
disclosed subject matter;
Fig. 1C depicts a 4-parameter affine motion model used in the VVC according to one
or more aspects of the disclosed subject matter;
Fig. ID depicts a 6-parameter affine motion model used in the VVC according to one
or more aspects of the disclosed subject matter;
Fig. 2 shows 4 (i.e. 2x2) sub-block motion vectors in a PU coded in affine mode and
bidirectional inter prediction;
Fig. 3 shows a reference block bounding box of 4 sub-block motion vectors in a PU
coded in affine mode and bi-directional inter prediction;
Fig. 4 shows a reference block bounding box of 2 sub-block motion vectors in a PU
coded in affine mode and unidirectional inter prediction (horizontal direction);
Fig. 5 shows a reference block bounding box of 2 sub-block motion vectors in a PU
coded in affine mode and unidirectional inter prediction (vertical direction);
Fig. 6 is an algorithmic flow chart of a method for controlling memory bandwidth consumption
of the affine mode according to one or more aspects of the disclosed subject matter;
Fig. 7 is an algorithmic flow chart of a method for controlling memory bandwidth consumption
of the affine mode according to one or more aspects of the disclosed subject matter;
Fig. 8 is an algorithmic flow chart of a method for controlling the sub-block motion
vector spread in the affine motion estimation on the encoder side according to one
or more aspects of the disclosed subject matter;
Fig. 9 is an algorithmic flow chart of a method for determining block size according
to one or more aspects of the disclosed subject matter; and
Fig. 10 is a block diagram showing an example of circuitry on which any of the preferred
embodiments, or any other embodiment, can be implemented.
DETAILED DESCRIPTION
[0028] The description set forth in detail with reference to the drawings, in which like
reference numerals refer to like elements or operations throughout.
[0029] Fig. 1A is a flow chart showing an overview of one or more aspects of the disclosed
subject matter. In step 102, the reference block bounding box size and at least one
threshold are determined. In step 104, it is determined whether the reference block
bounding box size exceeds the at least one threshold. If so, the affine motion compensation
is performed in step 106 using a first motion compensation operation. In response
to a determination that the reference block bounding size does not exceed the predefined
threshold, the affine motion compensation is performed in step 108 using a second
affine motion compensation operation that is different from the first motion compensation
operation. Either way, the results are output in step 110.
[0030] Fig. 1B shows an affine motion model according to one or more aspects of the disclosed
subject matter. Regarding the affine motion model, the origin (0, 0) of the x-y coordinate
system can be at the top-left corner point of a picture as illustrated in Fig. 1B.
Similarly, other drawings in the description having an x-y coordinate system can also
have the origin (0, 0) of the x-y coordinate at the top-left corner of a picture.
Generally, a PU coded in affine mode and affine merge mode can have uni-prediction
or bi-directional prediction. The uni-prediction can correspond to a list 0 or list
1 prediction, while the bi-directional prediction can correspond to a list 0 and list
1 bi-prediction. Although various algorithmic descriptions further described herein
focus on the uni-prediction mode to explain the algorithm, it should be appreciated
that if a PU is coded in bi-directional affine or bi-directional affine merge mode,
the process of affine mode and affine merge mode described herein are performed separately
for list 0 and list 1 predictions. In the affine motion model, the motion vector
v = (
vx,vy) at a sample position (
x,y) inside a PU is defined as follows:
where
a, b, c, d, e, f are the affine motion model parameters, which define a 6-parameter affine motion
model (see Fig. ID).
[0031] Fig. 1C shows a 4-parameter affine motion model according to one or more aspects
of the disclosed subject matter. The 4-parameter affine motion model can be a restricted
affine motion model because the parameters are restricted. For example, the 4-parameter
model can be described with four parameters by restricting
a =
d and
b =
-c in Equation 1:
[0032] In the 4-parameter affine motion model, the model parameters
a, b, e, f are determined by signaling two control point vectors at the top-left and top-right
corner of a PU. Fig. 1C shows two control point vectors
v0 = (
v0x,v0y) at sample position (
x0,
y0) and
v1 = (
v1x,
v1y) at sample positional(
x1,
y1). Accordingly, Equation 2 can be rewritten as:
[0033] It should be appreciated that in Fig. 1C, (
x1 -
x0) is equal to the PU width and
y1 =
y0. Accordingly, the two control point vectors do not have to be at the top-left and
top-right corner of a PU to derive the parameters of the 4-parameter affine motion
model. As long as the two control points have
x1 ≠
x0 and
y1 =
y0, Equation 3 is valid.
[0034] Fig. ID shows a 6-parameter affine motion model according to one or more aspects
of the disclosed subject matter. The model parameters for the 6-parameter affine motion
model can be determined by signaling three control point vectors at the top-left,
top-right, and bottom-left corner of a PU. For example, with three control point vectors
v0 = (
v0x, v0y) at sample position (
x0,y0),
v1 = (
v1x, v1y) at sample positional(
x1,
y1) and
v2 = (
v2x, v2y) at sample position (
x2,
y2), Equation 1 can be rewritten as:
[0035] It should be appreciated that in Fig. ID, (
x1 -
x0) is equal to the PU width, (
y2 -
y0) is equal to the PU height,
y1 =
y0, and
x2 =
x0. Accordingly, the three control point vectors do not have to be at the top-left,
top-right and bottom-left corner of a PU as shown in Fig. ID to derive the parameters
of the 6-parameter affine motion model. As long as the three control points have
x1 ≠
x0,
y2 ≠
y0, y1 =
y0 and
x2 =
x0, Equation 4 is valid.
[0036] Further, to constrain the memory bandwidth consumption of the affine mode for motion
compensation, the motion vectors of a PU coded in affine mode are not derived for
each sample in a PU. For example, as shown in Fig. 1C and Fig. ID, all the samples
inside a sub-block (e.g. 4x4 block size) of the PU can share a same motion vector.
Deriving the motion vector can be based on a sub-block motion data derivation process
of the affine mode where the motion vector is derived at sample position (
x,y) chosen for the sub-block and by using Equation 3 or Equation 4, and whether to use
Equation 3 or Equation 4 depends on the type of affine motion model. In the current
VVC design, the sub-block size is fixed to 4x4 and the sample position chosen for
deriving the sub-block motion vector field of a PU coded in affine mode is the center
point of each 4x4 sub-block of the PU.
[0037] The concepts of determining the reference block bounding size will be described.
Fig. 2 shows 4 (i.e. 2x2) sub-block motion vectors in a PU coded in affine mode and
bi-directional inter prediction and depicts the geometric relationship of 4 sub-block
vectors in a PU coded in affine mode and bi-directional inter prediction. The 4 sub-block
vectors (with sub-block size
m ∗ n) can be any 4 sub-block vectors within the PU whose locations satisfy the following
conditions:
[0039] The parameters of the affine motion model, i.e. (
a, b, c d), can be calculated in any suitable way such as using Equation 3 or 4.
[0041] Based on the coordinates listed in Table 1, the coordinates of upper-left and bottom-right
corners of the reference block bounding box in Fig. 3, i.e. (
xbul,ybul) and (
xbbr,ybbr), are defined as:
where max() and min() are functions used to return the largest and the smallest value
from a set of data, respectively.
[0042] By using Equations 6 and 7, the width and height of the reference block bounding
box, i.e. (
bxW4
, bxH4) can be computed by:
[0043] Fig. 4 shows a reference block bounding box of 2 sub-block motion vectors in a PU
coded in affine mode and unidirectional inter prediction (horizontal direction). The
width and height of the reference block bounding box (
bxWh, bxHh) for 2 sub-block motion vectors in a PU, which is coded in affine mode and unidirectional
inter prediction, can be computed by:
[0044] If the PU coded in affine mode uses unidirectional inter prediction, the reference
block bounding box can also be drawn in the vertical direction. Fig. 5 shows a reference
block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and
unidirectional inter prediction (vertical direction). As shown in Fig. 5, in this
case, the reference block bounding box size, i.e. (
bxWv, bxHv), can be computed by (see also Table 1 sub-block 0 and 2):
[0045] From Equation 12, Equation 13 and Equation 14, it can be seen that the reference
block bounding block size is independent of sub-block locations inside the PU; it
purely depends on the parameters of the affine motion model (i.e.
a, b, c, d)
, sub-block size (i.e.
m ∗
n) and filter tap lengths (i.e.
fx ∗ fy) used for motion compensation.
[0046] In the current VVC design (JVET-P2001), the sub-block size used for the affine mode
is 4x4 (i.e.
m =
n = 4), and the filter tap used for luma motion compensation of the affine mode is
6x6 (i.e.
fx =
fy = 6). The reference block bounding box sizes for the VVC are defined in Equations
15, 16 and 17.
[0047] Now that the above concepts of reference block bounding box computation have been
explained, various aspects of the disclosed subject matter using some or all of these
concepts will be further described herein.
[0048] Fig. 6 is an algorithmic flow chart of a method for controlling memory bandwidth
consumption of the affine mode. Based on the CPMVs (control point motion vectors)
received for the PU, the decoder computes the reference block bounding box size and
switches to the fallback mode (from the affine mode) if the reference blocking box
size exceeds the pre-defined thresholds. In this variant, the reference block bounding
box is computed for 2x2 sub-block vectors if the PU uses bidirectional affine mode,
and for 2x1 and 1x2 sub-block vectors if the PU uses unidirectional affine mode. In
one implementation, the following steps may be applied:
- 1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, f) are computed in step 602.
- 2. If it is determined in step 604 that the PU uses bi-directional affine mode (i.e.
bi-pred), the reference block bounding box size bxW4 ∗ bxH4 of 2x2 sub-block vectors is computed in step 606 by:
where m ∗ n is the sub-block size, and fx ∗ fy is the filter tap size used in the luma motion compensation.
- 3. If the PU uses unidirectional affine mode (i.e. uni-pred), the reference block
bounding box size bxWh ∗ bxHh of 2x1 sub-block vectors and the reference block bounding box size bxWv ∗ bxHv of 1x2 sub-block vectors are computed in step 608 by:
where m ∗ n is the sub-block size, and fx ∗ fy is the filter tap size used in the luma motion compensation of the affine mode.
- 4. Thresholds Thredb, Thredh and Thredv may be set to values defined by:
where δx ∗ δy > 0 defines the margin for controlling the memory bandwidth consumption.
- 5. If the PU uses bi-directional affine mode (i.e. bi-pred) and it is determined in
step 610 that bxW4 ∗ bxH4 ≤ Thredb or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined
in step 612 that both bxWh ∗ bxHh ≤ Thredh, bxWv ∗ bxHv ≤ Thredv , the sub-block motion vectors of the PU, v = (vx,vy) at a sample position (x,y), are generated in step 614 by using the affine motion model:
where (x,y) for a sub-block vector can be the center location of the sub-block.
- 6. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred)
and bxW4 ∗ bxH4 > Thredb or if the PU uses unidirectional affine mode (i.e. uni-pred) and either bxWh ∗ bxHh > Thredh or bxWv ∗ bxHv > Thredv, the fallback mode is triggered, and in a corresponding one of steps 616 and 618,
the sub-block motion vectors of the PU, v = (vx,vy) at a sample position (x,y), are set to a same motion vector. For example,
where (x0,y0) is the coordinate of the center point of the PU. (x0,y0) can be set to other locations of the PU. For example, if (x0,y0) is set to the coordinate of the top-left corner of the PU, then all the sub-block
vectors v = (vx,vy) of the PU are actually set to the control point motion vector of the PU at the top-left
PU corner location.
By setting all the sub-block vectors to a same vector, the reference block bounding
box size of 2x2, 2x1 and 1x2 sub-block vectors in the fallback mode is (fx + 2m - 1) ∗ (fy + 2n - 1), (fx + 2m - 1) ∗ (fy + n - 1) and (fx + m - 1) ∗ (fy + 2n - 1), respectively, which is guaranteed to be smaller than the pre-defined thresholds
Thredb, Thredh and Thredv.
- 7. The generated sub-block vectors are passed to the motion compensation and the rest
of the decoder processing in step 620.
[0049] In the affine mode of the current VVC design, the sub-block size is fixed to 4x4
(i.e.
m =
n = 4) and the filter-tap is fixed to 8x8 (i.e.
fx = fy = 6). If
δx and
δy are set to
δx =
δy = 2, thresholds
Thredb, Thredh and
Thredv become
Which means the memory bandwidth consumption of the affine mode controlled by the
algorithm described in Fig. 6 won't exceed the worst case memory bandwidth consumption
of the HEVC, which uses 8-tap interpolation filters for the motion compensation of
8x8 bidirectional PUs and 8x4/4x8 unidirectional PUs. The reference block size for
an 8x8 directional PU using 8-tap interpolation filters is 15
∗15 for both the list0 and list1 prediction, and the reference block size for an 8x4/4x8
unidirectional PU is 15
∗11/11
∗15 for list0 or list1 prediction. Those values match the thresholds set in Equation
19.
[0050] Fig. 7 is an algorithmic flow chart of a method for controlling memory bandwidth
consumption of the affine mode. Based on the CPMVs (control point motion vectors)
received for the PU, the decoder computes the reference block bounding box size and
switches to the fallback mode (from the affine mode) if the reference blocking box
size exceeds the pre-defined thresholds. In this embodiment, the reference block bounding
box is always computed for 2x2 sub-block vectors independent of the PU prediction
type (unidirectional or bidirectional prediction). In one implementation, the following
steps may be applied:
- 1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, f) are computed in step 702.
- 2. Compute the reference block bounding box size bxW4 ∗ bxH4 of 2x2 sub-block vectors in step 704 by
where m ∗ n is the sub-block size, and fx ∗ fy is the filter tap size used in the luma motion compensation.
- 3. Thresholds Thredb and Thredu may be both set to (fx + 2m - 1 + δx) ∗ (fy + 2n - 1 + δy), or set to different values based on prediction type (unidirectional prediction
or bidirectional), such as
where δx ∗ δy > 0 defines the margin for controlling the memory bandwidth consumption.
- 4. If it is determined in step 706 the PU uses bi-directional affine mode (i.e. bi-pred)
and it is determined in step 708 that bxW4 ∗ bxH4 ≤ Thredb or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined
in step 710 that bxW4 ∗ bxH4 ≤ Thredu, the sub-block motion vectors of the PU, v = (vx,vy) at a sample position (x,y), are generated in step 712 by using the affine motion model:
where (x,y) for a sub-block vector can be the center location of the sub-block.
- 5. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred)
and bxW4 ∗ bxH4 > Thredb or if the PU uses unidirectional affine mode (i.e. uni-pred) and bxW4 ∗ bxH4 > Thredu, the fallback mode is triggered in a corresponding one of steps 714 and 716, and the
sub-block motion vectors of the PU, v = (vx,vy) at a sample position (x,y), are set to a same motion vector. For example,
where (x0,y0) is the coordinate of the center point of the PU. (x0,y0) can be set to other locations of the PU. For example, if (x0,y0) is set to the coordinate of the top-left corner of the PU, then all the sub-block
vectors v = (vx,vy) of the PU are actually set to the control point motion vector of the PU at the top-left
PU corner location.
By setting all the sub-block vectors to a same vector, the reference block bounding
box size of 2x2 sub-block vectors in the fallback mode is (fx + 2m - 1) ∗ (fy + 2n - 1), which is guaranteed to be smaller than the pre-defined threshold values of
Thredb and Thredu.
- 6. The generated sub-block vectors are passed to the motion compensation and the rest
of the decoder processing in step 718.
[0051] It should be appreciated that in the bi-directional affine mode, a PU has both list0
and list1 predictions. In the disclosed subject matter, the reference bounding box
size
bxW4 ∗
bxH4 is computed independently for list0 and list1 prediction with the respective list0/list1
affine motion model parameters (a, b, c, d) of the PU, and the threshold
Thredb is set separately for list0 and list1 prediction (though the values of the threshold
could be the same). With the memory bandwidth control algorithms described above,
the following four combinations are possible for a PU coded in bi-directional affine
mode: 1) the regular sub-block motion vector fields are used for both the list0 and
list1 motion compensation of the PU; 2) the regular sub-block motion vector field
is used for list0 motion compensation but the fallback mode (i.e. a single vector
for list1 prediction of the entire PU) is used for list1 motion compensation; 3) the
fallback mode (i.e. a single vector for list0 prediction of the entire PU) is used
for list0 motion compensation but the regular sub-block motion vector field is used
for list1 motion compensation; and 4) the fallback mode (i.e. a first single vector
for list0 prediction and a second single vector for list1 prediction of the entire
PU) is used for both list0 and list1 motion compensation.
[0052] Fig. 8 is an algorithmic flow chart of a method for controlling the sub-block motion
vector spread in the affine motion estimation on the encoder side. The basic idea
is to constrain the sub-block motion vector spread at PU level. The sub-block motion
vector spread can be measured by using the reference block bounding box size. During
the encoder decision process, the encoder guarantees that the affine mode, if it is
selected for a PU, has the sub-block motion vector spread constrained to meet the
pre-defined budget of the worst-case memory bandwidth consumption at PU level. During
the affine motion estimation in which a set of candidate CPMVs (control point motion
vectors) is evaluated for the affine mode of the PU, the following steps may be applied:
- 1. Based on the CPMVs of a candidate point, the affine motion model parameters (a, b, c, d, e, f) are computed in step 802.
- 2. If the PU uses bi-directional affine mode (i.e. bi-pred), as determined in step
804, the reference bounding box size bxW4 ∗ bxH4 is computed in step 806 by using Equation 12. Otherwise, if the PU uses unidirectional
affine mode, the reference bounding box size bxWh ∗ bxHh and bxWv ∗ bxHv are computed in step 808 by using Equation 13 and Equation 14, respectively.
- 3. To control the sub-block motion vector spread at PU level, the current candidate
point is skipped for cost evaluation in the following cases:
- a. If, as determined in step 810, the PU uses bi-directional affine mode (i.e. bi-pred)
and bxW4 ∗ bxH4 > Thred4.
- b. Or if, as determined in step 812, the PU uses uni-directional affine mode and either
bxWh ∗ bxHh > Thredh or bxWv * bxHv > Thredv, where Thred4, Thredh and Thredv may be pre-defined by using Equation 18.
- 4. Otherwise, the cost of the current candidate point is computed in step 814 by using
the affine motion model parameters (a, b, c, d, e, f), the reference block data and the original PU data.
- 5. The cost of the best match point is then updated in step 816.
- 6. Repeat steps 802 through 816 to loop through all the candidate points for the PU
to obtain the estimated CPMVs for the PU that deliver the best rate-distortion cost.
[0053] The estimated affine CPMVs for the PU, if available, may be further evaluated against
the costs of regular motion vectors and intra prediction mode estimated for the PU
to decide whether the current PU should be encoded in the affine mode.
[0054] By using the proposed method, the worst-case memory bandwidth consumption for the
affine mode is restricted not only at sub-block level but also at PU level. Note that
the sub-block motion vector spread within a PU of affine mode is determined by the
affine parameters (
a, b, c, d).
[0055] It should be appreciated that in in step 810 the decision on whether the reference
bounding box size exceeds the pre-defined threshold is done independently for the
list0 and list1 prediction of a PU coded in bi-directional affine mode.
[0056] The restriction can also be imposed by a bitstream constraint. For example, the bitstream
constraint can be specified as follows:
A bitstream conforming to the VVC standard shall satisfy the following conditions:
- 1. If the PU uses bi-directional affine mode (i.e. bi-pred), the reference block bounding
box size bxW4 ∗ bxH4 is less than or equal to Thred4.
- 2. Otherwise, if the PU uses uni-directional affine mode, the reference block bounding
box size bxWh ∗ bxHh and bxWv ∗ bxHv is less than or equal to Thredh and hredv, respectively.
[0057] The implementation depicted in Fig. 8 just exhibits one example of the embodiment;
other variants can be implemented. For example:
- 1. Instead of using the bounding boxes in both the horizontal direction (i.e. bxWh, bxHh and Thredh) and the vertical direction (i.e. bxWv, bxHv and Thredv) in the case of unidirectional affine mode, the bounding box either in the horizontal
or in the vertical direction may be used.
- 2. Instead of computing the threshold and bounding box size for 4 sub-block vectors
(2x2 consecutive sub-block vectors) in the case of bi-directional affine mode, and
for 2 sub-block vectors (in horizontal and/or vertical direction) in the case of unidirectional
affine mode, the threshold and bounding box size may be computed for other numbers
of consecutive sub-block vectors with a PU.
- 3. The constraint may be imposed independent of the prediction type (i.e. bi-directional
or unidirectional affine mode). In one implementation, the constraint may be simply
expressed as a bitstream conforming to the VVC standard shall stratify that the reference
block bounding box size bxW4 ∗ bxH4 is less than or equal to Thred4.
- 4. Instead of using the fixed sub-block size m ∗ n (e. g. 4x4), the sub-block size may be further made adaptive to impose additional constraints
at the sub-block level.
[0058] Fig. 9 is an algorithmic flow chart of a method for determining block size. In step
902, the reference block bounding box size and the at least one threshold are determined.
In step 904, the reference block bounding box size and the at least one threshold
are compared. If the reference block bounding size exceeds the at least one threshold,
then in step 906, the sub-block size
m' ∗
n' (or as appropriate,
m' ∗
n or
m ∗
n', with m' > m,
and n' >
n) is used for the generation of the affine sub-block motion vector field and for affine
motion compensation. Otherwise, in step 908, the sub-block size m
∗n is used. Any combination of bi-directional, horizontal unidirectional and vertical
unidirectional modes can be used. The results are output in step 910.
[0059] For example, the memory bandwidth control algorithm can be modified based on the
shape of the PU coded in the affine mode:
- 1. If the PU coded in affine mode uses bi-directional inter prediction and the reference
block bounding box size of four sub-block vectors exceeds a pre-defined threshold,
i.e. bxW4 ∗ bxH4 > Thred4, then sub-block size m' ∗ n' (with m' > m, and n' > n) is used for the generation of the sub-block motion data field for the motion compensation
of the PU. Otherwise, sub-block size m ∗ n is used.
- 2. Otherwise, if the PU coded in affine mode uses unidirectional inter prediction,
and if the reference block bounding box size of two sub-block vectors exceeds a pre-defined
threshold in both horizontal and vertical directions, i.e. bxWh ∗ bxHh > Thredh and bxWv ∗ bxHv > Thredv, then sub-block size m' ∗ n (with m' > m) is used if PU width is larger than PU height, or sub-block size m ∗ n' (with n'
> n) is used PU width is less than or equal to PU height.
[0060] Otherwise, sub-block size
m ∗
n is used for the generation of the sub-block motion data field for the motion compensation
of the PU.
[0061] In another variation, instead of adaptively selecting the sub-block size based on
the size of reference block bounding box, the selection may be based on the width
and/or height of the reference block bounding box, and/or based on the DDR burst size
and alignments, or based on any combinations of above mentioned parameters (i.e. size,
with and height of the reference block bounding box, the DDR burst size and alignments,
and etc.).
[0062] It should be appreciated that in the disclosed matter the bitsteam constraints and/or
sub-block size adaptation are done independently for the list0 and list1 prediction
of a PU coded in bi-directional affine mode.
[0063] In another variation, the sub-block vectors of affine mode used for motion compensation
may be the same as or different from the ones used for (affine) merge/AMVP list derivation
(used as spatial neighboring candidates), for de-blocking filter and for storage of
temporal motion vectors (TMVPs). For example, the sub-block motion vectors of the
affine mode for motion compensation may be generated by the algorithm that adaptively
selects sub-block sizes (e.g. 8x8/8x4 adaptively), while the sub-block motion vectors
of the affine mode for (affine) merge/AMVP list derivation (used as spatial neighboring
candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs)
may be generated by using a fixed sub-block size (e.g. 4x4). In another example, the
sub-block motion vectors of the affine mode for motion compensation, for (affine)
merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking
filter and for storage of temporal motion vectors (TMVPs) may be generated by the
algorithm described herein that adaptively selects sub-block sizes (e.g. 8x8/8x4 adaptively).
[0064] A hardware description of a computer/device (e.g., the image processing device 1000)
according to exemplary embodiments, or any other embodiment, which is used to encode
and/or decode video is described with reference to Fig. 10. In Fig. 10, the image
processing device 1000 includes a CPU 1002 which performs one or more of the processes
described above. The process data and instructions may be stored in memory 1004, which
may be transitory, non-transitory, or a combination of the two. The video (data) to
be encoded/decoded can original from any source such as an external memory, a network
1006, or other location/device. These processes and instructions may also be stored
on a storage medium disk 1008 such as a hard drive (HDD) or portable storage medium
or may be stored remotely. Further, the claimed advancements are not limited by the
form of the computer-readable media on which the instructions of the inventive process
are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory,
RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device
with which the image processing device 1000 communicates, such as an image processing
device or computer.
[0065] Further, the claimed advancements may be provided as a utility application, background
daemon, or component of an operating system, or combination thereof, executing in
conjunction with CPU 1002 and an operating system such as Microsoft Windows, UNIX,
Solaris, LINUX, Apple MAC-OS and other suitable operating system.
[0066] The image processing device 1000 may be a general-purpose computer or a particular,
special-purpose machine. In one embodiment, the image processing device 1000 becomes
a particular, special-purpose machine when the processor 1002 is programmed to perform
network performance testing. The image processing device may be implemented as an
encoder, a decoder, or a device which both encodes and decodes images. The image processing
device can be implemented in a mobile phone, a laptop, a tablet, a general purpose
computer, a set-top box, a video decoding device such as an Amazon Fire TV Stick or
device, a Roku device, a television, a video monitor, a still camera, a video camera,
a scanner, a multifunction printer, an automobile display, or any desired device.
[0067] Alternatively, or additionally, the CPU 1002 may be implemented on an FPGA, ASIC,
PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize.
Further, CPU 1002 may be implemented as multiple processors cooperatively working
in parallel to perform the instructions of the inventive processes described above.
[0068] The image processing device 1000 in Fig. 10 also includes a network controller 1010,
such as an Intel Ethernet PRO network interface card from Intel Corporation of America,
for interfacing with network 1006. As can be appreciated, the network 1006 can be
a public network, such as the Internet, or a private network such as an LAN or WAN
network, or any combination thereof and can also include PSTN or ISDN sub-networks.
The network 1006 can also be wired, such as an Ethernet network, or can be wireless
such as a cellular network including EDGE, 3G and 4G wireless cellular systems. The
wireless network can also be WiFi, Bluetooth, or any other wireless form of communication
that is known.
[0069] The image processing device 1000 further includes a display controller 1012, such
as a graphics card or graphics adaptor for interfacing with display 1014, such as
a monitor. A general purpose I/O interface 1016 interfaces with a keyboard and/or
mouse 1018 as well as a touch screen panel 1020 on or separate from display 1014.
General purpose I/O interface also connects to a variety of peripherals 1022 including
printers and scanners.
[0070] A sound controller 1024 is also provided in the image processing device 1000 to interface
with speakers/microphone 1026 thereby providing sounds and/or music.
[0071] The general-purpose storage controller 1028 connects the storage medium disk 1008
with communication bus 1030, which may be an ISA, EISA, VESA, PCI, or similar, for
interconnecting all of the components of the image processing device 1000. A description
of the general features and functionality of the display 1014, keyboard and/or mouse
1018, as well as the display controller 1012, storage controller 1028, network controller
1010, sound controller 1024, and general purpose I/O interface 1016 is omitted herein
for brevity.
[0072] The exemplary circuit elements described in the context of the present disclosure
may be replaced with other elements and structured differently than the examples provided
herein. Moreover, circuitry configured to perform features described herein may be
implemented in multiple circuit units (e.g., chips), or the features may be combined
in circuitry on a single chipset. For that matter, any hardware and/or software capable
of implementing any of the above embodiments, or any other embodiment, can be used
instead of, or in addition to, what is disclosed above.
[0073] The functions and features described herein may also be executed by various distributed
components of a system. For example, one or more processors may execute these system
functions, wherein the processors are distributed across multiple components communicating
in a network. The distributed components may include one or more client and server
machines, which may share processing, in addition to various human interface and communication
devices (e.g., display monitors, smart phones, tablets, personal digital assistants
(PDAs)). The network may be a private network, such as a LAN or WAN, or may be a public
network, such as the Internet. Input to the system may be received via direct user
input and received remotely either in real-time or as a batch process. Additionally,
some implementations may be performed on modules or hardware not identical to those
described. Accordingly, other implementations are within the scope that may be claimed.
[0074] While preferred embodiments have been set forth above, those skilled in the art who
have reviewed the present disclosure will readily appreciate that other embodiments
can be realized within the scope of the disclosure. For example, disclosures of numerical
values and of specific technologies are illustrative rather than limiting. Also, whenever
technically feasible, features from different embodiments can be combined, and the
order in which operations are performed can be varied. Further, wherever technically
feasible, any feature disclosed herein can be used for encoding, decoding or both.
The one or more aspects of the disclosed subject matter are not limited to VVC implementations
and can be utilized with any video encoding/decoding system. Therefore, the one or
more aspects of the disclosed subject matter should be construed as limited only by
the appended claims.
1. A device for video encoding or decoding, comprising:
circuitry configured to
input digital video;
perform an encoding or decoding of a digital video which has been input; and output
the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.
2. The device of claim 1, wherein the circuitry is further configured to calculate the
reference block bounding box size by determining coordinates of an upper-left and
bottom-right corner of the reference block bounding box of a plurality of consecutive
sub-block vectors of the PU, and calculating a width and height of the reference block
bounding box based on the coordinates.
3. The device of claim 2, wherein the reference block bounding box size is calculated
based on a prediction type being unidirectional prediction or bidirectional prediction.
4. The device of claim 1, wherein the at least one predetermined threshold is set based
on a prediction type being unidirectional prediction or bidirectional prediction.
5. The device of claim 1, wherein the circuitry is further configured to
calculate the reference block bounding box size to determine whether the reference
block bounding size exceeds the at least one predetermined threshold, and
perform the first motion compensation operation or the second motion compensation
operation in response to a determination separately for list0 and list1 prediction
of the PU.
6. The device of claim 1, wherein, in the first motion compensation operation, all sub-block
vectors of the PU are set to a same vector, the same vector being an affine motion
vector for a single point in the PU.
7. The device of claim 1, wherein, in the second motion compensation operation, an affine
sub-block motion vector field for affine motion compensation is generated based on
a sub-block size.
8. The device of claim 1, wherein, in the first motion compensation operation, an affine
sub-block motion vector field for motion compensation is generated by using a larger
sub-block size.
9. The device of claim 1, wherein the circuitry is further configured to
select the control point motion vectors for the PU so that the resulting reference
block bounding box size does not exceed one or more predetermined thresholds.
10. A method of video encoding or decoding, comprising:
inputting digital video;
performing an encoding or decoding of the digital video which has been input; and
outputting the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.
11. The method of claim 10, further comprising:
calculating the reference block bounding box size by determining coordinates of an
upper-left and bottom-right corner of the reference block bounding box of a plurality
of consecutive sub-block vectors of the PU, and calculating a width and height of
the reference block bounding box based on the coordinates.
12. The method of claim 11, further comprising:
calculating the reference block bounding block size based on a prediction type being
unidirectional prediction or bidirectional prediction.
13. The method of claim 10, further comprising:
setting the at least one predetermined threshold based on a prediction type being
unidirectional prediction or bidirectional prediction.
14. The method of claim 10, further comprising:
calculating the reference block bounding box to
determine whether the reference block bounding size exceeds the at least one predetermined
threshold; and
perform the first motion compensation operation or the second motion compensation
operation in response to a determination separately for list0 and list1 prediction
of the PU.
15. A non-transitory, computer-readable storage medium storing instructions that, when
executed on one or more processors, control the one or more processors to perform
a method of video encoding or decoding, comprising:
inputting digital video;
performing an encoding or decoding of the digital video which has been input; and
outputting the digital video which has been encoded or decoded, wherein:
the encoding or decoding includes performing affine motion compensation in an affine
mode in which a prediction unit ("PU") of the digital video coded in the affine mode
uses inter prediction and a reference block bounding box size and determining whether
the reference block bounding size exceeds at least one predetermined threshold;
in response to a determination that the reference block bounding size exceeds the
at least one predetermined threshold, the affine motion compensation is performed
using a first motion compensation operation; and
in response to a determination that the reference block bounding size does not exceed
the at least one predetermined threshold, the affine motion compensation is performed
using a second motion compensation operation that is different from the first motion
compensation operation.