CONTROL OF MEMORY BANDWIDTH CONSUMPTION OF AFFINE MODE IN VERSATILE VIDEO CODING

Abstract

 Video encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds a predefined threshold. In response to a determination that the reference block bounding size exceeds the predefined threshold, the affine motion compensation is performed using a first motion compensation operation. In response to a determination that the reference block bounding size does not exceed the predefined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

Description

[0001] The present application claims the benefit of United States Provisional Patent Application Nos. 62/757,004, filed November 7, 2018; 62/769,875, filed November 20, 2018, and 62/792,195, filed January 14, 2019, whose disclosures are hereby incorporated by reference in their entireties into the present disclosure.

[0002] One or more aspects of the disclosed subject matter are directed to video encoding or decoding and more specifically to video encoding or decoding with memory bandwidth conservation when an affine motion model is used.

[0003] The VVC (Versatile Video Coding) is a new video compression standard being developed by the joint video experts team (JVET) jointly established by ISO/IEO MPEG and ITU-T. The VVC standard for single layer coding will be finalized by the end of 2020, with a design goal of being at least 50% more efficient than the previous standard MPEG HEVC/ITU-T H.265 Main-10 profile.

[0004] Among proposed coding tools to VVC under consideration, the affine motion compensation prediction introduces a more complex motion model for better compression efficiency. In previous standards such as HEVC, only a translational motion model is considered, in which all the sample positions inside a PU (prediction unit) may have a same translational motion vector for motion compensated prediction. However, in the real world, there are many kinds of motion, e.g., zoom in/out, rotation, perspective motions and other irregular motions. The affine motion model supports different motion vectors at different sample positions inside a PU, which effectively captures more complex motion. Different sample positions inside a PU, such as four corner points of the PU, may have different motion vectors as supported by the affine mode. A PU coded in affine mode and affine merge mode may have uni-prediction (list 0 or list 1 prediction) or bi-directional prediction (i.e. list 0 and list 1 bi-prediction).

[0005] In the current VVC design (see JVET-P2001, "Versatile Video Coding (Draft 7)"), the sub-block size for the affine mode is fixed to 4x4, which creates the 4x4 bi-directional prediction for the worst-case memory bandwidth consumption of motion compensation. In the HEVC, the worst-case memory bandwidth consumption for motion compensation is 8x8 bidirectional prediction, while 8x4 and 4x8 PUs use uni-prediction only. The increased memory bandwidth budget can never catch up with the path of sample rate increase (e.g., HEVC is typically for 4K video at 60 fps, while VVC will be used for 8K video at 60 fps, another factor of 4 increase in terms of sample processing rate).

[0006] According to an aspect, a device for video encoding or decoding is provided, comprising:

circuitry configured to

input digital video;

perform an encoding or decoding of a digital video which has been input; and output the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

[0007] Advantageously, the circuitry is further configured to calculate the reference block bounding box size by determining coordinates of an upper-left and bottom-right corner of the reference block bounding box of a plurality of consecutive sub-block vectors of the PU, and calculating a width and height of the reference block bounding box based on the coordinates.

[0008] Advantageously, the reference block bounding box size is calculated based on a prediction type being unidirectional prediction or bidirectional prediction.

[0009] Advantageously, the at least one predetermined threshold is set based on a prediction type being unidirectional prediction or bidirectional prediction.

[0010] Advantageously, the circuitry is further configured to
calculate the reference block bounding box size to determine whether the reference block bounding size exceeds the at least one predetermined threshold, and
perform the first motion compensation operation or the second motion compensation operation in response to a determination separately for list0 and list1 prediction of the PU.

[0011] Advantageously, in the first motion compensation operation, all sub-block vectors of the PU are set to a same vector, the same vector being an affine motion vector for a single point in the PU.

[0012] Advantageously, in the second motion compensation operation, an affine sub-block motion vector field for affine motion compensation is generated based on a sub-block size.

[0013] Advantageously, in the first motion compensation operation, an affine sub-block motion vector field for motion compensation is generated by using a larger sub-block size.

[0014] Advantageously, the circuitry is further configured to
select the control point motion vectors for the PU so that the resulting reference block bounding box size does not exceed one or more predetermined thresholds.

[0015] According to an aspect, a method of video encoding or decoding is provided, comprising:

inputting digital video;

performing an encoding or decoding of the digital video which has been input; and

outputting the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

[0016] Advantageously, the method further comprises:
calculating the reference block bounding box size by determining coordinates of an upper-left and bottom-right corner of the reference block bounding box of a plurality of consecutive sub-block vectors of the PU, and calculating a width and height of the reference block bounding box based on the coordinates.

[0017] Advantageously, the method further comprises:
calculating the reference block bounding block size based on a prediction type being unidirectional prediction or bidirectional prediction.

[0018] Advantageously, the method further comprises:
setting the at least one predetermined threshold based on a prediction type being unidirectional prediction or bidirectional prediction.

[0019] Advantageously, the method further comprises:
calculating the reference block bounding box to

determine whether the reference block bounding size exceeds the at least one predetermined threshold; and

perform the first motion compensation operation or the second motion compensation operation in response to a determination separately for list0 and fist1 prediction of the PU.

[0020] Advantageously, in the first motion compensation operation, all sub-block vectors of the PU are set to a same vector, the same vector being an affine motion vector for a single point in the PU.

[0021] Advantageously, in the second motion compensation operation, an affine sub-block motion vector field for affine motion compensation is generated based on a sub-block size.

[0022] Advantageously, in the first motion compensation operation, an affine sub-block motion vector field for motion compensation is generated by using a larger sub-block size.

[0023] Advantageously, the method further comprises:
selecting the control point motion vectors for the PU so that the resulting reference block bounding box size does not exceed one or more predetermined thresholds.

[0024] According to an aspect, a non-transitory, computer-readable storage medium is provided storing instructions that, when executed on one or more processors, control the one or more processors to perform a method of video encoding or decoding, comprising:

inputting digital video;

performing an encoding or decoding of the digital video which has been input; and

outputting the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

[0025] Advantageously, the non-transitory, computer-readable storage medium further comprises:
calculating the reference block bounding box size by determining coordinates of an upper-left and bottom-right corner of the reference block bounding box of a plurality of consecutive sub-block vectors of the PU, and calculating a width and height of the reference block bounding box based on the coordinates.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] One or more aspects of the disclosed subject matter will be set forth in detail with reference to the drawings, in which:

[0027] The present application claims the benefit of United States Provisional Patent Application Nos. 62/757,004, filed November 7, 2018; 62/769,875, filed November 20, 2018, and 62/792,195, filed January 14, 2019, whose disclosures are hereby incorporated by reference in their entireties into the present disclosure.

Fig. 1A is a flow chart showing an overview of one or more aspects of the disclosed subject matter;

Fig. 1B illustrates an affine motion model according to one or more aspects of the disclosed subject matter;

Fig. 1C depicts a 4-parameter affine motion model used in the VVC according to one or more aspects of the disclosed subject matter;

Fig. ID depicts a 6-parameter affine motion model used in the VVC according to one or more aspects of the disclosed subject matter;

Fig. 2 shows 4 (i.e. 2x2) sub-block motion vectors in a PU coded in affine mode and bidirectional inter prediction;

Fig. 3 shows a reference block bounding box of 4 sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction;

Fig. 4 shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (horizontal direction);

Fig. 5 shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (vertical direction);

Fig. 6 is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode according to one or more aspects of the disclosed subject matter;

Fig. 7 is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode according to one or more aspects of the disclosed subject matter;

Fig. 8 is an algorithmic flow chart of a method for controlling the sub-block motion vector spread in the affine motion estimation on the encoder side according to one or more aspects of the disclosed subject matter;

Fig. 9 is an algorithmic flow chart of a method for determining block size according to one or more aspects of the disclosed subject matter; and

Fig. 10 is a block diagram showing an example of circuitry on which any of the preferred embodiments, or any other embodiment, can be implemented.

DETAILED DESCRIPTION

[0028] The description set forth in detail with reference to the drawings, in which like reference numerals refer to like elements or operations throughout.

[0029] Fig. 1A is a flow chart showing an overview of one or more aspects of the disclosed subject matter. In step 102, the reference block bounding box size and at least one threshold are determined. In step 104, it is determined whether the reference block bounding box size exceeds the at least one threshold. If so, the affine motion compensation is performed in step 106 using a first motion compensation operation. In response to a determination that the reference block bounding size does not exceed the predefined threshold, the affine motion compensation is performed in step 108 using a second affine motion compensation operation that is different from the first motion compensation operation. Either way, the results are output in step 110.

[0030] Fig. 1B shows an affine motion model according to one or more aspects of the disclosed subject matter. Regarding the affine motion model, the origin (0, 0) of the x-y coordinate system can be at the top-left corner point of a picture as illustrated in Fig. 1B. Similarly, other drawings in the description having an x-y coordinate system can also have the origin (0, 0) of the x-y coordinate at the top-left corner of a picture. Generally, a PU coded in affine mode and affine merge mode can have uni-prediction or bi-directional prediction. The uni-prediction can correspond to a list 0 or list 1 prediction, while the bi-directional prediction can correspond to a list 0 and list 1 bi-prediction. Although various algorithmic descriptions further described herein focus on the uni-prediction mode to explain the algorithm, it should be appreciated that if a PU is coded in bi-directional affine or bi-directional affine merge mode, the process of affine mode and affine merge mode described herein are performed separately for list 0 and list 1 predictions. In the affine motion model, the motion vector v = (v_x,v_y) at a sample position (x,y) inside a PU is defined as follows:


where a, b, c, d, e, f are the affine motion model parameters, which define a 6-parameter affine motion model (see Fig. ID).

[0031] Fig. 1C shows a 4-parameter affine motion model according to one or more aspects of the disclosed subject matter. The 4-parameter affine motion model can be a restricted affine motion model because the parameters are restricted. For example, the 4-parameter model can be described with four parameters by restricting a = d and b = -c in Equation 1:

[0032] In the 4-parameter affine motion model, the model parameters a, b, e, f are determined by signaling two control point vectors at the top-left and top-right corner of a PU. Fig. 1C shows two control point vectors v₀ = (v_0x,v_0y) at sample position (x₀,y₀) and v₁ = (v_1x,v_1y) at sample positional(x₁,y₁). Accordingly, Equation 2 can be rewritten as:

[0033] It should be appreciated that in Fig. 1C, (x₁ - x₀) is equal to the PU width and y₁ = y₀. Accordingly, the two control point vectors do not have to be at the top-left and top-right corner of a PU to derive the parameters of the 4-parameter affine motion model. As long as the two control points have x₁ ≠ x₀ and y₁ = y₀, Equation 3 is valid.

[0034] Fig. ID shows a 6-parameter affine motion model according to one or more aspects of the disclosed subject matter. The model parameters for the 6-parameter affine motion model can be determined by signaling three control point vectors at the top-left, top-right, and bottom-left corner of a PU. For example, with three control point vectors v₀ = (v_0x, v_0y) at sample position (x₀,y₀), v₁ = (v_1x, v_1y) at sample positional(x₁, y₁) and v₂ = (v_2x, v_2y) at sample position (x₂, y₂), Equation 1 can be rewritten as:

[0035] It should be appreciated that in Fig. ID, (x₁ - x₀) is equal to the PU width, (y₂ - y₀) is equal to the PU height, y₁ = y₀, and x₂ = x₀. Accordingly, the three control point vectors do not have to be at the top-left, top-right and bottom-left corner of a PU as shown in Fig. ID to derive the parameters of the 6-parameter affine motion model. As long as the three control points have x₁ ≠ x₀, y₂ ≠ y₀, y₁ = y₀ and x₂ = x₀, Equation 4 is valid.

[0036] Further, to constrain the memory bandwidth consumption of the affine mode for motion compensation, the motion vectors of a PU coded in affine mode are not derived for each sample in a PU. For example, as shown in Fig. 1C and Fig. ID, all the samples inside a sub-block (e.g. 4x4 block size) of the PU can share a same motion vector. Deriving the motion vector can be based on a sub-block motion data derivation process of the affine mode where the motion vector is derived at sample position (x,y) chosen for the sub-block and by using Equation 3 or Equation 4, and whether to use Equation 3 or Equation 4 depends on the type of affine motion model. In the current VVC design, the sub-block size is fixed to 4x4 and the sample position chosen for deriving the sub-block motion vector field of a PU coded in affine mode is the center point of each 4x4 sub-block of the PU.

[0037] The concepts of determining the reference block bounding size will be described. Fig. 2 shows 4 (i.e. 2x2) sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction and depicts the geometric relationship of 4 sub-block vectors in a PU coded in affine mode and bi-directional inter prediction. The 4 sub-block vectors (with sub-block size m ^∗ n) can be any 4 sub-block vectors within the PU whose locations satisfy the following conditions:

[0038] By substituting Equation 5 into the affine motion model, the 4 sub-block vectors are be derived by

[0039] The parameters of the affine motion model, i.e. (a, b, c d), can be calculated in any suitable way such as using Equation 3 or 4.

[0040] For motion compensation, reference blocks are loaded around the co-located sub-block locations with the offsets determined by the sub-block motion vectors. Fig. 3 shows a reference block bounding box of 4 sub-block motion vectors in a PU coded in affine mode and bidirectional inter prediction. Assuming (x₀, y₀) are the coordinate of the sub-block 0 in Fig. 3 and all the sub-blocks in the PU have equal size m ^∗ n, and f_x ^∗ f_y are filter taps used in the motion compensation, the coordinates of upper-left and bottom-right locations of the reference blocks are listed in Table 1 below (using Equations 7 to 9):

Table 1: Coordinates of upper-left and bottom-right corners of reference blocks of four sub-block vectors

Sub-block	Coordinates of upper-left corner of the sub-block	Coordinates of upper-left corner of the reference block	Coordinates of bottom-right corner of the reference block
0	(x0,y0)
1	(x0 + m, y0)
2	(x0,y0 + n)
3	(x0 + m,y0 + n)

[0041] Based on the coordinates listed in Table 1, the coordinates of upper-left and bottom-right corners of the reference block bounding box in Fig. 3, i.e. (xb_ul,yb_ul) and (xb_br,yb_br), are defined as:




where max() and min() are functions used to return the largest and the smallest value from a set of data, respectively.

[0042] By using Equations 6 and 7, the width and height of the reference block bounding box, i.e. (bxW4, bxH4) can be computed by:

[0043] Fig. 4 shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (horizontal direction). The width and height of the reference block bounding box (bxW_h, bxH_h) for 2 sub-block motion vectors in a PU, which is coded in affine mode and unidirectional inter prediction, can be computed by:

[0044] If the PU coded in affine mode uses unidirectional inter prediction, the reference block bounding box can also be drawn in the vertical direction. Fig. 5 shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (vertical direction). As shown in Fig. 5, in this case, the reference block bounding box size, i.e. (bxW_v, bxH_v), can be computed by (see also Table 1 sub-block 0 and 2):

[0045] From Equation 12, Equation 13 and Equation 14, it can be seen that the reference block bounding block size is independent of sub-block locations inside the PU; it purely depends on the parameters of the affine motion model (i.e. a, b, c, d), sub-block size (i.e. m ∗ n) and filter tap lengths (i.e. f_x ∗ f_y) used for motion compensation.

[0046] In the current VVC design (JVET-P2001), the sub-block size used for the affine mode is 4x4 (i.e. m = n = 4), and the filter tap used for luma motion compensation of the affine mode is 6x6 (i.e. f_x = f_y = 6). The reference block bounding box sizes for the VVC are defined in Equations 15, 16 and 17.

[0047] Now that the above concepts of reference block bounding box computation have been explained, various aspects of the disclosed subject matter using some or all of these concepts will be further described herein.

[0048] Fig. 6 is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode. Based on the CPMVs (control point motion vectors) received for the PU, the decoder computes the reference block bounding box size and switches to the fallback mode (from the affine mode) if the reference blocking box size exceeds the pre-defined thresholds. In this variant, the reference block bounding box is computed for 2x2 sub-block vectors if the PU uses bidirectional affine mode, and for 2x1 and 1x2 sub-block vectors if the PU uses unidirectional affine mode. In one implementation, the following steps may be applied:

1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, f) are computed in step 602.

2. If it is determined in step 604 that the PU uses bi-directional affine mode (i.e. bi-pred), the reference block bounding box size bxW4 ∗ bxH4 of 2x2 sub-block vectors is computed in step 606 by:


where m ∗ n is the sub-block size, and f_x ∗ f_y is the filter tap size used in the luma motion compensation.

3. If the PU uses unidirectional affine mode (i.e. uni-pred), the reference block bounding box size bxW_h ∗ bxH_h of 2x1 sub-block vectors and the reference block bounding box size bxW_v ∗ bxH_v of 1x2 sub-block vectors are computed in step 608 by:




where m ∗ n is the sub-block size, and f_x ∗ f_y is the filter tap size used in the luma motion compensation of the affine mode.

4. Thresholds Thred_b, Thred_h and Thred_v may be set to values defined by:


where δ_x ∗ δ_y > 0 defines the margin for controlling the memory bandwidth consumption.

5. If the PU uses bi-directional affine mode (i.e. bi-pred) and it is determined in step 610 that bxW4 ∗ bxH4 ≤ Thred_b or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined in step 612 that both bxW_h ∗ bxH_h ≤ Thred_h, bxW_v ∗ bxH_v ≤ Thred_v , the sub-block motion vectors of the PU, v = (v_x,v_y) at a sample position (x,y), are generated in step 614 by using the affine motion model:


where (x,y) for a sub-block vector can be the center location of the sub-block.

6. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4 ∗ bxH4 > Thred_b or if the PU uses unidirectional affine mode (i.e. uni-pred) and either bxW_h ∗ bxH_h > Thred_h or bxW_v ∗ bxH_v > Thred_v, the fallback mode is triggered, and in a corresponding one of steps 616 and 618, the sub-block motion vectors of the PU, v = (v_x,v_y) at a sample position (x,y), are set to a same motion vector. For example,


where (x₀,y₀) is the coordinate of the center point of the PU. (x₀,y₀) can be set to other locations of the PU. For example, if (x₀,y₀) is set to the coordinate of the top-left corner of the PU, then all the sub-block vectors v = (v_x,v_y) of the PU are actually set to the control point motion vector of the PU at the top-left PU corner location.
By setting all the sub-block vectors to a same vector, the reference block bounding box size of 2x2, 2x1 and 1x2 sub-block vectors in the fallback mode is (f_x + 2m - 1) ∗ (f_y + 2n - 1), (f_x + 2m - 1) ∗ (f_y + n - 1) and (f_x + m - 1) ∗ (f_y + 2n - 1), respectively, which is guaranteed to be smaller than the pre-defined thresholds Thred_b, Thred_h and Thred_v.

7. The generated sub-block vectors are passed to the motion compensation and the rest of the decoder processing in step 620.

[0049] In the affine mode of the current VVC design, the sub-block size is fixed to 4x4 (i.e. m = n = 4) and the filter-tap is fixed to 8x8 (i.e. f_x = f_y = 6). If δ_x and δ_y are set to δ_x = δ_y = 2, thresholds Thred_b, Thred_h and Thred_v become

Which means the memory bandwidth consumption of the affine mode controlled by the algorithm described in Fig. 6 won't exceed the worst case memory bandwidth consumption of the HEVC, which uses 8-tap interpolation filters for the motion compensation of 8x8 bidirectional PUs and 8x4/4x8 unidirectional PUs. The reference block size for an 8x8 directional PU using 8-tap interpolation filters is 15^∗15 for both the list0 and list1 prediction, and the reference block size for an 8x4/4x8 unidirectional PU is 15^∗11/11^∗15 for list0 or list1 prediction. Those values match the thresholds set in Equation 19.

[0050] Fig. 7 is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode. Based on the CPMVs (control point motion vectors) received for the PU, the decoder computes the reference block bounding box size and switches to the fallback mode (from the affine mode) if the reference blocking box size exceeds the pre-defined thresholds. In this embodiment, the reference block bounding box is always computed for 2x2 sub-block vectors independent of the PU prediction type (unidirectional or bidirectional prediction). In one implementation, the following steps may be applied:

1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, f) are computed in step 702.

2. Compute the reference block bounding box size bxW4 ∗ bxH4 of 2x2 sub-block vectors in step 704 by



where m ∗ n is the sub-block size, and f_x ∗ f_y is the filter tap size used in the luma motion compensation.

3. Thresholds Thred_b and Thred_u may be both set to (f_x + 2m - 1 + δ_x) ∗ (f_y + 2n - 1 + δ_y), or set to different values based on prediction type (unidirectional prediction or bidirectional), such as


where δ_x ∗ δ_y > 0 defines the margin for controlling the memory bandwidth consumption.

4. If it is determined in step 706 the PU uses bi-directional affine mode (i.e. bi-pred) and it is determined in step 708 that bxW4 ∗ bxH4 ≤ Thred_b or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined in step 710 that bxW4 ∗ bxH4 ≤ Thred_u, the sub-block motion vectors of the PU, v = (v_x,v_y) at a sample position (x,y), are generated in step 712 by using the affine motion model:


where (x,y) for a sub-block vector can be the center location of the sub-block.

5. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4 ∗ bxH4 > Thred_b or if the PU uses unidirectional affine mode (i.e. uni-pred) and bxW4 ∗ bxH4 > Thred_u, the fallback mode is triggered in a corresponding one of steps 714 and 716, and the sub-block motion vectors of the PU, v = (v_x,v_y) at a sample position (x,y), are set to a same motion vector. For example,


where (x₀,y₀) is the coordinate of the center point of the PU. (x₀,y₀) can be set to other locations of the PU. For example, if (x₀,y₀) is set to the coordinate of the top-left corner of the PU, then all the sub-block vectors v = (v_x,v_y) of the PU are actually set to the control point motion vector of the PU at the top-left PU corner location.
By setting all the sub-block vectors to a same vector, the reference block bounding box size of 2x2 sub-block vectors in the fallback mode is (f_x + 2m - 1) ∗ (f_y + 2n - 1), which is guaranteed to be smaller than the pre-defined threshold values of Thred_b and Thred_u.

6. The generated sub-block vectors are passed to the motion compensation and the rest of the decoder processing in step 718.

[0051] It should be appreciated that in the bi-directional affine mode, a PU has both list0 and list1 predictions. In the disclosed subject matter, the reference bounding box size bxW4 ∗ bxH4 is computed independently for list0 and list1 prediction with the respective list0/list1 affine motion model parameters (a, b, c, d) of the PU, and the threshold Thred_b is set separately for list0 and list1 prediction (though the values of the threshold could be the same). With the memory bandwidth control algorithms described above, the following four combinations are possible for a PU coded in bi-directional affine mode: 1) the regular sub-block motion vector fields are used for both the list0 and list1 motion compensation of the PU; 2) the regular sub-block motion vector field is used for list0 motion compensation but the fallback mode (i.e. a single vector for list1 prediction of the entire PU) is used for list1 motion compensation; 3) the fallback mode (i.e. a single vector for list0 prediction of the entire PU) is used for list0 motion compensation but the regular sub-block motion vector field is used for list1 motion compensation; and 4) the fallback mode (i.e. a first single vector for list0 prediction and a second single vector for list1 prediction of the entire PU) is used for both list0 and list1 motion compensation.

[0052] Fig. 8 is an algorithmic flow chart of a method for controlling the sub-block motion vector spread in the affine motion estimation on the encoder side. The basic idea is to constrain the sub-block motion vector spread at PU level. The sub-block motion vector spread can be measured by using the reference block bounding box size. During the encoder decision process, the encoder guarantees that the affine mode, if it is selected for a PU, has the sub-block motion vector spread constrained to meet the pre-defined budget of the worst-case memory bandwidth consumption at PU level. During the affine motion estimation in which a set of candidate CPMVs (control point motion vectors) is evaluated for the affine mode of the PU, the following steps may be applied:

1. Based on the CPMVs of a candidate point, the affine motion model parameters (a, b, c, d, e, f) are computed in step 802.

2. If the PU uses bi-directional affine mode (i.e. bi-pred), as determined in step 804, the reference bounding box size bxW4 ∗ bxH4 is computed in step 806 by using Equation 12. Otherwise, if the PU uses unidirectional affine mode, the reference bounding box size bxW_h ∗ bxH_h and bxW_v ∗ bxH_v are computed in step 808 by using Equation 13 and Equation 14, respectively.

3. To control the sub-block motion vector spread at PU level, the current candidate point is skipped for cost evaluation in the following cases:

a. If, as determined in step 810, the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4 ∗ bxH4 > Thred4.

b. Or if, as determined in step 812, the PU uses uni-directional affine mode and either bxW_h ∗ bxH_h > Thred_h or bxW_v * bxH_v > Thred_v, where Thred4, Thred_h and Thred_v may be pre-defined by using Equation 18.

4. Otherwise, the cost of the current candidate point is computed in step 814 by using the affine motion model parameters (a, b, c, d, e, f), the reference block data and the original PU data.

5. The cost of the best match point is then updated in step 816.

6. Repeat steps 802 through 816 to loop through all the candidate points for the PU to obtain the estimated CPMVs for the PU that deliver the best rate-distortion cost.

[0053] The estimated affine CPMVs for the PU, if available, may be further evaluated against the costs of regular motion vectors and intra prediction mode estimated for the PU to decide whether the current PU should be encoded in the affine mode.

[0054] By using the proposed method, the worst-case memory bandwidth consumption for the affine mode is restricted not only at sub-block level but also at PU level. Note that the sub-block motion vector spread within a PU of affine mode is determined by the affine parameters (a, b, c, d).

[0055] It should be appreciated that in in step 810 the decision on whether the reference bounding box size exceeds the pre-defined threshold is done independently for the list0 and list1 prediction of a PU coded in bi-directional affine mode.

[0056] The restriction can also be imposed by a bitstream constraint. For example, the bitstream constraint can be specified as follows:
A bitstream conforming to the VVC standard shall satisfy the following conditions:

1. If the PU uses bi-directional affine mode (i.e. bi-pred), the reference block bounding box size bxW4 ∗ bxH4 is less than or equal to Thred4.

2. Otherwise, if the PU uses uni-directional affine mode, the reference block bounding box size bxW_h ∗ bxH_h and bxW_v ∗ bxH_v is less than or equal to Thred_h and hred_v, respectively.

[0057] The implementation depicted in Fig. 8 just exhibits one example of the embodiment; other variants can be implemented. For example:

1. Instead of using the bounding boxes in both the horizontal direction (i.e. bxWh, bxH_h and Thred_h) and the vertical direction (i.e. bxW_v, bxH_v and Thred_v) in the case of unidirectional affine mode, the bounding box either in the horizontal or in the vertical direction may be used.

2. Instead of computing the threshold and bounding box size for 4 sub-block vectors (2x2 consecutive sub-block vectors) in the case of bi-directional affine mode, and for 2 sub-block vectors (in horizontal and/or vertical direction) in the case of unidirectional affine mode, the threshold and bounding box size may be computed for other numbers of consecutive sub-block vectors with a PU.

3. The constraint may be imposed independent of the prediction type (i.e. bi-directional or unidirectional affine mode). In one implementation, the constraint may be simply expressed as a bitstream conforming to the VVC standard shall stratify that the reference block bounding box size bxW4 ∗ bxH4 is less than or equal to Thred4.

4. Instead of using the fixed sub-block size m ∗ n (e. g. 4x4), the sub-block size may be further made adaptive to impose additional constraints at the sub-block level.

[0058] Fig. 9 is an algorithmic flow chart of a method for determining block size. In step 902, the reference block bounding box size and the at least one threshold are determined. In step 904, the reference block bounding box size and the at least one threshold are compared. If the reference block bounding size exceeds the at least one threshold, then in step 906, the sub-block size m' ∗ n' (or as appropriate, m' ∗ n or m ∗ n', with m' > m, and n' > n) is used for the generation of the affine sub-block motion vector field and for affine motion compensation. Otherwise, in step 908, the sub-block size m^∗n is used. Any combination of bi-directional, horizontal unidirectional and vertical unidirectional modes can be used. The results are output in step 910.

[0059] For example, the memory bandwidth control algorithm can be modified based on the shape of the PU coded in the affine mode:

1. If the PU coded in affine mode uses bi-directional inter prediction and the reference block bounding box size of four sub-block vectors exceeds a pre-defined threshold, i.e. bxW4 ∗ bxH4 > Thred₄, then sub-block size m' ∗ n' (with m' > m, and n' > n) is used for the generation of the sub-block motion data field for the motion compensation of the PU. Otherwise, sub-block size m ∗ n is used.

2. Otherwise, if the PU coded in affine mode uses unidirectional inter prediction, and if the reference block bounding box size of two sub-block vectors exceeds a pre-defined threshold in both horizontal and vertical directions, i.e. bxW_h ∗ bxH_h > Thred_h and bxW_v ∗ bxH_v > Thred_v, then sub-block size m' ∗ n (with m' > m) is used if PU width is larger than PU height, or sub-block size m ∗ n' (with n' > n) is used PU width is less than or equal to PU height.

[0060] Otherwise, sub-block size m ∗ n is used for the generation of the sub-block motion data field for the motion compensation of the PU.

[0061] In another variation, instead of adaptively selecting the sub-block size based on the size of reference block bounding box, the selection may be based on the width and/or height of the reference block bounding box, and/or based on the DDR burst size and alignments, or based on any combinations of above mentioned parameters (i.e. size, with and height of the reference block bounding box, the DDR burst size and alignments, and etc.).

[0062] It should be appreciated that in the disclosed matter the bitsteam constraints and/or sub-block size adaptation are done independently for the list0 and list1 prediction of a PU coded in bi-directional affine mode.

[0063] In another variation, the sub-block vectors of affine mode used for motion compensation may be the same as or different from the ones used for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs). For example, the sub-block motion vectors of the affine mode for motion compensation may be generated by the algorithm that adaptively selects sub-block sizes (e.g. 8x8/8x4 adaptively), while the sub-block motion vectors of the affine mode for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs) may be generated by using a fixed sub-block size (e.g. 4x4). In another example, the sub-block motion vectors of the affine mode for motion compensation, for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs) may be generated by the algorithm described herein that adaptively selects sub-block sizes (e.g. 8x8/8x4 adaptively).

[0064] A hardware description of a computer/device (e.g., the image processing device 1000) according to exemplary embodiments, or any other embodiment, which is used to encode and/or decode video is described with reference to Fig. 10. In Fig. 10, the image processing device 1000 includes a CPU 1002 which performs one or more of the processes described above. The process data and instructions may be stored in memory 1004, which may be transitory, non-transitory, or a combination of the two. The video (data) to be encoded/decoded can original from any source such as an external memory, a network 1006, or other location/device. These processes and instructions may also be stored on a storage medium disk 1008 such as a hard drive (HDD) or portable storage medium or may be stored remotely. Further, the claimed advancements are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the image processing device 1000 communicates, such as an image processing device or computer.

[0065] Further, the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 1002 and an operating system such as Microsoft Windows, UNIX, Solaris, LINUX, Apple MAC-OS and other suitable operating system.

[0066] The image processing device 1000 may be a general-purpose computer or a particular, special-purpose machine. In one embodiment, the image processing device 1000 becomes a particular, special-purpose machine when the processor 1002 is programmed to perform network performance testing. The image processing device may be implemented as an encoder, a decoder, or a device which both encodes and decodes images. The image processing device can be implemented in a mobile phone, a laptop, a tablet, a general purpose computer, a set-top box, a video decoding device such as an Amazon Fire TV Stick or device, a Roku device, a television, a video monitor, a still camera, a video camera, a scanner, a multifunction printer, an automobile display, or any desired device.

[0067] Alternatively, or additionally, the CPU 1002 may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU 1002 may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes described above.

[0068] The image processing device 1000 in Fig. 10 also includes a network controller 1010, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with network 1006. As can be appreciated, the network 1006 can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN sub-networks. The network 1006 can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless form of communication that is known.

[0069] The image processing device 1000 further includes a display controller 1012, such as a graphics card or graphics adaptor for interfacing with display 1014, such as a monitor. A general purpose I/O interface 1016 interfaces with a keyboard and/or mouse 1018 as well as a touch screen panel 1020 on or separate from display 1014. General purpose I/O interface also connects to a variety of peripherals 1022 including printers and scanners.

[0070] A sound controller 1024 is also provided in the image processing device 1000 to interface with speakers/microphone 1026 thereby providing sounds and/or music.

[0071] The general-purpose storage controller 1028 connects the storage medium disk 1008 with communication bus 1030, which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the image processing device 1000. A description of the general features and functionality of the display 1014, keyboard and/or mouse 1018, as well as the display controller 1012, storage controller 1028, network controller 1010, sound controller 1024, and general purpose I/O interface 1016 is omitted herein for brevity.

[0072] The exemplary circuit elements described in the context of the present disclosure may be replaced with other elements and structured differently than the examples provided herein. Moreover, circuitry configured to perform features described herein may be implemented in multiple circuit units (e.g., chips), or the features may be combined in circuitry on a single chipset. For that matter, any hardware and/or software capable of implementing any of the above embodiments, or any other embodiment, can be used instead of, or in addition to, what is disclosed above.

[0073] The functions and features described herein may also be executed by various distributed components of a system. For example, one or more processors may execute these system functions, wherein the processors are distributed across multiple components communicating in a network. The distributed components may include one or more client and server machines, which may share processing, in addition to various human interface and communication devices (e.g., display monitors, smart phones, tablets, personal digital assistants (PDAs)). The network may be a private network, such as a LAN or WAN, or may be a public network, such as the Internet. Input to the system may be received via direct user input and received remotely either in real-time or as a batch process. Additionally, some implementations may be performed on modules or hardware not identical to those described. Accordingly, other implementations are within the scope that may be claimed.

[0074] While preferred embodiments have been set forth above, those skilled in the art who have reviewed the present disclosure will readily appreciate that other embodiments can be realized within the scope of the disclosure. For example, disclosures of numerical values and of specific technologies are illustrative rather than limiting. Also, whenever technically feasible, features from different embodiments can be combined, and the order in which operations are performed can be varied. Further, wherever technically feasible, any feature disclosed herein can be used for encoding, decoding or both. The one or more aspects of the disclosed subject matter are not limited to VVC implementations and can be utilized with any video encoding/decoding system. Therefore, the one or more aspects of the disclosed subject matter should be construed as limited only by the appended claims.

Claims

1. A device for video encoding or decoding, comprising:

circuitry configured to

input digital video;

perform an encoding or decoding of a digital video which has been input; and output the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

2. The device of claim 1, wherein the circuitry is further configured to calculate the reference block bounding box size by determining coordinates of an upper-left and bottom-right corner of the reference block bounding box of a plurality of consecutive sub-block vectors of the PU, and calculating a width and height of the reference block bounding box based on the coordinates.

3. The device of claim 2, wherein the reference block bounding box size is calculated based on a prediction type being unidirectional prediction or bidirectional prediction.

4. The device of claim 1, wherein the at least one predetermined threshold is set based on a prediction type being unidirectional prediction or bidirectional prediction.

5. The device of claim 1, wherein the circuitry is further configured to
calculate the reference block bounding box size to determine whether the reference block bounding size exceeds the at least one predetermined threshold, and
perform the first motion compensation operation or the second motion compensation operation in response to a determination separately for list0 and list1 prediction of the PU.

6. The device of claim 1, wherein, in the first motion compensation operation, all sub-block vectors of the PU are set to a same vector, the same vector being an affine motion vector for a single point in the PU.

7. The device of claim 1, wherein, in the second motion compensation operation, an affine sub-block motion vector field for affine motion compensation is generated based on a sub-block size.

8. The device of claim 1, wherein, in the first motion compensation operation, an affine sub-block motion vector field for motion compensation is generated by using a larger sub-block size.

9. The device of claim 1, wherein the circuitry is further configured to
select the control point motion vectors for the PU so that the resulting reference block bounding box size does not exceed one or more predetermined thresholds.

10. A method of video encoding or decoding, comprising:

inputting digital video;

performing an encoding or decoding of the digital video which has been input; and

outputting the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

11. The method of claim 10, further comprising:
calculating the reference block bounding box size by determining coordinates of an upper-left and bottom-right corner of the reference block bounding box of a plurality of consecutive sub-block vectors of the PU, and calculating a width and height of the reference block bounding box based on the coordinates.

12. The method of claim 11, further comprising:
calculating the reference block bounding block size based on a prediction type being unidirectional prediction or bidirectional prediction.

13. The method of claim 10, further comprising:
setting the at least one predetermined threshold based on a prediction type being unidirectional prediction or bidirectional prediction.

14. The method of claim 10, further comprising:
calculating the reference block bounding box to

determine whether the reference block bounding size exceeds the at least one predetermined threshold; and

perform the first motion compensation operation or the second motion compensation operation in response to a determination separately for list0 and list1 prediction of the PU.

15. A non-transitory, computer-readable storage medium storing instructions that, when executed on one or more processors, control the one or more processors to perform a method of video encoding or decoding, comprising:

inputting digital video;

performing an encoding or decoding of the digital video which has been input; and

outputting the digital video which has been encoded or decoded, wherein:

the encoding or decoding includes performing affine motion compensation in an affine mode in which a prediction unit ("PU") of the digital video coded in the affine mode uses inter prediction and a reference block bounding box size and determining whether the reference block bounding size exceeds at least one predetermined threshold;

in response to a determination that the reference block bounding size exceeds the at least one predetermined threshold, the affine motion compensation is performed using a first motion compensation operation; and

in response to a determination that the reference block bounding size does not exceed the at least one predetermined threshold, the affine motion compensation is performed using a second motion compensation operation that is different from the first motion compensation operation.

Drawing