Decoders shall produce output frames that are identical in all respects and have the same output order as those produced by the decoding process specified herein.
The input to this process is a sequence of open bitstream units (OBUs).
The output from this process is a sequence of decoded frames.
For each OBU in turn the syntax elements are extracted as specified in [section 5.1][].
The syntax tables include function calls indicating when the remaining block decode processes are triggered.
This process is triggered by a call to decode_frame from within the syntax tables.
If show_existing_frame is equal to 0, the process first performs any post processing filtering by the following ordered steps:
-
If loop_filter_level[ 0 ] is not equal to 0 or loop_filter_level[ 1 ] is not equal to 0, the loop filter process specified in [section 7.11][] is invoked once the coded frame has been decoded.
-
The CDEF process specified in [section 7.12][] is invoked.
-
The upscaling process specified in [section 7.13][] is invoked with CdefFrame as input and the output is assigned to UpscaledCdefFrame.
-
The upscaling process specified in [section 7.13][] is invoked with CurrFrame as input and the output is assigned to UpscaledCurrFrame.
-
The loop restoration process specified in [section 7.14][] is invoked.
-
The reference frame update process as specified in [section 7.16][] is invoked.
If show_existing_frame is equal to 1 or show_frame is equal to 1, the output process as specified in [section 7.15][] is invoked.
Note: Although we specify that all samples in CurrFrame are upscaled, at most 2 lines above and below each stripe (defined by StripeStartY and StripeEndY) will end up being read. Implementations may wish to avoid upscaling the unused lines. {:.alert .alert-info }
This section describes additional constraints on a bitstream that result from the choice of frame types.
If a coded video sequence starts with a key frame, then the additional constraints are automatically satisfied.
Otherwise, the sequence can start with a number of non key frames and extra care has to be taken to ensure that the decoding process is well defined. It is a requirement of bitstream conformance that the following applies:
- When ref_frame_idx[ i ] is decoded, it must identify an initialized reference frame, i.e. there must have been an earlier decoded frame within the same coded video sequence which had (refresh_frame_flags >> ref_frame_idx[ i ])&1 equal to 1.
Note: This constraint means that the first coded frame cannot be an inter frame. {:.alert .alert-info }
A bitstream conforming to this specification consists of one or more coded video sequences.
A coded video sequence consists of one or more temporal units. A temporal unit consists of a series of OBUs starting from a temporal delimiter, optional sequence headers, optional metadata OBUs, a sequence of one or more frame headers, each followed by zero or more tile group OBUs as well as optional padding OBUs.
A frame header and its associated tile group OBUs within a temporal unit must use the same value of obu_extension_flag (i.e., either both include or both not include the optional OBU extension header).
The value of temporal_id must be the same in all OBU extension headers that are contained in the same temporal unit.
If a video sequence contains at least one enhancement layer (OBUs with ehancement_id greater than 0) then all frame headers and tile group OBUs associated with base (enhancement_id equals 0) and enhancement layer (enhancement_id greater than 0) data must include the OBU extension header.
OBUs with enhancement level IDs (enhancement_id) greater than 0 must appear within a temporal unit in increasing order of the enhancement level ID values.
Temporal units containing key frames must contain a sequence header before the frame header. If multiple sequence headers are contained in a temporal unit containing a key frame then they must all indicate identical parameter values.
Sequence header OBUs may appear in any order within an OBU sequence. The values of the parameters must be identical each time the sequence header appears.
Note: A bitstream can change the sequence header parameters during decode, but this is interpreted as a video consisting of multiple independent coded video sequences. {:.alert .alert-info }
One or more metadata and padding OBUs may appear in any order within an OBU sequence (unless constrained by semantics provided elsewhere in this specification). Specific metadata types may be required or recommended to be placed in specific locations, as identified in their corresponding definitions.
OBU types that are not defined in this specification can be ignored by a decoder.
This process is triggered when the function update_cdf is called from the tile group syntax table.
The frame CDF arrays are set equal to the final tile CDF arrays for the largest tile (measured in bytes) as follows.
A copy is made of the saved CDF values for each of the CDF arrays mentioned in the semantics for setup_past_independence. The name of the destination for the copy is the name of the CDF array with no prefix. The name of the source for the copy is the name of the CDF array prefixed with "Saved". For example, the array YModeCdf will be updated with values equal to the contents of SavedYModeCdf.
This process is triggered during the call to load_previous.
A linear projection model is employed to create a motion field estimation that is able to capture high velocity temporal motion trajectories.
The motion field is estimated based on the saved motion vectors from the reference frames and the relative frame distances.
As the frame distances depend on the frame being referenced, a separate motion field is estimated for each reference frame used by the current frame.
A stack of up to three motion vectors (for each reference frame type) is prepared at each location on an 8x8 pixel grid.
The variable w8 (representing the width of the motion field in units of 8x8 pixels) is set equal to MiCols >> 1.
The variable h8 (representing the height of the motion field in units of 8x8 pixels) is set equal to MiRows >> 1.
As the linear projection can create a field with holes, the motion fields are initialized to an invalid motion vector of -32768, -32768 as follows:
for ( ref = LAST_FRAME; ref <= ALTREF_FRAME; ref++ )
for ( y = 0; y < h8 ; y++ )
for ( x = 0; x < w8; x++ )
for ( i = 0; i < MFMV_STACK_SIZE; i++ )
for ( j = 0; j < 2; j++ )
MotionFieldMvs[ ref ][ y ][ x ][ i ][ j ] = -1 << 15The variable refStamp (representing where in the stack to place projected motion vectors) is set equal to MFMV_STACK_SIZE - 1.
The variable lastIdx (representing which reference frame is used for LAST_FRAME) is set equal to ref_frame_idx[ 0 ].
The variable curGoldOrderHint (representing the expected output order for GOLDEN_FRAME of the current frame) is set equal to OrderHints[ GOLDEN_FRAME ].
The variable lastAltOrderHint (representing the expected output order for ALTREF_FRAME of LAST_FRAME) is set equal to SavedOrderHints[ lastIdx ][ ALTREF_FRAME ].
The variable useLast (representing whether to project the motion vectors from LAST_FRAME) is set equal to ( lastAltOrderHint != curGoldOrderHint ).
If useLast is equal to 1, the projection process in [section 7.6.1][] is invoked with src equal to LAST_FRAME, dir equal to 1, and refStamp also passed as an input.
The variable refStamp is set equal to MFMV_STACK_SIZE - 2.
Note: refStamp is decreased even if the projection process is not invoked. {:.alert .alert-info }
The variable useBwd is set equal to ( OrderHints[ BWDREF_FRAME ] > OrderHint ).
If useBwd is equal to 1, the projection process in [section 7.6.1][] is invoked with src equal to BWDREF_FRAME, dir equal to 0, and refStamp also passed as an input.
If useBwd is equal to 1, refStamp is set equal to refStamp - 1.
The variable useAlt2 is set equal to ( OrderHints[ ALTREF2_FRAME ] > OrderHint ).
If useAlt2 is equal to 1, the projection process in [section 7.6.1][] is invoked with src equal to ALTREF2_FRAME, dir equal to 0, and refStamp also passed as an input.
If useAlt2 is equal to 1, refStamp is set equal to refStamp - 1.
The variable useAlt is set equal to ( OrderHints[ ALTREF_FRAME ] > OrderHint ).
If useAlt is equal to 1 and (refStamp >= 0), the projection process in [section 7.6.1][] is invoked with src equal to ALTREF_FRAME, dir equal to 0, and refStamp also passed as an input.
The inputs to this process are:
-
a variable src specifying which reference frame's motion vectors should be projected,
-
a variable refStamp specifying which entry in the stack to store the motion vectors,
-
a variable dir specifying whether the forward (dir equal to 0) or backward motion vectors in the reference frame should be used.
The process projects the motion vectors from a whole reference frame (which will always be one of the backward reference types) and stores the results in MotionFieldMvs.
The variable srcIdx (representing which reference frame is used) is set equal to ref_frame_idx[ src - LAST_FRAME ].
The variable w8 (representing the width of the motion field in units of 8x8 pixels) is set equal to MiCols >> 1.
The variable h8 (representing the height of the motion field in units of 8x8 pixels) is set equal to MiRows >> 1.
If RefMiRows[ srcIdx ] is not equal to MiRows, RefMiCols[ srcIdx ] is not equal to MiCols, or RefFrameType[ srcIdx ] is equal to INTRA_ONLY_FRAME, the process exits at this point.
The process is specified as follows:
for ( y8 = 0; y8 < h8; y8++ ) {
for ( x8 = 0; x8 < w8; x8++ ) {
row = 2 * y8 + 1
col = 2 * x8 + 1
srcRef = SavedRefFrames[ srcIdx ][ row ][ col ][ dir ]
if ( srcRef > INTRA_FRAME ) {
mv = SavedMvs[ srcIdx ][ row ][ col ][ dir ]
projMv = get_mv_projection( mv, srcRef, src, src, dir, 1 )
posValid = get_block_position( y8, x8, projMv )
if ( posValid ) {
for ( dst = LAST_FRAME; dst <= ALTREF_FRAME; dst++ ) {
projMv = get_mv_projection( mv, srcRef, src, dst, dir, -1 )
MotionFieldMvs[ dst ][ PosY8 ][ PosX8 ][ refStamp ] = projMv
}
}
}
}
}When the function get_mv_projection is called, the get mv projection process specified in [section 7.6.2][] is invoked and the output assigned to projMv.
When the function get_block_position is called, the get block position process specified in [section 7.6.3][] is invoked and the output assigned to posValid. This process also sets up the variables PosY8 and PosX8 representing the projected location in the motion field.
The inputs to this process are:
-
a length two array mv specifying a motion vector,
-
a variable ref specifying the reference frame associated with the motion vector,
-
a variable src specifying which reference frame's motion vectors should be projected,
-
a variable dst specifying which reference frame to target when adjusting the motion vector.
-
a variable dir specifying whether the forward (dir equal to 0) or backward motion vectors in the reference frame are being used,
-
a variable dstSign specifying whether to negate the motion vector direction before projection.
The outputs of this process are:
- a length 2 array projMv containing a projected motion vector
This process starts with a motion vector mv from a previous frame (src). This motion vector gives the displacement expected when moving a certain number of frames (between src and ref). We wish to use the motion vector for predictions using a different reference frame (dst), so we need to scale the length of the motion vector.
The variable srcIdx (representing the reference frame where we obtained the motion vector) is set equal to ref_frame_idx[ src - LAST_FRAME ].
The variable srcOrderHint (representing the expected output order of the reference frame where we obtained the motion vector) is set equal to OrderHints[ src ].
The variable refOrderHint (representing the expected output order of the frame referenced by the motion vector) is set equal to SavedOrderHints[ srcIdx ][ ref ].
The variable sign is set equal to ( dir ? -1 : 1 ).
The variable mvOffset (representing the number of frames covered by the motion vector) is set equal to Max( 1, sign * (srcOrderHint - refOrderHint) ).
The variable dstOrderHint (representing the expected output order of the reference frame we wish to use) is set equal to OrderHints[ dst ].
The variable dstOffset (representing the number of frames that we wish the motion vector to cover) is set equal to dstSign * sign * (dstOrderHint - OrderHint).
It is a requirement of bitstream conformance that mvOffset is in the range 0 to 63. It is a requirement of bitstream conformance that dstOffset is in the range -31 to 31.
The projected motion vector is specified as follows:
for ( i = 0; i < 2; i++ )
projMv[ i ] = Round2Signed( mv[ i ] * dstOffset * Div_Mult[ mvOffset ], 14 )where Div_Mult is a constant lookup table specified as:
Div_Mult[64] = {
0, 16384, 8192, 5461, 4096, 3276, 2730, 2340, 2048, 1820, 1638,
1489, 1365, 1260, 1170, 1092, 1024, 963, 910, 862, 819, 780,
744, 712, 682, 655, 630, 606, 585, 564, 546, 528, 512,
496, 481, 468, 455, 442, 431, 420, 409, 399, 390, 381,
372, 364, 356, 348, 341, 334, 327, 321, 315, 309, 303,
297, 292, 287, 282, 277, 273, 268, 264, 260,
}The inputs to this process are:
-
variables x8 and y8 specifying a location in units of 8x8 pixels,
-
a length two array projMv specifying a projected motion vector,
The process generates global variables PosX8 and PosY8 representing the projected location in units of 8x8 pixels.
The process returns a flag posValid that indicates if the position should be used.
Note: posValid is specified such that only blocks within a certain distance of the current location need to be projected. {:.alert .alert-info }
The variable posValid is set equal to 1.
The variable PosY8 is set equal to project(y8, projMv[ 0 ], MiRows >> 1, MAX_OFFSET_HEIGHT).
The variable PosX8 is set equal to project(x8, projMv[ 1 ], MiCols >> 1, MAX_OFFSET_WIDTH).
where the function project is specified as follows:
project( v8, delta, max8, maxOff8 ) {
base8 = (v8 >> 3) << 3
if (delta >= 0) {
offset8 = delta >> ( 3 + 1 + MI_SIZE_LOG2 )
} else {
offset8 = -( ( -delta ) >> ( 3 + 1 + MI_SIZE_LOG2 ) )
}
v8 += offset8
if ( v8 < 0 ||
v8 >= max8 ||
v8 <= base8 - maxOff8 ||
v8 >= base8 + 8 + maxOff8 ) {
posValid = 0
}
return v8
}The project function clears posValid if the resulting position is offset too far.
The following sections define the processes used for predicting the motion vectors.
These processes are triggered at points defined by function calls to find_mv_list, compound_context, and find_mv_stack in the inter block mode info syntax described in [section 5.8.22][].
The input to this process is a variable refList specifying which set of motion vectors to predict.
This process is triggered by a function call to find_mv_list and adds additional entries to the array RefListMv[ refList ] (up to a maximum of 2 motion vector candidates).
The Find MV Stack process is run before this and will already have found motion vectors based on candidates with exactly matching reference frames and placed them into RefListMv. This process now fills up the list by searching for non-exact candidates and scaling the motion vectors appropriately.
The variable bw8 (representing the width of the block in units of 8 pixels) is set equal to Num_8x8_Blocks_Wide[ MiSize ].
The variable bh8 (representing the height of the block in units of 8 pixels) is set equal to Num_8x8_Blocks_High[ MiSize ].
An array offMvs8 (representing offsets in units of 8 pixels to search locations) is defined as:
offMvs8[ 9 ][ 2 ] = {
{ bh8 - 1, -1 },
{ -1, bw8 - 1 },
{ -1, (bw8 - 1) >> 1 },
{ (bh8 - 1) >> 1, -1 },
{ -1, -1 },
{ -1, ( bw8 == bh8 ) ? 0 : bw8 },
{ ( bw8 == bh8 ) ? 0 : bh8, -1 },
{ -1, -3 },
{ bh8 - 1, -3 }
}For i = 0..8 the spatial motion vector candidate process in [section 7.7.1.2][] is invoked with mv8 equal to offMvs8[ i ] and refList as inputs.
The input to this process is a variable refList specifying which set of motion vectors to predict.
The output is a motion vector mv representing global motion for this block.
The variable ref (specifying the reference frame) is set equal to RefFrame[ refList ].
The variable typ (specifying the type of global motion) is set equal to gm_type[ ref ].
The variable bw (representing the width of the block in units of pixels) is set equal to Block_Width[ MiSize ].
The variable bh (representing the height of the block in units of pixels) is set equal to Block_Height[ MiSize ].
The output motion vector mv is specified by projecting the central pixel of the block as follows:
if (ref == INTRA_FRAME || typ == IDENTITY) {
mv[0] = 0
mv[1] = 0
} else if (typ == TRANSLATION) {
mv[0] = gm_params[ref][0] >> (WARPEDMODEL_PREC_BITS - 3)
mv[1] = gm_params[ref][1] >> (WARPEDMODEL_PREC_BITS - 3)
} else {
x = MiCol * MI_SIZE + bw / 2 - 1
y = MiRow * MI_SIZE + bh / 2 - 1
xc = (gm_params[ref][2] - (1 << WARPEDMODEL_PREC_BITS)) * x +
gm_params[ref][3] * y +
gm_params[ref][0]
yc = gm_params[ref][4] * x +
(gm_params[ref][5] - (1 << WARPEDMODEL_PREC_BITS)) * y +
gm_params[ref][1]
if (allow_high_precision_mv) {
mv[0] = Round2Signed(xc, WARPEDMODEL_PREC_BITS - 3)
mv[1] = Round2Signed(yc, WARPEDMODEL_PREC_BITS - 3)
} else {
mv[0] = Round2Signed(xc, WARPEDMODEL_PREC_BITS - 2) * 2
mv[1] = Round2Signed(yc, WARPEDMODEL_PREC_BITS - 2) * 2
}
}
lower_mv_precision( mv )where the call to lower_mv_precision invokes the lower precision process specified in [section 7.7.3.9][].
The inputs to this process are:
-
a 2 element array mv8 specifying the offset in 8 pixel units to the candidate location,
-
a variable refList specifying which set of motion vectors to predict.
This process examines the motion vector at a position in the current frame and adds it to the candidate list if certain conditions are met.
The candidate location in 4x4 pixel units is derived as:
-
the variable mvRow is set equal to MiRow + 2 * mv8[0]
-
the variable mvCol is set equal to MiCol + 2 * mv8[1]
The variable size (representing the size of the superblock in units of 4x4 pixels) is set as follows:
-
If use_128x128_superblock is equal to 1, size is set equal to Num_4x4_Blocks_Wide[ BLOCK_128X128 ].
-
Otherwise (use_128x128_superblock is equal to 0), size is set equal to Num_4x4_Blocks_Wide[ BLOCK_64X64 ].
This process terminates immediately if any of the following are true:
-
the function is_inside( mvRow, mvCol ) is equal to 0 (this tests if the candidate is in a different tile from which we are not allowed to retrieve motion vectors)
-
(MiRow % size) + 2 * mv8[0] is greater than or equal to size (this tests if the candidate comes from the superblock row below)
-
(MiCol % size) + 2 * mv8[1] is greater than or equal to size (this tests if the candidate comes from the superblock column to the right)
-
RefFrames[ mvRow ][ mvCol ][ 0 ] has not yet been written for this frame (this tests if the candidate location has not been decoded yet)
-
IsInters[ mvRow ][ mvCol ] is equal to 0 (this tests if the candidate is for an inter coded block)
For list=0..1 the following ordered steps apply:
- The variable candRef is set equal to RefFrames[ mvRow ][ mvCol ][ list ]
- The 2 element array candMv is set to a copy of Mvs[ mvRow ][ mvCol ][ list ]
- Invoke the candidate add process in [section 7.7.1.3][] with candRef, candMv, refList as inputs.
The inputs to this process are:
-
a 2 element array candMv specifying the candidate motion vector,
-
a variable candRef specifying the candidate reference frame,
-
a variable refList specifying which set of motion vectors to predict.
This process terminates immediately if candRef is not equal to RefFrame[refList].
This process terminates immediately if NumMvFound is greater than or equal to 2 (this means we have already found enough motion vector candidates).
If RefFrameSignBias[ candRef ] is not equal to RefFrameSignBias[ RefFrame[refList] ], then the candidate motion vector is inverted as follows:
- candMv[ 0 ] *= -1
- candMv[ 1 ] *= -1
The variable bw (representing the width of the block rounded up to be a multiple of 8) is set equal to 8 * Num_8x8_Blocks_Wide[ MiSize ].
The variable bh (representing the height of the block rounded up to be a multiple of 8) is set equal to 8 * Num_8x8_Blocks_High[ MiSize ].
The motion vector is clamped to a legal range as follows:
- candMv[ 0 ] is set equal to clamp_mv_row( candMv[ 0 ], MV_BORDER + bh * 8)
- candMv[ 1 ] is set equal to clamp_mv_col( candMv[ 1 ], MV_BORDER + bw * 8)
If NumMvFound is equal to 0 or candMv is not equal to RefListMv[ refList ][ 0 ], then the following ordered steps apply:
- RefListMv[ refList ][ NumMvFound ] is set equal to candMv
- NumMvFound is set equal to NumMvFound + 1
This process is triggered by a function call to compound_context and constructs a variable CompoundModeContext that is used for decoding the compound_mode syntax element.
The variable bw8 (representing the width of the block in units of 8 pixels) is set equal to Num_8x8_Blocks_Wide[ MiSize ].
The variable bh8 (representing the height of the block in units of 8 pixels) is set equal to Num_8x8_Blocks_High[ MiSize ].
An array offMvs8 (representing offsets in units of 8 pixels to search locations) is defined as:
offMvs8[ 2 ][ 2 ] = {
{bh8 - 1, -1},
{-1, bw8 - 1}
}The CompoundModeContext is constructed by testing the mode of the neighbouring blocks at the top-right and bottom-left corners as follows:
counter = 0
for (i = 0; i < 2; i++) {
mvRow = MiRow + 2 * offMvs8[i][0]
mvCol = MiCol + 2 * offMvs8[i][1]
if (is_inside(mvCol, mvRow)) {
counter += Mode_To_Counter[ YModes[ mvRow ][ mvCol ] ]
}
}
CompoundModeContext = Counter_To_Context[counter]The Mode_To_Counter table is defined as:
Mode_To_Counter[ MB_MODE_COUNT ] = {
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 0, 0, 3, 1
}The Counter_To_Context table is defined as:
Counter_To_Context[ 19 ] = {
BOTH_PREDICTED,
NEW_PLUS_NON_INTRA,
BOTH_NEW,
ZERO_PLUS_PREDICTED,
NEW_PLUS_NON_INTRA,
INVALID_CASE,
BOTH_ZERO,
INVALID_CASE,
INVALID_CASE,
INTRA_PLUS_NON_INTRA,
INTRA_PLUS_NON_INTRA,
INVALID_CASE,
INTRA_PLUS_NON_INTRA,
INVALID_CASE,
INVALID_CASE,
INVALID_CASE,
INVALID_CASE,
INVALID_CASE,
BOTH_INTRA
}This process is triggered by a function call to find_mv_stack.
The input to this process is a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction.
This process constructs an array RefStackMv containing motion vector candidates. This is similar to the RefListMv list of motion vectors, except that the stack only contains candidates that exactly match in their use of reference frames, while RefListMv can also include partial matches. For single prediction, the candidates in the stack are used to overwrite entries in RefListMv.
The process also prepares the value of the contexts used when decoding inter prediction syntax elements.
The variable bw4 specifying the width of the block in 4x4 pixels is set equal to Num_4x4_Blocks_Wide[ MiSize ].
The following ordered steps apply:
-
The variable NumMvFound (representing the number of motion vector candidates in RefStackMv) is set equal to 0.
-
The variable NumNewMvFound (representing the number of candidates in RefStackMv that used NEWMV encoding) is set equal to 0.
-
If refList is greater than or equal to 0, the following applies:
-
The setup zero mv process specified in [section 7.7.1.1][] is invoked with the input refList and the output is assigned to ZeroMvs[ refList ].
-
The array RefListMv[ refList ] is initialized to have both entries equal to ZeroMvs[ refList ].
-
-
The scan row process in [section 7.7.3.1][] is invoked with deltaRow equal to -1 and refList as inputs.
-
The scan col process in [section 7.7.3.2][] is invoked with deltaCol equal to -1 and refList as inputs.
-
The scan point process in [section 7.7.3.3][] is invoked with deltaRow equal to -1, deltaCol equal to bw4, and refList as inputs.
-
The variable numNearest (representing the number of motion vectors found in the immediate neighbourhood) is set equal to NumMvFound
-
The variable numNew (representing the number of NEWMV candidates found in the immediate neighbourhood) is set equal to NumNewMvFound
-
If numNearest is greater than 0, WeightStack[ idx ] is incremented by REF_CAT_LEVEL for idx = 0..(numNearest-1).
-
The variable ZeroMvContext is set equal to 0.
-
If refList is greater than or equal to 0 and can_use_previous is equal to 1, the temporal scan process in [section 7.7.3.4][] is invoked with refList as input (the temporal scan process affects ZeroMvContext).
-
The scan point process in [section 7.7.3.3][] is invoked with deltaRow equal to -1, deltaCol equal to -1, and refList as inputs.
-
The scan row process in [section 7.7.3.1][] is invoked with deltaRow equal to -3 and refList as inputs.
-
The scan col process in [section 7.7.3.2][] is invoked with deltaCol equal to -3 and refList as inputs.
-
The scan row process in [section 7.7.3.1][] is invoked with deltaRow equal to -5 and refList as inputs.
-
The scan col process in [section 7.7.3.2][] is invoked with deltaCol equal to -5 and refList as inputs.
-
The scan col process in [section 7.7.3.2][] is invoked with deltaCol equal to -7 and refList as inputs.
-
The sorting process in [section 7.7.3.10][] is invoked with start equal to 0 and end equal to numNearest.
-
The sorting process in [section 7.7.3.10][] is invoked with start equal to numNearest and end equal to NumMvFound.
-
The context and clamping process in [section 7.7.3.11][] is invoked with refList, numNearest, and numNew as input.
The inputs to this process are:
-
a variable deltaRow specifying (in units of 4x4 pixels) how far above to look for motion vectors,
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction.
The variable bw4 specifying the width of the block in 4x4 pixels is set equal to Num_4x4_Blocks_Wide[ MiSize ].
The variable deltaCol is set equal to 0.
The variable useStep16 is set equal to (bw4 >= 16).
Note: useStep16 is equal to 1 when the block is 64 pixels wide. This means only 4 locations will be searched in this case. However, a 32 pixel wide block may still search 8 locations. {:.alert .alert-info }
If Abs(deltaRow) is greater than 1, the offset is adjusted as follows:
deltaRow += MiRow & 1
deltaCol = 1 - (MiCol & 1)Note: These adjustments reduce the number of motion vectors that need to be kept in memory {:.alert .alert-info }
A series of motion vector locations is scanned as follows:
i = 0
while ( i < bw4 ) {
mvRow = MiRow + deltaRow
mvCol = MiCol + deltaCol + i
if (!is_inside(mvRow,mvCol))
break
len = Min(bw4, Num_4x4_Blocks_Wide[ MiSizes[ mvRow ][ mvCol ] ])
if (Abs(deltaRow) > 1)
len = Max(2, len)
if (useStep16)
len = Max(4, len)
add_ref_mv_candidate( mvRow, mvCol, refList, len)
i += len
}where the call to add_ref_mv_candidate invokes the process in [section 7.7.3.6][].
The inputs to this process are:
-
a variable deltaCol specifying (in units of 4x4 pixels) how far left to look for motion vectors,
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction.
The variable bh4 specifying the height of the block in 4x4 pixels is set equal to Num_4x4_Blocks_High[ MiSize ].
The variable deltaRow is set equal to 0.
The variable useStep16 is set equal to (bh4 >= 16).
If Abs(deltaCol) is greater than 1, the offset is adjusted as follows:
deltaRow = 1 - (MiRow & 1)
deltaCol += MiCol & 1A series of motion vector locations is scanned as follows:
i = 0
while ( i < bh4 ) {
mvRow = MiRow + deltaRow + i
mvCol = MiCol + deltaCol
if (!is_inside(mvRow,mvCol))
break
len = Min(bh4, Num_4x4_Blocks_High[ MiSizes[ mvRow ][ mvCol ] ])
if (Abs(deltaCol) > 1)
len = Max(2, len)
if (useStep16)
len = Max(4, len)
add_ref_mv_candidate( mvRow, mvCol, refList, len)
i += len
}where the call to add_ref_mv_candidate invokes the process in [section 7.7.3.6][].
The inputs to this process are:
-
a variable deltaRow specifying (in units of 4x4 pixels) how far above to look for a motion vector,
-
a variable deltaCol specifying (in units of 4x4 pixels) how far left to look for a motion vector,
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction.
The variable mvRow is set equal to MiRow + deltaRow.
The variable mvCol is set equal to MiCol + deltaCol.
The variable len is set equal to 8.
If is_inside( mvRow, mvCol ) is equal to 1 and RefFrames[ mvRow ][ mvCol ][ 0 ] has been written for this frame (this checks that the candidate location has been decoded), the add reference motion vector process in [section 7.7.3.6][] is invoked with mvRow, mvCol, refList, len as inputs.
The input to this process is a variable refList specifying which set of motion vectors to predict.
This process scans the motion vectors in a previous frame looking for candidates which use the same reference frame.
The variable bw4 specifying the width of the block in 4x4 pixels is set equal to Num_4x4_Blocks_Wide[ MiSize ].
The variable bh4 specifying the height of the block in 4x4 pixels is set equal to Num_4x4_Blocks_High[ MiSize ].
The process scans the locations within the block as follows:
for ( deltaRow = 0; deltaRow < bh4 ; deltaRow += 2 ) {
for ( deltaCol = 0; deltaCol < bw4 ; deltaCol += 2 ) {
add_tpl_ref_mv( deltaRow, deltaCol, refList)
}
}where the call to add_tpl_ref_mv invokes the temporal sample process in [section 7.7.3.5][].
If bh4 is less than 2 or bw4 is less than 2, the process exits at this point.
Otherwise, the process then scans positions around the block (but still within the same superblock) as follows:
for ( i = 0; i < 3; i++ ) {
deltaRow = tplSamplePos[ i ][ 0 ]
deltaCol = tplSamplePos[ i ][ 1 ]
if ( check_sb_border( deltaRow, deltaCol ) ) {
add_tpl_ref_mv( deltaRow, deltaCol, refList)
}
}where tplSamplePos contains the offsets to search (in units of 4x4 pixels) and is specified as:
tplSamplePos[3][2] = {
{ bh4, -2 }, { bh4, bw4 }, { bh4 - 2, bw4 }
}and check_sb_border checks that the position is within the same superblock as follows:
check_sb_border( deltaRow, deltaCol ) {
sbSize = use_128x128_superblock ? BLOCK_128X128 : BLOCK_64X64
sbSize4 = Num_4x4_Blocks_Wide[ sbSize ]
row = (MiRow % sbSize4) + deltaRow
col = (MiCol % sbSize4) + deltaCol
return ( row >= 0 && row < sbSize4 && col >= 0 && col < sbSize4 )
}The inputs to this process are:
-
variables deltaRow and deltaCol specifying (in units of 4x4 pixels) the offset to the candidate location,
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction.
This process looks up a motion vector from the motion field and adds it into the stack.
The variable mvRow is set equal to (MiRow + deltaRow) | 1.
The variable mvCol is set equal to (MiCol + deltaCol) | 1.
If is_inside( mvRow, mvCol ) is equal to 0, this process terminates immediately.
The variable x8 is set equal to mvCol >> 1.
The variable y8 is set equal to mvRow >> 1.
(x8 and y8 represent the position of the candidate in units of 8x8 pixels.)
The process is specified as follows:
foundValidMv = 0
if ( refList >= 0 ) {
for ( i = 0; i < MFMV_STACK_SIZE; i++ ) {
candMv = MotionFieldMvs[ RefFrame[ refList ] ][ y8 ][ x8 ][ i ]
if ( candMv[ 0 ] == -1 << 15 )
continue
foundValidMv = 1
lower_mv_precision( candMv )
if ( deltaRow == 0 && deltaCol == 0 ) {
if ( Abs( candMv[ 0 ] ) >= 16 || Abs( candMv[ 1 ] ) >= 16 )
ZeroMvContext = 1
}
for ( idx = 0; idx < NumMvFound; idx++ ) {
if ( Abs( candMv[ 0 ] - RefStackMv[ idx ][ 0 ][ 0 ] ) < 4 &&
Abs( candMv[ 1 ] - RefStackMv[ idx ][ 0 ][ 1 ] ) < 4 )
break
}
if ( idx < NumMvFound ) {
WeightStack[ idx ] += 2
} else if ( NumMvFound < MAX_REF_MV_STACK_SIZE ) {
RefStackMv[ NumMvFound ][ 0 ] = candMv
PredStackMv[ NumMvFound ][ 0 ] = candMv
PredStackMv[ NumMvFound ][ 0 ][ 0 ] += 4
WeightStack[ NumMvFound ] = 2
NumMvFound += 1
}
}
} else {
for ( i = 0; i < MFMV_STACK_SIZE; i++ ) {
candMv0 = MotionFieldMvs[ RefFrame[ 0 ] ][ y8 ][ x8 ][ i ]
if ( candMv0[ 0 ] == -1 << 15 )
continue
candMv1 = MotionFieldMvs[ RefFrame[ 1 ] ][ y8 ][ x8 ][ i ]
if ( candMv1[ 0 ] == -1 << 15 )
continue
foundValidMv = 1
lower_mv_precision( candMv0 )
lower_mv_precision( candMv1 )
if ( deltaRow == 0 && deltaCol == 0 ) {
if ( Abs( candMv0[ 0 ] ) >= 16 || Abs( candMv0[ 1 ] ) >= 16 ||
Abs( candMv1[ 0 ] ) >= 16 || Abs( candMv1[ 1 ] ) >= 16 )
ZeroMvContext = 1
}
for ( idx = 0; idx < NumMvFound; idx++ ) {
if ( Abs( candMv0[ 0 ] - RefStackMv[ idx ][ 0 ][ 0 ] ) < 4 &&
Abs( candMv0[ 1 ] - RefStackMv[ idx ][ 0 ][ 1 ] ) < 4 &&
Abs( candMv1[ 0 ] - RefStackMv[ idx ][ 1 ][ 0 ] ) < 4 &&
Abs( candMv1[ 1 ] - RefStackMv[ idx ][ 1 ][ 1 ] ) < 4)
break
}
if ( idx < NumMvFound ) {
WeightStack[ idx ] += 2
} else if ( NumMvFound < MAX_REF_MV_STACK_SIZE ) {
RefStackMv[ NumMvFound ][ 0 ] = candMv0
PredStackMv[ NumMvFound ][ 0 ] = candMv0
PredStackMv[ NumMvFound ][ 0 ][ 0 ] += 4
RefStackMv[ NumMvFound ][ 1 ] = candMv1
PredStackMv[ NumMvFound ][ 1 ] = candMv1
PredStackMv[ NumMvFound ][ 1 ][ 0 ] += 4
WeightStack[ NumMvFound ] = 2
NumMvFound += 1
}
}
}
if ( deltaRow == 0 && deltaCol == 0 ) {
if ( !foundValidMv )
ZeroMvContext = 1
}where the call to lower_mv_precision invokes the lower precision process specified in [section 7.7.3.9][].
Note: The motion field does not store values for PredMv, so the PredStackMv values are constructed as a simple offset of the candidate motion vectors. {:.alert .alert-info }
The inputs to this process are:
-
variables mvRow and mvCol specifying (in units of 4x4 pixels) the candidate location,
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction,
-
a variable len specifying the weight attached to this motion vector.
This process examines the candidate to find matching reference frames.
If IsInters[ mvRow ][ mvCol ] is equal to 0, this process terminates immediately.
If refList is greater than or equal to 0, the following applies for candList = 0..1:
- if RefFrames[ mvRow ][ mvCol ][ candList ] is equal to RefFrame[ refList ], the search stack process in [section 7.7.3.7][] is invoked with mvRow, mvCol, len, candList, refList as inputs
Otherwise (refList is equal to -1), the following applies:
- if RefFrames[ mvRow ][ mvCol ][ 0 ] is equal to RefFrame[ 0 ] and RefFrames[ mvRow ][ mvCol ][ 1 ] is equal to RefFrame[ 1 ], the compound search stack process in [section 7.7.3.8][] is invoked with mvRow, mvCol, refList, len as inputs
The inputs to this process are:
-
variables mvRow and mvCol specifying (in units of 4x4 pixels) the candidate location,
-
a variable candList specifying which list in the candidate matches our reference frame,
-
a variable len specifying the weight attached to this motion vector,
-
a variable refList specifying which set of motion vectors to predict.
This process adds motion vectors to the stack if they are not already present.
The variable candMode is set equal to YModes[ mvRow ][ mvCol ].
The variable candSize is set equal to MiSizes[ mvRow ][ mvCol ].
The variable large is set equal to ( Min( Block_Width[ candSize ],Block_Height[ candSize ] ) >= 8 ).
The candidate motion vector candMv is set as follows:
-
If ( candMode == GLOBALMV || candMode == GLOBAL_GLOBALMV) and ( gm_type[ RefFrame[ refList ] ] > TRANSLATION ) and ( large == 1 ), candMv is set equal to ZeroMvs[ refList ].
-
Otherwise, candMv is set equal to Mvs[ mvRow ][ mvCol ][ candList ].
The lower precision process specified in [section 7.7.3.9][] is invoked with candMv.
The variable weight is set equal to len * 2.
The process depends on whether the candidate motion vector is already in the stack as follows:
-
If candMv is already equal to RefStackMv[ idx ][ 0 ] for some idx less than NumMvFound, then WeightStack[ idx ] is increased by weight
-
Otherwise, if NumMvFound is less than MAX_REF_MV_STACK_SIZE, the following ordered steps apply:
a. RefStackMv[ NumMvFound ][ 0 ] is set equal to candMv
b. PredStackMv[ NumMvFound ][ 0 ] is set equal to PredMvs[ mvRow ][ mvCol ][ 0 ]
c. WeightStack[ NumMvFound ] is set equal to weight
d. NumMvFound is set equal to NumMvFound + 1.
e. If candMode is equal to NEWMV, NumNewMvFound is set equal to NumNewMvFound + 1.
-
Otherwise, (NumMvFound is greater than or equal to MAX_REF_MV_STACK_SIZE), the process has no effect.
The inputs to this process are:
-
variables mvRow and mvCol specifying (in units of 4x4 pixels) the candidate location,
-
a variable candList specifying which list in the candidate matches our reference frame,
-
a variable len specifying the weight attached to this motion vector,
This process adds motion vectors to the stack if they are not already present.
The array candMvs (containing two motion vectors) is set equal to Mvs[ mvRow ][ mvCol ].
The variable candMode is set equal to YModes[ mvRow ][ mvCol ].
The variable candSize is set equal to MiSizes[ mvRow ][ mvCol ].
The variable large is set equal to ( Min( Block_Width[ candSize ],Block_Height[ candSize ] ) >= 8 ).
If candMode is equal to GLOBAL_GLOBALMV and large is equal to 1, for refList = 0..1 the following applies:
- If gm_type[ RefFrame[ refList ] ] > TRANSLATION, candMvs[ refList ] is set equal to ZeroMvs[ refList ].
For i = 0..1, the lower precision process specified in [section 7.7.3.9][] is invoked with candMvs[ i ].
The variable weight is set equal to len * 2.
The process depends on whether the candidate motion vector is already in the stack as follows:
-
If candMvs[ 0 ] is equal to RefStackMv[ idx ][ 0 ] and candMvs[ 1 ] is equal to RefStackMv[ idx ][ 1 ] for some idx less than NumMvFound, then WeightStack[ idx ] is increased by weight
-
Otherwise, if NumMvFound is less than MAX_REF_MV_STACK_SIZE, the following ordered steps apply:
a. RefStackMv[ NumMvFound ][ i ] is set equal to candMvs[ i ] for i = 0..1
b. PredStackMv[ NumMvFound ][ i ] is set equal to PredMvs[ mvRow ][ mvCol ][ i ] for i = 0..1
b. WeightStack[ NumMvFound ] is set equal to weight
c. NumMvFound is set equal to NumMvFound + 1.
-
Otherwise, (NumMvFound is greater than or equal to MAX_REF_MV_STACK_SIZE), the process has no effect.
NumNewMvFound is incremented by 1 if all of the following are true:
-
NumMvFound is increased during this process
-
YModes[ mvRow ][ mvCol ] is equal to NEW_NEWMV
The input to this process is a reference candMv to a motion vector array.
This process modifies the contents of the input motion vector to remove the least significant bit when high precision is not allowed, and all three fractional bits when force_integer_mv is equal to 1.
If allow_high_precision_mv is equal to 1, this process terminates immediately.
For i = 0..1, the following applies:
if (force_integer_mv) {
a = Abs( candMv[ i ] )
aInt = (a + 3) >> 3
if (candMv[ i ] > 0)
candMv[ i ] = aInt << 3
else
candMv[ i ] = -( aInt << 3 )
} else {
if (candMv[ i ] & 1) {
if (candMv[ i ] > 0)
candMv[ i ]--
else
candMv[ i ]++
}
}The inputs to this process are:
-
a variable start representing the first position to be sorted,
-
a variable end representing the length of the array.
This process performs a stable sort of part of the stack of motion vectors according to the corresponding weight.
Entries in RefStackMv from start (inclusive) to end (exclusive) are sorted.
The sorting process is specified as:
while ( end > start ) {
newEnd = start
for ( idx = start + 1; idx < end; idx++ ) {
if ( WeightStack[ idx - 1 ] < WeightStack[ idx ] ) {
swap(idx - 1, idx)
newEnd = idx
}
}
end = newEnd
}When the function swap is invoked, the entries at locations idx and idx - 1 should be swapped in each of the following arrays:
- WeightStack
- RefStackMv
- PredStackMv
The inputs to this process are:
-
a variable refList specifying which set of motion vectors to predict, or -1 to signal compound prediction,
-
a variable numNearest specifying the number of candidates found in the immediate neighbourhood,
-
a variable numNew specifying the number of NEWMV candidates found in the immediate neighbourhood.
This process computes contexts to be used when decoding syntax elements, and overwrites the entries in RefListMv with clamped candidates from RefStackMv.
The variable bw (representing the width of the block in units of pixels) is set equal to Block_Width[ MiSize ].
The variable bh (representing the height of the block in units of pixels) is set equal to Block_Height[ MiSize ].
The variable numLists specifying the number of reference frames used for this block is set equal to ( (refList < 0) ? 2 : 1 ).
The arrays DrlCtxStack and MvCtxStack contain one or two contexts (depending on whether we are doing single or compound prediction) for each entry in the stack.
The array MvCtxStack is set as follows:
for ( list = 0; list < numLists; list++ ) {
for ( idx = 0; idx < NumMvFound ; idx++ ) {
refMv = RefStackMv[ idx ][ list ]
predMv = PredStackMv[ idx ][ list ]
v = 2
for ( j = 0; j < 2; j++ ) {
if ( Abs( refMv[ j ] - predMv[ j ] ) > 4 )
v = 1
}
MvCtxStack[ idx ][ list ] = v
}
}The array DrlCtxStack is set as follows:
for ( idx = 0; idx < NumMvFound ; idx++ ) {
z = 0
if ( idx + 1 < NumMvFound ) {
w0 = WeightStack[ idx ]
w1 = WeightStack[ idx + 1 ]
if (w0 >= REF_CAT_LEVEL) {
if (w1 < REF_CAT_LEVEL) {
z = 2
}
} else {
if (w1 < REF_CAT_LEVEL) {
z = 3
}
}
}
DrlCtxStack[ idx ] = z
}Note: It is impossible for DrlCtxStack to contain the value 1. {:.alert .alert-info }
The motion vectors are clamped as follows:
for ( list = 0; list < numLists; list++ ) {
for ( idx = 0; idx < NumMvFound ; idx++ ) {
refMv = RefStackMv[ idx ][ list ]
refMv[ 0 ] = clamp_mv_row( refMv[ 0 ], MV_BORDER + bh * 8)
refMv[ 1 ] = clamp_mv_col( refMv[ 1 ], MV_BORDER + bw * 8)
if ( refList < 0 ) {
RefStackMv[ idx ][ list ] = refMv
} else if (idx < 2) {
RefListMv[ refList ][ idx ] = refMv
}
}
}The variables RefMvContext and NewMvContext are set as follows:
if ( numNearest == 0 ) {
NewMvContext = Min( NumMvFound, 1 ) // 0,1
RefMvContext = Min( NumMvFound, 2 ) // 0,1,2
} else if ( numNearest == 1 ) {
NewMvContext = 3 - Min( numNew, 1 ) // 2,3
RefMvContext = Min( 3 * NumMvFound, 4) // 0,3,4
} else {
NewMvContext = 6 - Min( numNew, 2 ) // 4,5,6
RefMvContext = 5
}Note: It is impossible for RefMvContext to be equal to 6 or 7 or 8+6 or 8+7. {:.alert .alert-info }
This process is triggered by a call to has_overlappable_candidates.
It returns 1 to indicate that the block has neighbours suitable for use by overlapped motion compensation, or 0 otherwise.
The process looks to see if there are any inter blocks to the left or above.
The check is only made at 8x8 granularity.
The process is specified as:
has_overlappable_candidates( ) {
if ( AvailU ) {
w4 = Num_4x4_Blocks_Wide[ MiSize ]
for ( x4 = MiCol; x4 < Min( MiCols, MiCol + w4 ); x4 += 2 ) {
if ( RefFrames[ MiRow - 1 ][ x4 | 1 ][ 0 ] > INTRA_FRAME )
return 1
}
}
if ( AvailL ) {
h4 = Num_4x4_Blocks_High[ MiSize ]
for ( y4 = MiRow; y4 < Min( MiRows, MiRow + h4 ); y4 += 2 ) {
if ( RefFrames[ y4 | 1 ][ MiCol - 1 ][ 0 ] > INTRA_FRAME )
return 1
}
}
return 0
}This process is triggered when the find_warp_samples function is invoked.
The process examines the neighbouring inter predicted blocks and estimates a local warp transformation based on the motion vectors.
The process produces a variable NumSamples containing the number of valid candidates found, and an array CandList containing sorted candidates.
The variables NumSamples and NumSamplesScanned are both set equal to 0.
The variable w4 specifying the width of the block in 4x4 pixels is set equal to Num_4x4_Blocks_Wide[ MiSize ].
The variable h4 specifying the height of the block in 4x4 pixels is set equal to Num_4x4_Blocks_High[ MiSize ].
The following ordered steps apply:
-
The add sample process in [section 7.7.5.1][] is invoked with deltaRow equal to -1, and deltaCol equal to 0..(w4 - 1) (this adds candidates above the current block).
-
The add sample process in [section 7.7.5.1][] is invoked with deltaCol equal to -1, and deltaRow equal to 0..(h4 - 1) (this adds candidates to the left of the current block).
-
The add sample process in [section 7.7.5.1][] is invoked with deltaRow equal to -1, and deltaCol equal to -1 (this adds a candidate to the top left).
-
The add sample process in [section 7.7.5.1][] is invoked with deltaRow equal to -1, and deltaCol equal to w4 (this adds a candidate to the top right).
-
If NumSamples is equal to 0 and NumSamplesScanned is greater than 0, NumSamples is set equal to 1.
The inputs to this process are:
-
a variable deltaRow specifying (in units of 4x4 pixels) how far above to look for a motion vector,
-
a variable deltaCol specifying (in units of 4x4 pixels) how far left to look for a motion vector.
The output of this process is to add a new sample to the list of candidates if it is a valid candidate and has not been seen before.
If NumSamplesScanned is greater than or equal to LEAST_SQUARES_SAMPLES_MAX, this process immediately exits.
The variable mvRow is set equal to MiRow + deltaRow.
The variable mvCol is set equal to MiCol + deltaCol.
If is_inside( mvRow, mvCol ) is equal to 0, then this process immediately returns.
If RefFrames[ mvRow ][ mvCol ][ 0 ] has not been written for this frame, then this process immediately returns.
If RefFrames[ mvRow ][ mvCol ][ 0 ] is not equal to RefFrame[ 0 ], then this process immediately returns.
If RefFrames[ mvRow ][ mvCol ][ 1 ] is not equal to NONE, then this process immediately returns.
The variable candSz is set equal to MiSizes[ mvRow ][ mvCol ].
The variable candW4 is set equal to Num_4x4_Blocks_Wide[ candSz ].
The variable candH4 is set equal to Num_4x4_Blocks_High[ candSz ].
The variable candRow is set equal to mvRow & ~(candH4 - 1).
The variable candCol is set equal to mvCol & ~(candW4 - 1).
The variable midY is set equal to candRow * 4 + candH4 * 2 - 1.
The variable midX is set equal to candCol * 4 + candW4 * 2 - 1.
The variable threshold is set equal to Clip3( 16, 112, Max( Block_Width[ MiSize ], Block_Height[ MiSize ] ) ).
The variable mvDiffRow is set equal to Abs( Mvs[ candRow ][ candCol ][ 0 ][ 0 ] - Mv[ 0 ][ 0 ] ).
The variable mvDiffCol is set equal to Abs( Mvs[ candRow ][ candCol ][ 0 ][ 1 ] - Mv[ 0 ][ 1 ] ).
The variable valid is set equal to ( ( mvDiffRow + mvDiffCol ) <= threshold ).
Note: candRow and candCol give the top-left position of the candidate block in units of 4x4 blocks. midX and midY give the central position of the candidate block in units of pixels. {:.alert .alert-info }
A candidate array (representing source and destination locations in units of 1/8 pixels) is specified as:
cand[ 0 ] = midY * 8
cand[ 1 ] = midX * 8
cand[ 2 ] = midY * 8 + Mvs[ candRow ][ candCol ][ 0 ][ 0 ]
cand[ 3 ] = midY * 8 + Mvs[ candRow ][ candCol ][ 0 ][ 1 ]If this array cand is equal (in all 4 locations) to CandList[ i ] for any i < NumSamples, then this process immediately returns.
Otherwise, the following ordered steps apply:
-
NumSamplesScanned is increased by 1.
-
If valid is equal to 0 and NumSamplesScanned is greater than 1, the process exits.
-
CandList[ NumSamples ][ j ] is set equal to cand[ j ] for j=0..3.
-
If valid is equal to 1, NumSamples is increased by 1.
Note: The test for cand being equal to CandList[i] ensures that each candidate block is only added once to the list. If the first two entries are equal, all four entries will automatically be equal. The reference implementation shows how to use additional logic to avoid calling the add sample process more than once per candidate. {:.alert .alert-info }
The following sections define the processes used for predicting the sample values.
These processes are triggered at points defined by function calls to predict_intra, predict_inter, predict_chroma_from_luma, and predict_palette in the residual syntax table described in [section 5.8.32][].
The intra prediction process is invoked for intra coded blocks to predict a part of the block corresponding to a transform block. When the transform size is smaller than the block size, this process can be invoked multiple times within a single block for the same plane, and the invocations are in raster order within the block.
The inputs to this process are:
-
the already reconstructed samples in the current frame CurrFrame,
-
a variable plane specifying which plane is being predicted,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the current transform block,
-
a variable haveLeft that is equal to 1 if there are valid samples to the left of this transform block,
-
a variable haveAbove that is equal to 1 if there are valid samples above this transform block,
-
a variable haveAboveRight that is equal to 1 if there are valid samples above the transform block to the right of this transform block,
-
a variable haveBelowLeft that is equal to 1 if there are valid samples to the left of the transform block below this transform block,
-
a variable mode specifying the type of intra prediction to apply,
-
variable log2W specifying the base 2 logarithm of the width of the region to be predicted,
-
variable log2H specifying the base 2 logarithm of the height of the region to be predicted.
The outputs of this process are intra predicted samples in the current frame CurrFrame.
The variable w is set equal to 1 << log2W.
The variable h is set equal to 1 << log2H.
The variable w4 is set equal to w >> 2.
The variable h2 is set equal to h >> 1.
The variable maxX is set equal to ( MiCols * MI_SIZE ) - 1.
The variable maxY is set equal to ( MiRows * MI_SIZE ) - 1.
If plane is greater than 0, then:
-
maxX is set equal to ( ( MiCols * MI_SIZE ) >> subsampling_x ) - 1.
-
maxY is set equal to ( ( MiRows * MI_SIZE ) >> subsampling_y ) - 1.
The array AboveRow[ i ] for i = 0..w + h - 1 is derived as follows:
-
If haveAbove is equal to 0 and haveLeft is equal to 1, AboveRow[ i ] is set equal to CurrFrame[ plane ][ y ][ x - 1 ].
-
Otherwise, if haveAbove is equal to 0 and haveLeft is equal to 0, AboveRow[ i ] is set equal to ( 1 << ( BitDepth - 1 ) ) - 1.
-
Otherwise, the following applies:
-
The variable aboveLimit is set equal to Min( maxX, x + ( haveAboveRight ? 2 * w : w ) - 1 ).
-
AboveRow[ i ] is set equal to CurrFrame[ plane ][ y-1 ][ Min(aboveLimit, x+i) ].
-
The array LeftCol[ i ] for i = 0..w + h - 1 is derived as follows:
-
If haveLeft is equal to 0 and haveAbove is equal to 1, LeftCol[ i ] is set equal to CurrFrame[ plane ][ y - 1 ][ x ].
-
Otherwise, if haveLeft is equal to 0 and haveAbove is equal to 0, LeftCol[ i ] is set equal to ( 1 << ( BitDepth - 1 ) ) + 1.
-
Otherwise, the following applies:
-
The variable leftLimit is set equal to Min( maxY, y + ( haveBelowLeft ? 2 * h : h ) - 1 ).
-
LeftCol[ i ] is set equal to CurrFrame[ plane ][ Min(leftLimit, y+i) ][ x-1 ].
-
The array AboveRow[ i ] for i = -1 is specified by:
-
If haveAbove is equal to 1 and haveLeft is equal to 1, AboveRow[ -1 ] is set equal to CurrFrame[ plane ][ y-1 ][ x-1 ].
-
Otherwise if haveAbove is equal to 1, AboveRow[ -1 ] is set equal to CurrFrame [ plane ][ y - 1 ][ x ].
-
Otherwise if haveLeft is equal to 1, AboveRow[ -1 ] is set equal to CurrFrame [ plane ][ y ][ x - 1 ].
-
Otherwise, AboveRow[ -1 ] is set equal to 1 << ( BitDepth - 1 ).
The array LeftCol[ i ] for i = -1 is set equal to AboveRow[ -1 ].
A 2D array named pred containing the intra predicted samples is constructed as follows:
-
If plane is equal to 0 and use_filter_intra is true, the following steps apply for i2 = 0..h2-1, for j4 = 0..w4-1:
-
The array p is derived as follows for i = 0..6:
-
If i is less than 5, p[ i ] is derived as follows:
-
If i2 is equal to 0, p[ i ] is set equal to AboveRow[ ( j4 << 2 ) + i - 1 ].
-
Otherwise (i2 is not equal to 0), p[ i ] is set equal to pred[ ( i2 << 1 ) - 1 ][ ( j4 << 2 ) + i - 1 ].
-
-
Otherwise (i is greater than or equal to 5), p[ i ] is derived as follows:
-
If j4 is equal to 0, p[ i ] is set equal to LeftCol[ ( i2 << 1 ) + i - 5 ].
-
Otherwise (j4 is not equal to 0), p[ i ] is set equal to pred[ ( i2 << 1 ) + i - 5 ][ ( j4 << 2 ) - 1 ].
-
-
-
The following steps apply for i1 = 0..1, j1 = 0..3:
-
The variable pr is set equal to 0.
-
The variable pr is incremented by Intra_Filter_Taps[ filter_intra_mode ][ ( i1 << 2 ) + j1 ][ i ] * p[ i ] for i = 0..6.
-
pred[ ( i2 << 1 ) + i1 ][ ( j4 << 2 ) + j1 ] is set equal to Clip1( Round2Signed( pr, INTRA_FILTER_SCALE_BITS ) ).
-
-
-
Otherwise, if is_directional_mode( mode ) is true, the following steps apply:
-
The variable angleDelta is derived as follows:
-
If plane is equal to 0, angleDelta is set equal to AngleDeltaY.
-
Otherwise (plane is not equal to 0), angleDelta is set equal to AngleDeltaUV.
-
-
The variable pAngle is set equal to ( Mode_To_Angle[ mode ] + angleDelta * ANGLE_STEP ).
-
If pAngle is not equal to 90 and pAngle is not equal to 180, the following applies:
-
If (pAngle > 90) and (pAngle < 180) and (w + h) >= 24), the filter corner process specified in [section 7.8.1.1][] is invoked and the output assigned to both LeftCol[ -1 ] and AboveRow[ -1 ].
-
The intra filter type process specified in [section 7.8.1.2][] is invoked and the output assigned to filterType.
-
If haveAbove is equal to 1, the following steps apply:
-
The intra edge filter strength selection process specified in [section 7.8.1.3][] is invoked with w, h, filterType, and pAngle - 90 as inputs, and the output assigned to the variable strength.
-
The variable numPx is set equal to Min( ( ( pAngle < 90 ) && haveAboveRight ) ? 2 * w : w, maxX - x + 1 ) + 1.
-
The intra edge filter process specified in [section 7.8.1.6][] is invoked with the parameters numPx, strength, and 0 as inputs.
-
-
If haveLeft is equal to 1, the following steps apply:
-
The intra edge filter strength selection process specified in [section 7.8.1.3][] is invoked with w, h, filterType, and pAngle - 180 as inputs, and the output assigned to the variable strength.
-
The variable numPx is set equal to Min( ( ( pAngle > 180 ) && haveBelowLeft ) ? 2 * h : h, maxY - y + 1 ) + 1.
-
The intra edge filter process specified in [section 7.8.1.6][] is invoked with the parameters numPx, strength, and 1 as inputs.
-
-
The intra edge upsample selection process specified in [section 7.8.1.4][] is invoked with w, h, filterType, and pAngle - 90 as inputs, and the output assigned to the variable upsampleAbove.
-
The variable numPx is set equal to ( w + (pAngle < 90 ? h : 0) ).
-
If upsampleAbove is equal to 1, the intra edge upsample process specified in [section 7.8.1.5][] is invoked with the parameters numPx and 0 as inputs.
-
The intra edge upsample selection process specified in [section 7.8.1.4][] is invoked with w, h, filterType, and pAngle - 180 as inputs, and the output assigned to the variable upsampleLeft.
-
The variable numPx is set equal to ( h + (pAngle < 180 ? w : 0) ).
-
If upsampleLeft is equal to 1, the intra edge upsample process specified in [section 7.8.1.5][] is invoked with the parameters numPx and 1 as inputs.
-
-
The variable dx is derived as follows:
-
If pAngle is greater than 0 and less than 90, dx is set equal to Dr_Intra_Derivative[ pAngle ].
-
Otherwise, if pAngle is greater than 90 and less than 180, dx is set equal to Dr_Intra_Derivative[ 180 - pAngle ].
-
Otherwise, dx is undefined.
-
-
The variable dy is derived as follows:
-
If pAngle is greater than 90 and less than 180, dy is set equal to Dr_Intra_Derivative[ pAngle - 90 ].
-
Otherwise, if pAngle is greater than 180 and less than 270, dy is set equal to Dr_Intra_Derivative[ 270 - pAngle ].
-
Otherwise, dy is undefined.
-
-
If pAngle is greater than 0 and less than 90, the following steps apply for i = 0..h-1, for j = 0..w-1:
-
The variable idx is set equal to ( i + 1 ) * dx.
-
The variable base is set equal to (idx >> ( 6 - upsampleAbove ) ) + (j << upsampleAbove).
-
The variable shift is set equal to ( (idx << upsampleAbove) >> 1 ) & 0x1F.
-
The variable maxBaseX is set equal to (w + h - 1) << upsampleAbove.
-
If base is less than maxBaseX, pred[ i ][ j ] is set equal to Clip1( Round2( AboveRow[ base ] * ( 32 - shift ) + AboveRow[ base + 1 ] * shift, 5 ) ).
-
Otherwise (base is greater than or equal to maxBaseX), pred[ i ][ j ] is set equal to AboveRow[ maxBaseX ].
-
-
Otherwise, if pAngle is greater than 90 and pAngle is less than 180, the following steps apply for i = 0..h-1, for j = 0..w-1:
-
The variable idx is set equal to ( j << 6 ) - ( i + 1 ) * dx.
-
The variable base is set equal to idx >> ( 6 - upsampleAbove ).
-
If base is greater than or equal to -(1 << upsampleAbove), the following steps apply:
-
The variable shift is set equal to ( ( idx << upsampleAbove ) >> 1 ) & 0x1F.
-
pred[ i ][ j ] is set equal to Clip1( Round2( AboveRow[ base ] * ( 32 - shift ) + AboveRow[ base + 1 ] * shift, 5 ) ).
-
-
Otherwise (base is less than -(1 << upsampleAbove), the following steps apply:
-
The variable idx is set equal to ( i << 6 ) - ( j + 1 ) * dy.
-
The variable base is set equal to idx >> ( 6 - upsampleLeft ).
-
The variable shift is set equal to ( ( idx << upsampleLeft ) >> 1 ) & 0x1F.
-
pred[ i ][ j ] is set equal to Clip1( Round2( LeftCol[ base ] * ( 32 - shift ) + LeftCol[ base + 1 ] * shift, 5 ) ).
-
-
-
Otherwise, if pAngle is greater than 180 and pAngle is less than 270, the following steps apply for i = 0..h-1, for j = 0..w-1:
-
The variable idx is set equal to ( j + 1 ) * dy.
-
The variable base is set equal to ( idx >> ( 6 - upsampleLeft ) ) + ( i << upsampleLeft ).
-
The variable shift is set equal to ( ( idx << upsampleLeft ) >> 1 ) & 0x1F.
-
The variable maxBaseY is set equal to ( w + h - 1 ) << upsampleLeft.
-
If base is less than maxBaseY, pred[ i ][ j ] is set equal to Clip1( Round2( LeftCol[ base ] * ( 32 - shift ) + LeftCol[ base + 1 ] * shift, 5 ) ).
-
Otherwise (base is greater than or equal to maxBaseY), pred[ i ][ j ] is set equal to LeftCol[ maxBaseY ].
-
-
Otherwise, if pAngle is equal to 90, pred[ i ][ j ] is set equal to AboveRow[ j ] with j = 0..w-1 and i = 0..h-1 (each row of the block is filled with a copy of AboveRow).
-
Otherwise, if pAngle is equal to 180, pred[ i ][ j ] is set equal to LeftCol[ i ] with j = 0..w-1 and i = 0..h-1 (each column of the block is filled with a copy of LeftCol).
-
-
Otherwise if mode is equal to V_PRED, pred[ i ][ j ] is set equal to AboveRow[ j ] with j = 0..w-1 and i = 0..h-1 (each row of the block is filled with a copy of AboveRow).
-
Otherwise if mode is equal to H_PRED, pred[ i ][ j ] is set equal to LeftCol[ i ] with j = 0..w-1 and i = 0..h-1 (each column of the block is filled with a copy of LeftCol).
-
Otherwise if mode is equal to D207_PRED, the following applies for i = 0..h-1:
-
pred[ i ][ j ] = Round2( LeftCol[ i + j/2 ] + LeftCol[ i + 1 + j/2 ], 1 ) for even values of j in the range 0..w-2
-
pred[ i ][ j ] = Round2( LeftCol[ i + j/2 ] + 2 * LeftCol[ i + 1 + j/2 ] + LeftCol[ i + 2 + j/2 ], 2 ) for odd values of j in the range 0..w-1
-
-
Otherwise if mode is equal to D45_PRED, the following applies for i = 0..h-1, for j = 0..w-1:
-
pred[ i ][ j ] = Round2( AboveRow[ i + j ] + 2 * AboveRow[ i + j + 1 ] + AboveRow[ i + j + 1 ], 2 ) if (i + j + 2 >= w + h)
-
pred[ i ][ j ] = Round2( AboveRow[ i + j ] + 2 * AboveRow[ i + j + 1 ] + AboveRow[ i + j + 2 ], 2 ) if (i + j + 2 < w + h)
-
-
Otherwise if mode is equal to D63_PRED, the following applies for i = 0..h-1, for j = 0..w-1:
-
pred[ i ][ j ] = Round2( AboveRow[ i/2 + j ] + AboveRow[ i/2 + j + 1 ], 1 ) for even values of i in the range 0..h-2
-
pred[ i ][ j ] = Round2( AboveRow[ i/2 + j ] + 2 * AboveRow[ i/2 + j + 1 ] + AboveRow[ i/2 + j + 2 ], 2 ) for odd values of i in the range 0..h-1.
-
-
Otherwise if mode is equal to D117_PRED, the following applies:
-
pred[ 0 ][ j ] = Round2( AboveRow[ j - 1 ] + AboveRow[ j ], 1 ) for j = 0..w-1
-
pred[ 1 ][ 0 ] = Round2( LeftCol[ 0 ] + 2 * AboveRow[ -1 ] + AboveRow[ 0 ], 2 )
-
pred[ 1 ][ j ] = Round2( AboveRow[ j - 2 ] + 2 * AboveRow[ j - 1 ] + AboveRow[ j ], 2 ) for j = 1..w-1
-
pred[ 2 ][ 0 ] = Round2( AboveRow[ -1 ] + 2 * LeftCol[ 0 ] + LeftCol[ 1 ], 2 )
-
pred[ i ][ 0 ] = Round2( LeftCol[ i - 3 ] + 2 * LeftCol[ i - 2 ] + LeftCol[ i - 1 ], 2 ) for i = 3..h-1
-
pred[ i ][ j ] = pred[ i - 2 ][ j - 1 ] for i = 2..h-1, for j = 1..w-1
-
-
Otherwise if mode is equal to D135_PRED, the following applies:
-
pred[ 0 ][ 0 ] = Round2( LeftCol[ 0 ] + 2 * AboveRow[ -1 ] + AboveRow[ 0 ], 2 )
-
pred[ 0 ][ j ] = Round2( AboveRow[ j - 2 ] + 2 * AboveRow[ j - 1 ] + AboveRow[ j ], 2 ) for j = 1..w-1
-
pred[ 1 ][ 0 ] = Round2( AboveRow[ -1 ] + 2 * LeftCol[ 0 ] + LeftCol[ 1 ], 2 )
-
pred[ i ][ 0 ] = Round2( LeftCol[ i - 2 ] + 2 * LeftCol[ i - 1 ] + LeftCol[ i ], 2 ) for i = 2..h-1
-
pred[ i ][ j ] = pred[ i - 1 ][ j - 1 ] for i = 1..h-1, for j = 1..w-1
-
-
Otherwise if mode is equal to D153_PRED, the following applies:
-
pred[ 0 ][ 0 ] = Round2( LeftCol[ 0 ] + AboveRow[ -1 ], 1 )
-
pred[ i ][ 0 ] = Round2( LeftCol[ i - 1 ] + LeftCol[ i ], 1 ) for i = 1..h-1
-
pred[ 0 ][ 1 ] = Round2( LeftCol[ 0 ] + 2 * AboveRow[ -1 ] + AboveRow[ 0 ], 2 )
-
pred[ 1 ][ 1 ] = Round2( AboveRow[ -1 ] + 2 * LeftCol[ 0 ] + LeftCol[ 1 ], 2 )
-
pred[ i ][ 1 ] = Round2( LeftCol[ i - 2 ] + 2 * LeftCol[ i - 1 ] + LeftCol[ i ], 2 ) for i = 2..h-1
-
pred[ 0 ][ j ] = Round2( AboveRow[ j - 3 ] + 2 * AboveRow[ j - 2 ] + AboveRow[ j - 1 ], 2 ) for j = 2..w-1
-
pred[ i ][ j ] = pred[ i - 1 ][ j - 2 ] for i = 1..h-1, for j = 2..w-1
-
-
Otherwise if mode is equal to TM_PRED, the following ordered steps apply for i = 0..h-1, for j = 0..w-1:
-
The variable base is set equal to AboveRow[ j ] + LeftCol[ i ] - AboveRow[ -1 ].
-
The variable pLeft is set equal to Abs( base - LeftCol[ i ]).
-
The variable pTop is set equal to Abs( base - AboveRow[ j ]).
-
The variable pTopLeft is set equal to Abs( base - AboveRow[ -1 ] ).
-
If pLeft <= pTop and pLeft <= pTopLeft, pred[ i ][ j ] is set equal to LeftCol[ i ].
-
Otherwise, if pTop <= pTopLeft, pred[ i ][ j ] is set equal to AboveRow[ j ].
-
Otherwise, pred[ i ][ j ] is set equal to AboveRow[ -1 ].
-
-
Otherwise if mode is equal to SMOOTH_PRED, the following ordered steps apply for i = 0..h-1, for j = 0..w-1:
-
The array smWeightsX is set dependent on the value of log2W according to the following table:
log2W smWeightsX 2 Sm_Weights_Tx_4x4 3 Sm_Weights_Tx_8x8 4 Sm_Weights_Tx_16x16 5 Sm_Weights_Tx_32x32 6 Sm_Weights_Tx_64x64 {:.table .table-sm .table-bordered } -
The array smWeightsY is set dependent on the value of log2H according to the following table:
log2H smWeightsY 2 Sm_Weights_Tx_4x4 3 Sm_Weights_Tx_8x8 4 Sm_Weights_Tx_16x16 5 Sm_Weights_Tx_32x32 6 Sm_Weights_Tx_64x64 {:.table .table-sm .table-bordered } -
The variable smoothPred is set as follows:
smoothPred = smWeightsY[ i ] * AboveRow[ j ] + ( 256 - smWeightsY[ i ] ) * LeftCol[ h - 1 ] + smWeightsX[ j ] * LeftCol[ i ] + ( 256 - smWeightsX[ j ] ) * AboveRow[ w - 1 ]
-
pred[ i ][ j ] is set equal to Clip1( Round2( smoothPred, 9 ) ).
-
-
Otherwise if mode is equal to SMOOTH_V_PRED, the following ordered steps apply for i = 0..h-1, for j = 0..w-1:
-
The array smWeights is set dependent on the value of log2H according to the following table:
log2H smWeights 2 Sm_Weights_Tx_4x4 3 Sm_Weights_Tx_8x8 4 Sm_Weights_Tx_16x16 5 Sm_Weights_Tx_32x32 6 Sm_Weights_Tx_64x64 {:.table .table-sm .table-bordered } -
The variable smoothPred is set as follows:
smoothPred = smWeights[ i ] * AboveRow[ j ] + ( 256 - smWeights[ i ] ) * LeftCol[ h - 1 ]
-
pred[ i ][ j ] is set equal to Clip1( Round2( smoothPred, 8 ) ).
-
-
Otherwise if mode is equal to SMOOTH_H_PRED, the following ordered steps apply for i = 0..h-1, for j = 0..w-1:
-
The array smWeights is set dependent on the value of log2W according to the following table:
log2W smWeights 2 Sm_Weights_Tx_4x4 3 Sm_Weights_Tx_8x8 4 Sm_Weights_Tx_16x16 5 Sm_Weights_Tx_32x32 6 Sm_Weights_Tx_64x64 {:.table .table-sm .table-bordered } -
The variable smoothPred is set as follows:
smoothPred = smWeights[ j ] * LeftCol[ i ] + ( 256 - smWeights[ j ] ) * AboveRow[ w - 1 ]
-
pred[ i ][ j ] is set equal to Clip1( Round2( smoothPred, 8 ) ).
-
-
Otherwise if mode is equal to DC_PRED and haveLeft is equal to 1 and haveAbove is equal to 1, pred[ i ][ j ] is set equal to avg with i = 0..h-1 and j = 0..w-1. The variable avg (the average of the samples in union of AboveRow and LeftCol) is specified as follows:
sum = 0 for ( k = 0; k < h; k++ ) sum += LeftCol[ k ] for ( k = 0; k < w; k++ ) sum += AboveRow[ k ] longLog2 = Max( log2W, log2h ) shortLog2 = Min( log2W, log2h ) diffLog2 = longLog2 - shortLog2 if ( diffLog2 == 1 ) { avg = ( sum * 0xab ) >> ( 9 + shortLog2 ) } else if ( diffLog2 == 2 ) { avg = ( sum * 0xcd ) >> ( 10 + shortLog2 ) } else { avg = (sum + ( ( w + h ) >> 1 ) ) / ( w + h ) } avg = Clip1( avg )
Note: We want to divide by w+h without using a division. 0xab is round(512/3) and 0xcd is round(1024,5). {:.alert .alert-info }
-
Otherwise if mode is equal to DC_PRED and haveLeft is equal to 1 and haveAbove is equal to 0, pred[ i ][ j ] is set equal to leftAvg with i = 0..h-1 and j = 0..w-1. The variable leftAvg is specified as follows:
sum = 0 for ( k = 0; k < h; k++ ) { sum += LeftCol[ k ] } leftAvg = (sum + (1 << (log2H - 1) ) ) >> log2H
-
Otherwise if mode is equal to DC_PRED and haveLeft is equal to 0 and haveAbove is equal to 1, pred[ i ][ j ] is set equal to aboveAvg with i = 0..h-1 and j = 0..w-1. The variable aboveAvg is specified as follows:
sum = 0 for ( k = 0; k < w; k++ ) { sum += AboveRow[ k ] } aboveAvg = (sum + (1 << (log2W - 1) ) ) >> log2W
-
Otherwise (mode is DC_PRED), pred[ i ][ j ] is set equal to 1 << ( BitDepth - 1 ) with i = 0..h-1 and j = 0..w-1.
The current frame is updated as follows:
- CurrFrame[ plane ][ y + i ][ x + j ] is set equal to pred[ i ][ j ] for i = 0..h-1 and j = 0..w-1.
This process uses a three tap filter to compute the value to be used for the top-left corner.
The variable s is set equal to LeftCol[ 0 ] * 5 + AboveRow[ -1 ] * 6 + AboveRow[ 0 ] * 5.
The output of this process is Round2(s, 4).
The output of this process is a variable filterType that is set to 1 if either the block above or to the left uses a smooth prediction mode.
The process is specified as follows:
get_filter_type( ) {
aboveSmooth = 0
leftSmooth = 0
if (AvailU) {
aboveSmooth = is_smooth( YModes[ MiRow - 1 ][ MiCol ] )
}
if (AvailL) {
leftSmooth = is_smooth( YModes[ MiRow ][ MiCol - 1 ] )
}
return aboveSmooth || leftSmooth
}where the function is_smooth indicates if a prediction mode is one of the smooth intra modes and is specified as:
is_smooth( mode ) {
return (mode == SMOOTH_PRED || mode == SMOOTH_V_PRED || mode == SMOOTH_H_PRED)
}The inputs to this process are:
-
a variable w containing the width of the transform in samples,
-
a variable h containing the height of the transform in samples,
-
a variable filterType that is 0 or 1 that controls the strength of filtering,
-
a variable delta containing an angle difference in degrees.
The output is an intra edge filter strength from 0 to 3 inclusive.
The variable d is set equal to Abs( delta ).
The variable blkWh (containing the sum of the dimensions) is set equal to w + h.
The output variable strength is specified as follows:
strength = 0
if (filterType == 0) {
if (blkWh <= 8) {
if (d >= 56) strength = 1
} else if (blkWh <= 12) {
if (d >= 40) strength = 1
} else if (blkWh <= 16) {
if (d >= 40) strength = 1
} else if (blkWh <= 24) {
if (d >= 8) strength = 1
if (d >= 16) strength = 2
if (d >= 32) strength = 3
} else if (blkWh <= 32) {
if (d >= 1) strength = 1
if (d >= 4) strength = 2
if (d >= 32) strength = 3
} else {
if (d >= 1) strength = 3
}
} else {
if (blkWh <= 8) {
if (d >= 40) strength = 1
if (d >= 64) strength = 2
} else if (blkWh <= 16) {
if (d >= 20) strength = 1
if (d >= 48) strength = 2
} else if (blkWh <= 24) {
if (d >= 4) strength = 3
} else {
if (d >= 1) strength = 3
}
}The inputs to this process are:
-
a variable w containing the width of the transform in samples,
-
a variable h containing the height of the transform in samples,
-
a variable filterType that is 0 or 1 that controls the strength of filtering,
-
a variable delta containing an angle difference in degrees.
The output is a flag useUpsample that is true if upsampling should be applied to the edge.
The variable d is set equal to Abs( delta ).
The variable blkWh (containing the sum of the dimensions) is set equal to w + h.
The output variable useUpsample is specified as follows:
if (d <= 0 || d >= 40) {
useUpsample = 0
} else if ( filterType == 0 ) {
useUpsample = (blkWh <= 16)
} else {
useUpsample = (blkWh <= 8)
}The inputs to this process are:
-
a variable numPx specifying the number of samples to filter,
-
a variable dir containing 0 when filtering the above samples, and 1 when filtering the left samples.
The output of this process are upsampled samples in the AboveRow and LeftCol arrays.
The variable buf is set depending on dir:
-
If dir is equal to 0, buf is set equal to a reference to AboveRow.
-
Otherwise (dir is equal to 1), buf is set equal to a reference to LeftCol.
Note: buf is a reference to either AboveRow or LeftCol. "reference" indicates that modifying values in buf modifies values in the original array. {:.alert .alert-info }
When the process starts, entries -1 to numPx-1 are valid in buf and contain the original values. When the process completes, entries -2 to 2*numPx-2 are valid in buf and contain the upsampled values.
An array dup of length numPx+3 is generated by extending buf by one sample at the start and end as follows:
dup[ 0 ] = buf[ -1 ]
for (i = -1; i < numPx; i++) {
dup[ i + 2 ] = buf[ i ]
}
dup[ numPx + 2 ] = buf[ numPx - 1 ]The upsampling process (modifying values in buf) is specified as follows:
buf[-2] = dup[0]
for (i = 0; i < numPx; i++) {
s = -dup[i] + (9 * dup[i + 1]) + (9 * dup[i + 2]) - dup[i + 3]
s = Clip1( Round2(s, 4) )
buf[ 2 * i - 1 ] = s
buf[ 2 * i ] = dup[i + 2]
}The inputs to this process are:
-
a size sz,
-
a filter strength strength between 0 and 3 inclusive,
-
an edge direction left (when equal to 1, it specifies a vertical edge; when equal to 0, it specifies a horizontal edge.
The process filters the LeftCol (if left is equal to 1) or AboveRow (if left is equal to 0) arrays.
If strength is equal to 0, the process returns without doing anything.
Otherwise (strength is not equal to 0), the following ordered steps apply for i = 1..sz-2:
-
The variable s is set equal to 0.
-
The following steps now apply for j = 0..INTRA_EDGE_TAPS-1.
a. The variable k is set equal to Clip3( 0, sz - 1, i - 2 + j ).
b. The variable s is incremented by Intra_Edge_Kernel[ strength - 1 ][ j ] * ( left ? LeftCol[ k - 1 ] : AboveRow[ k - 1 ] ).
-
If left is equal to 1, LeftCol[ i - 1 ] is set equal to ( s + 8 ) >> 4.
-
If left is equal to 0, AboveRow[ i - 1 ] is set equal to ( s + 8 ) >> 4.
The array Intra_Edge_Kernel is specified as follows:
Intra_Edge_Kernel[INTRA_EDGE_KERNELS][INTRA_EDGE_TAPS] = {
{ 0, 4, 8, 4, 0 },
{ 0, 5, 6, 5, 0 },
{ 2, 4, 4, 4, 2 }
}The inter prediction process is invoked for inter coded blocks and interintra blocks. The inputs to this process are:
-
a variable plane specifying which plane is being predicted,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables w and h specifying the width and height of the region to be predicted,
-
variables candRow and candCol specifying the location (in units of 4x4 blocks) of the motion vector information to be used.
The outputs of this process are predicted samples in the current frame CurrFrame.
This process is triggered by a function call to predict_inter.
The variable isInterInter is set equal to RefFrames[ candRow ][ candCol ][ 1 ] > INTRA_FRAME.
The variable isCompound is set equal to RefFrames[ candRow ][ candCol ][ 1 ] >= INTRA_FRAME.
The prediction arrays are formed by the following ordered steps:
-
The variable refList is set equal to 0.
-
The variable useWarp is derived as follows:
-
If w < 8 or h < 8, useWarp is set equal to 0.
-
Otherwise, if force_integer_mv is equal to 1, useWarp is set equal to 0.
-
Otherwise, if motion_mode is equal to LOCALWARP, useWarp is set equal to 1.
-
Otherwise, if (YMode == GLOBALMV || YMode == GLOBAL_GLOBALMV) and gm_type[ RefFrame[ refList ] ] > TRANSLATION, useWarp is set equal to 1.
-
Otherwise, useWarp is set equal to 0.
-
-
If setting useWarp equal to 1 would result in the bitstream not being conformant (due to either the requirement in the warp estimation process in [section 7.8.2.6][] about the determinant being non-zero, or the requirement in the setup shear process in [section 7.8.2.4][] about the size of the shear) then useWarp is set equal to 0.
-
If plane is equal to 0 and motion_mode is equal to LOCALWARP and useWarp is equal to 1, the warp estimation process in [section 7.8.2.6][] is invoked.
-
The motion vector array mv is set equal to Mvs[ candRow ][ candCol ][ refList ].
-
The variable refIdx specifying which reference frame is being used is set as follows:
-
If use_intrabc is equal to 0, refIdx is set equal to ref_frame_idx[ RefFrame[ refList ] - LAST_FRAME ].
-
Otherwise (use_intrabc is equal to 1), refIdx is set equal to -1 and RefFrameWidth[ -1 ] is set equal to FrameWidth and RefFrameHeight[ -1 ] is set equal to FrameHeight.
-
-
The motion vector scaling process in [section 7.8.2.1][] is invoked with plane, refIdx, x, y, mv as inputs and the output being the initial location startX, startY, and the step sizes stepX, stepY.
-
If useWarp is equal to 1, the block warp process in [section 7.8.2.3][] is invoked with plane, refList, x, y, i8, j8, w, h as inputs and the output is merged into the 2D array preds[ refList ] for i8 = 0..((h-1) >> 3) and for j8 = 0..((w-1) >> 3). (Each invocation fills in a block of output of size w by h at x offset j8 * 8 and y offset i8 * 8.)
-
If useWarp is equal to 0, the block inter prediction process in [section 7.8.2.2][] is invoked with plane, refIdx, startX, startY, stepX, stepY, w, h, candRow, candCol as inputs and the output is assigned to the 2D array preds[ refList ].
-
If isInterInter is equal to 1, then the variable refList is set equal to 1 and steps 4 to 9 are repeated to form the prediction for the second reference.
If IsInterIntra is equal to 1, the following ordered steps apply:
-
The variable mode is set as follows:
-
If interintra_mode is equal to II_DC_PRED, mode is set equal to DC_PRED.
-
Otherwise if interintra_mode is equal to II_V_PRED, mode is set equal to V_PRED,
-
Otherwise if interintra_mode is equal to II_H_PRED, mode is set equal to H_PRED,
-
Otherwise (interintra_mode is equal to II_SMOOTH_PRED), mode is set equal to SMOOTH_PRED.
-
-
The variable haveLeft is set equal to AvailL
-
The variable haveAbove is set equal to AvailU
-
The variable haveAboveRight is set equal to 0
-
The variable haveBelowLeft is set equal to 0
-
The variable log2W is set equal to log2( w )
-
The variable log2H is set equal to log2( h )
-
The intra prediction process specified in [section 7.8.1][] is invoked with plane, x, y, haveLeft, haveAbove, haveAboveRight, haveBelowLeft, mode, log2W, log2H as inputs. (This will write intra predicted samples into CurrFrame.)
An array named Mask is prepared as follows:
-
If compound_type is equal to COMPOUND_WEDGE and plane is equal to 0, the wedge mask process in [section 7.8.2.9][] is invoked with w, h as inputs.
-
Otherwise if compound_type is equal to COMPOUND_INTRA, the inter intra mask process in [section 7.8.2.11][] is invoked with preds, plane, x, y, w, h as inputs.
-
Otherwise if compound_type is equal to COMPOUND_SEG and plane is equal to 0, the segment mask process in [section 7.8.2.10][] is invoked with preds, w, h as inputs.
-
Otherwise, no mask array is needed.
If compound_type is equal to COMPOUND_DISTANCE, the distance weights process in [section TODO] is invoked with candRow and candCol as inputs.
The inter predicted samples are then derived as follows:
-
If isCompound is equal to 0, CurrFrame[ plane ][ y + i ][ x + j ] is set equal to Clip1( Round2( preds[ 0 ][ i ][ j ], 9) ) for i = 0..h-1 and j = 0..w-1.
-
Otherwise if compound_type is equal to COMPOUND_AVERAGE, CurrFrame[ plane ][ y + i ][ x + j ] is set equal to Clip1( Round2( preds[ 0 ][ i ][ j ] + preds[ 1 ][ i ][ j ], 10 ) ) for i = 0..h-1 and j = 0..w-1.
-
Otherwise if compound_type is equal to COMPOUND_DISTANCE, CurrFrame[ plane ][ y + i ][ x + j ] is set equal to Clip1( Round2( Round2( FwdWeight * preds[ 0 ][ i ][ j ] + BckWeight * preds[ 1 ][ i ][ j ], 3 ), 10 ) ) for i = 0..h-1 and j = 0..w-1.
-
Otherwise, the mask blend process in [section 7.8.2.12][] is invoked with preds, plane, x, y, w, h as inputs.
If motion_mode is equal to OBMC, the overlapped motion compensation in [section 7.8.2.7][] is invoked with plane, x, y, w, h as inputs.
The inputs to this process are:
-
a variable plane specifying which plane is being predicted,
-
a variable refIdx specifying which reference frame is being used,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
a variable mv specifying the clamped motion vector (in units of 1/8 th of a luma pixel, i.e. with 3 fractional bits).
The outputs of this process are the variables startX and startY giving the reference block location in units of 1/1024 th of a sample, and variables xStep and yStep giving the step size in units of 1/1024 th of a sample.
This process is responsible for computing the sampling locations in the reference frame based on the motion vector. The sampling locations are also adjusted to compensate for any difference in the size of the reference frame compared to the current frame.
Note: When intra block copy is being used, refIdx will be equal to -1 to signal that we need to predict from the frame currently being decoded. The arrays RefFrameWidth and RefFrameHeight include values at index -1 giving the dimensions of the current frame. {:.alert .alert-info }
It is a requirement of bitstream conformance that all the following conditions are satisfied:
-
2 * FrameWidth >= RefFrameWidth[ refIdx ]
-
2 * FrameHeight >= RefFrameHeight[ refIdx ]
-
FrameWidth <= 16 * RefFrameWidth[ refIdx ]
-
FrameHeight <= 16 * RefFrameHeight[ refIdx ]
A variable xScale is set equal to (RefFrameWidth[ refIdx ] << REF_SCALE_SHIFT + ( FrameWidth / 2 ) ) / FrameWidth.
A variable yScale is set equal to (RefFrameHeight[ refIdx ] << REF_SCALE_SHIFT + ( FrameHeight / 2 ) ) / FrameHeight.
(xScale and yScale specify the size of the reference frame relative to the current frame in units where (1 << 14) is equivalent to both frames having the same size.)
The variables subX and subY are set equal to the subsampling for the current plane as follows:
-
If plane is equal to 0, subX is set equal to 0 and subY is set equal to 0.
-
Otherwise, subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The variable halfSample (representing half the size of a sample in units of 1/16 th of a sample) is set equal to ( 1 << ( SUBPEL_BITS - 1 ) ).
The variable origX is set equal to ( (x << SUBPEL_BITS) + ( ( 2 * mv[1] ) >> subX ) + halfSample ).
The variable origY is set equal to ( (y << SUBPEL_BITS) + ( ( 2 * mv[0] ) >> subY ) + halfSample ).
(origX and origY specify the location of the centre of the sample at the top-left corner of the reference block in the current frame's coordinate system in units of 1/16 th of a sample, i.e. with SUBPEL_BITS=4 fractional bits.)
The variable baseX is set equal to (origX * xScale - ( halfSample << REF_SCALE_SHIFT ) ).
The variable baseY is set equal to (origY * yScale - ( halfSample << REF_SCALE_SHIFT ) ).
(baseX and baseY specify the location of the top-left corner of the block in the reference frame in the reference frame's coordinate system with 18 fractional bits.)
The variable off (containing a rounding offset for the filter tap selection) is set equal to ( ( 1 << (SCALE_SUBPEL_BITS - SUBPEL_BITS) ) / 2 ).
The output variable startX is set equal to (Round2Signed( baseX, REF_SCALE_SHIFT + SUBPEL_BITS - SCALE_SUBPEL_BITS) + off).
The output variable startY is set equal to (Round2Signed( baseY, REF_SCALE_SHIFT + SUBPEL_BITS - SCALE_SUBPEL_BITS) + off).
(startX and startY specify the location of the top-left corner of the block in the reference frame in the reference frame's coordinate system with SCALE_SUBPEL_BITS=10 fractional bits.)
The output variable stepX is set equal to Round2Signed( xScale, REF_SCALE_SHIFT - SCALE_SUBPEL_BITS).
The output variable stepY is set equal to Round2Signed( yScale, REF_SCALE_SHIFT - SCALE_SUBPEL_BITS).
(stepX and stepY are the size of one current frame sample in the reference frame's coordinate system with 10 fractional bits.)
The inputs to this process are:
-
a variable plane,
-
a variable refIdx specifying which reference frame is being used (or -1 for intra block copy),
-
variables x and y giving the block location with in units of 1/1024 th of a sample,
-
variables xStep and yStep giving the step size in units of 1/1024 th of a sample,
-
variables w and h giving the width and height of the block in units of samples,
-
variables candRow and candCol specifying the location (in units of 4x4 blocks) of the motion vector information to be used.
The output from this process is the 2D array named pred containing inter predicted samples.
A variable ref specifying the reference frame contents is set as follows:
-
If refIdx is equal to -1, ref is set equal to CurrFrame.
-
Otherwise (refIdx is greater than or equal to 0), ref is set equal to FrameStore[ refIdx ].
The variables subX and subY are set equal to the subsampling for the current plane as follows:
-
If plane is equal to 0, subX is set equal to 0 and subY is set equal to 0.
-
Otherwise, subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The variable lastX is set equal to ( (RefFrameWidth[ refIdx ] + subX) >> subX) - 1.
The variable lastY is set equal to ( (RefFrameHeight[ refIdx ] + subY) >> subY) - 1.
(lastX and lastY specify the coordinates of the bottom right sample of the reference plane.)
The variable intermediateHeight specifying the height required for the intermediate array is set equal to (((h - 1) * yStep + (1 << SCALE_SUBPEL_BITS) - 1) >> SCALE_SUBPEL_BITS) + 8.
The sub-sample interpolation is effected via two one-dimensional convolutions. First a horizontal filter is used to build up a temporary array, and then this array is vertically filtered to obtain the final prediction. The fractional parts of the motion vectors determine the filtering process. If the fractional part is zero, then the filtering is equivalent to a straight sample copy.
The filtering is applied as follows:
-
The array intermediate is specified as follows:
interpFilter = InterpFilters[ candRow ][ candCol ][ 1 ] if ( w <= 4 ) { if (interpFilter == EIGHTTAP || interpFilter == EIGHTTAP_SHARP) { interpFilter = 4 } else if (interpFilter == EIGHTTAP_SMOOTH) { interpFilter = 5 } } for ( r = 0; r < intermediateHeight; r++ ) { for ( c = 0; c < w; c++ ) { s = 0 p = x + xStep * c for ( t = 0; t < 8; t++ ) s += Subpel_Filters[ interpFilter ][ (p >> 6) & 15 ][ t ] * ref[ plane ] [ Clip3( 0, lastY, (y >> 10) + r - 3 ) ] [ Clip3( 0, lastX, (p >> 10) + t - 3 ) ] intermediate[ r ][ c ] = Round2(s, 5) } }
-
The array pred is specified as follows:
interpFilter = InterpFilters[ candRow ][ candCol ][ 0 ] if ( h <= 4 ) { if (interpFilter == EIGHTTAP || interpFilter == EIGHTTAP_SHARP) { interpFilter = 4 } else if (interpFilter == EIGHTTAP_SMOOTH) { interpFilter = 5 } } for ( r = 0; r < h; r++ ) { for ( c = 0; c < w; c++ ) { s = 0 p = (y & 1023) + yStep * r for ( t = 0; t < 8; t++ ) s += Subpel_Filters[ interpFilter ][ (p >> 6) & 15 ][ t ] * intermediate[ (p >> 10) + t ][ c ] pred[ r ][ c ] = s } }
where the constant array Subpel_Filters is specified as:
Subpel_Filters[ 6 ][ 16 ][ 8 ] = { { { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 2, -6, 126, 8, -2, 0, 0 }, { 0, 2, -10, 122, 18, -4, 0, 0 }, { 0, 2, -12, 116, 28, -8, 2, 0 }, { 0, 2, -14, 110, 38, -10, 2, 0 }, { 0, 2, -14, 102, 48, -12, 2, 0 }, { 0, 2, -16, 94, 58, -12, 2, 0 }, { 0, 2, -14, 84, 66, -12, 2, 0 }, { 0, 2, -14, 76, 76, -14, 2, 0 }, { 0, 2, -12, 66, 84, -14, 2, 0 }, { 0, 2, -12, 58, 94, -16, 2, 0 }, { 0, 2, -12, 48, 102, -14, 2, 0 }, { 0, 2, -10, 38, 110, -14, 2, 0 }, { 0, 2, -8, 28, 116, -12, 2, 0 }, { 0, 0, -4, 18, 122, -10, 2, 0 }, { 0, 0, -2, 8, 126, -6, 2, 0 } }, { { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 2, 28, 62, 34, 2, 0, 0 }, { 0, 0, 26, 62, 36, 4, 0, 0 }, { 0, 0, 22, 62, 40, 4, 0, 0 }, { 0, 0, 20, 60, 42, 6, 0, 0 }, { 0, 0, 18, 58, 44, 8, 0, 0 }, { 0, 0, 16, 56, 46, 10, 0, 0 }, { 0, -2, 16, 54, 48, 12, 0, 0 }, { 0, -2, 14, 52, 52, 14, -2, 0 }, { 0, 0, 12, 48, 54, 16, -2, 0 }, { 0, 0, 10, 46, 56, 16, 0, 0 }, { 0, 0, 8, 44, 58, 18, 0, 0 }, { 0, 0, 6, 42, 60, 20, 0, 0 }, { 0, 0, 4, 40, 62, 22, 0, 0 }, { 0, 0, 4, 36, 62, 26, 0, 0 }, { 0, 0, 2, 34, 62, 28, 2, 0 } }, { { 0, 0, 0, 128, 0, 0, 0, 0 }, { -2, 2, -6, 126, 8, -2, 2, 0 }, { -2, 6, -12, 124, 16, -6, 4, -2 }, { -2, 8, -18, 120, 26, -10, 6, -2 }, { -4, 10, -22, 116, 38, -14, 6, -2 }, { -4, 10, -22, 108, 48, -18, 8, -2 }, { -4, 10, -24, 100, 60, -20, 8, -2 }, { -4, 10, -24, 90, 70, -22, 10, -2 }, { -4, 12, -24, 80, 80, -24, 12, -4 }, { -2, 10, -22, 70, 90, -24, 10, -4 }, { -2, 8, -20, 60, 100, -24, 10, -4 }, { -2, 8, -18, 48, 108, -22, 10, -4 }, { -2, 6, -14, 38, 116, -22, 10, -4 }, { -2, 6, -10, 26, 120, -18, 8, -2 }, { -2, 4, -6, 16, 124, -12, 6, -2 }, { 0, 2, -2, 8, 126, -6, 2, -2 } }, { { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 0, 0, 120, 8, 0, 0, 0 }, { 0, 0, 0, 112, 16, 0, 0, 0 }, { 0, 0, 0, 104, 24, 0, 0, 0 }, { 0, 0, 0, 96, 32, 0, 0, 0 }, { 0, 0, 0, 88, 40, 0, 0, 0 }, { 0, 0, 0, 80, 48, 0, 0, 0 }, { 0, 0, 0, 72, 56, 0, 0, 0 }, { 0, 0, 0, 64, 64, 0, 0, 0 }, { 0, 0, 0, 56, 72, 0, 0, 0 }, { 0, 0, 0, 48, 80, 0, 0, 0 }, { 0, 0, 0, 40, 88, 0, 0, 0 }, { 0, 0, 0, 32, 96, 0, 0, 0 }, { 0, 0, 0, 24, 104, 0, 0, 0 }, { 0, 0, 0, 16, 112, 0, 0, 0 }, { 0, 0, 0, 8, 120, 0, 0, 0 } }, { { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 0, -4, 126, 8, -2, 0, 0 }, { 0, 0, -8, 122, 18, -4, 0, 0 }, { 0, 0, -10, 116, 28, -6, 0, 0 }, { 0, 0, -12, 110, 38, -8, 0, 0 }, { 0, 0, -12, 102, 48, -10, 0, 0 }, { 0, 0, -14, 94, 58, -10, 0, 0 }, { 0, 0, -12, 84, 66, -10, 0, 0 }, { 0, 0, -12, 76, 76, -12, 0, 0 }, { 0, 0, -10, 66, 84, -12, 0, 0 }, { 0, 0, -10, 58, 94, -14, 0, 0 }, { 0, 0, -10, 48, 102, -12, 0, 0 }, { 0, 0, -8, 38, 110, -12, 0, 0 }, { 0, 0, -6, 28, 116, -10, 0, 0 }, { 0, 0, -4, 18, 122, -8, 0, 0 }, { 0, 0, -2, 8, 126, -4, 0, 0 } }, { { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 0, 30, 62, 34, 2, 0, 0 }, { 0, 0, 26, 62, 36, 4, 0, 0 }, { 0, 0, 22, 62, 40, 4, 0, 0 }, { 0, 0, 20, 60, 42, 6, 0, 0 }, { 0, 0, 18, 58, 44, 8, 0, 0 }, { 0, 0, 16, 56, 46, 10, 0, 0 }, { 0, 0, 14, 54, 48, 12, 0, 0 }, { 0, 0, 12, 52, 52, 12, 0, 0 }, { 0, 0, 12, 48, 54, 14, 0, 0 }, { 0, 0, 10, 46, 56, 16, 0, 0 }, { 0, 0, 8, 44, 58, 18, 0, 0 }, { 0, 0, 6, 42, 60, 20, 0, 0 }, { 0, 0, 4, 40, 62, 22, 0, 0 }, { 0, 0, 4, 36, 62, 26, 0, 0 }, { 0, 0, 2, 34, 62, 30, 0, 0 } } }
Note: All the values in Subpel_Filters are even. The last two filter types are used for small blocks and only have four filter taps. The filter at index 4 has a four tap version of the EIGHTTAP filter. The filter at index 5 has a four tap version of the EIGHTAP_SMOOTH filter. {:.alert .alert-info }
The inputs to this process are:
-
a variable plane,
-
a variable refList specifying that we should predict from RefFrame[ refList ],
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables i8 and j8 specifying the offset (in units of 8 samples) relative to the top left sample,
-
variables w and h giving the width and height of the block in units of samples.
The output from this process is the 2D array named pred containing warped inter predicted samples.
The process only updates a section of the pred array. The size of the updated section is 8x8 samples, clipped to the size of the block. Variables i8 and j8 give the location of the section to update.
A variable refIdx specifying which reference frame is being used is set equal to ref_frame_idx[ RefFrame[ refList ] - LAST_FRAME ].
A variable ref specifying the reference frame contents is set equal to FrameStore[ refIdx ].
The variables subX and subY are set equal to the subsampling for the current plane as follows:
-
If plane is equal to 0, subX is set equal to 0 and subY is set equal to 0.
-
Otherwise, subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The variable lastX is set equal to ( (RefFrameWidth[ refIdx ] + subX) >> subX) - 1.
The variable lastY is set equal to ( (RefFrameHeight[ refIdx ] + subY) >> subY) - 1.
(lastX and lastY specify the coordinates of the bottom right sample of the reference plane.)
The variable srcX is set equal to (j8 * 8 + 4) << subX.
The variable srcY is set equal to (i8 * 8 + 4) << subY.
(srcX and srcY specify a location in the luma plane that will be projected using the warp parameters.)
The array warpParams is specified as follows:
-
If motion_mode is equal to LOCALWARP, warpParams is set equal to LocalWarpParams.
-
Otherwise (motion_mode is not equal to LOCALWARP), warpParams is set equal to gm_params[ RefFrame[ refList ] ].
The variable dstX is set equal to warpParams[2] * srcX + warpParams[3] * srcY + warpParams[0].
The variable dstY is set equal to warpParams[4] * srcX + warpParams[5] * srcY + warpParams[1].
(dstX and dstY specify the destination location in the luma plane using WARPEDMODEL_PREC_BITS bits of precision).
The setup shear process specified in [section 7.8.2.4][] is invoked with warpParams as input, and the outputs are assigned to alpha, beta, gamma, and delta.
The sub-sample interpolation is effected via two one-dimensional convolutions. First a horizontal filter is used to build up a temporary array, and then this array is vertically filtered to obtain the final prediction.
The filtering is applied as follows:
-
The array intermediate is specified as follows:
x4 = dstX >> subX y4 = dstY >> subY ix4 = x4 >> WARPEDMODEL_PREC_BITS sx4 = x4 & ((1 << WARPEDMODEL_PREC_BITS) - 1) iy4 = y4 >> WARPEDMODEL_PREC_BITS sy4 = y4 & ((1 << WARPEDMODEL_PREC_BITS) - 1) for (i1 = -7; i1 < 8; i1++) { for (i2 = -4; i2 < 4; i2++) { sx = sx4 + alpha * i2 + beta * i1 offs = Round2(sx, WARPEDDIFF_PREC_BITS) + WARPEDPIXEL_PREC_SHIFTS s = 0 for (i3 = 0; i3 < 8; i3++) { s += Warped_Filters[ offs ][ i3 ] * ref[ plane ][ Clip3( 0, lastY, iy4 + i1 ) ] [ Clip3( 0, lastX, ix4 + i2 - 3 + i3 ) ] } intermediate[(i1 + 7)][(i2 + 4)] = Round2(s, 7) } }
-
The array pred is specified as follows:
for (i1 = -4; i1 < Min(4, h - i8 * 8 - 4); i1++) { for (i2 = -4; i2 < Min(4, w - j8 * 8 - 4); i2++) { sy = sy4 + gamma * i2 + delta * i1 offs = Round2(sy, WARPEDDIFF_PREC_BITS) + WARPEDPIXEL_PREC_SHIFTS s = 0 for (i3 = 0; i3 < 8; i3++) { s += Warped_Filters[offs][i3] * intermediate[(i1 + i3 + 4)][(i2 + 4)] } pred[ i8 * 8 + i1 + 4 ][ j8 * 8 + i2 + 4 ] = s } }
where the constant array Warped_Filters is specified as:
Warped_Filters[WARPEDPIXEL_PREC_SHIFTS * 3 + 1][8] = { { 0, 0, 127, 1, 0, 0, 0, 0 }, { 0, - 1, 127, 2, 0, 0, 0, 0 }, { 1, - 3, 127, 4, - 1, 0, 0, 0 }, { 1, - 4, 126, 6, - 2, 1, 0, 0 }, { 1, - 5, 126, 8, - 3, 1, 0, 0 }, { 1, - 6, 125, 11, - 4, 1, 0, 0 }, { 1, - 7, 124, 13, - 4, 1, 0, 0 }, { 2, - 8, 123, 15, - 5, 1, 0, 0 }, { 2, - 9, 122, 18, - 6, 1, 0, 0 }, { 2, -10, 121, 20, - 6, 1, 0, 0 }, { 2, -11, 120, 22, - 7, 2, 0, 0 }, { 2, -12, 119, 25, - 8, 2, 0, 0 }, { 3, -13, 117, 27, - 8, 2, 0, 0 }, { 3, -13, 116, 29, - 9, 2, 0, 0 }, { 3, -14, 114, 32, -10, 3, 0, 0 }, { 3, -15, 113, 35, -10, 2, 0, 0 }, { 3, -15, 111, 37, -11, 3, 0, 0 }, { 3, -16, 109, 40, -11, 3, 0, 0 }, { 3, -16, 108, 42, -12, 3, 0, 0 }, { 4, -17, 106, 45, -13, 3, 0, 0 }, { 4, -17, 104, 47, -13, 3, 0, 0 }, { 4, -17, 102, 50, -14, 3, 0, 0 }, { 4, -17, 100, 52, -14, 3, 0, 0 }, { 4, -18, 98, 55, -15, 4, 0, 0 }, { 4, -18, 96, 58, -15, 3, 0, 0 }, { 4, -18, 94, 60, -16, 4, 0, 0 }, { 4, -18, 91, 63, -16, 4, 0, 0 }, { 4, -18, 89, 65, -16, 4, 0, 0 }, { 4, -18, 87, 68, -17, 4, 0, 0 }, { 4, -18, 85, 70, -17, 4, 0, 0 }, { 4, -18, 82, 73, -17, 4, 0, 0 }, { 4, -18, 80, 75, -17, 4, 0, 0 }, { 4, -18, 78, 78, -18, 4, 0, 0 }, { 4, -17, 75, 80, -18, 4, 0, 0 }, { 4, -17, 73, 82, -18, 4, 0, 0 }, { 4, -17, 70, 85, -18, 4, 0, 0 }, { 4, -17, 68, 87, -18, 4, 0, 0 }, { 4, -16, 65, 89, -18, 4, 0, 0 }, { 4, -16, 63, 91, -18, 4, 0, 0 }, { 4, -16, 60, 94, -18, 4, 0, 0 }, { 3, -15, 58, 96, -18, 4, 0, 0 }, { 4, -15, 55, 98, -18, 4, 0, 0 }, { 3, -14, 52, 100, -17, 4, 0, 0 }, { 3, -14, 50, 102, -17, 4, 0, 0 }, { 3, -13, 47, 104, -17, 4, 0, 0 }, { 3, -13, 45, 106, -17, 4, 0, 0 }, { 3, -12, 42, 108, -16, 3, 0, 0 }, { 3, -11, 40, 109, -16, 3, 0, 0 }, { 3, -11, 37, 111, -15, 3, 0, 0 }, { 2, -10, 35, 113, -15, 3, 0, 0 }, { 3, -10, 32, 114, -14, 3, 0, 0 }, { 2, - 9, 29, 116, -13, 3, 0, 0 }, { 2, - 8, 27, 117, -13, 3, 0, 0 }, { 2, - 8, 25, 119, -12, 2, 0, 0 }, { 2, - 7, 22, 120, -11, 2, 0, 0 }, { 1, - 6, 20, 121, -10, 2, 0, 0 }, { 1, - 6, 18, 122, - 9, 2, 0, 0 }, { 1, - 5, 15, 123, - 8, 2, 0, 0 }, { 1, - 4, 13, 124, - 7, 1, 0, 0 }, { 1, - 4, 11, 125, - 6, 1, 0, 0 }, { 1, - 3, 8, 126, - 5, 1, 0, 0 }, { 1, - 2, 6, 126, - 4, 1, 0, 0 }, { 0, - 1, 4, 127, - 3, 1, 0, 0 }, { 0, 0, 2, 127, - 1, 0, 0, 0 }, { 0, 0, 0, 127, 1, 0, 0, 0}, { 0, 0, -1, 127, 2, 0, 0, 0}, { 0, 1, -3, 127, 4, -2, 1, 0}, { 0, 1, -5, 127, 6, -2, 1, 0}, { 0, 2, -6, 126, 8, -3, 1, 0}, {-1, 2, -7, 126, 11, -4, 2, -1}, {-1, 3, -8, 125, 13, -5, 2, -1}, {-1, 3, -10, 124, 16, -6, 3, -1}, {-1, 4, -11, 123, 18, -7, 3, -1}, {-1, 4, -12, 122, 20, -7, 3, -1}, {-1, 4, -13, 121, 23, -8, 3, -1}, {-2, 5, -14, 120, 25, -9, 4, -1}, {-1, 5, -15, 119, 27, -10, 4, -1}, {-1, 5, -16, 118, 30, -11, 4, -1}, {-2, 6, -17, 116, 33, -12, 5, -1}, {-2, 6, -17, 114, 35, -12, 5, -1}, {-2, 6, -18, 113, 38, -13, 5, -1}, {-2, 7, -19, 111, 41, -14, 6, -2}, {-2, 7, -19, 110, 43, -15, 6, -2}, {-2, 7, -20, 108, 46, -15, 6, -2}, {-2, 7, -20, 106, 49, -16, 6, -2}, {-2, 7, -21, 104, 51, -16, 7, -2}, {-2, 7, -21, 102, 54, -17, 7, -2}, {-2, 8, -21, 100, 56, -18, 7, -2}, {-2, 8, -22, 98, 59, -18, 7, -2}, {-2, 8, -22, 96, 62, -19, 7, -2}, {-2, 8, -22, 94, 64, -19, 7, -2}, {-2, 8, -22, 91, 67, -20, 8, -2}, {-2, 8, -22, 89, 69, -20, 8, -2}, {-2, 8, -22, 87, 72, -21, 8, -2}, {-2, 8, -21, 84, 74, -21, 8, -2}, {-2, 8, -22, 82, 77, -21, 8, -2}, {-2, 8, -21, 79, 79, -21, 8, -2}, {-2, 8, -21, 77, 82, -22, 8, -2}, {-2, 8, -21, 74, 84, -21, 8, -2}, {-2, 8, -21, 72, 87, -22, 8, -2}, {-2, 8, -20, 69, 89, -22, 8, -2}, {-2, 8, -20, 67, 91, -22, 8, -2}, {-2, 7, -19, 64, 94, -22, 8, -2}, {-2, 7, -19, 62, 96, -22, 8, -2}, {-2, 7, -18, 59, 98, -22, 8, -2}, {-2, 7, -18, 56, 100, -21, 8, -2}, {-2, 7, -17, 54, 102, -21, 7, -2}, {-2, 7, -16, 51, 104, -21, 7, -2}, {-2, 6, -16, 49, 106, -20, 7, -2}, {-2, 6, -15, 46, 108, -20, 7, -2}, {-2, 6, -15, 43, 110, -19, 7, -2}, {-2, 6, -14, 41, 111, -19, 7, -2}, {-1, 5, -13, 38, 113, -18, 6, -2}, {-1, 5, -12, 35, 114, -17, 6, -2}, {-1, 5, -12, 33, 116, -17, 6, -2}, {-1, 4, -11, 30, 118, -16, 5, -1}, {-1, 4, -10, 27, 119, -15, 5, -1}, {-1, 4, -9, 25, 120, -14, 5, -2}, {-1, 3, -8, 23, 121, -13, 4, -1}, {-1, 3, -7, 20, 122, -12, 4, -1}, {-1, 3, -7, 18, 123, -11, 4, -1}, {-1, 3, -6, 16, 124, -10, 3, -1}, {-1, 2, -5, 13, 125, -8, 3, -1}, {-1, 2, -4, 11, 126, -7, 2, -1}, { 0, 1, -3, 8, 126, -6, 2, 0}, { 0, 1, -2, 6, 127, -5, 1, 0}, { 0, 1, -2, 4, 127, -3, 1, 0}, { 0, 0, 0, 2, 127, -1, 0, 0}, { 0, 0, 0, 1, 127, 0, 0, 0 }, { 0, 0, 0, - 1, 127, 2, 0, 0 }, { 0, 0, 1, - 3, 127, 4, - 1, 0 }, { 0, 0, 1, - 4, 126, 6, - 2, 1 }, { 0, 0, 1, - 5, 126, 8, - 3, 1 }, { 0, 0, 1, - 6, 125, 11, - 4, 1 }, { 0, 0, 1, - 7, 124, 13, - 4, 1 }, { 0, 0, 2, - 8, 123, 15, - 5, 1 }, { 0, 0, 2, - 9, 122, 18, - 6, 1 }, { 0, 0, 2, -10, 121, 20, - 6, 1 }, { 0, 0, 2, -11, 120, 22, - 7, 2 }, { 0, 0, 2, -12, 119, 25, - 8, 2 }, { 0, 0, 3, -13, 117, 27, - 8, 2 }, { 0, 0, 3, -13, 116, 29, - 9, 2 }, { 0, 0, 3, -14, 114, 32, -10, 3 }, { 0, 0, 3, -15, 113, 35, -10, 2 }, { 0, 0, 3, -15, 111, 37, -11, 3 }, { 0, 0, 3, -16, 109, 40, -11, 3 }, { 0, 0, 3, -16, 108, 42, -12, 3 }, { 0, 0, 4, -17, 106, 45, -13, 3 }, { 0, 0, 4, -17, 104, 47, -13, 3 }, { 0, 0, 4, -17, 102, 50, -14, 3 }, { 0, 0, 4, -17, 100, 52, -14, 3 }, { 0, 0, 4, -18, 98, 55, -15, 4 }, { 0, 0, 4, -18, 96, 58, -15, 3 }, { 0, 0, 4, -18, 94, 60, -16, 4 }, { 0, 0, 4, -18, 91, 63, -16, 4 }, { 0, 0, 4, -18, 89, 65, -16, 4 }, { 0, 0, 4, -18, 87, 68, -17, 4 }, { 0, 0, 4, -18, 85, 70, -17, 4 }, { 0, 0, 4, -18, 82, 73, -17, 4 }, { 0, 0, 4, -18, 80, 75, -17, 4 }, { 0, 0, 4, -18, 78, 78, -18, 4 }, { 0, 0, 4, -17, 75, 80, -18, 4 }, { 0, 0, 4, -17, 73, 82, -18, 4 }, { 0, 0, 4, -17, 70, 85, -18, 4 }, { 0, 0, 4, -17, 68, 87, -18, 4 }, { 0, 0, 4, -16, 65, 89, -18, 4 }, { 0, 0, 4, -16, 63, 91, -18, 4 }, { 0, 0, 4, -16, 60, 94, -18, 4 }, { 0, 0, 3, -15, 58, 96, -18, 4 }, { 0, 0, 4, -15, 55, 98, -18, 4 }, { 0, 0, 3, -14, 52, 100, -17, 4 }, { 0, 0, 3, -14, 50, 102, -17, 4 }, { 0, 0, 3, -13, 47, 104, -17, 4 }, { 0, 0, 3, -13, 45, 106, -17, 4 }, { 0, 0, 3, -12, 42, 108, -16, 3 }, { 0, 0, 3, -11, 40, 109, -16, 3 }, { 0, 0, 3, -11, 37, 111, -15, 3 }, { 0, 0, 2, -10, 35, 113, -15, 3 }, { 0, 0, 3, -10, 32, 114, -14, 3 }, { 0, 0, 2, - 9, 29, 116, -13, 3 }, { 0, 0, 2, - 8, 27, 117, -13, 3 }, { 0, 0, 2, - 8, 25, 119, -12, 2 }, { 0, 0, 2, - 7, 22, 120, -11, 2 }, { 0, 0, 1, - 6, 20, 121, -10, 2 }, { 0, 0, 1, - 6, 18, 122, - 9, 2 }, { 0, 0, 1, - 5, 15, 123, - 8, 2 }, { 0, 0, 1, - 4, 13, 124, - 7, 1 }, { 0, 0, 1, - 4, 11, 125, - 6, 1 }, { 0, 0, 1, - 3, 8, 126, - 5, 1 }, { 0, 0, 1, - 2, 6, 126, - 4, 1 }, { 0, 0, 0, - 1, 4, 127, - 3, 1 }, { 0, 0, 0, 0, 2, 127, - 1, 0 }, { 0, 0, 0, 0, 2, 127, - 1, 0 } }
The input for this process is an array warpParams representing an affine transformation.
The outputs from this process are variables alpha, beta, gamma, delta representing two shearing operations that combine to make the full affine transformation.
The variable alpha0 is set equal to Clip3( -32768, 32767, warpParams[ 2 ] - (1 << WARPEDMODEL_PREC_BITS) ).
The variable beta0 is set equal to Clip3( -32768, 32767, warpParams[ 3 ] ).
The resolve divisor process specified in [section 7.8.2.5][] is invoked with warpParams[ 2 ] as input, and the outputs assigned to divShift and divFactor.
The variable v is set equal to ( warpParams[ 4 ] << WARPEDMODEL_PREC_BITS ).
The variable gamma0 is set equal to Clip3( -32768, 32767, Round2Signed( v * divFactor, divShift ) ).
The variable w is set equal to ( warpParams[ 3 ] * warpParams[ 4 ] ).
The variable delta0 is set equal to Clip3( -32768, 32767, warpParams[ 5 ] - Round2Signed( w * divFactor, divShift ) - (1 << WARPEDMODEL_PREC_BITS) ).
It is a requirement of bitstream conformance that 4 * Abs( alpha0 ) + 7 * Abs( beta0 ) is less than (1 << WARPEDMODEL_PREC_BITS).
It is a requirement of bitstream conformance that 4 * Abs( gamma0 ) + 4 * Abs( delta0 ) is less than (1 << WARPEDMODEL_PREC_BITS).
The output variables are set as follows:
alpha = Round2Signed( alpha0, WARP_PARAM_REDUCE_BITS ) << WARP_PARAM_REDUCE_BITS
beta = Round2Signed( beta0, WARP_PARAM_REDUCE_BITS ) << WARP_PARAM_REDUCE_BITS
gamma = Round2Signed( gamma0, WARP_PARAM_REDUCE_BITS ) << WARP_PARAM_REDUCE_BITS
delta = Round2Signed( delta0, WARP_PARAM_REDUCE_BITS ) << WARP_PARAM_REDUCE_BITSThe input for this process is a variable d.
The outputs for this process are variables divShift and divFactor that can be used to perform an approximate division by d via multiplying by divFactor and shifting right by divShift.
The variable n (representing the location of the most signficant bit in Abs(d) ) is set equal to floor( log2( Abs(d) ) ).
The variable e is set equal to Abs( d ) - ( 1 << n ).
The variable f is set as follows:
-
If n > DIV_LUT_BITS, f is set equal to Round2( e, n - DIV_LUT_BITS ).
-
Otherwise, f is set equal to e << ( DIV_LUT_BITS - n ).
The output variable divShift is set equal to ( n + DIV_LUT_PREC_BITS ).
The output variable divFactor is set as follows:
-
If d is less than 0, divFactor is set equal to -Div_Lut[ f ].
-
Otherwise, divFactor is set equal to Div_Lut[ f ].
The lookup table Div_Lut is specified as:
Div_Lut[DIV_LUT_NUM] = {
16384, 16320, 16257, 16194, 16132, 16070, 16009, 15948, 15888, 15828, 15768,
15709, 15650, 15592, 15534, 15477, 15420, 15364, 15308, 15252, 15197, 15142,
15087, 15033, 14980, 14926, 14873, 14821, 14769, 14717, 14665, 14614, 14564,
14513, 14463, 14413, 14364, 14315, 14266, 14218, 14170, 14122, 14075, 14028,
13981, 13935, 13888, 13843, 13797, 13752, 13707, 13662, 13618, 13574, 13530,
13487, 13443, 13400, 13358, 13315, 13273, 13231, 13190, 13148, 13107, 13066,
13026, 12985, 12945, 12906, 12866, 12827, 12788, 12749, 12710, 12672, 12633,
12596, 12558, 12520, 12483, 12446, 12409, 12373, 12336, 12300, 12264, 12228,
12193, 12157, 12122, 12087, 12053, 12018, 11984, 11950, 11916, 11882, 11848,
11815, 11782, 11749, 11716, 11683, 11651, 11619, 11586, 11555, 11523, 11491,
11460, 11429, 11398, 11367, 11336, 11305, 11275, 11245, 11215, 11185, 11155,
11125, 11096, 11067, 11038, 11009, 10980, 10951, 10923, 10894, 10866, 10838,
10810, 10782, 10755, 10727, 10700, 10673, 10645, 10618, 10592, 10565, 10538,
10512, 10486, 10460, 10434, 10408, 10382, 10356, 10331, 10305, 10280, 10255,
10230, 10205, 10180, 10156, 10131, 10107, 10082, 10058, 10034, 10010, 9986,
9963, 9939, 9916, 9892, 9869, 9846, 9823, 9800, 9777, 9754, 9732,
9709, 9687, 9664, 9642, 9620, 9598, 9576, 9554, 9533, 9511, 9489,
9468, 9447, 9425, 9404, 9383, 9362, 9341, 9321, 9300, 9279, 9259,
9239, 9218, 9198, 9178, 9158, 9138, 9118, 9098, 9079, 9059, 9039,
9020, 9001, 8981, 8962, 8943, 8924, 8905, 8886, 8867, 8849, 8830,
8812, 8793, 8775, 8756, 8738, 8720, 8702, 8684, 8666, 8648, 8630,
8613, 8595, 8577, 8560, 8542, 8525, 8508, 8490, 8473, 8456, 8439,
8422, 8405, 8389, 8372, 8355, 8339, 8322, 8306, 8289, 8273, 8257,
8240, 8224, 8208, 8192
}This process produces the array LocalWarpParams based on NumSamples candidates in CandList by performing a least squares fit.
A 2x2 matrix A, and two length 2 arrays Bx and By are constructed as follows:
for (i = 0; i < 2; i++) {
for (j = 0; j < 2; j++) {
A[i][j] = 0
}
Bx[i] = 0
By[i] = 0
}
w4 = Num_4x4_Blocks_Wide[MiSize]
h4 = Num_4x4_Blocks_High[MiSize]
midY = MiRow * 4 + h4 * 2 - 1
midX = MiCol * 4 + w4 * 2 - 1
suy = midY * 8
sux = midX * 8
duy = suy + Mv[0][0]
dux = sux + Mv[0][1]
for (i = 0; i < NumSamples; i++) {
sy = CandList[i][0] - suy
sx = CandList[i][1] - sux
dy = CandList[i][2] - duy
dx = CandList[i][3] - dux
if (Abs(sx - dx) < LS_MV_MAX && Abs(sy - dy) < LS_MV_MAX) {
A[0][0] += ls_product(sx, sx) + 8
A[0][1] += ls_product(sx, sy) + 4
A[1][1] += ls_product(sy, sy) + 8
Bx[0] += ls_product(sx, dx) + 8
Bx[1] += ls_product(sy, dx) + 4
By[0] += ls_product(sx, dy) + 4
By[1] += ls_product(sy, dy) + 8
}
}where ls_product is specified as:
ls_product(a, b) {
return ( (a * b) >> 2) + (a + b)
}Note: The matrix A is symmetric so entry A[1][0] is omitted. {:.alert .alert-info }
The variable det (containing the determinant of the matrix A) is set equal to A[0][0] * A[1][1] - A[0][1] * A[0][1].
It is a requirement of bitstream conformance that det is not equal to 0.
The resolve divisor process specified in [section 7.8.2.5][] is invoked with det as input, and the outputs assigned to divShift and divFactor.
The local warp parameters in LocalWarpParams are derived as follows:
divShift -= WARPEDMODEL_PREC_BITS
if (divShift < 0) {
divFactor = divFactor << (-divShift)
divShift = 0
}
LocalWarpParams[2] = diag( A[1][1] * Bx[0] - A[0][1] * Bx[1])
LocalWarpParams[3] = nondiag( -A[0][1] * Bx[0] + A[0][0] * Bx[1])
LocalWarpParams[4] = nondiag( A[1][1] * By[0] - A[0][1] * By[1])
LocalWarpParams[5] = diag( -A[0][1] * By[0] + A[0][0] * By[1])
mvx = Mv[ 0 ][ 1 ]
mvy = Mv[ 0 ][ 0 ]
vx = mvx * (1 << (WARPEDMODEL_PREC_BITS - 3)) -
(midX * (LocalWarpParams[2] - (1 << WARPEDMODEL_PREC_BITS)) + midY * LocalWarpParams[3])
vy = mvy * (1 << (WARPEDMODEL_PREC_BITS - 3)) -
(midX * LocalWarpParams[4] + midY * (LocalWarpParams[5] - (1 << WARPEDMODEL_PREC_BITS)))
LocalWarpParams[0] = Clip3(-WARPEDMODEL_TRANS_CLAMP, WARPEDMODEL_TRANS_CLAMP - 1, vx)
LocalWarpParams[1] = Clip3(-WARPEDMODEL_TRANS_CLAMP, WARPEDMODEL_TRANS_CLAMP - 1, vy)where diag and nondiag are specified to divide and clamp using divFactor and divShift as follows:
nondiag(v) {
return Clip3(-WARPEDMODEL_NONDIAGAFFINE_CLAMP + 1,
WARPEDMODEL_NONDIAGAFFINE_CLAMP - 1,
Round2Signed(v * divFactor, divShift))
}
diag(v) {
return Clip3((1 << WARPEDMODEL_PREC_BITS) - WARPEDMODEL_NONDIAGAFFINE_CLAMP + 1,
(1 << WARPEDMODEL_PREC_BITS) + WARPEDMODEL_NONDIAGAFFINE_CLAMP - 1,
Round2Signed(v * divFactor, divShift))
}The inputs to this process are:
-
a variable plane specifying which plane is being predicted,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables w and h specifying the width and height of the region to be predicted.
The outputs of this process are modified inter predicted samples in the current frame CurrFrame.
This process blends the inter predicted samples for the current block with inter predicted samples based on motion vectors from the above and left blocks.
The maximum number of overlapped predictions is limited based on the size of the block.
For small blocks, only the left neighbour will be used to form the prediction.
The variables subX and subY describing the subsampling of the current plane are derived as follows:
-
If plane is equal to 0, subX and subY are set equal to 0.
-
Otherwise (plane is not equal to 0), subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The process is specified as:
if ( AvailU ) {
if ( get_plane_residual_size( MiSize, plane ) >= BLOCK_8X8 ) {
pass = 0
w4 = Num_4x4_Blocks_Wide[ MiSize ]
x4 = MiCol
y4 = MiRow
nCount = 0
nLimit = Min(4, Mi_Width_Log2[ MiSize ])
while ( nCount < nLimit && x4 < Min( MiCols, MiCol + w4 ) ) {
candRow = MiRow - 1
candCol = x4 | 1
candSz = MiSizes[ candRow ][ candCol ]
step4 = Clip3( 2, 16, Num_4x4_Blocks_Wide[ candSz ] )
if ( RefFrames[ candRow ][ candCol ][ 0 ] > INTRA_FRAME ) {
nCount += 1
predW = Min( w, ( step4 * MI_SIZE ) >> subX )
predH = Min( h >> 1, 32 >> subY )
mask = get_obmc_mask( predH )
predict_overlap( )
}
x4 += step4
}
}
}
if ( AvailL ) {
pass = 1
h4 = Num_4x4_Blocks_High[ MiSize ]
x4 = MiCol
y4 = MiRow
nCount = 0
nLimit = Min(4, Mi_Height_Log2[ MiSize ])
while ( nCount < nLimit && y4 < Min( MiRows, MiRow + h4 ) ) {
candCol = MiCol - 1
candRow = y4 | 1
candSz = MiSizes[ candRow ][ candCol ]
step4 = Clip3( 2, 16, Num_4x4_Blocks_High[ candSz ] )
if ( RefFrames[ candRow ][ candCol ][ 0 ] > INTRA_FRAME ) {
nCount += 1
predW = Min( w >> 1, 32 >> subX )
predH = Min( h, ( step4 * MI_SIZE ) >> subY )
mask = get_obmc_mask( predW )
predict_overlap( )
}
y4 += step4
}
}When the function predict_overlap is invoked, the following ordered steps apply to form the overlap prediction for a region of size predW by predH based on the candidate motion vector:
-
The motion vector mv is set equal to Mvs[ candRow ][ candCol ][ 0 ].
-
The variable refIdx is set equal to ref_frame_idx[ RefFrames[ candRow ][ candCol ][ 0 ] - LAST_FRAME ].
-
The variable predX is set equal to (x4 * 4) >> subX.
-
The variable predY is set equal to (y4 * 4) >> subY.
-
The motion vector scaling process in [section 7.8.2.1][] is invoked with plane, refIdx, predX, predY, mv as inputs and the output being the initial location startX, startY, and the step sizes stepX, stepY.
-
The block inter prediction process in [section 7.8.2.2][] is invoked with plane, refIdx, startX, startY, stepX, stepY, predW, predH, candRow, candCol as inputs and the output is assigned to the 2D array obmcPred.
-
obmcPred[ i ][ j ] is set equal to Clip1( Round2( obmcPred[ i ][ j ], 9) ) for i = 0..predH-1 and j = 0..predW-1.
-
The blending process in [section 7.8.2.8][] is invoked with plane, predX, predY, predW, predH, pass, obmcPred, and mask as inputs.
The function get_obmc_mask returns a blending mask as follows:
get_obmc_mask( length ) {
if (length == 1) {
return Obmc_Mask_1
} else if (length == 2) {
return Obmc_Mask_2
} else if (length == 4) {
return Obmc_Mask_4
} else if (length == 8) {
return Obmc_Mask_8
} else if (length == 16) {
return Obmc_Mask_16
} else if (length == 32) {
return Obmc_Mask_32
}
}The blending masks are defined as follows:
Obmc_Mask_1[1] = { 64 }
Obmc_Mask_2[2] = { 45, 64 }
Obmc_Mask_4[4] = { 39, 50, 59, 64 }
Obmc_Mask_8[8] = { 36, 42, 48, 53, 57, 61, 64, 64 }
Obmc_Mask_16[16] = { 34, 37, 40, 43, 46, 49, 52, 54,
56, 58, 60, 61, 64, 64, 64, 64 }
Obmc_Mask_32[32] = { 33, 35, 36, 38, 40, 41, 43, 44,
45, 47, 48, 50, 51, 52, 53, 55,
56, 57, 58, 59, 60, 60, 61, 62,
64, 64, 64, 64, 64, 64, 64, 64 }The inputs to this process are:
-
a variable plane specifying which plane is being predicted,
-
variables predX and predY specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables predW and predH specifying the width and height of the region to be predicted,
-
a variable pass equal to 0 if blending above samples, or equal to 1 if blending left samples,
-
a 2d array obmcPred containing the samples predicted from a neighbouring motion vector,
-
an array mask containing the blending weights.
The outputs of this process are modified inter predicted samples in the current frame CurrFrame.
For i = 0..(predH - 1) and j = 0..(predW - 1), the following ordered steps apply:
-
The variable m specifying the blending factor is specifed as follows:
-
If pass is equal to 0 (blend from above), m is set equal to mask[ i ].
-
Otherwise (pass is equal to 0 meaning blend from left), m is set equal to mask[ j ].
-
-
CurrFrame[ plane ][ predY + i ][ predX + j ] is set equal to Round2( m * CurrFrame[ plane ][ predY + i ][ predX + j ] + (64 - m) * obmcPred[ i ][ j ], 6)
The input to this process is:
- variables w and h specifying the width and height of the region to be predicted.
This process sets up a mask array for the luma samples.
The mask is specified as:
for (i = 0; i < h; i++) {
for (j = 0; j < w; j++) {
Mask[ i ][ j ] = WedgeMasks[ MiSize ][ wedge_sign ][ wedge_index ][ i ][ j ]
}
}where WedgeMasks is a fixed lookup table that is specified as:
w = MASK_MASTER_SIZE
h = MASK_MASTER_SIZE
for (j = 0; j < w; j ++) {
shift = MASK_MASTER_SIZE / 4
for (i = 0; i < h; i += 2) {
MasterMask[ WEDGE_OBLIQUE63 ][ i ][ j ] = Wedge_Master_Oblique_Even[ Clip3( 0, MASK_MASTER_SIZE - 1, j - shift ) ]
shift -= 1
MasterMask[ WEDGE_OBLIQUE63 ][ i + 1][ j ] = Wedge_Master_Oblique_Even[ Clip3( 0, MASK_MASTER_SIZE - 1, j - shift ) ]
MasterMask[ WEDGE_VERTICAL ][ i ][ j ] = Wedge_Master_Vertical[ j ]
MasterMask[ WEDGE_VERTICAL ][ i + 1 ][ j ] = Wedge_Master_Vertical[ j ]
}
}
for (i = 0; i < h; i++) {
for (j = 0; j < w; j++) {
msk = MasterMask[WEDGE_OBLIQUE63][ i ][ j ]
MasterMask[WEDGE_OBLIQUE27][j ][ i ] = msk
MasterMask[WEDGE_OBLIQUE117][ i ][ w - 1 - j ] = 64 - msk
MasterMask[WEDGE_OBLIQUE153][ w - 1 - j ][ i ] = 64 - msk
MasterMask[WEDGE_HORIZONTAL][ j ][ i ] = MasterMask[WEDGE_VERTICAL][ i ][ j ]
}
}
for (bsize = BLOCK_8X8; bsize < BLOCK_SIZES; bsize++) {
if ( Wedge_Bits[ bsize ] > 0 ) {
w = Block_Width[ MiSize ]
h = Block_Height[ MiSize ]
for (wedge = 0; wedge < WEDGE_TYPES; wedge++) {
dir = get_wedge_direction(bsize, wedge)
xoff = MASK_MASTER_SIZE / 2 - ((get_wedge_xoff(bsize, wedge) * w) >> 3)
yoff = MASK_MASTER_SIZE / 2 - ((get_wedge_yoff(bsize, wedge) * h) >> 3)
sum = 0
for (i = 0; i < w; i++)
sum += MasterMask[dir][ yoff ][ xoff+i ]
for (i = 1; i < h; i++)
sum += MasterMask[dir][ yoff+i ][ xoff ]
avg = (sum + (w + h - 1) / 2) / (w + h - 1)
flipSign = (avg < 32)
for (i = 0; i < h; i++) {
for (j = 0; j < w; j++) {
WedgeMasks[bsize][flipSign][wedge][ i ][ j ] = MasterMask[ dir ][ yoff+i ][ xoff+j ]
WedgeMasks[bsize][!flipSign][wedge][ i ][ j ] = 64 - MasterMask[ dir ][ yoff+i ][ xoff+j ]
}
}
}
}
}The 1d lookup tables are defined as:
Wedge_Master_Oblique_Odd[MASK_MASTER_SIZE] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 6, 18,
37, 53, 60, 63, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64
}
Wedge_Master_Oblique_Even[MASK_MASTER_SIZE] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 4, 11, 27,
46, 58, 62, 63, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64
}
Wedge_Master_Vertical[MASK_MASTER_SIZE] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 7, 21,
43, 57, 62, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64
}The get_wedge functions are defined as:
block_shape(bsize) {
w4 = Num_4x4_Blocks_Wide[bsize]
h4 = Num_4x4_Blocks_High[bsize]
if (h4 > w4)
return 0
else if (h4 < w4)
return 1
else
return 2
}
get_wedge_direction(bsize, index) {
return Wedge_Codebook[block_shape(bsize)][index][0]
}
get_wedge_xoff(bsize, index) {
return Wedge_Codebook[block_shape(bsize)][index][1]
}
get_wedge_yoff(bsize, index) {
return Wedge_Codebook[block_shape(bsize)][index][2]
}
Wedge_Codebook[3][16][3] = {
{
{ WEDGE_OBLIQUE27, 4, 4 }, { WEDGE_OBLIQUE63, 4, 4 },
{ WEDGE_OBLIQUE117, 4, 4 }, { WEDGE_OBLIQUE153, 4, 4 },
{ WEDGE_HORIZONTAL, 4, 2 }, { WEDGE_HORIZONTAL, 4, 4 },
{ WEDGE_HORIZONTAL, 4, 6 }, { WEDGE_VERTICAL, 4, 4 },
{ WEDGE_OBLIQUE27, 4, 2 }, { WEDGE_OBLIQUE27, 4, 6 },
{ WEDGE_OBLIQUE153, 4, 2 }, { WEDGE_OBLIQUE153, 4, 6 },
{ WEDGE_OBLIQUE63, 2, 4 }, { WEDGE_OBLIQUE63, 6, 4 },
{ WEDGE_OBLIQUE117, 2, 4 }, { WEDGE_OBLIQUE117, 6, 4 },
},
{
{ WEDGE_OBLIQUE27, 4, 4 }, { WEDGE_OBLIQUE63, 4, 4 },
{ WEDGE_OBLIQUE117, 4, 4 }, { WEDGE_OBLIQUE153, 4, 4 },
{ WEDGE_VERTICAL, 2, 4 }, { WEDGE_VERTICAL, 4, 4 },
{ WEDGE_VERTICAL, 6, 4 }, { WEDGE_HORIZONTAL, 4, 4 },
{ WEDGE_OBLIQUE27, 4, 2 }, { WEDGE_OBLIQUE27, 4, 6 },
{ WEDGE_OBLIQUE153, 4, 2 }, { WEDGE_OBLIQUE153, 4, 6 },
{ WEDGE_OBLIQUE63, 2, 4 }, { WEDGE_OBLIQUE63, 6, 4 },
{ WEDGE_OBLIQUE117, 2, 4 }, { WEDGE_OBLIQUE117, 6, 4 },
},
{
{ WEDGE_OBLIQUE27, 4, 4 }, { WEDGE_OBLIQUE63, 4, 4 },
{ WEDGE_OBLIQUE117, 4, 4 }, { WEDGE_OBLIQUE153, 4, 4 },
{ WEDGE_HORIZONTAL, 4, 2 }, { WEDGE_HORIZONTAL, 4, 6 },
{ WEDGE_VERTICAL, 2, 4 }, { WEDGE_VERTICAL, 6, 4 },
{ WEDGE_OBLIQUE27, 4, 2 }, { WEDGE_OBLIQUE27, 4, 6 },
{ WEDGE_OBLIQUE153, 4, 2 }, { WEDGE_OBLIQUE153, 4, 6 },
{ WEDGE_OBLIQUE63, 2, 4 }, { WEDGE_OBLIQUE63, 6, 4 },
{ WEDGE_OBLIQUE117, 2, 4 }, { WEDGE_OBLIQUE117, 6, 4 },
}
}The input to this process is:
-
an array preds containing the predicted samples,
-
variables w and h specifying the width and height of the region to be predicted.
This process prepares an array Mask containing the blending weights for the luma samples.
The process sets the array based on the difference between the two predictions as follows:
for (i = 0; i < h; i++) {
for (j = 0; j < w; j++) {
diff = Abs(preds[ 0 ][ i ][ j ] - preds[ 1 ][ i ][ j ])
diff = Round2(diff, (BitDepth - 8) + 9)
m = Clip3(0, 64, 38 + diff / 16)
if (mask_type)
Mask[ i ][ j ] = 64 - m
else
Mask[ i ][ j ] = m
}
}The input to this process is:
-
a variable plane specifying which plane is being predicted,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables w and h specifying the width and height of the region to be predicted.
This process prepares an array Mask containing the blending weights for the luma samples.
The process sets the array based on the mode used for intra prediction as follows:
sizeScale = MAX_SB_SIZE / Max( h, w )
for (i = 0; i < h; i++) {
for (j = 0; j < w; j++) {
if (interintra_mode == II_V_PRED) {
Mask[ i ][ j ] = Ii_Weights_1d[ i * sizeScale ]
} else if (interintra_mode == II_H_PRED) {
Mask[ i ][ j ] = Ii_Weights_1d[ j * sizeScale ]
} else if (interintra_mode == II_SMOOTH_PRED) {
Mask[ i ][ j ] = Ii_Weights_1d[ Min(i, j) * sizeScale ]
} else {
Mask[ i ][ j ] = 32
}
}
}where the table Ii_Weights_1d is defined as:
Ii_Weights_1d[MAX_SB_SIZE] = {
60, 58, 56, 54, 52, 50, 48, 47, 45, 44, 42, 41, 39, 38, 37, 35, 34, 33, 32,
31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 22, 21, 20, 19, 19, 18, 18, 17, 16,
16, 15, 15, 14, 14, 13, 13, 12, 12, 12, 11, 11, 10, 10, 10, 9, 9, 9, 8,
8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 4, 4,
4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
}The inputs to this process are
-
an array preds containing the predicted samples,
-
a variable plane specifying which plane is being predicted,
-
variables dstX and dstY specifying the location of the top left sample in the CurrFrame[ plane ] array of the region to be predicted,
-
variables w and h specifying the width and height of the region to be predicted.
The process combines two predictions according to the mask.
It makes use of an array Mask containing the blending weights to apply (the weights are defined for the current plane samples if compound_type is equal to COMPOUND_INTRA, or the luma plane otherwise).
The variables subX and subY describing the subsampling of the current plane are derived as follows:
-
If plane is equal to 0, subX and subY are set equal to 0.
-
Otherwise (plane is not equal to 0), subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The process is specified as follows:
for (y = 0; y < h; y++) {
for (x = 0; x < w; x++) {
if (!subX && !subY) {
m = Mask[ y ][ x ]
} else if (subX && !subY) {
m = Round2(Mask[ y ][ 2*x ] + Mask[ y ][ 2*x+1 ], 1)
} else if (!subX && subY) {
m = Round2(Mask[ 2*y ][ x ] + Mask[ 2*y+1 ][ x ], 1)
} else {
m = Round2(Mask[ 2*y ][ 2*x ] + Mask[ 2*y ][ 2*x+1 ] +
Mask[ 2*y+1 ][ 2*x ] + Mask[ 2*y+1 ][ 2*x+1 ], 2)
}
if (interintra) {
pred0 = Clip1(Round2(preds[ 0 ][ y ][ x ], 9))
pred1 = CurrFrame[plane][x+dstX][y+dstY]
CurrFrame[plane][x+dstX][y+dstY] = Round2( m * pred1 + (64 - m) * pred0, 6 )
} else {
pred0 = preds[ 0 ][ y ][ x ]
pred1 = preds[ 1 ][ y ][ x ]
CurrFrame[plane][x+dstX][y+dstY] = Clip1( Round2( Round2(m * pred0 + (64 - m) * pred1, 6), 9 ) )
}
}
}The inputs to this process are variables candRow and candCol specifying the location (in units of 4x4 blocks) of the motion vector information to be used.
This process computes weights to be used for blending predictions together based on the expected output times of the reference frames.
The weights are computed as follows:
for ( refList = 0; refList < 2; refList++ ) {
h = OrderHints[ RefFrames[ candRow ][ candCol ][ refList ] ]
dist[ refList ] = Clip3( 0, MAX_FRAME_DISTANCE, Abs( h - OrderHint ) )
}
d0 = dist[ 1 ]
d1 = dist[ 0 ]
order = d0 <= d1
if (d0 == 0 || d1 == 0) {
FwdWeight = Quant_Dist_Lookup[ 3 ][ order ]
BckWeight = Quant_Dist_Lookup[ 3 ][ 1 - order ]
} else {
for (i = 0; i < 3; ++i) {
c0 = Quant_Dist_Weight[ i ][ order ]
c1 = Quant_Dist_Weight[ i ][ 1 - order ]
if ( order ) {
if ( d0 * c0 > d1 * c1 )
break
} else {
if ( d0 * c0 < d1 * c1 )
break
}
}
FwdWeight = Quant_Dist_Lookup[ i ][ order ]
BckWeight = Quant_Dist_Lookup[ i ][ 1 - order ]
}where the tables Quant_Dist_Lookup and Quant_Dist_Weight are specified as:
Quant_Dist_Weight[ 4 ][ 2 ] = {
{ 2, 3 }, { 2, 5 }, { 2, 7 }, { 1, MAX_FRAME_DISTANCE }
}
Quant_Dist_Lookup[ 4 ][ 2 ] = {
{ { 9, 7 }, { 11, 5 }, { 12, 4 }, { 13, 3 } }
}The palette prediction process is invoked for palette coded intra blocks to predict a part of the block using the limited palette.
The inputs to this process are:
- a variable plane specifying which plane is being predicted,
- variables startX and startY specifying the location of the top left sample in the CurrFrame[ plane ] array of the current transform block,
- variables x and y specifying the location in 4x4 units relative to the top left sample of the current transform block,
- a variable txSz, specifying the size of the current transform block.
The outputs of this process are palette predicted samples in the current frame CurrFrame.
The variable w specifying the width of the transform block is set equal to Tx_Width[ txSz ].
The variable h specifying the height of the transform block is set equal to Tx_Height[ txSz ].
The variable palette is specified as follows:
- If plane is 0, palette is set to palette_colors_y.
- Otherwise, if plane is 1, palette is set to palette_colors_u.
- Otherwise (plane is 2), palette is set to palette_colors_v.
The variable map is specified as follows:
- If plane is 0, map is set to ColorMapY.
- Otherwise (plane is not 0), map is set to ColorMapUV.
The current frame is updated as follows:
- CurrFrame[ plane ][ startY + i ][ startX + j ] is set equal to palette[ map[ y * 4 + i ][ x * 4 + j ] ] for i = 0..h-1 and j = 0..w-1.
The chroma from luma process uses luma predicted samples to form a prediction for the chroma samples. The high frequencies are taken from the luma prediction and combined with DC predicted chroma samples.
The inputs to this process are:
-
a variable plane (greater than zero) specifying which plane is being predicted,
-
variables startX and startY specifying the location of the top left sample in the CurrFrame[ plane ] array of the current transform block,
-
a variable txSz, specifying the size of the current transform block.
The outputs of this process are modified chroma predicted samples in the current frame CurrFrame.
The variable w specifying the width of the transform block is set equal to Tx_Width[ txSz ].
The variable h specifying the height of the transform block is set equal to Tx_Height[ txSz ].
The variable subX is set equal to subsampling_x.
The variable subY is set equal to subsampling_y.
The variable alpha depends on the plane as follows:
-
If plane is equal to 1, alpha is set equal to CflAlphaU.
-
Otherwise (plane is equal to 2), alpha is set equal to CflAlphaV.
An array L (containing subsampled luma pixels with 3 fractional bits of precision) and lumaAvg (representing the average luma intensity with 3 fractional bits of precision) is specified as:
lumaAvg = 0
for ( i = 0; i < h; i++ ) {
for ( j = 0; j < w; j++ ) {
lumaX = (startX + j) << subX
lumaY = (startY + i) << subY
lumaX = Min( lumaX, MaxLumaW - (1 << subX) )
lumaY = Min( lumaY, MaxLumaH - (1 << subY) )
t = 0
for ( dy = 0; dy <= subY; dy += 1 )
for ( dx = 0; dx <= subX; dx += 1 )
t += CurrFrame[ 0 ][ lumaY + dy ]
[ lumaX + dx ]
v = t << ( 3 - subX - subY )
L[ i ][ j ] = v
lumaAvg += v
}
}
lumaAvg >>= Tx_Width_Log2[ txSz ] + Tx_Height_Log2[ txSz ]The predicted chroma pixels are specified as:
for ( i = 0; i < h; i++ ) {
for ( j = 0; j < w; j++ ) {
dc = CurrFrame[ plane ][ startY + i ][ startX + j ]
scaledLuma = Round2Signed( alpha * ( L[ i ][ j ] - lumaAvg ), 6 )
CurrFrame[ plane ][ startY + i ][ startX + j ] = Clip1( dc + scaledLuma )
}
}This section details the process of reconstructing a block of coefficients using dequantization and inverse transforms.
This section defines the functions get_dc_quant and get_ac_quant that are needed by the dequantization process.
The quantization parameters are derived from lookup tables.
The function dc_q( b ) is specified as Dc_Qlookup[ (BitDepth-8) >> 1 ][ Clip3( 0, 255, b ) ] where Dc_Qlookup is defined as follows:
Dc_Qlookup[ 3 ][ 256 ] = {
{
4, 8, 8, 9, 10, 11, 12, 12, 13, 14, 15, 16,
17, 18, 19, 19, 20, 21, 22, 23, 24, 25, 26, 26,
27, 28, 29, 30, 31, 32, 32, 33, 34, 35, 36, 37,
38, 38, 39, 40, 41, 42, 43, 43, 44, 45, 46, 47,
48, 48, 49, 50, 51, 52, 53, 53, 54, 55, 56, 57,
57, 58, 59, 60, 61, 62, 62, 63, 64, 65, 66, 66,
67, 68, 69, 70, 70, 71, 72, 73, 74, 74, 75, 76,
77, 78, 78, 79, 80, 81, 81, 82, 83, 84, 85, 85,
87, 88, 90, 92, 93, 95, 96, 98, 99, 101, 102, 104,
105, 107, 108, 110, 111, 113, 114, 116, 117, 118, 120, 121,
123, 125, 127, 129, 131, 134, 136, 138, 140, 142, 144, 146,
148, 150, 152, 154, 156, 158, 161, 164, 166, 169, 172, 174,
177, 180, 182, 185, 187, 190, 192, 195, 199, 202, 205, 208,
211, 214, 217, 220, 223, 226, 230, 233, 237, 240, 243, 247,
250, 253, 257, 261, 265, 269, 272, 276, 280, 284, 288, 292,
296, 300, 304, 309, 313, 317, 322, 326, 330, 335, 340, 344,
349, 354, 359, 364, 369, 374, 379, 384, 389, 395, 400, 406,
411, 417, 423, 429, 435, 441, 447, 454, 461, 467, 475, 482,
489, 497, 505, 513, 522, 530, 539, 549, 559, 569, 579, 590,
602, 614, 626, 640, 654, 668, 684, 700, 717, 736, 755, 775,
796, 819, 843, 869, 896, 925, 955, 988, 1022, 1058, 1098, 1139,
1184, 1232, 1282, 1336
},
{
4, 9, 10, 13, 15, 17, 20, 22, 25, 28, 31, 34,
37, 40, 43, 47, 50, 53, 57, 60, 64, 68, 71, 75,
78, 82, 86, 90, 93, 97, 101, 105, 109, 113, 116, 120,
124, 128, 132, 136, 140, 143, 147, 151, 155, 159, 163, 166,
170, 174, 178, 182, 185, 189, 193, 197, 200, 204, 208, 212,
215, 219, 223, 226, 230, 233, 237, 241, 244, 248, 251, 255,
259, 262, 266, 269, 273, 276, 280, 283, 287, 290, 293, 297,
300, 304, 307, 310, 314, 317, 321, 324, 327, 331, 334, 337,
343, 350, 356, 362, 369, 375, 381, 387, 394, 400, 406, 412,
418, 424, 430, 436, 442, 448, 454, 460, 466, 472, 478, 484,
490, 499, 507, 516, 525, 533, 542, 550, 559, 567, 576, 584,
592, 601, 609, 617, 625, 634, 644, 655, 666, 676, 687, 698,
708, 718, 729, 739, 749, 759, 770, 782, 795, 807, 819, 831,
844, 856, 868, 880, 891, 906, 920, 933, 947, 961, 975, 988,
1001, 1015, 1030, 1045, 1061, 1076, 1090, 1105, 1120, 1137, 1153, 1170,
1186, 1202, 1218, 1236, 1253, 1271, 1288, 1306, 1323, 1342, 1361, 1379,
1398, 1416, 1436, 1456, 1476, 1496, 1516, 1537, 1559, 1580, 1601, 1624,
1647, 1670, 1692, 1717, 1741, 1766, 1791, 1817, 1844, 1871, 1900, 1929,
1958, 1990, 2021, 2054, 2088, 2123, 2159, 2197, 2236, 2276, 2319, 2363,
2410, 2458, 2508, 2561, 2616, 2675, 2737, 2802, 2871, 2944, 3020, 3102,
3188, 3280, 3375, 3478, 3586, 3702, 3823, 3953, 4089, 4236, 4394, 4559,
4737, 4929, 5130, 5347
},
{
4, 12, 18, 25, 33, 41, 50, 60,
70, 80, 91, 103, 115, 127, 140, 153,
166, 180, 194, 208, 222, 237, 251, 266,
281, 296, 312, 327, 343, 358, 374, 390,
405, 421, 437, 453, 469, 484, 500, 516,
532, 548, 564, 580, 596, 611, 627, 643,
659, 674, 690, 706, 721, 737, 752, 768,
783, 798, 814, 829, 844, 859, 874, 889,
904, 919, 934, 949, 964, 978, 993, 1008,
1022, 1037, 1051, 1065, 1080, 1094, 1108, 1122,
1136, 1151, 1165, 1179, 1192, 1206, 1220, 1234,
1248, 1261, 1275, 1288, 1302, 1315, 1329, 1342,
1368, 1393, 1419, 1444, 1469, 1494, 1519, 1544,
1569, 1594, 1618, 1643, 1668, 1692, 1717, 1741,
1765, 1789, 1814, 1838, 1862, 1885, 1909, 1933,
1957, 1992, 2027, 2061, 2096, 2130, 2165, 2199,
2233, 2267, 2300, 2334, 2367, 2400, 2434, 2467,
2499, 2532, 2575, 2618, 2661, 2704, 2746, 2788,
2830, 2872, 2913, 2954, 2995, 3036, 3076, 3127,
3177, 3226, 3275, 3324, 3373, 3421, 3469, 3517,
3565, 3621, 3677, 3733, 3788, 3843, 3897, 3951,
4005, 4058, 4119, 4181, 4241, 4301, 4361, 4420,
4479, 4546, 4612, 4677, 4742, 4807, 4871, 4942,
5013, 5083, 5153, 5222, 5291, 5367, 5442, 5517,
5591, 5665, 5745, 5825, 5905, 5984, 6063, 6149,
6234, 6319, 6404, 6495, 6587, 6678, 6769, 6867,
6966, 7064, 7163, 7269, 7376, 7483, 7599, 7715,
7832, 7958, 8085, 8214, 8352, 8492, 8635, 8788,
8945, 9104, 9275, 9450, 9639, 9832, 10031, 10245,
10465, 10702, 10946, 11210, 11482, 11776, 12081, 12409,
12750, 13118, 13501, 13913, 14343, 14807, 15290, 15812,
16356, 16943, 17575, 18237, 18949, 19718, 20521, 21387
}
}The function ac_q( b ) is specified as Ac_Qlookup[ (BitDepth-8) >> 1 ][ Clip3( 0, 255, b ) ] where Ac_Qlookup is defined as follows:
Ac_Qlookup[ 3 ][ 256 ] = {
{
4, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78,
79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102,
104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126,
128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150,
152, 155, 158, 161, 164, 167, 170, 173, 176, 179, 182, 185,
188, 191, 194, 197, 200, 203, 207, 211, 215, 219, 223, 227,
231, 235, 239, 243, 247, 251, 255, 260, 265, 270, 275, 280,
285, 290, 295, 300, 305, 311, 317, 323, 329, 335, 341, 347,
353, 359, 366, 373, 380, 387, 394, 401, 408, 416, 424, 432,
440, 448, 456, 465, 474, 483, 492, 501, 510, 520, 530, 540,
550, 560, 571, 582, 593, 604, 615, 627, 639, 651, 663, 676,
689, 702, 715, 729, 743, 757, 771, 786, 801, 816, 832, 848,
864, 881, 898, 915, 933, 951, 969, 988, 1007, 1026, 1046, 1066,
1087, 1108, 1129, 1151, 1173, 1196, 1219, 1243, 1267, 1292, 1317, 1343,
1369, 1396, 1423, 1451, 1479, 1508, 1537, 1567, 1597, 1628, 1660, 1692,
1725, 1759, 1793, 1828
},
{
4, 9, 11, 13, 16, 18, 21, 24, 27, 30, 33, 37,
40, 44, 48, 51, 55, 59, 63, 67, 71, 75, 79, 83,
88, 92, 96, 100, 105, 109, 114, 118, 122, 127, 131, 136,
140, 145, 149, 154, 158, 163, 168, 172, 177, 181, 186, 190,
195, 199, 204, 208, 213, 217, 222, 226, 231, 235, 240, 244,
249, 253, 258, 262, 267, 271, 275, 280, 284, 289, 293, 297,
302, 306, 311, 315, 319, 324, 328, 332, 337, 341, 345, 349,
354, 358, 362, 367, 371, 375, 379, 384, 388, 392, 396, 401,
409, 417, 425, 433, 441, 449, 458, 466, 474, 482, 490, 498,
506, 514, 523, 531, 539, 547, 555, 563, 571, 579, 588, 596,
604, 616, 628, 640, 652, 664, 676, 688, 700, 713, 725, 737,
749, 761, 773, 785, 797, 809, 825, 841, 857, 873, 889, 905,
922, 938, 954, 970, 986, 1002, 1018, 1038, 1058, 1078, 1098, 1118,
1138, 1158, 1178, 1198, 1218, 1242, 1266, 1290, 1314, 1338, 1362, 1386,
1411, 1435, 1463, 1491, 1519, 1547, 1575, 1603, 1631, 1663, 1695, 1727,
1759, 1791, 1823, 1859, 1895, 1931, 1967, 2003, 2039, 2079, 2119, 2159,
2199, 2239, 2283, 2327, 2371, 2415, 2459, 2507, 2555, 2603, 2651, 2703,
2755, 2807, 2859, 2915, 2971, 3027, 3083, 3143, 3203, 3263, 3327, 3391,
3455, 3523, 3591, 3659, 3731, 3803, 3876, 3952, 4028, 4104, 4184, 4264,
4348, 4432, 4516, 4604, 4692, 4784, 4876, 4972, 5068, 5168, 5268, 5372,
5476, 5584, 5692, 5804, 5916, 6032, 6148, 6268, 6388, 6512, 6640, 6768,
6900, 7036, 7172, 7312
},
{
4, 13, 19, 27, 35, 44, 54, 64,
75, 87, 99, 112, 126, 139, 154, 168,
183, 199, 214, 230, 247, 263, 280, 297,
314, 331, 349, 366, 384, 402, 420, 438,
456, 475, 493, 511, 530, 548, 567, 586,
604, 623, 642, 660, 679, 698, 716, 735,
753, 772, 791, 809, 828, 846, 865, 884,
902, 920, 939, 957, 976, 994, 1012, 1030,
1049, 1067, 1085, 1103, 1121, 1139, 1157, 1175,
1193, 1211, 1229, 1246, 1264, 1282, 1299, 1317,
1335, 1352, 1370, 1387, 1405, 1422, 1440, 1457,
1474, 1491, 1509, 1526, 1543, 1560, 1577, 1595,
1627, 1660, 1693, 1725, 1758, 1791, 1824, 1856,
1889, 1922, 1954, 1987, 2020, 2052, 2085, 2118,
2150, 2183, 2216, 2248, 2281, 2313, 2346, 2378,
2411, 2459, 2508, 2556, 2605, 2653, 2701, 2750,
2798, 2847, 2895, 2943, 2992, 3040, 3088, 3137,
3185, 3234, 3298, 3362, 3426, 3491, 3555, 3619,
3684, 3748, 3812, 3876, 3941, 4005, 4069, 4149,
4230, 4310, 4390, 4470, 4550, 4631, 4711, 4791,
4871, 4967, 5064, 5160, 5256, 5352, 5448, 5544,
5641, 5737, 5849, 5961, 6073, 6185, 6297, 6410,
6522, 6650, 6778, 6906, 7034, 7162, 7290, 7435,
7579, 7723, 7867, 8011, 8155, 8315, 8475, 8635,
8795, 8956, 9132, 9308, 9484, 9660, 9836, 10028,
10220, 10412, 10604, 10812, 11020, 11228, 11437, 11661,
11885, 12109, 12333, 12573, 12813, 13053, 13309, 13565,
13821, 14093, 14365, 14637, 14925, 15213, 15502, 15806,
16110, 16414, 16734, 17054, 17390, 17726, 18062, 18414,
18766, 19134, 19502, 19886, 20270, 20670, 21070, 21486,
21902, 22334, 22766, 23214, 23662, 24126, 24590, 25070,
25551, 26047, 26559, 27071, 27599, 28143, 28687, 29247
}
}The function get_qindex( ignoreDeltaQ, segmentId ) returns the quantizer index for the current block and is specified by the following:
-
If seg_feature_active_idx( segmentId, SEG_LVL_ALT_Q ) is equal to 1 the following ordered steps apply:
-
Set the variable data equal to FeatureData[ segmentId ][ SEG_LVL_ALT_Q ].
-
Set data equal to base_q_idx + data
-
Return Clip3( 0, 255, data ).
-
-
Otherwise, if ignoreDeltaQ is equal to 0 and delta_q_present is equal to 1, return CurrentQIndex.
-
Otherwise, return base_q_idx.
The function get_dc_quant( plane ) returns the quantizer value for the dc coefficient for a particular plane and is derived as follows:
-
If plane is equal to 0, return dc_q( get_qindex( 0, segment_id ) + DeltaQYDc ).
-
Otherwise if plane is equal to 1, return dc_q( get_qindex( 0, segment_id ) + DeltaQUDc ).
-
Otherwise (plane is equal to 2), return dc_q( get_qindex( 0, segment_id ) + DeltaQVDc ).
The function get_ac_quant( plane ) returns the quantizer value for the ac coefficient for a particular plane and is derived as follows:
-
If plane is equal to 0, return ac_q( get_qindex( 0, segment_id ) ).
-
Otherwise if plane is equal to 1, return ac_q( get_qindex( 0, segment_id ) + DeltaQUAc ).
-
Otherwise (plane is equal to 2), return ac_q( get_qindex( 0, segment_id ) + DeltaQVAc ).
The reconstruct process is invoked to perform dequantization, inverse transform and reconstruction. This process is triggered at a point defined by a function call to reconstruct in the transform block syntax table described in [section 5.8.33][].
The inputs to this process are:
-
a variable plane specifying which plane is being reconstructed,
-
variables x and y specifying the location of the top left sample in the CurrFrame[ plane ] array of the current transform block,
-
a variable txSz, specifying the size of the transform block.
The outputs of this process are reconstructed samples in the current frame CurrFrame.
The reconstruction and dequantization process is defined as follows:
The variable transpose is set equal to ( Tx_Width[ txSz ] > Tx_Height[ txSz ] ).
The variable txSize is derived as follows:
-
If transpose is equal to 1, txSize is set equal to Transpose_Tx_Size[ txSz ].
-
Otherwise (transpose is equal to 0), txSize is set equal to txSz.
The variable txType is derived as follows:
-
If transpose is equal to 1, txType is set equal to Transpose_Tx_Type[ PlaneTxType ].
-
Otherwise (transpose is equal to 0), txType is set equal to PlaneTxType.
The variable dqDenom is derived as follows:
-
If txSz is equal to TX_32X32, TX_16X32, TX_32X16, TX_16X64, or TX_64X16, dqDenom is set equal to 2.
-
Otherwise, if txSz is equal to TX_64X64, TX_32X64, or TX_64X32, dqDenom is set equal to 4.
-
Otherwise, dqDenom is set equal to 1.
The variable log2W (specifying the base 2 logarithm of the width of the transform block) is set equal to Tx_Width_Log2[ txSz ].
The variable log2H (specifying the base 2 logarithm of the height of the transform block) is set equal to Tx_Height_Log2[ txSz ].
The variable w (specifying the width of the transform block) is set equal to 1 << log2W.
The variable h (specifying the height of the transform block) is set equal to 1 << log2H.
The variable tw is set equal to Min( 32, w ).
The variable th is set equal to Min( 32, h ).
The variable flipUD is derived as follows. If txType is equal to one of FLIPADST_DCT, FLIPADST_ADST, V_FLIPADST, or FLIPADST_FLIPADST, flipUD is set equal to 1. Otherwise, flipUD is set equal to 0.
The variable flipLR is derived as follows. If txType is equal to one of DCT_FLIPADST, ADST_FLIPADST, H_FLIPADST, or FLIPADST_FLIPADST, flipLR is set equal to 1. Otherwise, flipLR is set equal to 0.
The following ordered steps apply:
-
For i = 0..(th-1), for j = 0..(tw-1), the following ordered steps apply:
a. The variable q is derived as follows:
-
If i is equal to 0 and j is equal to 0, the variable q is set equal to get_dc_quant( plane ).
-
Otherwise (i, j or both are not equal to 0), the variable q is set equal to get_ac_quant( plane ).
b. The variable q2 is derived as follows:
-
If using_qmatrix is equal to 1, PlaneTxType is less than IDTX, and SegQMLevel[ segment_id ] is less than 15, q2 is set equal to Round2( q * Quantizer_Matrix[ SegQMLevel[ segment_id ] ][ plane > 0 ][ Qm_Offset[ txSz ] + i * tw + j ], 4 ).
-
Otherwise, q2 is set equal to q.
c. Dequant[ i ][ j ] is set equal to ( Quant[ i * tw + j ] * q2 ) / dqDenom.
-
-
Invoke the 2D inverse transform block process defined in [section 7.10.2][] with the variables txSize, txType and transpose as input. The inverse transform outputs are stored in the Residual buffer.
-
For i = 0..(h-1), for j = 0..(w-1), the following applies:
-
The variable xx is set equal to flipLR ? ( w - j - 1 ) : j.
-
The variable yy is set equal to flipUD ? ( h - i - 1 ) : i.
-
CurrFrame[ plane ][ y + yy ][ x + xx ] is set equal to Clip1( CurrFrame[ plane ][ y + yy ][ x + xx ] + ( transpose ? Residual[ j ][ i ] : Residual[ i ][ j ] ) ).
-
It is a requirement of bitstream conformance that the values written into the Dequant and Residual arrays in steps 1 and 2 are representable by a signed integer with 8 + BitDepth bits.
This section details the inverse transforms used during the reconstruction processes detailed in [section 7.9][].
This section defines the butterfly functions B, H, SB, SH used by the 1D transform processes.
The inverse transform process works by writing values into an array T. It is a requirement of bitstream conformance that the values saved into the array T are representable by a signed integer using 8 + BitDepth bits of precision.
Note: The inverse asymmetric discrete sine transforms also make use of an intermediate array named S. The values in this array require higher precision to avoid overflow. Using signed integers with 24 + BitDepth bits of precision is enough to avoid overflow. {:.alert .alert-info }
The function brev(numBits, x) returns the bit-reversal of numBits of x and is specified as follows:
brev( numBits, x ) {
t = 0
for ( i = 0; i < numBits; i++ ) {
bit = (x >> i) & 1
t += bit << (numBits - 1 - i)
}
return t
}The function B( a, b, angle, highPrecision, 0 ) performs a butterfly rotation specified by the following ordered steps:
-
The variable x is set equal to T[ a ] * cos128( highPrecision, angle ) - T[ b ] * sin128( highPrecision, angle ).
-
The variable y is set equal to T[ a ] * sin128( highPrecision, angle ) + T[ b ] * cos128( highPrecision, angle ).
-
T[ a ] is set equal to Round2( x, highPrecision ? 13 : 12 ).
-
T[ b ] is set equal to Round2( y, highPrecision ? 13 : 12 ).
It is a requirement of bitstream conformance that the values saved into the array T by this function are representable by a signed integer using 8 + BitDepth bits of precision.
The function cos128( highPrecision, angle ) is specified for integer values of the input angle by the following ordered steps:
-
Set a variable angle2 equal to angle & 255.
-
If angle2 is greater than or equal to 0 and less than or equal to 64, return Cos128_Lookup[ highPrecision ][ angle2 ].
-
If angle2 is greater than 64 and less than or equal to 128, return Cos128_Lookup[ highPrecision ][ 128 - angle2 ] * -1.
-
If angle2 is greater than 128 and less than or equal to 192, return Cos128_Lookup[ highPrecision ][ angle2 - 128 ] * -1.
-
Otherwise (if angle2 is greater than 192 and less than 256), return Cos128_Lookup[ highPrecision ][ 256 - angle2 ].
Where Cos128_Lookup is a constant lookup table defined as:
Cos128_Lookup[ 2 ][ 65 ] = {
{ 4096, 4095, 4091, 4085, 4076, 4065, 4052, 4036,
4017, 3996, 3973, 3948, 3920, 3889, 3857, 3822,
3784, 3745, 3703, 3659, 3612, 3564, 3513, 3461,
3406, 3349, 3290, 3229, 3166, 3102, 3035, 2967,
2896, 2824, 2751, 2675, 2598, 2520, 2440, 2359,
2276, 2191, 2106, 2019, 1931, 1842, 1751, 1660,
1567, 1474, 1380, 1285, 1189, 1092, 995, 897,
799, 700, 601, 501, 401, 301, 201, 101, 0 },
{ 8192, 8190, 8182, 8170, 8153, 8130, 8103, 8071,
8035, 7993, 7946, 7895, 7839, 7779, 7713, 7643,
7568, 7489, 7405, 7317, 7225, 7128, 7027, 6921,
6811, 6698, 6580, 6458, 6333, 6203, 6070, 5933,
5793, 5649, 5501, 5351, 5197, 5040, 4880, 4717,
4551, 4383, 4212, 4038, 3862, 3683, 3503, 3320,
3135, 2948, 2760, 2570, 2378, 2185, 1990, 1795,
1598, 1401, 1202, 1003, 803, 603, 402, 201, 0 },
}The function sin128( highPrecision, angle ) is defined to be cos128( highPrecision, angle - 64 ).
Note: The cos128 function implements the expression round( ( highPrecision ? 8192 : 4096 ) * cos( angle * pi / 128 ) ). The sin128 function implements the expression round( ( highPrecision ? 8192 : 4096 ) * sin( angle * pi / 128 ) ). {:.alert .alert-info }
When the angle is equal to 32 + 64 * k for integer k the butterfly rotation can be equivalently performed with two fewer multiplications (because the magnitude of cos128(highPrecision, 32 + 64 * k) is always equal to that of sin128(highPrecision, 32 + 64 * k)) by the following process:
-
The variable v is set equal to (angle & 64) ? T[ a ] + T[ b ] : T[ a ] - T[ b ].
-
The variable w is set equal to (angle & 64) ? -T[ a ] + T[ b ] : T[ a ] + T[ b ].
-
The variable x is set equal to v * cos128( highPrecision, angle ).
-
The variable y is set equal to w * cos128( highPrecision, angle ).
-
T[ a ] is set equal to Round2( x, highPrecision ? 13 : 12 ).
-
T[ b ] is set equal to Round2( y, highPrecision ? 13 : 12 ).
It is a requirement of bitstream conformance that the angle is equal to 16 + 32 * k for integer k, the variables v and w are representable by a signed integer using 8 + BitDepth bits of precision.
The function B( a ,b, angle, highPrecision, 1 ) performs a butterfly rotation and flip specified by the following ordered steps:
-
The function B( a, b, angle, 0 ) is invoked.
-
The contents of T[ a ] and T[ b ] are exchanged.
The function H( a, b, 0 ) performs a Hadamard rotation specified by the following ordered steps:
-
The variable x is set equal to T[ a ].
-
The variable y is set equal to T[ b ].
-
T[ a ] is set equal to x + y.
-
T[ b ] is set equal to x - y.
It is a requirement of bitstream conformance that the values saved into the array T by this function are representable by a signed integer using 8 + BitDepth bits of precision.
The function H( a, b, 1 ) performs a Hadamard rotation with flipped indices and is specified as follows:
- The function H( b, a, 0 ) is invoked.
This process clamps the contents of the array T of length 2n.
The input to this process is a variable n that specifies the base 2 logarithm of the length of the input array.
T[ i ] is set equal to Clip3( - ( 1 << 15 ), ( 1 << 15 ) - 1, T[ i ] ) for i = 0..((1 << n) - 1).
This process performs an in-place permutation of the array T of length 2n for 2 ≤ n ≤ 5 which is required before execution of the inverse DCT process.
The input to this process is a variable n that specifies the base 2 logarithm of the length of the input array.
A temporary array named copyT is set equal to T.
T[ i ] is set equal to copyT[ brev( n, i ) ] for i = 0..((1 << n) - 1).
This process performs an in-place inverse discrete cosine transform of the permuted array T which is of length 2n for 2 ≤ n ≤ 6.
The inputs to this process are:
-
a variable n that specifies the base 2 logarithm of the length of the input array.
-
a variable highPrecision, specifying whether high or low precision sine and cosine values should be used.
-
a variable clamp that specifies whether intermediate values should be clamped.
The following ordered steps apply:
-
If n is equal to 6, invoke B( 32 + i, 63 - i, 63 - 4 * brev( 4, i ), highPrecision, 0 ) for i = 0..15.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 5, invoke B( 16 + i, 31 - i, 6 + ( brev( 3, 7 - i ) << 3 ), highPrecision, 0 ) for i = 0..7.
-
If n is equal to 6, invoke H( 32 + i * 2, 33 + i * 2, i & 1 ) for i = 0..15.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 4, invoke B( 8 + i, 15 - i, 12 + ( brev( 2, 3 - i ) << 4 ) ), highPrecision, 0) for i = 0..3.
-
If n is greater than or equal to 5, invoke H( 16 + 2 * i, 17 + 2 * i, i & 1 ) for i = 0..7.
-
If n is equal to 6, invoke B(62 - i * 4 - j, 33 + i * 4 + j, 60 - 16 * brev( 2, i ) + 64 * j, highPrecision, 1) for i = 0..3, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 3, invoke B( 4 + i, 7 - i, 56 - 32 * i, highPrecision, 0 ) for i = 0..1.
-
If n is greater than or equal to 4, invoke H( 8 + 2 * i, 9 + 2 * i, i & 1 ) for i = 0..3.
-
If n is greater than or equal to 5, invoke B( 30 - 4 * i - j, 17 + 4 * i + j, 24 + (j << 6) + ( ( 1 - i ) << 5 ), highPrecision, 1 ) for i = 0..1, for j=0..1.
-
If n is equal to 6, invoke H( 32 + i * 4 + j, 35 + i * 4 - j, i & 1 ) for i = 0..7, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
Invoke B( 2 * i, 2 * i + 1, 32 + 16 * i, highPrecision, 1 - i ) for i = 0..1.
-
If n is greater than or equal to 3, invoke H( 4 + 2 * i, 5 + 2 * i, i ) for i = 0..1.
-
If n is greater than or equal to 4, invoke B( 14 - i, 9 + i, 48 + 64 * i, highPrecision, 1 ) for i = 0..1.
-
If n is greater than or equal to 5, invoke H( 16 + 4 * i + j, 19 + 4 * i - j, i & 1 ) for i = 0..3, for j = 0..1.
-
If n is equal to 6, invoke B( 61 - i * 8 - j, 34 + i * 8 + j, 56 - i * 32 + ( j >> 1 ) * 64, highPrecision, 1 ) for i = 0..1, for j = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
Invoke H( i, 3 - i, 0 ) for i = 0..1.
-
If n is greater than or equal to 3, invoke B( 6, 5, 32, highPrecision, 1 ).
-
If n is greater than or equal to 4, invoke H( 8 + 4 * i + j, 11 + 4 * i - j, i ) for i = 0..1, for j = 0..1.
-
If n is greater than or equal to 5, invoke B( 29 - i, 18 + i, 48 + ( i >> 1 ) * 64, highPrecision, 1 ) for i = 0..3.
-
If n is equal to 6, invoke H( 32 + 8 * i + j, 39 + 8 * i - j, i & 1) for i = 0..3, for j = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 3, invoke H( i, 7 - i, 0 ) for i = 0..3.
-
If n is greater than or equal to 4, invoke B( 13 - i, 10 + i, 32, highPrecision, 1 ) for i = 0..1.
-
If n is greater than or equal to 5, invoke H( 16 + i * 8 + j, 23 + i * 8 - j, i ) for i = 0..1, for j = 0..3.
-
If n is equal to 6, invoke B( 59 - i, 36 + i, i < 4 ? 48 : 112, highPrecision, 1 ) for i = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 4, invoke H( i, 15 - i, 0 ) for i = 0..7.
-
If n is greater than or equal to 5, invoke B( 27 - i, 20 + i, 32, highPrecision, 1 ) for i = 0..3.
-
If n is equal to 6, the following steps apply for i = 0..7:
-
Invoke H( 32 + i, 47 - i, 0 ).
-
Invoke H( 48 + i, 63 - i, 1 ).
-
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is greater than or equal to 5, invoke H( i, 31 - i, 0 ) for i = 0..15.
-
If n is equal to 6, invoke B( 55 - i, 40 + i, 32, highPrecision, 1 ) for i = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
-
If n is equal to 6, invoke H( i, 63 - i, 0 ) for i = 0..31.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n.
This process performs the in-place permutation of the array T of length 2n which is required as the first step of the inverse ADST, where 3 ≤ n ≤ 5.
The input to this process is a variable n that specifies the base 2 logarithm of the length of the input array.
The variable n0 is set equal to 1 << n.
A temporary array named copyT is set equal to T.
If n is less than 5, the following steps apply for i = 0..(n0-1):
-
The variable a is set equal to ( i + 1 ) & 1.
-
The variable b is set equal to a ^ ( ( i >> 1 ) & 1 ).
-
The variable c is set equal to a ^ ( ( i >> 2 ) & 1 ).
-
The variable d is set equal to a ^ ( ( i >> 3 ) & 1 ).
-
The variable idx is set equal to ( ( d << 3 ) | ( c << 2 ) | ( b << 1 ) | a ) & ( n0 - 1 ).
-
T[ i ] is set equal to copyT[ idx ].
Otherwise (n is equal to 5), the following steps apply for i = 0..(n0-1):
-
The variable a is set equal to i & 1.
-
The variable b is set equal to a ^ ( ( i >> 1 ) & 1 ).
-
The variable c is set equal to b ^ ( ( i >> 2 ) & 1 ).
-
The variable d is set equal to c ^ ( ( i >> 3 ) & 1 ).
-
The variable e is set equal to d ^ ( ( i >> 4 ) & 1 ).
-
The variable idx is set equal to ( a << 4 ) | ( b << 3 ) | ( c << 2 ) | ( d << 1 ) | e.
-
T[ i ] is set equal to e ? ( - copyT[ idx ] ) : copyT[ idx ].
This process performs the in-place permutation of the array T of length 2n which is required before the final step of the inverse ADST, where 3 ≤ n ≤ 5.
The input to this process is a variable n that specifies the base 2 logarithm of the length of the input array.
The variable n0 is set equal to 1 << n.
A temporary array named copyT is set equal to T.
If n is less than 5, the following steps apply for i = 0..(n0-1):
-
The variable a is set equal to ( ( i >> 3 ) & 1 ).
-
The variable b is set equal to ( ( i >> 2 ) & 1 ) ^ ( ( i >> 3 ) & 1 ).
-
The variable c is set equal to ( ( i >> 1 ) & 1 ) ^ ( ( i >> 2 ) & 1 ).
-
The variable d is set equal to ( i & 1 ) ^ ( ( i >> 1 ) & 1 ).
-
The variable idx is set equal to ( ( d << 3 ) | ( c << 2 ) | ( b << 1 ) | a ) >> ( 4 - n ).
-
T[ i ] is set equal to ( i & 1 ) ? ( - copyT[ idx ] ) : copyT[ idx ].
Otherwise (n is equal to 5), the following steps apply for i = 0..(n0-1):
- T[ i ] is set equal to ( i & 1 ) ? copyT[ n0 - i - 1 ] : copyT[ i + 1 ].
This process performs an in-place inverse ADST process on the array T of size 4.
The inputs to this process are:
-
a variable highPrecision, specifying whether high or low precision sine values should be used,
-
a variable clamp that specifies whether intermediate values should be clamped.
The following ordered steps apply:
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 2.
-
The following applies:
s0 = SINPI_1_9 * T[ 0 ] s1 = SINPI_2_9 * T[ 0 ] s2 = SINPI_3_9 * T[ 1 ] s3 = SINPI_4_9 * T[ 2 ] s4 = SINPI_1_9 * T[ 2 ] s5 = SINPI_2_9 * T[ 3 ] s6 = SINPI_4_9 * T[ 3 ] s7 = T[ 0 ] - T[ 2 ] if ( BitDepth == 8 ) s7 = Clip3( - ( 1 << 15 ), ( 1 << 15 ) - 1, s7 ) s7 = s7 + T[ 3 ] if ( clamp ) s7 = Clip3( - ( 1 << 15 ), ( 1 << 15 ) - 1, s7 ) x0 = s0 + s3 + s5 x1 = s1 - s4 - s6 x2 = SINPI_3_9 * s7 x3 = s2 T[ 0 ] = Round2( x0 + x3, highPrecision ? 13 : 12 ) T[ 1 ] = Round2( x1 + x3, highPrecision ? 13 : 12 ) T[ 2 ] = Round2( x2, highPrecision ? 13 : 12 ) T[ 3 ] = Round2( x0 + x1 - x3, highPrecision ? 13 : 12 )
where the constants used are defined as follows, depending on the value of highPrecision:
Symbol Value when highPrecision is 0 Value when highPrecision is 1 SINPI_1_9 1321 2642 SINPI_2_9 2482 4965 SINPI_3_9 3344 6689 SINPI_4_9 3803 7606 {:.table .table-sm .table-bordered } -
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 2.
This process performs an in-place inverse ADST process on the array T of size 8.
The inputs to this process are:
-
a variable highPrecision, specifying whether high or low precision sine and cosine values should be used.
-
a variable clamp that specifies whether intermediate values should be clamped.
The following ordered steps apply:
-
Invoke the ADST input array permutation process specified in [section 7.10.1.5][] with the input variable n set to 3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke B( 2 * i, 2 * i + 1, 60 - 16 * i, highPrecision, 1 ) for i = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke H( i, 4 + i, 0 ) for i = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke B( 4 + 3 * i, 5 + i, 48 - 32 * i, 1 ) for i = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke H( 4 * j + i, 2 + 4 * j + i, 0 ) for i = 0..1, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke B( 2 + 4 * i, 3 + 4 * i, 32, 1 ) for i = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
-
Invoke the ADST output array permutation process specified in [section 7.10.1.6][] with the input variable n set to 3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 3.
This process performs an in-place inverse ADST process on the array T of size 16.
The inputs to this process are:
-
a variable highPrecision, specifying whether high or low precision sine and cosine values should be used.
-
a variable clamp that specifies whether intermediate values should be clamped.
The following ordered steps apply:
-
Invoke the ADST input array permutation process specified in [section 7.10.1.5][] with the input variable n set to 4.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke B( 2 * i, 2 * i + 1, 62 - 8 * i, highPrecision, 1 ) for i = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke H( i, 8 + i, 0 ) for i = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke B( 8 + 2 * i, 9 + 2 * i, 56 - 32 * i, highPrecision, 1 ) and B( 13 + 2 * i, 12 + 2 * i, 8 + 32 * i, highPrecision, 1 ) for i = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke H( 8 * j + i, 4 + 8 * j + i, 0 ) for i = 0..3, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke B( 4 + 8 * j + 3 * i, 5 + 8 * j + i, 48 - 32 * i, highPrecision, 1 ) and B( 12 + 3 * i, 13 + i, 48 - 32 * i, highPrecision, 1 ) for i = 0..1, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke H( 4 * j + i, 2 + 4 * j + i, 0 ) for i = 0..1, for j = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke B( 2 + 4 * i, 3 + 4 * i, 32, highPrecision, 1 ) for i = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
-
Invoke the ADST output array permutation process specified in [section 7.10.1.6][] with the input variable n set to 4.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 4.
This process performs an in-place inverse ADST process on the array T of size 32.
The inputs to this process are:
-
a variable highPrecision, specifying whether high or low precision sine and cosine values should be used.
-
a variable clamp that specifies whether intermediate values should be clamped.
The following ordered steps apply:
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke the ADST input array permutation process specified in [section 7.10.1.5][] with the input variable n set to 5.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke B( 2 + 4 * i, 3 + 4 * i, 32, highPrecision, 1 ) for i = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke H( 4 * j + i, 2 + 4 * j + i, 0 ) for i = 0..1, for j = 0..7.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke B( 4 + 8 * j + 3 * i, 5 + 8 * j + i, 48 - 32 * i, highPrecision, 1 ) for i = 0..1, for j = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke H( 8 * j + i, 4 + 8 * j + i, 0 ) for i = 0..3, for j = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke B( 8 + 16 * j + 2 * i, 9 + 16 * j + 2 * i, 56 - 32 * i, highPrecision, 1 ) for i = 0..1, j = 0..1.
-
Invoke B( 13 + 16 * j + 2 * i, 12 + 16 * j + 2 * i, 8 + 32 * i, highPrecision, 1 ) for i = 0..1, j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke H( 16 * j + i, 8 + 16 * j + i, 0 ) for i = 0..7, for j = 0..1.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke B( 16 + 2 * i, 17 + 2 * i, 60 - 16 * i, highPrecision, 1 ) for i = 0..3.
-
Invoke B( 25 + 2 * i, 24 + 2 * i, 4 + 16 * i, highPrecision, 1 ) for i = 0..3.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke H( i, 16 + i, 0 ) for i = 0..15.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke B( 2 * i, 1 + 2 * i, 63 - 4 * i, highPrecision, 1 ) for i = 0..15.
-
If clamp is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to 5.
-
Invoke the ADST output array permutation process specified in [section 7.10.1.6][] with the input variable n set to 5.
This process performs an in-place inverse ADST process on the array T of size 2n for 2 ≤ n ≤ 5.
The inputs to this process are:
-
a variable n that specifies the base 2 logarithm of the length of the input array.
-
a variable clamp that specifies whether intermediate values should be clamped.
-
a variable highPrecision, specifying whether high or low precision sine and cosine values should be used.
The following steps apply:
-
If n is equal to 2, the following steps apply.
- Invoke the inverse ADST4 process in [section 7.10.1.7][], with the input variables highPrecision and clamp.
-
Otherwise, if n is equal to 3, invoke the inverse ADST8 process in [section 7.10.1.8][], with the input variables highPrecision and clamp.
-
Otherwise, if n is equal to 4, invoke the inverse ADST16 process in [section 7.10.1.9][], with the input variable highPrecision and clamp.
-
Otherwise (n is equal to 5), invoke the inverse ADST32 process in [section 7.10.1.10][], with the input variable highPrecision and clamp.
The input to this process is a variable shift that specifies the amount of pre-scaling.
This process does an in-place transform of the array T (of length 4) by the following ordered steps:
a = T[ 0 ] >> shift
c = T[ 1 ] >> shift
d = T[ 2 ] >> shift
b = T[ 3 ] >> shift
a += c
d -= b
e = (a - d) >> 1
b = e - b
c = e - c
a -= b
d += c
T[ 0 ] = a
T[ 1 ] = b
T[ 2 ] = c
T[ 3 ] = dThe process does an in-place transform of the array T (of length 4) by the following calculation for i = 0..3:
T[ i ] = Round2( T[ i ] * 23170, 14 )The process does an in-place transform of the array T (of length 8) by the following calculation for i = 0..7:
T[ i ] = T[ i ] * 2The process does an in-place transform of the array T (of length 16) by the following calculation for i = 0..15:
T[ i ] = Round2( T[ i ] * 46340, 14 )The process does an in-place transform of the array T (of length 32) by the following calculation for i = 0..31:
T[ i ] = T[ i ] * 4The process does an in-place transform of the array T (of length 64) by the following calculation for i = 0..63:
T[ i ] = Round2( T[ i ] * 92680, 14 )This process performs an in-place identity transform process (with a size-dependent scaling factor) on the array T of size 2n for 2 ≤ n ≤ 6.
The input to this process is a variable n that specifies the base 2 logarithm of the length of the input array.
The process to invoke depends on n as follows:
-
If n is equal to 2, invoke the inverse identity transform 4 process in [section 7.10.1.13][].
-
Otherwise, if n is equal to 3, invoke the inverse identity transform 8 process in [section 7.10.1.14][].
-
Otherwise, if n is equal to 4, invoke the inverse identity transform 16 process in [section 7.10.1.15][].
-
Otherwise if n is equal to 5, invoke the inverse identity transform 32 process in [section 7.10.1.16][].
-
Otherwise (n is equal to 6), invoke the inverse identity transform 64 process in [section 7.10.1.17][].
This process performs a 2D inverse transform for an array of coefficients stored in the 2D array Dequant. The output is placed in the 2D array Residual.
The inputs to this process are:
-
a variable txSz that specifies the transform size,
-
a variable txType that specifies the transform type,
-
a variable transpose, which, if equal to 1, signifies that the transform should be transposed.
Set the variable log2W equal to Tx_Width_Log2[ txSz ].
Set the variable log2H equal to Tx_Height_Log2[ txSz ].
Set the variable w equal to 1 << log2W.
Set the variable h equal to 1 << log2H.
Set the variable whMax equal to Max(w, h).
Set the variable clampCol equal to BitDepth == 8 || BitDepth == 10.
Set the variable clampRow equal to BitDepth == 8 || ( BitDepth == 10 && ( txSz == TX_4X4 || txSz == TX_4X8 || txSz == TX_8X4 ) ).
Set the variable highPrecisionRow equal to ( log2W < 4 && log2H < 4 ).
Set the variable colShift equal to 4.
Set the variable rowShift equal to Transform_Row_Shift[ txSz ].
The row transforms with i = 0..(h-1) are applied as follows:
-
T[ j ] is derived as follows for j = 0..(w-1):
-
If i and j are both less than 32, T[ j ] is set equal to ( transpose ? Dequant[ j ][ i ] : Dequant[ i ][ j ] ).
-
Otherwise, T[ j ] is set equal to 0.
-
-
If Abs( log2W - log2H ) is equal to 1, T[ j ] is set equal to Round2( T[ j ] * 11585, 14 ) for j = 0..(w-1).
-
If Lossless is equal to 1, invoke the Inverse WHT process as specified in [section 7.10.1.12][] with shift equal to 2.
-
Otherwise, if txType is equal to one of DCT_DCT, ADST_DCT, FLIPADST_DCT or H_DCT, apply an inverse DCT as follows:
-
If clampRow is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to log2W.
-
Invoke the inverse DCT permutation process as specified in [section 7.10.1.3][] with the input variable n equal to log2W.
-
Invoke the inverse DCT process as specified in [section 7.10.1.4][] with the input variable n equal to log2W, the input variable highPrecision set equal to highPrecisionRow, and the input variable clamp equal to clampRow.
-
-
Otherwise if txType is equal to one of DCT_ADST, ADST_ADST, DCT_FLIPADST, FLIPADST_FLIPADST, ADST_FLIPADST, FLIPADST_ADST, H_ADST, or H_FLIPADST, invoke the inverse ADST process as specified in [section 7.10.1.11][] with input variable n equal to log2W, the input variable highPrecision set equal to highPrecisionRow, and the input variable clamp set equal to clampRow.
-
Otherwise, invoke the inverse identity transform process specified in [section 7.10.1.18][] with the input variable n equal to log2W.
-
Set T[ j ] equal to Round2( T[ j ], rowShift ) for j = 0..(w-1).
-
If BitDepth is equal to 8, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to log2W.
-
Residual[ i ][ j ] is set equal to T[ j ] for j = 0..(w-1).
The column transforms with j = 0..(w-1) are applied as follows:
-
Set T[ i ] equal to Residual[ i ][ j ] for i = 0..(h-1).
-
If Lossless is equal to 1, invoke the Inverse WHT process as specified in [section 7.10.1.12][] with shift equal to 0.
-
Otherwise, if txType is equal to one of DCT_DCT, DCT_ADST, DCT_FLIPADST or V_DCT, apply an inverse DCT as follows:
-
If clampCol is equal to 1, invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to log2H.
-
Invoke the inverse DCT permutation process as specified in [section 7.10.1.3][] with the input variable n equal to log2H.
-
Invoke the inverse DCT process as specified in [section 7.10.1.4][] with the input variable n equal to log2H, the input variable highPrecision equal to 1, and the input variable clamp equal to clampCol.
-
-
Otherwise, if txType is equal to one of ADST_DCT, ADST_ADST, FLIPADST_DCT, FLIPADST_FLIPADST, ADST_FLIPADST, FLIPADST_ADST, V_ADST, or V_FLIPADST, invoke the inverse ADST process as specified in [section 7.10.1.11][] with input variable n equal to log2H, the input variable highPrecision equal to 1, and the input variable clamp equal to clampCol.
-
Otherwise, invoke the inverse identity transform process specified in [section 7.10.1.18][] with the input variable n equal to log2H.
-
T[ i ] is set equal to Round2( T[ i ], colShift ) for i = 0..(h-1).
-
Invoke the intermediate clamping process specified in [section 7.10.1.2][] with the input variable n equal to log2H.
-
Residual[ i ][ j ] is set equal to T[ i ] for i = 0..(h-1).
where Transform_Row_Shift is defined as:
Transform_Row_Shift[ TX_SIZES_ALL ] = {
0, 1, 2, 2, 2, 0, 0, 1, 1,
1, 1, 1, 1, 1, 1, 2, 2, 2, 2
}Input to this process is the array CurrFrame of reconstructed samples.
Output from this process is a modified array CurrFrame containing deblocked samples.
The purpose of the loop filter is to eliminate (or at least reduce) visually objectionable artifacts associated with the semi-independence of the coding of super blocks and their constituent sub-blocks.
The loop filter is applied on all vertical boundaries followed by all horizontal boundaries as follows:
for ( plane = 0; plane < NumPlanes; plane++ ) {
if ( ( plane == 0 && ( loop_filter_level[ 0 ] || loop_filter_level[ 1 ] ) ) ||
( plane > 0 && loop_filter_level[ 1 + plane ] ) ) {
for ( pass = 0; pass < 2; pass++ ) {
rowStep = ( plane > 0 ) ? 1 : ( 1 << subsampling_y )
colStep = ( plane > 0 ) ? 1 : ( 1 << subsampling_x )
for ( row = 0; row < MiRows; row += rowStep )
for ( col = 0; col < MiCols; col += colStep )
loop_filter_edge( plane, pass, row, col )
}
}
}When the function loop_filter_edge is called, the edge loop filter process specified in [section 7.11.1][] is invoked with the variables plane, pass, row, and col as inputs.
Note: The loop filter is an integral part of the decoding process, in that the results of loop filtering are used in the prediction of subsequent frames. {:.alert .alert-info }
Note: The loop filtering is designed so that any order of filtering for the edges will give identical results, provided that the vertical boundaries are filtered before the horizontal boundaries. {:.alert .alert-info }
Note: The loop filter applies after the macroblocks have been "reconstructed" (i.e., had their prediction summed with their residual); correct decoding is predicated on the fact that already-constructed portions of the current frame referenced via intra prediction are not yet filtered. {:.alert .alert-info }
The inputs to this process are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
a variable pass specifying the direction of the edges. pass equal to 0 means we are filtering vertical block boundaries, and pass equal to 1 means we are filtering horizontal block boundaries,
-
variables row and col specifying the location of the edge in units of 4x4 blocks in the luma plane.
The outputs of this process are modified values in the array CurrFrame.
The variables subX and subY describing the subsampling of the current plane are derived as follows:
-
If plane is equal to 0, subX and subY are set equal to 0.
-
Otherwise (plane is not equal to 0), subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The variables dx and dy are derived as follows:
-
If pass is equal to 0, then dx is set equal to 1, dy is set equal to 0.
-
Otherwise (pass is equal to 1), dy is set equal to 1, dx is set equal to 0.
dx and dy specify the offset between the samples to be filtered.
The variable x is set equal to col * MI_SIZE.
The variable y is set equal to row * MI_SIZE.
x and y contain the location in luma coordinates.
The variable onScreen (equal to 1 if the samples on both sides of the boundary lie in the visible area) is derived as follows:
-
If x is greater than or equal to FrameWidth, onScreen is set equal to 0.
-
Otherwise, if y is greater than or equal to FrameHeight, onScreen is set equal to 0.
-
Otherwise, if pass is equal to 0 and x is equal to 0, onScreen is set equal to 0.
-
Otherwise, if pass is equal to 1 and y is equal to 0, onScreen is set equal to 0.
-
Otherwise, onScreen is set equal to 1.
If onScreen is equal to 0, then this process immediately returns and no filtering is applied to this edge.
The variables xP and yP (containing the location in the current plane) are derived as follows:
-
Set xP equal to x >> subX
-
Set yP equal to y >> subY
The variables prevRow and prevCol (containing the location of the mode info block on the other side of the boundary) are derived as follows:
-
Set prevRow equal to row - ( dy << subY )
-
Set prevCol equal to col - ( dx << subX )
Set the variable MiSize equal to MiSizes[ row ][ col ].
Set the variable txSz equal to LoopfilterTxSizes[ plane ][ row ][ col ].
Set the variable planeSize equal to get_plane_residual_size( MiSize, plane )
Set the variable skip equal to Skips[ row ][ col ].
Set the variable isIntra equal to RefFrames[ row ][ col ][ 0 ] <= INTRA_FRAME.
Set the variable prevTxSz equal to get_tx_size( plane, TxSizes[ prevRow ][ prevCol ] ).
Set the variable prevSkip equal to Skips[ prevRow ][ prevCol ].
Set the variable prevIsIntra equal to RefFrames[ prevRow ][ prevCol ][ 0 ] <= INTRA_FRAME.
The variable isBlockEdge (equal to 1 if the samples cross a prediction block edge) is derived as follows:
-
If pass is equal to 0 and xP is an exact multiple of Block_Width[ planeSize ], isBlockEdge is set equal to 1.
-
Otherwise, if pass is equal to 1 and yP is an exact multiple of Block_Height[ planeSize ], isBlockEdge is set equal to 1.
-
Otherwise, isBlockEdge is set equal to 0.
The variable isTxEdge (equal to 1 if the samples cross a transform block edge) is derived as follows:
-
If pass is equal to 0 and xP is an exact multiple of Tx_Width[ txSz ], isTxEdge is set equal to 1.
-
Otherwise, if pass is equal to 1 and yP is an exact multiple of Tx_Height[ txSz ], isTxEdge is set equal to 1.
-
Otherwise, isTxEdge is set equal to 0.
The variable isTileEdge (equal to 1 if the samples come from different tiles) is derived as follows:
-
If TileNums[ prevRow ][ prevCol ] is equal to TileNums[ row ][ col ], isTileEdge is set equal to 0.
-
Otherwise, isTileEdge is set equal to 1
The variable applyFilter (equal to 1 if the samples are filtered) is derived as follows:
-
If isTxEdge is equal to 0, applyFilter is set equal to 0.
-
Otherwise, if pass is equal to 0 and loop_filter_across_tiles_v is equal to 0 and isTileEdge is equal to 1, applyFilter is set equal to 0
-
Otherwise, if pass is equal to 1 and loop_filter_across_tiles_h is equal to 0 and isTileEdge is equal to 1, applyFilter is set equal to 0
-
Otherwise, if isBlockEdge is equal to 1 or skip is equal to 0 or isIntra is equal to 1 or prevSkip is equal to 0 or prevIsIntra is equal to 1, applyFilter is set equal to 1.
-
Otherwise applyFilter is set equal to 0.
The filter size process specified in [section 7.11.2][] is invoked with the inputs txSz, prevTxSz, pass, subX, subY, and plane, and the output assigned to the variable filterSize (containing the maximum filter size that can be used).
The adaptive filter strength process specified in [section 7.11.3][] is invoked with the inputs row, col, plane, and pass, and the output assigned to the variables lvl, limit, blimit, and thresh.
If lvl is equal to 0, the adaptive filter strength process specified in [section 7.11.3][] is invoked with the inputs prevRow, prevCol, plane, and pass, and the output assigned to the variables lvl, limit, blimit, and thresh.
For the variable i taking values from 0 to MI_SIZE - 1, the following applies:
- If applyFilter is equal to 1 and lvl is greater than zero, the sample filtering process specified in [section 7.11.5][] is invoked with the input variable x set equal to xP + dy * i, the input variable y set equal to yP + dx * i, and the variables plane, limit, blimit, thresh, dx, dy, and filterSize supplied as inputs.
Note: the vector (dx,dy) represents the direction of the filter, while (dy,dx) represents the direction of the boundary. {:.alert .alert-info }
The inputs to this process are:
-
a variable txSz specifying the size of the transform block,
-
a variable prevTxSz specifying the size of the transform block on the other side of the boundary,
-
a variable pass specifying the direction of the edges,
-
variables subX and subY describing the subsampling of the current plane,
-
a variable plane specifying whether we are filtering Y, U, or V samples.
The output of this process is the variable filterSize containing the maximum filter size that can be used.
The purpose of this process is to reduce the width of the chroma filters and to ensure that different boundaries can be filtered in parallel.
The variable baseSize is set equal to Min(prevTxSz, txSz).
The output variable filterSize is derived as follows:
-
If plane is equal to 0, filterSize is set equal to Min( TX_16X16, baseSize ),
-
Otherwise, (plane is greater than 0), filterSize is set equal to Min( TX_8X8, baseSize ).
The inputs to this process are:
-
the variables row and col specifying the luma location in units of 4x4 blocks,
-
the variable plane specifying whether we are filtering Y, U or V samples,
-
the variable pass specifying the direction of the edge being filtered. pass equal to 0 means we are filtering vertical block boundaries, and pass equal to 1 means we are filtering horizontal block boundaries.
The outputs of this process are the variables lvl, limit, blimit, and thresh.
The output variable lvl is derived as follows:
-
The variable segment is set equal to SegmentIds[ row ][ col ].
-
The variable ref is set equal to RefFrames[ row ][ col ][ 0 ].
-
The variable mode is set equal to YModes[ row ][ col ].
-
The variable modeType is derived as follows:
-
If mode is equal to NEARESTMV or NEARMV or NEWMV, modeType is set equal to 1.
-
Otherwise (if mode is an intra type or GLOBALMV or GLOBAL_GLOBALMV), modeType is set equal to 0.
-
-
The variable deltaLF is derived as follows:
-
If delta_lf_multi is equal to 0, deltaLF is set equal to DeltaLFs[ row ][ col ][ 0 ].
-
Otherwise (delta_lf_multi is equal to 1), deltaLF is set equal to DeltaLFs[ row ][ col ][ ( plane == 0 ) ? pass : ( plane + 1 ) ].
-
-
The adaptive filter strength selection process specified in [section 7.11.4][] is invoked, with segment, ref, modeType, deltaLF, plane, and pass as inputs, and the output being the output variable lvl.
The variable shift is derived as follows:
-
If loop_filter_sharpness is greater than 4, shift is set equal to 2.
-
Otherwise, if loop_filter_sharpness is greater than 0, shift is set equal to 1.
-
Otherwise, shift is set equal to 0.
The output variable limit is derived as follows:
-
If loop_filter_sharpness is greater than 0, limit is set equal to Clip3( 1, 9 - loop_filter_sharpness, lvl >> shift ).
-
Otherwise, limit is set equal to Max( 1, lvl >> shift ).
The output variable blimit is set equal to 2 * (lvl + 2) + limit.
The output variable thresh is set equal to lvl >> 4.
The inputs to this process are:
-
The variable segment, specifying the current segment id,
-
The variable ref, specifying the reference frame type (INTRA_FRAME, LAST_FRAME, etc.),
-
The variable modeType, specifying the loop filter mode type,
-
The variable deltaLF, specifying the loop filter delta value,
-
The variable plane, specifying whether we are filtering Y, U or V samples,
-
The variable pass, specifying the direction of the edge being filtered. pass equal to 0 means we are filtering vertical block boundaries, and pass equal to 1 means we are filtering horizontal block boundaries.
The output of this process is a filter strength level.
This process is invoked to select a loop filter strength level.
The variable i is set equal to ( plane == 0 ) ? pass : ( plane + 1 ).
The variable baseFilterLevel is set equal to Clip3( 0, MAX_LOOP_FILTER, deltaLF + loop_filter_level[ i ] ).
The following ordered steps apply:
-
The variable lvlSeg is set equal to baseFilterLevel.
-
The variable feature is set equal to SEG_LVL_ALT_LF_Y_V + i.
-
If seg_feature_active_idx( segment, feature ) is equal to 1 the following ordered steps apply:
a. lvlSeg is set equal to FeatureData[ segment ][ feature ] + lvlSeg.
b. lvlSeg is set equal to Clip3( 0, MAX_LOOP_FILTER, lvlSeg ).
-
If loop_filter_delta_enabled is equal to 1, then the following ordered steps apply:
a. The variable nShift is set equal to lvlSeg >> 5.
b. If ref is equal to INTRA_FRAME, then lvlSeg is set equal to lvlSeg + ( loop_filter_ref_deltas[ INTRA_FRAME ] << nShift ).
c. Otherwise, if ref is not equal to INTRA_FRAME, then lvlSeg is set equal to lvlSeg + ( loop_filter_ref_deltas[ ref ] << nShift ) + ( loop_filter_mode_deltas[ modeType ] << nShift ).
d. lvlSeg is set equal to Clip3(0, MAX_LOOP_FILTER, lvlSeg).
-
Return lvlSeg.
The inputs to this process are:
-
variables x and y specifying the location within CurrFrame[ plane ],
-
a variable plane specifying whether the block is the Y, U or V plane,
-
variables limit, blimit, thresh that specify the strength of the filtering operation,
-
variables dx and dy specifying the direction perpendicular to the edge being filtered,
-
a variable filterSize of specifying the maximum size of filter allowed.
The outputs of this process are modified values in the array CurrFrame.
First the filter mask process specified in [section 7.11.5.1][] is invoked with the inputs x, y, plane, limit, blimit, thresh, dx, dy, and filterSize, and the output is assigned to the variables hevMask, filterMask, flatMask, and flatMask2.
Then the appropriate filter process is invoked with the inputs x, y, plane, dx, dy as follows:
-
If filterMask is equal to 0, no filter is invoked.
-
Otherwise, if filterSize is equal to TX_4X4 or flatMask is equal to 0, the narrow filter process specified in [section 7.11.5.2][] is invoked with the additional input variable hevMask.
-
Otherwise, if filterSize is equal to TX_8X8 or flatMask2 is equal to 0, the wide filter process specified in [section 7.11.5.3][] is invoked with the additional input variable log2Size set to 3.
-
Otherwise, the wide filter process specified in [section 7.11.5.3][] is invoked with the additional input variable log2Size set to 4.
The inputs to this process are:
-
variables x and y specifying the location within CurrFrame[ plane ],
-
a variable plane specifying whether the block is the Y, U or V plane,
-
variables limit, blimit, thresh that specify the strength of the filtering operation,
-
variables dx and dy specifying the direction perpendicular to the edge being filtered,
-
a variable filterSize of specifying the maximum size of filter allowed.
The outputs from this process are the variables:
-
hevMask,
-
filterMask,
-
flatMask, (only used if filterSize >= TX_8X8),
-
flatMask2 (only used if filterSize >= TX_16X16).
The values output for these masks depend on the differences between samples on either side of the specified boundary. These samples are specified as follows:
q0 = CurrFrame[ plane ][ y ][ x ]
q1 = CurrFrame[ plane ][ y + dy ][ x + dx ]
q2 = CurrFrame[ plane ][ y + dy * 2 ][ x + dx * 2 ]
q3 = CurrFrame[ plane ][ y + dy * 3 ][ x + dx * 3 ]
q4 = CurrFrame[ plane ][ y + dy * 4 ][ x + dx * 4 ]
q5 = CurrFrame[ plane ][ y + dy * 5 ][ x + dx * 5 ]
q6 = CurrFrame[ plane ][ y + dy * 6 ][ x + dx * 6 ]
p0 = CurrFrame[ plane ][ y - dy ][ x - dx ]
p1 = CurrFrame[ plane ][ y - dy * 2 ][ x - dx * 2 ]
p2 = CurrFrame[ plane ][ y - dy * 3 ][ x - dx * 3 ]
p3 = CurrFrame[ plane ][ y - dy * 4 ][ x - dx * 4 ]
p4 = CurrFrame[ plane ][ y - dy * 5 ][ x - dx * 5 ]
p5 = CurrFrame[ plane ][ y - dy * 6 ][ x - dx * 6 ]
p6 = CurrFrame[ plane ][ y - dy * 7 ][ x - dx * 7 ]Note: Samples q4, q5, q6, p4, p5, and p6 are only used if filterSize is equal to TX_16X16. {:.alert .alert-info }
The value of hevMask indicates whether the sample has high edge variance. It is calculated as follows:
hevMask = 0
threshBd = thresh << (BitDepth - 8)
hevMask |= (Abs( p1 - p0 ) > threshBd)
hevMask |= (Abs( q1 - q0 ) > threshBd)The variable filterLen, representing the number of taps each side of the central sample in the filter, is derived as follows:
-
If filterSize is equal to TX_4X4, filterLen is set equal to 4.
-
Otherwise, if plane is not equal to 0, filterLen is set equal to 6.
-
Otherwise, if filterSize is equal to TX_8X8, filterLen is set equal to 8.
-
Otherwise, filterLen is set equal to 16.
The value of filterMask indicates whether adjacent samples close to the edge (within four samples either side of the specified boundary) vary by less than the limits given by limit and blimit. It is used to determine if any filtering should occur and is calculated as follows:
limitBd = limit << (BitDepth - 8)
blimitBd = blimit << (BitDepth - 8)
mask = 0
if ( filterLen >= 4 ) {
mask |= (Abs( p1 - p0 ) > limitBd)
mask |= (Abs( q1 - q0 ) > limitBd)
mask |= (Abs( p0 - q0 ) * 2 + Abs( p1 - q1 ) / 2 > blimitBd)
} else if ( filterLen >= 6 ) {
mask |= (Abs( p2 - p1 ) > limitBd)
mask |= (Abs( q2 - q1 ) > limitBd)
} else if ( filterLen >= 8 ) {
mask |= (Abs( p3 - p2 ) > limitBd)
mask |= (Abs( q3 - q2 ) > limitBd)
}
filterMask = (mask == 0)The value of flatMask is only required when filterSize >= TX_8X8. It measures whether samples from each side of the specified boundary are in a flat region. That is whether those samples are at most (1 << (BitDepth - 8)) different from the sample on the boundary. It is calculated as follows:
thresholdBd = 1 << (BitDepth - 8)
if ( filterSize >= TX_8X8 ) {
mask = 0
mask |= (Abs( p1 - p0 ) > thresholdBd)
mask |= (Abs( q1 - q0 ) > thresholdBd)
mask |= (Abs( p2 - p0 ) > thresholdBd)
mask |= (Abs( q2 - q0 ) > thresholdBd)
if ( filterLen >= 8 ) {
mask |= (Abs( p3 - p0 ) > thresholdBd)
mask |= (Abs( q3 - q0 ) > thresholdBd)
}
flatMask = (mask == 0)
}The value of flatMask2 is only required when filterSize >= TX_16X16. It measures whether at least seven samples from each side of the specified boundary are in a flat region assuming the first four on each side are (so the full region is flat if flatMask & flatMask2 == 0). The value of flatMask2 is calculated as follows:
thresholdBd = 1 << (BitDepth - 8)
if ( filterSize >= TX_16X16 ) {
mask = 0
mask |= (Abs( p6 - p0 ) > thresholdBd)
mask |= (Abs( q6 - q0 ) > thresholdBd)
mask |= (Abs( p5 - p0 ) > thresholdBd)
mask |= (Abs( q5 - q0 ) > thresholdBd)
mask |= (Abs( p4 - p0 ) > thresholdBd)
mask |= (Abs( q4 - q0 ) > thresholdBd)
flatMask2 = (mask == 0)
}The inputs to this filter are:
-
a variable hevMask specifying whether this is a high edge variance case,
-
variables x, y specifying the the location within CurrFrame[ plane ],
-
a variable plane specifying whether the block is the Y, U or V plane,
-
variables limit, blimit, thresh that specify the strength of the filtering operation,
-
variables dx and dy specifying the direction perpendicular to the edge being filtered.
This process modifies up to two samples on each side of the specified boundary depending on the value of hevMask as follows:
-
If hevMask is equal to 0 (i.e. the samples do not have high edge variance), this process modifies two samples on each side of the specified boundary, using a filter constructed from just the inner two (one from each side of the specified boundary).
-
Otherwise (the samples do have high edge variance), this process only modifies the one value on each side of the specified boundary, using a filter constructed from four input samples (two from each side of the specified boundary).
The process subtracts 0x80 << (BitDepth - 8) from the input sample values so that they are in the range -(1 << (BitDepth - 1)) to (1 << (BitDepth - 1)) - 1 inclusive. Intermediate values are made to be in this range by the following function:
filter4_clamp( value ) {
return Clip3( -(1 << (BitDepth - 1)), (1 << (BitDepth - 1)) - 1, value )
}The process is specified as follows:
q0 = CurrFrame[ plane ][ y ][ x ]
q1 = CurrFrame[ plane ][ y + dy ][ x + dx ]
p0 = CurrFrame[ plane ][ y - dy ][ x - dx ]
p1 = CurrFrame[ plane ][ y - dy * 2 ][ x - dx * 2 ]
ps1 = p1 - (0x80 << (BitDepth - 8))
ps0 = p0 - (0x80 << (BitDepth - 8))
qs0 = q0 - (0x80 << (BitDepth - 8))
qs1 = q1 - (0x80 << (BitDepth - 8))
filter = hevMask ? filter4_clamp( ps1 - qs1 ) : 0
filter = filter4_clamp( filter + 3 * (qs0 - ps0) )
filter1 = filter4_clamp( filter + 4 ) >> 3
filter2 = filter4_clamp( filter + 3 ) >> 3
oq0 = filter4_clamp( qs0 - filter1 ) + (0x80 << (BitDepth - 8))
op0 = filter4_clamp( ps0 + filter2 ) + (0x80 << (BitDepth - 8))
CurrFrame[ plane ][ y ][ x ] = oq0
CurrFrame[ plane ][ y - dy ][ x - dx ] = op0
if ( !hevMask ) {
filter = Round2( filter1, 1 )
oq1 = filter4_clamp( qs1 - filter ) + (0x80 << (BitDepth - 8))
op1 = filter4_clamp( ps1 + filter ) + (0x80 << (BitDepth - 8))
CurrFrame[ plane ][ y + dy ][ x + dx ] = oq1
CurrFrame[ plane ][ y - dy * 2 ][ x - dx * 2 ] = op1
}The inputs to this filter are:
-
variables x, y specifying the the location within CurrFrame[ plane ],
-
a variable plane specifying whether the block is the Y, U or V plane,
-
variables dx and dy specifying the direction perpendicular to the edge being filtered,
-
a variable log2Size specifying the base 2 logarithm of the number of taps.
This filter is only applied when samples from each side of the boundary are detected to be in a flat region.
The variable n (specifying the number of filter taps on each side of the central sample) is set as follows:
-
If log2Size is equal to 4, n is set equal to 6.
-
Otherwise if plane is equal to 0, n is set equal to 3.
-
Otherwise (log2Size is equal to 3 and plane is greater than 0), n is set equal to 2.
The variable n2 (specifying the number of filter taps equal to 2 on each side of the central sample needed to give a unity DC gain) is set as follows:
-
If log2Size is equal to 3 and plane is equal to 0, n2 is set equal to 0.
-
Otherwise (log2Size is equal to 4 or plane is greater than 0), n2 is set equal to 1.
This process modifies the samples on each side of the specified boundary by applying a low pass filter as follows:
for( i = -n; i < n; i++ ) {
t = 0
for( j = -n; j <= n; j++ ) {
p = Clip3( -( n + 1 ), n, i + j )
tap = ( Abs( j ) <= n2 ) ? 2 : 1
t += CurrFrame[ plane ][ y + p * dy ][ x + p * dx ] * tap
}
F[ i ] = Round2( t, log2Size )
}
for( i = -n; i < n; i++ )
CurrFrame[ plane ][ y+i * dy ][ x+i * dx ] = F[ i ]where F is an array with indices from -n to n - 1 used to store the filtered results.
Input to this process is the array CurrFrame of reconstructed samples.
Output from this process is the array CdefFrame containing deringed samples.
The purpose of CDEF is to perform deringing based on the detected direction of blocks.
CDEF parameters are stored for each 64 by 64 block of pixels.
The CDEF filter is applied on each 8 by 8 block of pixels as follows:
step4 = Num_4x4_Blocks_Wide[ BLOCK_8X8 ]
cdefSize4 = Num_4x4_Blocks_Wide[ BLOCK_64X64 ]
cdefMask4 = ~(cdefSize4 - 1)
for (r = 0; r < MiRows; r += step4) {
for (c = 0; c < MiCols; c += step4) {
baseR = r & cdefMask4
baseC = c & cdefMask4
idx = cdef_idx[ baseR ][ baseC ]
cdef_block(r, c, idx)
}
}When the cdef_block function is called, the CDEF block process specified in [section 7.12.1][] is invoked with r, c, and idx as inputs.
The inputs to this process are:
-
variables r and c specifying the location of an 8x8 block in units of 4x4 blocks in the luma plane,
-
a variable idx specifying which set of CDEF parameters to use, or -1 to signal that no filtering should be applied.
The block is first copied to the CdefFrame as follows:
startY = r * MI_SIZE
endY = startY + MI_SIZE * 2
startX = c * MI_SIZE
endX = startX + MI_SIZE * 2
for( y = startY; y < endY; y++ ) {
for( x = startX; x < endX; x++ ) {
CdefFrame[ 0 ][ y ][ x ] = CurrFrame[ 0 ][ y ][ x ]
}
}
startY >>= subsampling_y
endY >>= subsampling_y
startX >>= subsampling_x
endX >>= subsampling_x
for( y = startY; y < endY; y++ ) {
for( x = startX; x < endX; x++ ) {
CdefFrame[ 1 ][ y ][ x ] = CurrFrame[ 1 ][ y ][ x ]
CdefFrame[ 2 ][ y ][ x ] = CurrFrame[ 2 ][ y ][ x ]
}
}Note If CDEF filtering turns out to be needed, then the contents of CdefFrame will be overwritten later in this process. {:.alert .alert-info }
If idx is equal to -1, then the process returns immediately after performing this copy.
The variable coeffShift is set equal to BitDepth - 8.
The variable skip is set equal to Skips[ r ][ c ].
If skip is equal to 0, the CDEF direction process specified in [section 7.12.2][] is invoked with r and c as inputs, and the outputs assigned to variables yDir and var.
If skip is equal to 0, the following ordered steps apply:
-
The variable priStr is set equal to cdef_y_pri_strength[ idx ] << coeffShift.
-
The variable secStr is set equal to cdef_y_sec_strength[ idx ] << coeffShift.
-
The variable dir is set equal to ( priStr == 0 ) ? 0 : yDir.
-
The variable varStr is set equal to ( var >> 6 ) ? Min( floor( log2( var >> 6 ) ), 12) : 0.
-
The variable priStr is set equal to ( var ? ( priStr * ( 4 + varStr ) + 8 ) >> 4 : 0 ).
-
The variable damping is set equal to CdefDamping + coeffShift.
-
The CDEF filter process specified in [section 7.12.3][] is invoked with plane equal to 0, r, c, priStr, secStr, damping, and dir as input.
-
If NumPlanes is equal to 1, the process terminates at this point (i.e. filtering is not done for the U and V planes).
-
The variable priStr is set equal to cdef_uv_pri_strength[ idx ] << coeffShift.
-
The variable secStr is set equal to cdef_uv_sec_strength[ idx ] << coeffShift.
-
The variable dir is set equal to ( priStr == 0 ) ? 0 : Cdef_Uv_Dir[ subsampling_x ][ subsampling_y ][ yDir ].
-
The variable damping is set equal to CdefDamping + coeffShift - 1.
-
The CDEF filter process specified in [section 7.12.3][] is invoked with plane equal to 1, r, c, priStr, secStr, damping, and dir as input.
-
The CDEF filter process specified in [section 7.12.3][] is invoked with plane equal to 2, r, c, priStr, secStr, damping, and dir as input.
Cdef_Uv_Dir is a constant lookup table defined as:
Cdef_Uv_Dir[ 2 ][ 2 ][ 8 ] = {
{ {0, 1, 2, 3, 4, 5, 6, 7},
{1, 2, 2, 2, 3, 4, 6, 0} },
{ {7, 0, 2, 4, 5, 6, 6, 6},
{0, 1, 2, 3, 4, 5, 6, 7} }
}The inputs to this process are variables r and c specifying the location of an 8x8 block in units of 4x4 blocks in the luma plane.
The outputs of this process are:
-
a variable yDir containing the direction of this block,
-
a variable var containing the variance for this block.
This block uses luma samples to measure the direction and variance of a block.
The process is specified as:
for (i = 0; i < 8; i++) {
cost[i] = 0
for (j = 0; j < 15; j++)
partial[i][j] = 0
}
bestCost = 0
yDir = 0
x0 = c << MI_SIZE_LOG2
y0 = r << MI_SIZE_LOG2
for (i = 0; i < 8; i++) {
for (j = 0; j < 8; j++) {
x = (CurrFrame[ 0 ][y0 + i][x0 + j] >> (BitDepth - 8)) - 128
partial[0][i + j] += x
partial[1][i + j / 2] += x
partial[2][i] += x
partial[3][3 + i - j / 2] += x
partial[4][7 + i - j] += x
partial[5][3 - i / 2 + j] += x
partial[6][j] += x
partial[7][i / 2 + j] += x
}
}
for (i = 0; i < 8; i++) {
cost[2] += partial[2][i] * partial[2][i]
cost[6] += partial[6][i] * partial[6][i]
}
cost[2] *= Div_Table[8]
cost[6] *= Div_Table[8]
for (i = 0; i < 7; i++) {
cost[0] += (partial[0][i] * partial[0][i] +
partial[0][14 - i] * partial[0][14 - i]) *
Div_Table[i + 1]
cost[4] += (partial[4][i] * partial[4][i] +
partial[4][14 - i] * partial[4][14 - i]) *
Div_Table[i + 1]
}
cost[0] += partial[0][7] * partial[0][7] * Div_Table[8]
cost[4] += partial[4][7] * partial[4][7] * Div_Table[8]
for (i = 1; i < 8; i += 2) {
for (j = 0; j < 4 + 1; j++) {
cost[i] += partial[i][3 + j] * partial[i][3 + j]
}
cost[i] *= Div_Table[8]
for (j = 0; j < 4 - 1; j++) {
cost[i] += (partial[i][j] * partial[i][j] +
partial[i][10 - j] * partial[i][10 - j]) *
Div_Table[2 * j + 2]
}
}
for (i = 0; i < 8; i++) {
if (cost[i] > bestCost) {
bestCost = cost[i]
yDir = i
}
}
var = (bestCost - cost[(yDir + 4) & 7]) >> 10where the Div_Table is a constant lookup table specified as:
Div_Table[9] = {
0, 840, 420, 280, 210, 168, 140, 120, 105
}The inputs to this process are:
-
a variable plane specifying which plane is being predicted,
-
variables r and c specifying the location of an 8x8 block in units of 4x4 blocks in the luma plane,
-
a variable priStr specifying the primary filter strength,
-
a variable secStr specifying the secondary filter strength,
-
a variable damping specifying a shift used for damping,
-
a variable dir specifying the detected direction of the block.
The process modifies samples in CdefFrame based on filtering samples from CurrFrame.
MiColStart, MiRowStart, MiColEnd, MiRowEnd are set equal to the values they had when the syntax element MiSizes[ r ][ c ] was written.
Note: These variables are used by the is_inside_filter_region function to determine which samples are available for use in filtering. {:.alert .alert-info }
The variable coeffShift is set equal to BitDepth - 8.
The filtering is applied as follows:
subX = (plane > 0) ? subsampling_x : 0
subY = (plane > 0) ? subsampling_y : 0
x0 = (c * MI_SIZE ) >> subX
y0 = (r * MI_SIZE ) >> subY
w = 8 >> subX
h = 8 >> subY
for ( i = 0; i < h; i++ ) {
for (j = 0; j < w; j++) {
sum = 0
x = CurrFrame[plane][y0 + i][x0 + j]
max = x
min = x
for (k = 0; k < 2; k++) {
for (sign = -1; sign <= 1; sign += 2) {
p = cdef_get_at(plane, x0, y0, i, j, dir, k, sign, subX, subY)
sum += Cdef_Pri_Taps[(priStr >> coeffShift) & 1][k] * constrain(p - x, priStr, damping)
if (p != CDEF_VERY_LARGE) {
max = Max(p, max)
min = Min(p, min)
}
for ( dirOff = -2; dirOff <= 2; dirOff += 4) {
s = cdef_get_at(plane, x0, y0, i, j, (dir + dirOff) & 7, k, sign, subX, subY)
sum += Cdef_Sec_Taps[(priStr >> coeffShift) & 1][k] * constrain(s - x, secStr, damping)
if (s != CDEF_VERY_LARGE) {
max = Max(s, max)
min = Min(s, min)
}
}
}
}
CdefFrame[plane][y0 + i][x0 + j] = Clip3(min, max, x + ((8 + sum - (sum < 0)) >> 4) )
}
}where Cdef_Pri_Taps and Cdef_Sec_Taps are constant lookup tables specified as:
Cdef_Pri_Taps[2][2] = {
{ 4, 2 }, { 3, 3 }
}
Cdef_Sec_Taps[2][2] = {
{ 2, 1 }, { 2, 1 }
}constrain is specified as:
constrain(diff, threshold, damping) {
if ( !threshold )
return 0
dampingAdj = Max(0, damping - floor( log2( threshold ) ) )
sign = (diff < 0) ? -1 : 1
return sign * Clip3(0, Abs(diff), threshold - (Abs(diff) >> dampingAdj) )
}and cdef_get_at fetches a pixel from CurrFrame if it is available, or returns CDEF_VERY_LARGE otherwise, and is defined as:
cdef_get_at(plane, x0, y0, i, j, dir, k, sign, subX, subY) {
if ( subX != subY )
dir = subX ? Cdef_422Directions[ dir ] : Cdef_440Directions[ dir ]
y = y0 + i + sign * Cdef_Directions[dir][k][0]
x = x0 + j + sign * Cdef_Directions[dir][k][1]
candidateR = (y << subY) >> MI_SIZE_LOG2
candidateC = (x << subX) >> MI_SIZE_LOG2
if ( is_inside_filter_region( candidateR, candidateC ) )
return CurrFrame[ plane ][ y ][ x ]
else
return CDEF_VERY_LARGE
}where Cdef_Directions, Cdef_422Directions and Cdef_440Directions are constant lookup tables defined as:
Cdef_Directions[8][2][2] = {
{ { -1, 1 }, { -2, 2 } },
{ { 0, 1 }, { -1, 2 } },
{ { 0, 1 }, { 0, 2 } },
{ { 0, 1 }, { 1, 2 } },
{ { 1, 1 }, { 2, 2 } },
{ { 1, 0 }, { 2, 1 } },
{ { 1, 0 }, { 2, 0 } },
{ { 1, 0 }, { 2, -1 } }
}
Cdef_422Directions[8] = { 7, 0, 2, 4, 5, 6, 6, 6 }
Cdef_440Directions[8] = { 1, 2, 2, 2, 3, 4, 6, 0 }Input to this process is an array inputFrame of width FrameWidth and height FrameHeight.
The output of this process is a horizontally upscaled frame of width UpscaledWidth and height FrameHeight.
If use_superres is equal to 0, no upscaling is required and this process returns inputFrame.
This process is specified as:
for (plane = 0; plane < NumPlanes; plane++) {
if ( plane > 0 ) {
subX = subsampling_x
subY = subsampling_y
} else {
subX = 0
subY = 0
}
downscaledPlaneW = Round2(FrameWidth, subX)
upscaledPlaneW = Round2(UpscaledWidth, subX)
planeH = Round2(FrameHeight, subY)
stepX = ((downscaledPlaneW << SUPERRES_SCALE_BITS) + (upscaledPlaneW / 2)) / upscaledPlaneW
err = (upscaledPlaneW * stepX) - (downscaledPlaneW << SUPERRES_SCALE_BITS)
initialSubpelX =
(-((upscaledPlaneW - downscaledPlaneW) << (SUPERRES_SCALE_BITS - 1)) + upscaledPlaneW / 2) / upscaledPlaneW +
(1 << (SUPERRES_EXTRA_BITS - 1)) - err / 2
initialSubpelX &= SUPERRES_SCALE_MASK
miW = MiCols >> subX
for (i = 0; i < TileCols; i++) {
downscaledX0 = MiColStarts[i] << (MI_SIZE_LOG2 - subX)
upscaledX0 = (downscaledX0 * superres_denom) / SUPERRES_NUM
if (i == TileCols - 1) {
fullTileX1 = miW * MI_SIZE
upscaledX1 = upscaledPlaneW
} else {
fullTileX1 = MiColStarts[i+1] << (MI_SIZE_LOG2 - subX)
upscaledX1 = (fullTileX1 * superres_denom) / SUPERRES_NUM
}
if (loop_filter_across_tiles_v) {
minX = 0
maxX = miW * MI_SIZE - 1
} else {
minX = downscaledX0
maxX = fullTileX1 - 1
}
for (y = 0; y < planeH; y++) {
for (x = upscaledX0; x < upscaledX1; x++) {
srcX = -(1 << SUPERRES_SCALE_BITS) + initialSubpelX + x*stepX
srcXPx = (srcX >> SUPERRES_SCALE_BITS)
srcXSubpel = (srcX & SUPERRES_SCALE_MASK) >> SUPERRES_EXTRA_BITS
sum = 0
for (k = 0; k < SUPERRES_FILTER_TAPS; k++) {
sampleX = Clip3(minX, maxX, srcXPx + (k - SUPERRES_FILTER_OFFSET))
px = frame[plane][sampleX][y]
sum += px * Upscale_Filter[srcXSubpel][k]
}
outputFrame[plane][x][y] = Clip1(Round2(sum, FILTER_BITS))
}
}
}
}where Upscale_Filter is specified as:
Upscale_Filter[SUPERRES_FILTER_SHIFTS][SUPERRES_FILTER_TAPS] = {
{ 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 0, -1, 128, 2, -1, 0, 0 },
{ 0, 1, -3, 127, 4, -2, 1, 0 }, { 0, 1, -4, 127, 6, -3, 1, 0 },
{ 0, 2, -6, 126, 8, -3, 1, 0 }, { 0, 2, -7, 125, 11, -4, 1, 0 },
{ -1, 2, -8, 125, 13, -5, 2, 0 }, { -1, 3, -9, 124, 15, -6, 2, 0 },
{ -1, 3, -10, 123, 18, -6, 2, -1 }, { -1, 3, -11, 122, 20, -7, 3, -1 },
{ -1, 4, -12, 121, 22, -8, 3, -1 }, { -1, 4, -13, 120, 25, -9, 3, -1 },
{ -1, 4, -14, 118, 28, -9, 3, -1 }, { -1, 4, -15, 117, 30, -10, 4, -1 },
{ -1, 5, -16, 116, 32, -11, 4, -1 }, { -1, 5, -16, 114, 35, -12, 4, -1 },
{ -1, 5, -17, 112, 38, -12, 4, -1 }, { -1, 5, -18, 111, 40, -13, 5, -1 },
{ -1, 5, -18, 109, 43, -14, 5, -1 }, { -1, 6, -19, 107, 45, -14, 5, -1 },
{ -1, 6, -19, 105, 48, -15, 5, -1 }, { -1, 6, -19, 103, 51, -16, 5, -1 },
{ -1, 6, -20, 101, 53, -16, 6, -1 }, { -1, 6, -20, 99, 56, -17, 6, -1 },
{ -1, 6, -20, 97, 58, -17, 6, -1 }, { -1, 6, -20, 95, 61, -18, 6, -1 },
{ -2, 7, -20, 93, 64, -18, 6, -2 }, { -2, 7, -20, 91, 66, -19, 6, -1 },
{ -2, 7, -20, 88, 69, -19, 6, -1 }, { -2, 7, -20, 86, 71, -19, 6, -1 },
{ -2, 7, -20, 84, 74, -20, 7, -2 }, { -2, 7, -20, 81, 76, -20, 7, -1 },
{ -2, 7, -20, 79, 79, -20, 7, -2 }, { -1, 7, -20, 76, 81, -20, 7, -2 },
{ -2, 7, -20, 74, 84, -20, 7, -2 }, { -1, 6, -19, 71, 86, -20, 7, -2 },
{ -1, 6, -19, 69, 88, -20, 7, -2 }, { -1, 6, -19, 66, 91, -20, 7, -2 },
{ -2, 6, -18, 64, 93, -20, 7, -2 }, { -1, 6, -18, 61, 95, -20, 6, -1 },
{ -1, 6, -17, 58, 97, -20, 6, -1 }, { -1, 6, -17, 56, 99, -20, 6, -1 },
{ -1, 6, -16, 53, 101, -20, 6, -1 }, { -1, 5, -16, 51, 103, -19, 6, -1 },
{ -1, 5, -15, 48, 105, -19, 6, -1 }, { -1, 5, -14, 45, 107, -19, 6, -1 },
{ -1, 5, -14, 43, 109, -18, 5, -1 }, { -1, 5, -13, 40, 111, -18, 5, -1 },
{ -1, 4, -12, 38, 112, -17, 5, -1 }, { -1, 4, -12, 35, 114, -16, 5, -1 },
{ -1, 4, -11, 32, 116, -16, 5, -1 }, { -1, 4, -10, 30, 117, -15, 4, -1 },
{ -1, 3, -9, 28, 118, -14, 4, -1 }, { -1, 3, -9, 25, 120, -13, 4, -1 },
{ -1, 3, -8, 22, 121, -12, 4, -1 }, { -1, 3, -7, 20, 122, -11, 3, -1 },
{ -1, 2, -6, 18, 123, -10, 3, -1 }, { 0, 2, -6, 15, 124, -9, 3, -1 },
{ 0, 2, -5, 13, 125, -8, 2, -1 }, { 0, 1, -4, 11, 125, -7, 2, 0 },
{ 0, 1, -3, 8, 126, -6, 2, 0 }, { 0, 1, -3, 6, 127, -4, 1, 0 },
{ 0, 1, -2, 4, 127, -3, 1, 0 }, { 0, 0, -1, 2, 128, -1, 0, 0 },
}It is a requirement of bitstream conformance that upscaledPlaneW is strictly greater than downscaledPlaneW.
The output of this process is equal to outputFrame.
Input to this process are the arrays UpscaledCurrFrame (of reconstructed samples) and UpscaledCdefFrame (of deringed samples).
Output from this process is the array LrFrame of loop restored samples.
Note: Although in this process we loop over 4x4 blocks, loop restoration is designed to work in stripes 64 luma pixels high without needing additional line buffers. Samples within the current stripe are fetched from UpscaledCdefFrame. Samples outside the current stripe are fetched from UpscaledCurrFrame (these samples will be deblocked, but will not have CDEF filtering applied). {:.alert .alert-info }
The array LrFrame is set equal to a copy UpscaledCdefFrame. (The contents of LrFrame will later be overwritten for blocks that require restoration filtering.)
If allow_intrabc is equal to 1, then the process returns immediately after performing this copy.
Otherwise, loop restoration is applied for each 4x4 block of each plane as follows:
for ( row = 0; row < MiRows; row++ ) {
for ( col = 0; col * MI_SIZE < UpscaledWidth; col++ ) {
for ( plane = 0; plane < NumPlanes; plane++ ) {
loop_restore_block( plane, row, col)
}
}
}When loop_restore_block is called, the loop restore block process in [section 7.14.1][] is invoked with plane, row, and col as inputs.
The inputs to this block are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
variables row and col specifying the location of the block in units of 4x4 blocks in the upscaled luma plane.
The output of this process are samples in LrFrame[ plane ].
The variables TileNum, MiColStart, MiRowStart, MiColEnd, MiRowEnd are set equal to the values they had when the syntax element MiSizes[ row ][ col ] was written.
The variable lumaY is set equal to row * MI_SIZE.
The variable stripeNum (specifying the zero-based index of the current stripe) is set equal to (lumaY + 8) / 64.
Note: The stripes are offset upwards by 8 luma pixels to make pipelined implementations more efficient. When a row of superblocks has been received, enough rows of deblocked output can be produced to allow loop restoration of the corresponding stripes. {:.alert .alert-info }
The variables subX and subY are set equal to the subsampling for the current plane as follows:
-
If plane is equal to 0, subX is set equal to 0 and subY is set equal to 0.
-
Otherwise, subX is set equal to subsampling_x and subY is set equal to subsampling_y.
The variable StripeStartY (specifying the start of the stripe in units of samples in the current plane) is set equal to ( (-8 + stripeNum * 64) >> subY ).
The variable StripeEndY (specifying the end of the stripe in units of samples in the current plane) is set equal to StripeStartY + (64 >> subY) - 1.
Note: StripeStartY and StripeEndY are used by the get source sample process to decide whether to fetch from UpscaledCurrFrame or UpscaledCdefFrame. {:.alert .alert-info }
The variable unitSize (specifying the size of restoration units in units of samples in the current plane) is set equal to LoopRestorationSize[ plane ].
The variable unitRows (specifying the number of restoration units down the tile) is set equal to count_units_in_tile( unitSize, (MiRowEnd - MiRowStart) * MI_SIZE >> subY ).
The variable tileLeft (specifying the left edge of the tile in units of samples in the current plane) is set equal to upscale_mi_to_pos(MiColStart, plane).
The variable tileRight (specifying the right edge of the tile in units of samples in the current plane) is set equal to upscale_mi_to_pos(MiColEnd, plane).
The variable unitCols (specifying the number of restoration units across the tile) is set equal to count_units_in_tile( unitSize, tileRight - tileLeft ).
Note: The number of restoration units in a tile can be different for chroma and luma. {:.alert .alert-info }
The variable unitRow (specifying the vertical index of the current loop restoration unit) is set equal to Min( unitRows - 1, ( (row - MiRowStart) * MI_SIZE >> subY ) / unitSize).
The variable unitCol (specifying the horizontal index of the current loop restoration unit) is set equal to Min( unitCols - 1, (col * MI_SIZE - tileLeft) / unitSize).
The horizontal extent of the space allowed for filtering is configured depending on loop_filter_across_tiles_v as follows:
-
If loop_filter_across_tiles_v is equal to 1, the following applies:
-
TileStartX is set equal to 0
-
TileEndX is set equal to Round2( UpscaledWidth, subX) - 1
-
-
Otherwise (loop_filter_across_tiles_v is equal to 0), the following applies:
-
TileStartX is set equal to tileLeft
-
TileEndX is set equal to tileRight - 1
-
The vertical extent of the space allowed for filtering is configured depending on loop_filter_across_tiles_h as follows:
-
If loop_filter_across_tiles_h is equal to 1, the following applies:
-
TileStartY is set equal to 0
-
TileEndY is set equal to Round2( FrameHeight, subY ) - 1
-
-
Otherwise (loop_filter_across_tiles_h is equal to 0), the following applies:
-
TileStartY is set equal to (MiRowStart * MI_SIZE >> subY)
-
TileEndY is set equal to Round2( Min( FrameHeight, MiRowEnd * MI_SIZE ), subY) - 1
-
The variable x is set equal to ( col * MI_SIZE >> subX ).
The variable y is set equal to ( row * MI_SIZE >> subY ).
The variable w is set equal to ( MI_SIZE >> subX ).
The variable h is set equal to ( MI_SIZE >> subY ).
(Variables x and y represent the position of the block in samples relative to the top-left corner of the current plane. Variables w and h represent the size of the block in samples.)
Note: Although we describe the filter as operating on small blocks, the output will be the same if larger blocks are used - provided all contained samples belong to the same loop restoration unit. {:.alert .alert-info }
The variable rType (specifying the loop restoration type) is set equal to LrType[ TileNum ][ plane ][ unitRow ][ unitCol ].
The filter to used depends on rType as follows:
-
If rType is equal to RESTORE_WIENER, the Wiener filter process specified in [section 7.14.4][] is invoked with plane, unitRow, unitCol, x, y, w, and h as inputs.
-
Otherwise, if rType is equal to RESTORE_SGRPROJ, the self guided filter process specified in [section 7.14.2][] is invoked with plane, unitRow, unitCol, x, y, w, and h as inputs.
-
Otherwise (rType is equal to RESTORE_NONE), no filtering is applied.
The inputs to this block are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
variables unitRow and unitCol specifying the offset of the loop restoration unit with the current tile,
-
variables x and y specifying the position of the block in samples relative to the top-left corner of the current plane,
-
variables w and h specifying the size of the block in samples.
The arrays flt1 and flt2 are prepared by the following ordered steps:
-
The variable set is set equal to LrSgrSet[ TileNum ][ plane ][ unitRow ][ unitCol ].
-
The variable pass is set equal to 0.
-
The box filter process specified in [section 7.14.3][] is invoked with plane, x, y, w, h, set, and pass as inputs, and the output assigned to flt1.
-
The variable pass is set equal to 1.
-
The box filter process specified in [section 7.14.3][] is invoked with plane, x, y, w, h, set, and pass as inputs, and the output assigned to flt2.
The restoration process is then applied for each sample as follows:
xqd0 = LrSgrXqd[ TileNum ][ plane ][ unitRow ][ unitCol ][ 0 ]
xqd1 = LrSgrXqd[ TileNum ][ plane ][ unitRow ][ unitCol ][ 1 ]
xq0 = xqd0
xq1 = (1 << SGRPROJ_PRJ_BITS) - xq0 - xqd1
for ( i = 0; i < h; i++ ) {
for ( j = 0; j < w; j++ ) {
u = UpscaledCdefFrame[ plane ][ y + i ][ x + j ] << SGRPROJ_RST_BITS
f1 = flt1[ i ][ j ] - u
f2 = flt2[ i ][ j ] - u
v = xq0 * f1 + xq1 * f2 + (u << SGRPROJ_PRJ_BITS)
w = Round2( v, SGRPROJ_RST_BITS + SGRPROJ_PRJ_BITS )
LrFrame[ plane ][ y + i ][ x + j ] = Clip1( w )
}
}The inputs to this process are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
variables x and y specifying the position of the block in samples relative to the top-left corner of the current plane,
-
variables w and h specifying the size of the block in samples,
-
a variable set specifying the strength of the filtering,
-
a variable pass (equal to 0 or 1), specifying if we are generating the first or second filtered output.
The output of this process is a 2d array F.
The variable r (specifying that the box filters have side length 2*r+1) is set equal to Sgr_Params[ set ][ pass * 2 + 0 ].
The variable eps (specifying a scaling for the output) is set equal to Sgr_Params[ set ][ pass * 2 + 1 ].
The 2d arrays A and B (note these arrays are valid for coordinates including an extra sample around the boundary) are prepared as follows:
n = ( 2 * r + 1 ) * ( 2 * r + 1 )
n2e = n * n * eps
s = (((1 << SGRPROJ_MTABLE_BITS) + n2e / 2) / n2e)
for ( i = -1; i < h + 1; i++ ) {
for ( j = -1; j < w + 1; j++ ) {
a = 0
b = 0
for ( dy = -r ; dy <= r; dy++ ) {
for ( dx = -r; dx <= r; dx++ ) {
c = get_source_sample( plane, x + j + dx, y + i + dy )
a += c * c
b += c
}
}
a = Round2( a, 2 * (BitDepth - 8) )
b = Round2( b, BitDepth - 8 )
p = Max( 0, a * n - b * b )
z = Round2( p * s, SGRPROJ_MTABLE_BITS )
if ( z >= 255 )
a2 = 256
else
a2 = ((z << SGRPROJ_SGR_BITS) + (z/2)) / (z + 1)
oneOverN = ((1 << SGRPROJ_RECIP_BITS) + (n/2)) / n
b2 = ( (1 << SGRPROJ_SGR_BITS) - a2 ) * b * oneOverN
A[ i ][ j ] = a2
B[ i ][ j ] = Round2( b2, SGRPROJ_RECIP_BITS )
}
}Note: When pass is equal to 0, only odd rows (i.e. entries A[ i ][ j ] and B[ i ][ j ] with i odd) will be used to generate the output.
{:.alert .alert-info }
where the call to get_source_sample specifies that the get source sample process specified in [section 7.14.6][] should be invoked and the output assigned to variable c.
After A and B are prepared, the output array F is generated as follows:
for ( i = 0; i < h; i++ ) {
shift = 5
if ( pass == 0 && ( i & 1 ) ) {
shift = 4
}
for ( j = 0; j < w; j++ ) {
a = 0
b = 0
for ( dy = -1 ; dy <= 1; dy++ ) {
for ( dx = -1; dx <= 1; dx++ ) {
if ( pass == 0 ) {
if ( (i + dy) & 1 ) {
weight = (dx == 0) ? 6 : 5
} else {
weight = 0
}
} else {
weight = (dx == 0 || dy == 0) ? 4 : 3
}
a += weight * A[ i + dy ][ j + dx ]
b += weight * B[ i + dy ][ j + dx ]
}
}
v = a * UpscaledCdefFrame[ plane ][ y + i ][ x + j ] + b
F[ i ][ j ] = Round2( v, SGRPROJ_SGR_BITS + shift - SGRPROJ_RST_BITS)
}
}Note: When pass is equal to 0, the weights for even rows of A and B are always equal to 0. {:.alert .alert-info }
The constant lookup table Sgr_Params is specified as:
Sgr_Params[ (1 << SGRPROJ_PARAMS_BITS) ][ 4 ] = {
{ 2, 12, 1, 4 }, { 2, 15, 1, 6 }, { 2, 18, 1, 8 }, { 2, 20, 1, 9 },
{ 2, 22, 1, 10 }, { 2, 25, 1, 11 }, { 2, 35, 1, 12 }, { 2, 45, 1, 13 },
{ 2, 55, 1, 14 }, { 2, 65, 1, 15 }, { 2, 75, 1, 16 }, { 2, 30, 1, 6 },
{ 2, 50, 1, 12 }, { 2, 60, 1, 13 }, { 2, 70, 1, 14 }, { 2, 80, 1, 15 }
}The inputs to this block are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
variables unitRow and unitCol specifying the offset of the loop restoration unit with the current tile,
-
variables x and y specifying the position of the block in samples relative to the top-left corner of the current plane,
-
variables w and h specifying the size of the block in samples.
The output from this process are modified pixels in LrFrame.
The sub-sample interpolation is effected via two one-dimensional convolutions. First a horizontal filter is used to build up a temporary array, and then this array is vertically filtered to obtain the final prediction.
The Wiener coefficient process specified in [section 7.14.5][] is invoked with an input of LrWiener[ TileNum ][ plane ][ unitRow ][ unitCol ][ 0 ] and the output assigned to vfilter.
The Wiener coefficient process specified in [section 7.14.5][] is invoked with an input of LrWiener[ TileNum ][ plane ][ unitRow ][ unitCol ][ 1 ] and the output assigned to hfilter.
Note: The horizontal filter needs to be applied before the vertical filter, but the horizontal coefficients are sent after the vertical coefficients. {:.alert .alert-info }
The filtering is applied as follows:
-
The array intermediate is specified as follows:
offset = (1 << (BitDepth + EXTRAPREC_BITS - 1)) limit = (1 << (BitDepth + 1 + EXTRAPREC_BITS)) - 1 for ( r = 0; r < h + 7; r++ ) { for ( c = 0; c < w; c++ ) { s = 0 for ( t = 0; t < 7; t++ ) s += hfilter[ t ] * get_source_sample( plane, x + c + t - 3, y + r - 3 ) v = Round2(s, FILTER_BITS - EXTRAPREC_BITS) intermediate[ r ][ c ] = Clip3( -offset, limit - offset, v) } }
Where the call to get_source_sample specifies that the get source sample process specified in [section 7.14.6][] should be invoked.
Note: The intermediate result is clipped so that ( intermediate[ r ][ c ] + offset ) fits in an unsigned variable with (BitDepth + 3) bits. {:.alert .alert-info }
-
The output samples are written as follows:
for ( r = 0; r < h; r++ ) { for ( c = 0; c < w; c++ ) { s = 0 for ( t = 0; t < 7; t++ ) s += vfilter[ t ] * intermediate[ r + t ][ c ] v = Round2( s, FILTER_BITS + EXTRAPREC_BITS ) LrFrame[ plane ][ y + r ][ x + c ] = Clip1( v ) } }
The input to this process is an array coeff containing 3 coefficients.
The output from this process is an an array containing 7 coefficients.
The Wiener filter is always symmetrical and has a unit DC gain, so there are only three coefficients that need to be explicitly coded.
This process computes the full set of coefficients as follows:
filter[ 3 ] = 128
for ( i = 0; i < 3; i++ ) {
c = coeff[ i ]
filter[ i ] = c
filter[ 6 - i ] = c
filter[ 3 ] -= 2 * c
}The output of the process is the array filter.
Note: When chroma is being filtered, coeff[ 0 ] will always be equal to 0, therefore filter[ 0 ] and filter[ 6 ] will always be equal to 0. In other words, luma uses a 7-tap filter, while chroma uses a 5-tap filter. {:.alert .alert-info }
The inputs to this process are:
-
a variable plane specifying whether we are filtering Y, U, or V samples,
-
variables x and y specifying the location in the current plane in units of samples.
This process makes sure samples are taken from within the allowed extent for loop restoration filtering.
Samples within the current stripe are taken after Cdef filtering has been applied, samples outside the current stripe are taken before Cdef filtering.
The sample to return is specified as follows:
x = Min(TileEndX, x)
x = Max(TileStartX, x)
y = Min(TileEndY, y)
y = Max(TileStartY, y)
if (y < StripeStartY) {
y = Max(StripeStartY - 2,y)
return UpscaledCurrFrame[ plane ][ y ][ x ]
} else if (y > StripeEndY) {
y = Min(StripeEndY + 2,y)
return UpscaledCurrFrame[ plane ][ y ][ x ]
} else {
return UpscaledCdefFrame[ plane ][ y ][ x ]
}Note: This process can be called for samples on the three lines above and three lines below the current stripe. However, the coordinates are cropped such that only two lines above and below the stripe need to be fetched. In other words, requests for the third line (above or below) are given a copy of the second line. {:.alert .alert-info }
This process is invoked to optionally display a frame.
If show_existing_frame is equal to 1, then the decoder should output a previously decoded frame as follows:
-
The variable w is set equal to RefFrameWidth[ frame_to_show_map_idx ].
-
The variable h is set equal to RefFrameHeight[ frame_to_show_map_idx ].
-
The variable subX is set equal to RefSubsamplingX[ frame_to_show_map_idx ].
-
The variable subY is set equal to RefSubsamplingY[ frame_to_show_map_idx ].
-
The Y plane is w samples across by h samples down and the sample at location x samples across and y samples down is given by FrameStore[ frame_to_show_map_idx ][ 0 ][ y ][ x ] with x = 0..w - 1 and y = 0..h - 1.
-
The U plane is (w + subX) >> subX samples across by (h + subY) >> subY samples down and the sample at location x samples across and y samples down is given by FrameStore[ frame_to_show_map_idx ][ 1 ][ y ][ x ] with x = 0..((w + subX) >> subX) - 1 and y = 0..((h + subY) >> subY) - 1.
-
The V plane is (w + subX) >> subX samples across by (h + subY) >> subY samples down and the sample at location x samples across and y samples down is given by FrameStore[ frame_to_show_map_idx ][ 2 ][ x ][ y ] with x = 0..((w + subX) >> subX) - 1 and y = 0..((h + subY) >> subY) - 1.
-
The bit depth for each sample is RefBitDepth[ frame_to_show_map_idx ].
Otherwise, if show_frame is equal to 1, then the decoder should output the current frame as follows:
-
The variable w is set equal to FrameWidth.
-
The variable h is set equal to FrameHeight.
-
The variable subX is set equal to subsampling_x.
-
The variable subY is set equal to subsampling_y.
-
The Y plane is w samples across by h samples down and the sample at location x samples across and y samples down is given by LrFrame[ 0 ][ y ][ x ] with x = 0..w - 1 and y = 0..h - 1.
-
The U plane is (w + subX) >> subX samples across by (h + subY) >> subY samples down and the sample at location x samples across and y samples down is given by LrFrame[ 1 ][ y ][ x ] with x = 0..((w + subX) >> subX) - 1 and y = 0..((h + subY) >> subY) - 1.
-
The V plane is (w + subX) >> subX samples across by (h + subY) >> subY samples down and the sample at location x samples across and y samples down is given by LrFrame[ 2 ][ x ][ y ] with x = 0..((w + subX) >> subX) - 1 and y = 0..((h + subY) >> subY) - 1.
-
The bit depth for each sample is BitDepth.
Note: If NumPlanes is equal to 1, then the U and V planes should be ignored. {:.alert .alert-info }
This process is invoked as the final step in decoding a frame.
The inputs to this process are the decoded samples for the current frame LrFrame[ plane ][ x ][ y ].
The output from this process is an updated set of reference frames and previous motion vectors.
For each value of i from 0 to NUM_REF_FRAMES - 1, the following applies if bit i of refresh_frame_flags is equal to 1 (i.e. if (refresh_frame_flags >> i) & 1 is equal to 1):
-
RefFrameWidth[ i ] is set equal to FrameWidth.
-
RefFrameHeight[ i ] is set equal to FrameHeight.
-
RefMiCols[ i ] is set equal to MiCols.
-
RefMiRows[ i ] is set equal to MiRows.
-
RefFrameType[ i ] is set equal to frame_type.
-
RefSubsamplingX[ i ] is set equal to subsampling_x.
-
RefSubsamplingY[ i ] is set equal to subsampling_y.
-
RefBitDepth[ i ] is set equal to BitDepth.
-
SavedOrderHints[ i ][ j ] is set equal to RefOrderHint[ j ] for j = 0..NUM_REF_FRAMES-1.
-
FrameStore[ i ][ 0 ][ y ][ x ] is set equal to LrFrame[ 0 ][ y ][ x ] for x = 0..FrameWidth-1, for y = 0..FrameHeight-1.
-
FrameStore[ i ][ plane ][ y ][ x ] is set equal to LrFrame[ plane ][ y ][ x ] for plane = 1..2, for x = 0..((FrameWidth + subsampling_x) >> subsampling_x) - 1, for y = 0..((FrameHeight + subsampling_y) >> subsampling_y) - 1.
-
The following applies for row = 0..MiRows-1, for col = 0..MiCols-1:
SavedRefFrames[ i ][ row ][ col ][ 0 ] = NONE
SavedRefFrames[ i ][ row ][ col ][ 1 ] = NONE
for ( list = 0; list < 2; list++ )
r = RefFrames[ row ][ col ][ list ]
if ( r > INTRA_FRAME && RefOrderHint[ r ] != OrderHint ) {
refIdx = ( RefOrderHint[ r ] > OrderHint ) ? 1 : 0
mvRow = Mvs[ row ][ col ][ list ][ 0 ]
mvCol = Mvs[ row ][ col ][ list ][ 1 ]
if ( Abs( mvRow ) <= REFMVS_LIMIT && Abs( mvCol ) <= REFMVS_LIMIT ) {
SavedRefFrames[ i ][ row ][ col ][ refIdx ] = r
SavedMvs[ i ][ row ][ col ][ list ][ 0 ] = mvRow
SavedMvs[ i ][ row ][ col ][ list ][ 1 ] = mvCol
}
}
}-
SavedGmParams[ i ][ ref ][ j ] is set equal to gm_params[ ref ][ j ] for ref = LAST_FRAME..ALTREF_FRAME, for j = 0..5.
-
SavedSegmentIds[ i ][ row ][ col ] is set equal to SegmentIds[ row ][ col ] for row = 0..MiRows-1, for col = 0..MiCols-1.
-
The function save_cdfs( i ) is invoked (see below).
For each value of i from 0 to NUM_REF_FRAMES - 1, the following applies if bit i of refresh_frame_flags is equal to 1 (i.e. if (refresh_frame_flags >> i) & 1 is equal to 1):
- RefOrderHint[ i ] is set equal to OrderHint.
save_cdfs( ctx ) is a function call that indicates that all the CDF arrays are saved into frame context number ctx in the range 0 to (NUM_REF_FRAMES - 1). When this function is invoked the following takes place:
- A copy of each CDF array mentioned in the semantics for setup_past_independence is saved in an area of memory indexed by ctx.
Note: Although this process stores all the motion vectors, only the values where row and col are both odd will be used again. {:.alert .alert-info }