Advanced Encoding Guide

Dirty lines from A Silent Voice (2016)'s intro. On mouseover: fixed with ContinuityFixer and FillBorders.

One of the more common issues you may encounter are 'dirty lines', these are usually found on the borders of video where a row or column of pixels exhibits inconsistent luma values comparative to its surroundings. Oftentimes, this is the due to improper downscaling, for example downscaling after applying borders. Dirty lines can also occur because the compressionist doesn't consider that whilst they're working with 4:2:2 chroma subsampling (meaning their height doesn't have to be mod2), consumer video will be 4:2:0, leading to extra black rows that you can't get rid of during cropping if the main clip isn't placed properly. Another form of dirty lines is exhibited when the chroma planes are present on black bars. Usually, these should be cropped out. The opposite can also occur, however, where the planes with legitimate luma information lack chroma information.

It's important to remember that sometimes your source will have fake lines (often referred to as 'dead' lines), meaning ones without legitimate information. These will usually just mirror the next row/column. Do not bother fixing these, just crop them instead. An example:

Similarly, when attempting to fix dirty lines you should thoroughly check that your fix has not caused unwanted problems, such as smearing (common with overzealous ContinuityFixer values) or flickering (especially on credits, it is advisable to omit credit reels from your fix in most cases). If you cannot figure out a proper fix it is completely reasonable to either crop off the dirty line(s) or leave them unfixed. A bad fix is worse than no fix!

Here are five commonly used methods for fixing dirty lines:

rektlvls

From rekt. This is basically FixBrightnessProtect3 and FixBrightness from AviSynth in one, although unlike FixBrightness, not the entire frame is processed. Its values are quite straightforward. Raise the adjustment values to brighten, lower to darken. Set prot_val to None and it will function like FixBrightness, meaning the adjustment values will need to be changed.

from rekt import rektlvls
fix = rektlvls(src, rownum=None, rowval=None, colnum=None, colval=None, prot_val=[16, 235])

If you'd like to process multiple rows at a time, you can enter a list (e.g. rownum=[0, 1, 2]).

To illustrate this, let's look at the dirty lines in the black and white Blu-ray of Parasite (2019)'s bottom rows:

In this example, the bottom four rows have alternating brightness offsets from the next two rows. So, we can use rektlvls to raise luma in the first and third row from the bottom, and again to lower it in the second and fourth:

fix = rektlvls(src, rownum=[803, 802, 801, 800], rowval=[27, -10, 3, -3])

In this case, we are in FixBrightnessProtect3 mode. We aren't taking advantage of prot_val here, but people usually use this mode regardless, as there's always a chance it might help. The result:

bbmod

From awsmfunc. This is a mod of the original BalanceBorders function. While it doesn't preserve original data nearly as well as rektlvls, it will lead to decent results with high blur and thresh values and is easy to use for multiple rows, especially ones with varying brightness, where rektlvls is no longer useful. If it doesn't produce decent results, these can be changed, but the function will get more destructive the lower you set them. It's also significantly faster than the versions in havsfunc and sgvsfunc, as only necessary pixels are processed.

import awsmfunc as awf
bb = awf.bbmod(src=clip, left=0, right=0, top=0, bottom=0, thresh=[128, 128, 128], blur=[20, 20, 20], planes=[0, 1, 2], scale_thresh=False, cpass2=False)

The arrays for thresh and blur are again y, u, and v values. It's recommended to try blur=999 first, then lowering that and thresh until you get decent values.
thresh specifies how far the result can vary from the input. This means that the lower this is, the better. blur is the strength of the filter, with lower values being stronger, and larger values being less aggressive. If you set blur=1, you're basically copying rows. If you're having trouble with chroma, you can try activating cpass2, but note that this requires a very low thresh to be set, as this changes the chroma processing significantly, making it quite aggressive.

For our example, I've created fake dirty lines, which we will fix:

To fix this, we can apply bbmod with a low blur and a high thresh, meaning pixel values can change significantly:

fix = awf.bbmod(src, top=6, thresh=90, blur=20)

Our output is already a lot closer to what we assume the source should look like. Unlike rektlvls, this function is quite quick to use, so lazy people (i.e. everyone) can use this to fix dirty lines before resizing, as the difference won't be noticeable after resizing.

While you can use rektlvls on as many rows/columns as necessary, the same doesn't hold true for bbmod. Unless you are resizing after, you should only use bbmod on two rows/pixels for low blur values (\(\approx 20\)) or three for higher blur values. If you are resizing after, you can change the maximum value according to: \[ max_\mathrm{resize} = max \times \frac{resolution_\mathrm{source}}{resolution_\mathrm{resized}} \]

FillBorders

From fb. This function pretty much just copies the next column/row in line. While this sounds, silly, it can be quite useful when downscaling leads to more rows being at the bottom than at the top, and one having to fill one up due to YUV420's mod2 height.

fill = core.fb.FillBorders(src=clip, left=0, right=0, bottom=0, top=0, mode="fixborders")

A very interesting use for this function is one similar to applying ContinuityFixer only to chroma planes, which can be used on gray borders or borders that don't match their surroundings no matter what luma fix is applied. This can be done with the following script:

fill = core.fb.FillBorders(src=clip, left=0, right=0, bottom=0, top=0, mode="fixborders")
merge = core.std.Merge(clipa=clip, clipb=fill, weight=[0,1])

You can also split the planes and process the chroma planes individually, although this is only slightly faster. A wrapper that allows you to specify per-plane values for fb is FillBorders in awsmfunc.

Note that you should only ever fill single columns/rows with FillBorders. If you have more black lines, crop them! If there are frames requiring different crops in the video, don't fill these up. More on this at the end of this chapter.

To illustrate what a source requiring FillBorders might look like, let's look at Parasite (2019)'s SDR UHD once again, which requires an uneven crop of 277. However, we can't crop this due to chroma subsampling, so we need to fill one row. To illustrate this, we'll only be looking at the top rows. Cropping with respect to chroma subsampling nets us:

crp = src.std.Crop(top=276)

Obviously, we want to get rid of the black line at the top, so let's use FillBorders on it:

fil = crp.fb.FillBorders(top=1, mode="fillmargins")

This already looks better, but the orange tones look washed out. This is because FillBorders only fills one chroma if two luma are fixed. So, we need to fill chroma as well. To make this easier to write, let's use the awsmfunc wrapper:

fil = awf.fb(crp, top=1)

Our source is now fixed. Some people may want to resize the chroma to maintain original aspect ratio performing lossy resampling on chroma, but whether this is the way to go is not generally agreed upon. If you want to go this route:

top = 1
bot = 1
new_height = crp.height - (top + bot)
fil = awf.fb(crp, top=top, bottom=bot)
out = fil.resize.Spline36(crp.width, new_height, src_height=new_height, src_top=top) 

ContinuityFixer

From cf. ContinuityFixer works by comparing the rows/columns specified to the amount of rows/columns specified by range around it and finding new values via least squares regression. Results are similar to bbmod, but it creates entirely fake data, so it's preferable to use rektlvls or bbmod with a high blur instead. Its settings look as follows:

fix = core.cf.ContinuityFixer(src=clip, left=[0, 0, 0], right=[0, 0, 0], top=[0, 0, 0], bottom=[0, 0, 0], radius=1920)

This is assuming you're working with 1080p footage, as radius's value is set to the longest set possible as defined by the source's resolution. I'd recommend a lower value, although not going much lower than 3, as at that point, you may as well be copying pixels (see FillBorders below for that). What will probably throw off most newcomers is the array I've entered as the values for rows/columns to be fixed. These denote the values to be applied to the three planes. Usually, dirty lines will only occur on the luma plane, so you can often leave the other two at a value of 0. Do note an array is not necessary, so you can also just enter the amount of rows/columns you'd like the fix to be applied to, and all planes will be processed.

As ContinuityFixer is less likely to keep original data in tact, it's recommended to prioritize bbmod over it.

Let's look at the bbmod example again and apply ContinuityFixer:

fix = src.cf.ContinuityFixer(top=[6, 6, 6], radius=10)

Let's compare this with the bbmod fix (remember to mouse-over to compare):

Just like bbmod, ContinuityFixer shouldn't be used on more than two rows/columns. Again, if you're resizing, you can change this maximum accordingly: \[ max_\mathrm{resize} = max \times \frac{resolution_\mathrm{source}}{resolution_\mathrm{resized}} \]

ReferenceFixer

From edgefixer. This requires the original version of edgefixer (cf is just an old port of it, but it's nicer to use and processing hasn't changed). I've never found a good use for it, but in theory, it's quite neat. It compares with a reference clip to adjust its edge fix as in ContinuityFixer.:

fix = core.edgefixer.Reference(src, ref, left=0, right=0, top=0, bottom=0, radius = 1920)

Notes

Too many rows/columns

One thing that shouldn't be ignored is that applying these fixes (other than rektlvls) to too many rows/columns may lead to these looking blurry on the end result. Because of this, it's recommended to use rektlvls whenever possible or carefully apply light fixes to only the necessary rows. If this fails, it's better to try bbmod before using ContinuityFixer.

Resizing

It's important to note that you should always fix dirty lines before resizing, as not doing so will introduce even more dirty lines. However, it is important to note that, if you have a single black line at an edge that you would use FillBorders on, you should remove that using your resizer.

For example, to resize a clip with a single filled line at the top to \(1280\times536\) from \(1920\times1080\):

top_crop = 138
bot_crop = 138
top_fill = 1
bot_fill = 0
src_height = src.height - (top_crop + bot_crop) - (top_fill + bot_fill)
crop = core.std.Crop(src, top=top_crop, bottom=bot_crop)
fix = core.fb.FillBorders(crop, top=top_fill, bottom=bot_fill, mode="fillmargins")
resize = core.resize.Spline36(fix, 1280, 536, src_top=top_fill, src_height=src_height)

An easier way of doing the above is using the relevant parameters in awsmfunc's zresize.

So the last line in the above example would become:

resize = awf.zresize(fix, preset=720, top=1)

Similarly, if you filled 1 line each on the left and right side, you would use:

resize = awf.zresize(fix, preset=720, left=1, right=1)

A significant benefit of using zresize is that it automatically calculates the most appropriate target resolution, to minimize the AR error.

Diagonal borders

If you're dealing with diagonal borders, the proper approach here is to mask the border area and merge the source with a FillBorders call. An example of this (from the Your Name (2016)):

Fix compared with unmasked in fillmargins mode and contrast adjusted for clarity:

Code used (note that this was detinted after):

mask = core.std.ShufflePlanes(src, 0, vs.GRAY).std.Binarize(43500)
cf = core.fb.FillBorders(src, top=6, mode="mirror").std.MaskedMerge(src, mask)

Finding dirty lines

Dirty lines can be quite difficult to spot. If you don't immediately spot any upon examining borders on random frames, chances are you'll be fine. If you know there are frames with small black borders on each side, you can use something like the following script:

def black_detect(clip, thresh=None):
    if thresh:
        clip = core.std.ShufflePlanes(clip, 0, vs.GRAY).std.Binarize(
            "{0}".format(thresh)).std.Invert().std.Maximum().std.Inflate( ).std.Maximum().std.Inflate()
    l = core.std.Crop(clip, right=clip.width / 2)
    r = core.std.Crop(clip, left=clip.width / 2)
    clip = core.std.StackHorizontal([r, l])
    t = core.std.Crop(clip, top=clip.height / 2)
    b = core.std.Crop(clip, bottom=clip.height / 2)
    return core.std.StackVertical([t, b])

This script will make values under the threshold value (i.e. the black borders) show up as vertical or horizontal white lines in the middle on a mostly black background. If no threshold is given, it will simply center the edges of the clip. You can just skim through your video with this active. An automated alternative would be dirtdtct, which scans the video for you.

Other kinds of variable dirty lines are a bitch to fix and require checking scenes manually.

Variable borders

An issue very similar to dirty lines is unwanted borders. During scenes with different crops (e.g. IMAX or 4:3), the black borders may sometimes not be entirely black, or be completely messed up. In order to fix this, simply crop them and add them back. You may also want to fix dirty lines that may have occurred along the way:

crop = core.std.Crop(src, left=100, right=100)
clean = core.cf.ContinuityFixer(crop, left=2, right=2, top=0, bottom=0, radius=25)
out = core.std.AddBorders(clean, left=100, right=100)

If you're resizing, you should crop these off before resizing, then add the borders back, as leaving the black bars in during the resize will create dirty lines:

crop = src.std.Crop(left=100, right=100)
clean = crop.cf.ContinuityFixer(left=2, right=2, top=2, radius=25)
resize = awf.zresize(clean, preset=720)
border_size = (1280 - resize.width) / 2
bsize_mod2 = border_size % 2
out = resize.std.AddBorders(left=border_size - bsize_mod2, right=border_size + bsize_mod2)

In the above example, we have to add more to one side than the other to reach our desired width. Ideally, your border_size will be mod2 and you won't have to do this.

If you know you have borders like these, you can use brdrdtct from awsmfunc similarly to dirtdtct to scan the file for them.