Patch Sizes

View this page on the UpscalerJS website

Demonstrates the use of patch sizes with UpscalerJS.

Open in Codesandbox.

Background & Motivation behind Patch Sizes

Inference requests with Tensorflow.js models are synchronous and blocking, which makes achieving UI responsiveness difficult, particularly with larger images or on older hardware.

A solution is to slice the upcoming image into pieces and upscale each one individually.

Splitting an image into patches

However, upscaling models have a tendency to perform poorly on edges, resulting in noticeable artifacting:

Example of artifacting along the edges of patches

A solution is to add a bit of padding to each patch size, and then slice off the resulting padding when stitching our image back together. Here's what that looks like (note, we are explicitly not removing the padding in this demonstration):

Example of adding padding to the image (UpscalerJS slices off the excess padding)

UpscalerJS provides an easy mechanism for working with patch sizes, no math required.

Code

Specify our patch size and padding in the request to upscale:

import Upscaler from 'upscaler'
import image from '/path/to/image.png'
const upscaler = new Upscaler()
upscaler.upscale(image, {
  patchSize: 32,
  padding: 2,
})

A padding of 2 or greater is generally sufficient to avoid noticeable artifacting.

If we do not explicitly provide a padding argument, UpscalerJS will emit a warning. Generally, a patchSize argument should always be accompanied by a padding to avoid the artifacting demonstrated above. To avoid this warning, pass an explicit argument of 0 as the padding:

upscaler.upscale(image, {
  patchSize: 32,
  padding: 0,
})

info

You may notice, if you inspect the patch sizes received from the progress example, that all image patches are the same size, even if the numbers don't match up precisely; for example, if you specify a patch size of 10 for a 12x12 image, you'll receive 4 10x10 patches.

This is done purposefully to improve performance. Every new tensor size requires another "warm up" of the model on the GPU. Because of this, it's usually faster to process images that are consistent sizes than it is to process a novel image size (even if it's smaller).

You can read more about patch sizes and performance here.