Demonstrates the use of patch sizes with UpscalerJS.Open in Stackblitz.
Background & Motivation behind Patch Sizes
Inference requests with Tensorflow.js models are synchronous and blocking, which makes achieving UI responsiveness difficult, particularly with larger images or on older hardware.
A solution is to slice the upcoming image into pieces and upscale each one individually.
However, upscaling models have a tendency to perform poorly on edges, resulting in noticeable artifacting:
A solution is to add a bit of padding to each patch size, and then slice off the resulting padding when stitching our image back together. Here's what that looks like (note, we are explicitly not removing the padding in this demonstration):
UpscalerJS provides an easy mechanism for working with patch sizes, no math required.
Specify our patch size and padding in the request to upscale:
import Upscaler from 'upscaler'
import image from '/path/to/image.png'
const upscaler = new Upscaler()
2 or greater is generally sufficient to avoid noticeable artifacting.
If we do not explicitly provide a
padding argument, UpscalerJS will emit a warning. Generally, a
patchSize argument should always be accompanied by a
padding to avoid the artifacting demonstrated above. To avoid this warning, pass an explicit argument of
0 as the padding:
You may notice, if you inspect the patch sizes received from the progress example, that all image patches are the same size, even if the numbers don't match up precisely; for example, if you specify a patch size of 10 for a 12x12 image, you'll receive 4 10x10 patches.
This is done purposefully to improve performance. Every new tensor size requires another "warm up" of the model on the GPU. Because of this, it's usually faster to process images that are consistent sizes than it is to process a novel image size (even if it's smaller).