[@thi.ng/webgl] Best practice to save screenshot

[nkint]

Hi! I'm looking for some best practice to save screenshot of any desired size.

Right now I'm relying on a global variable that during the update push a new canvas dimension/aspect ratio/viewport, save the screenshot, pop back the dimensions.

Honestly I don't like that approach, or at least I'm looking for something more modular with less boilerplate to easily use it in other project.

Any help/ideas are super usefull! 💡

// NOTE: resize and retina display are not handled

import { start } from '@thi.ng/hdom';
import { canvasWebGL } from '@thi.ng/hdom-components';
import {
  draw,
  GLMat4,
  ModelSpec,
  compileModel,
  LAMBERT,
  shader,
  cube,
  GLVec3,
} from '@thi.ng/webgl';
import { perspective, lookAt } from '@thi.ng/matrices';
import { normalize } from '@thi.ng/vectors';

export const globalOpts = {
  doSave: false,
  width: 200,
  height: 200,
  widthExport: 1500,
  heightExport: 1500,

  get w() {
    return this.doSave ? this.widthExport : this.width;
  },

  get h() {
    return this.doSave ? this.heightExport : this.height;
  },
};

function getMyCoolModel(gl: WebGLRenderingContext, proj: GLMat4, view: GLMat4) {
  return compileModel(gl, {
    ...cube(),
    shader: shader(gl, LAMBERT()),
    uniforms: {
      proj,
      view,
      lightDir: <GLVec3>normalize(null, [0.5, 0.75, 1]),
    },
  });
}

const app = () => {
  let model: ModelSpec;

  const view = <GLMat4>lookAt([], [10, 10, 10], [0, 0, 0], [0, 1, 0]);

  const canvas = canvasWebGL({
    init(_, gl) {
      let proj = <GLMat4>perspective([], 45, globalOpts.w / globalOpts.h, 0.1, 100);

      // init other fbo/texture/post-process relying on `globalOpts.w` and `globalOpts.h`

      model = getMyCoolModel(gl, proj, view);
    },
    update(canvas, gl, __, time) {
      if (globalOpts.doSave) {
        // set export settings
        gl.canvas.width = globalOpts.w;
        gl.canvas.height = globalOpts.h;
        gl.viewport(0, 0, globalOpts.w, globalOpts.h);
        this.init(canvas, gl);
      }

      gl.clearColor(0.8, 0, 0, 1);
      gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

      draw(model);

      if (globalOpts.doSave) {
        saveScreenshot(canvas, 'image.png');

        // get back to screen settings
        globalOpts.doSave = false;
        gl.canvas.width = globalOpts.w;
        gl.canvas.height = globalOpts.h;
        gl.viewport(0, 0, globalOpts.w, globalOpts.h);
        this.init(canvas, gl);
      }
    },
  });
  return [canvas, { width: globalOpts.width, height: globalOpts.height }];
};

addEventListener('keypress', evt => {
  if (evt.key === 's') {
    globalOpts.doSave = true;
  }
});

function saveScreenshot(canvas: Readonly<HTMLCanvasElement>, fileName: Readonly<string>) {
  function dataURItoBlob(dataURI: string) {
    // https://github.com/graingert/datauritoblob/blob/master/dataURItoBlob.js
    var byteString = atob(dataURI.split(',')[1]);
    var mimeString = dataURI
      .split(',')[0]
      .split(':')[1]
      .split(';')[0];
    // write the bytes of the string to an ArrayBuffer
    var ab = new ArrayBuffer(byteString.length);
    var dw = new DataView(ab);
    for (var i = 0; i < byteString.length; i++) {
      dw.setUint8(i, byteString.charCodeAt(i));
    }
    // write the ArrayBuffer to a blob, and you're done
    return new Blob([ab], { type: mimeString });
  }

  const dataURI = canvas.toDataURL('image/png');
  const blob = dataURItoBlob(dataURI);

  const link = document.createElement('a');
  link.download = fileName;
  link.href = window.URL.createObjectURL(blob);
  link.onclick = () =>
    setTimeout(() => {
      window.URL.revokeObjectURL(blob as any);
      link.removeAttribute('href');
    }, 0);

  link.click();
}

let cancel = start(app());

if (process.env.NODE_ENV !== 'production') {
  const hot = (<any>module).hot;
  hot && hot.dispose(cancel);
}

[postspectacular]

Hi @nkint - instead of having to resize the mounted onscreen canvas and re-initialize the whole scene, you could just use an FBO, bind it, render to it, then read its pixels (via readPixels), release the FBO, then create a new 2d canvas and put the read pixel array as image data. The @thi.ng/pixel can simplify/reduce boilerplate for some of those steps. We could also add a higher-order function to the webgl pkg to make this more re-usable...

Also, I don't understand the need for using canvas.toDataURL() and the conversion to Blob in your example, as you could just use canvas.toBlob() and be done with it. I've been using the latter together with this download function (also used in a few other examples) and already have a note to bundle this functionality as new package soon.

canvas.toBlob((blob) => download("image.png", blob), "image/png");

[nkint]

Hi @postspectacular ! Thank you for the answer.

So I made a quick trial...

It seems to work well... but:
I'm currently using a barycentric wireframe shader that needs gl.enable(gl.SAMPLE_ALPHA_TO_COVERAGE); and the desired alpha does not work inside the frame buffer and I download images with some faces that should be transparent but they are not.

I'm posting the code here as a note for future investigations:

import {
  TextureOpts,
  RBO,
  texture,
  fbo,
  RboOpts,
  FboOpts,
  rbo,
  readPixels,
  TextureFormat,
} from '@thi.ng/webgl';
import { isBoolean, isNil } from '@thi.ng/checks';
import { download } from '../download';

export const createWebglScreenShot = (
  gl: WebGLRenderingContext,
  width: number,
  height: number,
  fileName: string = 'image.png',
  optsColorTexture?: Partial<TextureOpts>,
  depthBuffer: RBO | RboOpts | boolean = true,
) => {
  const optsTexture: Partial<TextureOpts> = {
    width,
    height,
    image: null,
    filter: gl.LINEAR,
    wrap: gl.CLAMP_TO_EDGE,
    ...optsColorTexture,
  };
  const colorTexture = texture(gl, optsTexture);
  const optsFbo: Partial<FboOpts> = { tex: [colorTexture] };

  if (isNil(depthBuffer)) {
    // doing nothing
  } else if (depthBuffer instanceof RBO) {
    optsFbo.depth = depthBuffer;
  } else if (isBoolean(depthBuffer)) {
    optsFbo.depth = rbo(gl, { width, height });
  } else if (depthBuffer.width && depthBuffer.height) {
    optsFbo.depth = rbo(gl, depthBuffer);
  }

  const frameBuffer = fbo(gl, optsFbo);
  frameBuffer.unbind();

  return {
    capture(draw: () => void) {
      const canvas = document.createElement('canvas');
      canvas.width = width;
      canvas.height = height;
      const context = canvas.getContext('2d');
      const imageData = context.getImageData(0, 0, width, height);

      frameBuffer.bind();

      const prevViewport = gl.getParameter(gl.VIEWPORT);
      gl.viewport(0, 0, width, height);
      draw();

      const buffer = new Uint8Array(width * height * 4);
      readPixels(gl, 0, 0, width, height, TextureFormat.RGBA, colorTexture.type, buffer);

      imageData.data.set(buffer);
      context.putImageData(imageData, 0, 0);

      canvas.toBlob(blob => download(fileName, blob), 'image/png');

      // frameBuffer.release();
      // colorTexture.release();
      frameBuffer.unbind();

      gl.viewport(...(prevViewport as [number, number, number, number]));
    },
  };
};

export type WebglScreenShot = ReturnType<typeof createWebglScreenShot>;

[postspectacular]

Ciao @nkint - haven't tried this myself yet, but if you're using WebGL2 you could try adding this manual RBO multi-sample init/config step before initializing the actual FBO:

gl.renderbufferStorageMultisample(gl.RENDERBUFFER, gl.getParameter(gl.MAX_SAMPLES), gl.RGBA8, width, height);