Unable to import gdal3.js (a gdal port)

Community Help
1

I’ve tried to load the library gdal3.js, a port of gdal, without success (with import, with require, with help from the module require debugger notebook):

Maybe because it rely on WebAssembly and it need an initialization step.
Is there a way to make it works in Observable ?

Thanks!

2

Hi @TomBor, that’s a tricky one!

I can’t claim that I got it working, but at least I got through to initialization (where it then throws a console error due to a missing callback “next”). Maybe this will help you to move forward:

{
  const path = 'https://cdn.jsdelivr.net/npm/gdal3.js@2.1.0/dist/package';
  const aliased = require.alias({fs: {}, path: {}, string_decoder: {}});
  const init = await aliased(`${path}/gdal3.js`);

  const paths = {
    js: await makeLocal(`${path}/gdal3.js`),
    wasm: `${path}/gdal3WebAssembly.wasm`,
    data: `${path}/gdal3WebAssembly.data`,
  };

  return init({paths});

  async function makeLocal(url, type = 'text/javascript') {
    const src = await (await fetch(url)).text();
    return URL.createObjectURL(new Blob([src], {type}));
  }
}
2 Likes
3

Thank you @mootari !
That’s definitely some progress.
I’ve tried to applied your code in this notebook and succeed to list available gdal drivers as in gdal3 documentation but I still can’t open a raster or vector file.

4

Well, that was annoying, but I figured it out eventually:

  1. When gdal3.js is configured to use its worker (which is the default, unless you pass useWorker: false), it will return a Proxy. This proxy passes every property access to the worker indiscriminately.
  2. Observable’s Runtime uses ducktyping to figure out what type of value a cell returned. It tries to detect promises by checking for .then(), and generators by checking for .next() and .return(). You can probably guess by now where this is going.
  3. All of these checks get passed to the gdal worker, where they produce the previously mentioned errors.
  4. To get things going we need to prevent access to these props. Sadly Proxy instances are immutable, so we cannot simply redefine the props. Instead we have to wrap the Proxy in yet another Proxy. Fun!

Another downside of the worker/proxy combo is that you can no longer see the actual API. I’ve added some URL flags to disable the worker, load the dev build and enable/inject debugger statements. I recommend that you try to get things working with the worker disabled first.

Now, for the files:

  1. If you pass in a string (like you did with the FileAttachment URL), gdal will treat it as internal path reference, meaning the file has to be present in its virtual filesystem (at least that’s what I took away from looking at the source).
  2. Instead you need to pass in an object with a specific interface, which FileAttachment already matches: gdal expects .name and .arrayBuffer(), which you might recognize as properties of File instances.
  3. gdal will add the file to its internal filesystem, where you can then reference it simply via its name.

Be aware that there doesn’t seem to be much of a point in calling .open() multiple times, as gdal.open() will happily accept an array of files.

3 Likes
5

Fantastic!
Thanks for insisting on finding a solution and for all the explanations. All this tricks to make it work are still mysterious to me, but I hope It will inspire other to try this port of gdal. I have proposed you as co-author, as you have done all the hard part!
Hello gdal3.js

1 Like
6

@TomBor I’ve made a few more tweaks. Can you please republish?

7

Done. Thanks again!