Data table cell arrow data support

cn00 · September 4, 2024, 6:21am

I use @uwdata/vgplot duckdb-wasm to query data from http,
if vgplot query returns array will accept by data table cell, but if query returns arrow, data table cell report this error:

mootari · September 4, 2024, 10:33am

Are you able to share your notebook (or a simplified example)?

cn00 · September 9, 2024, 3:09am

this is an example: Untitled / sn | Observable

mootari · September 9, 2024, 10:01am

Can you share your repro steps? I only manage to trigger the error every once in a while if I randomly select and deselect columns.

cn00 · September 10, 2024, 7:30am

sorry, I had set the default data source to vg_data_array_is_ok, did you change the data cell source to vg_data_arrow, I meet this error all the time

mootari · September 10, 2024, 8:52am

Looks like the file parquet file isn’t served correctly. I can see CORS errors in the network tab.

cn00 · September 10, 2024, 10:18am

sorry, the CORS was revoked by IaC, please try again, or you can pass a file url parameter to override the default parquet file witch fit your network like this https://observablehq.com/@cnsn/data-preview2?url=https//www.to/file.parquet

mootari · September 10, 2024, 11:00am

Now I’m also getting the error consistently. I suspect that some race conditions and/or side effects might be at play. Perhaps you can try to model the dependencies more explicitely?

It looks like vg_data_arrow does not register as table:

(await DuckDBClient.of({vg_data_arrow, penguins})).sql`show tables`

mootari · September 10, 2024, 7:19pm

After changing the type to json and playing around some more with other Arrow examples I can see now that this is likely a bug on our end.

I’ve filed an internal issue, but I’m afraid I don’t know how soon we’ll be able to fix it.

mootari · September 10, 2024, 7:40pm

I suspect this is caused by

github.com/duckdb/duckdb-wasm

Connection.insertArrowTable fails for esbuild typescript

opened 08:11PM - 18 Apr 24 UTC

rob-blackbourn

### What happens? The method `Connection.insertArrowTable` silently fails in a …typescript program. ### To Reproduce ## bare-node works Change the query in `examples/bare-node/index.cjs` from: ```js const conn = await db.connect(); await conn.query(`SELECT count(*)::INTEGER as v FROM generate_series(0, 100) t(v)`); await conn.close(); ``` to: ```js const conn = await db.connect(); const json = [ { a: 1, b: 11 }, { a: 2, b: 22 }, ]; const table = arrow.tableFromJSON(json); await conn.insertArrowTable(table, { name: 'local_table', create: true, }); try { const result = await conn.query(`SELECT * FROM local_table`); console.log(result.toString()); } catch (error) { console.log(error); } await conn.close(); ``` The table should be created and the query succeeds. ## esbuild-node fails Change the query in `examples/esbuild-node/index.ts` from: ```ts const conn = await db.connect(); await conn.query<{ v: arrow.Int }>(`SELECT count(*)::INTEGER as v FROM generate_series(0, 100) t(v)`); await conn.close(); ``` to: ```ts const conn = await db.connect(); const json = [ { a: 1, b: 11 }, { a: 2, b: 22 }, ]; const table = arrow.tableFromJSON(json); await conn.insertArrowTable(table, { name: 'local_table', create: true, }); try { const result = await conn.query<{ a: arrow.Int; b: arrow.Int }>(`SELECT * FROM local_table`); console.log(result.toString()); } catch (error) { console.log(error); } await conn.close(); ``` This should *fail*. ## Investigation When I step through the code I notice the failure is in `apache-arrow/ipc/writer.js` in the function `writeAll` at the statement `if (input instanceof table_js_1.Table) {`. In the `bare-node` example this is `true` while for `esbuild-node` this is `false`. Furthermore for `bare-node` `input.__proto__ === table_js_1.Table.prototype` is `true`, but for `esbuild-node` it is `false`. It appears the internal version of the `apache-arrow` `Table` class is different from that of the locally imported one. I note the `bundle.mjs` in `duckdb-wasm` patches the `appache-arrow/package.json` due to the export strategy of the `appache-arrow` package. Could this be involved? ### Browser/Environment: Chrome 123.0.6312.123 ### Device: MacBook Pro (ARM) ### DuckDB-Wasm Version: Latest clone (also 1.28.1-dev179.0) ### DuckDB-Wasm Deployment: The duckdb-wasm repo ### Full Name: Rob Blackbourn ### Affiliation: None

cn00 · September 11, 2024, 2:21am

I find that Input.table can accept vg_data_arrow correctly:

mootari · September 11, 2024, 10:02am

Data table cells use DuckDB under the hood, so they inherit some of the bugs. Inputs.table() simply requires an Iterable of objects and does not look at the schema at all.

As a workaround you can use

vg_data_array = vg.coordinator().query(sql, {type: "json", /* ... */})

(By the way, you don’t need await at the top level - Observable’s Runtime automatically awaits Promises returned by cells.)

As mentioned, the underlying problem is that three different module instances (and versions) of Apache Arrow are being loaded. Mosaic uses a different arrow module instance to produce the Arrow Table than Observable’s Stdlib, which then fails DuckDB’s instance checks. Observable’s DuckDBClient.of correctly detects Arrow Tables regardless of version because it uses duck typing (no pun intended), but DuckDB itself will then fail to detect the Table and skips it.

To give an example, try the following:

// Import the same arrow module URL as used by Observable's DuckDB - modules are singletons
arrow11 = import("https://cdn.observableusercontent.com/npm/apache-arrow@11.0.0/+esm")

// Convert to an arrow 11 Table
vg_data_arrow = arrow11.tableFromJSON(
  (await vg.coordinator().query(sql, {type: 'arrow', cache: false})).toArray()
)

Your data tables will now let you select “vg_data_arrow” as table and correctly show the data.

I also tried to mask the instance by proxying getPrototypeOf, but unfortunately it seems that the format has changed too much to be backwards compatible to 11.

mootari · September 11, 2024, 12:41pm

I was misremembering. It’s actually Arrow that performs the instanceof checks:

github.com

apache/arrow/blob/27acf8bf6e16d4c53e1ed59d5ef46ac7db0306ea/js/src/ipc/writer.ts#L172


      
              return this;
          }
          
          public write(payload?: Table<T> | RecordBatch<T> | Iterable<RecordBatch<T>> | null) {
              let schema: Schema<T> | null = null;
          
              if (!this._sink) {
                  throw new Error(`RecordBatchWriter is closed`);
              } else if (payload == null) {
                  return this.finish() && undefined;
              } else if (payload instanceof Table && !(schema = payload.schema)) {
                  return this.finish() && undefined;
              } else if (payload instanceof RecordBatch && !(schema = payload.schema)) {
                  return this.finish() && undefined;
              }
          
              if (schema && !compareSchemas(schema, this._schema)) {
                  if (this._started && this._autoDestroy) {
                      return this.close();
                  }
                  this.reset(this._sink, schema);

cn00 · September 12, 2024, 5:12am

Thank a lot for your time. The performance is not the first level for me at this time . I can use the array instead of arrow. I’m using vg.coordinator().query() just for get data from http on AWS s3. I had find the solution you had answer in the other channel, thanks again!

Topic		Replies	Views
Do Parquet files in DuckDB connections use HTTP ranged requests? Feedback	2	458	February 9, 2023
DuckDB is not returning rows for Parquet files Community Help	15	217	July 31, 2024
DuckDB: load multiple remote parquet files dinamically Community Help	7	758	June 5, 2024
Populating a DuckDb database cell with remote binary file (parquet) Community Help	2	258	January 10, 2023
duckdb error within Observable Framework Community Help	5	186	May 22, 2024

Data table cell arrow data support

Related Topics