Reading parquet file multiple times. Is there a better way?

Community Help
1

I’m using multiple inputs in a page and I’m getting the data from the .parquet file (i.e. weekly_overview), with blocks, crops and varieties. This is how I’m doing it now.

const indicators = await sql([ SELECT DISTINCT indicator FROM "weekly_overview_23" ORDER BY indicator;]);

```js
const variedades = await sql([`
  SELECT DISTINCT block, crop, variety
  FROM weekly_overview_23
  ORDER BY block, crop, variety`]);

const bloques = await sql([`
  SELECT DISTINCT block
  FROM weekly_overview_23
  ORDER BY block`]);

const cultivos = await sql([`
  SELECT DISTINCT crop
  FROM weekly_overview_23
  ORDER BY crop`]);

The terminal is showing that I’m reading the parquet file multiple times. Wondering if there is a better way.
Thanks,

2

Can you check if the requests to the parquet file have Range headers set? The sql feature of Framework uses DuckDB, and DuckDB will only request the parts of the file that are needed for specific queries.

For example, in your query that reads SELECT DISTINCT crop, it is very likely that DuckDB is only reading the crop table from the parquet file, and not the entire file. You should also see that the response size of each request is smaller than the entire file, and in theory the sum of all the requests should be at most the size of the original parquet file.