# DuckDB is not returning rows for Parquet files

**URL:** <https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426>\
**Category:** Help\
**Created:** [June 24, 2024, 9:04pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426 "2024-06-24T21:04:33Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 24, 2024, 9:04pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/1 "2024-06-24T21:04:33Z")

</div>

Hi all, I’ve been using Observable Framework for a few weeks, and I ran into a strange behaviour with SQL.

I have a parquet file that I download from Google BigQuery created from a Python data loader. From what I can tell, the Parquet file is fine, and has all the fields, the right schema, the right data, the works.

With my code, I pop this SQL in… This works, and confirms that the whole thing is ok.

```sql
select
    resource
from
    metric_detail
where
    owner = ${owner}

```

I’d like to expand the columns. So I run the actual query I want…

```sql
select
    resource,
    title
from
    metric_detail
where
    owner = ${owner}

```

This is where the problem comes in – I get 0 rows. Upon further testing, I tried this…

```sql
select
    *
from
    metric_detail
where
    owner = ${owner}

```

The headings update, and they show all the columns, yet again, no data returned.

I did observe something interesting. The query

```auto
select distinct title
from metric_detail

```

did give me a weird error in the browser console.

```auto
Error: Invalid Error: TProtocolException: Invalid data
    at U.onMessage (_esm.js:7:11151)

```

Some additional info – changing the data format from parquet to json has worked. Json is not desirable as it is way too big.

Any suggestions on what might be the issue with the parquet file?

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [June 24, 2024, 10:12pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/2 "2024-06-24T22:12:49Z")

</div>

Does the problem go away if you reload the page? And does changing the value for `owner` dynamically (e.g. via a text input) cause similar problems?

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 24, 2024, 10:27pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/3 "2024-06-24T22:27:59Z")

</div>

I tried reloading, restarting, the works. No change. It is not a problem with the SQL code. It is something to do with the ingestion of a parquet file format. Changing to json solved the issue, so I know it is not an SQL issue.

Something with the way Observable is ingesting the Parquet file is causing it to do something weird with the file. I viewed the file with a Parquet explorer. All the fields are there, all the data is there.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [June 24, 2024, 10:47pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/4 "2024-06-24T22:47:07Z")

</div>

Can you narrow down what’s causing the issue? E.g., is it the way you select columns? Or is it the specific column types?

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 25, 2024, 1:42am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/5 "2024-06-25T01:42:23Z")

</div>

Not yet. Column types are VARCHAR. When SELECT’ing those columns, it returns 0 rows. The WHERE is not even in play at all. I will be focussing my efforts on how the parquet is being generated, forcing column types, and seeing if it has any impact.

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 26, 2024, 8:48pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/6 "2024-06-26T20:48:04Z")

</div>

Hi all, started with a new round of testing.

Querying the parquet file directly in DuckDb worked exactly as expected. All queries worked, and seems that the Parquet file is healthy, and DuckDB is fully capable of reading and parsing it.

Next up, I created a new `test.md` file, and queried both the `json` and `parquet` file in a `sql` block, simply just a `select * from table` for each of them.

- On first run, everything worked as expected. Both json and parquet data files were able to display the result of the SQL query.
- I navigated to one of the other pages in the project, and it still worked.
- Navigated back to my test page, and the Parquet query failed. It did not return any rows.

# Conclusion thus far

This is not a data or a SQL issue. This has something to do with concurrency, or caching of the parquet data in memory. I don’t believe it is a DuckDB issue, because even while this issue is occurring, I am able to query the Parquet file directly.

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 28, 2024, 7:55am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/7 "2024-06-28T07:55:47Z")

</div>

Still no luck… Upon further investigation, I did come across this issue in Github.

> <https://github.com/duckdb/duckdb/issues/6837>
>
> \### What happens?
> 
> Intermittent issues with reading CSV file and parquet files o…ver http(f)s, using duckdb CLI v0.7.1 b00b93f0b1 and duckdb-CLI v0.7.2-dev986 b6eb596089. 
> 
> Maybe this is related to the server providing the files (?), in this case https://csvbase.com/calpaterson/iris.csv and https://csvbase.com/calpaterson/iris.parquet, using SQL like:
> 
> \`\`\`sql
> from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> \`\`\`
> 
> Error messages can be:
> 
> \`\`\`
> // v0.7.1
> Error: IO Error: Server did not send Content-Length header, can not read from this file.
> Error: Invalid Error: stoll
> 
> // v0.7.2-dev986
> Error: Invalid Input Error: No magic bytes found at end of file 'https://csvbase.com/calpaterson/iris.parquet'
> Error: Invalid Error: TProtocolException: Invalid data
> Segmentation fault (core dumped)
> 
> \`\`\`
> 
> Eventually the SQL statement succeeds, often on the third attempt (v0.7.1).
> Or the statement succeeds intially, but repeating it causes errors (v0.7.2-dev986).
> 
> 
> Possibly related to #5924
> 
> \### To Reproduce
> 
> \`\`\`bash
> root@e5b47b35d70a:/data# duckdb
> \-- Loading resources from /root/.duckdbrc
> v0.7.1 b00b93f0b1
> Enter ".help" for usage hints.
> Connected to a transient in-memory database.
> Use ".open FILENAME" to reopen on a persistent database.
> 
> D from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> Error: IO Error: Server did not send Content-Length header, can not read from this file.
> 
> D from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> Error: Invalid Error: stoll
> 
> D from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> ┌────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬─────────────┐
> │ csvbase\_row\_id │ sepal length │ sepal width │ petal length │ petal width │ class │
> │ int64 │ double │ double │ double │ double │ varchar │
> ├────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼─────────────┤
> │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 3 │ 4.7 │ 3.2 │ 1.3 │ 0.2 │ Iris-setosa │
> └────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴─────────────┘
> 
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> Error: IO Error: Server did not send Content-Length header, can not read from this file.
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> Error: Invalid Error: stoll
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> ┌────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬─────────────┐
> │ csvbase\_row\_id │ sepal length │ sepal width │ petal length │ petal width │ class │
> │ int64 │ double │ double │ double │ double │ varchar │
> ├────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼─────────────┤
> │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 3 │ 4.7 │ 3.2 │ 1.3 │ 0.2 │ Iris-setosa │
> └────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴─────────────┘
> \`\`\`
> 
> When using duckdb-CLI v0.7.2-dev986 b6eb596089, downloaded from https://github.com/duckdb/duckdb/actions/runs/4497501281:
> 
> It works much better initially, but there are still some errors reported when doing multiple fetches:
> 
> \`\`\`bash
> ./duckdb -unsigned
> v0.7.2-dev986 b6eb596089
> Enter ".help" for usage hints.
> D install 'httpfs.duckdb\_extension';
> D from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> ┌────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬─────────────┐
> │ csvbase\_row\_id │ sepal length │ sepal width │ petal length │ petal width │ class │
> │ int64 │ double │ double │ double │ double │ varchar │
> ├────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼─────────────┤
> │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 3 │ 4.7 │ 3.2 │ 1.3 │ 0.2 │ Iris-setosa │
> └────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴─────────────┘
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> ┌────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬─────────────┐
> │ csvbase\_row\_id │ sepal length │ sepal width │ petal length │ petal width │ class │
> │ int64 │ double │ double │ double │ double │ varchar │
> ├────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼─────────────┤
> │ 1 │ 5.1 │ 3.5 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 2 │ 4.9 │ 3.0 │ 1.4 │ 0.2 │ Iris-setosa │
> │ 3 │ 4.7 │ 3.2 │ 1.3 │ 0.2 │ Iris-setosa │
> └────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴─────────────┘
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> Error: Invalid Input Error: No magic bytes found at end of file 'https://csvbase.com/calpaterson/iris.parquet'
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> Error: Invalid Error: TProtocolException: Invalid data
> 
> D from 'https://csvbase.com/calpaterson/iris.parquet' limit 3;
> Error: Invalid Error: TProtocolException: Invalid data
> 
> D from 'https://csvbase.com/calpaterson/iris.csv' limit 3;
> Segmentation fault (core dumped)
> \`\`\`
> 
> \### OS:
> 
> duckdb CLI (amd-64) running on Debian (docker container)
> 
> \### DuckDB Version:
> 
> CLI v0.7.1 b00b93f and CLI v0.7.2-dev986 b6eb596089
> 
> \### DuckDB Client:
> 
> CLI
> 
> \### Full Name:
> 
> Markus Skyttner
> 
> \### Affiliation:
> 
> KTH Royal Institute of Technology
> 
> \### Have you tried this on the latest \`master\` branch?
> 
> \- \[X\] I agree
> 
> \### Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
> 
> \- \[X\] I agree

Not sure if it is related, but will keep digging.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [June 28, 2024, 10:13am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/8 "2024-06-28T10:13:43Z")

</div>

Could you share the Parquet file that you’re testing with, or maybe even publish a small test repo?

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 28, 2024, 10:26am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/9 "2024-06-28T10:26:52Z")

</div>

Seems to be related to this issue…

> <https://github.com/duckdb/duckdb-wasm/issues/1658>
>
> \### What happens?
> 
> Executing twice the same request, after reloading the shell p…age, yields an error.
> 
> \### To Reproduce
> 
> in https://shell.duckdb.org/, execute :
> \`\`\`
> FROM 'https://static.data.gouv.fr/resources/communes-2023-format-parquet/20240122-085355/communes2023.parquet' 
> SELECT codgeo WHERE epci = '200039865' ;
> \`\`\`
> then reload the browser and execute that same query again.
> 
> On windows and with Chrome or Edge, i get:
> \`Invalid Error: TProtocolException: Invalid data\`
> 
> \`codgeo \`column, which is also the first of the dataset, seems to be responsible.
> 
> With Firefox, no issue.
> 
> \### OS:
> 
> Win11
> 
> \### DuckDB Version:
> 
> 10.0.0
> 
> \### DuckDB Client:
> 
> shell wasm1.28.1-dev159.0
> 
> \### Full Name:
> 
> eric mauviere
> 
> \### Affiliation:
> 
> icem7
> 
> \### Have you tried this on the latest \[nightly build\](https://duckdb.org/docs/installation/?version=main)?
> 
> I have tested with a nightly build
> 
> \### Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
> 
> \- \[X\] Yes, I have

Busy with more testing. Seems to be Windows related.

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [June 28, 2024, 10:45am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/10 "2024-06-28T10:45:18Z")

</div>

**How to reproduce the issue**

- Start up an Observable Framework project that has at least 1 parquet file being consumed
- Have at least 2 pages that will query the Parquet file
- Run the Framework dev instance
  - `npm run dev`

- Navigate to the first page using **Google Chrome** on **Windows 11**.
- Navigate to the 2nd page - The error message will appear

> `Error: Invalid Error: TProtocolException: Invalid data` will be shown.

**Workaround**

- Use Firefox
- Use `json` or `csv` files instead of `parquet`

**Related topics**

> <https://github.com/duckdb/duckdb-wasm/issues/1658>
>
> \### What happens?
> 
> Executing twice the same request, after reloading the shell p…age, yields an error.
> 
> \### To Reproduce
> 
> in https://shell.duckdb.org/, execute :
> \`\`\`
> FROM 'https://static.data.gouv.fr/resources/communes-2023-format-parquet/20240122-085355/communes2023.parquet' 
> SELECT codgeo WHERE epci = '200039865' ;
> \`\`\`
> then reload the browser and execute that same query again.
> 
> On windows and with Chrome or Edge, i get:
> \`Invalid Error: TProtocolException: Invalid data\`
> 
> \`codgeo \`column, which is also the first of the dataset, seems to be responsible.
> 
> With Firefox, no issue.
> 
> \### OS:
> 
> Win11
> 
> \### DuckDB Version:
> 
> 10.0.0
> 
> \### DuckDB Client:
> 
> shell wasm1.28.1-dev159.0
> 
> \### Full Name:
> 
> eric mauviere
> 
> \### Affiliation:
> 
> icem7
> 
> \### Have you tried this on the latest \[nightly build\](https://duckdb.org/docs/installation/?version=main)?
> 
> I have tested with a nightly build
> 
> \### Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
> 
> \- \[X\] Yes, I have

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [July 1, 2024, 10:59am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/11 "2024-07-01T10:59:34Z")

</div>

I noticed I am not the only one who found this issue… [“No results” after refreshing site when using some parquet files · Issue #1470 · observablehq/framework (github.com)](https://github.com/observablehq/framework/issues/1470)

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [July 1, 2024, 11:08am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/12 "2024-07-01T11:08:24Z")

</div>

As mentioned earlier, if you want someone else to look into this as well then it would be helpful to have a test repository with the exact setup required to reproduce the bug.

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [July 1, 2024, 11:14am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/13 "2024-07-01T11:14:52Z")

</div>

I agree… The problem is the data file I have that is not working contains sensitive data, and I am not able to reproduce this with a separate parquet file. What adds to the issue, is once I run it in cognito mode, or clear the browser cache completely, it works fine. So on face value it is not an issue with the parquet file in itself, but rather how the data gets cached.

We do have another user with the [same issue](https://github.com/observablehq/framework/issues/1470). I will continue to follow the issue on Github.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [July 1, 2024, 11:38am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/14 "2024-07-01T11:38:23Z")

</div>

Does the size of the file matter? E.g., can you still reproduce the problem with only a subset of the data?

And if part of the data suffices for a repro, could you perhaps scramble/obfuscate it?

Edit: Nvm, I just saw that the issue author shared repro steps for DuckDB’s “weather.parquet” example file: ["No results" after refreshing site when using some parquet files · Issue #1470 · observablehq/framework · GitHub](https://github.com/observablehq/framework/issues/1470#issuecomment-2174720653)

---

<div class="post-metadata">

**Author:** ![massyn](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/massyn/32/8995_2.png) [@massyn](https://talk.observablehq.com/u/massyn)\
**Post date:** [July 2, 2024, 7:55am UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/15 "2024-07-02T07:55:03Z")

</div>

Here is my test site where the same issue is reproduced, using the `weather.parquet` file from your site [DuckDB Client for Observable / CMU Data Interaction Group | Observable](https://observablehq.com/@cmudig/duckdb-client)

> **[Testing parquet bug | Test](http://static.massyn.net.s3-website-ap-southeast-2.amazonaws.com/observablehq-test/)**

Full source code are available here

> **[GitHub - massyn/observablehq-test](https://github.com/massyn/observablehq-test)**
>
> Contribute to massyn/observablehq-test development by creating an account on GitHub.

# To reproduce the issue

- Use Google Chrome on a Windows system
- Open [Testing parquet bug | Test](http://static.massyn.net.s3-website-ap-southeast-2.amazonaws.com/observablehq-test/)
- Click on Page 2
- You may need to go back to the Index page.
- You will notice the table changes to “No results”, when there should be a result.

---

<div class="post-metadata">

**Author:** ![GuillaumeChretien](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/guillaumechretien/32/9094_2.png) [@GuillaumeChretien](https://talk.observablehq.com/u/GuillaumeChretien)\
**Post date:** [July 31, 2024, 5:34pm UTC](https://talk.observablehq.com/t/duckdb-is-not-returning-rows-for-parquet-files/9426/16 "2024-07-31T17:34:56Z")

</div>

I have the same issue in a different context… so +1 !