Loading a duckdb database

Community Help
1

Hello,
I noticed that I can create DuckDB using FileAttachments loading data from json, tsv etc but not directly from a duckdb database? Is this possible?

Thanks much
Saptarshi

1 Like
2

Since v1.4.0, yes

3

Thanks! The reason I ask is because I have a costly process that produces two csv which I thought could keep in two tables in a duckdb table. I would stream this out to standard output

In the markdown I would use FileAttachment that reads “mydata.dudkdb.py” and produce a DuckDb database connection in the observable framework side of things

Is this the right way of thinking of the changes?

1 Like
4

It appears to elude me. In the markdown I have

const db = new  DuckDBClient(FileAttachment("./data/download_data.duckdb"))

and the corresponding python download_data.duckdb.py code ends with

    conn = duckdb.connect(database=tempfile)   
    conn.execute("CREATE TABLE t1 AS SELECT * FROM dcsv")
    conn.execute("CREATE TABLE t2 AS SELECT * FROM dproj") 
    conn.close()
    with open(tempfile, 'rb') as file:
        sys.stdout.buffer.write(file.read())

This works, but it looks I have an empty client.

Screenshot 2024-03-22 at 11.39.55 PM
Screenshot 2024-03-22 at 11.39.55 PM700×582 59.2 KB

When I try submitting a query, I get RuntimeError: this._db.connect is not a function. (In 'this._db.connect()', 'this._db.connect' is undefined)
thanks much
Saptarshi

5

Looks like the documentation has the answer

You can also attach a complete database saved as DuckDB file, typically using the .db file extension (or .ddb or .duckdb). In this case, the associated name (below base) is a schema name rather than a table name.

const db2 = await DuckDBClient.of({base: FileAttachment("quakes.db")});

(DuckDB | Observable Framework)
thank you again.

6

Lastly,(sorry for the noise!), I have used duckdb 0.10 python API and confirmed the database I created can be opened by duckdb CLI client. But in Observable framework, I see

RuntimeError: INTERNAL Error: Unsupported compression function type
7

The DuckDB format has been changing with every version. They’ve promised to have backward compatibility starting with 0.10.0, but Framework is still using 0.9.3 or something (it’s easy to get lost in this maze of version numbers). So for now I’d recommend to … wait a few weeks.

1 Like
8

Thanks much! Currently experimenting so can wait.