File attachments!

Hi @sourqe, I’m not sure if this will be helpfull to you but here a the few config files in my project https://github.com/maliky/images/tree/master/config-observable-react-app

Maybe you could explain more in detail what is not working for you ?

1 Like

Thank you for sharing your config files. This is exactly what I needed to get going!

@mbostock @tom if I try to query 5 column table with numbers with 2,000,000 rows it takes very very long. If files would be easier to upload I would love to do that. I have 50mb file as a result of that query. I would love to pay 10$ a month to be able to store files in Observable up to 50mb and 500mb total. $20 a month for 1gb total and 100mb per upload.

Ability to work with larger data could be monetized :wink:

1 Like

Not to redirect your question, but do you have an index over those columns?

You could still have your data fetched from external sources:

  • Obserable Secrets allow you to store API keys that can be used in private notebooks
  • Some hosters may offer the creation of unprivileged access keys, or randomize file URLs so that the path must be known.

@mootari thank you for the fast response. I’m using Observable bigquery connector. In BigQuery panel the request takes 10 times (emotionally) less time that in Observable. So lets think that the problem not in the DB. I think getting and parsing the data takes time. And if the file upload can help with that I’m willing to pay.

Is that 50MB in JSON or something more compact like CSV? I can think of some (silly) hacks to workaround the 15MB limit (like encoding your 4 integers as RGBA in a PNG) :slight_smile:

But, your point is valid. The initial limits were motivated by preventing abuse of uploads and we haven’t had time yet to think through what pricing should really look like.

I still have the same problem with a simple create react app

hi @nicola I guess this issue was addressed on this thread

and this is Mike’s github project with a step by step solution

When I downloaded my notebook, the Javascript file contains the following:

const fileAttachments = new Map([["international_data_KW.csv","https://static.observableusercontent.com/files/e36bf4125fef283d70395849db9c0e81da778efd25fbf8a3ff1b07357c065434210cf10757f2c1f459052edf56589089422df9763cebb1d4731c26cd83391001"]]);

When I try to run it, I get a CORS error as github won’t allow the fetching of this file.

When I replace the line with the following:

const fileAttachments = new Map([["international_data_KW.csv","files/e36bf4125fef283d70395849db9c0e81da778efd25fbf8a3ff1b07357c065434210cf10757f2c1f459052edf56589089422df9763cebb1d4731c26cd83391001"]]);

Like this, the data can load correctly.

Is this intentional? For my workflow it would be much better to just load the files from the local files folder. Did I forget to set some settings?

2 Likes

Apologies; this is a recent bug. It’s supposed to be a reference to a local file. A fix will be coming shortly.

2 Likes

This bug has been fixed. Thanks for letting us know.

3 Likes

Hi @mbostock Thanks for fixing this. Now I’m getting this line when I use the “Download code” link:

const fileAttachments = new Map([["chart1@9.csv",new URL("./files/2e4ad02b5e4a76f86eafed44467d0d2f6f03b55bd5ea1604fef2d082d67ce815542296c3ebfe9ae908b40080c0c4864c8275ce8532c6c3411804b6a7c2f79bab",import.meta.url)]]);

But now I’m getting this error when I compile my Gatsby project:

Generating development JavaScript bundle failed
Unexpected token (4:204)

The reason of it is that now import.meta.url is using.
How can I handle this? Many thanks for any help.

1 Like

Some bundlers don’t fully support modern JavaScript features such as import.meta. You’ll either need to fix your bundler (e.g., Babel’s @babel/plugin-syntax-import-meta plugin), or rewrite the fileAttachments declaration to point to wherever you are hosting your files. You can see an example of the latter here:

2 Likes

Is there some way to bulk delete unused file attachments? I’m often attaching files as I go along prototyping, don’t end up using them all, and don’t want to publish a notebook with lots of extra unused stuff still attached.
Thanks

4 Likes

Agreed! We have that in our backlog, along with some other bulk attachment actions, thanks!

5 Likes

thanks - appreciated!

When you fork the notebook, the fork is rid of all the unused attachments.

3 Likes
  • and its history. :confused:
1 Like

True: “unused” file attachments are a part of history, like those cells you added and then removed. It can be a positive too in some cases, to have a clean notebook without all the tinkering.

1 Like

great tip - thank you

1 Like