File attachments!

mike · October 30, 2019, 7:48pm

We just deployed and announced a new feature: file attachments!

You can now upload and attach files directly to notebooks, making it much easier to work with small-ish datasets. We’ve updated many of our example notebooks to use this new feature.

Please try it out and let us know what you think!

mootari · October 30, 2019, 8:42pm

Can you share more details on the rolling quota?

mike · October 30, 2019, 8:50pm

Not currently. We’ll be evaluating the rolling quota based on usage.

mootari · October 30, 2019, 9:01pm

Is it a space or bandwidth quota? If I should ever hit the quota, can I just delete files? And does the storage backend deduplicate artifacts (like git LFS)?

Edit: I just realized that responding to “not currently” with the above questions may appear ignorant, so please let me add the following introduction:

If you can’t share details on the imposed limits, perhaps you can still shed some light on the following questions? […]

mootari · October 30, 2019, 9:20pm

~~@mike I just pasted a URL into a cell and it got automatically turned into an attachment. This was absolutely not my intention. Please disable this “feature” or make it optional.~~

Nevermind, I’m dumb. Had the image in the clipboard instead of the URL. Nice feature!

Edit: However, I’d still love to see a setting to disable this feature or a confirmation dialog. Removing an accidentally added attachment currently requires six interaction steps.

mootari · October 30, 2019, 10:19pm

The paste feature still needs polish. Here’s the result of keeping cmd+v pressed for approx. two seconds, while an image (2606x3240, link) was in the clipboard:

notebook freezes while 37 OPTIONS requests are made, in intervals of ~7 seconds.
after ~ 2 min: notebook becomes responsive again, upload starts. Displayed progress appears to switch between multiple files (goes back and forth).
after ~5 min: notebook crashes with an uncaught error inside a progress event, apparently triggered by a CORS error (hit the quota?).
after reloading: 32 files were added. Deleting a single file requires three clicks.

tom · October 30, 2019, 10:49pm

We’ll look into showing progress UI faster when you hit paste. In the meantime we recommend not holding down paste - like GitHub, this forum, and any text field, holding down paste keeps pasting.

mootari · October 30, 2019, 11:20pm

My intention was to demonstrate how little of an effort is needed to cause results that may take significantly longer to clean up. Some notebooks make use of the clipboard, and in these the right or wrong focus makes all the difference.

I’d suggest the following additions:

don’t respond to repeated pastes of the same blob until the first instance has been uploaded and mapped,
hash blobs and map them to existing files via hashes (would require roundtrip before post),
allow bulk deletion of files.

jrus · October 31, 2019, 6:32pm

Great new feature folks!

Can files be renamed? It seems a bit limiting to need to pick the name on disk before upload.

Future feature request: use the same system for uploaded notebook thumbnails.

Tiny note: in the docs, the contents list at the top includes a link to https://github.com/observablehq/stdlib#file_attachments while the anchor link down the page uses a hyphen instead of an underscore https://github.com/observablehq/stdlib#file-attachments FIXED

mike · October 31, 2019, 2:09pm

Not currently, sorry! We’ll likely add this in the future.

One wrinkle is that files are currently immutable, so “renaming” would create a new alias for that file and replace references to the old name with the new name. However, we would eventually like to allow file mutability while preserving notebook history, so stay tuned.

j-f1 · November 1, 2019, 12:09am

One option would be to do it in a way similar to Git, where you have each version reference the hash of each attachment:

// in the notebook data
"files": {
  "foo.txt": "9206ac42b532ef8e983470c251f4e1a365fd636c",
  "bar.json": "e706dccea74c3b564eda845fed6e2e725f680a91"
}

and you could download the files from the CDN by hash.

This would allow updating files by changing the hash in the new version (and uploading the new file to the CDN) and attaching the new file. Users could also go back in the history and delete the files, which would delete them from the CDN and cause the FileAttachment call to fail when inspecting previous versions.

This would also allow renaming files since you could point a new name to the same hash in a subsequent version.

jipexu · November 4, 2019, 10:28am

Great feature !
I have probably a stupid question …
if you download the code of a notebook with some file attached, to your serveur :
is this file dowloaded to the server ?
if this file is updated on server side (for example by php from mysql query)

the final result of the js will take the change ??

Thanks

mike · November 4, 2019, 6:01pm

If you download your notebook’s tarball, the tarball will include all of your notebook’s files (and any transitive import’s files, too). FileAttachment will pass a relative path to fetch to load the file from your server. The file names are hashes, but if you look at the generated code you can determine which hashes corresponds to which file names.

You can replace the downloaded file in the tarball with different contents, or you can edit the code to point to a different file, as you prefer.

jipexu · November 4, 2019, 7:16pm

ok many thanks Mike its great !

aaronkyle · November 5, 2019, 10:41pm

I’ve been disconnected from the Internet for most of the past month, and therefore slow to realize the impact of this feature: it’s a huge deal. Thank you! It’s really nice to be able to work with data files without having to find a way to host and link them outside Observable. While Observable, at launch, helped reduce the barriers to entry for me learning JS (and this community helps so much too), the many innovations you’ve made in the short time since launch have further eliminated several of the sticking points that originally stymied me. I can’t express enough my gratitude for this gift.

Thank you so much!

mike · November 5, 2019, 11:07pm

Thanks @aaronkyle! We appreciate you sticking with us.

maliky · November 9, 2019, 7:07pm

Thank you @aaronkyle for expressing what I feel too towards the observable team :o)

Concerning the file attachements, beside the announcement, I noticed the change in an unfortunate way.

I had successfully bundled a notebook referring to @jashkenas/inputs but since the header image was added as a file attachement src="${await FileAttachment("capstan.gif").url()}" I failed because I need to do some additionnal configuration with webpack.

The error message of yarn build is the following.

ERROR in ./node_modules/@maliky/3-filtres-pour-les-donnees-des-mooc/e93997d5089d7165@2200.js 4:203
Module parse failed: Unexpected token (4:203)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| export default function define(runtime, observer) {
|   const main = runtime.module();
>   const fileAttachments = new Map([["capstan.gif",new URL("./files/c051fbc024553912e31968b35e537d4ad3592201b5f8e7bd13fd9d02e38599c5d541a704d0858c676328babb3e5c9c35dd7c6d67240090d094882a1cad8eece4",import.meta.url)]]);
|   main.builtin("FileAttachment", runtime.fileAttachments(name => fileAttachments.get(name)));
|   main.variable(observer()).define(["md","FileAttachment"], async function(md,FileAttachment){return(
 @ ./node_modules/@maliky/3-filtres-pour-les-donnees-des-mooc/482865d740b971e4@946.js 3:0-49 414:32-39
 @ ./static/rdb/js/src/collapse.index.js
error Command failed with exit code 2.

The e93997d5089d7165@2200.js is Jemery’s inputs notebook. and the error is related to webpack not been able to handle the file attachement type.

I’ve juste started to look into a solution to this and install bable loader for markdown, file and images (including gif) but something else is blocking the bundling process. Does it have to do with the Map or URL keyword ?

I’m sure that once I’ll have resolved this webpack configuration issue I get a lot of benefits from file attachements.

mike · November 9, 2019, 10:23pm

Does webpack support import.meta? You might need this plugin, or similar:

github.com

open-wc/open-wc/blob/master/packages/webpack-import-meta-loader/webpack-import-meta-loader.js

/* eslint-disable */
// @ts-nocheck
const toBrowserPath = require('./src/to-browser-path');
const regex = /import\.meta/g;

/**
 * Webpack loader to rewrite `import.meta` in modules with url data to the source code file location.
 *
 * @example
 * return import.meta;
 * // becomes: return ({ url: `${window.location.protocol}//${window.location.host}/relative/path/to/file.js` });
 *
 * return import.meta.url;
 * // becomes: return ({ url: `${window.location.protocol}//${window.location.host}/relative/path/to/file.js` }).url;
 */
module.exports = function(source) {
  const path = require('path');

  const relativePath = this.context.substring(
    this.context.indexOf(this.rootContext) + this.rootContext.length + 1,

This file has been truncated. show original

maliky · November 11, 2019, 6:31am

I was stuck trying to set-up two loaders: ‘babel-loader’ and ‘webpack-import-meta-loader’ for the same ‘js’ file in webpack.config.js

So I also looked at @babel/plugin-syntax-import-meta but installing it and adding the adequate line in for .babelrc was not enough.

I came back to the webpack.config.js and just added the two loaders one after another as in :

 module: {
   rules: [
     { test: /\.css$/, use: "css-loader" },
     {
       test: /\.js$/,
       loader: require.resolve("@open-wc/webpack-import-meta-loader")
     },
     {
       test: /.(js|jsx)$/,
       include: [path.resolve(__dirname, "static/rdb/js/src")],
       use: {
         loader: "babel-loader"
       }
     }
   ]
 }

And it works…
Thank you