🏠 back to Observable

backup method?


#21

Wow, thanks for noticing this. I’ve updated my script just now too.


#22

PSA: Observable has dropped the “beta.” subdomain. Be sure to update the site path and cookie domain in your scripts.


#23

I’ve added this block in @bgchen’s version of the script:

      case 'backup-user': {
        // backup public documents for @user
        const user = (process.argv[3] || "").replace(/^@/, "");
        const dir = process.argv[4] || "data";
        if (process.argv[3]) {
          let before = "";
          const dirName = `${dir}/${user}`;
          try {
            fs.mkdirSync(dirName);
          } catch(e) {}
          do {
            const nbdat = await api.get(`/documents/@${user}${before}`);
            for (const nb of nbdat) {
              const fileName = `${dirName}/${nb.slug.replace("/", ".v")}.json`;
              let savedContent;
              try {
                savedContent = JSON.parse(fs.readFileSync(fileName, 'utf8'));
              } catch(e) {}
              if (savedContent && savedContent.version >= nb.version) {
                console.log(`Skipping ${nb.title}`);
              } else {
                console.log(`Downloading ${nb.title}`);
                const nbdat = await api.get(`/document/${nb.id}`);
                fs.writeFileSync(fileName, JSON.stringify(nbdat), {flag:'w'});
              }
            }
            before = nbdat.length ? `?before=${nbdat.pop().update_time}` : "";
          } while (before)
          break;
        }
      }

Usage:

> node index.js backup-user @fil

will create a data/fil/ directory containing all my published notebooks. Note that, if the notebook is published but has been modified since last publication, what I receive and save is the most current version.

(Should me move this thread to a github project?)


#24

I’m almost done setting up a repo, just wanted to clean up a few things beforehand.

Thanks for sharing, I was wondering about your backup requirements when I tried to plan the high-level API and helpers.


#25

this has been failing with

(node:35847) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'value' of undefined
    at ObservableAPI.authorizeWithGithub (/Users/fil/Source/observable/backup/api.js:100:29)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

(last time it worked was about a week ago)


#26

I noticed this too. I think the issue is that the Observable page now generates the “T” token client-side and posts that to the server, whereas the scripts have been getting this token from the cookie (?).

(I’ve been waiting for @mootari to share his repo so that we’ll have something nicer to build from than my crude edit :smile: )


#27

Yep, the relevant code is:

      onClick: ()=>{
        n && n(),
        window.location.assign(function(e) {
          return `https://github.com/login/oauth/authorize?scope=user:email&client_id=1a8619df27715d9d2c97&state=${pu()}&redirect_uri=${`https://api.observablehq.com/github/oauth?path=/loggedin${e}`}`
        }(r))
      }

and

  function pu() {
    const e = document.cookie.match(/(?:^|;)\s*T\s*=\s*([0-9a-f]{32})(?:$|;)/);
    if (e)
      return e[1];
    const t = (n = 16,
    Array.from(crypto.getRandomValues(new Uint8Array(n)), e=>e.toString(16).padStart(2, "0")).join(""));
    var n;
    const r = new Date(Date.now() + 1728e5);
    return document.cookie = `T=${t}; Domain=.observablehq.com; Path=/; Secure; Expires=${r.toUTCString()}`,
    t
  }

So, so sorry about the delay. These last days I’ve been either too swamped or too tired to finish setting up the repo. I’ll set it up this weekend, promise. :slight_smile:


#28

I’ve updated the gist to have ensureToken() generate the token by itself instead of fetching it from the server.


#29

Thanks! I used your work to update my version just now:

I made one change to your new ensureToken. Instead of:

ensureToken(regenerate = false) {
  if (!regenerate && this.getToken()) return;

I use:

ensureToken(regenerate = false) {
  if (!regenerate && this.getToken() && this.getToken().value !== '') return;

The third clause is necessary since after authentication, the response from https://observablehq.com/loggedin has a set-cookie header which reads set-cookie: T=; Max-Age=0; Domain=.observablehq.com; HttpOnly; Path=/; Secure.

Edit: updated per @mootari’s comment below.


#30

I noticed that too, but wrongly dismissed it as irrelevant. I’d add the check against value to getToken() though, otherwise code might fetch a token cookie with an empty value. I’ve updated my gist accordingly.