# backup method?

**URL:** <https://talk.observablehq.com/t/backup-method/1628>\
**Category:** Help\
**Created:** [January 31, 2019, 1:58pm UTC](https://talk.observablehq.com/t/backup-method/1628 "2019-01-31T13:58:31Z")\
**Posts on this page:** 18\
**Page:** 2

<div class="post-metadata">

**Author:** ![bgchen](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/bgchen/32/784_2.png) [@bgchen](https://talk.observablehq.com/u/bgchen)\
**Post date:** [February 24, 2019, 5:39pm UTC](https://talk.observablehq.com/t/backup-method/1628/21 "2019-02-24T17:39:56Z")

</div>

Wow, thanks for noticing this. I’ve updated my script just now too.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [February 26, 2019, 11:25pm UTC](https://talk.observablehq.com/t/backup-method/1628/22 "2019-02-26T23:25:47Z")

</div>

PSA: Observable has dropped the “beta.” subdomain. Be sure to update the site path and cookie domain in your scripts.

---

<div class="post-metadata">

**Author:** ![Fil](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/fil/32/207_2.png) [@Fil](https://talk.observablehq.com/u/Fil)\
**Post date:** [March 3, 2019, 5:48pm UTC](https://talk.observablehq.com/t/backup-method/1628/23 "2019-03-03T17:48:12Z")

</div>

I’ve added this block in @bgchen’s version of the script:

```auto
      case 'backup-user': {
        // backup public documents for @user
        const user = (process.argv[3] || "").replace(/^@/, "");
        const dir = process.argv[4] || "data";
        if (process.argv[3]) {
          let before = "";
          const dirName = `${dir}/${user}`;
          try {
            fs.mkdirSync(dirName);
          } catch(e) {}
          do {
            const nbdat = await api.get(`/documents/@${user}${before}`);
            for (const nb of nbdat) {
              const fileName = `${dirName}/${nb.slug.replace("/", ".v")}.json`;
              let savedContent;
              try {
                savedContent = JSON.parse(fs.readFileSync(fileName, 'utf8'));
              } catch(e) {}
              if (savedContent && savedContent.version >= nb.version) {
                console.log(`Skipping ${nb.title}`);
              } else {
                console.log(`Downloading ${nb.title}`);
                const nbdat = await api.get(`/document/${nb.id}`);
                fs.writeFileSync(fileName, JSON.stringify(nbdat), {flag:'w'});
              }
            }
            before = nbdat.length ? `?before=${nbdat.pop().update_time}` : "";
          } while (before)
          break;
        }
      }

```

Usage:

`> node index.js backup-user @fil`

will create a data/fil/ directory containing all my published notebooks. Note that, if the notebook is published but has been modified since last publication, what I receive and save is the most current version.

(Should me move this thread to a github project?)

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [March 3, 2019, 6:27pm UTC](https://talk.observablehq.com/t/backup-method/1628/24 "2019-03-03T18:27:18Z")

</div>

> [@Fil](#):
>
> (Should me move this thread to a github project?)

I’m almost done setting up a repo, just wanted to clean up a few things beforehand.

> [@Fil](#):
>
> I’ve added this block in @bgchen’s version of the script:

Thanks for sharing, I was wondering about your backup requirements when I tried to plan the high-level API and helpers.

---

<div class="post-metadata">

**Author:** ![Fil](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/fil/32/207_2.png) [@Fil](https://talk.observablehq.com/u/Fil)\
**Post date:** [March 19, 2019, 7:54am UTC](https://talk.observablehq.com/t/backup-method/1628/25 "2019-03-19T07:54:45Z")

</div>

this has been failing with

```auto
(node:35847) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'value' of undefined
    at ObservableAPI.authorizeWithGithub (/Users/fil/Source/observable/backup/api.js:100:29)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

```

(last time it worked was about a week ago)

---

<div class="post-metadata">

**Author:** ![bgchen](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/bgchen/32/784_2.png) [@bgchen](https://talk.observablehq.com/u/bgchen)\
**Post date:** [March 19, 2019, 2:32pm UTC](https://talk.observablehq.com/t/backup-method/1628/26 "2019-03-19T14:32:08Z")

</div>

I noticed this too. I think the issue is that the Observable page now generates the “T” token client-side and posts that to the server, whereas the scripts have been getting this token from the cookie (?).

(I’ve been waiting for @mootari to share his repo so that we’ll have something nicer to build from than my crude edit 😄 )

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [March 19, 2019, 7:05pm UTC](https://talk.observablehq.com/t/backup-method/1628/27 "2019-03-19T19:05:26Z")

</div>

> [@bgchen](#):
>
> I noticed this too. I think the issue is that the Observable page now generates the “T” token client-side and posts that to the server, whereas the scripts have been getting this token from the cookie (?).

Yep, the relevant code is:

```auto
      onClick: ()=>{
        n && n(),
        window.location.assign(function(e) {
          return `https://github.com/login/oauth/authorize?scope=user:email&client_id=1a8619df27715d9d2c97&state=${pu()}&redirect_uri=${`https://api.observablehq.com/github/oauth?path=/loggedin${e}`}`
        }(r))
      }

```

and

```auto
  function pu() {
    const e = document.cookie.match(/(?:^|;)\s*T\s*=\s*([0-9a-f]{32})(?:$|;)/);
    if (e)
      return e[1];
    const t = (n = 16,
    Array.from(crypto.getRandomValues(new Uint8Array(n)), e=>e.toString(16).padStart(2, "0")).join(""));
    var n;
    const r = new Date(Date.now() + 1728e5);
    return document.cookie = `T=${t}; Domain=.observablehq.com; Path=/; Secure; Expires=${r.toUTCString()}`,
    t
  }

```

> [@bgchen](#):
>
> (I’ve been waiting for @mootari to share his repo so that we’ll have something nicer to build from than my crude edit 😄 )

So, so sorry about the delay. These last days I’ve been either too swamped or too tired to finish setting up the repo. I’ll set it up this weekend, promise. 🙂

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [March 24, 2019, 12:05am UTC](https://talk.observablehq.com/t/backup-method/1628/28 "2019-03-24T00:05:21Z")

</div>

I’ve updated the gist to have `ensureToken()` generate the token by itself instead of fetching it from the server.

> <https://gist.github.com/mootari/511b751e325db8316bb3138dcb0a7393/revisions?diff=split>

---

<div class="post-metadata">

**Author:** ![bgchen](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/bgchen/32/784_2.png) [@bgchen](https://talk.observablehq.com/u/bgchen)\
**Post date:** [March 24, 2019, 1:53am UTC](https://talk.observablehq.com/t/backup-method/1628/29 "2019-03-24T01:53:12Z")

</div>

Thanks! I used your work to update my version just now:

> <https://gist.github.com/bryangingechen/9d86f1e5ec01674a32dd9c54bdc13947/revisions?diff=split>

I made one change to your new `ensureToken`. Instead of:

```auto
ensureToken(regenerate = false) {
  if (!regenerate && this.getToken()) return;

```

I use:

```auto
ensureToken(regenerate = false) {
  if (!regenerate && this.getToken() && this.getToken().value !== '') return;

```

The third clause is necessary since after authentication, the response from `https://observablehq.com/loggedin` has a `set-cookie` header which reads `set-cookie: T=; Max-Age=0; Domain=.observablehq.com; HttpOnly; Path=/; Secure`.

**Edit** : updated per @mootari’s comment below.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [March 24, 2019, 7:09pm UTC](https://talk.observablehq.com/t/backup-method/1628/30 "2019-03-24T19:09:09Z")

</div>

> [@bgchen](#):
>
> The third clause is necessary since after authentication

I noticed that too, but wrongly dismissed it as irrelevant. I’d add the check against value to getToken() though, otherwise code might fetch a token cookie with an empty value. I’ve updated my gist accordingly.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [March 26, 2019, 10:00pm UTC](https://talk.observablehq.com/t/backup-method/1628/31 "2019-03-26T22:00:17Z")

</div>

The repo is now available:

> **[mootari/observable-client](https://github.com/mootari/observable-client)**
>
> Contribute to mootari/observable-client development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Fil](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/fil/32/207_2.png) [@Fil](https://talk.observablehq.com/u/Fil)\
**Post date:** [August 13, 2020, 1:58pm UTC](https://talk.observablehq.com/t/backup-method/1628/32 "2020-08-13T13:58:55Z")

</div>

Is this still working for you? I have stopped using it for quite a while and now it doesn’t want to connect anymore.

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [August 13, 2020, 4:34pm UTC](https://talk.observablehq.com/t/backup-method/1628/33 "2020-08-13T16:34:06Z")

</div>

Never got around actually to using it. 🙂 Does the authentication fail, or can you at least retreive a token?

Edit: The login has now an extra step, and the CSRF token only gets set after one inputs the name. 🤔

---

<div class="post-metadata">

**Author:** ![Fil](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/fil/32/207_2.png) [@Fil](https://talk.observablehq.com/u/Fil)\
**Post date:** [August 13, 2020, 4:51pm UTC](https://talk.observablehq.com/t/backup-method/1628/34 "2020-08-13T16:51:25Z")

</div>

Don’t worry—I just wanted to check if it was just broken for me, but I don’t really _need_ it for the moment. I still would like to be able to bulk download easily for backup and grep 🙂

---

<div class="post-metadata">

**Author:** ![mootari](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/mootari/32/581_2.png) [@mootari](https://talk.observablehq.com/u/mootari)\
**Post date:** [August 13, 2020, 4:54pm UTC](https://talk.observablehq.com/t/backup-method/1628/35 "2020-08-13T16:54:21Z")

</div>

> [@Fil](#):
>
> but I don’t really _need_ it for the moment

Noone else has complained either. I may have to assume that noone is using it. 😢

Anyway, can I ask you to open an issue in [GitHub - mootari/observable-client](https://github.com/mootari/observable-client) ?

---

<div class="post-metadata">

**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\
**Post date:** [May 6, 2021, 1:08pm UTC](https://talk.observablehq.com/t/backup-method/1628/36 "2021-05-06T13:08:18Z")

</div>

I dunno if I should necro these threads but it makes sense to have only one backup thread IMHO. My backup solution exports to storage, ordered by update timestamp and checks version ids so it can stop early if nothing has changed.

> **[Notebook Backup](https://observablehq.com/@tomlarkworthy/notebook-backups)**
>
> Provision storage Use mootari's notebook list to find recently published notebook Backup: Iterate through list starting at most recent Save the version in file metadata, so we skip backing up the same file everyday

I’ll probably automate it with cron once I gat some confidence with it.

I am not really planning on making this a service, but its pretty easy to copy if people want.

---

<div class="post-metadata">

**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\
**Post date:** [February 5, 2022, 7:52pm UTC](https://talk.observablehq.com/t/backup-method/1628/37 "2022-02-05T19:52:24Z")

</div>

I did not like the previous approach in the end, it requires too much manual triggering and it was hard to setup in the first place, plus the end result was a tar archive that was hard to interact with. So based on the learnings from the previous one, I have done a fresh approach.

This new backup solution triggers a Github Action to unpacks the tar code, syncs with Github, and runs after every publish automatically, it also works with non-public team notebooks! You can point everything at a common repository, because the notebooks are unpacked into a directory mirroring their URL, you can take a look [here](https://github.com/endpointservices/observable-notebooks). It only took me 270 Github action attempts before I got it!

> **[Automatically Backup Observable notebooks to Github](https://observablehq.com/@tomlarkworthy/github-backups)**
>
> Take control of your data and relax. Backup your public and team Observable notebooks to a Github repository automatically when published.
> By using a combination of on version hook which executes after a notebook is published, and repository dispatch...

---

<div class="post-metadata">

**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\
**Post date:** [May 11, 2022, 8:01am UTC](https://talk.observablehq.com/t/backup-method/1628/38 "2022-05-11T08:01:59Z")

</div>

The intent of the github backups notebook is that you could set it up once in a personal backup notebook that can then be transitively imported everywhere you need backups and avoid having to configure the github token each time.

This was not working under certain conditions, thanks @jimpick for reporting the issue, which is now solved. As you can see I personally have quite the collection of backups now:- [observable-notebooks/@endpointservices at main · endpointservices/observable-notebooks · GitHub](https://github.com/endpointservices/observable-notebooks/tree/main/%40endpointservices)

which only requires me to import my footer notebook

> **[Endpoint Services Footer](https://observablehq.com/@endpointservices/footer-with-backups)**
>
> Generic services to apply to all Endpoint Service notebooks, e.g. error monitoring, analytics and backups. Endpoint Services collects usage metrics through Plausible Analytics and publishes them here. Notebooks are monitored for errors using...

that footer does a few other useful things like install an error reporting framework and usage analytics.

[Previous page](https://talk.observablehq.com/t/backup-method/1628.md?page=1)