If Observable authenticates via GitHub, can I easily write to it?

[aaronkyle]

Is it possible to write to an existing GitHub file from Observable? How about creating a file?

[mootari]

Assuming you’re talking about notebook actions, the answer is no. Your notebook context has no knowledge of and access to your user session on observablehq.com. Both are strictly separated.

[j-f1]

However, you could add a personal access token to your secrets and access it by calling Secret('GH_TOKEN').

[aaronkyle]

@j-f1

However, you could add a personal access token to your secrets and access it by calling Secret('GH_TOKEN') .

Is this a set-up that anyone would care to demonstrate? It would be great to set up an simple interface to write in and out of files stored on GitHub.

[j-f1]

Here’s a quick example:

GitHub API

[aaronkyle]

Awesome. Now to my task of understanding!

I see how one points to different pages, but when pointing to my own page, entering a token, pressing ‘submit’ and seeing the cells dance to indicate that they are updated, I don’t seem to be able to write to the text. Is this because it’s showing the rendered text? And instead we’d need some sort of input field?

[mootari]

Turns out you don’t even need a token to list files:

List Github Repo Contents / Fabian Iwand / Observable

[j-f1]

@mootari

Turns out you don’t even need a token to list files

Yep — I just put the token there to make it possible to write to the files when the time comes.

[bgchen]

Just for fun, here’s a notebook in my favorites (by @asg017!) on using github in Observable:

Github API in Observable (v3 and v4)

[aaronkyle]

This is all really great stuff! Thank you for the guidance and example notebooks.

I welcome any further examples of writing to GitHub-hosted files, as I still don’t quite get how this would be done. I’ll also read more about the API. As for the goal: It would be especially cool to be able to host a .csv file with information describing some thing (like number of eggs in a basket) to be able to edit and save a value in the GitHub .csv file from an Observable input cell. It would similarly be cool to have the text of a markdown document rendered into an Observable textarea input cell, so that clicking ‘save’ writes the changed text back to the markdown file.

I welcome any further examples and guidance! Thanks again!

[mootari]

@aaronkyle

It would similarly be cool to have the text of a markdown document rendered into an Observable textarea input cell, so that clicking ‘save’ writes the changed text back to the markdown file.

May I ask why you want to edit your Github hosted files via Observable? Asking because Github’s UI already lets you edit markdown files:

[j-f1]

@aaronkyle

I welcome any further examples of writing to GitHub-hosted files

Here’s GitHub’s docs for doing just that: Repository contents - GitHub Docs

[aaronkyle]

@mootari

May I ask why you want to edit your Github hosted files via Observable? Asking because Github’s UI already lets you edit markdown files

I was waiting for someone to ask this.

In essence, I’d like to be able to expose some GitHub pages for editing (like those terms) from a rendered-out version of the page like this one [without having to go back to GitHub].

But this probably makes more sense in the CSV context - like if I were to host the CSV back-end for this work-in-progress (and please forgive me when you see this page… if it ever loads… you’ve already demonstrated a far better approach… I just haven’t caught up with you yet).

@j-f1

Here’s GitHub’s docs for doing just that: Repository contents - GitHub Docs

Thanks for this, Jed. I must admit - each time I try reading and working through these API guides I hit a wall. It’s still too much for my novice understanding of data manipulation and management. I’ll keep going (I’ve learned a lot this year!), but maybe I’m over-reaching…

[j-f1]

@aaronkyle

I must admit - each time I try reading and working through these API guides I hit a wall.

It looks like @asg017’s notebook currently only supports GET requests at the moment, so you’ll have to roll your own client. Here’s a starter bit of code for you:

const headers = { Authorization: `bearer ${token}` }
const url = `https://api.github.com/repos/${owner}/${repo}/contents/${path}`
// To create a file:
const body = {
  message: "Add a file. This is the commit message.",
  content: btoa('Hello, world! This is the content of the file.'),
  /* optional, defaults to the user who owns the auth token: */
  committer: { name, email },
  /* optional, defaults to the repository’s default branch */
  branch: 'master'
}
fetch(url, { method: 'PUT', body: JSON.stringify(body), headers })

// To update a file:
const file = await fetch(url, { headers }).then(res => res.json())
const body = {
  sha: file.sha,
  message: "Update a file. This is the commit message.",
  content: btoa('Hello, world! This is the new content of the file.'),
  /* optional, defaults to the user who owns the auth token: */
  committer: { name, email },
  /* optional, defaults to the repository’s default branch */
  branch: 'master'
}
fetch(url, { method: 'PUT', body: JSON.stringify(body), headers })

[aaronkyle]

@aaronkyle

if it ever loads…

In case that page doesn’t load… Here are a couple of more links to show you where I am going:

1. a super simple notebook about naming cases - the idea here is the same as the CSV-driven input page (with without CSV). The idea is that data would be stored in some database, etc, and re-presented as pre-populated input fields. Currently I am only demonstrating the concept, but don’t have a way to ‘save’ the change from the UI (without editing the code, that is).

2. a mock UI with toggle buttons to edit - this is how I’m envisioning a page that would allow users to edit information about a case (note that only the active ‘overview’ tab is modeling the behaviors I want - with buttons to the right).

… To be clear - I don’t really need to do this with GitHub (and given how much more work it involves than I was hoping, I probably I won’t). To step back a bit: I am trying to come up with a way to manage information that is totally transparent (and ideally free for anyone to replicate / use). I am enjoying learning more about JS and availing of all that Observable has to offer to work out proofs-of-concept for user interfaces, but I don’t have any knowledge of how one goes about connecting UIs to databases, etc. Where I am currently at is to save and store this information in .csv files, which I manage in Excel… but this quickly gets tedious, so I am seeking a better way.

I expect there are tools in the Observable pipeline that will help me with this, like the promised Google Drive connector, which I anticipate will allow me to use Google Sheets as the CSV / TSV data store, as well as the database client announced by Visnu. But databases still scare me, and using GitHub as a database seems like a reasonable, friendly idea.

[mootari]

@aaronkyle

In essence, I’d like to be able to expose some GitHub pages for editing (like those terms) from a rendered-out version of the page like this one [without having to go back to GitHub].

Who will be able to edit these? How do you plan to restrict access?

[aaronkyle]

@mootari

Who will be able to edit these? How do you plan to restrict access?

I would expect that only I would be able to edit my own repos (or team members for an organizational repo). Otherwise, you’d just fork my repo, add in your own access token to some form of connector, and then edit yourself

[mootari]

@aaronkyle

I would expect that only I would be able to edit my own repos

Then what exactly is the point of making them editable through Observable? From what I understand I’d have to fork your notebook(s), repository/ies, create access tokens, add them and then tie everything together?

[aaronkyle]

@mootari

Then what exactly is the point of making them editable through Observable?

The idea would be to use Observable to quickly build out visualizations from the data, and also to be able to correct / update data on the fly. You’re correct that it’d be some work to fork and re-connect the underlying repository, but I suppose this is where the ‘organizational’ / team repos that GitHub offers would be advantageous.

It’ll take me some time to bring all of this together in a way that exemplifies this more clearly, but I am working on it. To try to elaborate a bit more:

This notebook renders out data fields for all cases historically received by the office where I work. This is the same information that is available here. This information changes regularly, and rather than updating the CSV that drives the Observable notebook separately and re-upload to a hosting environment each time, it’d be awesome to make and save those edits within Observable. I started working on my slow-to-load UI template notebook above to model how this would help in terms of organizing how information is collected, and then routing collected information into various reports.

I am getting kicked out of my office by movers now, so I apologize that this is a rushed explanation.

[aaronkyle]

@mootari

Then what exactly is the point of making them editable through Observable?

Ok - toward a rationale for writing from Observable to Github (with examples of use case):

Let’s say I don’t want to deal with a CMS, but I have lots of different types of content that I wish to publish to the web. GitHub has done a lot to make this easy and convenient, but Observable is even more convenient – especially when testing out visualizations that cannot be viewed from the GitHub UI.

If I had an easy toolbox to edit and manipulate GH pages from within Observable, I could test changes within Observable, and then, without ever leaving my Observable notebook context, I could save changes to my visualizations – and the data that drives them.

As one minor example:

In this Observable notebook I create a series of inputs that describe cases received by CAO (CAO receives complaints by people affected by environmental and social impacts of development projects). As the office processes complaints in phases, it will regularly add and edit the information that describes a case. To keep on top of this information, rather than having the people working on cases try to directly edit the CSV data source (which is cumbersome, as each case has over 500 attributes that describe it, and the office has around 75 active cases at the moment), I’d like to define a subset of the Observable UI notebook via HTML pages, and from these–the part I haven’t figured out yet–save changes back into the database.

Currently the database is a single CSV file. If we put this CSV file on GitHub using an organization account, we’d encourage transparent access to information and enhance accountability for changes to the data set. That is, the organizational data record would be public, and only users with organizational access tokens could edit this ‘official’ version of our CSV data.

Once I have patterns from writing from Observable to GitHub in this way, I could also more quickly review and edit historical data (which is pretty messy still) as I use Observable to ask questions of the data – at which point I often find that a value is missing somewhere, or there’s some edit. So again – the trick here would be to edit and save changes on-the-fly without having leave the Observable context. That is, rather than to dig into a specific cell in a massive/sprawling CSV data source, I could just call some shortcut I made to a UI for that cell within Observable, save the revised value, and go on my merry way

Still more cool / meta - if I were to open an editor to an HTML page on GitHub that embeds Observable cells, I could also edit / extend this page from within the very notebook that it references – meaning that I could use Observable as an effective CMS ‘front-end’ for a GitHub ‘back end’.

Seems to me like a nice set up