Accessing the BacDive API

Community Help
1
  • I would like to access the Bacterial Diversity database - Bacdive - in an Observable Notebook by using the Bacdive API, but I am having trouble setting this up.
  • I have tried using the Oauth 2.0 Client from Tom Larkworthy, but am having trouble getting it to work.

From my reading of the code for the REST client for BacDive web service in Python the bacdive authorisation process is based on keycloak. Has anyone ever tried authorisation via keycloack? Do you think Tom’s Oauth client could help me if I did it right?

Much appreciated.

Observable notebook

2

yeah its not obvious what all the server URL settings should be. I would use the python client to run a successful login and read all the network traffic to understand where the token endpoint and stuff is. The keycloak client is initialized with few arguments, a base URL, and not the more detailed stuff, so its leaning on the URL defaults for keycloak, which I do not know.

python-keycloak package depends on “requests”, so to record the network history in python you can use

you could read the python-keycload source too and figure this out maybe (if there is not some magic auto-configuration), but I prefer direct measurement of an empirically verifiably working process over static reasoning

3

BTW the oauth notebook was successfully used by cornhundred to do a google login for biometric dataset too. The main point of that notebook is to hide the client-secret, but given that endpoint doesn’t use a secret you can probably get it all working without that notebook in raw fetch… if you can figure out the server URL pattern. For that, monkey patching requests will get you required authoritative data.

If you get stuck I can help you in a call.

4

Thanks, I’ll see if I can work it out with your advices and let you know how it goes.

5

As you (@tomlarkworthy) advised, I used the python client and spied out the whole connection process, but it’s still too abstruse for me. Here is the logging:
→ the python code :
client = bacdive.BacdiveClient(‘myUsername’, ‘myPassword’)
→ the logging:
DEBUG:urllib3.util.retry:Converted retries value: 1 → Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.util.retry:Converted retries value: 1 → Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): sso.dsmz.de:443 DEBUG:http.client:send: b’CONNECT sso.dsmz.de:443 HTTP/1.1\r\nProxy-Authorization: Basic bGluYXJkLXk6MWtjaW5uQVkhITcy\r\nHost: sso.dsmz.de:443\r\n\r\n’ DEBUG:http.client:header: DEBUG:http.client:send: b’POST /auth/realms/dsmz/protocol/openid-connect/token HTTP/1.1\r\nHost: sso.dsmz.de\r\nUser-Agent: python-requests/2.31.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: /\r\nConnection: keep-alive\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 148\r\n\r\n’ DEBUG:http.client:send: b’username=myUsername&password=myPassword&client_id=api.bacdive.public&grant_type=password&code=&redirect_uri=&scope=openid’ DEBUG:http.client:reply: ‘HTTP/1.1 200 OK\r\n’ DEBUG:http.client:header: cache-control: no-store DEBUG:http.client:header: set-cookie: KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/DSMZ/; Secure; HttpOnly DEBUG:http.client:header: set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/DSMZ/; Secure; HttpOnly DEBUG:http.client:header: x-xss-protection: 1; mode=block DEBUG:http.client:header: pragma: no-cache DEBUG:http.client:header: x-frame-options: SAMEORIGIN DEBUG:http.client:header: referrer-policy: no-referrer DEBUG:http.client:header: date: Tue, 04 Feb 2025 15:47:48 GMT DEBUG:http.client:header: strict-transport-security: max-age=31536000; includeSubDomains DEBUG:http.client:header: x-content-type-options: nosniff DEBUG:http.client:header: content-type: application/json DEBUG:http.client:header: content-length: 2737 DEBUG:urllib3.connectionpool:https://sso.dsmz.de:443 “POST /auth/realms/dsmz/protocol/openid-connect/token HTTP/1.1” 200 2737

– Authentication successful –

6 
POST /auth/realms/dsmz/protocol/openid-connect/token

server URL

Host: sso.dsmz.de:443

Things to note, its using form encoding

Content-Type: application/x-www-form-urlencoded

Here are the params, note code and redirect_uri are blank, and there is no client_secret

username=myUsername&password=myPassword&client_id=api.bacdive.public&grant_type=password&code=&redirect_uri=&scope=openid

So it does that to start the conversation.

Then it maybe it does a post with the response at the end to swap a token

POST /auth/realms/dsmz/protocol/openid-connect/token HTTP/1.1

Yeah, you don’t need the oauth notebook for this, this can be replicated with vanilla fetch. Note it sends grant=password which identifies the protocol exactly that it is using, documented here: Password Grant - OAuth 2.0 Simplified

Here is my best guess at the first part but I am getting 401 because I guess the password is wrong Community Help / Tom Larkworthy | Observable

7

Do you mean something like that (for the first part):

await fetch(‘https://sso.dsmz.de:443’, {
method: ‘POST’,
body: grant_type=password&username=${user}&password=${password}&client_id=api.bacdive.public&scope=openid,
headers: {
‘Content-Type’: ‘application/x-www-form-urlencoded’
}
})

8

Yeah and I would expect a token back, some hex string or something

9

This doesn’t work. I’m not getting anywhere. I’ll try other ways by importing the keycloak.js module…

10

ok, I am happy to do a call too. I would not expect it to take long.