d3.csv of https://file.csv returns `property undefined` within script, but OK if defined in separate cell ?

Community Help
1

Hi Community.

I continue to try to re-convert notebooks back to HTML. In this process, I discovered an oddity:

It doesn’t work if I try D3.csv to transform a CSV to an array within the script:

   var url_source = d3.csv('https://...')
  
   var input = document.getElementById('exampleInputEmail3')
   input.value = url_source[0].url_orig;

I get the error: TypeError: Cannot read property 'url_orig' of undefined (and I know url_orig is a property of the referenced url_source array… you can see this in the notebook linked at the bottom).

It does, however, work when I define a CSV in the script element:

   var name = d3.csvParse(`email_name,
                   Name1,
                   Name2,
                   Name3`)
  
   var input = document.getElementById('exampleInputEmail1')
   input.value = name[0].email_name;

And it also works if I use D3.csv to transform a CSV to an array, then reference the array:

   var url = external_data
  
   var input = document.getElementById('exampleInputEmail2')
   input.value = url[0].url_orig;

external_data = d3.csv('https:...')

Two questions:

  1. Any insight as to why I can in-line a csv, or reference an array created with d3.csv in another cell, but I can’t d3.csv an externally-hosted CSV in within the same cell / script?
  2. How do I correctly write a ‘combined’ script that both reads in the external array and then runs the function? That is, I in regular HTML, I can’t seem to do either of these two things:

option 1 won’t work in HTML – same as Observable:

<script>
   var url = d3.csv('https://...')
   var input = document.getElementById('exampleInputEmail1')
   input.value = url[0].url_orig;

</script>

option 2 won’t work in HTML - though OK if separate Observable cells

<script>
   var url = d3.csv('https://...')
</script>
<script>  
   var url2 = url
   var input = document.getElementById('exampleInputEmail1')
   input.value = url2[0].url_orig;
</script>

Here’s a notebook reproducing the successes and errors discussed above.

Thanks in advance for your insights!!

2

So I still don’t understand why two similar (identical?) methods produce different results, but to get the form to pre-populate with data from an external array, this works:

{
  var case_data = d3.csv('https://....csv')
.then(function(data) {
  document.getElementById('exampleInput').value = data[0].url_orig;
       });
}
3

Hi Aaron, the methods you sketched above are not equivalent, and it’s important to understand why.

First, d3.csv() returns a Promise, not an array. So the way JS evaluates this:

   var url_source = d3.csv('https://...')
  
   var input = document.getElementById('exampleInputEmail3')
   input.value = url_source[0].url_orig;

is roughly as follows.

  1. url_source is created as a Promise. It will resolve when the request to the URL succeeds (or fails), however, that will take a fairly long time compared to the evaluation of other code, so JS doesn’t wait for it and moves to the next lines:

  2. input is set to the html element with Id exampleInputEmail3, and then

  3. input.value is set to url_source[0].url_orig. At this point, url_source is still an unresolved Promise, so url_source[0] is undefined. Then .url_orig doesn’t exist, so you get the error that you quoted above.

  4. At some point, url_source resolves, but the above steps have already been performed, so this has no effect.

One thing you can do to get that cell to work in Observable is to throw an await in front:

   var url_source = await d3.csv('https://...')
  
   var input = document.getElementById('exampleInputEmail3')
   input.value = url_source[0].url_orig;

That causes the evaluation of the JS to pause until url_source has resolved. I believe this is what happens implicitly in Observable when you separate the request into different cells:

   var url = external_data
  
   var input = document.getElementById('exampleInputEmail2')
   input.value = url[0].url_orig;

external_data = d3.csv('https:...')

Observable waits for external_data to resolve before evaluating the first cell. Note that separating code into different Observable cells is not equivalent to putting code in different script tags!

In any case, this is not a good solution for a page on the web since what this does is to lock the other code as well as the rendering of the page until the request finishes, which will lead to poor performance and a bad user experience.

A much better approach is to make use of the Promise API as you did in your followup post. The then syntax performs the given function automatically when the request resolves without blocking the rest of the code on the page.

Promises are a bit confusing but very important to the way we get data from other sources on the web, so I recommend spending some time with the docs or looking at some other tutorials (there are many out there!).

2 Likes
4

Nice explanation, Bryan, thank you! I didn’t realize it was so involved. :open_mouth:

I will definitely read about promises. At this point, I am still having trouble to recognize and understand underlying concepts, so my approach to problem-solving is very much a ‘brute force’ methods of sifting through web pages and trying different things until eventually I figure something out (or not). I had no idea, for example, that my follow-up post utilized the ‘Promise API’… I just found some other notebook where this pattern was used, tried it, and there it was! [To wit: This was a rather long process of staring by googling “javascript replace value with array object”, which essentially led nowhere… eventually I found the solution just clicking into Observable notebooks…]

So again - I owe you a huge debt of thanks and :clap: kudos for being an all-star teacher :1st_place_medal:

5

No problem, seeing that my posts are useful is reward enough! :+1:

Here’s a tip for future learning. It’s generally a bad idea to use code that you don’t understand at least at the level of the relevant library / API docs (though I’ve certainly broken this rule!). A huge part of learning to program is building up a mental model for what the computer does when given a piece of code, and though it sometimes seems that way, there’s really no magic in JS – if you have the code, trustworthy docs, and (a lot of) patience and focus, you can figure out what happens when you run it.

The D3 docs are generally quite clear and precise (if a bit dense for new learners). In this case, I think chasing down the terms in the documentation for d3.csv might have led you more quickly to the answer. Granted, the text there links to the Fetch Spec which seems to have been written for machines rather than humans, but once you know the magic word “Fetch”, you might find the more readable page on MDN, which mentions promises.

1 Like
6

I ran in the same problem a few days ago, but had already forgotten that d3.csv returned a promise. So thank you for your question. It was timely reminder for me. :o)

7

I highly recommend familiarizing yourself with your browser’s debugger. Anytime you don’t understand why the code is doing what it is, you can step through it line-by-line and inspect values to see where things start to go astray. It’s not always a magic bullet, but it’s a powerful tool for understanding and fixing code. Like here, you could have seen pretty quickly that d3.csv returned a Promise object rather than the expected array.

You can use your debugger within Observable, too. Add a debugger statement to a cell, pop open the debugger, then hit the play button or Shift-Enter. For example:

{
  debugger;
  for (let i = 0; i < 10; ++i) {
    yield i;
  }
}
7 Likes
8

Holy cow! I didn’t know about the debugger statement. The dev tools just became a thousand times easier to use.

2 Likes