Prep data for time line chart

Community Help
1

I am just learning d3 and this is my first post!
I have question about prepping data in an optimal way:
I have a csv data like :

Screen Shot 2021-01-19 at 11.24.05 PM
Screen Shot 2021-01-19 at 11.24.05 PM2076×290 34.7 KB

after parsing it using d3.csv().parse(stringData)
it will output an array of Object each object like
{
“Date”: “2020-08-17T00:00-0500”,
“key1”: “11.34833333333333”,
“key2”: “-1.1275”,
“key3”: “-0.73833333333333”
}

I want to change the structure of of output object to be like all of these info to be in one object to be like

{
“name”: “key1”,
“value”: 11.34833333333333,
“date”: new Date(“2015-05-25T18:51:05.734Z”),
“groupName”: “y1”
},
{
“name”: “key2”,
“value”: -1.1275,
“date”: new Date(“2015-05-25T18:51:05.734Z”),
“groupName”: “y1”
},
{
“name”: “key3”,
“value”: -0.73833333333333,
“date”: new Date(“2015-05-25T18:51:05.734Z”),
“groupName”: “y1”
},

so basically each parsed object is really contains info of 3 obj
I know I can achieve it using map functions in javascript but it became ugly and lots of loops- made it hard for performance. any idea how can I use d3 methods to get what I want ?

1 Like
2

Arquero seems to suit your needs to “tidy” the data: Introducing Arquero / UW Interactive Data Lab / Observable. It’s the JavaScript equivalent of dplyr in R.

1 Like
3

Something like this? Reformatting data / Brett Cooper | Observable
There is a lot of data missing or I don’t get the rules, so I hope this helps.

2 Likes
4

@hellonearthis Thanks for the example - one thing to point out is that the key names I hardcoded them for example as key1,key2,key3 but in the application they are dynamic. and the data is like 3000 so looks like by doing that I will have to go over all over them. I was thinking if there is a method or some sort that can clean this up

@severo I will look at this library and see if it suits me

2 Likes
5

Is there a way to tell which keys are to be extracted, have they a unique quality that they all share, like all starting with key? That would make it easy to generate the hardcode array of key names.
Working out the group name is hard as there is not obvious way to computer it from the csv data set.

6

@hellonearthis
I solved it by add this function to your solution

columns = test.columns.splice(1);
res = columns.flatMap(key => test.map(d => ({name: d.name, value: d[key], name: key, groupName: ‘y1’, date: d.Date})));

it output this
0: Object {name: “key1”, value: 11.34833333333333, groupName: “y1”, date: “2020-08-17T00:00-0500”}
1: Object {name: “key1”, value: 1.34333, groupName: “y1”, date: “2019-08-17T00:00-0501”}
2: Object {name: “key2”, value: -1.1275, groupName: “y1”, date: “2020-08-17T00:00-0500”}
3: Object {name: “key2”, value: -2.1275, groupName: “y1”, date: “2019-08-17T00:00-0501”}
4: Object {name: “key3”, value: -0.73833333333333, groupName: “y1”, date: “2020-08-17T00:00-0500”}
5: Object {name: “key3”, value: -2.73333, groupName: “y1”, date: “2019-08-17T00:00-0501”}

3 Likes
7

@hellonearthis I was really impressed of how cool this platform is - its easy to test things out. thanks once again

2 Likes