Read a CSV file composed of 2 tables / nrows / skiprows

Community Help
1

Hello,

I’d like to read this file https://raw.githubusercontent.com/scls19fr/krp_python_telemetry/main/Logdata%20Essay%20mini60%202023-10-31.csv (ideally as 2 Danfo.js dataframes)

With Python Pandas I’m doing

def load_krp_file(fname):
    nrows = 12
    df_head = pd.read_csv(fname, nrows=nrows, names=["key", "value"])
    df_head = df_head.set_index("key")
    df = pd.read_csv(fname, skiprows=nrows)
    units = df.iloc[0, :]
    units.name = "units"
    cols = df.columns
    df = df.drop(0)
    for col in df.columns:
        df[col] = df[col].astype(float)
    df["Lap"] = ((df["Distance"] - df["Distance"].shift()) < 0).astype(int).cumsum()
    #df["Laptime"] = pd.NaT
    df["Starttime"] = np.where((df["Distance"] - df["Distance"].shift()) < 0, df["Time"], np.NaN)
    #df["Starttime"].iloc[0] = 0
    df.loc[df.index[0], "Starttime"] = 0
    df["Starttime"] = df["Starttime"].ffill()
    #df["Starttime"] = df["Starttime"].fillna(method="ffill")
    df["Laptime"] = df["Time"] - df["Starttime"]

    df["Time"] = pd.to_datetime(df["Time"], unit="s")
    #df["Time"] = pd.to_timedelta(df["Time"], unit="s")

    #df["Laptime"] = pd.to_datetime(df["Laptime"], unit="s")
    
    df = df.set_index("Time")
    laps = df["Lap"].unique()
    
    laptimes = df.groupby("Lap")["Laptime"].last()[0:-1]
    
    return df_head, units.to_frame(), df, laps, laptimes

unfortunately I don’t see a nrows option to define the number of rows to read, nor a skiprows option to skip a given number of rows.

What are according your JS experience the best way of reading such a file and to plot in Plotly graphs?

Kind regards

2

Hi @scelles, I’m not familiar with danfo.js, but does this help you get started?

fetch("https://raw.githubusercontent.com/scls19fr/krp_python_telemetry/main/Logdata%20Essay%20mini60%202023-10-31.csv")
.then(response => response.text())
.then(text => {
  // Normalize line breaks and split segments by double line breaks.
  const [metaText, columns, rows] = text.replace(/\r?\n/g, "\n").split(/\n\n/);
  // Parse first meta section as csv, then convert to an object
  const meta = Object.fromEntries(d3.csvParseRows(metaText));
  // Parse the two lines from the headers segment
  const [headers, units] = d3.csvParseRows(columns);
  // Parse the rows segment into an untyped array of arrays.
  const data = d3.csvParseRows(rows);
  return {meta, headers, units, data};
})
3

Thanks @mootari your answer helped me !

But now I don’t know how to create a Danfo.js DataFrame.

I tried creating it from array of array (data) Creating a DataFrame - Danfo.js like so

dfd.DataFrame(parsed["data"], {columns: parsed["headers"]})

but it raises

TypeError: this.loadArrayIntoNdframe is not a function

My notebook is here KRP Telemetry / Sébastien Celles | Observable

4

I haven’t look at it in detail yet, but I want to point out that you’re missing one important cell in your notebook:

import {print} from "@visnup/hello-danfo-js"

The print function that you currently call is window.print which triggers an actual print dialog (and is not available in notebooks).