Working with large file- Framework or other options?

Community Help
1

The timing of this framework release is eery. I am trying to solve a problem where I have a very large dataset (1.5G .csv or 300MB .zip). I need to dynamically filter this dataset based on 2 or 3 user inputs and then summarizes over US counties, and then plot on a choropleth. Not complicated.

Whenever I load this local file in the notebook, it basically crashes the notebook (which makes sense). Ideally I’d do the filtering outside the browser using R and then pass the summarized data back to Observable, but then I lose interactivity b/c it’s constantly taking inputs from JS/Observable, re-rerunning, etc.

The filtering and aggregation are simple. I’m just counting unique rows that meet the criteria. It’s weather data, so to filter in R and aggregate (using data.table syntax):
dat[wind_speed >10 & relative_humidity >75,][,length(date_iso),by=countyState]

The JS equivalent isn’t complicated so I’ll omit it, but is Framework the only way to work with data this size in a semi-interactive manner? Can JS filter as is reads as file?

2

It might pay off to consider using the parquet format (see Sylvain Lesage: "You should use Parquet for your data. Example #54…" - Mastodon). If your queries are “simple” (a and b, like described above), you could also try a “data cube” strategy where the data is pre-aggregated at a certain (small) precision step; alternatively, splitting the dataset into chunks that you wouldn’t need to load all at the same time.