Hi,
I am dabbling with tick data for cryptocurrencies from binance.
I am testing the waters with data from 2 months: 250 million rows x 9 columns.
I am trying multiple variations of code, but the problem is the repeated use of all my RAM, and eventual crashing of R studio. This happens in both duckdb and arrow or mixed pipelines.
My question i a nutshell, I currently have 32 GB RAM. is this generally too little for such data, should i upgrade? or do i need to improve/optimize on my code?
Sample code that aborts R session after 11 minutes:
library(tidyverse)
library(duckdb)
library(arrow)
library(here)
schema_list <- list(
trade_id = int64(),
price = float64(),
qty =