How is memory modeled in projects like Apache Spark or Druid?

In the past couple of months I've worked with both Apache Spark and Druid for some projects. As I've gone through the process of learning how to use these tools, I've spent some time reading through their documentation and peaking into their inner workings (the under-the-hood code rather than the API level code).

In both projects, they mention both on-heap memory and -- to use a project generic term -- "worker-memory" (the amount of memory a particular worker has when performing a job/task). I'm interested in learning more about the latter.

In these projects, did the developers create classes to model contiguous memory itself (perhaps using Arrays)? If not, I can't quite figure out how they'd go about allowing the "worker-memory" setting to be defined in the first place.

If anyone has any insight into the particulars revolving around this, I'd be interested in learning about it.

Posted Oct 22, 2016, 2:01 PM UTC

542
vekst42•

I'm not totally clear on the details myself. I know spark at least does a lot of low level memory management you would never normally see with stuff in the jvm. These slides might provide a starting point for looking into it in more detail (https://deanwampler.github.io/polyglotprogramming/papers/ScalaJVMBigData-SparkLessons.pdf).

2
dmuxOP•

Thanks for the reply. I've dug a little deeper into the Spark code, and I believe I've found the packages concerned with memory management and Memory Consumers.

It does appear they're using byte arrays for task memory.

2
-pv1-•

I've been working with druid for the past year, i'm not that familiar with the term "worker memory". Are you referring to the memory used by workers running real-time or batch ingestion tasks? But I know that druid folks are contemplating using off the heap memory for several use cases instead of ByteBuffers. There is a proposal being created ot instead use "native memory" from here -https://github.com/DataSketches/sketches-core/tree/master/memory/src/main/java/com/yahoo/memory

1