What do you use for parallelism: R, Python, Scala, or Other?
I'm currently spending some time learning Scala. I already know a bit of Python and a lot of R, and across these languages, the thing I'm most interested in is parallel computation. My normal routine for to implement a lot of parallel jobs on a cluster in R is something like:
cl <- makePSOCKcluster(c('node1', 'node2', 'node_n') # etc
results <- clusterApplyLB(cl, X = objects, function(object) {
local_cl <- makeForkCluster(detectCores())
result <- clusterApplyLB(cl, X = lists_in_object, myFunction)
stopCluster(local_cl)
return(result)
})
)
stopCluster(cl)
save(results, file = 'results.RData')
I've heard people talk about how R isn't great with concurrency and parallel code, but to me, this workflow seems really straightforward. I've done something similar in Python with the multiprocessing module, but I have yet to use a Python module which allows for connections to remote machines as easily as R's makePSOCKcluster function.
My question is this - for those of you doing parallel computation in Python and Scala across lots of machines, how are you doing it? Is there a similarly straightforward or even more straightforward implementation of this style of code in either Python or Scala?
Thanks!
Posted Feb 22, 2016, 4:01 AM UTC