r/datascience•
McHeathen

What do you use for parallelism: R, Python, Scala, or Other?

I'm currently spending some time learning Scala. I already know a bit of Python and a lot of R, and across these languages, the thing I'm most interested in is parallel computation. My normal routine for to implement a lot of parallel jobs on a cluster in R is something like:

cl <- makePSOCKcluster(c('node1', 'node2', 'node_n') # etc
results <- clusterApplyLB(cl, X = objects, function(object) {
    local_cl <- makeForkCluster(detectCores())
    result <- clusterApplyLB(cl, X = lists_in_object, myFunction)
    stopCluster(local_cl)

    return(result)
    })
)

stopCluster(cl)
save(results, file = 'results.RData')

I've heard people talk about how R isn't great with concurrency and parallel code, but to me, this workflow seems really straightforward. I've done something similar in Python with the multiprocessing module, but I have yet to use a Python module which allows for connections to remote machines as easily as R's makePSOCKcluster function.

My question is this - for those of you doing parallel computation in Python and Scala across lots of machines, how are you doing it? Is there a similarly straightforward or even more straightforward implementation of this style of code in either Python or Scala?

Thanks!

Posted Feb 22, 2016, 4:01 AM UTC

411
Loading comments…