[question] How does the given algorithm incorporate machine learning techniques??? (no code included, but there is a description)
For a report (the final, big report we have in our "big data" class at my uni), we had to use Hadoop with a machine learning algorithm of our choice (random forest, L-SVM, etc). He had provided code for what he calls "random forest with hadoop," and we were essentially supposed to simply modify this code to run with our dataset, and compare it to a version of the same code that doesn't incorporate Hadoop.
That being said, he gave us some R code with a mapreduce call in it and said basically, "here is random forest with Hadoop, compare with a Hadoop-less version." Since the class is moreso about Hadoop and HDFS than machine learning, I can't exactly tell whether this code actually does what random forest does. Without posting the code itself (afraid of copyright issues since it's in his textbook), I want to describe what some of this code does.
Random Forest (mapper): The mapper seems to be creating two bootstrap samples, combining them with rbind(), assigning the bootstrap sample index to the key and the combined samples themselves as the val.
RF (reducer): The reducer is where the code takes one feature--specified by the coder--and compares each entry in the feature against the feature mean, assigning a class to that feature (either a 0 or 1 for his example), which I have modified to a feature in question for my dataset, and introduced one more possible class. The original feature values are then replaced with their assigned classes, and... That's it.
The rest of the code: The rest of the code is mine, and is only for the purpose of debugging: it takes the output, compares those assigned classes against the actual classes, and calculates "information gain."
The bootstrap samples are never used anywhere else in his code, no bagging is apparent, only one feature is used, and the metric by which that feature is classified is hard-coded and, seemingly, specific to that particular example dataset. No "trees" are made, and there is certainly no vote tally function for said trees' outputs.
The code that performs this function isn't supposed to be some incomplete "fill-in-the-blanks" type of situation; rather, it's straight out of a textbook he wrote on the subject that he is using as an example of code for random forest, to help describe how it works. I'm at a loss--I don't understand how this at all accomplishes what random forest is supposed to accomplish, seeing as there is no "training, validation, and testing"--the algorithm simply uses mapreduce to compare observation attributes against the feature mean.
His decision tree example seems to be ridden with manual coder-specified decisions as well, such as a mapper with comments like
"#split needed on left side"
(code)
"#split not needed on the right side"
(code)
Shouldn't these algorithms be doing that work for me? (Thereby making the algorithms themselves much more complex?) When I use a package like Rpart, it generates a decision tree from my data, with little else to go on that I specify!
Once again, sorry for my ineptitude, I am not a machine learning programmer, though I would like to learn how these approaches work, instead of scratching my head at how a couple of manual comparisons qualify as ML.
Tl;dr: My professor gave us code for "machine learning" (random forest, decision tree) with hadoop that is suspiciously riddled with manually made decisions about the data/no apparent tree/forest structure (though I am not a machine learning programmer, so my scrutiny is nil).
Edit: tldr, formatting
Edit 2: I realized that the bootstrapping code was actually making multiple bootstrap samples, but the code just isn't going any further with them.
Posted Dec 2, 2015, 10:56 PM UTC · edited