Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ml xgboost workload #638

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Add ml xgboost workload #638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
45b71a9
Add XGBoost
xwu99 Jul 15, 2020
b81f847
Clean up code
xwu99 Jul 15, 2020
6210f92
Add num_workers and nthread params and check spark.task.cpus
xwu99 Jul 15, 2020
85851a6
modify README
xwu99 Jul 17, 2020
93b663b
Update README.md
xwu99 Jul 17, 2020
0e923f0
Merge commit 'refs/pull/628/head' of https://github.com/Intel-bigdata…
Jul 30, 2020
31c3b59
update xgboost configuration to use v1.1.0-scala2.12 by default
bobjiang82 Jul 30, 2020
5791a0f
refine XGBoost.scala, use pipeline
bobjiang82 Jul 30, 2020
205a4e2
Merge branch 'master' into addxgboost
bobjiang82 Aug 3, 2020
38fc921
Update hibench.conf
bobjiang82 Aug 7, 2020
a213d75
Update run-sparkbench.md
bobjiang82 Aug 7, 2020
b92c8cc
Update benchmarks.lst
bobjiang82 Aug 14, 2020
7902f59
Update benchmarks_ml.lst
bobjiang82 Aug 14, 2020
b928bb9
Merge pull request #1 from Intel-bigdata/master
bobjiang82 Sep 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion conf/benchmarks.lst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,6 @@ ml.linear
ml.lda
ml.svm
ml.gmm
ml.xgboost

graph.nweight
graph.nweight
2 changes: 1 addition & 1 deletion conf/workloads/ml/xgboost.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ hibench.xgboost.features ${hibench.xgboost.${hibench.scale.pr
hibench.xgboost.partitions ${hibench.default.map.parallelism}

hibench.xgboost.numClasses 2
hibench.xgboost.maxDepth 30
hibench.xgboost.maxDepth 8
hibench.xgboost.maxBins 32
hibench.xgboost.numIterations 20
hibench.xgboost.learningRate 0.1
Expand Down
11 changes: 11 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,17 @@
<name>Scala-tools Maven 2 Repository</name>
<url>https://oss.sonatype.org/content/groups/scala-tools/</url>
</repository>
<repository>
<id>xgboostrepo</id>
<name>XGBoost Maven Repo</name>
<url>https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
Expand Down
4 changes: 2 additions & 2 deletions sparkbench/ml/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,12 @@
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j_${scala.binary.version}</artifactId>
<version>1.0.0</version>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j-spark_${scala.binary.version}</artifactId>
<version>1.0.0</version>
<version>1.1.0</version>
</dependency>
</dependencies>
</project>
12 changes: 6 additions & 6 deletions sparkbench/ml/src/main/scala/com/intel/sparkbench/ml/XGBoost.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,9 @@
package com.intel.hibench.sparkbench.ml

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.tree.GradientBoostedTrees
import org.apache.spark.mllib.tree.configuration.BoostingStrategy
import org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.feature.{LabeledPoint => NewLabeledPoint}
import ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
Expand All @@ -33,7 +31,7 @@ object XGBoost {

case class Params(
numClasses: Int = 2,
maxDepth: Int = 30,
maxDepth: Int = 8,
maxBins: Int = 32,
numIterations: Int = 20,
learningRate: Double = 0.1,
Expand Down Expand Up @@ -93,7 +91,7 @@ object XGBoost {
val mllibRDD: RDD[LabeledPoint] = sc.objectFile(dataPath)
// Convert to ML LabeledPoint and to DataFrame
val mlRDD: RDD[NewLabeledPoint] = mllibRDD.map { p => NewLabeledPoint(p.label, p.features.asML) }
val data = mlRDD.toDF
val data = mlRDD.toDF("label", "features")

// Split the data into training and test sets (30% held out for testing)
val splits = data.randomSplit(Array(0.7, 0.3))
Expand Down Expand Up @@ -123,7 +121,9 @@ object XGBoost {
setFeaturesCol("features").
setLabelCol("label")

val model = xgbClassifier.fit(trainingData)
val pipeline = new Pipeline().setStages(Array(xgbClassifier))

val model = pipeline.fit(trainingData)

// Make predictions.
val predictions = model.transform(testData)
Expand Down
1 change: 1 addition & 0 deletions travis/benchmarks_ml.lst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ ml.linear
ml.lda
ml.svm
ml.gmm
ml.xgboost