Skip to content

Commit 025e2a6

Browse files
authored
chore: Use native_datafusion scan in benchmark scripts (#3460)
1 parent 8724b76 commit 025e2a6

2 files changed

Lines changed: 2 additions & 0 deletions

File tree

dev/benchmarks/comet-tpcds.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ $SPARK_HOME/bin/spark-submit \
4040
--conf spark.executor.extraClassPath=$COMET_JAR \
4141
--conf spark.plugins=org.apache.spark.CometPlugin \
4242
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
43+
--conf spark.comet.scan.impl=native_datafusion \
4344
--conf spark.comet.expression.Cast.allowIncompatible=true \
4445
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
4546
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain \

dev/benchmarks/comet-tpch.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ $SPARK_HOME/bin/spark-submit \
4040
--conf spark.executor.extraClassPath=$COMET_JAR \
4141
--conf spark.plugins=org.apache.spark.CometPlugin \
4242
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
43+
--conf spark.comet.scan.impl=native_datafusion \
4344
--conf spark.comet.exec.replaceSortMergeJoin=true \
4445
--conf spark.comet.expression.Cast.allowIncompatible=true \
4546
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \

0 commit comments

Comments
 (0)