spark中broadcast oom 解决方案

1. 错误信息

aused by: java.util.concurrent.ExecutionException: org.apache.spark.util.SparkFatalException

 at java.util.concurrent.FutureTask.report(FutureTask.java:122)

 at java.util.concurrent.FutureTask.get(FutureTask.java:206)

 at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$doExecuteBroadcast$2.apply$mcVI$sp(BroadcastExchangeExec.scala:152)

 at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$doExecuteBroadcast$2.apply(BroadcastExchangeExec.scala:1502

  2. 解决方案

   方案1） 设置一下spark.sql.autoBroadcastJoinThreshold=-1

   方案2） 调低 spark.sql.autoBroadcastJoinThreshold=10485760

标签：BroadcastExchangeExec,util,java,oom,broadcast,sql,FutureTask,spark
来源： https://blog.csdn.net/hzp666/article/details/123060867