首页 > TAG信息列表 > anonfun

sqprk集群上使用自定义udf函数，出现无法序列化的错误

在spark集群上，将读取到的csv文件生成的datafream，需要对其中一列进行转化，内置的udf函数已经不能满足需求 所以需要自定义一个udf，但是在使用的时候报错，如下 Exception in thread "main" org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.Clos

java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String

问题 最近遇到一个问题，在用spark读取tensorflow的文件并写入时 //读数据 val df = spark.read.format("tfrecords").load(readPath).toDF() //写数据 df.write.mode(SaveMode.Overwrite).format("tfrecords").option("recordType","Example").save(writePath)

Caused by: java.sql.SQLException: Can‘t call commit when autocommit=true 踩坑

在使用spark将hive中的数据导出到Mysql的时候遇到这个一个错误 Caused by: java.sql.SQLException: Can't call commit when autocommit=true at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934) at com.mysql.jdbc.SQLError.createSQLExcep

启动spark-shell显示不匹配错误

错误信息 [ERROR] Failed to construct terminal; falling back to unsupported java.lang.NumberFormatException: For input string: "0x100" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Inte

启动dr-elephant失败问题

启动一直报错 SLF4J: Found binding in [jar:file:/data/src/dr-elephant-2.1.7/lib/ch.qos.logback.logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data/src/dr-elephant-2.1.7/lib/org.slf4j.slf4j-simple-1.6.6.j

java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark

ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of &

Spark跑在Yarn上出现错误，原因是jdk的版本问题

./bin/spark-shell --master yarn 　　 2019-07-01 12:20:13 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicableSetting default log level to "WARN".To adjust logging leve

异常-User class threw exception: java.lang.IllegalStateException: Cannot call methods on a stopped Spa

1 详细信息 User class threw exception: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:   org.apache.spark.SparkContext.<init>(SparkContext.scala:76) com.wm.bigdata.spark.etl.RentO