spark-submit:java.lang.NoSuchMethodError:org.apache.spark.SparkContext $。$ lessinit $ greater $ default $ 6()Lscala / collection / Map;

我正在Spark上运行一个简单的Scala示例。在这一行:

val sc = new SparkContext("spark://login.hpc-lan.recolic.org:7077","Recolic+Hust BigData Lab")

火花抛出:

root@ip-172-31-46-127 ~/h/hust-bigdata-lab# spark-submit --master spark://login.hpc-lan.recolic.org:7077 --class net.recolic.hust.bigdata.labx.Main  target/hust-bigdata-labx.jar 
19/11/07 07:37:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map;
    at net.recolic.hust.bigdata.labx.Main$.main(Main.scala:10)
    at net.recolic.hust.bigdata.labx.Main.main(Main.scala)
    at sun.reflect.NativeMethodaccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodaccessorImpl.invoke(NativeMethodaccessorImpl.java:62)
    at sun.reflect.DelegatingMethodaccessorImpl.invoke(DelegatingMethodaccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runmain(SparkSubmit.scala:845)
    at org.apache.spark.deploy.SparkSubmit.doRunmain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我的pom.xml是pasted here。我试图降级到Scala 1.10,试图构建一个包含所有依赖项的胖子,尝试使用sbt,但是它们都不能解决问题。

您能提出建议吗?

-更新

我的火花是2.4.4,使用scala 2.11.12。因此,我modified pom.xml使用spark sdk 2.4.4。但是,仍然是相同的错误。

CCup_LoveTodo 回答:spark-submit:java.lang.NoSuchMethodError:org.apache.spark.SparkContext $。$ lessinit $ greater $ default $ 6()Lscala / collection / Map;

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/3146714.html

大家都在问