我有一个 scala spark 项目由于某些依赖 hell 而失败。这是我的 build.sbt:
scalaVersion := "2.13.3"
val SPARK_VERSION = "3.2.0"
libraryDependencies ++= Seq(
"com.typesafe" % "config" % "1.3.1",
"com.github.pathikrit" %% "better-files" % "3.9.1",
"org.apache.commons" % "commons-compress" % "1.14",
"commons-io" % "commons-io" % "2.6",
"com.typesafe.scala-logging" %% "scala-logging" % "3.9.4",
"ch.qos.logback" % "logback-classic" % "1.2.3" exclude ("org.slf4j", "*"),
"org.plotly-scala" %% "plotly-render" % "0.8.1",
"org.apache.spark" %% "spark-sql" % SPARK_VERSION,
"org.apache.spark" %% "spark-mllib" % SPARK_VERSION,
// Test dependencies
"org.scalatest" %% "scalatest" % "3.2.10" % Test,
"com.amazon.deequ" % "deequ" % "2.0.0-spark-3.1" % Test,
"org.awaitility" % "awaitility" % "3.0.0" % Test,
"org.apache.spark" %% "spark-core" % SPARK_VERSION % Test,
"org.apache.spark" %% "spark-sql" % SPARK_VERSION % Test
这是构建失败:
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] Total time: 5 s, completed Dec 19, 2021, 5:14:33 PM
[info] shutting down sbt server
这是因为我使用的是 Scala 2.13 吗?
最佳答案
我不得不做不可避免的事情并将其添加到我的 build.sbt 中:
ThisBuild / useCoursier := false
https://stackoverflow.com/questions/70413201/