"Modules were resolved with conflicting cross-version suffixes"

I get the following error when I try to import dependencies with the following in my build.sbt:

(update) Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, com.chuusai:shapeless, org.apache.spark:spark-sketch, org.apache.spark:spark-kvstore, org.json4s:json4s-ast, org.apache.spark:spark-catalyst, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.apache.spark:spark-sql, org.scala-lang.modules:scala-xml, org.json4s:json4s-jackson, org.typelevel:macro-compat, com.fasterxml.jackson.module:jackson-module-scala, org.scalanlp:breeze-macros, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.typelevel:machinist, org.json4s:json4s-scalap, org.scala-lang.modules:scala-parser-combinators, org.scalanlp:breeze, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common

scalaVersion := "2.12.12"
val sparkVersion = "3.0.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
libraryDependencies += "com.amazon.deequ" % "deequ" % "1.1.0_spark-3.0-scala-2.12"

However, it does compile properly when I provide the following configurations:

scalaVersion := "2.11.8"
val sparkVersion = "2.4.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
libraryDependencies += "com.amazon.deequ" % "deequ" % "1.1.0_spark-3.0-scala-2.12"

I’m building using sbt version 1.2.8 and I’ve tried importing these dependencies with other versions of sbt, too. Behaviour also persists if you use the sbt plugin or run sbt from the terminal directly.

I’ve also tried deleting invalidating caches and restarting the IDE.

IntelliJ Community 2019.2.4 macOS Mojave 10.14.6

Issue Analytics

State:
Created 3 years ago
Reactions:5
Comments:9 (1 by maintainers)

Top GitHub Comments

9reactions

hedibejaouicommented, Jan 25, 2021

@froocpu I was able to solve this error by excluding some of Deequ’s transitive libraries that are cross-compiled with Scala 2.11:

libraryDependencies += ("com.amazon.deequ" % "deequ" % "1.1.0_spark-3.0-scala-2.12")
        .exclude("org.scalanlp", "breeze_2.11")
        .exclude("com.chuusai", "shapeless_2.11")
        .exclude("org.apache.spark", "spark-core_2.11")
        .exclude("org.apache.spark", "spark-sql_2.11")

1reaction

hedibejaouicommented, Jan 26, 2021

@leopasta Try adding the breeze library explicitly as a test dependency.

Top Results From Across the Web

Why is the error Conflicting cross-version suffixes?

The conflicts appear because: you've specified your Scala version to be 2.11; you've explicitly specified the Scala version (2.10) for the reactivemongo and ......

Scala3 upgrade: modules were resolved with conflicting cross ...

How does one fix this problem? [error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("...."), "name"):

How to fix conflicting cross-version suffixes? - Google Groups

> In the end I still got stuck with versioning issues. I'm running into this same problem but I'm trying to use spray...

sbt project not importing correct scala version : SCL-14214

While trying to use scala 2.11 I keep getting 2.10 or 2.12 imported which is causing the below issue. [IJ]sbt:ubi-journeyData-validation[error] (:update) ...

sbt/sbt - Gitter

[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/Users/rpeters/Projects/some-project/"), "server"): [error] ...