Scala lambda only failing in AWS

Question

Im writing my first scala lambda, and locally everything connects and works fine. However, when I try to test my lambda in AWS, I get the following error.

{
  "errorMessage": "Error loading class FooBar.Main: scala/collection/Seq",
  "errorType": "java.lang.NoClassDefFoundError"
}

From my googling, its seems this is cause I needed to add the scala library to my dependencies, which I did.

name := "FooBar"
version := "0.1"
scalaVersion := "2.12.12"
javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")
lazy val root = (project in file(".")).
settings(
  name := "FooBar",
  version := "1.0",
  scalaVersion := "2.12.12",
  retrieveManaged := true
)
libraryDependencies += "software.amazon.awssdk" % "ec2" % "2.5.60"
libraryDependencies += "com.amazonaws" % "aws-lambda-java-core" % "1.2.0"
libraryDependencies += "com.amazonaws" % "aws-lambda-java-events" % "2.1.0"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-dynamodb" % "1.11.313"
libraryDependencies += "org.scalikejdbc" %% "scalikejdbc" % "3.4.0"
libraryDependencies += "org.apache.phoenix" % "phoenix-core" % "4.14.3-HBase-1.4"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.4.10"
libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.4.10"
libraryDependencies += "io.spray" %%  "spray-json" % "1.3.2"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"
libraryDependencies += "org.scala-lang" % "scala-library" % "2.12.12"

assemblyShadeRules in assembly := Seq(
  ShadeRule.keep("x.**").inAll,
  ShadeRule.keep("FooBar.**").inProject
)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

again, everything works fine locally, just can never execute on AWS. Anyone have an idea?

Answer 1

The sbt-assembly plugins shade rule ShadeRule.keep documentation states

The ShadeRule.keep rule marks all matched classes as "roots". If any keep rules are defined all classes which are not reachable from the roots via dependency analysis are discarded when writing the output jar.

https://github.com/sbt/sbt-assembly#shading

So in this case all the classes of the class path x.* and FooBar.* are retained while creating the fat jar. Rest all other classes including the classes in scala-library are discarded.

To fix this remove all the ShadeRule.keep rules and instead try ShadeRule.zap to selectively discard classes not required.

For example, the following shade rules removes all the HDFS classes from the far jar:

  assemblyShadeRules in assembly := Seq(
    ShadeRule.zap("org.apache.hadoop.hdfs.**").inAll
  )

PS: AWS Lambda had a hard limit of 256MB of code size after unzipping the fat jar.