RuntimeException: scala.Some is not a valid external type for schema of string

See original GitHub issue

I’m trying to read an Excel file in Java using the spark-excel library. It appears that the returned data is wrapped in scala.Some type.

Here is how I invoke the library:

        Dataset<Row> extractedExcel = sparkSession
                .read()
                .format("com.crealytics.spark.excel")
                .option("inferSchema", "false")
                .option("treatEmptyValuesAsNulls", "false")
                .option("useHeader", "true")
                // .schema(schema)
                .load(absPath);

Then, if I make any call on the resulting dataset, say extractedExcel.show(); or try to write the dataset out, I run into an exception:

Caused by: java.lang.RuntimeException: scala.Some is not a valid external type for schema of string
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.StaticInvoke_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:324)

The error message suggests that the returned dataset wraps values in a scala.Some. I don’t see how can the values be unwrapped and turned into plain strings.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:1
  • Comments:13

github_iconTop GitHub Comments

4reactions
EnverOsmanovcommented, May 18, 2020

I just did a release, should be available very soon (maybe already).

3reactions
xvinoshcommented, Aug 14, 2020

@EnverOsmanov Thanks for replying, the issue is solved after I ran my code with latest version of spark-excel(0.13.5)

Read more comments on GitHub >

github_iconTop Results From Across the Web

Spark 3.0 scala.None$ is not a valid external type for schema ...
It worked by setting elasticsearch-hadoop property es.field.read.empty.as.null = no .option("es.field.read.empty.as.null", "no").
Read more >
Spark job fails after upgrade from DSE 5.0 with error "is not a ...
RuntimeException : Error while encoding: java.lang.RuntimeException: scala.None$ is not a valid external type for schema of date ...
Read more >
UTF8String is not a valid external type for schema of string
I'm having this error using the 3.0.0-alpha2 with spark 3.0.0-preview2, below is a minimal example to run in the spark-shell and the stacktrace, ......
Read more >
Unable to read data from Elasticsearch with spark in Databricks.
Caused by: RuntimeException: scala.collection.convert.Wrappers$JListWrapper is not a valid external type for schema of string · _id · _index ...
Read more >
tFileInputParquet randomly changing the order of the schema ...
RuntimeException : Error while encoding: java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for schema of string.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found