Spark bug: java.lang.IllegalArgumentException: Illegal pattern component: XXX solution

1. Features:

1. This problem often occurs when spark2.1.x is upgraded to spark2.2.x. For example, when building a spark environment through maven, rely on maven for version upgrades.

2. This problem occurs when spark.read.json or csv is called.

2. Cause

When maven is upgraded, the complete dependency package is not automatically loaded. jsonAPI has special requirements for timeStampFormat, and the default is

The format of `yyyy-MM-dd'T'HH:mm:ss.SSSXXX` cannot be recognized by the scala-lang package.

* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the string that
   * indicates a timestamp format. Custom date formats follow the formats at
   * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>

Three. Solution

Method 1: Modify the option option to modify the default timeStampFormat

spark.read.json
.option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
.load

Method 2: Manually increase the dependency of the lang package, add in the pom file:

<dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-lang3</artifactId>
      <version>3.5</version>
</dependency>