Upgrade to Apache Spark 3.4.0 due to the following security updates:
In Apache Spark versions prior to 3.4.0, applications using spark-submit can specify a proxy-user to run as, limiting privileges. The application can execute code with the privileges of the submitting user, however, by providing malicious configuration-related classes on the classpath. This affects architectures relying on proxy-user, for example those using Apache Livy to manage submitted applications.
Apache Spark version 3.4.0 or later ensure that spark.submit.proxyUser.allowCustomClasspathInClusterMode is set to its
default of "false", and is not overridden by submitted applications.