Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”

Are you trying to submit a Spark job over YARN on an HDP Hadoop cluster and encounter these kinds of errors? (below) If so just add the following 2 lines to your [spark-home]/conf/spark-defaults.conf file:   ERRORS You will see the errors below, stem from the root issue that occurs on an Spark Executor node where […]

Read More Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”

Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”

Recently have been playing with HortonWorks HDP 2.2. Was starting to configure some oozie workflows and when submitting the job the first step’s Hive script failed with this error and stack. To fix this, SSH into your HDP instance VM and edit: /etc/hadoop/conf/core-site.xml and change the following config to add “localhost”. Save and restart the relevant services […]

Read More Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”