Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”

Are you trying to submit a Spark job over YARN on an HDP Hadoop cluster and encounter these kinds of errors? (below) If so just add the following 2 lines to your [spark-home]/conf/spark-defaults.conf file:   ERRORS You will see the errors below, stem from the root issue that occurs on an Spark Executor node where […]

Read More Fix: HDP, YARN, Spark “check your cluster UI to ensure that workers are registered and have sufficient resources”

Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”

Recently have been playing with HortonWorks HDP 2.2. Was starting to configure some oozie workflows and when submitting the job the¬†first step’s Hive script failed with this error and stack. To fix this, SSH into your HDP instance VM and edit:¬†/etc/hadoop/conf/core-site.xml and change the following config to add “localhost”. Save and restart the relevant services […]

Read More Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1”