After SSH’ing into your Sandbox, use su to login as root.
Now let’s get the bits using the command
wget http://public-repo-1.hortonworks.com/HDP-LABS/Projects/spark/1.2.0/spark-1.2.0.2.2.0.0-82-bin-2.6.0.2.2.0.0-2041.tgz
Then uncompress the archive with
tar xvfz spark-1.2.0.2.2.0.0-82-bin-2.6.0.2.2.0.0-2041.tgz
Move the resulting folder to /opt as just spark
mv spark-1.2.0.2.2.0.0-82-bin-2.6.0.2.2.0.0-2041 /opt/spark
also for good measure create a symbolic link from /usr/hdp/current/spark-client into /opt/spark
ln -s /opt/spark /usr/hdp/current/spark-client
Create a bash script called spark.sh in the /etc/profile.d with the following lines
export SPARK_HOME=/usr/hdp/current/spark-client
export YARN_CONF_DIR=/etc/hadoop/conf
export PATH=/usr/hdp/current/spark-client/bin:$PATH
Then run the script with
source /etc/profile.d/spark.sh
Create a file SPARK_HOME/conf/spark-defaults.conf and add the following settings:
spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0–2041
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0–2041
Now before we can test run a sample, use your browser to navigate to http://<hostname>:8080 for the Ambari interface. Use admin and admin as username and password respectively to login.
After you login ensure necessary services like Datanode, Nodemanager, ResourceManager, Namenode, HiveServer2, etc are running. If they are stopped, start them and wait for them to start before the next step.
Now we can test the validity of our installation by running the sample Pi calculator
cd /opt/spark
./bin/spark-submit --verbose --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10
You can check the value of the result by navigating to the URL above. Make sure you replace sandbox.hortonworks.com with the actual DNS name of your Sandbox VM. Also ensure that you have opened the port 8088 either on Azure or your Virtualization software.
Saptak Sen
If you enjoyed this post, you should check out my book: Starting with Spark.