Introduction
Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides central security policy administration across the core enterprise security requirements of authorization, accounting and data protection.
Apache Ranger already extends baseline features for coordinated enforcement across Hadoop workloads from batch, interactive SQL and real–time in Hadoop.
In this tutorial, we cover using Apache Ranger for HDP 2.3 to secure your Hadoop environment. We will walkthrough the following topics:
- Support for Knox authorization and audit
- Command line policies in Hive
- Command line policies in HBase
- REST APIs for policy manager
Prerequisite
The only prerequisite for this tutorial is that you have Hortonworks Sandbox.
Once you have Hortonworks Sandbox, login through SSH:
Starting Knox Service and Demo LDAP Service
From the Ambari console at http://localhost:8080/ (username and password is admin and admin respectively), select Knox from the list of Services on the left-hand side of the page.
Then click on Service Actions from the top right hand side of the page click on Start
![]http://www.dropbox.com/s/jhb30dgey8m30n6/Screenshot%202015-09-08%2010.27.06.png?dl=1)
From the following you can track the start of the Knox service to completion:
Then go back to the Service Actions button on the Knox service and click on Start Demo LDAP
You can track the start of the Demo LDAP Service from the following screen:
Knox access scenarios
Check if Ranger Admin console is running, at http://localhost:6080/from your host machine. The username is admin and the password is admin
If it is not running you can start from the command line using the command
sudo service ranger-admin start
Click on sandbox_knox link under Knox section in the main screen of Ranger Administration Portal
You can review policy details by a clicking on the policy name.
To start testing Knox policies, we would need to turn off the “global knox allow” policy.
Locate Sandbox for Guest policy on the Ranger Admin console and edit the policy
and enable policy named “Sandbox for Guest”
From your local SSHd terminal (not directly on the Sandbox), run this CURL command to access WebHDFS
curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
Go to Ranger Policy Manager tool → Audit screen and check the knox access (denied) being audited.
Now let us try the same CURL command using “guest” user credentials from the terminal
curl -k -u guest:guest-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16393,"group":"hadoop","length":0,"modificationTime":1439987528048,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":4,"fileId":16389,"group":"hdfs","length":0,"modificationTime":1439987809562,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17000,"group":"hdfs","length":0,"modificationTime":1439989173392,"owner":"hdfs","pathSuffix":"demo","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16398,"group":"hdfs","length":0,"modificationTime":1439987529660,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16394,"group":"hdfs","length":0,"modificationTime":1439987528532,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16396,"group":"hadoop","length":0,"modificationTime":1439987538099,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16954,"group":"hdfs","length":0,"modificationTime":1439988741413,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1440165443820,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":8,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1439988397561,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}
We can check the auditing in the Ranger Policy Manager → Audit screen
Ranger plugin for Knox intercepts any request made to Knox and enforces policies which are retrieved from the Ranger Administration Portal
You can configure the Knox policies in Ranger to restrict to a specific service (WebHDFS, WebHCAT etc) and to a specific user or a group and you can even bind user/group to an ip address
Hive grant/revoke permission scenarios
Ranger can support import of grant/revoke policies set through command line or Hue for Hive. Ranger can store these policies centrally along with policies created in the administration portal and enforce it in Hive using its plugin.
As a first step, disable the global access policy for Hive in Ranger Administration Portal
Let us try running a Grant operation using user Hive from the command line. Login into beeline tool using the following command
beeline -u "jdbc:hive2://sandbox.hortonworks.com:10000/default" -n it1 -p it1-d org.apache.hive.jdbc.HiveDriver
Then issue the GRANT command
grant select, update on table xademo.customer_details to user network1;
You should see the following error:
Let us check the audit log in the Ranger Administration Portal → Audit
You can see that access was denied for an admin operation for user it1.
We can create a policy in Ranger for user ‘it1’ to be an admin. Create a new policy from the Ranger Admin console and ensure the configuration matches the illustration below
We can try the beeline command again, once the policy has been saved.
GRANT select, update on table xademo.customer_details to user network1;
If the command goes through successfully, you will see the policy created/updated in Ranger Admin Portal → Policy Manager. It checks if there is an existing relevant policy to update, else it creates a new one.
What happened here?
Ranger plugin intercepts GRANT/REVOKE commands in Hive and creates corresponding policies in Admin portal. The plugin then uses these policies for enforcing Hive authorization (Hiveserver2)
Users can run further GRANT commands to update permissions and REVOKE commands to take away permissions.
HBase grant/revoke permission scenarios
Ranger can support import of grant/revoke policies set through command line in Hbase. Similar to Hive, Ranger can store these policies as part of the Policy Manager and enforce it in Hbase using its plugin.
Before you go further, ensure HBase is running from Ambari – http://127.0.0.1:8080 (username and password are admin).
If it is not go to Service Actions button on top right and Start the service
As a first step, let us try running a Grant operation using user Hbase.
Disable the public access policy “HBase Global Allow” in Ranger Administration Portal – policy manager
Login into HBase shell as ‘it1’ user
su - it1
[it1@sandbox ~]$ hbase shell
Run a grant command to give “Read”, “Write”, “Create” access to user mktg1 in table ‘iemployee’
hbase(main):001:0> grant 'mktg1', 'RWC', 'iemployee'
you should get a Acess Denied as below:
Go to Ranger Administration Portal → Policy Manager and create a new policy to assign “admin” rights to user it1
Save the policy and rerun the HBase command again
hbase(main):006:0> grant 'mktg1', 'RWC', 'iemployee'
0 row(s) in 0.8670 seconds
Check HBase policies in the Ranger Policy Administration portal. The grant permissions were added to an existing policy for table ‘iemployee’ that we created in previous step
You can revoke the same permissions and the permissions will be removed from Ranger admin. Try this in the same HBase shell
hbase(main):007:0> revoke 'mktg1', 'iemployee'
0 row(s) in 0.4330 seconds
You can check the existing policy and see if it has been changed
What happened here?
Ranger plugin intercepts GRANT/REVOKE commands in Hbase and creates corresponding policies in the Admin portal. The plugin then uses these policies for enforcing authorization
Users can run further GRANT commands to update permissions and REVOKE commands to take away permissions.
REST APIs for Policy Administration
Ranger policies administration can be managed through REST APIs. Users can use the APIs to create or update policies, instead of going into the Administration Portal.
####Running REST APIs from command line
From your local command line shell, run this CURL command. This API will create a policy with the name “hadoopdev-testing-policy2” within the HDFS repository “sandbox_hdfs”
curl -i --header "Accept:application/json" -H "Content-Type: application/json" --user admin:admin -X POST http://127.0.0.1:6080/service/public/api/policy -d '{ "policyName":"hadoopdev-testing-policy2","resourceName":"/demo/data/test","description":"Testing policy for /demo/data/test","repositoryName":"sandbox_hdfs","repositoryType":"HDFS","permMapList":[{"userList":["mktg1"],"permList":["Read"]},{"groupList":["IT"],"permList":["Read"]}],"isEnabled":true,"isRecursive":true,"isAuditEnabled":true,"version":"0.1.0","replacePerm":false}'
the policy manager and see the new policy named “hadoopdev-testing-policy2”
Click on the policy and check the permissions that has been created
The policy id is part of the URL of this policy detail pagehttp://127.0.0.1:6080/index.html#!/hdfs/1/policy/26
We can use the policy id to retrieve or change the policy.
Run the below CURL command to get policy details using API
curl -i --user admin:admin -X GET http://127.0.0.1:6080/service/public/api/policy/26
What happened here?
We created a policy and retrieved policy details using REST APIs. Users can now manage their policies using API tools or applications integrated with the Ranger REST APIs
Hopefully, through this whirlwind tour of Ranger, you were introduced to the simplicity and power of Ranger for security administration.
Saptak Sen
If you enjoyed this post, you should check out my book: Starting with Spark.