Apache Ranger安装配置

Apache Ranger安装配置

下载解压

下载:http://ranger.apache.org/download.html


清理编译

1
2
3
cd /ranger/root/path
mvn clean
mvn -DskipTests=false clean compile package install assembly:assembly

编译完成target文件夹中生成zip和tar.gz文档


服务器安装solr

  1. 下载压缩包解压:solr-7.3.0.zip
  2. ranger源码中:cd security-admin/contrib/solr_for_audit_setup 参考
  3. 编辑install.properties
  4. ./setup.sh
  5. 启动solr:$solr_home/$collection_name/scripts/start_solr.sh
  6. 安装说明:$SOLR_RANGER_HOME/install_notes.txt

备注:
Configure Standalone Solr

Property Sample value Description
JAVA_HOME Provide the path to where you have installed JDK. If it is Hadoop, then you can check /etc/hadoop/conf/hadoop-env.sh for the value of JAVA_HOME. Please note, Solr only support JDK 1.7 and above.
SOLR_USER solr The Linux user used to run Solr
SOLR_INSTALL_FOLDER /opt/solr Location where the Solr is installed. This is the same property used if you want setup.sh to install Solr
SOLR_RANGER_HOME /opt/solr/ranger_audit_server This is the location where Ranger related configuration and schema files will be copied
SOLR_RANGER_PORT 6083 The port you want Solr to listen on.
SOLR_DEPLOYMENT standalone The value standalone will configure solr to run as standalone.
SOLR_RANGER_DATA_FOLDER /opt/solr/ranger_audit_server/data This is the folder where you want the index data to be stored. It is important that the volume for this folder has enough disk space. It is recommended to have at least 1 TB free space for index data. Please take regular backup of this folder.
SOLR_LOG_FOLDER /var/log/solr/ranger_audits The folder where where want Solr logs to go. Make sure the volume for this folder has enough disk space. Please delete old log files on regular basis.
SOLR_MAX_MEM 2g This is the memory assigned for Solr. Make sure you provide adequate memory to the Solr process

Configure SolrCloud

Installing and configuring SolrCloud needs few additional steps. We need to the following:

  1. Add the Ranger Audit config (including schema.xml) to the ZooKeeper
  2. Create the collection in Solr.

First, modify the install.properties for the following properties:

Property Sample value Description
JAVA_HOME Provide the path to where you have installed JDK. If it is Hadoop, then you can check /etc/hadoop/conf/hadoop-env.sh for the value of JAVA_HOME. Please note, Solr only support JDK 1.7 and above.
SOLR_USER solr The Linux user used to run Solr process
SOLR_INSTALL_FOLDER /opt/solr Location where the Solr is installed. This is the same property used if you want setup.sh to install Solr
SOLR_RANGER_HOME /opt/solr/ranger_audit_server This is the location where the scripts and index data will be stored. Please note, in SolrCloud, there is no publicly configurable option to provide the location for storing the index data. So make sure you set the value to the folder where the volume as enough disk space.
SOLR_RANGER_PORT 6083 The port you want Solr to listen on.
SOLR_DEPLOYMENT solrcloud The value solrclould will configure solr to run as SolrCloud.
SOLR_ZK ${zk_host}:2181/ranger_audits It is recommended to give sub-folder to create the Ranger Audit related configurations. In this way, you can use ZooKeeper for other installations of Solr also. You have to give the zookeeper node only for the last node. E.g. zk1:2181,zk2:2182,zk3:2181/ranger_audits
SOLR_SHARDS 1 If you wish to distribute your audit logs, then you can use multiple shards. Make sure the number of shards is equal or less than the number of Solr nodes you will be running.
SOLR_REPLICATION 1 It is highly recommended to set up at least 2 nodes and replicate the indexes. This gives redundancy to index data and also load balancing of Solr queries. Please note, Solr recommends that you should have SOLR_SHARD * SOLR_REPLICATION Solr instances. E.g. if you have 3 shards and 2 replications, then you have 6 Solr instances.
SOLR_LOG_FOLDER /var/log/solr/ranger_audits The folder where where want Solr logs to go. Make sure the volume for this folder has enough disk space. Please delete old log files on regular basis.
SOLR_MAX_MEM 2g This is the memory assigned for Solr. Make sure you provide adequate memory to the Solr process. If you are using very high transaction/request Hadoop environment, then it might better to assign up to 32GB memory for Solr.

For configuring SolrCloud, you need to do the following:

  1. Using ./setup.sh script install and configure Solr for Ranger Audits on all other nodes also (don’t start it yet)
  2. Execute /opt/solr/ranger_audit_server/scripts/add_ranger_audits_conf_to_zk.sh (only once from any node where solr is installed)
  3. Start Solr on all nodes: /opt/solr/ranger_audit_server/scripts/start_solr.sh
  4. Create Ranger Audit collection: /opt/solr/ranger_audit_server/scripts/create_ranger_audits_collection.sh (only once \
    from any node where solr is installed)

Make sure you have enough disk space for index. It is recommended to have at least 1TB free.

After starting Solr for RangerAudit, Solr will listen at ${SOLR_PORT}. E.g Check Solr by accessing http://${SOLR_HOST}:6083 from your browser.


安装RangerAdmin

  1. 解压ranger-1.0.0-admin.tar.gz至/opt/ranger-1.0.0-admin
  2. 编辑install.properties:
    • SQL_CONNECTOR_JAR=/path/to/mysql-connector-java.jar
    • db_root_user=root
    • db_root_password=$password
    • db_host=localhost
    • db_name=ranger
    • db_user=rangeradmin
    • db_password=$password
    • audit_store=solr
    • audit_solr_urls=http://192.168.21.145:6083/solr/ranger_audits
    • unix_user=ranger
    • unix_user_pwd=ranger
    • unix_group=ranger
  3. 更新属性后即需要运行setup.sh
  4. 安装Ranger Admin service可通过以下命令控制:ranger-admin start/stop/restart

安装Ranger UserSync

  1. 解压ranger-1.0.0-usersync.tar.gz/opt/ranger-1.0.0-usersync
  2. 新建日志目录:
    • sudo mkdir -p /var/log/ranger-usersync
    • chown -R ranger:ranger /var/log/ranger-usersync
  3. 编辑install.properties:
    • POLICY_MGR_URL = http://localhost:6080
    • SYNC_SOURCE = unix
    • logdir = /var/log/ranger/usersync
  4. 每次更新完properties后运行setup.sh
  5. 可通过以下命令控制:ranger-usersync-services.sh start/stop/restart
  6. 重要:这里有个bug,导致同步失效。需手动修改后重启服务。修改/etc/ranger/usersync/conf/ranger-ugsync-site.xml中:
    1
    2
    3
    4
    <property>
    <name>ranger.usersync.enabled</name>
    <value>true</value>
    </property>

安装Ranger HDFS插件

  1. namenode节点解压ranger-1.0.0-hdfs-plugin/opt/ranger-1.0.0-hdfs-plugin
  2. 编辑install.properties
    • POLICY_MGR_URL=http://$ranger-server-ip:6080
    • REPOSITORY_NAME=hadoopdev
    • COMPONENT_INSTALL_DIR_NAME=/path/to/$hadoop_home
    • XAAUDIT.SOLR.ENABLE=true
    • XAAUDIT.SOLR.URL=http://$solr-server-ip:6083/solr/ranger_audits
    • CUSTOM_USER=root
    • CUSTOM_GROUP=root
  3. 启用插件:enable-hdfs-plugin.sh
  4. 链接HadoopConf:ln -s /path/to/$hadoop-home/etc/hadoop /path/to/$hadoop-home/conf
  5. 如果echo $HADOOP_HOME为空,则添加环境变量:echo "export HADOOP_HOME=/path/to/$hadoop-home" >> /etc/bashrc
  6. 执行./enable-hdfs-plugin.sh
  7. [待确认]cp /path/to/$hadoop-home/lib/*.jar /path/to/$hadoop-home/share/hadoop/hdfs/lib/
  8. [待确认] 日志权限
    • chown root:hadoop /path/to/$hadoop-home/logs
    • chmod g+w /path/to/$hadoop-home/logs
  9. 重启hdfs:
    • /path/to/$hadoop-home/sbin/stop-all.sh
    • /path/to/$hadoop-home/sbin/start-all.sh
      或者
      依次停止:
    • su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh stop namenode"
    • su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop-­daemon.sh stop secondarynamenode"
    • su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop-­daemon.sh stop datanode"
      依次启动:
    • su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start namenode"
    • su -­l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start secondarynamenode"
    • su ­-l hdfs ­-c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start secondarynamenode"
  10. 网页端注册服务:
    • 登录Ranger Admin
    • Access Manager -> Service Manager -> HDFS -> add service
    • Service Name: hadoopdev [install.properties->REPOSITORY_NAME]
    • Username&Password 系统登录用户名和密码
    • Namenode URL: /path/to/$hadoop-home/etc/hadoop/core-site.xml -> fs.defaultFS

安装Ranger HIVE插件

  1. HIVE节点解压ranger-1.0.0-hive-plugin/opt/ranger-1.0.0-hive-plugin
  2. 编辑install.properties:
    • POLICY_MGR_URL=http://$ranger-server-ip:6080
    • REPOSITORY_NAME=hivedev
    • COMPONENT_INSTALL_DIR_NAME=/path/to/$hive-home
    • XAAUDIT.SOLR.ENABLE=true
    • XAAUDIT.SOLR.URL=http://$solr-server-ip:6083/solr/ranger_audits
    • CUSTOM_USER=hive
    • CUSTOM_GROUP=hive
  3. 执行./enable-hive-plugin.sh
  4. 新建日志文件夹:mkdir /var/log/hive chown -R hive:hive /var/log/hive
  5. 确保配置单元用户有权访问配置文件
    • chown -­R hive:hadoop /usr/local/apache-­hive-­1.2.0-­bin/conf/hiveserver2-­site.xml
    • chown -­R hive:hadoop /usr/local/apache-­hive­-1.2.0­-bin/conf/hive-­log4j.properties
    • chown -­R hive:hadoop /usr/local/apache­-hive­-1.2.0­-bin/conf/hive­-site.xml
  6. 更改配置文件
    hiveserver2site.xml:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    	<configuration>
    <property>
    <name>hive.security.authorization.enabled</name>
    <value>true</value>
    </property>
    <property>
    <name>hive.security.authorization.manager</name>
    <value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>
    </property>
    <property>
    <name>hive.security.authenticator.manager</name>
    <value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
    </property>
    <property>
    <name>hive.conf.restricted.list</name>
    <value>hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager</value>
    </property>
    </configuration>

    hive-site.xml:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    <property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
    </property>
    <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive</value>
    </property>
    <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/tmp/hive_resources</value>
    </property>
    <property>
    <name>hive.scratch.dir.permission</name>
    <value>733</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    </property>
    <property>
    <name>hive.hwi.listen.host</name>
    <value>localhost</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
    </property>
  7. 重启Hive:

    • 停止hive:ps aux | awk '{print $1,$2}' | grep hive | awk '{print $2}' | xargs kill >/dev/null 2>&1
    • 启动MetaStore: nohup hive --­­service metastore > /var/log/hive/hive.out 2> /var/log/hive/hive.log &
    • 启动HiveServer2:hiveserver2 ­> /var/log/hive/hiveServer2.out 2>/var/log/hive/hiveServer2.log &
    • 测试: beeline ­-u "jdbc:hive2://localhost:10000" -­n hive ­-p hive
如果文章对您有帮助,感谢您的赞助支持!