最有效的网站推广方案,电子商务网站建设利益分析,做外贸需要用什么网站,全国一体化在线政务服务平台背景介绍
由于 CDH#xff08;Clouderas Distribution Hadoop #xff09;近几年已经开始收费并限制节点数量和版本升级#xff0c;最近使用开源的 hadoop 搭了一套测试集群#xff0c;其中的权限管理组件用到了Apache Ranger#xff0c;所以记录一下编译打包过程。
组件…背景介绍
由于 CDHClouderas Distribution Hadoop 近几年已经开始收费并限制节点数量和版本升级最近使用开源的 hadoop 搭了一套测试集群其中的权限管理组件用到了Apache Ranger所以记录一下编译打包过程。
组件版本
Hadoop 2.8.5Hive 2.3.7Hbase 1.4.9Ranger 1.2.0
注意由于 Ranger 2.0 版本之后支持的 Hadoop 是 3.x 的版本所以选择了 2.0 之前最新的 Ranger 1.2.0 版本
编译步骤
下载源码
# 克隆源码
git clone https://github.com/apache/ranger.git
# 切换分支
git checkout release-ranger-1.2.0
修改配置
在根目录的 pom.xml 中修改版本 Hadoop 从 2.7.1 修改到 2.8.5
Hive 从 2.3.4 修改到 2.3.7
Hbase 从 1.3.2 修改 1.4.9
各位可以根据需要进行修改当然仅修改这里很简单但不代表编译就兼容
进行编译
mvn clean compile package assembly:assembly install -Dmaven.test.skip -DskipTests -Drat.skiptrue -Dpmd.skiptrue
成功后可见如下信息 打包后的产物就在 target 目录中 这里面有我们部署的 admin 包还有各个支持 ranger 权限控制的插件
错误记录
1Hadoop API 不兼容
https://lists.apache.org/thread/rpk62z0x1drs7oqss1rmy0rrwnp1r5j6 修改 KMSAuthenticationFilter.java 类的
props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND,KMSClientProvider.TOKEN_KIND);
// 调整为Text 转成 toString
props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND,KMSClientProvider.TOKEN_KIND.toString());
2Hbase API 不兼容
a. 修改 ranger-hbase-plugin 子模块
RangerAuthorizationCoprocessor.java 新增实现接口 Overridepublic void preGetClusterStatus(ObserverContextMasterCoprocessorEnvironment observerContext) throws IOException {}Overridepublic void postGetClusterStatus(ObserverContextMasterCoprocessorEnvironment observerContext, ClusterStatus clusterStatus) throws IOException {}Overridepublic void preClearDeadServers(ObserverContextMasterCoprocessorEnvironment observerContext) throws IOException {}Overridepublic void postClearDeadServers(ObserverContextMasterCoprocessorEnvironment observerContext, ListServerName list, ListServerName list1) throws IOException {}Overridepublic void preMoveServers(ObserverContextMasterCoprocessorEnvironment ctx, SetAddress servers, String targetGroup) throws IOException {}Overridepublic void postMoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress servers, String s) throws IOException {}Overridepublic void preMoveServersAndTables(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, SetTableName set1, String s) throws IOException {}Overridepublic void postMoveServersAndTables(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, SetTableName set1, String s) throws IOException {}Overridepublic void preRemoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set) throws IOException {}Overridepublic void postRemoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set) throws IOException {} RangerAuthorizationCoprocessorBase.java 修改参数类型 public void preMoveServers(final ObserverContextMasterCoprocessorEnvironment ctx, SetHostAndPort servers, String targetGroup) throws IOException {}public void postMoveServers(ObserverContextMasterCoprocessorEnvironment ctx, SetHostAndPort servers, String targetGroup) throws IOException {}// 调整参数类型成如下public void preMoveServers(final ObserverContextMasterCoprocessorEnvironment ctx, SetAddress servers, String targetGroup) throws IOException {}public void postMoveServers(ObserverContextMasterCoprocessorEnvironment ctx, SetAddress servers, String targetGroup) throws IOException {} b.修改 ranger-hbase-plugin-shim 子模块
RangerAuthorizationCoprocessor.java 新增实现接口注释掉部分接口
// 新增Overridepublic void preCommitStoreFile(ObserverContextRegionCoprocessorEnvironment observerContext, byte[] bytes, ListPairPath, Path list) throws IOException {}Overridepublic void postCommitStoreFile(ObserverContextRegionCoprocessorEnvironment observerContext, byte[] bytes, Path path, Path path1) throws IOException {}Overridepublic void preMoveServersAndTables(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, SetTableName set1, String s) throws IOException {}Overridepublic void postMoveServersAndTables(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, SetTableName set1, String s) throws IOException {}Overridepublic void preRemoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set) throws IOException {}Overridepublic void postRemoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set) throws IOException {}public void preBalanceRSGroup(final ObserverContextMasterCoprocessorEnvironment ctx, String groupName) throws IOException {}Overridepublic void preGetClusterStatus(ObserverContextMasterCoprocessorEnvironment observerContext) throws IOException {}Overridepublic void postGetClusterStatus(ObserverContextMasterCoprocessorEnvironment observerContext, ClusterStatus clusterStatus) throws IOException {}Overridepublic void preClearDeadServers(ObserverContextMasterCoprocessorEnvironment observerContext) throws IOException {}Overridepublic void postClearDeadServers(ObserverContextMasterCoprocessorEnvironment observerContext, ListServerName list, ListServerName list1) throws IOException {}Overridepublic void preMoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, String s) throws IOException {}Overridepublic void postMoveServers(ObserverContextMasterCoprocessorEnvironment observerContext, SetAddress set, String s) throws IOException {}// 注释掉
// public void preMoveServers(final ObserverContextMasterCoprocessorEnvironment ctx, SetHostAndPort servers, String targetGroup) throws IOException {}
// public void postMoveServers(ObserverContextMasterCoprocessorEnvironment ctx, SetHostAndPort servers, String targetGroup) throws IOException {}
3Python 版本不兼容
[ERROR] around Ant part ...exec failοnerrοrtrue executablepython... 4:48
修复ranger 编译仅仅支持 python 2.x 版本如果我们是 python 3.x 的版本需要切换为 2 的版本如果你安装了 conda 也很简单
# 切换环境
conda activate py27
4PMD 规则校验不通过 这个直接打包时禁用掉 pmd 检查即可 -Dpmd.skiptrue
注意事项
经过上述调整过后我们的编译终于成功了但需要注意的是 API 的兼容仅仅是为了不报错如果真的需要到某个组件的功能时建议移值高版本的特性过来避免某些功能不能正常 work 问题
