hive-server2 CDH5 Cent66


Hive Serverを動かしとく。
入れとくと色々できる。

Microsoft® Hive ODBC Driver
http://www.microsoft.com/en-us/download/details.aspx?id=40886

Using the Microsoft Hive ODBC Driver in a .NET client
http://msdn.microsoft.com/en-us/library/dn749834.aspx

How to create a SQL Server Linked Server to HDInsight HIVE using Microsoft Hive ODBC Driver
http://blogs.msdn.com/b/igorpag/archive/2013/11/12/how-to-create-a-sql-server-linked-server-to-hdinsight-hive-using-microsoft-hive-odbc-driver.aspx

とりあえず新規インストール時から。

yum -y upgrade
vi /etc/hosts
ホスト名をループバックに記述 

yum -y install samba.x86_64

service nmb start
chkconfig nmb on

service iptables stop
chkconfig iptables off

setenforce 0
vi /etc/selinux/config
SELINUX=disabled  該当部に記述 

MetaStore用のPostgreSQL入れとく

rpm -ivh http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/pgdg-centos92-9.2-6.noarch.rpm

yum -y install postgresql92-server.x86_64

passwd postgres

service postgresql-9.2 initdb

vi /var/lib/pgsql/9.2/data/postgresql.conf
listen_addresses = '*'
port = 5432
standard_conforming_strings = off

chkconfig postgresql-9.2 on
service postgresql-9.2 start

Hadoop/Hiveのインストール

mount -t cifs //ipあどれす/shared /mnt -o username=ゆーざ,password=ぱすわーど
rpm -ivh /mnt/jdk-8u25-linux-x64.rpm
umount /mnt

export JAVA_HOME=/usr/java/default
java -version

cd /etc/yum.repos.d
yum -y install wget
wget http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo
rpm --import http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

cd

yum -y install hadoop-conf-pseudo hive

sudo -u hdfs hdfs namenode -format

service hadoop-hdfs-namenode start
service hadoop-hdfs-datanode start
service hadoop-yarn-resourcemanager start
service hadoop-yarn-nodemanager start

分散管理用のzookeeper入れとく

http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_zookeeper_package_install.html

yum -y install zookeeper
yum -y install zookeeper-server

mkdir -p /var/lib/zookeeper
chown -R zookeeper /var/lib/zookeeper/

service zookeeper-server init
service zookeeper-server start

Hive Serverいれて設定
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_hiveserver2_configure.html

yum -y install hive-server2 

vi /etc/hive/conf/hive-site.xml

<property>
  <name>hive.support.concurrency</name>
  <description>Enable Hive's Table Lock Manager Service</description>
  <value>true</value>
</property>
<property>
  <name>hive.zookeeper.quorum</name>
  <description>Zookeeper quorum used by Hive's Table Lock Manager</description>
  <value>localhost</value>
</property>

service hive-server2 restart

接続確認

beeline -u jdbc:hive2://

メタストアを設定

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_hive_metastore_configure.html

yum -y install postgresql-jdbc
ln -s /usr/share/java/postgresql-jdbc.jar /usr/lib/hive/lib/postgresql-jdbc.jar

sudo -u postgres psql

CREATE USER hiveuser WITH PASSWORD 'ぱすわーど';

CREATE DATABASE metastore;
\c metastore;
※ちゃんとHiveバージョンと一致しているか確認
\i /usr/lib/hive/scripts/metastore/upgrade/postgres/hive-schema-0.13.0.postgres.sql

alter role postgres with password 'ぱすわーど';

\pset tuples_only on
\o /tmp/grant-privs
SELECT
 'GRANT SELECT,INSERT,UPDATE,DELETE ON "'  || schemaname || '". "' ||tablename ||'" TO hiveuser ;'
FROM
 pg_tables
WHERE tableowner = CURRENT_USER and schemaname = 'public';
\o
\pset tuples_only off
\i /tmp/grant-privs

\q

pg_hba.confを全部md5にして一応接続確認

vi /var/lib/pgsql/9.2/data/pg_hba.conf

host    all         all         127.0.0.1/32    md5


service postgresql-9.2 restart

psql -h ほすとめい -U hiveuser -d metastore

hive-site.xmlを設定

vi /etc/hive/conf/hive-site.xml

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:postgresql://ほすとめい/metastore</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.postgresql.Driver</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hiveuser</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>ぱすわーど</value>
</property>
 
<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>false</value>
</property>

<property>
  <name>hive.metastore.schema.verification</name>
  <value>true</value>
</property>

<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>
 
<property>
  <name>datanucleus.autoStartMechanism</name> 
  <value>SchemaTable</value>
</property> 

yum -y install hive-metastore

vi /etc/hive/conf/hive-site.xml

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://127.0.0.1:9083</value>
  <description>IP address (or fully-qualified domain name) and port of the metastore host</description>
</property>


sudo -u hdfs hadoop fs -chmod -R 777 /

service hive-server2 restart

最終確認

hive
exit;
netstat -alnA inet

beeline

!connect jdbc:hive2://localhost:10000 hive hive org.apache.hive.jdbc.HiveDriver

カテゴリー: 開発 タグ: , パーマリンク