Apache Storm 설치하기
프로젝트에서 Apache Storm을 활용할 부분이 생겨서 기능 검증이 필요한 시점이다. AWS에 Apache Storm을 설치한 과정을 정리하고자 한다. EC2의 Instance Type 및 용도는 아래와 같다.
- Zookeeper #1 + Nimbus + Supervisor + Storm UI
- Zookeeper #2 + Supervisor
- Zookeeper #3 + Supervisor
#공통 (Each Node) 각 Instance에 JDK를 설치한다.
$sudo apt-add-repository ppa:webupd8team/java$sudo apt-get update
$sudo apt-get install oracle-java8-installer
#Zookeeper Cluster 설치 (Each Node)아래의 command를 수행한다.
$wget http://apache.mirror.cdnetworks.com/zookeeper/stable/zookeeper-3.4.8.tar.gz$tar xzvf zookeeper-3.4.8.tar.gz$mv zookeeper-3.4.8 zookeeper$cd zookeeper$mkdir data$cd conf
$cp zoo_sample.cfg zoo.cfg
zoo.cfg파일을 수정한다. Zookeeper #1 설정
# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=0.0.0.0:2888:3888server.2=2.2.2.2:2888:3888server.3=3.3.3.3:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
Zookeeper #2 설정
# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=1.1.1.1:2888:3888server.2=0.0.0.0:2888:3888server.3=3.3.3.3:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
Zookeeper #3 설정
# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=1.1.1.1:2888:3888server.2=2.2.2.2:2888:3888server.3=0.0.0.0:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
설정이 완료되었으면 Zookeeper를 실행한다. (각 서버별 수행)
$cd /home/ubuntu/zookeeper/bin
$./zkServer.sh start
#Apache Storm 설치 아래의 명령어를 모든 노드에서 수행한다.
$wget http://apache.mirror.cdnetworks.com/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz
$tar xzvf apache-storm-1.0.1.tar.gz$mv apache-storm-1.0.1 stormStorm configuration을 설정한다.
$vi storm/conf/storm.yaml
각 노드별로 설정이 완료되었으면, 아래의 과정을 수행한다.
Start Nimbus
$nohup ./storm/bin/storm nimbus >> nimbus.log 2>&1 &
Start Supervisor 슈퍼바이저는 각 노드에서 수행한다.
$nohup ./storm/bin/storm supervisor >> supervisor.log 2>&1 &
Start Storm UI 스톰 UI는 Nimbus 노드에서 수행한다. (Worker 노드에서 수행 금지, 접속은 브라우저에서 IP:8080)
$nohup ./storm/bin/storm ui >> ui.log 2>&1 &
Start Logviwer 각 노드에서 수행한다.
$nohup ./storm/bin/storm logviewer > logviewer.log &
Start Example Topology Nimbus 노드에서 수행한다.
$./storm/bin/storm jar ../examples/storm-starter/storm-starter-topologies-0.10.0.jar org.apache.storm.starter.RollingTopWords production-topology remote
Kill Example Topology Nimbus 노드에서 수행한다.
$./storm/bin/storm kill production-topology 1
이제 갖고 놀면 된다.
