install kylin(#2 install)

docker상에서 kylin 설치 그리고 superset으로 데이터 확인

1. 환경준비

    centos 7.6

    docker

2. Install and Execute

## make file & folder

[root@kylin kylin-docker]# docker build -t sungminyun/kylin-docker:152 .
Sending build context to Docker daemon 234.5 kB
Step 1/39 : FROM ubuntu:16.04
Trying to pull repository docker.io/library/ubuntu ...
16.04: Pulling from docker.io/library/ubuntu
f7277927d38a: Pull complete
8d3eac894db4: Pull complete
edf72af6d627: Pull complete
3e4f86211d23: Pull complete
Digest: sha256:97b54e5692c27072234ff958a7442dde4266af21e7b688e7fca5dc5acc8ed7d9
Status: Downloaded newer image for docker.io/ubuntu:16.04
 ---> 5e13f8dd4c1a
Step 2/39 : MAINTAINER sungminyun
 ---> Running in a59282f3b3f5
 ---> af92d9f062e1
Removing intermediate container a59282f3b3f5
Step 3/39 : WORKDIR /tmp
 ---> 47074a3af534
Removing intermediate container e89b332271c8

Step 4/39 : RUN set -x     && apt-get update && apt-get install -y wget vim telnet ntp     && update-rc.d ntp defaults
 ---> Running in 3c4dc3ae4b80

+ apt-get update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [921 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [576 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6119 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1304 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [983 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7942 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8807 B]
Fetched 16.0 MB in 8s (1935 kB/s)
Reading package lists...
+ apt-get install -y wget vim telnet ntp
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  ca-certificates file ifupdown iproute2 isc-dhcp-client isc-dhcp-common
  libatm1 libbsd0 libdns-export162 libedit2 libexpat1 libgdbm3 libgpm2
  libidn11 libisc-export160 libmagic1 libmnl0 libmpdec2 libopts25 libperl5.22
  libpython3.5 libpython3.5-minimal libpython3.5-stdlib libsqlite3-0
  libssl1.0.0 libxtables11 mime-support netbase openssl perl perl-modules-5.22
  rename vim-common vim-runtime
Suggested packages:
  ppp rdnssd iproute2-doc resolvconf avahi-autoipd isc-dhcp-client-ddns
  apparmor gpm ntp-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make ctags vim-doc vim-scripts vim-gnome-py2
  | vim-gtk-py2 | vim-gtk3-py2 | vim-athena-py2 | vim-nox-py2
The following NEW packages will be installed:
  ca-certificates file ifupdown iproute2 isc-dhcp-client isc-dhcp-common
  libatm1 libbsd0 libdns-export162 libedit2 libexpat1 libgdbm3 libgpm2
  libidn11 libisc-export160 libmagic1 libmnl0 libmpdec2 libopts25 libperl5.22
  libpython3.5 libpython3.5-minimal libpython3.5-stdlib libsqlite3-0
  libssl1.0.0 libxtables11 mime-support netbase ntp openssl perl
  perl-modules-5.22 rename telnet vim vim-common vim-runtime wget
0 upgraded, 38 newly installed, 0 to remove and 1 not upgraded.
Need to get 22.1 MB of archives.
After this operation, 108 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libatm1 amd64 1:2.5.1-1.5 [24.2 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmnl0 amd64 1.0.3-5 [12.0 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 libgdbm3 amd64 1.8.3-13.1 [16.9 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 libgpm2 amd64 1.20.4-6.1 [16.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 netbase all 5.3 [12.9 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 libbsd0 amd64 0.8.2-1 [41.7 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 libedit2 amd64 3.1-20150325-1ubuntu2 [76.5 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 libopts25 amd64 1:5.18.7-3 [57.8 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libssl1.0.0 amd64 1.0.2g-1ubuntu4.15 [1084 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ntp amd64 1:4.2.8p4+dfsg-3ubuntu5.9 [519 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 perl-modules-5.22 all 5.22.1-9ubuntu0.6 [2629 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libperl5.22 amd64 5.22.1-9ubuntu0.6 [3405 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 perl amd64 5.22.1-9ubuntu0.6 [237 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagic1 amd64 1:5.25-2ubuntu1.2 [216 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 file amd64 1:5.25-2ubuntu1.2 [21.2 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 iproute2 amd64 4.3.0-1ubuntu3.16.04.5 [523 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ifupdown amd64 0.8.10ubuntu1.4 [54.9 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libisc-export160 amd64 1:9.10.3.dfsg.P4-8ubuntu1.14 [153 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libdns-export162 amd64 1:9.10.3.dfsg.P4-8ubuntu1.14 [667 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 isc-dhcp-client amd64 4.3.3-5ubuntu12.10 [224 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 isc-dhcp-common amd64 4.3.3-5ubuntu12.10 [105 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1 amd64 2.1.0-7ubuntu0.16.04.4 [71.4 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libidn11 amd64 1.32-3ubuntu1.2 [46.5 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpdec2 amd64 2.4.2-1 [82.6 kB]
Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-minimal amd64 3.5.2-2ubuntu0~16.04.5 [524 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial/main amd64 mime-support all 3.59ubuntu1 [31.0 kB]
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libsqlite3-0 amd64 3.11.0-1ubuntu1.2 [397 kB]
Get:28 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-stdlib amd64 3.5.2-2ubuntu0~16.04.5 [2134 kB]
Get:29 http://archive.ubuntu.com/ubuntu xenial/main amd64 libxtables11 amd64 1.6.0-2ubuntu3 [27.2 kB]
Get:30 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 vim-common amd64 2:7.4.1689-3ubuntu1.3 [103 kB]
Get:31 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 openssl amd64 1.0.2g-1ubuntu4.15 [492 kB]
Get:32 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ca-certificates all 20170717~16.04.2 [167 kB]
Get:33 http://archive.ubuntu.com/ubuntu xenial/main amd64 telnet amd64 0.17-40 [63.5 kB]
Get:34 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 wget amd64 1.17.1-1ubuntu1.5 [299 kB]
Get:35 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5 amd64 3.5.2-2ubuntu0~16.04.5 [1360 kB]
Get:36 http://archive.ubuntu.com/ubuntu xenial/main amd64 rename all 0.20-4 [12.0 kB]
Get:37 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 vim-runtime all 2:7.4.1689-3ubuntu1.3 [5179 kB]
Get:38 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 vim amd64 2:7.4.1689-3ubuntu1.3 [1036 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 22.1 MB in 7s (2858 kB/s)
Selecting previously unselected package libatm1:amd64.
(Reading database ... 4777 files and directories currently installed.)

# Installing Hadoop

Step 5/39 : ARG MIRROR=mirror.bit.edu.cn
 ---> Running in 59d78c067946
 ---> dd42e139eac9
Removing intermediate container 59d78c067946
Step 6/39 : ARG HADOOP_VERSION=2.7.7
 ---> Running in d33684806cea
 ---> 1baf2cedc0b6
Removing intermediate container d33684806cea
Step 7/39 : RUN set -x     && wget -q http://${MIRROR}/apache/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz     && tar -xzvf hadoop-${HADOOP_VERSION}.tar.gz -C /usr/local/     && mv /usr/local/hadoop-${HADOOP_VERSION} /usr/local/hadoop
 ---> Running in 339d3d183e69

+ wget -q http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
+ tar -xzvf hadoop-2.7.7.tar.gz -C /usr/local/
hadoop-2.7.7/
hadoop-2.7.7/bin/
hadoop-2.7.7/bin/hadoop.cmd
hadoop-2.7.7/bin/rcc

+ mv /usr/local/hadoop-2.7.7 /usr/local/hadoop
 ---> 4adeff170230
Removing intermediate container 339d3d183e69
Step 8/39 : ENV HADOOP_HOME /usr/local/hadoop
 ---> Running in 7923d241a2d8
 ---> c554d088d4ce
Removing intermediate container 7923d241a2d8
Step 9/39 : ENV HADOOP_CONF_DIR $HADOOP_HOME/etc/hadoop
 ---> Running in c44351873dee
 ---> e07102e3d48d
Removing intermediate container c44351873dee
Step 10/39 : ENV YARN_CONF_DIR $HADOOP_HOME/etc/hadoop
 ---> Running in e3e7e1171f95
 ---> 8b94c89c3733
Removing intermediate container e3e7e1171f95

# Installing Spark

Step 11/39 : ARG SPARK_VERSION=2.4.3
 ---> Running in 9128c3e6e94d
 ---> 3e70186b6c61
Removing intermediate container 9128c3e6e94d
Step 12/39 : RUN set -x     && wget -q http://${MIRROR}/apache/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-without-hadoop.tgz     && tar -xzvf spark-${SPARK_VERSION}-bin-without-hadoop.tgz -C /usr/local/     && mv /usr/local/spark-${SPARK_VERSION}-bin-without-hadoop /usr/local/spark
 ---> Running in 80b11d3287aa

+ wget -q http://mirror.bit.edu.cn/apache/spark/spark-2.4.3/spark-2.4.3-bin-without-hadoop.tgz
+ tar -xzvf spark-2.4.3-bin-without-hadoop.tgz -C /usr/local/
spark-2.4.3-bin-without-hadoop/
spark-2.4.3-bin-without-hadoop/python/
spark-2.4.3-bin-without-hadoop/python/setup.cfg

+ mv /usr/local/spark-2.4.3-bin-without-hadoop /usr/local/spark
 ---> 4469da07513e
Removing intermediate container 80b11d3287aa
Step 13/39 : ENV SPARK_HOME /usr/local/spark
 ---> Running in bc6b855e9b1d
 ---> 5157f6566d0e
Removing intermediate container bc6b855e9b1d
Step 14/39 : ENV LD_LIBRARY_PATH $HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH
 ---> Running in ea25bdd42739
 ---> 686be14e0245
Removing intermediate container ea25bdd42739

# Installing Hive

Step 15/39 : ARG HIVE_VERSION=2.3.5
 ---> Running in e91da0674826
 ---> ab3357f0bc52
Removing intermediate container e91da0674826
Step 16/39 : RUN set -x     && wget -q http://${MIRROR}/apache/hive/hive-${HIVE_VERSION}/apache-hive-${HIVE_VERSION}-bin.tar.gz     && tar -xzvf apache-hive-${HIVE_VERSION}-bin.tar.gz -C /usr/local/     && mv /usr/local/apache-hive-${HIVE_VERSION}-bin /usr/local/hive
 ---> Running in 05411d47616c

+ wget -q http://mirror.bit.edu.cn/apache/hive/hive-2.3.5/apache-hive-2.3.5-bin.tar.gz
+ tar -xzvf apache-hive-2.3.5-bin.tar.gz -C /usr/local/
apache-hive-2.3.5-bin/conf/hive-log4j2.properties.template
apache-hive-2.3.5-bin/conf/hive-exec-log4j2.properties.template

+ mv /usr/local/apache-hive-2.3.5-bin /usr/local/hive
apache-hive-2.3.5-bin/lib/py/queryplan/__init__.py
apache-hive-2.3.5-bin/lib/py/queryplan/constants.py
apache-hive-2.3.5-bin/lib/py/queryplan/ttypes.py
apache-hive-2.3.5-bin/hcatalog/bin/common.sh

Step 17/39 : ENV HIVE_HOME /usr/local/hive
 ---> Running in 29d7381b077e
 ---> 6e6841a8c0ee
Removing intermediate container 29d7381b077e
Step 18/39 : ENV HCAT_HOME $HIVE_HOME/hcatalog
 ---> Running in a9f229becfa8
 ---> 61a69adfd15e
Removing intermediate container a9f229becfa8
Step 19/39 : ENV HIVE_CONF $HIVE_HOME/conf
 ---> Running in 3293ce19ad51
 ---> 4716bc51b9d4
Removing intermediate container 3293ce19ad51

# Installing HBase

Step 20/39 : ARG HBASE_VERSION=1.4.10
 ---> Running in 44fb2a28be66
 ---> 6d2ee9047f51
Removing intermediate container 44fb2a28be66
Step 21/39 : RUN set -x     && wget -q http://${MIRROR}/apache/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz     && tar -xzvf hbase-${HBASE_VERSION}-bin.tar.gz -C /usr/local/     && mv /usr/local/hbase-${HBASE_VERSION} /usr/local/hbase
 ---> Running in 14dd04714c65

+ wget -q http://mirror.bit.edu.cn/apache/hbase/1.4.10/hbase-1.4.10-bin.tar.gz
+ tar -xzvf hbase-1.4.10-bin.tar.gz -C /usr/local/
hbase-1.4.10/LICENSE.txt
hbase-1.4.10/NOTICE.txt

hbase-1.4.10/lib/hbase-it-1.4.10-tests.jar
hbase-1.4.10/lib/hbase-annotations-1.4.10-tests.jar
+ mv /usr/local/hbase-1.4.10 /usr/local/hbase
 ---> f7abf4959826
Removing intermediate container 14dd04714c65
Step 22/39 : ENV HBASE_HOME /usr/local/hbase
 ---> Running in 74f0b319ae42
 ---> 8b46eddf59a9
Removing intermediate container 74f0b319ae42

# Installing Kylin

Step 23/39 : ARG KYLIN_VERSION=2.6.3
 ---> Running in 8d196a94a37c
 ---> 3ec37054afd8
Removing intermediate container 8d196a94a37c
Step 24/39 : RUN set -x     && wget -q http://${MIRROR}/apache/kylin/apache-kylin-${KYLIN_VERSION}/apache-kylin-${KYLIN_VERSION}-bin-hbase1x.tar.gz     && tar -xzvf apache-kylin-${KYLIN_VERSION}-bin-hbase1x.tar.gz -C /usr/local/     && mv /usr/local/apache-kylin-${KYLIN_VERSION}-bin /usr/local/kylin
 ---> Running in dfebd3e3c2eb

+ wget -q http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.6.3/apache-kylin-2.6.3-bin-hbase1x.tar.gz
+ tar -xzvf apache-kylin-2.6.3-bin-hbase1x.tar.gz -C /usr/local/
apache-kylin-2.6.3-bin-hbase1x/
apache-kylin-2.6.3-bin-hbase1x/commit_SHA1
apache-kylin-2.6.3-bin-hbase1x/bin/

# Set env & excute

[root@kylin kylin-docker]# docker build -t sungminyun/kylin-docker:152 .
Sending build context to Docker daemon 231.9 kB
Step 1/17 : FROM ubuntu:16.04
 ---> 5e13f8dd4c1a
Step 2/17 : MAINTAINER sungminyun
 ---> Using cache
 ---> af92d9f062e1
Step 3/17 : WORKDIR /tmp
 ---> Using cache
 ---> 47074a3af534
Step 4/17 : ENV PATH $PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$KYLIN_HOME/bin
 ---> Running in 0c8c2824a751
 ---> 7f5da0ce925a
Removing intermediate container 0c8c2824a751
Step 5/17 : COPY client-conf/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml
 ---> 50c478482baa
Removing intermediate container 2eff04d1fa97
Step 6/17 : COPY client-conf/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml
 ---> e7e49e31d5e4
Removing intermediate container 80ea8c9da70a
Step 7/17 : COPY client-conf/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml
 ---> 2fc5b5b0fe43
Removing intermediate container f6706acc18e0
Step 8/17 : COPY client-conf/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml
 ---> 1d422616fc57
Removing intermediate container 0eb552ab9bcd
Step 9/17 : COPY client-conf/hbase-site.xml $HBASE_HOME/conf/hbase-site.xml
 ---> 8ea07efcf683
Removing intermediate container ff6303063255
Step 10/17 : COPY client-conf/hdfs-site.xml $HBASE_HOME/conf/hdfs-site.xml
 ---> 2561f67ae18e
Removing intermediate container b4427c8f920f
Step 11/17 : COPY client-conf/hive-site.xml $HIVE_HOME/conf/hive-site.xml
 ---> 076d7fc5bf54
Removing intermediate container bb82358656b1
Step 12/17 : COPY client-conf/mapred-site.xml $HIVE_HOME/conf/mapred-site.xml
 ---> 9bbd61f66bc0
Removing intermediate container 66dbdf136cff
Step 13/17 : RUN rm -rf /tmp/*
 ---> Running in 396958c3f41b

 ---> 0033e431f639
Removing intermediate container 396958c3f41b
Step 14/17 : WORKDIR /root
 ---> 8086ef1ffcc1
Removing intermediate container 9fd95239f1fb
Step 15/17 : EXPOSE 7070
 ---> Running in cf6585fc9bd8
 ---> a0e1c5258d31
Removing intermediate container cf6585fc9bd8
Step 16/17 : VOLUME /usr/local/kylin/logs
 ---> Running in ba927e1d838c
 ---> cf712e2dea40
Removing intermediate container ba927e1d838c
Step 17/17 : ENTRYPOINT sh -c /usr/local/kylin/bin/kylin.sh start; bash
 ---> Running in a2755ea1ef8e
 ---> 741f537ef391
Removing intermediate container a2755ea1ef8e
Successfully built 741f537ef391