using kylin (#1 install hadoop)

kylin 설치 그리고 superset과의 연계를 통해서 BI분석을 수행해보자.

1. 환경준비

    centos 7.6
    docker

    hadoop 3.1.2

2.install hadoop  on centos7

[root@kylin client-conf]# docker pull centos:7
Trying to pull repository docker.io/library/centos ...
7: Pulling from docker.io/library/centos
8ba884070f61: Pull complete
Digest: sha256:a799dd8a2ded4a83484bbae769d97655392b3f86533ceb7dd96bbac929809f3c
Status: Downloaded newer image for docker.io/centos:7

[root@kylin client-conf]# docker run --privileged -d --name namenode centos:7 init

4c865448f160ea24d79a15d687073b170cf9301af37f770a0e810a0ae83b9941

[root@kylin client-conf]#

[root@kylin client-conf]# docker exec -it namenode bash
[root@4c865448f160 /]#
[root@4c865448f160 /]# yum -y install net-tools
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.kakao.com
 * extras: mirror.kakao.com
 * updates: mirror.kakao.com
base                                                                                            | 3.6 kB  00:00:00
extras                                                                                          | 3.4 kB  00:00:00
updates                                                                                         | 3.4 kB  00:00:00
(1/4): extras/7/x86_64/primary_db                                                               | 205 kB  00:00:00
(2/4): base/7/x86_64/group_gz                                                                   | 166 kB  00:00:00
(3/4): updates/7/x86_64/primary_db                                                              | 7.4 MB  00:00:00
(4/4): base/7/x86_64/primary_db                                                                 | 6.0 MB  00:00:48
Resolving Dependencies
--> Running transaction check
---> Package net-tools.x86_64 0:2.0-0.24.20131004git.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================
 Package                   Arch                   Version                                   Repository            Size
=======================================================================================================================
Installing:
 net-tools                 x86_64                 2.0-0.24.20131004git.el7                  base                 306 k

Transaction Summary
=======================================================================================================================
Install  1 Package

Total download size: 306 k
Installed size: 918 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/net-tools-2.0-0.24.20131004git.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for net-tools-2.0-0.24.20131004git.el7.x86_64.rpm is not installed
net-tools-2.0-0.24.20131004git.el7.x86_64.rpm                                                   | 306 kB  00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-6.1810.2.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : net-tools-2.0-0.24.20131004git.el7.x86_64                                                           1/1
  Verifying  : net-tools-2.0-0.24.20131004git.el7.x86_64                                                           1/1

Installed:
  net-tools.x86_64 0:2.0-0.24.20131004git.el7


Complete!

[root@4c865448f160 /]# yum install java-1.8.0-openjdk-devel.x86_64
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.kakao.com
 * extras: mirror.kakao.com
 * updates: mirror.kakao.com
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk-devel.x86_64 1:1.8.0.222.b10-0.el7_6 will be installed
--> Processing Dependency: java-1.8.0-openjdk(x86-64) = 1:1.8.0.222.b10-0.el7_6 for package: 1:java-1.8.0-openjdk-devel-1.8.0.222.b10-0.el7_6.x86_64
--> Processing Dependency: libjvm.so()(64bit) for package: 1:java-1.8.0-openjdk-devel-1.8.0.222.b10-0.el7_6.x86_64
--> Processing Dependency: libjava.so()(64bit) for package: 1:java-1.8.0-openjdk-devel-1.8.0.222.b10-0.el7_6.x86_64
--> Processing Dependency: libX11.so.6()(64bit) for package: 1:java-1.8.0-openjdk-devel-1.8.0.222.b10-0.el7_6.x86_64

--> Running transaction check

Installed:
  java-1.8.0-openjdk-devel.x86_64 1:1.8.0.222.b10-0.el7_6

Dependency Installed:
  alsa-lib.x86_64 0:1.1.6-2.el7                                  atk.x86_64 0:2.28.1-1.el7
  avahi-libs.x86_64 0:0.6.31-19.el7                              cairo.x86_64 0:1.15.12-3.el7
  copy-jdk-configs.noarch 0:3.3-10.el7_5                         cups-libs.x86_64 1:1.6.3-35.el7
  dejavu-fonts-common.noarch 0:2.33-6.el7                        dejavu-sans-fonts.noarch 0:2.33-6.el7
  fontconfig.x86_64 0:2.13.0-4.3.el7                             fontpackages-filesystem.noarch 0:1.44-8.el7
  freetype.x86_64 0:2.8-12.el7_6.1                               fribidi.x86_64 0:1.0.2-1.el7
  gdk-pixbuf2.x86_64 0:2.36.12-3.el7                             giflib.x86_64 0:4.1.6-9.el7
  graphite2.x86_64 0:1.3.10-1.el7_3                              gtk-update-icon-cache.x86_64 0:3.22.30-3.el7
  gtk2.x86_64 0:2.24.31-1.el7                                    harfbuzz.x86_64 0:1.7.5-2.el7
  hicolor-icon-theme.noarch 0:0.12-7.el7                         hwdata.x86_64 0:0.252-9.1.el7
  jasper-libs.x86_64 0:1.900.1-33.el7                            java-1.8.0-openjdk.x86_64 1:1.8.0.222.b10-0.el7_6
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.222.b10-0.el7_6     javapackages-tools.noarch 0:3.4.1-11.el7
  jbigkit-libs.x86_64 0:2.0-11.el7                               libICE.x86_64 0:1.0.9-9.el7
  libSM.x86_64 0:1.2.2-2.el7                                     libX11.x86_64 0:1.6.5-2.el7
  libX11-common.noarch 0:1.6.5-2.el7                             libXau.x86_64 0:1.0.8-2.1.el7
  libXcomposite.x86_64 0:0.4.4-4.1.el7                           libXcursor.x86_64 0:1.1.15-1.el7
  libXdamage.x86_64 0:1.1.4-4.1.el7                              libXext.x86_64 0:1.3.3-3.el7
  libXfixes.x86_64 0:5.0.3-1.el7                                 libXft.x86_64 0:2.3.2-2.el7
  libXi.x86_64 0:1.7.9-1.el7                                     libXinerama.x86_64 0:1.1.3-2.1.el7
  libXrandr.x86_64 0:1.5.1-2.el7                                 libXrender.x86_64 0:0.9.10-1.el7
  libXtst.x86_64 0:1.2.3-1.el7                                   libXxf86vm.x86_64 0:1.1.4-1.el7
  libdrm.x86_64 0:2.4.91-3.el7                                   libfontenc.x86_64 0:1.1.3-3.el7
  libglvnd.x86_64 1:1.0.1-0.8.git5baa1e5.el7                     libglvnd-egl.x86_64 1:1.0.1-0.8.git5baa1e5.el7
  libglvnd-glx.x86_64 1:1.0.1-0.8.git5baa1e5.el7                 libjpeg-turbo.x86_64 0:1.2.90-6.el7
  libpciaccess.x86_64 0:0.14-1.el7                               libpng.x86_64 2:1.5.13-7.el7_2
  libthai.x86_64 0:0.1.14-9.el7                                  libtiff.x86_64 0:4.0.3-27.el7_3
  libwayland-client.x86_64 0:1.15.0-1.el7                        libwayland-server.x86_64 0:1.15.0-1.el7
  libxcb.x86_64 0:1.13-1.el7                                     libxshmfence.x86_64 0:1.2-1.el7
  libxslt.x86_64 0:1.1.28-5.el7                                  lksctp-tools.x86_64 0:1.0.17-2.el7
  mesa-libEGL.x86_64 0:18.0.5-4.el7_6                            mesa-libGL.x86_64 0:18.0.5-4.el7_6
  mesa-libgbm.x86_64 0:18.0.5-4.el7_6                            mesa-libglapi.x86_64 0:18.0.5-4.el7_6
  pango.x86_64 0:1.42.4-2.el7_6                                  pcsc-lite-libs.x86_64 0:1.8.8-8.el7
  pixman.x86_64 0:0.34.0-1.el7                                   python-javapackages.noarch 0:3.4.1-11.el7
  python-lxml.x86_64 0:3.2.1-4.el7                               ttmkfdir.x86_64 0:3.0.9-42.el7

  tzdata-java.noarch 0:2019b-1.el7                               xorg-x11-font-utils.x86_64 1:7.5-21.el7

[root@4c865448f160 /]# wget http://apache.tt.co.kr/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
bash: wget: command not found
[root@4c865448f160 /]# yum install wget
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.kakao.com
 * extras: mirror.kakao.com
 * updates: mirror.kakao.com
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed

--> Finished Dependency Resolution
[root@4c865448f160 /]# wget http://apache.tt.co.kr/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
--2019-08-03 01:23:55--  http://apache.tt.co.kr/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
Resolving apache.tt.co.kr (apache.tt.co.kr)... 211.47.69.77
Connecting to apache.tt.co.kr (apache.tt.co.kr)|211.47.69.77|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 332433589 (317M) [application/x-gzip]
Saving to: 'hadoop-3.1.2.tar.gz'


31% [=======================>                                                      ] 103,729,460 2.09MB/s  eta 84s

[root@4c865448f160 /]# tar zxvf hadoop-3.1.2.tar.gz
hadoop-3.1.2/
hadoop-3.1.2/LICENSE.txt
hadoop-3.1.2/NOTICE.txt

hadoop-3.1.2/README.txt

[root@4c865448f160 hadoop]# pwd
/hadoop-3.1.2/etc/hadoop
[root@4c865448f160 hadoop]# mkdir -p ./dfs/data
[root@4c865448f160 hadoop]# mkdir -p ./dfs/temp

[root@4c865448f160 hadoop]# mkdir -p ./dfs/name
[root@4c865448f160 hadoop-3.1.2]# vi ./etc/hadoop/hdfs-site.xml
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
                <property>
                <name>dfs.namenode.name.dir</name>
                <value>/hadoop-3.1.2/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/hadoop-3.1.2/dfs/data</value>
        </property>
        <property>
                <name>dfs.namenode.http.address</name>
                <value>namenode:50070</value>
        </property>
        <property>
                <name>dfs.secondary.http.address</name>
                <value>namenode:50090</value>
        </property>

</configuration>

[root@4c865448f160 hadoop-3.1.2]# vi ./etc/hadoop/core-site.xml

<configuration>

        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://namenode:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/hadoop-3.1.2/dfs/temp</value>
        </property>

</configuration>

[root@4c865448f160 hadoop-3.1.2]# ./bin/hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

2019-08-04 01:35:21,763 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = namenode/172.17.0.2
STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 3.1.2

[root@4c865448f160 hadoop-3.1.2]# ./sbin/start-all.sh
Starting namenodes on [namenode]
Last login: Sun Aug  4 01:37:23 UTC 2019
namenode: ssh: connect to host namenode port 22: Connection refused
Starting datanodes
Last login: Sun Aug  4 01:38:33 UTC 2019
localhost: ssh: connect to host localhost port 22: Connection refused

[root@4c865448f160 hadoop-3.1.2]# yum install openssh-server openssh-clients openssh-askpass
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.kakao.com
 * extras: mirror.kakao.com
 * updates: mirror.kakao.com
base                                                                                            | 3.6 kB  00:00:00
extras                                                                                          | 3.4 kB  00:00:00
updates                                                                                         | 3.4 kB  00:00:00
Resolving Dependencies
--> Running transaction check

---> Package openssh-askpass.x86_64 0:7.4p1-16.el7 will be installed

[root@4c865448f160 hadoop-3.1.2]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:L8wDHi/PxqpP8sRWYmY7un512vlwxgZyGj1qERZhxBA root@4c865448f160
The key's randomart image is:
+---[RSA 2048]----+
|      E==.       |
|       ...       |
|        o        |
|       . o       |
|      B S =      |
|     * X.O.+     |
|    . XoX+o.=    |
|     B.*+oo=     |
|   .==+oo  ..    |
+----[SHA256]-----+
[root@4c865448f160 hadoop-3.1.2]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@4c865448f160 hadoop-3.1.2]# chmod 0600 ~/.ssh/authorized_keys
[root@4c865448f160 hadoop-3.1.2]# ssh 0.0.0.0
ssh: connect to host 0.0.0.0 port 22: Connection refused
[root@4c865448f160 hadoop-3.1.2]# vi /etc/hosts.allow
[root@4c865448f160 hadoop-3.1.2]# systemctl restart sshd
[root@4c865448f160 hadoop-3.1.2]# ssh 0.0.0.0
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:cExnN7EVPc9wpV/NlzBwXrCWOAROHQjdlZ+JqrFX30w.
ECDSA key fingerprint is MD5:68:a1:00:24:6a:72:29:95:62:94:19:95:44:d8:2e:80.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.

Last login: Sun Aug  4 01:38:36 2019

### starting hadoop
[root@4c865448f160 hadoop-3.1.2]# ./sbin/start-all.sh
Starting namenodes on [namenode]
Last login: Sun Aug  4 01:41:51 UTC 2019
Starting datanodes
Last login: Sun Aug  4 01:42:01 UTC 2019
Starting secondary namenodes [namenode]
Last login: Sun Aug  4 01:42:04 UTC 2019
Starting resourcemanager
Last login: Sun Aug  4 01:42:09 UTC 2019
Starting nodemanagers
Last login: Sun Aug  4 01:42:19 UTC 2019

[root@4c865448f160 hadoop-3.1.2]#

[root@4c865448f160 ~]# jps
7810 DataNode
8470 NodeManager
8342 ResourceManager
8038 SecondaryNameNode
7687 NameNode

9087 Jps

[root@4c865448f160 bin]# ./hdfs dfs -mkdir /hdfstest
[root@4c865448f160 bin]# ./hdfs dfs -ls /
Found 1 items

drwxr-xr-x   - root supergroup          0 2019-08-04 01:49 /hdfstest