머신러닝을 활용한 빅데이터 분석 #1

-
분석 Suicide Rates Overview 1985 to 2016 

Suicide Rates Data를 csv형태로 취득한 후, json변환을 거쳐 메세지큐를 통해 Druid에 저장하고 Machine Learning으로 분석 가공한 후, 시각화처리를 수행합니다.
분석에 필요한 솔루션
http://kafka.apache.org/
https://nifi.apache.org/
http://druid.io/
https://scikit-learn.org/
https://superset.incubator.apache.org/



#1 Start Druid

./bin/supervise -c quickstart/tutorial/conf/tutorial-cluster.conf
**Druid 의 경우, zookeeper의 선행 시작이 필수.

#2 Start Kafka broker



./bin/kafka-server-start.sh config/server.properties

#3 Start Nifi


./bin/nifi.sh start



댓글

댓글 쓰기

이 블로그의 인기 게시물

error You have $NVM_DIR set to "/root/.nvm", but that directory does not exist. Check your profile files and environment.

-
#error [min@localhost ~]$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 13527 100 13527 0 0 17679 0 --:--:-- --:--:-- --:--:-- 17682 You have $NVM_DIR set to "/root/.nvm", but that directory does not exist. Check your profile files and environment. #fix error [min@localhost ~]$ unset NVM_DIR [min@localhost ~]$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 13527 100 13527 0 0 28270 0 --:--:-- --:--:-- --:--:-- 28240 => Downloading nvm from git to '/home/min/.nvm' => Cloning into '/home/min/.nvm'... remote: Enumerating objects: 290, done. remot
자세한 내용 보기

linux mint install xrdp

-
이미지
## install xrdp $ sudo apt install xrdp $ sudo adduser xrdp ssl-cert $ sudo ufw allow from 192.168.2.0/24 to any port 3389 $ sudo ufw reload $ sudo ufw status ## xrdp connection closes immediately? $ sudo vim /etc/xrdp/startwm.sh unset DBUS_SESSION_BUS_ADDRESS unset XDG_RUNTIME_DIR $ sudo systemctl restart xrdp ## xrdp connection closes immediately? $ sudo vim /etc/xrdp/startwm.sh #!/bin/sh # xrdp X session start script (c) 2015, 2017 mirabilos # published under The MirOS Licence if test -r /etc/profile; then . /etc/profile fi if test -r /etc/default/locale; then . /etc/default/locale test -z "${LANG+x}" || export LANG test -z "${LANGUAGE+x}" || export LANGUAGE test -z "${LC_ADDRESS+x}" || export LC_ADDRESS test -z "${LC_ALL+x}" || export LC_ALL test -z "${LC_COLLATE+x}" || export LC_COLLATE test -z "${LC_CTYPE+x}" || export LC_CTYPE test -z "${LC_IDENTIFICATION+x}" || export L
자세한 내용 보기

centos 6.x yum update problem making ssl connection

-
이미지
## yum update [root@localhost ~]# yum update Loaded plugins: fastestmirror Setting up Update Process Loading mirror speeds from cached hostfile https://vault.centos.org/centos/6/os/x86_64/repodata/repomd.xml: [Errno 14] problem making ssl connection Trying other mirror. Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again ## fix repo [root@localhost ~]# cd /etc/yum yum/ yum.conf yum.repos.d/ [root@localhost ~]# cd /etc/yum.repos.d/ [root@localhost ~]# yum clean all Loaded plugins: fastestmirror Cleaning repos: base update Cleaning up Everything Cleaning up list of fastest mirrors ## fix repo [root@localhost yum.repos.d]# vi CentOS-Base.repo [base] name= CentOS-6 - Base baseurl= http://mirror.nsc.liu.se/centos-store/6.6/os/x86_64/ gpgcheck=0 priority=1 protect=1 [update] name=CentOS-6 - Updates baseurl=https://archive.kernel.org/centos-vault/6.6/updates/x86_64/ gpgcheck=0 priority=1 protect=1
자세한 내용 보기