foreword
This article belongs to the column "Big Data Installation and Deployment" . This column is original by the author. Please indicate the source for the citation. Please point out the deficiencies and mistakes in the comment area, thank you!
1. Prepare three virtual machines
Please refer to my blog to install a three-node pseudo-distributed cluster of big data - a detailed tutorial on the installation of a three-node big data environment
2. Prepare cloudera installation package
Since it is deployed offline, the required files need to be downloaded in advance.
The installation package required in this article has been split into 3 compressed files (because a single resource of Station C supports up to 1000 MB), download the compressed files of the following 3 links, put them in the same folder, and then unzip the first file. Get the installation package we need.
CDH 5.14.2 Installation Package Part 1
CDH 5.14.2 Installation Package Part 2
CDH 5.14.2 Installation Package Part 3
The documents that need to be prepared are:
- Cloudera Manager 5
- File name: cloudera-manager-centos7-cm5.14.0_x86_64.tar.gz
- CDH installation package (Parecls package)
- The version number must correspond to Cloudera Manager
- The following 3 files are required:
-
- CDH-5.14.0-1.cdh5.14.0.p0.23-el7.parcel
-
- CDH-5.14.0-1.cdh5.14.0.p0.23-el7.parcel.sha1
-
- manifest.json
- MySQL JDBC driver
- File name: mysql-connector-java-5.1.49.jar
Upload the decompressed CDH directory to the /root directory of the three nodes.
3. Install dependent packages on all machines
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
4. Install mysql database
Install the mysql database on the first machine
You can refer to my blog to install mysql—— CentOS7 installs MySQL 5.7
5. Install cloudera server
5.1 Unzip the server management installation package
All nodes upload the cloudera-manager-centos7-cm5.14.2_x86_64.tar.gz file and decompress it
[root@node1 ~]# tar -zxvf cdh/cloudera-manager-centos7-cm5.14.2_x86_64.tar.gz -C /opt/
[root@node2 ~]# tar -zxvf cdh/cloudera-manager-centos7-cm5.14.2_x86_64.tar.gz -C /opt/
[root@node3 ~]# tar -zxvf cdh/cloudera-manager-centos7-cm5.14.2_x86_64.tar.gz -C /opt/
After decompression, you can see the files in the /opt directory
[root@node1 ~]# cd /opt/
[root@node1 opt]# ll
total 0
drwxr-xr-x 3 root root 19 Oct 11 20:00 bigdata
drwxr-xr-x 4 1106 4001 36 Apr 3 2018 cloudera
drwxr-xr-x 9 1106 4001 88 Apr 3 2018 cm-5.14.2
drwxr-xr-x. 8 root root 273 Oct 6 16:47 java
[root@node1 opt]# cd cloudera/
[root@node1 cloudera]# ll
total 0
drwxr-xr-x 2 1106 4001 6 Apr 3 2018 csd
drwxr-xr-x 2 1106 4001 6 Apr 3 2018 parcel-repo
[root@node1 cloudera]# cd ..
[root@node1 opt]# cd cm-5.14.2/
[root@node1 cm-5.14.2]# ll
total 0
drwxr-xr-x 7 1106 4001 104 Apr 3 2018 etc
drwxr-xr-x 5 1106 4001 89 Apr 3 2018 lib
drwxr-xr-x 3 1106 4001 17 Apr 3 2018 lib64
drwxr-xr-x 5 1106 4001 70 Apr 3 2018 log
drwxr-xr-x 3 1106 4001 33 Apr 3 2018 run
drwxr-xr-x 2 1106 4001 41 Apr 3 2018 sbin
drwxr-xr-x 3 1106 4001 17 Apr 3 2018 share
5.2 Create client running directory
Manually create folders for all nodes
[root@node1 ~]# mkdir /opt/cm-5.14.2/run/cloudera-scm-agent
[root@node2 ~]# mkdir /opt/cm-5.14.2/run/cloudera-scm-agent
[root@node3 ~]# mkdir /opt/cm-5.14.2/run/cloudera-scm-agent
5.3 Create cloudera-scm user
Create cloudera-scm user for all nodes
useradd --system --home=/opt/cm-5.14.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
5.4 Initialize the database
Upload the provided mysql driver package: mysql-connector-java-5.1.49.jar to the /root/cdh directory of node1, and then refer to the following to put the mysql jdbc driver into the corresponding location:
[root@node1 cdh]# cp mysql-connector-java-5.1.49.jar /opt/cm-5.14.2/share/cmf/lib/
Refer to the following commands to initialize the database (only need to be executed on the Cloudera Manager Server: node1 node)
[root@node1 cdh]# /opt/cm-5.14.2/share/cmf/schema/scm_prepare_database.sh mysql -h node1 -uroot -p'1qaz@WSX' --scm-host node1 scm scm '1qaz@WSX'
JAVA_HOME=/opt/java
Verifying that we can write to /opt/cm-5.14.2/etc/cloudera-scm-server
Thu Oct 13 21:56:25 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /opt/cm-5.14.2/etc/cloudera-scm-server
Executing: /opt/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.14.2/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.14.2/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Thu Oct 13 21:56:26 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
Script parameter explanation:
[database type]
-h [database node ip/hostname]
-u[database user name]
-p[database password]
–scm-host [Cloudera Manager Server node ip/hostname]
[database]
[user name ]
[password]
5.5 Modify all node client configurations
Modify the server_host parameter to the host name of the Cloudera Manager Server node
[root@node1 ~]# vim /opt/cm-5.14.2/etc/cloudera-scm-agent/config.ini
5.6 Upload CDH installation package
Put the following files into the Server node: node1's /opt/cloudera/parcel-repo/ directory:
- CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel
- CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1
- manifest.json
[root@node1 cdh]# cp CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel /opt/cloudera/parcel-repo/
[root@node1 cdh]# cp CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/
[root@node1 cdh]# cp manifest.json /opt/cloudera/parcel-repo/
Rename the sha1 file
[root@node1 parcel-repo]# mv CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1 CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha
5.7 Change the user group permissions of the installation directory
All nodes change users and user groups of cdh related folders
[root@node1 ~]# chown -R cloudera-scm:cloudera-scm /opt/cm-5.14.2
[root@node1 ~]# chown -R cloudera-scm:cloudera-scm /opt/cloudera
5.8 Start Cloudera Manager and agent
Server node (that is, node1 node)
[root@node1 ~]# /opt/cm-5.14.2/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server: [ OK ]
The client needs to be started on all nodes
[root@node1 ~]# /opt/cm-5.14.2/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent: [ OK ]
[root@node2 ~]# /opt/cm-5.14.2/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent: [ OK ]
[root@node3 ~]# /opt/cm-5.14.2/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent: [ OK ]
6. Service installation
Use a browser to log in to the web interface of cloudera-manager, the user name and password are both admin
After logging in, check the acceptance agreement on the agreement page and click Continue
Choose the free version, the free version can already meet our daily business needs, just choose the free version. Click to continue
As shown below, click to continue
As shown in the figure below, click the currently managed machine, then select the machine, and click Continue
As shown in the figure below, then select the package corresponding to the version of your parcel
After clicking, enter the installation page, wait a moment
As shown in the figure below, the cluster is being installed
As shown in the figure below, the distribution of the installation package is successful, click Continue
For such a warning, you need to enter the following command on each machine:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'vm.swappiness=10'>> /etc/sysctl.conf
sysctl vm.swappiness=10
echo never > /sys/kernel/mm/transparent_hugepage/defrag”
“echo never > /sys/kernel/mm/transparent_hugepage/enabled”
As shown in the figure below, then click to re-run. If you don’t think so, there will be no warnings. Click Finish to enter the installation of hadoop ecosystem service components
As shown in the figure below, select the custom service, and we first install the most basic service combination. Then before installation, if it involves the installation of hive and oozie, then first go to mysql, create the database by yourself, and grant permissions;
therefore:
create database hive;
create database oozie;
grant all on *.* to hive identified by '1qaz@WSX';
grant all on *.* to oozie identified by '1qaz@WSX';
If the following error occurs:
mysql> grant all on *.* to oozie identified by '1qaz@WSX';
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
mysql> update mysql.user set Grant_priv='Y',Super_priv='Y' where user = 'root' and host = 'localhost';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> flush privileges;
mysql> quit
Bye
You have new mail in /var/spool/mail/root
[root@node02 ~]# systemctl restart mysqld.service
[root@node02 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.27 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all on *.* to hive identified by '1qaz@WSX';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all on *.* to oozie identified by '1qaz@WSX';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>
Then install the software!
Then, choose a custom service, if we need other services later, we will add them
Then click Continue to enter the Select Service Add Assignment page and assign it
After the service is selected, as shown in the figure below, you can click to view the service division by host
After clicking continue, as shown in the figure below, enter the number database scm in the mysql database, user name scm, password 1qaz@WSX, click test connection, wait for about 30 seconds, it shows success, click continue
Click all the way to continue, the rest is to wait
As shown in the figure above, if the waiting time is too long, we can stop the machine where the manager is located (that is, node01) and adjust the memory to a larger size. It is recommended that if the notebook is more than 4G, and if the cloud environment is more than 8G, we first adjust it to 4G or more. Restart the server and agent of cloudera after restarting the node01 machine
[root@node1 ~]# cd /opt/cm-5.14.2/etc/init.d
#启动server
[root@node1 init.d]# ./cloudera-scm-server start
#启动agent
[root@node1 init.d]# ./cloudera-scm-agent start
7. Log in to cloudera manager again
After the login is successful, as shown in the figure below, restart the cluster, and then wait.
