Install Baidu Cloud Crawler under Centos 7

This crawler runs under MySQL, Python 2.7, and Mysql-python, so install MySQL and MySQL-python first.

1. Install MySQL installation dependencies

yum install libaio

Install MySQL

wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum localinstall mysql-community-release-el7-5.noarch.rpm yum install mysql-community-server

Start MySQL

systemctl start mysqld

Set MySQL password

mysql_secure_installation;

2. Firewall settings and installation of iptables

yum install iptables-services

Open port 3306

vi /etc/sysconfig/iptables

Add

-A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 3306 -j ACCEPT -A RH-Firewall-1-INPUT -m state –state NEW -m udp -p udp –dport 3306 -j ACCEPT

Restart iptables

service iptables restart

3. Install MySQL-python

yum install MySQL-python

4. Setup program

wget https://github.com/x-spiders/baiduyun-spider/archive/master.zip unzip master.zip cd baiduyun-spider-master

Set the account and password for connecting to the database

Open bin/spider.py and modify DB_HOST, DB_PORT, DB_USER, DB_PASS

Run the crawler for the first time

python bin/spider.py --seed-user

Run the crawler

pythonbin/spider.py

Source code source: https://geekspider.org/senior/215.html

Recommended site search: Korean server recommendation, mainland China permanent free cloud server, US server rental, .net space, Chinese international domain name, website domain name ip address query, virtual host rental, mainland China space, private server website space, Hong Kong vps Host rental,