2020-09-16
freebuf能力中心開(kāi)源了一套資產(chǎn)掃描系統(tǒng)(https://github.com/TophantTechnology/ARL)，提供docker版，但并無(wú)源碼搭建的文檔(無(wú)完整文檔)。于是在星光哥及小明哥指導(dǎo)下完成源碼搭建，在這里記錄一下。
環(huán)境
搭建過(guò)程
首先從github下載源碼，我下載到opt 目錄下。
phantomjs
先在官網(wǎng)下載phantomjs
# 下載
wget https:///ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
# 解壓
tar -jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2
# 修改環(huán)境變量
vim /etc/profile
# 刷新環(huán)境變量
source /etc/profile
# 驗(yàn)證
phantomjs --version
/etc/profile文件末尾增加如下內(nèi)容(路徑看自己解壓完放哪)
export PATH=$PATH:/opt/phantomjs-2.1.1-linux-x86_64/bin
安裝相關(guān)軟件包
yum install epel-release
yum install rabbitmq-server supervisor
yum install wqy-microhei-fonts fontconfig
rabbitmq-server
上面已經(jīng)安裝了rabbitmq-server
# 啟動(dòng)
systemctl start rabbitmq-server
# 配置
rabbitmqctl add_user arl arlpassword
rabbitmqctl add_vhost arlvhost
rabbitmqctl set_user_tags arl arltag
rabbitmqctl set_permissions -p arlvhost arl ".*" ".*" ".*"
nmap
下載nmap
https:///dist/nmap-7.80-1.x86_64.rpm 
# 安裝
rpm -ivh nmap-7.80-1.x86_64.rpm
mongoDB
vim /etc/yum.repos.d/mongodb-org-4.0.repo
[mngodb-org]
name=MongoDB Repository
baseurl=http://mirrors.aliyun.com/mongodb/yum/redhat/7Server/mongodb-org/4.0/x86_64/
gpgcheck=0
enabled=1
安裝yum -y install mongodb-org
開(kāi)啟無(wú)IP限制（方便看數(shù)據(jù)） 
vim /etc/mongod.conf
service mongod restart
mongodb 添加認(rèn)證
# mongo
> use arl
> db.createUser({user:'admin',pwd:'admin',roles:[{role:'dbOwner',db:'arl'}]})
> exit
開(kāi)啟登錄驗(yàn)證
# 關(guān)閉數(shù)據(jù)庫(kù)
service mongod stop
# 開(kāi)啟驗(yàn)證登錄
vim /etc/mongod.conf
security:
authorization: enabled
# service mongod start
添加賬號(hào)密碼
mongo -u admin -p admin --authenticationDatabase "arl"
use arl
db.user.insert({ username: 'admin123', password: hex_md5('arlsalt!@#'+'admin123') })
exit
GeoIP數(shù)據(jù)庫(kù)
由于官方政策更新請(qǐng)前往maxmind注冊(cè)下載GeoLite2-City.tar.gz，GeoLite2-ASN.tar.gz 解壓。
在config.yaml中配置好相關(guān)路徑。
我放在了/data/GeoLite2
config.yaml配置
添加并修改配置文件
cp app/config.yaml.example app/config.yaml
我的配置
CELERY:
BROKER_URL : "amqp://arl:arlpassword@localhost:5672/arlvhost"
MONGO:
URI : 'mongodb://admin:admin@localhost:27017/arl'
DB : 'arl'
RISKIQ:
EMAIL: "n1un1u2019@qq.com"
KEY: "03da3c29cce5152a536bae332f7b03a288154b37a13b93921225d3ade49c9b4c"
GEOIP:
CITY: '/data/GeoLite2/GeoLite2-City.mmdb'
ASN: '/data/GeoLite2/GeoLite2-ASN.mmdb'
FOFA:
EMAIL: ""
KEY: ""
ARL:
AUTH: true
API_KEY: ""
BLACK_IPS:
- 127.0.0.0/8
#- 0.0.0.0/8
#- 172.16.0.0/12
#- 100.0.0.0/8
#- 10.0.0.0/8
#- 192.168.0.0/16
nginx
# centOS7.6下安裝nginx
cd /yum install gcc gcc-c++ # 安裝依賴
wget http:///download/nginx-1.17.1.tar.gz # 下載nginx源碼
tar -zxvf nginx-1.17.1.tar.gz # 解壓
cd nginx-1.17.1 # 進(jìn)入解壓后的文件夾
./configure --prefix=/usr/local/nginx # 配置nginx安裝路徑
make
make install
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx # 創(chuàng)建nginx軟鏈接，若已被占用，則在 /usr/bin 下 rm-rf nginxnginx # 啟動(dòng)ngixnnginx -s stop # 停止nginx服務(wù)
導(dǎo)入配置nginx.conf，這邊直接用root啟動(dòng)了(根據(jù)自身修改用戶)。
user root;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /usr/local/nginx/conf/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http:///en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80;
server_name _;
#access_log logs/host.access.log main;
access_log off;
root /opt/ARL-master/docker/frontend;
location / {
try_files $uri $uri/ /index.html;
index index.html index.htm;
}
location /api/ {
proxy_pass http://127.0.0.1:5003/api/;
}
}
}
nginx啟動(dòng)。
無(wú)報(bào)錯(cuò)，前端則啟動(dòng)成功。
后端啟動(dòng)
安裝依賴
pip3 install -r requirements.txt
WEB 服務(wù)啟動(dòng)
gunicorn -b 0.0.0.0:5003 app.main:arl_app -w 3 --access-logfile arl_web.log
CELERY 啟動(dòng)
celery -A app.celerytask.celery worker -l info -Q arltask -n arltask -c 2 -O fair -f arl_worker.log
利用tmux+supervisior啟動(dòng)。
安裝tmux
yum install -y tmux
在項(xiàng)目根目錄編寫(xiě)supervisior文件
supervisior.conf
[program:web_api]
command=gunicorn -b 0.0.0.0:5003 app.main:arl_app -w 3 --access-logfile /var/log/arl_web.log
;stdout_logfile=/var/log/web_api_stdout.log
;stderr_logfile=/var/log/web_api_stderr.log
autostart=true
autorestart=true
[program:celery_scan]
command=celery -A app.celerytask.celery worker -l info -Q arltask -n arltask -c 2 -O fair -f arl_worker.log
;stdout_logfile=/var/log/celery_scan_stdout.log
;stderr_logfile=/var/log/celery_scan_stderr.log
autostart=true
autorestart=true
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
[supervisord]
logfile=/tmp/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=info ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=true ; start in foreground if true; default false
minfds=1024 ; min. avail startup file descriptors; default 1024
minprocs=200 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
# 新建一個(gè)窗口
tmux new -s sp
# 在項(xiàng)目根目錄運(yùn)行
supervisord -c supervisord.conf
# 會(huì)話后臺(tái)運(yùn)行
ctrl + b + d
# 進(jìn)入會(huì)話
tmux a -t sp
運(yùn)行成功
使用
直接訪問(wèn)該IP
admin123/admin123登錄
添加任務(wù)并看結(jié)果
查看API文檔并測(cè)試
結(jié)尾
使用了下，在資產(chǎn)探測(cè)收集還是很不錯(cuò)的，日站收集資產(chǎn)神器。而且是開(kāi)源，也提供了API出來(lái)。在后續(xù)二開(kāi)也很方便的。感謝開(kāi)源！
|
