前一段时间因为公司有日志集中管理平台系统搭建，为了入库统计方便，就把nginx的日志全部格式化成json了。结果近期想自主统计一下访问人数的时候发现awk解析json的日志很蛋疼，就写了一个python脚本。算是第一次真正用python来满足业务需求，记录一下。
写记一下nginx配置的log格式
log_format json_log escape=json '{"realip":"$remote_addr","@timestamp":"$time_iso8601","host":"$http_host","request":"$request","req_body":"$request_body","status":"$status","size":$body_bytes_sent,"ua":"$http_user_agent","req_time":"$request_time","uri":"$uri","referer":"$http_referer","xff":"$http_x_forwarded_for","ups_status":"$upstream_status","ups_addr":"$upstream_addr","ups_time":"$upstream_response_time"}';
在贴几行ngin的日志内容
{"realip":"111.13.193.51","@timestamp":"2019-10-17T12:20:37+08:00","host":"test.test.com","request":"GET /HTTPServer/token/fund?fund=110009371&token=9875581cc341bcf HTTP/1.1","req_body":"","status":"200","size":4,"ua":"PanGu/2.2.1 (Linux; Android 9;MI 6)","req_time":"0.016","uri":"/HTTPServer/token/fund","referer":"","xff":"","ups_status":"200","ups_addr":"10.42.220.117:8090","ups_time":"0.016"}
{"realip":"111.13.193.51","@timestamp":"2019-10-17T12:20:37+08:00","host":"test.test.com","request":"POST /HTTPServer/servlet HTTP/1.1","req_body":"{\"funcid\":\"383001\",\"token\":\"9875581cc341bcf\",\"parms\":{\"SEC_ID\":\"tpyzq\",\"FLAG\":\"true\"}}","status":"200","size":121,"ua":"PanGu/2.2.1 (Linux; Android 9;MI 6)","req_time":"0.101","uri":"/HTTPServer/servlet","referer":"","xff":"","ups_status":"200","ups_addr":"10.42.220.117:8090","ups_time":"0.100"}
{"realip":"111.13.193.51","@timestamp":"2019-10-17T12:20:39+08:00","host":"test.test.com","request":"POST /HTTPServer/servlet HTTP/1.1","req_body":"{\"funcid\":\"300742\",\"token\":\"9875581cc341bcf\",\"parms\":{\"POSITION_STR\":\"\",\"REQUEST_NUM\":\"30\",\"SEC_ID\":\"tpyzq\",\"FLAG\":\"true\",\"STOCK_CODE\":\"\"}}","status":"200","size":2283,"ua":"PanGu/2.2.1 (Linux; Android 9;MI 6)","req_time":"0.137","uri":"/HTTPServer/servlet","referer":"","xff":"","ups_status":"200","ups_addr":"10.42.220.117:8090","ups_time":"0.132"}
第一次用readlines来获取所有的内容，后来发现生产的nginx当天日志大小将近4个G，如果直接跑的话，会占用4G的系统内存，所以计划优化一下。搞成按大小来分次读取的。
本来计划的是可以用ip和ua来区分出人，可以统计UV，后来发现不太对，有可能用户在手机移动网络和WIFI的情况下，IP会变，因为我们做的是证券软件，用户随时随地都要盯盘，所以这种情况的概率应该不低，所以需要前端配合，在UA里面增加用户设备标示，这样的话可能统计结果更加精确
下面代码为还没有优化的一次性加载文章全部内容的代码
# -*- coding: utf-8 -*-
import logging
import json
from datetime import datetime
import time
logging.basicConfig(level=logging.DEBUG,#控制台打印的日志级别
filename='count_access.log',
filemode='w',##模式，有w和a，w就是写模式，每次都会重新写日志，覆盖之前的日志
#a是追加模式，默认如果不写的话，就是追加模式
format=
'%(asctime)s - %(pathname)s[line:%(lineno)d] - %(levelname)s: %(message)s'
#日志格式
)
path = "access_json.log_20191014"
# 定义时间时间间隔，单位s
rule_interval = (60*60*24*5)
user_set = set()
android_set = set()
ios_set = set()
curl_set = set()
okhttp_set = set()
secure_set = set()
cfnet_set = set()
bigdate_set = set()
other_set = set()
startTime = time.time()
with open(path, 'r') as load_f:
all_lines = load_f.readlines()
print("文件加载完成，耗时s："+str(time.time()-startTime))
record_count = 0
android_count = 0
ios_count = 0
curl_count = 0
okhttp_count = 0
secure_count = 0
cfnet_count = 0
bigdata_count = 0
other_count = 0
for line in all_lines[len(all_lines)::-1]:
try:
json_line = json.loads(line)
except UnicodeDecodeError:
print(line)
continue
timestamp = json_line['@timestamp'].encode()
req_time = time.mktime(datetime.strptime(timestamp.split('+')[0], '%Y-%m-%dT%H:%M:%S').timetuple())
now_tome = time.time()
time_interval = now_tome-req_time
if (time_interval < rule_interval):
user = "IP:" +str(json_line['realip'].encode())+",UA:"+(json_line['ua'].encode())
# print("TIME："+str(timestamp)+","+user)
user_set.add(user)
record_count = record_count+1
if(record_count%10000==0):
print(str(record_count)+"...")
if("Android" in user):
android_set.add(user)
android_count = android_count + 1
elif ("iOS" in user or "iPhone" in user):
ios_set.add(user)
ios_count = ios_count + 1
elif("curl" in user):
curl_set.add(user)
curl_count = curl_count + 1
elif("okhttp" in user):
okhttp_set.add(user)
okhttp_count = okhttp_count + 1
elif("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 Hutool" in user):
secure_set.add(user)
secure_count = secure_count + 1
elif("CFNetwork" in user):
cfnet_set.add(user)
cfnet_count = cfnet_count + 1
elif("Apache-HttpClient/4.3.5" in user):
bigdate_set.add(user)
bigdata_count = bigdata_count + 1
else:
other_set.add(user)
other_count = other_count + 1
continue
else:
break
logging.info("UV："+str(len(user_set)))
logging.info("PV："+str(record_count))
logging.info("ANDROID_UV：" + str(len(android_set)))
logging.info("ANDROID_PV：" + str(android_count))
logging.info("IOS_UV：" + str(len(ios_set)))
logging.info("IOS_PV：" + str(ios_count))
logging.info("CURL_UV：" + str(len(curl_set)))
logging.info("CURL_PV：" + str(curl_count))
logging.info("OKHTTP_UV：" + str(len(okhttp_set)))
logging.info("OKHTTP_PV：" + str(okhttp_count))
logging.info("SECURE_UV：" + str(len(secure_set)))
logging.info("SECURE_PV：" + str(secure_count))
logging.info("CFNET_UV：" + str(len(cfnet_set)))
logging.info("CFNET_PV：" + str(cfnet_count))
logging.info("BIGDATA_UV：" + str(len(bigdate_set)))
logging.info("BIGDATA_PV：" + str(bigdata_count))
logging.info("OTHER_UV：" + str(len(other_set)))
logging.info("OTHER_PV：" + str(other_count))
logging.info(other_set)
