神马是Fabric
- 一个让你通过命令行执行无参数Python函数的工具
- 一个让通过ssh执行的shell命令更加容器、更符合Python的命令库
- 自然而然地，大部分用户把这两件事结合着用，使用 Fabric 来写和执行 Python 函数或 task
安装
建议安装pyenv使用python的虚拟环境
pip安装,必须是python2.7版本以上不然不支持
pip3 install fabric3 #安装python3以上版本
pip install fabric #安装python3以下版本
fab常用参数
- fab选项参数
-l 显示定义好的任何函数列表
-f 执行fab入口文件，默认入口文件为fabfile.py
-g 指定网关
-H 指定目标主机，多个主机使用，分隔
-P 以异步并行方式运行多主机任务，默认为串行任务，也就是依次执行
-u 指定主机用户名
-P 执行主机密码
案例： fab -p 密码 -H 主机1,主机2... '命令'
- fab操作案例
(fabric) [fabric@MAILCCOD ~]$ fab -H root@192.168.127.1:22 -- "pwd" #远程执行pwd命令
[root@192.168.127.1:22] Executing task '<remainder>'
[root@192.168.127.1:22] run: pwd
[root@192.168.127.1:22] Login password for 'root':
[root@192.168.127.1:22] out: /root
[root@192.168.127.1:22] out:
Done.
编写fabfile文件
-
全局属性的设置
env对象的作用定义了fabfile的全局设置，支持多个属性，包括目标主机、用户、密码、角色
env.hosts定义多个目标主机、用ip或者主机名的列表
格式： env.hosts= ["192.168.127.1","192.168.127.2"]env.user： 定义用户名
格式： env.user = "root"env.port: 定义端口
格式： env.port = 22env.password:定义密码
格式：env.password = "1"env.warn_only执行报错不中断
env.warn_only = Trueenv.skip_bad_hosts连接服务器异常跳过
env.skip_bad_hosts = Trueenv.timeout设置执行命令超时时间，默认10s
env.timeout = 120env.passwords: 定义多台主机的用户名、ip地址、端口、密码
格式: env.passwords = { "root@192.168.127.1:22":"passwd", "root@192.168.127.2:22":"passwd", }env.gatewayd定义网关
格式：env.gateway = "192.168.127.1"env.roledefs:定义角色分组
格式： env.roledefs = { "web":["192.168.127.2","192.168.127.3"] "Mysql":["192.168.127.4","192.168.127.5"] }
单机操作
- 编写fabfile文件
fabfile.py
#coding:utf8
from fabric.api import * #导出api方法
env.hosts = ["127.0.0.1"] #声明定义host主机
env.user = "root" #声明定义用户名
env.password = "ChannelSoft*ccod4.5" #声明定义密码
env.port = 20001 #声明定义密码
@task #使用装饰器
def show(): #编写show函数，远程运行linux命令
run("ls /home")
run("ps -ef|rep resin")
@task
def Mem():
run("free -m")
@task
def run_all(): #执行所有任务
execute(show)
execute(Mem)
if __name__ == "__main__": #使用Python直接执行
execute(run_all)
- 运行fabfile文件
fab -f fabfile.py show运行show任务
fab -f fabfile.py -l显示执行的任务
./fabfile.py#执行调用
多机操作
- 多机操作
#!/usr/bin/python
#coding:utf8
from fabric.api import *
env.user = "root"
env.hosts = ["192.168.127.1","192.168.127.2"]
env.passwords = {
"root@192.168.127.1:22":"qnsoft",
"root@192.168.127.2:22":"qnsoft",
}
#roledefs是定义角色分组
env.roledefs = {
"Web":["192.168.127.1"],
"DB":["192.168.127.2"]
}
@task
def show():
run("df -h")
@task
@roles("Web") #使用角色
def Echo():
run("ifconfig ")
@task
@roles("DB")
def run_all():
execute(show) #此命令是调用上面定义的函数
run("ifconfig")
if __name__ == "__main__":
execute(run_all)
- 运行代码
[auto@mongodb1 fabric]$ fab -f Fabric1-0.py -l #使用fab命令显示可执行的任务
Available commands:
Echo
run_all
show
[auto@mongodb1 fabric]$ fab -f Fabric1-0.py Echo #执行Echo任务
常用的fabric的API
local: 执行本地命令
格式: local(本地命令)lcd: 切换到本地目录
格式: lcd(本地目录)cd: 切换远程目录
格式: cd(远程目录)run: 执行远程命令
格式: run(远程命令)sudo: sudo方式执行远程命令
格式: sudo(远程命令)put: 上传本地文件到远程主机
格式: put(本地文件,远程文件)get:从远程主机下载文件到本地
格式: get(远程文件,本地文件)prompt: 获取用户输入信息
格式: prompt("please input user password:")confirm: 获得提示信息确认
格式: confirm("continue[Y/N]?")reboot: 重启远程主机
格式： reboot()@task: 函数修饰符，标注过的
fab命令可见,非标注过的fab命令不可见@runs_once: 函数修饰符，标识的函数只会执行一次，不受多台主机影响
颜色输出
- from fabric.colors import * 导入颜色函数
- fabric.colors.blue(text, bold=False)
- fabric.colors.cyan(text, bold=False)
- fabric.colors.green(text, bold=False)
- fabric.colors.magenta(text, bold=False)
- fabric.colors.red(text, bold=False)
- fabric.colors.white(text, bold=False)
- fabric.colors.yellow(text, bold=False)
文件上传下载校验
- 源代码
upload_file.py
#!/usr/bin/python
#coding:utf8
from fabric.api import *
from fabric.contrib.console import confirm
from fabric.colors import * #导入颜色函数
env.user = "root"
env.hosts = ["192.168.127.26"]
env.passwords = {
"root@192.168.127.2:22":"ChannelSoft*ccod4.5"
}
#roledefs是定义角色分组
env.roledefs = {
"fps":["192.168.127.2"]
}
#定义文件上传任务
@task
def upload_file():
with settings(warn_only=True): #异常处理
local("tar zcf FPS.tar.gz /home/auto/fabric/Fps.py")
result = put("FPS.tar.gz","/root/FPS.tar.gz")
if result.failed and not confirm("continue[y/n]"): #异常处理，如果上传失败是否继续
abort("Put tar file Failed")
#获取本地和上传远端文件的m5d值是否一致
with settings(warn_only=True):
local_file = local("md5sum FPS.tar.gz",capture=True).split(" ")[0]
remote_file = run("md5sum /root/FPS.tar.gz").split(" ")[0]
if local_file == remote_file:
print(green("Yes upload Acss , M5D Ok"))
else:
print(read("Upload file Failed MD5 NO ！！！"))
#定义文件下载任务
@task
def download_file():
with settings(warn_only=True):
get("/root/FPS.tar.gz","FPS.tar.gz")
local("tar -zxf FPS.tar.gz -C ./ && cat Fps.py ")
@task
def run_all():
execute(upload_file)
execute(download_file)
if __name__ == "__main__":
execute(run_all)
并行执行
如何运作
默认情况下，fabric会默认顺序执行所有的任务，为了任务函数之间并不会产生交互，该功能实现是基于Python multiprocessing模块，它为每个主机和任务组合创建一个线程，同时提供一个可选的弹窗用于阻止创建过多的进程如何使用装饰器
由于并行执行影响的最小单位是任务，所有功能的启动和禁用也是以任务为单位使用parallel或者serial装饰器，如下所示：
#!/usr/bin/python
#coding:utf8
from fabric.api import *
@parallel
def runs_in_parallel():
pass
def runs_serially():
pass
如果这样执行：
$ fab -H host1,host2,host3 runs_in_parallel runs_serially
将会按照这样的流程执行：
runs_in_parallel 运行在 host1、host2 和 host3 上
runs_serially 运行在 host1 上
runs_serially 运行在 host2 上
runs_serially 运行在 host3 上
- 命令行操作
你也可以使用命令行选项 -P 或者环境变量 env.parallel <env-parallel>强制所有任务并行执行。不过被装饰器 `~fabric.decorators.serial 封装的任务会忽略该设置，仍旧保持顺序执行。
例如，下面的 fabfile 会产生和上面同样的执行顺序：
from fabric.api import *
def runs_in_parallel():
pass
@serial
def runs_serially():
pass
在这样调用时：
$ fab -H host1,host2,host3 -P runs_in_parallel runs_serially
和上面一样，runs_in_parallel 将会并行执行，runs_serially 顺序执行。
- bubble大小
主机列表很大时，用户的机器可能会因为并发运行了太多的 Fabric 进程而被压垮，因此，你可能会选择 moving bubble 方法来限制 Fabric 并发执行的活跃进程数。
默认情况下没有使用 bubble 限制，所有主机都运行在并发池中。你可以在任务级别指定parallel 的关键字参数 pool_size 来覆盖该设置，或者使用选项 -z 全局设置。
例如同时在 5 个主机上运行：
from fabric.api import *
@parallel(pool_size=5)
def heavy_task():
# lots of heavy local lifting or lots of IO here
或者不使用关键字参数 pool_size：
$ fab -P -z 5 heavy_task
- 案例
并行执行
#!/usr/bin/python
#coding:utf8
from fabric.api import *
from Hosts import *
@task
@roles("Web")
@parallel(pool_size=5) #意思说最大线程是为5个
def File():
with prefix("cd /home") :
run("ls -htrl")
#类似执行 cd /home && ls ccodrunner 的linux命令
if __name__ == "__main__":
execute(File)
上下文切换
-使用with语句的上下文管理器
#!/usr/bin/python
#coding:utf8
from fabric.api import *
env.hosts = ["127.0.0.1"] #声明定义host主机
env.user = "root" #声明定义用户名
env.password = "qnsoft" #声明定义密码
env.port = 10000 #声明定义密码
@task
def File():
with prefix("cd /home") : #with上下文切换
run("ls ccodrunner")
#类似执行 cd /home && ls ccodrunner 的linux命令
if __name__ == "__main__":
execute(File)
案例多服务器不同端口执行任务
- Host文件
#!/usr/local/python27/bin/python2
#coding:utf8
from fabric.api import *
from fabric.state import env
import ConfigParser,os
curpath=os.path.dirname(os.path.realpath(__file__))
cfgpath=os.path.join(curpath,"config.ini")
conf=ConfigParser.ConfigParser()
conf.read(cfgpath)
Fabric_Remote_execution_user=conf.get("FabricBaseConfig","Fabric_Remote_execution_user")
CmsGroupNameOne=conf.get("FabricBaseConfig","CmsGroupNameOne")
UmgGroupNameTwo=conf.get("FabricBaseConfig","UmgGroupNameTwo")
CheckcmsGroups=conf.get("CheckCms","CheckcmsGroups")
CMSCDRHostGroup=conf.get("FabricBaseConfig","CMSCDRHostGroup")
env.user = Fabric_Remote_execution_user
env.warn_only = True
env.passwords = { #此处就不需要添加端口，否则后面程序处理异常
'root@172.16.100.30':'******',
'root@172.16.100.31':'******',
'root@172.16.100.32':'******',
'root@172.16.186.11':'******',
'root@172.16.186.12':'******',
'root@172.16.186.13':'******',
'root@172.16.186.14':'******',
'root@172.16.100.84':'******',
'root@172.16.100.49':'******',
'root@172.16.100.50':'******',
'root@172.16.100.53':'******',
'root@172.16.100.86':'******',
'root@172.16.100.87':'******',
'root@172.16.100.98':'******',
'root@172.16.100.36':'******',
'root@172.16.100.53':'******',
'root@172.16.100.17':'******',
'root@172.16.100.25':'******'
}
#roledefs是定义角色分组
env.roledefs = {
CMSCDRHostGroup:["172.16.100.86","172.16.100.87","172.16.100.98","172.16.100.49","172.16.100.50"],
UmgGroupNameTwo:["172.16.186.12","172.16.186.13","172.16.100.84","172.16.100.25:50712"], #如果非22端口则在roledefs里面指定端口
CmsGroupNameOne:["172.16.100.49","172.16.100.50","172.16.100.53","172.16.100.17","172.16.100.87","172.16.100.98","172.16.100.86"],
CheckcmsGroups:["172.16.100.49","172.16.100.50","172.16.100.53","172.16.100.17","172.16.100.87","172.16.100.98","172.16.100.86","172.16.100.53","172.16.100.17"]
}
- 数据采集程序
#!/usr/local/python27/bin/python2
# -*- coding:utf-8 -*-
from fabric.api import *
from Hosts import *
import time,ConfigParser,os,logging.config,logging
from datetime import datetime, date, timedelta
@task
@roles(CmsGroupNameOne)
#获取SDR文件名
def GetSdrFileName():
Host = env.host_string
RunHost = SplitSrt(Host)
#RunNewHost = Fabric_Remote_execution_user + "@"+ RunHost +":22"
RunNewHost = Fabric_Remote_execution_user + "@"+ RunHost
#通过passwords获取对应的密码
RunNewPasswd = env.passwords.get(RunNewHost)
env.password = RunNewPasswd
Shell_1 = "find " + SdrPath + " -maxdepth " + EcursiveDirs + " -type d -name \"$(date -d yesterday +%Y%m)*\"|uniq -w " + LengthWeightRemoval
#pending目录
run(" ls -htrl " + SdrPath + "sdr/pending/*_" + Time + "* | grep _" + Time + " |grep SDR |awk '{print $NF}' |" + " grep -v " + TimeTo + " |sort|uniq")
run(" ls -htrl " + SdrPath + "sdr/SDR*" + Time + "* | grep _" + Time + " |grep SDR |awk '{print $NF}' |" + " grep -v " + TimeTo + " |sort|uniq")
ReturnMsg1 = run(Shell_1)
if "No such file or director" not in ReturnMsg1:
HuaDanDirList = ReturnMsg1.split("\r\n")
RemovalHuaDanDir = []
for HuaDanDir in HuaDanDirList:
NewHuaDanDir = HuaDanDir[:-2]
RemovalHuaDanDir.append(NewHuaDanDir)
RemovalHuaDanDir = list(set(RemovalHuaDanDir))
if RemovalHuaDanDir is not None:
for NewHuaDanDir in RemovalHuaDanDir:
print("当前Host: ",RunHost," 获取话单路径列表: ",NewHuaDanDir)
run(" ls -htrl " + NewHuaDanDir + "*/* | grep _" + Time + " |grep SDR |awk '{print $NF}' |" + " grep -v " + TimeTo + " |sort|uniq")
else:
print("当前Host: ",RunHost,"获取SDR话单路径列表为: None")
#获取Hostip
def SplitSrt(Host):
MewHost = Host.split(":")[0]
return MewHost
@task
@roles(UmgGroupNameTwo)
#获取CDR文件名
def GetCdrFileName():
Host = env.host_string
RunHost = SplitSrt(Host)
RunNewHost = Fabric_Remote_execution_user + "@"+ RunHost
RunNewPasswd = env.passwords.get(RunNewHost)
print(RunNewPasswd,"--->",env.passwords,RunNewHost)
env.password = RunNewPasswd
Shell_2 = "find " + CdrPath + " -maxdepth " + EcursiveDirs + " -type d -name \"$(date -d yesterday +%Y%m)*\"|grep DRFilesBak|grep cdr| uniq -w " + LengthWeightRemoval
ReturnCdrMsg = run(Shell_2)
if "No such file or director" not in ReturnCdrMsg:
CdrDirList = ReturnCdrMsg.split("\r\n")
RemovalCDRHuaDanDir = []
for HuaDanCdrDir in CdrDirList:
NewHuadanCdrDir = HuaDanCdrDir[:-2]
RemovalCDRHuaDanDir.append(NewHuadanCdrDir)
RemovalCDRHuaDanDir = list(set(RemovalCDRHuaDanDir))
if RemovalCDRHuaDanDir is not None:
for NewHuadanCdrDir in RemovalCDRHuaDanDir:
run(" ls -htrl " + NewHuadanCdrDir + "*/* | grep " + Time + " |grep CDR |awk '{print $NF}' |" + " grep -v " + TimeTo + " |sort|uniq")
else:
print("当前Host: ",RunHost,"获取CDR话单路径列表为: None")
@task
@roles(CMSCDRHostGroup)
#获取SDR服务器中的CDR话单文件列表
def GetSdrCdrFile():
Host = env.host_string
RunHost = SplitSrt(Host)
RunNewHost = Fabric_Remote_execution_user + "@"+ RunHost
RunNewPasswd = env.passwords.get(RunNewHost)
env.password = RunNewPasswd
Shell_3 = "find " + SdrCdrPath + " -maxdepth " + EcursiveDirs + " -type d -name \"$(date -d yesterday +%Y%m)*\"|grep cdr|grep YD| uniq -w " + LengthWeightRemoval
ReturnCmsCdrMsg = run(Shell_3)
if "No such file or director" not in ReturnCmsCdrMsg:
CmsCdrDirList = ReturnCmsCdrMsg.split("\r\n")
RemovalCmsCDRHuaDanDir = []
for CmsCdrDir in CmsCdrDirList:
GetNewCmsCdrDir = CmsCdrDir[:-2]
RemovalCmsCDRHuaDanDir.append(GetNewCmsCdrDir)
RemovalCmsCDRHuaDanDir = list(set(RemovalCmsCDRHuaDanDir))
if RemovalCmsCDRHuaDanDir is not None:
for GetCmsSdrDirPath in RemovalCmsCDRHuaDanDir:
run(" ls -htrl " + GetCmsSdrDirPath + "*/* | grep " + Time + " |grep CDR |awk '{print $NF}' |" + " grep -v " + TimeTo + " |sort|uniq")
else:
print("当前Host: ",RunHost,"获取CDR话单路径列表为: None")
if __name__ == "__main__":
curpath=os.path.dirname(os.path.realpath(__file__))
cfgpath=os.path.join(curpath,"config.ini")
conf=ConfigParser.ConfigParser()
conf.read(cfgpath)
Time = (date.today() + timedelta(days=-1)).strftime("%Y%m")
TimeTo = time.strftime("%Y%m%d", time.localtime())
SdrPath= conf.get("FabricBaseConfig","SdrPath")
CdrPath= conf.get("FabricBaseConfig","CdrPath")
SdrCdrPath= conf.get("FabricBaseConfig","SdrCdrPath")
DockerPath= conf.get("FabricBaseConfig","DockerPath")
CmsGroupNameOne = conf.get("FabricBaseConfig","CmsGroupNameOne")
UmgGroupNameTwo = conf.get("FabricBaseConfig","UmgGroupNameTwo")
EcursiveDirs = conf.get("FabricBaseConfig","EcursiveDirs")
LengthWeightRemoval = conf.get("FabricBaseConfig","LengthWeightRemoval")
Fabric_Remote_execution_user=conf.get("FabricBaseConfig","Fabric_Remote_execution_user")
CMSCDRHostGroup=conf.get("FabricBaseConfig","CMSCDRHostGroup")
if CmsGroupNameOne:
execute(GetSdrFileName)
if UmgGroupNameTwo:
execute(GetCdrFileName)
fabric免密操作
由于使用fabric第三方模块版本过老，centos7需要使用ssh-keygen -m PEM -t rsa特殊格式，生成密钥，然后进行免密认证,centos6系统使用《ssh-keygen -t rsa》,然后依次对服务器做免密认证。
实现-密码配置为空即可
#!/usr/local/python27/bin/python2
#coding:utf8
from fabric.api import *
from fabric.state import env
import ConfigParser,os
curpath=os.path.dirname(os.path.realpath(__file__))
cfgpath=os.path.join(curpath,"./cfg/config.ini")
conf=ConfigParser.ConfigParser()
conf.read(cfgpath)
Fabric_Remote_execution_user=conf.get("FabricBaseConfig","Fabric_Remote_execution_user")
CmsGroupNameOne=conf.get("FabricBaseConfig","CmsGroupNameOne")
UmgGroupNameTwo=conf.get("FabricBaseConfig","UmgGroupNameTwo")
#env.user = Fabric_Remote_execution_user
#env.hosts = ["192.168.127.8","10.100.0.60","10.100.0.53","10.100.0.56","10.100.0.49","10.100.0.25","10.100.0.3"]
#env.key_filename = "/home/ZLtest/.ssh/id_rsa"
#env.password = 'JSDXqnsoft!1024'
env.warn_only = True
env.skip_bad_hosts = True
env.passwords = {
"root@192.168.27.30":"",
"root@192.168.27.31":"",
"root@192.168.27.34":"",
"root@192.168.27.35":""
}
#roledefs是定义角色分组
env.roledefs = {
UmgGroupNameTwo:["192.168.27.34","192.168.27.35"],
CmsGroupNameOne:["192.168.27.30","192.168.27.31"]
}
