python logging多进程多线程输出到同一个日志文件的实战案例
作者:VincentZhu 时间:2021-01-08 16:25:17
参考官方案例:https://docs.python.org/zh-cn/3.8/howto/logging-cookbook.html
import logging
import logging.config
import logging.handlers
from multiprocessing import Process, Queue
import random
import threading
import time
def logger_thread(q):
while True:
record = q.get()
if record is None:
break
logger = logging.getLogger(record.name)
logger.handle(record)
def worker_process(q):
qh = logging.handlers.QueueHandler(q)
root = logging.getLogger()
root.setLevel(logging.DEBUG)
root.addHandler(qh)
levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR,
logging.CRITICAL]
loggers = ['foo', 'foo.bar', 'foo.bar.baz',
'spam', 'spam.ham', 'spam.ham.eggs']
for i in range(100):
lvl = random.choice(levels)
logger = logging.getLogger(random.choice(loggers))
logger.log(lvl, 'Message no. %d', i)
if __name__ == '__main__':
q = Queue()
d = {
'version': 1,
'formatters': {
'detailed': {
'class': 'logging.Formatter',
'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
},
'file': {
'class': 'logging.FileHandler',
'filename': 'mplog.log',
'mode': 'w',
'formatter': 'detailed',
'foofile': {
'filename': 'mplog-foo.log',
'errors': {
'filename': 'mplog-errors.log',
'level': 'ERROR',
'loggers': {
'foo': {
'handlers': ['foofile']
'root': {
'level': 'DEBUG',
'handlers': ['console', 'file', 'errors']
}
workers = []
for i in range(5):
wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
workers.append(wp)
wp.start()
logging.config.dictConfig(d)
lp = threading.Thread(target=logger_thread, args=(q,))
lp.start()
# At this point, the main process could do some useful work of its own
# Once it's done that, it can wait for the workers to terminate...
for wp in workers:
wp.join()
# And now tell the logging thread to finish up, too
q.put(None)
lp.join()
实战案例:
1、字典形式配置日志
log_conf_dict = {
'version': 1,
'formatters': {
'my_formatter': {
'class': 'logging.Formatter',
'format': '%(asctime)s %(processName)s(%(process)d) %(threadName)s(%(thread)d) %(filename)s[line:%(lineno)d] %(levelname)s %(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
'formatter': 'my_formatter',
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': '/log/test.log',
'maxBytes': 5*1024*1024,
'backupCount': 60,
'mode': 'w',
'delay': True,
'formatter': 'my_formatter',
'encoding': 'utf-8',
'level': 'INFO',
},
},
'loggers': {
'my_logger': {
'handlers': ['file']
}
},
'root': {
'level': _level,
'handlers': ['console', 'file']
},
}
2、主进程中开启独立的日志写入监听线程
"""主进程中开启独立的日志写入监听线程"""
queue = Queue(-1)
logging.config.dictConfig(dict)
log_thread = threading.Thread(target=logger_main, args=(queue,))
log_thread.start() """其他逻辑代码段""" queue.put(None) log_thread.join()
日志写入函数
def logger_main(q):
'''日志队列写入文件'''
while True:
record = q.get()
if record is None:
break
logger = logging.getLogger()
logger.handle(record)
3、子进程中将日志输入QueueHandler日志队列
def child_proc_main(queue):
lqh = logging.handlers.QueueHandler(queue)
lqh.set_name("my_queue_handler")
root = logging.getLogger() #很关键的一步,必须先清空,再加入。原因:多进程多线程复杂环境下,在window和linux平台运行表现不一致,linux会复制主进程的日志配置,造成同时输出多个日志文件。
root.handlers.clear()
root.addHandler(lqh)
root.setLevel(level)
来源:https://www.cnblogs.com/OnlyDreams/p/15923001.html
标签:python,logging,日志文件
![](/images/zang.png)
![](/images/jiucuo.png)
猜你喜欢
Python中实现变量赋值传递时的引用和拷贝方法
2021-12-26 04:05:54
详解ES6之async+await 同步/异步方案
2023-08-24 11:10:54
python列表的常用操作方法小结
2023-06-21 16:14:34
Python在不同场景合并多个Excel的方法
2021-11-03 02:52:00
![](https://img.aspxhome.com/file/2023/7/107867_0s.webp)
Python聚类算法之凝聚层次聚类实例分析
2023-08-01 05:30:14
![](https://img.aspxhome.com/file/2023/1/89371_0s.png)
php cookie中点号(句号)自动转为下划线问题
2023-09-07 11:05:04
Python 获得13位unix时间戳的方法
2022-12-08 18:26:25
python统计日志ip访问数的方法
2023-07-05 18:57:20
Go语言题解LeetCode724寻找数组的中心下标
2023-07-09 03:26:01
python微信跳一跳游戏辅助代码解析
2021-06-10 21:42:29
![](https://img.aspxhome.com/file/2023/7/112167_0s.png)
Python中最快的循环姿势实例详解
2022-12-16 01:30:50
![](https://img.aspxhome.com/file/2023/2/98942_0s.jpg)
Python决策树之基于信息增益的特征选择示例
2023-01-09 23:44:32
![](https://img.aspxhome.com/file/2023/0/101290_0s.png)
python 剪切移动文件的实现代码
2023-06-10 05:21:14
Python模块WSGI使用详解
2021-02-19 00:05:27
Python使用itchat 功能分析微信好友性别和位置
2023-09-24 15:57:12
![](https://img.aspxhome.com/file/2023/5/99455_0s.jpg)
django中的图片验证码功能
2022-06-10 00:07:54
![](https://img.aspxhome.com/file/2023/9/95469_0s.png)
asp如何通过表单创建一个Word?
2010-06-07 20:56:00
如何实现文件上传并自动归类功能?
2010-05-24 18:37:00
Pycharm新建模板默认添加个人信息的实例
2022-03-30 20:59:50
![](https://img.aspxhome.com/file/2023/5/80585_0s.jpg)
Python实现桶排序与快速排序算法结合应用示例
2022-10-26 23:33:39
![](https://img.aspxhome.com/file/2023/2/84032_0s.jpg)