python – 修复了线程锁定的strptime异常,但是减慢了程序的速度

我有以下代码,当在一个线程内运行时(完整的代码在这里 – https://github.com/eWizardII/homobabel/blob/master/lovebird.py)

 for null in range(0,1):
            while True:
                try:
                    with open('C:/Twitter/tweets/user_0_' + str(self.id) + '.json', mode='w') as f:
                        f.write('[')
                        threadLock.acquire()
                        for i, seed in enumerate(Cursor(api.user_timeline,screen_name=self.ip).items(200)):
                            if i>0:
                                f.write(", ")
                            f.write("%s" % (json.dumps(dict(sc=seed.author.statuses_count))))
                            j = j + 1
                        threadLock.release()
                        f.write("]")
                except tweepy.TweepError, e:
                    with open('C:/Twitter/tweets/user_0_' + str(self.id) + '.json', mode='a') as f:
                        f.write("]")
                    print "ERROR on " + str(self.ip) + " Reason: ", e
                    with open('C:/Twitter/errors_0.txt', mode='a') as a_file:
                        new_ii = "ERROR on " + str(self.ip) + " Reason: " + str(e) + "\n"
                        a_file.write(new_ii)
                break

现在没有线程锁定我生成以下错误：

Exception in thread Thread-117: Traceback (most recent call last):   File "C:\Python27\lib\threading.py", line 530, in __bootstrap_inner
    self.run()   File "C:/Twitter/homobabel/lovebird.py", line 62, in run
    for i, seed in enumerate(Cursor(api.user_timeline,screen_name=self.ip).items(200)): File "build\bdist.win-amd64\egg\tweepy\cursor.py", line 110, in next
    self.current_page = self.page_iterator.next()   File "build\bdist.win-amd64\egg\tweepy\cursor.py", line 85, in next
    items = self.method(page=self.current_page,
*self.args, **self.kargs)   File "build\bdist.win-amd64\egg\tweepy\binder.py", line 196, in _call
    return method.execute()   File "build\bdist.win-amd64\egg\tweepy\binder.py", line 182, in execute
    result = self.api.parser.parse(self, resp.read())   File "build\bdist.win-amd64\egg\tweepy\parsers.py", line 75, in parse
    result = model.parse_list(method.api, json)   File "build\bdist.win-amd64\egg\tweepy\models.py", line 38, in parse_list
    results.append(cls.parse(api, obj))   File "build\bdist.win-amd64\egg\tweepy\models.py", line 49, in parse
    user = User.parse(api, v)   File "build\bdist.win-amd64\egg\tweepy\models.py", line 86, in parse
    setattr(user, k, parse_datetime(v))   File "build\bdist.win-amd64\egg\tweepy\utils.py", line 17, in parse_datetime
    date = datetime(*(time.strptime(string, '%a %b %d %H:%M:%S +0000 %Y')[0:6]))   File "C:\Python27\lib\_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]   File "C:\Python27\lib\_strptime.py", line 300, in _strptime
    _TimeRE_cache = TimeRE()   File "C:\Python27\lib\_strptime.py", line 188, in __init__
    self.locale_time = LocaleTime()   File "C:\Python27\lib\_strptime.py", line 77, in __init__
    raise ValueError("locale changed during initialization") ValueError: locale changed during initialization

问题在于线程锁定,每个线程基本上都是串行运行的,并且它需要很长时间才能运行每个循环,因为这样就不再有线程了.因此,如果没有办法摆脱线程锁定,有没有办法让它更快地运行try语句中的for循环？

解决方法:

根据StackOverflow上的前一个Answer,time.strptime不是线程安全的.不幸的是,该问题中引用的错误与您遇到的错误不同.

他们的解决方案是在初始化任何线程之前调用time.strptime,然后在各种线程中对time.strptime的后续调用将起作用.

我认为在查看_strptime和locale标准库模块后,相同的解决方案可能适用于您的情况.我无法确定它是否可行,因为我无法在本地测试您的代码,但我想我会为您提供一个潜在的解决方案.

让我知道这个是否奏效.

编辑：

我做了一些研究,Python标准库在locale.h C头文件中调用setlocale.根据setlocale documentation,这不是线程安全的,并且在初始化线程之前应该调用setlocale,如前所述.

不幸的是,每次调用time.strptime时都会调用setlocale.所以,我建议如下：

>测试前面列出的解决方案,尝试在初始化线程和删除锁之前调用time.strptime.
>如果#1不起作用,您可能需要滚动自己的time.strptime函数,该函数是线程安全的,如locale模块的Python文档中所述.

标签：python,initialization,multithreading,locale,strptime
来源： https://codeday.me/bug/20190610/1211954.html