python-Sqlite executemany和DELETE

执行很多删除操作似乎很慢(插入很好),我想知道是否有人知道为什么要花这么长时间

考虑下面的代码

import sqlite3

db = sqlite3.connect("mydb")
c = db.cursor()
c.execute("DROP TABLE IF EXISTS testing ")
c.execute("CREATE TABLE testing (val INTEGER);")
my_vals2 = [[x] for x in range(1,10000)]

def insertmany(vals):
    c.executemany("INSERT INTO testing (val) VALUES (?)",vals)
    db.commit()

def deletemany1(vals):
    c.executemany("DELETE FROM testing WHERE val=?",vals)
    db.commit()

def deletemany2(vals): #this is fastest even though im looping over to convert to strings and again to join ...
    vals = ["'%s'"%v[0] for v in vals] 
    c.execute("DELETE FROM testing WHERE val IN (%s)"%",".join(vals))
    #DELETE FROM TABLE WHERE x in (1,2,3...)

和以下时间结果(timeit提供了有趣的数据,所以：/)来自ipython

%time insertmany(my_vals2) 
#CPU times: user 0.60 s, sys: 0.00 s, total: 0.60 s Wall time: 0.60 s

%time deletemany1(my_vals2)
#CPU times: user 3.58 s, sys: 0.00 s, total: 3.58 s Wall time: 3.58 s

%time deletemany2(my_vals2)
#CPU times: user 0.02 s, sys: 0.00 s, total: 0.02 s Wall time: 0.02 s

为了完整起见,这里是timeit结果(但我认为timeit在第二次测试时被破坏了(即ms不同于第一次测试的单位)

%timeit insertmany(my_vals2) 
#1 loops, best of 3: 358 ms per loop

%timeit deletemany1(my_vals2)
#1 loops, best of 3: 8.34 ms per loop  <- this is not faster than the above!!!! (timeit lies?)

%timeit deletemany2(my_vals2)
#100 loops, best of 3: 2.3 ms per loop  

那么,为什么executemany soooooo与deletes比较慢？

解决方法:

SQLite将表记录存储在B树中,并按rowid排序.

使用自动生成的rowid进行插入时,所有记录都将附加在表的末尾.
但是,删除时,SQLite必须首先搜索记录.如果id列未建立索引,这将很慢；创建一个明确的索引(由John提议),或将该列声明为INTEGER PRIMARY KEY以使其成为rowid.

如果不使用索引,即仅在批量插入之后创建索引,则使用索引插入会变得更快.

您的最后一条delete命令可一次删除所有记录.如果您知道要删除表中的所有记录,则可以通过仅使用DELETE FROM测试来进一步加快速度,该测试根本不需要查看任何记录.

标签：sqlite,ipython,executemany,python
来源： https://codeday.me/bug/20191031/1977085.html