summaryrefslogtreecommitdiff
path: root/crocoite/__init__.py
diff options
context:
space:
mode:
authorLars-Dominik Braun <lars@6xq.net>2017-12-19 10:52:23 +0100
committerLars-Dominik Braun <lars@6xq.net>2017-12-19 10:52:23 +0100
commitd20528cf1f5ee85162c449d9c74c2c862ab3d4ff (patch)
treed78cab05988dc2aa2161cd522a7c4781d487c0b6 /crocoite/__init__.py
parent8e939f8922815bd917f4dd750aa5f8a17a8f750c (diff)
downloadcrocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.gz
crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.bz2
crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.zip
Serialize WARC writing
Logger and SiteWriter both access .write_record() concurrently, which can corrupt WARC files. Move the writer to its own thread and decouple it with a queue. Since we’re probably I/O-bound this may speed up writeback as well.
Diffstat (limited to 'crocoite/__init__.py')
0 files changed, 0 insertions, 0 deletions