diff options
author | Lars-Dominik Braun <lars@6xq.net> | 2017-12-19 10:52:23 +0100 |
---|---|---|
committer | Lars-Dominik Braun <lars@6xq.net> | 2017-12-19 10:52:23 +0100 |
commit | d20528cf1f5ee85162c449d9c74c2c862ab3d4ff (patch) | |
tree | d78cab05988dc2aa2161cd522a7c4781d487c0b6 /crocoite/__init__.py | |
parent | 8e939f8922815bd917f4dd750aa5f8a17a8f750c (diff) | |
download | crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.gz crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.bz2 crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.zip |
Serialize WARC writing
Logger and SiteWriter both access .write_record() concurrently, which
can corrupt WARC files. Move the writer to its own thread and decouple
it with a queue. Since we’re probably I/O-bound this may speed up
writeback as well.
Diffstat (limited to 'crocoite/__init__.py')
0 files changed, 0 insertions, 0 deletions