diff options
| author | Lars-Dominik Braun <lars@6xq.net> | 2017-12-19 10:52:23 +0100 |
|---|---|---|
| committer | Lars-Dominik Braun <lars@6xq.net> | 2017-12-19 10:52:23 +0100 |
| commit | d20528cf1f5ee85162c449d9c74c2c862ab3d4ff (patch) | |
| tree | d78cab05988dc2aa2161cd522a7c4781d487c0b6 /crocoite/data/extract-links.js | |
| parent | 8e939f8922815bd917f4dd750aa5f8a17a8f750c (diff) | |
| download | crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.gz crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.tar.bz2 crocoite-d20528cf1f5ee85162c449d9c74c2c862ab3d4ff.zip | |
Serialize WARC writing
Logger and SiteWriter both access .write_record() concurrently, which
can corrupt WARC files. Move the writer to its own thread and decouple
it with a queue. Since we’re probably I/O-bound this may speed up
writeback as well.
Diffstat (limited to 'crocoite/data/extract-links.js')
0 files changed, 0 insertions, 0 deletions