diff options
author | Lars-Dominik Braun <lars@6xq.net> | 2018-06-20 11:13:37 +0200 |
---|---|---|
committer | Lars-Dominik Braun <lars@6xq.net> | 2018-06-20 11:17:25 +0200 |
commit | 7730e0d64ec895091a0dd7eb0e3c6ce2ed02d981 (patch) | |
tree | 15d0ca2e0374b7d00a05d5dd5de1e48838e71feb /crocoite/data/extract-links.js | |
parent | 06a06463c0367718b2ed1b2b7f081cff6ca998a0 (diff) | |
download | crocoite-7730e0d64ec895091a0dd7eb0e3c6ce2ed02d981.tar.gz crocoite-7730e0d64ec895091a0dd7eb0e3c6ce2ed02d981.tar.bz2 crocoite-7730e0d64ec895091a0dd7eb0e3c6ce2ed02d981.zip |
Synchronous SiteLoader event handling
Previously a browser crash stalled the entire grab, since events from
pychrome were handled asynchronously in a different thread and
exceptions were not propagated to the main thread.
Now all browser events are stored in a queue and processed by the main
thread, allowing us to handle browser crashes gracefully (more or less).
This made the following additional changes necessary:
- Clear separation between producer (browser) and consumer (WARC, stats,
…)
- Behavior scripts now yield events as well, instead of accessing the
WARC writer
- WARC logging was removed (for now) and WARC writer does not require
serialization any more
Diffstat (limited to 'crocoite/data/extract-links.js')
0 files changed, 0 insertions, 0 deletions