crocoite.git - Web archiving using Google Chrome

Age	Commit message (Collapse)	Author	Files	Lines
2018-06-20	Add __slots__ to classes	Lars-Dominik Braun	1	-0/+22
	This is mainly a quality of life change
2018-06-20	Synchronous SiteLoader event handling	Lars-Dominik Braun	1	-99/+161
	Previously a browser crash stalled the entire grab, since events from pychrome were handled asynchronously in a different thread and exceptions were not propagated to the main thread. Now all browser events are stored in a queue and processed by the main thread, allowing us to handle browser crashes gracefully (more or less). This made the following additional changes necessary: - Clear separation between producer (browser) and consumer (WARC, stats, …) - Behavior scripts now yield events as well, instead of accessing the WARC writer - WARC logging was removed (for now) and WARC writer does not require serialization any more
2018-05-05	Extract only visible and clickable links	Lars-Dominik Braun	1	-1/+1

2018-05-04	Add distributed recursive crawls	Lars-Dominik Braun	1	-5/+17

2018-05-04	Add support for recursive crawls	Lars-Dominik Braun	1	-0/+100
	Only local right now, not distributed.
2018-05-04	behavior: Add link extraction script	Lars-Dominik Braun	1	-1/+11

2018-05-04	Move page archiving logic to SinglePageController	Lars-Dominik Braun	1	-0/+103
	In preparation for recursive crawls.