crocoite.git - Web archiving using Google Chrome

Age	Commit message (Collapse)	Author	Files	Lines
2018-06-20	Synchronous SiteLoader event handling	Lars-Dominik Braun	1	-5/+4
	Previously a browser crash stalled the entire grab, since events from pychrome were handled asynchronously in a different thread and exceptions were not propagated to the main thread. Now all browser events are stored in a queue and processed by the main thread, allowing us to handle browser crashes gracefully (more or less). This made the following additional changes necessary: - Clear separation between producer (browser) and consumer (WARC, stats, …) - Behavior scripts now yield events as well, instead of accessing the WARC writer - WARC logging was removed (for now) and WARC writer does not require serialization any more
2018-05-04	sopel: Use recursive, distributed controller	Lars-Dominik Braun	1	-2/+7

2018-05-04	IRC plugin: Use argparse	Lars-Dominik Braun	1	-17/+33

2018-05-04	Move page archiving logic to SinglePageController	Lars-Dominik Braun	1	-15/+12
	In preparation for recursive crawls.
2018-02-22	irc plugin: Serialize celery operations	Lars-Dominik Braun	1	-68/+105
	This is a workaround for https://github.com/celery/celery/issues/4480
2017-12-25	Increase default body size	Lars-Dominik Braun	1	-4/+4

2017-12-24	Refactor behavior scripts	Lars-Dominik Braun	1	-10/+7
	No functional changes, just cleanup. Replaces onload and onsnapshot events. Move screen metric emulation, DOM snapshots and screenshots here as well.
2017-12-22	Add simple stats-keeping SiteLoader	Lars-Dominik Braun	1	-1/+14

2017-12-19	Select default behavior scripts by site URL	Lars-Dominik Braun	1	-2/+24

2017-12-17	Add distributed archiving	Lars-Dominik Braun	1	-0/+144
	Using celery. Also adds a plugin for the IRC bot sopel. Code still needs some love, but it should work.