crocoite.git - Web archiving using Google Chrome

Age	Commit message (Collapse)	Author	Files	Lines
2019-07-02	Stabilize WARC headers	Lars-Dominik Braun	1	-6/+2
	In preparation for 1.0 release: - Correct mime types - Add X-Crocoite-Type, so logs, scripts, dom-snapshots and screenshots can be identified easily - Remove random WARC headers like X-Chrome-Initiator. We don’t want to maintain those. - Remove non-standard urn-based package URLs. Can’t use them without a urn-registration
2018-12-25	warc: Add tests	Lars-Dominik Braun	1	-2/+2
	Using hyothesis-based testcase generation. This is quite nice compared to manual test data generation, since it catches alot more corner cases (if done right). This commit also fixes a few issues, including: - log records will only be written if the log is nonempty - properly quote packageUrl path’s - drop old thread checking code - use placeholder url for scripts without name
2018-12-24	Use f-strings where possible	Lars-Dominik Braun	1	-2/+2
	Replaces str.format, which is less readable due to its separation of format and arguments.
2018-12-21	util: Skip missing source files	Lars-Dominik Braun	1	-1/+1
	Requirement extraction fails if the package is an .egg file (i.e. not extracted). Do not try to compute checksum/file length for them.
2018-12-21	Parse URLs by default	Lars-Dominik Braun	1	-7/+15
	Use library yarl (already pulled in by aiohttp). No URL processed should be a string.
2018-12-08	tools: Add version info to merged WARCs	Lars-Dominik Braun	1	-1/+13
	In preparation for #9. I was hoping to reuse one of schema.org’s microdata schema’s, but neither Action (archival action) nor SoftwareApplication (version information) seem to be suitable.
2018-12-01	util: Remove unused function	Lars-Dominik Braun	1	-5/+0

2018-11-19	Coding style	Lars-Dominik Braun	1	-1/+1
	Fix a few random issues pointed out by pylint, mainly unused imports.
2018-11-06	Switch single mode to asyncio	Lars-Dominik Braun	1	-2/+2
	This is a direct port to asyncio without any design changes. These need to happen in further refinements. Fixes issue #1.
2018-08-04	Add package information to warcinfo	Lars-Dominik Braun	1	-1/+44
	Change warcinfo record format to JSON (this is permitted by the specs) and add Python version, dependencies and their versions as well as file hashes. This should give us enough information to figure out the exact environment used to create the WARC.
2018-06-25	warc: Save DOM-/image screenshot as WARC conversion	Lars-Dominik Braun	1	-0/+6
	Judging from the docs this is the proper way to store these resources. Enable both for the IRC bot by default, since they won’t interfere with IA’s wayback machine.
2017-12-24	Refactor behavior scripts	Lars-Dominik Braun	1	-0/+43
	No functional changes, just cleanup. Replaces onload and onsnapshot events. Move screen metric emulation, DOM snapshots and screenshots here as well.