Age | Commit message (Collapse) | Author | Files | Lines | |
---|---|---|---|---|---|
2017-12-22 | Add simple stats-keeping SiteLoader | Lars-Dominik Braun | 4 | -10/+60 | |
2017-12-22 | SiteLoader: Save entire finished response | Lars-Dominik Braun | 1 | -2/+9 | |
2017-12-22 | Don’t write WARC record if body cannot be retrieved | Lars-Dominik Braun | 1 | -19/+48 | |
+refactoring. | |||||
2017-12-20 | Increase hardcoded max timeouts | Lars-Dominik Braun | 1 | -2/+2 | |
We need a better solution for this. Sites loading a lot of responsive images easily need a minute after resizing. | |||||
2017-12-20 | Fix HTTP headers using the same key more than once | Lars-Dominik Braun | 1 | -2/+15 | |
This is an undocumented DevTools feature. | |||||
2017-12-19 | Serialize WARC writing | Lars-Dominik Braun | 2 | -3/+38 | |
Logger and SiteWriter both access .write_record() concurrently, which can corrupt WARC files. Move the writer to its own thread and decouple it with a queue. Since we’re probably I/O-bound this may speed up writeback as well. | |||||
2017-12-19 | Select default behavior scripts by site URL | Lars-Dominik Braun | 5 | -3/+75 | |
2017-12-18 | README: Add related project | Lars-Dominik Braun | 1 | -0/+9 | |
2017-12-17 | Add Twitter fixups | Lars-Dominik Braun | 1 | -0/+17 | |
2017-12-17 | Extend README | Lars-Dominik Braun | 1 | -10/+25 | |
2017-12-17 | Don’t fetch redirected request body | Lars-Dominik Braun | 2 | -8/+13 | |
We can’t do that safely due to a race-condition. | |||||
2017-12-17 | Add distributed archiving | Lars-Dominik Braun | 5 | -154/+407 | |
Using celery. Also adds a plugin for the IRC bot sopel. Code still needs some love, but it should work. | |||||
2017-12-06 | Start Chrome browser instance | Lars-Dominik Braun | 2 | -44/+101 | |
Unless --browser argument is given. Uses sane settings and a temporary profile directory. | |||||
2017-12-06 | Add flags to disable screenshot/DOM snapshot | Lars-Dominik Braun | 1 | -5/+9 | |
2017-12-03 | Add note about Range: requests | Lars-Dominik Braun | 1 | -0/+2 | |
2017-12-03 | Fix UTF-8 encoding name | Lars-Dominik Braun | 1 | -1/+1 | |
HTMLSerializer uses the exact string given in <meta charset=X>, thus it should be with hyphen. | |||||
2017-12-03 | Add page screenshot to WARC | Lars-Dominik Braun | 1 | -0/+14 | |
2017-11-29 | Add missing timestamp to response data for redirects | Lars-Dominik Braun | 1 | -1/+1 | |
Fixes 6f628ca24ac2b243dd4a611ff1ecff2d35aaa019 | |||||
2017-11-29 | argparse: Add metavar | Lars-Dominik Braun | 1 | -7/+7 | |
2017-11-29 | Use Chrome’s timestamps as WARC-Date | Lars-Dominik Braun | 2 | -8/+14 | |
2017-11-29 | Refactoring | Lars-Dominik Braun | 5 | -403/+571 | |
Reusable browser communication and WARC writing. | |||||
2017-11-26 | DOM snapshot: Generate valid HTML5 | Lars-Dominik Braun | 2 | -9/+31 | |
Some tags are “void”, i.e. cannot contain contents and don’t have a closing tag. | |||||
2017-11-25 | Ignore duplicate URLs when saving DOM snapshot | Lars-Dominik Braun | 1 | -1/+10 | |
2017-11-25 | Workaround broken device metrics reset | Lars-Dominik Braun | 1 | -1/+3 | |
Apparently neither width=0, height=0 nor clearDeviceMetricsOverride() do what they should, so manually reset to 1080p screen size. | |||||
2017-11-25 | Strip on* HTML attributes | Lars-Dominik Braun | 2 | -1/+111 | |
They can carry JavaScript as well and should not be allowed for DOM snapshots. | |||||
2017-11-25 | Rename --run-before-snapshot and document --on* options | Lars-Dominik Braun | 2 | -4/+20 | |
2017-11-24 | DOM snapshot: Save frames/subdocuments as well | Lars-Dominik Braun | 1 | -13/+36 | |
Request all subdocuments with pierce=True, split the result and save each document. Playback with pywb works, because timestamps of the snapshots are close to each other. | |||||
2017-11-24 | Reset device metrics | Lars-Dominik Braun | 1 | -2/+5 | |
2017-11-24 | Save onsnapshot script to WARC | Lars-Dominik Braun | 1 | -4/+8 | |
2017-11-22 | Make <canvas> static before DOM snapshot | Lars-Dominik Braun | 3 | -9/+31 | |
Use --run-before-snapshot=canvas-snapshot.js. Replaces <canvas> with image snapshot. We could use .captureStream() as well. | |||||
2017-11-22 | Emulate different screen sizes | Lars-Dominik Braun | 2 | -3/+25 | |
Causes the browser to load CSS assets and <img> srcset, for example. | |||||
2017-11-22 | Add example fixups for Instagram | Lars-Dominik Braun | 3 | -3/+32 | |
2017-11-21 | Move base64 metadata into WARC header | Lars-Dominik Braun | 1 | -1/+1 | |
2017-11-21 | Graceful page load timeout | Lars-Dominik Braun | 2 | -11/+32 | |
Stop scrolling script, wait for remaining resources to load. | |||||
2017-11-20 | Add page created from DOM snapshot | Lars-Dominik Braun | 3 | -9/+119 | |
2017-11-17 | Initial import | Lars-Dominik Braun | 7 | -0/+419 | |