summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2018-05-04Move page archiving logic to SinglePageControllerLars-Dominik Braun7-160/+211
In preparation for recursive crawls.
2018-05-04Move header unfolding into ItemLars-Dominik Braun2-21/+24
2018-05-04Fetch request POST bodyLars-Dominik Braun2-8/+20
If there is any and it was not included in the response already.
2018-05-04Test chained redirectsLars-Dominik Braun1-12/+32
2018-04-20Add screenshot extraction script to contrib/Lars-Dominik Braun1-0/+54
2018-04-20Save screenshot of entire pageLars-Dominik Braun1-6/+16
…and not just the current viewport. Due to limitations within Chrome it may be necessary to manually stitch multiple images if the page height exceeds 16k pixels.
2018-04-14Fix base64 body detectionLars-Dominik Braun2-10/+10
Broken by commit a21d7332e33a3e47a363004196451721d449e70b
2018-04-14Add timeout to request body fetchLars-Dominik Braun1-3/+4
When something goes wrong, these block the entire grab.
2018-04-14Handle JavaScript dialogsLars-Dominik Braun1-2/+37
alert, confirm and prompt and beforeunload
2018-04-04behavior: Add selector for YouTube.Lars-Dominik Braun1-0/+6
2018-03-30Add click selectors for InstagramLars-Dominik Braun1-0/+8
Load more comments/images for posts.
2018-03-29Travis: Run tests with pypy3Lars-Dominik Braun1-0/+1
2018-03-29Use setuptoolsLars-Dominik Braun1-1/+1
2018-03-25Add Travis CILars-Dominik Braun2-1/+17
2018-03-25Add a few simple testsLars-Dominik Braun1-0/+190
To be expanded, but it’s a start…
2018-03-25Replace deprecated logger.warnLars-Dominik Braun1-3/+3
2018-03-25ChromeService: Close listening socketLars-Dominik Braun1-0/+1
We passed it to the child and don’t need it any more.
2018-03-25Move getResponseBody call to Item wrapperLars-Dominik Braun2-13/+21
2018-03-18browser: Don’t overwrite LogEntry’s argsLars-Dominik Braun1-1/+1
2018-03-18behavior: Add click selectors for redditLars-Dominik Braun1-7/+27
This is slightly obnoxious, since their JavaScript rate-limits clicks to ≤3 Hz and simply ignores everything beyond that.
2018-03-05Add generic click behavior scriptLars-Dominik Braun3-37/+119
Configureable. Clicks elements matching one (or more) CSS selectors once or multiple times. Currently supported: Facebook, Twitter, Disqus (embedded iframe)
2018-03-04Remove instagram behavior scriptLars-Dominik Braun2-27/+1
The “load more” button does not exist any more.
2018-02-23README: Add Squidwarc to related projectsLars-Dominik Braun1-0/+5
2018-02-22irc plugin: Serialize celery operationsLars-Dominik Braun1-68/+105
This is a workaround for https://github.com/celery/celery/issues/4480
2018-01-20behavior: Scroll all DOM elementsLars-Dominik Braun1-0/+6
One example is Twitter, which uses a popover div for individual tweets. Scrolling the page won’t scroll that div’s content, which is required to load more replies.
2018-01-20twitter: Expand “more replies” linksLars-Dominik Braun1-8/+21
Click them periodically.
2017-12-27Log messages from browser consoleLars-Dominik Braun1-0/+12
2017-12-25Increase default body sizeLars-Dominik Braun4-9/+38
2017-12-24Refactor behavior scriptsLars-Dominik Braun8-194/+302
No functional changes, just cleanup. Replaces onload and onsnapshot events. Move screen metric emulation, DOM snapshots and screenshots here as well.
2017-12-23Set fake finished response for redirectsLars-Dominik Braun1-1/+4
Fixes bcfbdd9b45b7e872ee77e1366197443d855d8c7c
2017-12-23Drain tab event queue before stoppingLars-Dominik Braun1-0/+2
2017-12-22Add simple stats-keeping SiteLoaderLars-Dominik Braun4-10/+60
2017-12-22SiteLoader: Save entire finished responseLars-Dominik Braun1-2/+9
2017-12-22Don’t write WARC record if body cannot be retrievedLars-Dominik Braun1-19/+48
+refactoring.
2017-12-20Increase hardcoded max timeoutsLars-Dominik Braun1-2/+2
We need a better solution for this. Sites loading a lot of responsive images easily need a minute after resizing.
2017-12-20Fix HTTP headers using the same key more than onceLars-Dominik Braun1-2/+15
This is an undocumented DevTools feature.
2017-12-19Serialize WARC writingLars-Dominik Braun2-3/+38
Logger and SiteWriter both access .write_record() concurrently, which can corrupt WARC files. Move the writer to its own thread and decouple it with a queue. Since we’re probably I/O-bound this may speed up writeback as well.
2017-12-19Select default behavior scripts by site URLLars-Dominik Braun5-3/+75
2017-12-18README: Add related projectLars-Dominik Braun1-0/+9
2017-12-17Add Twitter fixupsLars-Dominik Braun1-0/+17
2017-12-17Extend READMELars-Dominik Braun1-10/+25
2017-12-17Don’t fetch redirected request bodyLars-Dominik Braun2-8/+13
We can’t do that safely due to a race-condition.
2017-12-17Add distributed archivingLars-Dominik Braun5-154/+407
Using celery. Also adds a plugin for the IRC bot sopel. Code still needs some love, but it should work.
2017-12-06Start Chrome browser instanceLars-Dominik Braun2-44/+101
Unless --browser argument is given. Uses sane settings and a temporary profile directory.
2017-12-06Add flags to disable screenshot/DOM snapshotLars-Dominik Braun1-5/+9
2017-12-03Add note about Range: requestsLars-Dominik Braun1-0/+2
2017-12-03Fix UTF-8 encoding nameLars-Dominik Braun1-1/+1
HTMLSerializer uses the exact string given in <meta charset=X>, thus it should be with hyphen.
2017-12-03Add page screenshot to WARCLars-Dominik Braun1-0/+14
2017-11-29Add missing timestamp to response data for redirectsLars-Dominik Braun1-1/+1
Fixes 6f628ca24ac2b243dd4a611ff1ecff2d35aaa019
2017-11-29argparse: Add metavarLars-Dominik Braun1-7/+7
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-01 07:30:41 +0000