summaryrefslogtreecommitdiff
path: root/crocoite/browser.py
AgeCommit message (Collapse)AuthorFilesLines
2018-05-04Support --browser again for local crawlsLars-Dominik Braun1-1/+1
Broken by commit 75019eac4545bb2e8b90033834e91beef614cdf3
2018-05-04browser: Replace context manager decoratorLars-Dominik Braun1-51/+66
Use an actual class that supports multiple invokations.
2018-05-04Move header unfolding into ItemLars-Dominik Braun1-0/+22
2018-05-04Fetch request POST bodyLars-Dominik Braun1-1/+15
If there is any and it was not included in the response already.
2018-05-04Test chained redirectsLars-Dominik Braun1-12/+32
2018-04-14Fix base64 body detectionLars-Dominik Braun1-9/+9
Broken by commit a21d7332e33a3e47a363004196451721d449e70b
2018-04-14Add timeout to request body fetchLars-Dominik Braun1-3/+4
When something goes wrong, these block the entire grab.
2018-04-14Handle JavaScript dialogsLars-Dominik Braun1-2/+37
alert, confirm and prompt and beforeunload
2018-03-25Add a few simple testsLars-Dominik Braun1-0/+190
To be expanded, but it’s a start…
2018-03-25Replace deprecated logger.warnLars-Dominik Braun1-3/+3
2018-03-25ChromeService: Close listening socketLars-Dominik Braun1-0/+1
We passed it to the child and don’t need it any more.
2018-03-25Move getResponseBody call to Item wrapperLars-Dominik Braun1-2/+19
2018-03-18browser: Don’t overwrite LogEntry’s argsLars-Dominik Braun1-1/+1
2017-12-27Log messages from browser consoleLars-Dominik Braun1-0/+12
2017-12-23Set fake finished response for redirectsLars-Dominik Braun1-1/+4
Fixes bcfbdd9b45b7e872ee77e1366197443d855d8c7c
2017-12-23Drain tab event queue before stoppingLars-Dominik Braun1-0/+2
2017-12-22Add simple stats-keeping SiteLoaderLars-Dominik Braun1-2/+33
2017-12-22SiteLoader: Save entire finished responseLars-Dominik Braun1-2/+9
2017-12-17Add distributed archivingLars-Dominik Braun1-6/+15
Using celery. Also adds a plugin for the IRC bot sopel. Code still needs some love, but it should work.
2017-12-06Start Chrome browser instanceLars-Dominik Braun1-0/+52
Unless --browser argument is given. Uses sane settings and a temporary profile directory.
2017-11-29Add missing timestamp to response data for redirectsLars-Dominik Braun1-1/+1
Fixes 6f628ca24ac2b243dd4a611ff1ecff2d35aaa019
2017-11-29Use Chrome’s timestamps as WARC-DateLars-Dominik Braun1-8/+8
2017-11-29RefactoringLars-Dominik Braun1-0/+209
Reusable browser communication and WARC writing.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-05 22:12:11 +0000