summaryrefslogtreecommitdiff
path: root/crocoite
AgeCommit message (Collapse)AuthorFilesLines
2018-09-25Log extracted linksLars-Dominik Braun2-2/+25
2018-08-21Remove celery and recursionLars-Dominik Braun3-317/+23
Gonna rewrite that properly.
2018-08-17behavior: Load more comments from FacebookLars-Dominik Braun1-0/+4
2018-08-05test_browser: Properly handle failed requestsLars-Dominik Braun2-15/+14
Fixes test failures. Very fragile code unfortunately.
2018-08-04Properly handle failure to retrieve request bodyLars-Dominik Braun3-5/+50
Just truncate the WARC record like we do with responses. Also add a few tests, but they’re not covering the call to getRequestPostData. Not sure what we have to do here.
2018-08-04Reference warcinfo record in every other recordLars-Dominik Braun1-18/+30
2018-08-04Add package information to warcinfoLars-Dominik Braun3-8/+65
Change warcinfo record format to JSON (this is permitted by the specs) and add Python version, dependencies and their versions as well as file hashes. This should give us enough information to figure out the exact environment used to create the WARC.
2018-08-04Reintroduce WARC loggingLars-Dominik Braun9-76/+337
Commit 7730e0d64ec895091a0dd7eb0e3c6ce2ed02d981 removed logging to WARC files. Add it again, but with a different implementation.. Credits to structlog for inspiration.
2018-06-25browser: Fix testcase race conditionLars-Dominik Braun1-0/+4
2018-06-25warc: Add metadata to truncated recordsLars-Dominik Braun1-22/+28
Specifically for a) redirects (body missing) b) bodies larger than size limit and c) whenever we couldn’t fetch the response body for whatever reason. We gave it our best shot, but still failed miserably. Future generations will certainly appreciate that. Eh, maybe. Hopefully. Will they?
2018-06-25warc: Save DOM-/image screenshot as WARC conversionLars-Dominik Braun6-37/+72
Judging from the docs this is the proper way to store these resources. Enable both for the IRC bot by default, since they won’t interfere with IA’s wayback machine.
2018-06-21Fix a few issues pointed out by pylintLars-Dominik Braun5-22/+10
2018-06-21browser: Add a few more testsLars-Dominik Braun1-3/+31
Increase coverage.
2018-06-20Move tests to pytestLars-Dominik Braun2-162/+177
It just seems a little nicer than plain old unittest
2018-06-20Add __slots__ to classesLars-Dominik Braun5-1/+56
This is mainly a quality of life change
2018-06-20Synchronous SiteLoader event handlingLars-Dominik Braun6-509/+514
Previously a browser crash stalled the entire grab, since events from pychrome were handled asynchronously in a different thread and exceptions were not propagated to the main thread. Now all browser events are stored in a queue and processed by the main thread, allowing us to handle browser crashes gracefully (more or less). This made the following additional changes necessary: - Clear separation between producer (browser) and consumer (WARC, stats, …) - Behavior scripts now yield events as well, instead of accessing the WARC writer - WARC logging was removed (for now) and WARC writer does not require serialization any more
2018-06-08browser: Replace --remote-debugging-socket-fdLars-Dominik Braun1-23/+19
It was replaced by --remote-debugging-pipe in version 67. pychrome does not support that out of the box, so instead we’ll let Chrome choose its own port and poll a file in its user-data-dir.
2018-06-03behavior: Wrap extract links script in anonymous namespaceLars-Dominik Braun2-2/+5
Otherwise it may clash with symbols defined by the page.
2018-05-20behavior: Patreon: Load more comments/repliesLars-Dominik Braun1-0/+4
2018-05-20behavior: Click Patreon’s “load more” buttonLars-Dominik Braun1-0/+6
2018-05-05Rename command line toolsLars-Dominik Braun1-0/+97
Move contrib/ scripts to .tools and add entry points to setup.py, rename crocoite-standalone to crocoite-grab.
2018-05-05Extract only visible and clickable linksLars-Dominik Braun2-4/+29
2018-05-04Share recursive argument parserLars-Dominik Braun2-14/+15
2018-05-04Support --browser again for local crawlsLars-Dominik Braun2-2/+6
Broken by commit 75019eac4545bb2e8b90033834e91beef614cdf3
2018-05-04Add distributed recursive crawlsLars-Dominik Braun3-31/+91
2018-05-04Add support for recursive crawlsLars-Dominik Braun2-2/+115
Only local right now, not distributed.
2018-05-04browser: Replace context manager decoratorLars-Dominik Braun1-51/+66
Use an actual class that supports multiple invokations.
2018-05-04behavior: Add link extraction scriptLars-Dominik Braun4-5/+43
2018-05-04Move page archiving logic to SinglePageControllerLars-Dominik Braun5-144/+198
In preparation for recursive crawls.
2018-05-04Move header unfolding into ItemLars-Dominik Braun2-21/+24
2018-05-04Fetch request POST bodyLars-Dominik Braun2-8/+20
If there is any and it was not included in the response already.
2018-05-04Test chained redirectsLars-Dominik Braun1-12/+32
2018-04-20Save screenshot of entire pageLars-Dominik Braun1-6/+16
…and not just the current viewport. Due to limitations within Chrome it may be necessary to manually stitch multiple images if the page height exceeds 16k pixels.
2018-04-14Fix base64 body detectionLars-Dominik Braun2-10/+10
Broken by commit a21d7332e33a3e47a363004196451721d449e70b
2018-04-14Add timeout to request body fetchLars-Dominik Braun1-3/+4
When something goes wrong, these block the entire grab.
2018-04-14Handle JavaScript dialogsLars-Dominik Braun1-2/+37
alert, confirm and prompt and beforeunload
2018-04-04behavior: Add selector for YouTube.Lars-Dominik Braun1-0/+6
2018-03-30Add click selectors for InstagramLars-Dominik Braun1-0/+8
Load more comments/images for posts.
2018-03-25Add a few simple testsLars-Dominik Braun1-0/+190
To be expanded, but it’s a start…
2018-03-25Replace deprecated logger.warnLars-Dominik Braun1-3/+3
2018-03-25ChromeService: Close listening socketLars-Dominik Braun1-0/+1
We passed it to the child and don’t need it any more.
2018-03-25Move getResponseBody call to Item wrapperLars-Dominik Braun2-13/+21
2018-03-18browser: Don’t overwrite LogEntry’s argsLars-Dominik Braun1-1/+1
2018-03-18behavior: Add click selectors for redditLars-Dominik Braun1-7/+27
This is slightly obnoxious, since their JavaScript rate-limits clicks to ≤3 Hz and simply ignores everything beyond that.
2018-03-05Add generic click behavior scriptLars-Dominik Braun3-37/+119
Configureable. Clicks elements matching one (or more) CSS selectors once or multiple times. Currently supported: Facebook, Twitter, Disqus (embedded iframe)
2018-03-04Remove instagram behavior scriptLars-Dominik Braun2-27/+1
The “load more” button does not exist any more.
2018-01-20behavior: Scroll all DOM elementsLars-Dominik Braun1-0/+6
One example is Twitter, which uses a popover div for individual tweets. Scrolling the page won’t scroll that div’s content, which is required to load more replies.
2018-01-20twitter: Expand “more replies” linksLars-Dominik Braun1-8/+21
Click them periodically.
2017-12-27Log messages from browser consoleLars-Dominik Braun1-0/+12
2017-12-25Increase default body sizeLars-Dominik Braun3-5/+34
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-27 11:23:14 +0000