1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
|
crocoite
========
Archive websites using `headless Google Chrome`_ and its DevTools protocol.
.. image:: https://travis-ci.org/PromyLOPh/crocoite.svg?branch=master
:target: https://travis-ci.org/PromyLOPh/crocoite
.. _headless Google Chrome: https://developers.google.com/web/updates/2017/04/headless-chrome
Quick start
-----------
The following dependencies must be present to run crocoite:
- Python 3
- pychrome_
- warcio_
- html5lib_
.. _pychrome: https://github.com/fate0/pychrome
.. _warcio: https://github.com/webrecorder/warcio
.. _html5lib: https://github.com/html5lib/html5lib-python
It is recommended to prepare a virtualenv and let pip handle the dependency
resolution for Python packages instead:
.. code:: bash
cd crocoite
virtualenv -p python3 sandbox
source sandbox/bin/activate
pip install .
One-shot commandline interface and pywb_ playback:
.. code:: bash
crocoite-grab --output example.com.warc.gz http://example.com/
rm -rf collections && wb-manager init test && wb-manager add test example.com.warc.gz
wayback &
$BROWSER http://localhost:8080
.. _pywb: https://github.com/ikreymer/pywb
Rationale
---------
Most modern websites depend heavily on executing code, usually JavaScript, on
the user’s machine. They also make use of new and emerging Web technologies
like HTML5, WebSockets, service workers and more. Even worse from the
preservation point of view, they also require some form of user interaction to
dynamically load more content (infinite scrolling, dynamic comment loading,
etc).
The naive approach of fetching a HTML page, parsing it and extracting
links to referenced resources therefore is not sufficient to create a faithful
snapshot of these web applications. A full browser, capable of running scripts and
providing modern Web API’s is absolutely required for this task. Thankfully
Google Chrome runs without a display (headless mode) and can be controlled by
external programs, allowing them to navigate and extract or inject data.
This section describes the solutions crocoite offers and explains design
decisions taken.
crocoite captures resources by listening to Chrome’s `network events`_ and
requesting the response body using `Network.getResponseBody`_. This approach
has caveats: The original HTTP requests and responses, as sent over the wire,
are not available. They are reconstructed from parsed data. The character
encoding for text documents is changed to UTF-8. And the content body of HTTP
redirects cannot be retrieved due to a race condition.
.. _network events: https://chromedevtools.github.io/devtools-protocol/1-3/Network
.. _Network.getResponseBody: https://chromedevtools.github.io/devtools-protocol/1-3/Network#method-getResponseBody
But at the same time it allows crocoite to rely on Chrome’s well-tested network
stack and HTTP parser. Thus it supports HTTP version 1 and 2 as well as
transport protocols like SSL and QUIC. Depending on Chrome also eliminates the
need for a man-in-the-middle proxy, like warcprox_, which has to decrypt SSL
traffic and present a fake certificate to the browser in order to store the
transmitted content.
.. _warcprox: https://github.com/internetarchive/warcprox
WARC records generated by crocoite therefore are an abstract view on the
resource they represent and not necessarily the data sent over the wire. A URL
fetched with HTTP/2 for example will still result in a HTTP/1.1
request/response pair in the WARC file. This may be undesireable from
an archivist’s point of view (“save the data exactly like we received it”). But
this level of abstraction is inevitable when dealing with more than one
protocol.
crocoite also interacts with and therefore alters the grabbed websites. It does
so by injecting `behavior scripts`_ into the site. Typically these are written
in JavaScript, because interacting with a page is easier this way. These
scripts then perform different tasks: Extracting targets from visible
hyperlinks, clicking buttons or scrolling the website to to load more content,
as well as taking a static screenshot of ``<canvas>`` elements for the DOM
snapshot (see below).
.. _behavior scripts: https://github.com/PromyLOPh/crocoite/tree/master/crocoite/data
Replaying archived WARC’s can be quite challenging and might not be possible
with current technology (or even at all):
- Some sites request assets based on screen resolution, pixel ratio and
supported image formats (webp). Replaying those with different parameters
won’t work, since assets for those are missing. Example: missguided.com.
- Some fetch different scripts based on user agent. Example: youtube.com.
- Requests containing randomly generated JavaScript callback function names
won’t work. Example: weather.com.
- Range requests (Range: bytes=1-100) are captured as-is, making playback
difficult
crocoite offers two methods to work around these issues. Firstly it can save a
DOM snapshot to the WARC file. It contains the entire DOM in HTML format minus
``<script>`` tags after the site has been fully loaded and thus can be
displayed without executing scripts. Obviously JavaScript-based navigation
does not work any more. Secondly it also saves a screenshot of the full page,
so even if future browsers cannot render and display the stored HTML a fully
rendered version of the website can be replayed instead.
Related projects
----------------
brozzler_
Uses Google Chrome as well, but intercepts traffic using a proxy. Supports
distributed crawling and immediate playback.
Squidwarc_
Communicates with headless Google Chrome and uses the Network API to
retrieve requests like crocoite. Supports recursive crawls and page
scrolling, but neither custom JavaScript nor distributed crawling.
.. _brozzler: https://github.com/internetarchive/brozzler
.. _Squidwarc: https://github.com/N0taN3rd/Squidwarc
|
