httrack - offline browser : copy websites to a local directory
httrack [ url ]... [ -filter ]... [ +filter ]... [ -O, --path ] [ -w, --mirror ] [ -W, --mirror-wizard ] [ -g, --get-files ] [ -i, --continue ] [ -Y, --mirrorlinks ] [ -P, --proxy ] [ -%f, --httpproxy-ftp[=0] ] [ -%b, --bind ] [ -rN, --depth[=N] ] [ -%eN, --ext-depth[=N] ] [ -mN, --max-files[=N] ] [ -MN, --max-size[=N] ] [ -EN, --max-time[=N] ] [ -AN, --max-rate[=N] ] [ -%cN, --connection-per-second[=N] ] [ -%G, --pause ] [ -%JN, --max-retry-after[=N] ] [ -GN, --max-pause[=N] ] [ -cN, --sockets[=N] ] [ -TN, --timeout[=N] ] [ -RN, --retries[=N] ] [ -JN, --min-rate[=N] ] [ -HN, --host-control[=N] ] [ -%P, --extended-parsing[=0] ] [ -n, --near ] [ -t, --test ] [ -%L, --list ] [ -%S, --urllist ] [ -%m, --sitemap ] [ -NN, --structure[=N] ] [ -%N, --delayed-type-check[=N] ] [ -%D, --cached-delayed-type-check[=0] ] [ -%M, --mime-html[=0] ] [ -%Z, --single-file[=0] ] [ -LN, --long-names[=N] ] [ -KN, --keep-links[=N] ] [ -x, --replace-external ] [ -%x, --disable-passwords[=0] ] [ -%q, --include-query-string[=0] ] [ -%g, --strip-query ] [ -%C, --host-alias ] [ -o, --generate-errors[=N] ] [ -X, --purge-old[=0] ] [ -%p, --preserve ] [ -%T, --utf8-conversion[=0] ] [ -bN, --cookies[=N] ] [ -%K, --cookies-file ] [ -%Y, --why ] [ -u, --check-type[=N] ] [ -j, --parse-java[=N] ] [ -sN, --robots[=N] ] [ -%h, --http-10[=0] ] [ -%k, --keep-alive[=0] ] [ -%z, --disable-compression[=0] ] [ -%B, --tolerant[=0] ] [ -%s, --updatehack[=0] ] [ -%u, --urlhack[=0] ] [ -%A, --assume ] [ -@iN, --protocol[=N] ] [ -@mN, --mptcp[=0] ] [ -%w, --disable-module ] [ -F, --user-agent ] [ -%R, --referer ] [ -%E, --from ] [ -%F, --footer ] [ -%l, --language ] [ -%a, --accept ] [ -%X, --headers ] [ -%Q, --browser-headers[=0] ] [ -C, --cache[=N] ] [ -k, --store-all-in-cache ] [ -%r, --warc ] [ -%d, --changes[=0] ] [ -%n, --do-not-recatch[=0] ] [ -%v, --display[=N] ] [ -Q, --do-not-log ] [ -q, --quiet ] [ -z, --extra-log ] [ -Z, --debug-log ] [ -v, --verbose ] [ -f, --file-log ] [ -f2, --single-log ] [ -I, --index[=0] ] [ -%i, --build-top-index[=0] ] [ -%I, --search-index[=N] ] [ -pN, --priority[=N] ] [ -S, --stay-on-same-dir ] [ -D, --can-go-down ] [ -U, --can-go-up ] [ -B, --can-go-up-and-down ] [ -a, --stay-on-same-address ] [ -d, --stay-on-same-domain ] [ -l, --stay-on-same-tld ] [ -e, --go-everywhere ] [ -%H, --debug-headers ] [ -%!, --disable-security-limits[=0] ] [ -V, --userdef-cmd ] [ -%W, --callback ] [ -y, --background-on-suspend[=0] ] [ -K, --keep-links[=N] ]
httrack allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site’s relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads.
httrack www.example.com/bob/
mirror site www.example.com/bob/ and only this site
httrack www.example.com/bob/
other.example.com/mike/ +*.com/*.jpg
-mime:application/*
mirror the two sites together (with shared links) and accept any .jpg files on .com sites
httrack
www.example.com/bob/bobby.html +* -r6
httrack www.example.com/bob/bobby.html --spider -P
proxy.example.com:8080
httrack --update
httrack
httrack --continue
|
-O |
path for mirror/logfiles+cache (-O path_mirror[,path_cache_and_logfiles]) (--path <param>) |
|
-w |
*mirror web sites (--mirror) | ||
|
-W |
mirror web sites, semi-automatic (asks questions) (--mirror-wizard) | ||
|
-g |
just get files (saved in the current directory) (--get-files) | ||
|
-i |
continue an interrupted mirror using the cache (--continue) | ||
|
-Y |
mirror ALL links located in the first level pages (mirror links) (--mirrorlinks) |
|
-P |
proxy use (-P [socks5://|connect://][user:pass@]proxy:port) (--proxy <param>) | ||
|
-%f |
*use proxy for ftp (f0 don’t use) (--httpproxy-ftp[=0]) | ||
|
-%b |
use this local hostname to make/send requests (-%b hostname) (--bind <param>) |
|
-rN |
set the mirror depth to N (* r9999) (--depth[=N]) | ||
|
-%eN |
set the external links depth to N (* %e0) (--ext-depth[=N]) | ||
|
-mN |
maximum file length for a non-html file (--max-files[=N]) | ||
|
-mN,N2 |
maximum file length for non html (N) and html (N2) | ||
|
-MN |
maximum overall size that can be uploaded/scanned (--max-size[=N]) | ||
|
-EN |
maximum mirror time in seconds (60=1 minute, 3600=1 hour) (--max-time[=N]) | ||
|
-AN |
maximum transfer rate in bytes/seconds (1000=1KB/s max) (--max-rate[=N]) | ||
|
-%cN |
maximum number of connections/seconds (*%c5) (--connection-per-second[=N]) | ||
|
-%G |
random pause of MIN[:MAX] seconds between files (e.g. %G5:10) (--pause <param>) | ||
|
-%JN |
longest Retry-After obeyed, in seconds (*%J60, 0=no wait) (--max-retry-after[=N]) | ||
|
-GN |
pause transfer if N bytes reached, and wait until lock file is deleted (--max-pause[=N]) |
|
-cN |
number of multiple connections (*c4) (--sockets[=N]) | ||
|
-TN |
timeout, number of seconds after a non-responding link is shutdown; also bounds host name resolution (--timeout[=N]) | ||
|
-RN |
number of retries, in case of timeout or non-fatal errors (*R2) (--retries[=N]) | ||
|
-JN |
traffic jam control, minimum transfert rate (bytes/seconds) tolerated for a link (--min-rate[=N]) | ||
|
-HN |
host is abandoned if: 0=never, 1=timeout, 2=slow, 3=timeout or slow (--host-control[=N]) |
|
-%P |
*extended parsing, attempt to parse all links, even in unknown tags or Javascript (%P0 don’t use) (--extended-parsing[=0]) | ||
|
-n |
get non-html files ’near’ an html file (ex: an image located outside) (--near) | ||
|
-t |
test all URLs (even forbidden ones) (--test) | ||
|
-%L |
<file> add all URL located in this text file (one URL per line) (--list <param>) | ||
|
-%S |
<file> add all scan rules located in this text file (one scan rule per line) (--urllist <param>) | ||
|
-%m |
seed the crawl from the site’s sitemap (robots.txt Sitemap:, then /sitemap.xml); --sitemap-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (--sitemap) |
|
-NN |
structure type (0 *original structure, 1+: see below) (--structure[=N]) |
or user defined structure (-N
"%h%p/%n%q.%t", --structure or --user-structure)
--structure takes one preset or template: -N1L0 clusters
more short options, --structure=1L0 is refused
|
-%N |
delayed type check, don’t make any link test but wait for files download to start instead (experimental) (%N0 don’t use, %N1 use for unknown extensions, * %N2 always use) (--delayed-type-check[=N]) | ||
|
-%D |
cached delayed type check, don’t wait for remote type during updates, to speedup them (%D0 wait, * %D1 don’t wait) (--cached-delayed-type-check[=0]) | ||
|
-%M |
generate a RFC MIME-encapsulated full-archive (.mht) (--mime-html[=0]) | ||
|
-%Z |
after the mirror, rewrite each saved page with its stylesheets, scripts, images and fonts inlined as data: URIs, so any page opens by double-click anywhere (links between pages stay relative; audio and video stay links); --single-file-max-size N caps each asset (default 10485760 bytes). %M is the better container where a Chromium-family browser is a given: one archive, no base64 tax on text, a shared asset stored once (--single-file[=0]) | ||
|
-%t |
keep the original file extension, don’t rewrite it from the MIME type (%t0 rewrite) | ||
|
-LN |
long names (L1 *long names / L0 8-3 conversion / L2 ISO9660 compatible) (--long-names[=N]) | ||
|
-KN |
keep original links (e.g. http://www.adr/link) (K0 *relative link, K absolute links, K4 original links, K3 absolute URI links, K5 transparent proxy link) (--keep-links[=N]) | ||
|
-x |
replace external html links by error pages (--replace-external) | ||
|
-%x |
do not include any password for external password protected websites (%x0 include) (--disable-passwords[=0]) | ||
|
-%q |
*include query string for local files (useless, for information purpose only) (%q0 don’t include) (--include-query-string[=0]) | ||
|
-%g |
strip query keys for dedup ([host/pattern=]key1,key2,...) (--strip-query <param>) | ||
|
-%C |
fold other hostnames of one site onto it ([scheme://]alias[,...]=[scheme://]host) (--host-alias <param>) | ||
|
-o |
*save the server’s error pages (404..) (o0 discard them) (--generate-errors[=N]) | ||
|
-X |
*purge old files after update (X0 keep delete) (--purge-old[=0]) | ||
|
-%p |
preserve html files ’as is’ (identical to ’-K4 -%F ""’) (--preserve) | ||
|
-%T |
links conversion to UTF-8 (--utf8-conversion[=0]) |
|
-bN |
accept cookies in cookies.txt (0=do not accept,* 1=accept) (--cookies[=N]) | ||
|
-%K |
use a Netscape cookies.txt as the jar, rewritten at the end (--cookies-file <param>) |
<URL>#httrack-bundle=<name>
crawls <URL> with the user agent,
headers and cookies of the login bundle <name>.hts,
read from the project folder or from --bundle-dir
<dir>
|
-%Y |
explain which filter rule accepts or rejects a URL, then exit (--why <param>) | ||
|
-u |
check document type if unknown (cgi,asp..) (u0 don’t check, * u1 check but /, u2 check always) (--check-type[=N]) | ||
|
-j |
*parse scripts (j0 don’t parse, bitmask: |1 parse default, |4 don’t parse .js |8 don’t be aggressive) (--parse-java[=N]) | ||
|
-sN |
follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always, 3=always (even strict rules)) (--robots[=N]) | ||
|
-%h |
force HTTP/1.0 requests (reduce update features, only for old servers or proxies) (--http-10[=0]) | ||
|
-%k |
use keep-alive if possible, greately reducing latency for small files and test requests (%k0 don’t use) (--keep-alive[=0]) | ||
|
-%z |
do not request compressed content (%z0 request) (--disable-compression[=0]) | ||
|
-%B |
tolerant requests (accept bogus responses on some servers, but not standard!) (--tolerant[=0]) | ||
|
-%s |
update hacks: keep a file already on disk when its size matches, even on a first crawl, and accept some bogus responses (--updatehack[=0]) | ||
|
-%u |
url hacks: various hacks to limit duplicate URLs (strip //, www.foo.com==foo.com..) (--urlhack[=0]) |
opt out of one url-hack part: --keep-www-prefix (www.foo.com<>foo.com), --keep-double-slashes (//), --keep-query-order (?b&a)
|
-%A |
assume that a type (cgi,asp..) is always linked with a mime type (-%A php3,cgi=text/html;dat,bin=application/x-zip) (--assume <param>) |
shortcut: ’--assume
standard’ is equivalent to -%A php2 php3 php4 php cgi
asp jsp pl cfm nsf=text/html
can also be used to force a specific file type: --assume
foo.cgi=text/html
|
-@iN |
internet protocol (0=both ipv6+ipv4, 4=ipv4 only, 6=ipv6 only) (--protocol[=N]) | ||
|
-@mN |
carry connections over Multipath TCP where the system offers it (0=no, 1=yes) (--mptcp[=0]) | ||
|
-%w |
disable a specific external mime module (-%w httrack-plugin) (--disable-module <param>) |
|
-F |
user-agent field sent in HTTP headers (-F "user-agent name") (--user-agent <param>) | ||
|
-%R |
default referer field sent in HTTP headers (--referer <param>) | ||
|
-%E |
from email address sent in HTTP headers (--from <param>) | ||
|
-%F |
footer string in Html code (-%F "Mirrored from {url} on {date}"; fields {addr} {path} {url} {date} {lastmodified} {version} {mime} {charset} {status} {size}, or legacy %s) (--footer <param>) | ||
|
-%l |
preferred language (-%l "fr, en, jp, *" (--language <param>) | ||
|
-%a |
accepted formats (-%a "text/html,image/png;q=0.9,*/*;q=0.1"; repeat as -%a "KIND=formats" for one kind of link: document iframe frame image style script font audio video object embed (--accept <param>) | ||
|
-%X |
additional HTTP header line (-%X "X-Magic: 42"; an Accept line replaces every computed Accept, per-kind values included; Content-Length, Transfer-Encoding, Connection, Keep-Alive, TE, Upgrade and Expect are dropped (--headers <param>) | ||
|
-%Q |
send the headers a browser -F implies: its Accept values, header order and Sec-Fetch-*, and client hints over HTTPS (%Q0 don’t) (--browser-headers[=0]) |
|
-C |
create/use a cache for updates and retries (C0 no cache,* C1 cache is prioritary,C2 test update before) (--cache[=N]) | ||
|
-k |
store all files in cache (not useful if files on disk) (--store-all-in-cache) | ||
|
-%r |
write an ISO-28500 WARC/1.1 archive; --warc-file NAME sets the output name, --warc-max-size N rotates segments past N bytes, --warc-cdx also writes a sorted CDXJ index, --wacz packages it all as a WACZ file (--warc) | ||
|
-%d |
write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror (--changes[=0]) | ||
|
-%n |
do not re-download locally erased files (--do-not-recatch[=0]) | ||
|
-%v |
display on screen filenames downloaded (in realtime) - * %v1 short version - %v2 full animation (--display[=N]) | ||
|
-Q |
no log - quiet mode (--do-not-log) | ||
|
-q |
no questions - quiet mode (--quiet) | ||
|
-z |
log - extra infos (--extra-log) | ||
|
-Z |
log - debug (--debug-log) | ||
|
-v |
log on screen (--verbose) | ||
|
-f |
log in files (--file-log) | ||
|
-f2 |
*one single log file (--single-log) | ||
|
-I |
*make an index (I0 don’t make) (--index[=0]) | ||
|
-%i |
make a top index for a project folder (* %i0 don’t make) (--build-top-index[=0]) | ||
|
-%I |
make an searchable index for this mirror (* %I0 don’t make) (--search-index[=N]) |
|
-pN |
priority mode: (* p3) (--priority[=N]) |
p0 just scan, don’t save
anything (for checking links)
p1 save only html files
p2 save only non html files
*p3 save all files
p7 get html files before, then treat other files
|
-S |
stay on the same directory (--stay-on-same-dir) | ||
|
-D |
*can only go down into subdirs (--can-go-down) | ||
|
-U |
can only go to upper directories (--can-go-up) | ||
|
-B |
can both go up&down into the directory structure (--can-go-up-and-down) | ||
|
-a |
*stay on the same address (--stay-on-same-address) | ||
|
-d |
stay on the same principal domain (--stay-on-same-domain) | ||
|
-l |
stay on the same TLD (eg: .com) (--stay-on-same-tld) | ||
|
-e |
go everywhere on the web (--go-everywhere) | ||
|
-%H |
debug HTTP headers in logfile (--debug-headers) |
|
-#test |
list engine self-tests (run one with -#test=NAME [args]) | ||
|
-#C |
cache list (-#C ’*.com/spider*.gif’ (--debug-cache <param>) | ||
|
-#R |
cache repair (damaged cache) (--repair-cache) | ||
|
-#d |
debug parser (--debug-parsing) | ||
|
-#E |
extract new.zip cache meta-data in meta.zip (--extract-cache) | ||
|
-#f |
always flush log files (--advanced-flushlogs) | ||
|
-#FN |
maximum number of filters (--advanced-maxfilters[=N]) | ||
|
-#h |
version info (--version) | ||
|
-#K |
scan stdin (debug) (--debug-scanstdin) | ||
|
-#L |
maximum number of links (-#L1000000) (--advanced-maxlinks[=N]) | ||
|
-#p |
display ugly progress information (--advanced-progressinfo) | ||
|
-#P |
catch URL (--catch-url) | ||
|
-#T |
generate transfer ops. log every minutes (--debug-xfrstats) | ||
|
-#u |
wait time (--advanced-wait[=N]) | ||
|
-#Z |
generate transfer rate statistics every minutes (--debug-ratestats) |
|
-%! |
bypass built-in security limits aimed to avoid bandwidth abuses (bandwidth, simultaneous connections) (--disable-security-limits[=0]) |
IMPORTANT NOTE: DANGEROUS
OPTION, ONLY SUITABLE FOR EXPERTS
USE IT WITH EXTREME CARE
|
-V |
execute system command after each files ($0 is the filename: -V "rm \$0") (--userdef-cmd <param>) |
$0 reaches the shell as a
parameter, so a command that evaluates its
argument as an expression (let, declare -i, eval) reads it
as code
|
-%W |
use an external library function as a wrapper (-%W myfoo.so[,myparameters]) (--callback <param>) | ||
|
-y |
go to background when suspended (y0 don’t) (--background-on-suspend[=0]) |
|
-N0 |
Site-structure (default) | ||
|
-N1 |
HTML in web/, images/other files in web/images/ | ||
|
-N2 |
HTML in web/HTML, images/other in web/images | ||
|
-N3 |
HTML in web/, images/other in web/ | ||
|
-N4 |
HTML in web/, images/other in web/xxx, where xxx is the file extension (all gif will be placed onto web/gif, for example) | ||
|
-N5 |
Images/other in web/xxx and HTML in web/HTML | ||
|
-N99 |
All files in web/, with random names (gadget !) | ||
|
-N100 |
Site-structure, without www.domain.xxx/ | ||
|
-N101 |
Identical to N1 except that "web" is replaced by the site’s name | ||
|
-N102 |
Identical to N2 except that "web" is replaced by the site’s name | ||
|
-N103 |
Identical to N3 except that "web" is replaced by the site’s name | ||
|
-N104 |
Identical to N4 except that "web" is replaced by the site’s name | ||
|
-N105 |
Identical to N5 except that "web" is replaced by the site’s name | ||
|
-N199 |
Identical to N99 except that "web" is replaced by the site’s name | ||
|
-N1001 |
Identical to N1 except that there is no "web" directory | ||
|
-N1002 |
Identical to N2 except that there is no "web" directory | ||
|
-N1003 |
Identical to N3 except that there is no "web" directory (option set for g option) | ||
|
-N1004 |
Identical to N4 except that there is no "web" directory | ||
|
-N1005 |
Identical to N5 except that there is no "web" directory | ||
|
-N1099 |
Identical to N99 except that there is no "web" directory |
|
-%n |
Name of file without file type (ex: image) |
|||
|
-%N |
Name of file, including file type (ex: image.gif) |
|||
|
-%t |
File type (ex: gif) |
|||
|
-%p |
Path [without ending /] (ex: /someimages) |
|||
|
-%h |
Host name (ex: www.example.com) |
|||
|
-%M |
URL MD5 (128 bits, 32 ascii bytes) |
|||
|
-%Q |
query string MD5 (128 bits, 32 ascii bytes) |
|||
|
-%k |
full query string |
|||
|
-%r |
protocol name (ex: http) |
|||
|
-%q |
small query string MD5 (16 bits, 4 ascii bytes) |
’%s?’ Short name version (ex: %sN)
-%[param]
param variable in query string
-%[param:before:after:empty:notfound]
advanced variable extraction
%[param:before:after:empty:notfound]
param : parameter name
before : string to prepend if the parameter was found
after : string to append if the parameter was found
notfound : string replacement if the parameter could not be
found
empty : string replacement if the parameter was empty
all fields, except the first one (the parameter name), can
be empty
|
-K0 |
foo.cgi?q=45 -> foo4B54.html?q=45 (relative URI, default) | ||
|
-K |
-> http://www.example.com/folder/foo.cgi?q=45 (absolute URL) (--keep-links[=N]) | ||
|
-K3 |
-> /folder/foo.cgi?q=45 (absolute URI) | ||
|
-K4 |
-> foo.cgi?q=45 (original URL) | ||
|
-K5 |
-> http://www.example.com/folder/foo4B54.html?q=45 (transparent proxy URL) |
--mirror
<URLs> *make a mirror of site(s) (default)
|
--get |
<URLs> get the files indicated, do not seek other URLs (-qg) |
|||
|
--list |
<text file> add all URL located in this text file (-%L) |
--mirrorlinks
<URLs> mirror all links in 1st level pages (-Y)
--testlinks
<URLs> test links in pages (-r1p0C0I0t)
--spider
<URLs> spider site(s), to test links: reports Errors & Warnings (-p0C0I0t)
--testsite
<URLs> identical to --spider
--skeleton
<URLs> make a mirror, but gets only html files (-p1)
--update
update a mirror, without confirmation (-iC2)
--continue
continue a mirror, without confirmation (-iC1)
--catchurl
create a temporary proxy to capture an URL or a form post URL
--clean
erase cache & log files
--http10
force http/1.0 requests (-%h)
|
0 |
The mirror ran to the end, or stopped because it was asked to: a signal, the interactive shell, or a transfer budget set with --max-time or --max-size. Meeting a budget is the outcome that was asked for, not a failure, so it is not an abort even though the log file calls it one. | ||
|
3 |
A mirror started and the engine gave up before the end: a full disk, a cache write failure, or a link table that hit the -#L cap. Unlike a time or size budget, that cap cuts short work that was asked for rather than granting it, which is why the two kinds of user-set limit report differently. The reason is in the log file, and what was mirrored before the abort is kept. This is the process exit status; the engine’s own exit_xh is a separate channel with its own values, and the two are kept apart so that neither can be read as the other. | ||
|
1, 255 |
httrack refused the command line, or a cache operation it asked for failed. No mirror ran. |
/etc/httrack.conf
The system wide configuration file.
cookies.txt
The cookie jar, in Netscape format. Without --cookies-file, the mirror reads the one in its output directory and the one in the current directory, and writes the jar back to its output directory at the end. With --cookies-file, that file alone is the jar. HTTrack rewrites it at the end and keeps its old content as <file>.bak. HTTrack leaves the file unchanged and warns if a line could not be loaded or a cookie repeats. It does the same when the file holds a comment that no known tool writes. An empty expiry field means a session cookie.
The file given to --cookies-file can also hold a document.cookie string, such as a=1; b=2, as a browser console prints it, or a pasted Cookie: request header. Those cookies are sent to the registrable domain of the first start URL and its subdomains, or to that host alone when it is an address or a public suffix. They are Secure when that URL is https. HTTrack never rewrites such a file. HttpOnly cookies are not in it. A cookies.txt export has them, and so does the Cookie header from DevTools "Copy as cURL", pasted here.
<name>.hts
A login bundle, named by a start URL that ends with #httrack-bundle=<name>, such as https://example.com/members/#httrack-bundle=login1. The name holds letters, digits, ., _ and - only. HTTrack reads the file in the project folder, or in the folder given to --bundle-dir <dir>, and never writes it. doit.log keeps that folder as an absolute path. HTTrack reads the suffix on the command line, in a -%L list and in doit.log, never on a crawled link. A missing file, or one that does not parse, stops the run before any request.
The first line is # HTTrack bundle 1. Then come url lines, each one more start URL, at most one ua line, and header and scope lines. A cookies line ends them, and a Netscape cookie jar follows it.
The bundle’s user agent and headers replace -F and -%X for each host in its scope, and only there. A bundle with header lines but no ua line sends the -F user agent. When any start URL naming the bundle, or any of its url lines, is https, its headers go over https alone, and so do those of the bundles it merges with. By default the scope is the registrable domains of the start URLs naming the bundle and of its url lines. A host with no registrable domain, such as a public suffix, a single label or an address, covers itself alone. Scope lines replace that default. HTTrack drops a cookie outside the scope and gives the count in one warning. HTTrack refuses a Host, Cookie, Proxy-* or framing header, a scope line naming a public suffix, and a line that holds a control byte or is too long, in the cookie jar too and even under -b0. A second ua line, header lines over 8192 bytes, or a full cookie jar stops the run too. A repeated cookie line or an unknown key only warns, and so does an unknown # line in the cookie jar. Before the cookies line, a # line is a comment. Messages give the file and the line number, never a value. Two bundles whose scopes overlap must give the same user agent, headers and cookie values, or the run stops.
The bundle’s cookies go into the cookie jar, which HTTrack writes at the end of the crawl. Of the bundle and the jar, the newer file wins a cookie both hold. A new capture therefore wins over the jar, and a cookie the server refreshed wins over an older bundle. On equal times the bundle wins.
A running mirror
also watches its own output directory for these. Creating
one is how a script drives an engine it has no other handle
on. On Windows, which has no cross-process kill(1),
it is the only way.
hts-in_progress.lock
Written when the mirror starts and deleted when it ends. It records the command line, and names the two request files below.
hts-stop.lock
Create it to pause the mirror. The engine deletes it, lets the transfers already in flight finish, writes hts-paused.lock and waits. Delete that one to carry on.
hts-abort.lock
Create it to stop the mirror and keep what it has downloaded, so that a later --continue resumes from there rather than starting over. The engine deletes it and stops.
The engine acts on either request only when the file is stamped later than hts-in_progress.lock, which the mirror dates one second back when it starts. A request created after the mirror started is therefore heard at once, on every filesystem. One left behind by an earlier run is ignored. A request the engine cannot delete is ignored too, because it would otherwise fire on every later poll.
|
HOME |
Is being used if you defined in /etc/httrack.conf the line path ˜/websites/# |
Errors/Warnings are reported to hts-log.txt by default, or to stderr if the -v option was specified.
These are the principals limits of HTTrack for that moment. Note that we did not heard about any other utility that would have solved them. - Several scripts generating complex filenames may not find them (ex: img.src=’image’+a+Mobj.dst+’.gif’) - Cgi-bin links may not work properly in some cases (parameters needed). To avoid them: use filters like -*cgi-bin*
Please reports bugs to <bugs@httrack.com>. Include a complete, self-contained example that will allow the bug to be reproduced, and say which version of httrack you are using. Do not forget to detail options used, OS version, and any other information you deem necessary.
Copyright (C) 1998-2026 Xavier Roche and other contributors
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
The most recent released version of httrack can be found at: https://www.httrack.com
Xavier Roche <roche@httrack.com>
The HTML documentation (available online at https://www.httrack.com/html/ ) contains more detailed information. Please also refer to the httrack FAQ (available online at https://www.httrack.com/html/faq.html )
The graphical interface is webhttrack(1).