13 Linux wget command usage examples

Wget is a computer software package for retrieving content from web servers using HTTP, HTTPS and FTP protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc. Its features include recursive download, conversion of links for offline viewing of local HTML, and support for proxies.

Lets have a look at some examples of using wget:

1. Download a webpage


$ wget http://www.examplewebsite.com/

2. Download a file from ftp server


$ wget ftp://ftp.examplewebsite.com/source-code.tar.gz

If you specify a directory, Wget will retrieve the directory listing, parse it and convert it to html


$ wget ftp://ftp.examplewebsite.com/

3. To download specifying the host port number use the following syntax


$ http://host[:port]/directory/file

$ ftp://host[:port]/directory/file

4. To download providing username and password use the follow syntax


$ ftp://user:password@host/path

$ http://user:password@host/path

Either user or password, or both, may be left out. If you leave out either the http username or password, no authentication will be sent. If you leave out the ftp username, ‘anonymous’ will be used. If you leave out the ftp password, your email address will be supplied as a default password.

5. Downloading from FTP in ASCII mode

By default, ftp documents are retrieved in the binary mode (type ‘i’), which means that they are downloaded unchanged. Another useful mode is the ‘a’ (ASCII) mode, which converts the line delimiters between the different operating systems, and is thus useful for text files.

To dowload in ASCII mode use the Wget type feature as below


$ ftp://host/directory/file;type=a

6. Log all messages to logfile


$ wget http://www.examplewebsite.com/ -o logfilename

7. Append all log messages to an existing logfile


$ wget http://www.examplewebsite.com/ -a logfilename

8. Download reading URLs from a file


$ wget -i file

9. Download reading URLs from a standard inputterminated by Ctrl-d


$ wget -i -

10. To turn on debug output use the -d option as below:


$ wget -d http://www.examplewebsite.com/

Please note that ‘-d’ will not work if wget has not been complied with debug support

 11. Download a single web page along with the assets like images and style sheets needed to display the page, and convert the URLs inside it to refer to locally available content


$ wget -p --convert-links http://www.examplewebsite.com/dir/examplepage.html

or


$ wget -p -k http://www.examplewebsite.com/dir/examplepage.html

12. Download the entire contents of examplewesbite.com


wget -r -l 0 http://www.examplewesbite.com/

or


wget -r -l inf http://www.examplewesbite.com/

`-l inf` stands for infinite recursion depth

`-l 0` is equivalent to `-l inf`

13. Mirror website to a static copy converting html files saved under extensions other than ‘.html’ to `.html’


$ wget --mirror -w 2 -p --html-extension --convert-links -P <dir> http://www.examplewebsite.com

 

The complete wget manual can be downloaded at:

https://www.gnu.org/software/wget/manual/wget.pdf

Leave a Comment

Back to top