Install wget and curl
To install wget, enter the following command:
sudo apt-get install wget
To install curl, enter the following command:
sudo apt-get install curl
Example: In the following example, we will HTTP/HTTPS request through curl and wget and download a website page.
Using the curl command, save a webpage.
curl https://w3wiki.org -o geeks.html
Output: The file is downloaded as geeks.html
Using the wget command, save a webpage.
wget https://practice.w3wiki.org/jobs
Output
We have the jobs file from wget and geeks.html using the curl command.
Example 2: In the following example, we will learn FTP protocol requests through curl and wget and download files.
To download files from a domain name like the w3wiki logo from wget, use the following command.
wget https://media.w3wiki.org/wp-content/cdn-uploads/20210420155809/gfg-new-logo.png
The output is as follows:-
Using the curl command, we can specify the name, for example, logo, and download as follows:
curl https://media.w3wiki.org/wp-content/cdn-uploads/20210420155809/gfg-new-logo.png -o logo.png
The output is as follows:
curl also supports uploading files to the web. We need to add the flag -T to specify uploading. We use the following command to upload files to any URL.
curl -T “geeks_logo.png” ftp://www.w3wiki.org/upload/to/url/
Example 3: Recursive downloading
wget utility tool supports recursive downloading and we add the flag –recursive for that.
wget –recursive https://practice.w3wiki.org/jobs
This command downloads all the related resources to a folder named at the web page URL. The output is as follows:
Terminal(Download in progress)
Files and Folders downloaded
The files path added to robots.txt are ignored. To turn this feature off, add the -e flag as follows:
wget -e robots=off https://practice.w3wiki.org/jobs
The documents downloaded are in form of breadth-first-search. But it can be overridden by mentioning the depth by -l flag and setting the depth. The default maximum depth is 5.
wget ‐l=1 ‐‐recursive ‐‐no-parent https://practice.w3wiki.org/jobs
Difference Between wget VS curl
wget and curl are the commands that are used to HTTP requests without any GUI or software, rather we use the Terminal in Linux that provides the respective output or message. The commands are very useful for web crawling, web scraping, testing RESTful APIs, etc.