Downloading and Archiving Files

wget

Basics

• Quiet

wget -q <address>

• Not verbose (mostly quite)

wget -nv <address>

• Chagne name

wget -O <filename> <address>

• Download multiple URLs

wget <address> <address>

• create file.txt with links (one per line)

wget -i file.txt

• Throttle download speeds

wget --limit-rate 400k <address>

• Resume download

wget -c <address>

• Download in the background

wget -b <address>

Advanced

• Mirror website

• Log output to a file

wget -o file.log -r -l 3 --spider <address> wget -a file.log <address>

• Find broken links

grep -B 2 '404' file.log | grep "http" | cut -d " " -f 4 | sort -u

• Change user agent

wget -U '<string>' <address>

• View response headers

wget -S --spider <address>

• Add username and password

wget ‐‐http-user=daniel ‐‐http-password=JZXGAo34mz <address>

• Save cookies

wget ‐‐cookies=on ‐‐save-cookies cookies.txt ‐‐keep-session-cookies ‐‐post-data 'user=daniel&password=JZXGAo34mz' <address>

• Load cookies

wget ‐‐cookies=on ‐‐load-cookies cookies.txt ‐‐keep-session-cookies <address>

Quiz

• Create a directory in / called LinuxTestDrills/downloads.
  • The rest of the Quiz should be done in this folder. Remove files that are not part of the instructions if you make a mistake.

1. Download https://en.wikipedia.org/wiki/Penrose_tiling#/media/File:Penrose_Tiling_(Rhombi).svg and save it as awesome.svg
2. Download concatenate linuxtest.danielc.us/downloads/topSite and linuxtest.danielc.us/downloads/secondTopSite into a single file called links.
3. Download the sites in links. Open index.html with a web browser and click on one of the links.
4. Mirror https://dizer.netlify.app/ with one level of recursion with links that you can browse it from your computer. Open it with a browser and click around.
5. Finish downloading with full recursion, and click around again.
6. Download linuxtest.danielc.us/downloads/quizFiles/quiz and linuxtest.danielc.us/downloads/quizFiles/wget.
7. Execute the command quizFiles/quiz wget

tar

tar -cf files.tar ​files/ (create tar from dir)

tar tvf files.tar (list files in tar)

tar --delete -f files.tar file (deletes file from tar)

tar --update -f files.tar file (adds file to tar)

tar -jcf files.tar.bz2 files.tar (create bz2 from tar)

tar -Jcf files.tar.xz files/ (create tar.xz from dir)

unzip files.zip -d files/ (unzip to dir)

tar xf files.tar.bz2 -C <DIR> --same-permissions (extract to dir)

wget file.tar.xz -O - | tar xz (Download and extract)

Quiz

1. Archive LinuxTestDrills/downloads into a file LinuxTestDrills/archives/downloads.tar.
2. List the files in LinuxTestDrills/archives/downloads.tar to make sure you have only the correct files.
3. Download the file at https://danielc.us/lfcs-prep/testFiles.tar.xz to LinuxTestDrills/downloadsand extract it into /LinuxTestDrills, in a single command, maintaining original ownership. You can change ownership as needed during the drills.
4. Add LinuxTestDrills/downloads/testFiles.tar.xz into LinuxTestDrills/archives/downloads.tar and delete everything in the folder LinuxTestDrills/downloads.
5. Compress LinuxTestDrills/archives/downloads.tar with bz2, xz, and zip. Look at the resulting files.
6. Run quizFiles/checkQuiz1a.sh