Essential Command Line Tools: Curl, Git, Wget, Lynx

Other maintenance tools curl, git, wget, lynx

Curl • What is it? Command line file transfer tool using URL syntax • Supports various protocols like, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP.

Curl Examples • Download a Single File: curl http://www.centos.org • Save the CURL Output to a file: curl -o mygettext.html http://www.gnu.org/software/gettext/manual/gettext.html curl -O http://www.gnu.org/software/gettext/manual/gettext.html • Fetch Multiple Files at a time: curl -O URL1 -O URL2 • Follow HTTP Location Headers with -L option (for redirects): curl -L http://www.google.com • Continue/Resume a Previous Download: curl -O http://www.gnu.org/software/gettext/manual/gettext.html • Pass HTTP Authentication in CURL: curl -u username:password URL

Curl Examples • Download Files from FTP server: curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php .. a specific file curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/ .. list directory • Upload Files to FTP Server: curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com .. multiople files curl -u ftpuser:ftppass -T - ftp://ftp.testserver.com/myfile_1.txt .. from STDIN • More Information using Verbose and Trace Option: curl -v http://google.co.in curl -trace -v http://google.com • Using a proxy: curl -x proxysever.test.com:3128 http://google.com • To send email: curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com

git • What is it? A file versioning system similar used for program project files similar to RCCS. • Also sometimes used for downloading project files. Not widely used except for program developers or sometimes for kernel packages. • Also provides for checksum/hashes for project files.

wget What is it? GNU wget software package for command line retrievial of files using HTTP, HTTPS and FTP. Some features: • Can resume aborted downloads, using REST and RANGE • Can use filename wild cards and recursively mirror directories • NLS-based message files for many different languages • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally • Runs on most UNIX-like operating systems as well as Microsoft Windows • Supports HTTP proxies • Supports HTTP cookies • Supports persistent HTTP connections • Unattended / background operation • Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring

wget examples • Info: wget --version wget --help • Download a web page: wget linux.about.com • With retries: wget -t 10 linux.about.com • Download a file: wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz wget -O wget.zip http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz .. with a different name • Multiple files, multiple protocols: wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig • Read URLs from a file: wget -i /wget/tmp.txt

wget examples • Resume a download: wget -c http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso • Unsername, password accessed files: wget --http-user=narad --http-password=password http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso wget --ftp-user=narad --ftp-password=password ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso • Log activity: wget -o logfile linux.about.com • Recurse a website to retrieve all content: wget -r linux.about.com -o logfile • Convert the links to point to downloaded files: wget --convert-links -r linux.about.com -o logfile • Ensure all linked documents are downloaded: wget -p --convert-links -r linux.about.com -o logfile

lynx • Command line web browser, Text-based Web Browser. • Configurable. • Support for SSL and many features of HTML • Highlight the chosen link. • Number all the links on a web page and open links using number assigned. • No support for JavaScript. • No support for HTTP Cookies. • Configuration through commands in terminal or configuration files. • See also links product

lynx examples • Get a page: lynx google.com • With cookies: lynx -accept_all_cookies http://www.yahoo.com • With a password: -auth=id:password -auth=mydomain\\myuser:mypass • Dump a page contents: lynx -dump "http://www.google.com" ><a title="test" href="http://www.giglig.com/test">test</a>.txt • Escape a string on the command line by using single ticks: lynx 'http://www.example.com/This(URL)is anExample.html'

ldd command • What is it? Linux utility used determine the shared library dependencies of an executable or even that of a shared library. • See also: dpkg –L, yum provides, rpm-qf, rpm –qcf, rpm -qdf • Option Description -v Print all information. -d Perform relocations and report any missing objects (ELF only). -r Perform relocations for both data objects and functions, and report any missing objects or functions (ELF only). -u –unused Print unused direct dependencies. –help Usage information.

ldd examples • Basic SO libraries info: ldd /bin/ls ldd libshared.so • Expanded: ldd -v /bin/ls: • Unused: ldd -u /bin/ls • Dynamic relocatable: ldd -r assert.o

Essential Command Line Tools: Curl, Git, Wget, Lynx