Search google (and update twitter) from the command line

August 20, 2009admin No Comments »

Hello there! If you are new here, you might want to subscribe to the RSS feed for updates on this topic.

There are some of us who prefer to do as much as possible from the command line. For whatever reason, this type of person wants to prove to the world the GUI is not necessary. For that reason, I thought I would illustrate how, with the help of curl, you can search Google and update your twitter status from the command line.

Curl is a tool that transfers data to or from a server. Curl supports HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE protocols and works without user interaction. But the neat thing about curl is that it offers a ton of various tricks for the clever end user. One of those tricks is fetching and putting data to/from a server – like Google and Twitter.

What you’ll need

In order to achieve these tasks you will need to have installed both curl and html2text. Since I am showing you how to use these tools for command line tricks, there’s no need to figure up the GUI tools. So we will install both tools from the command line. To do so you need to open up your terminal and issue a command similar to:

sudo apt-get install curl html2text

You will be prompted for your sudo password and then asked to verify the installation. Hit ‘Y’ to continue on with the installation.

Once these two pieces of software are installed, it’s time to work some “magic”.

Searching Google

In order to do this you need to pipe one command through another. Piping is done with the help of the “|” character separating the commands. In this case you are going to pipe the curl command through the html2text command. What this will do is render the html fetched from Google into plain text.

In order to get curl to search a web site you will have to use a single argument with the command. The argument is:

-A

which instructs curl to send the user agent string to a web site so the site thinks it is actually a web browser doing the search. Crafty.

So the command looks like this:

curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80

Notice I use the single quotes. If you are searching for a single word this is not needed. If there is a space between two words, put them in single quotes. The results from this search would look like:

9. **** My Review of Jack Wallen's Post: ?10 Reasons Why Linux Will ... **** Windows 7 may be generating some positive buzz, but Jack Wallen remains skeptical. In fact, he says it's only a matter of time before Linux takes its ... techxav.com/.../my-review-of-10-reasons-why-linux-will-triumph-over- windows/ -Cached - Similar 10. **** Jack Wallen - news **** Jack Wallen asks Why aren't schools adopting open source? and the answer is simply exposure to open source. Getting thousands of students to call and write ... www.wikio.co.uk/news/Jack+Wallen -Cached - Similar

Because these results will fly by you might want to do one more trick with this command by sending it to a file instead. To do this we will use the “>” character and a file name like so:

curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80 > search

Once the search is done, issue the command less search and you will see the results of your search.

Update Twitter status

The same kind of trick is done with your twitter status. To do this you would issue a command like:

curl -u user:password -d status='Your tweet' http://twitter.com/statuses/update.xml

Where ‘Your tweet’ is the status update you want to enter.

Right after you issue that command a bunch of information will be returned. This information is your profile rendered in XML. I have yet to find a way to turn that off.

Final thoughts

The command line is an incredibly powerful tool. Imagine what you can do with these two examples. You could set up a cron job to update your status automatically. Or you could create a script to automatically gather data from Google searches. The possibilites are limitless.

Tags: command line tricks, command-line, curl, twitter