projects

for hire

sketches

spider

Spider is an application which invites the user to interact with a visual representation of a web crawler.

The first component, the web-crawler is a piece of software that starts from a given URL on the web, and then proceeds to follow all links from there to new pages, then follows all links from thoses pages and so on, until it encounters a dead-end where it has exhausted all possible HTML connections.

The spider is interested by the links, but also with the textual content of websites. Every word encountered during the parsing process is indexed and saved in a database. An occurence count is kept for each word, so that at any given time, the spider can report the top 8 most frequently used words.

The second component, the interface, depicts the software bot as a visual component in the form of a spider. It can be dragged around the screen to represent the act of crawling the web. If left idle, the spider will finish processing the current webpage, but will wait until it gets moved again to progress to the next link in the list. The graphical spider leaves behind a trail of every word encountered, but keeps attached to each of it's leg the 8 most frequent words.

Every time a website has been visited, the spider also drops a node in the background. Nodes are inter-connected with each other if they share the same domain name.

This piece is written in Java 1.4.1. At the moment, Spider has to run as a standalone application, because of Java's security features. A future implementation would turn Spider into web-browser friendly applet.