RFC2186 Internet Cache Protocol (ICP), version 2. [c. 1997/09/01]

RFC2187 Application of Internet Cache Protocol (ICP), version 2. [c. 1997/09/01]

SOFTWARE: Squid proxy caching server (monthly posting)

BASAR: A framework for integrating agents in the World Wide Web ( Christoph G. Thomas ; IEEE Computer Magazine 1995-05)

Topical AND Keyword Based Search Engines? [ 2000/10/16]

Unusual HTTP Requests For robots.txt? [ 2000/09/22]

How About an Intelligent Open Source Filter? [ 2000/04/04]

Is Spidering Content from the Web Illegal? [ 1999/11/09]

How do I configure my machine to use a proxy server? [ 1999/09/01]

Questions and Answers

Dead Link Check (DLC) - DLC - HTTP link checker written in Perl. Can generate HTML output for easy checking of results and process a link cache file to hasten multiple requests. Initially created as an extension to Public Bookmark Generator (PBM); can be used alone. {(L)GPL}

p5-WWW-Link-0.030 - Maintain information about the state of links

narval-1.1 - Network Assistant Reasoning with a Validating Agent Language

larbin-2.6.1 - A powerful HTTP crawler with an easy interface

ja-navi2ch-emacs21-1.5.1,1 - 2ch.net client for Emacsen

googolplex-0.1.0 - Query Google, parse it and returns the result as a list

dailystrips-1.0.22 - Utility to download or view your favorite online comic strips daily

crawl-0.3 - A small, efficient web crawler with advanced features

p5-Image-Grab-1.4 - Perl extension for Grabbing images off the Internet

p5-WWW-Robot-0.022 - Perl interface to a generic web traversal engine

junkbuster-2.0.2 - An HTTP proxy server that eliminates ads

adzapper-0.4.0 - A filtering proxy that can block ads from being displayed

surfraw-1.0.7 - Command line interface to popular WWW search engines

htdump-0.9x - A tool to retrieve WWW data

decss-1.0 - Strip cascading style sheets from webpages

p5-HTML-Summary-0.017 - Produces summaries from the textual content of web pages

puf-0.91b6a - puf is a "parallel url fetcher" for UN*X systems

quotes-1.7.2 - Quote, currency, and Slashdot headline fetcher based on Perl

downloader-1.30 - Program for downloading via ftp or http with GUI

checkbot-1.67 - A WWW link verifier, similar like momspider

momspider-1.00 - WWW Spider for multi-owner maintenance.

xquote-1.1 - A quote retrieval tool for X. [X]

wget-1.8.1_1 - Retrieve files from the 'net via HTTP and FTP

urlview-0.9 - URL extractor/launcher

comline-4.0D - W3C Command Line WWW Tool

harvest-1.5 - Collect information from all over the Internet

transproxy-1.2 - Transparent HTTP proxy for ipfw's fwd rule (or IPFILTER's ipnat command)

wcolEpre-1999.01.10 - A prefetching proxy server for WWW

webcopy-0.98b7 - A Web Mirroring Program

webglimpse-1.6 - WWW interface to Glimpse search engine.

site-dater.pl - Generates a table of web links within a local hierarchy sorted by date. {PD}

SiteMap - Creates an HTML SiteMap of your *.*htm* files {GPL}

ht://Dig - Complete world wide web indexing and searching system {GPL}

Checklinks - HTML link checker that supports SSI, many Apache options, and more (in Perl 5) {OpenSource}

notify - Notify (website) visitors of changes to your site. {GPL}

CheckURL - Sends notification e-mails for changed URLs {GPL}

DejaSearch - DejaSearch is a frontend to DejaNews, the leading Usenet archive {GPL}

Web Secretary - Web page monitoring software {GPL}

netcomics - A perl script that downloads today's comics from the Web {GPL}

sitecopy - Maintain remote copies of locally stored web sites {GPL}

DraE Tracking - Allows servers to provide free tracking to web sites {GPL}

FastLink - FastLink is a free Java Applet that displays mirror sites sorted by their respon {GPL}

The Internet Junkbuster - The Internet Junkbuster v2.0.2 {GPL}

EHeadlines - Root Menu news system. {x,GPL}

gtkMeat - A Freshmeat new submissions ticker {x,GPL}

gtkSlash - Gtk+ based Slashdot headlines news ticker {x,GPL}

Kget - KDE app to get files from the internet {x,GPL}

asScotch - The days UserFriendly comic strip in your AfterStep rootmenu {x,GPL}

asTequila - The AfterStep Resource Page (TARP) headlines in your AfterStep rootmenu {x,GPL}

Squid - High performance Web proxy cache {GPL}

w3mir - HTTP copying and mirroring program {Artistic}

WWWOFFLE - Simple proxy server with special features for use with dial-up internet links {GPL}

freshmeat newsletter to HTML converter - procmail filter to convert freshmeat email newsletter to HTML {Artistic}

webcrawl {PD}

ECLiPt-Mirror - Full-featured mirroring script {GPL}

pavuk - Webgrabber with an optional Xt or GTK GUI {GPL}

snarf - Command-line URL retrieval tool with some unique features. {GPL}

ticker - Configurable text scroller, with slashdot and freshmeat modules {GPL}

curl - Tiny command line client for getting data from a URL {GPL}

swebget - Prints a webpage to stdout {GPL}

GNU Wget - Network utility to retrieve files from the World Wide Web {GPL}

PathFinder - A personal web search engine {GPL}

HTTPGate - A Filtering HTTP Gateway {GPL}

Muffin - Filtering proxy server for the World Wide Web written entirely in Java {GPL}

tinyproxy - A small, lightweight, easy-to-configure HTTP proxy. {GPL}

Internet Junkbuster - Blocks unwanted banner ads and protects your privacy {GPL}

Kticker - News ticker widget that downloads news headlines and displays them periodically {x,GPL}

urlredir - URL redirector for use with the squid proxy server {GPL}

DailyUpdate - Grabs dynamic information from the internet and integrates itinto your webpage {GPL}

Web User Interface - Builds a list of all available personal homepages. {GPL}

CGIProxy - Anonymizing, filter-bypassing HTTP proxy in a CGI script (in Perl) {OpenSource}

Get Right - HTTP resume for failed transfers. {GPL}

Web Tree Scanner - A program to visualize the tree of a WWW server and check the links [X] {GPL}

nss - Netscape Startup Script. Script to handle Netscape launches. [X] {freely distributable}

Slashdot Reader - Slashdot Reader written in Pike/GTK. [X] {PD}

httptunnel-3.3 - Tunnel a tcp/ip connection through a http/tcp/ip connection

RabbIT - Mutating, caching webproxy to speed up surfing over slow links {freely distributable}

World Engine - Java Search Engine Front End

fresh-split - Perl scripts for splitting freshmeat news

Submitwolf Pro 4.02 By Trellian

asGin - Linux Today headlines in your AfterStep root menu [X] {GPL}

urlmon - URL monitoring and report tool {GPL}

Cyberscrub Professional Edition 1.5

Net Nanny 4.0

LinkBot Personal Edition 6.0

Applications and Utilities

p5-WWW-Search-AltaVista-2.05 - Perl WWW::Search class for searching AltaVista

libstocks-0.5.0 - A C library which can be used to fetch stocks quotes

p5-HTTP-GHTTP-1.06 - Perl interface to the gnome ghttp library

ruby-rss-0.9.1 - Ruby library for parsing, creating, downloading, and caching RSS

HTTP::Status - Processes status codes sent over HTTP, e.g. "403 Forbidden", "4040 Not Found", or "402 Payment required". Part of the libwww bundle. [Perl] {oss}

LWP::RobotUA - Create your own Web robot. Part of the libwww bundle. [Perl] {oss}

WWW::Robot - A traversal engine for your Web robot. [Perl] {oss}

WWW::RobotRules - Nice Web robots, as they scour the Net for treasure, heed a robots.txt file if they find one. Information about the Robot standard can be found in http://info.webcrawler.com/mak/projects/robots/norobots.html. [Perl] {oss}

ARS - A Web client for Remedy's ARS system. Useful only if you're already using ARSPerl. [Perl] {oss}

Libraries and Functions

(The following links to subjects at this site retain your personalized selections.)

WWW Servers - Respond to HTTP requests

WWW authoring - Creating HTML, CGI

WWW Browsers - User interface for accessing the WWW

Up to: World Wide Web - HTTP, HTML, standards, browsers, transfer utilities, servers, et al.

freshmeat.net : Topic : Internet : WWW/HTTP : Site Management : Link Checking

Www - - WEB utilities (browers, HTTP servers, etc).

Computers : Programming : Agents :

(Metalab at UNC) /pub/linux/apps/www/indexing/ - indexing and search tools for the Web

(Metalab at UNC) /pub/linux/apps/www/mirroring/ - mirroring and batch retrieval