ht://Dig Copyright © 1995-2004 The ht://Dig Group
	  Please see the file COPYING for
	  license information.
	
There are several programs in the ht://Dig package.
	  Digging is the first step in creating a search database. This
	  system uses the word digging while other systems call
	  it harvesting or gathering. In the ht://Dig
	  system, the program htdig performs
	  the information gathering stage. In this process, the program
	  will act as a regular web user, except that it will follow
	  all hyperlinks that it comes across. (Actually, it
	  will not follow all of them, just those that are within the
	  domain it needs to gather information on...)
	   Each document it goes to is examined and all the unique
	  words in this document are extracted and stored.
	
The digging process will only follow links and has no notion of JavaScript, applets, or user-input forms.
	  Searching is where the users actually get to use all the
	  information that was gathered during the dig and merge
	  stages. The 
	  htsearch program performs the actual searches. It typically
	  produces HTML output which will be seen by the
	  users, though other text formats could be generated by
	  editing the output templates.
	
Merging does exactly that--it merges one database into another. In previous versions of ht://Dig, the htmerge program also formed databases for use by htsearch from the htdig output. This process is now largely unnecessary except for removal of invalid URLs which is now done by the htpurge program.
Purging removes documents and the associated words from the databases. This should be done after running htdig to remove invalid URLs, documents marked not to be indexed, old versions of modified documents, etc. You can also specify specific URLs to be removed explicitly by htpurge.
Loading involves importing the contents of the databases from formatted ASCII text documents as created by htdump or the -t flag from htdig. This is, of course, destructive by nature and data from the text files will replace any conflicting data in the databases.
Dumping involves exporting the contents of the databases to formatted ASCII text documents. This can be useful for backups, transferring databases between different operating systems, changing the compression or encodings in the ht://Dig configuration, parsing by external utilities. It is not recommended to edit these files by hand, so be warned! (Minor edits will probably be fine.)
The htstat program returns statistics on the databases, similar to the -s flags for some of the programs. In addition, it can return a list of URLs in the databases.
	  The ht://Dig system includes a handy reminder service which
	  allows HTML authors to add some ht://Dig specific meta
	  information in HTML documents. This meta information is
	  used to email authors after a specified date. Very useful
	  to maintain lists that contain those annoying "new"
	  graphics with new items. (Hint: Things really aren't all
	  that new anymore after 6 months!)
	
To allow the searches to use "fuzzy" algorithms to match words, the htfuzzy program can create indexes for several different algorithms.