NAME
    INSTALL - Swish-e Installation Instructions

OVERVIEW
    This document describes how to download, build and install Swish-e from
    source. Also below is a basic overview of using Swish-e to index
    documents with pointers to other more advanced examples.

    This document also provides instructions on how to get help installing
    and using Swish-e (and the important information you should provide when
    asking for help). Please read these instructions before requesting help
    on the Swish-e discussion list. See "QUESTIONS AND TROUBLESHOOTING".

    Although building from source is recommended, some OS distributions
    (e.g. Debian) provide pre-compiled binaries. Check with your
    distribution for available packages. Build from source if your
    distribution does not offer the current version of swish-e.

    Also, please read the Swish-e FAQ SWISH-FAQ as it answers many
    frequently asked questions.

    Swish-e knows how to index HTML, XML, and plain text documents. Helper
    applications and other tools are used to convert documents such as PDF
    or MS Word into a format that swish-e can index. These additional
    applications and tools (listed below) must be installed separately. The
    process of converting douments is called "filtering."

    NOTE: Swish-e version 4.2.0 installs a lot more files when running "make
    install". Be aware that the Swish-e documentation may thus include
    errors about where files are located. Please notify the swish-e
    discussion list of any documentation errors.

  Upgrading from previous versions of Swish-e
    If upgrading from a previous version of Swish-e read the CHANGES page
    before installing. The Swish-e index format may have changed and
    existing indexes may not work with the newer version of Swish-e.

    If you have existing indexes you may need to reindex your data before
    running the "make install" step described below. Swish-e may be run from
    the build directory after compiling but before installation.

  Windows Users
    A Windows binary version is available as a separate download from the
    Swish-e site (http://swish-e.org). Many of the installation instructions
    below will not apply to Windows users; the Windows version is
    pre-compiled and includes libxml2, zlib, xpdf and catdoc.

    A number of Perl modules may also be needed. These can be installed with
    ActiveState's PPM utility.

       libwww-perl   - the LWP modules (for spidering)
       HTML-Tagset   - used by web spider
       HTML-Parser   - used by web spider
       MIME-Types    - used for filtering documents when not spidering
       HTML-Template - formatting output from swish.cgi (optional)
       HTML-FillInForm (if HTML-Template is used)

SYSTEM REQUIREMENTS
    Swish-e makes use of a number of libraries and tools that are not
    distributed with Swish-e. Some libraries need to be installed before
    building Swish-e from source, where other tools can be installed at any
    time. See below for details.

  Software Requirements
    Swish-e is written in C and up to this time has been tested on a number
    of platforms, including Sun/Solaris, Dec Alpha, BSD, Linux, OS X, and
    Open VMS.

    The GNU C compiler, gcc, and GNU make are strongly recommended. Repeat:
    you will find life easier if you use the GNU tools.

  Optional but Recommended Packages
    Most of these packages listed below are available as easily installable
    packages. Check your operating system vendor, or install from source.
    Most are very common packages that may already be installed on your
    computer.

    As noted below, some packages need to be installed before building
    Swish-e from source, while others may be added after Swish-e is
    installed.

    * Libxml2
        Libxml2 is very strongly recommended. It is used for parsing both
        HTML and XML files. Swish-e can be built and installed without
        libxml2, but the HTML parser built into swish-e is not as accurate
        as libxml2.

            http://xmlsoft.org/

        For swish-e to use libxml2 it must be installed before building
        swish-e.

        If libzml2 is installed in a non-standard location (e.g. libxml2 is
        built with --prefix $HOME/local) then make sure you add the bin
        directory to your $PATH before building swish-e. Swish-e's configure
        script uses a program created by libxml2 (xml2-config) to find the
        location of libxml2. Use "which xml2-config" to verify the program
        can be found where expected.

    * Zlib Compression
        The Zlib compression library is commonly installed on most systems
        and is recommended for use with Swish-e. Zlib is used for
        compressing text stored in the swish-e index.

            http://www.gzip.org/zlib/

        Zlib must be installed before building swish-e.

    * Perl Modules
        Although Swish-e is a compiled C program, many support features use
        Perl. For example, the web spiders are written in Perl, and modules
        to help with filtering documents are also written in Perl.

        The following Perl modules may be required. Check your current Perl
        installation as many may already be installed.

            LWP
            URI
            HTML::Parser
            HTML::Tagset
            MIME::Types (optional)

        Not that installing Bundle::LWP with the CPAN module

            perl -MCPAN -e 'install Bundle::LWP'

        will install many of the above.

        If you wish to use HTML-Template with swish.cgi to generate output:

            HTML::Template
            HTML::FillInForm

        If you wish to use Template-Toolkit with swish.cgi to generate
        output install:

            Template

        Questions about installing these modules may be sent to the swish-e
        discussion list.

        The "search.cgi" example script requires both Template-Toolkit and
        HTML::FillInForm.

    * Indexing PDF Documents
        Indexing PDF files requires the xpdf package. This is a common
        package available with most operating systems and often provided as
        a package.

            http://www.foolabs.com/xpdf/

        Xpdf may be added after swish-e is installed.

    * Indexing MS Word Documents
        Indexing MS Word documents requires the Catdoc program.

            http://www.45.free.net/~vitus/ice/catdoc

        Catdoc may be added after swish-e is installed.

    * Indexing MP3 ID3 Tags
        Indexing MP3 ID3 Tags requires the MP3::Tag Perl module. See
        http://search.cpan.org. MP3::Tag may be installed after swish-e is
        installed.

    * Indexing MS Excel Files
        Indexing MS Excel files is supported by the following Perl modules,
        also available at http://search.cpan.org.

            Spreadsheet::ParseExcel
            HTML::Entities

        These Perl modules may be installed after swish-e is installed.

INSTALLATION
    Here are brief installation instructions that should work in most cases.
    Following this section are more detailed instructions and examples.

  Building Swish-e
    Download swish-e using your favorite web browser or a utility like wget,
    lynx, or lwp-download. Unpack and build using the following steps:

    Note: "swish-e-2.4.0" is used as an example. Download the most current
    version available and adjust the commands below! Also, if running
    Debian, see notes below on building a .deb package from the swish-e
    source package.

    The "$" symbol indicates steps run as an unprivileged user. The "#"
    indicates steps run as the superuser (root).

        $ wget http://swish-e.org/Download/swish-e-2.4.0.tar.gz
        $ gzip -dc swihs-e-2.4.0.tar.gz | tar xof -
        $ cd swish-e-2.4.0  (this directory will depend on the version of Swish-e)

        $ ./configure
        $ make
        $ make check
        ...
        ==================
        All 3 tests passed
        ==================

        $ su root  (or use sudo)
        (enter password)

        # make install
        # exit
        $ swish-e -V
        SWISH-E 2.4.0

    IMPORTANT: Once installed do not run swish-e as the superuser (root) --
    root is only required during the installation step when installing into
    system directories. Please do not break this rule.

    NOTE: If you are upgrading from an older version of swish-e be sure and
    review the CHANGES file. Old index files may not be compatible with the
    newer version of swish-e. After building swish-e (before running "make
    install"), swish-e can be run from the build directory:

        $ src/swish-e -V

    To minimize downtime, create new index files before running "make
    install" by using swish-e from the build directory. Then copy the index
    files to the live location and run "make install":

        $ src/swish-e -c /path/to/config -f index.new

    Keep in mind that where you index from may effect the paths stored in
    the index file.

  Installing without root access
    Here's another installation example. This might be used if you do not
    have root access or you wish to install swish someplace other than
    /usr/local.

    This example also shows building Swish-e in a "build" directory separate
    from where the source files are located. This is the recommended way to
    build Swish-e, but requires GNU Make. Without GNU Make you will likely
    need to build from within the source directory as shown in the previous
    example.

        $ tar zxof swish-e-2.4.0.tar.gz  (GNU tar with "z" option)
        $ mkdir build
        $ cd build

    Note that the current directory is not where Swish-e was unpacked.

    Swish-e uses a configure script. configure has many options, but uses
    reasonable and standard defaults. Running

        $ ../swish-e-2.4.0/configure --help

    will display the options.

    Two options are of common interest: --prefix sets the top-level
    installation directory, and --disable-shared will link swish-e
    statically, which may be needed on some platforms (Solaris 2.6 perhaps).

    Platforms vary on how to link against libraries when libraries are
    installed in non-standard locations. Swish-e uses the GNU autoconf tools
    for building the package. Autoconf is good at building and testing, but
    still requires you to provide information appropriate for your platform.
    This may mean reading the manual page for your compiler and linker to
    see how to specify non-standard file locations.

    For most unix-type platforms you use LDFLAGS and CPPFLAGS environment
    variables to specify paths to include (header) files and libraries not
    in standard locations.

    Now, in this example we do not have root access, and we have installed
    libxml2 and libz in $HOME/local. Swish-e will also be installed in
    $HOME/local (by using the --prefix setting).

    In this case you would need to add $HOME/local/bin to the start of your
    shell's PATH setting. This is required because libxml2 installs a
    program that is used when running the configure script. Before running
    configure type:

        $ which xml2-config

    And it should list $HOME/local/bin/xml2-config.

    Now run configure (remember, we are in a separate "build" directory):

        $ ../swish-e-2.4.0/configure \
            --prefix=$HOME/local \
            CPPFLAGS=-I$HOME/local/include \
            LDFLAGS="-R$HOME/local/lib -L$HOME/local/lib"

        $ make >/dev/null  (redirect output to only see warnings and errors)

        $ make check
        ...
        ==================
        All 3 tests passed
        ==================

        $ make install
        $ $HOME/local/bin/swish-e -V 
        SWISH-E 2.4.0

    Note the use of dobule quotes in the LDFLAGS line above. This allows
    $HOME to be expanded.

  Run-time paths
    The -R option says to add that path to the paths used to find shared
    libraries at run time. This path is stored in the swish-e binary. When
    swish-e is run it will look in this directory for shared libraries.

    Some platforms may not support the -R option and you may instead have to
    set the LD_RUN_PATH environment variable before running make.

    Some systems such as Redhad do not look in /usr/local/lib for libraries.
    In these cases you can either use -R as above when building swish-e or
    add /usr/local/lib to /etc/ld.so.conf and run ldconfig as root.

    If all else fails, you may need to actually read the man pages for your
    platform.

  Building a Debian Package
    The Swish-e distribution includes the files requires to build a Debian
    pakcage.

        $ tar zxof swish-e-2.4.0.tar.gz  (GNU tar with "z" option)
        $ cd swish-e-2.4.0
        $ fakeroot debian/rules binary
        [lots of output]
        dpkg-deb: building package `swish-e' in `../swish-e_2.4.0-0_i386.deb'.
        $ su
        # dpkg -i ../swish-e_2.4.0-0_i386.deb

  What's installed
    Swish installs a number of files. By default all files are installed
    below /usr/local, but this can be changed by setting --prefix when
    running configure (as shown above). Individual paths may also be set.
    Run configure --help for details.

       $prefix/bin/swish-e         The swish-e binary program
       $prefix/share/doc/swish-e/  Full documentation and examples
       $prefix/lib/libswish-e      The swish-e C library
       $prefix/include/swish-e.h   The library header file
       $prefix/man/man1/           Documentation as manual pages
       $prefix/lib/swish-e/        Helper programs (spider.pl, swishspider, swish.cgi)
       $prefix/lib/swish-e/perl/   Perl helper modules

    Note that the Perl modules are *not* installed in the system Perl
    library. Swish-e and the Perl scripts that require the modules know
    where to find the modules, but the perldoc program used for reading
    documentation does not. This can be corrected by adding
    $prefix/lib/swish-e and $prefix/lib/swish-e/perl to the PERL5LIB
    environment variable.

  Documentation
    Documentation can be found in the $prefix/share/doc/swish-e directory.
    Documentation can also be read on-line at the Swish-e web site:

        http://swish-e.org/

  The Swish-e documentation as man(1) pages
    Running "make install" installs some of the Swish-e documentation as man
    pages. The following man pages are installed:

        SWISH-FAQ(1)
        SWISH-CONFIG(1)
        SWISH-RUN(1)
        SWISH-LIBRARY(1)

    The man pages are installed in the system man directory. This directory
    is determined by running ./configure and can be set by passing the
    directory when running ./configure.

    For example,

        ./configure --mandir=/usr/local/doc/man

    The man director is relative to the --prefix setting. If you use
    --prefix you do not normally need to also specify --mandir.

    Information on running ./configure can be found by typing:

        ./configure --help

  Join the Swish-e discussion list
    The final step when installing Swish-e is to join the Swish-e discussion
    list.

    The Swish-e discussion list is the place to ask questions about
    installing and using Swish-e, see or post bug fixes or security
    announcements, and a place where you can offer help to others. Please do
    not contact the developers directly.

    The list is typically *very low traffic*, so it won't overload your
    inbox. Please take time to subscribe. See http://Swish-e.org.

    If you are using Swish-e on a public site, please let the list know so
    it can be added to the list of sites that use Swish-e!

    Please review the next section before posting a question to the Swish-e
    list.

QUESTIONS AND TROUBLESHOOTING
    Support for installation, configuration and usage is available via the
    Swish-e discussion list. Visit http://swish-e.org for information. Do
    not contact developers directly for help -- always post your question to
    the list.

    It's very important to provide the right information when asking for
    help.

    Please search the Swish-e list archive before posting a question, and
    check the SWISH-FAQ to see if your question hasn't already been asked.

    Before posting use tools available to narrow down the problem.

    Swish-e has the -T, -v, and -k switches that may help resolve issues.
    These switches are described on the SWISH-RUN page. For example, if you
    cannot find a document by a keyword that you believe should be indexed
    try indexing just that single file, and use the -T INDEXED_WORDS option
    to see if the word is actually being indexed. First try without any
    changes to default settings:

        swish-e -i testdoc.html -T indexed_words | less

    if that works then add in your configuration file:

        swish-e -i testdoc.html -c swish.conf -T indexed_words | less

    If that still isn't working as you expect try to reduce the test
    document to a very small example. This will be very helpful when asking
    for help.

    Other tools are to use -H9 when searching to display full headers in
    search results. Look at the "Parsed Words" header to see what words
    swish-e is searching for.

  When posting please provide the following information:
    Use these guidelines when asking for help. The most important tip is to
    provide the least amount of information that can be used to reproduce
    your problem. Do not paraphrase output -- copy-and-paste -- but trim
    text that is not necessary.

    *   The exact version of Swish-e that you are using. Running Swish-e
        with the "-V" switch will print the version number. Also, supply the
        output from "uname -a" or similar command that identifies the
        operating system you are running on. If you are running an old
        version of swish be prepared for a response to your question of
        "upgrade."

    *   A summary of the problem. This should include the commands issued
        (e.g. for indexing or searching) and their output, and why you don't
        think it's working correctly. Please cut-n-paste the exact commands
        and their output instead of retyping to avoid errors.

    *   Include a copy of the configuration file you are using, if any.
        Swish-e has reasonable defaults so in many cases you can run it
        without using a configuration file. But, if you need to use a
        configuration file, reduce it down to the absolute minimum number of
        commands required to demonstrate your problem. Again, cut-n-paste.

    *   A small copy of a source document that demonstrates the problem.

        If you are having problems spidering a web server, use lwp-download
        or wget to copy the file locally to make sure you can index the
        document using the file system method. This will help determine if
        the problem is with spidering or with indexing.

        If you expect help with spidering, don't post fake URLs, as it makes
        it impossible to test. If you don't want to expose your web page to
        the people on the Swish-e list, find some other site to test
        spidering on. If that works, but you still cannot spider your own
        site then post your real URL if you want help, or make a test
        document available via some other source.

    *   If you are having trouble building Swish-e please cut-n-paste the
        output from make (or from ./configure if that's where the problem
        is).

    The key is to provide enough information so that others may reproduce
    the problem.

ADDITIONAL INSTALLATION OPTIONS
    These steps are not required for normal use of Swish-e.

  The SWISH::API Perl Module
    The Swish-e distribution includes a module that provides a Perl
    interface to the Swish-e C library. This module provides a way to search
    a Swish-e index without running the swish-e program. Searching an index
    will be many times faster when running under a persistent environment
    such as Apache/mod_perl with the SWISH::API module.

    See the perl/README file for information on installing and using the
    SWISH::API Perl module.

  Creating PDF and Postscript documentation
    The Swish-e documentation in HTML format was created with
    Pod::HtmlPsPdf, a package of Perl modules written and/or modified by
    Stas Bekman to automate the conversion of documents in pod format (see
    perldoc perlpod) to HTML, Postscript, and PDF. A slightly modified
    version of this package is included with the Swish-e distribution and
    used for building the HTML.

    If your system has the necessary tools to build Postscript and the
    converter ps2pdf installed, you may be able to build the Postscript and
    PDF versions of the documentation. After you have run ./configure, type
    from the doc directory of the distribution:

        make pdf

    And with any luck you will end up with the these two files in the
    top-level directory:

        swish-e_documentation.pdf
        swish-e_documentation.ps

    Most people find reading the documentation in HTML most convenient.

GENERAL CONFIGURATION AND USAGE
    This section should give you a basic overview of indexing and searching
    with Swish-e. Other examples can be found in the conf directory which
    will step you through a number of different configurations. Also, please
    review the SWISH-FAQ.

    Swish-e is a command line program. The program is controlled by passing
    switches on the command line. A configuration file may be used, but
    often is not required. Swish-e does not include a graphical user
    interface. There are example CGI scripts provided in the distribution,
    but they require additional setup to use.

  Introduction to Indexing and Searching
    Swish-e can index files on the local file system. For example, running:

         swish-e -i /var/www/htdocs

    will index *all* files in the /var/www/htdocs directory. You may specify
    one or more files or directories with the -i option. By default this
    will create an index (which is made up of more than one file) in the
    current directory called index.swish-e.

    Then to search the resulting index for a given word:

         swish-e -w apache

    This will find the word "apache" in the body or title of the indexed
    documents.

    As mentioned above, Swish-e will index all files in a directory unless
    instructed otherwise. So if /var/www/htdocs contains non-HTML then you
    will need a configuration file to limit the files that Swish-e indexes.
    Create a file called "swish.conf":

        # Example configuration file

        # Tell swish what to index (same as -i switch above)
        IndexDir /var/www/htdocs

        # Only index HTML and text files
        IndexOnly .htm .html .txt

        # Tell swish that .txt files are to use the text parser.
        IndexContents TXT* .txt

        # Otherwise, use the HTML parser
        DefaultContents HTML*

    Save that as "swish.conf" and reindex:

        swish-e -c swish.conf

    The Swish-e configuration settings are described in the SWISH-CONFIG
    manual page. Order of statements in the configuration file is typically
    not important, although some statements depend on previously set
    statements. There are many possible settings. Good advice is to use as
    few settings as possible when first starting out with Swish-e.

    The runtime options (switches) are described in the SWISH-RUN manual
    page. You may also see a summary of options by running:

        swish-e -h

    Swish-e has two other methods reading input files. One method uses a
    Perl helper script and the LWP Perl library to spider remote web sites:

        swish-e -S http -i http://localhost/index.html -v2

    This will spider the web server running on the local host. The "-S"
    option defines the input source method to be "http", "-i" specifies the
    URL to spider, and "-v" sets the verbose level to two. There are a
    number of configuration options specific to the "-S" http input source.
    See SWISH-CONFIG. Note that only files of Content-Type text/* will be
    indexed.

    The "-S http" method is deprecated in favor of the next input method.

    The other method is a general purpose input method where Swish-e reads
    input from a program that produces documents in a special format. The
    program might read and format data stored in a database, or parse and
    format messages in a mailing list archive, or run a program that spiders
    web sites like the previous method.

    The Swish-e distribution includes a spider program that uses this method
    of input. This spider program is much more configurable and feature-rich
    than the previous -S http method.

    To duplicate the previous example create a configuration file called
    "swish2.conf"

        # Example for spidering
        # Use the "spider.pl" program included with Swish-e
        IndexDir spider.pl

        # Define what site to index
        SwishProgParameters default http://localhost/index.html

    Then create the index using the command:

        swish-e -S prog -c swish2.conf

    This says to use the "-S prog" input source method. Note that in this
    case the IndexDir settings does not list a file or directory to index,
    but a program name run. This program, spider.pl, does the work of
    fetching the documents from the web server and passing them to Swish-e
    for indexing.

    The SwishProgParameters options is a special feature that allows passing
    command line parameters to the program specified with IndexDir. In this
    case passing the word "default" which tells spider.pl to use default
    settings, and the URL to spider.

    Running a script under Windows requires specifying the interpreter (e.g.
    perl.exe) and then use SwishPropParameters to specify the script and the
    script's parameters. See *Notes when using -S prog on MS Windows* on the
    SWISH-RUN page.

    The advantage of the "-S prog" method of spidering (over the previous
    "-S http" method) is that the Perl code is only compiled once instead of
    for every document fetched from the web server. In addition it is a much
    more advanced spider with many, many features. Still, as used here,
    spider.pl will automatically index PDF or MS Word documents if (when)
    Xpdf and Catdoc are installed.

    A special form of the "-S prog" input source method is:

        ./myprog --option | swish-e -S prog -i stdin -c config

    This allows running Swish-e from a program (instead of running the
    external program from Swish-e). Thus, this also can be done:

        ./myprog --option > outfile
        swish-e -S prog -i stdin -c config < outfile

    or

        ./myprog --option > outfile
        cat outfile | swish-e -S prog -i stdin -c config

    One final note about the "-S prog" input source method. The program
    specified with -i or IndexDir needs to be an absolute path. The
    exception is when the program is installed in the "libexecdir" directory
    and then a plain program name may be specified (as in the example
    showing spider.pl above).

    All three input source methods are described in more detail on the
    SWISH-RUN page.

  Metanames and Properties
    There's two key Swish-e concepts that you need to be familiar with:
    Metanames and Properties.

    * Metanames
        Swish-e creates a reverse index. Just like an index in a book, you
        look up a word and it lists the pages (or documents) where that word
        can be found.

        Swish-e can create multiple index tables within the same index file.
        For example, you might want to create an index of just words in HTML
        titles so searches can be limited to just titles. Or you might have
        descriptive words in a meta tag called "keywords" you would like to
        search.

        Some database systems might call these different "fields" or
        "columns", but swish-e calls them *MetaNames* (as a result of first
        indexing HTML meta tags).

        To find documents with "foo" in their title you might run:

            swish-e -w swishtitle=foo

        or more advanced:

            swish-e -w swishtitle=(foo or bar) or swishdefault=(baz)

        The Metaname "swishdefault" is the name used by Swish-e if no other
        name is specified. The following two searches are the same:

            swish-e -w foo
            swish-e -w swishdefault=foo

        When indexing HTML documents Swish-e indexes words in the body and
        title under the Metaname "swishdefault".

    * Properties
        Swish-e search results is a list of files -- actually internally
        swish uses file numbers. Data can be associated with each file
        number when indexing. For example, by default Swish-e associates the
        file's name, title, last modified date, and size with the file
        number and these items can be printed in search results. In Swish-e
        this associated data is called a file's *Properties*. Properties can
        be any data you wish to associated with a document -- even the
        entire text of the document can be stored in the index. What data is
        stored as a Property is controlled by the *PropertyNames* (and
        others) configuration directive.

        What properties are printed with search results depends on the -x or
        -p switches. By default Swish-e returns the rank, path/URL, title
        and file size in bytes for each result.

  Getting Started With Swish-e
    Swish-e reads a configuration file (see SWISH-CONFIG) for directives
    that control what and how Swish-e indexes files. Swish-e is also
    controlled by command line arguments (see SWISH-RUN). Many of the
    command line arguments have equivalent configuration directives (e.g. -i
    and IndexDir).

    Swish-e does not require a configuration file, but most people need to
    change the default behavior by placing settings in a configuration file.

    To try the examples below you may change to the tests subdirectory of
    the distribution. The tests will use the *.html files in this directory
    when creating the test index. You may wish to review these *.html files
    to get an idea of the various native file formats that Swish-e supports.

    You may also use your own test documents. It's recommended to use small
    test documents when first using Swish-e.

  Step 1: Create a Configuration File
    The configuration file controls what and how Swish-e indexes. The
    configuration file consists of directives, comments, and blank lines.
    The configuration file can be any name you like.

    This example will work with the documents in the tests directory. You
    may wish to review the tests/test.config configuration file used for the
    "make test" tests.

    For example, a simple configuration file (swish-e.conf):

        # Example Swish-e Configuration file

        # Define *what* to index
        # IndexDir can point to a directories and/or a files
        # Here it's pointing to the current directory
        # Swish-e will also recurse into sub-directories.
        IndexDir .

        # But only index the .html files
        IndexOnly .html

        # Show basic info while indexing
        IndexReport 1

    And that's a simple configuration file. It says to index all the .html
    files in the current directory and sub-directories, if any, and provide
    some basic output while indexing.

    As mentioned above, the complete list of all configuration file
    directives are described in SWISH-CONFIG.

  Step 2: Index your Files
    Run Swish-e using the "-c" switch to specify the name of the
    configuration file.

        swish-e -c swish-e.conf

        Indexing Data Source: "File-System"
        Indexing "."
        Removing very common words...
        no words removed.
        Writing main index...
        Sorting words ...
        Sorting 55 words alphabetically
        Writing header ...
        Writing index entries ...
          Writing word text: Complete
          Writing word hash: Complete
          Writing word data: Complete
        55 unique words indexed.
        4 properties sorted.                                              
        5 files indexed.  1252 total bytes.  140 total words.
        Elapsed time: 00:00:00 CPU time: 00:00:00
        Indexing done!

    This created the index file index.swish-e. This is the default index
    file name unless the IndexFile directive is specified in the
    configuration file:

        IndexFile ./website.index

    You may use the -f switch to specify a index file at indexing time. The
    -f option overrides a IndexFile setting in the configuration file.

  Step 3: Search
    You specify your search terms with the "-w" switch. For example, to find
    the files that contain the word sample you would issue the command:

        swish-e -w sample

    This example assumes that you are in the tests directory. Swish-e
    returns in response to that command the following:

        swish-e -w sample

        # SWISH format: 2.4.0
        # Search words: sample
        # Number of hits: 2
        # Search time: 0.000 seconds
        # Run time: 0.005 seconds
        1000 ./test_xml.html "If you are seeing this, the METATAG XML search was successful!" 159
        1000 ./test.html "If you are seeing this, the test was successful!" 437
        .

    So the word sample was found in two documents. The first number shown is
    the relevance or rank of the search term, followed by the file
    containing the search term, the title of the document, and finally the
    length of the document.

    The period (".") alone at the end marks the end of results.

    Much more information may be retrieved while searching by using the "-x"
    and "-H" switches (see SWISH-RUN) and by using Document Properties (see
    SWISH-CONFIG).

  Phrase Searching
    To search for a phrase in a document use double-quotes to delimit your
    search terms. (The default phrase delimiter is set in src/swish.h.)

    You must protect the quotes from the shell.

    For example, under Unix:

        swish-e -w '"this is a phrase" or (this and that)'
        swish-e -w 'meta1=("this is a phrase") or (this and that)'

    Or under Windows command.com shell.

        swish-e -w \"this is a phrase\" or (this and that)

    The phrase delimiter can be set with the "-P" switch.

  Boolean Searching
    You can use the Boolean operators and, or, or not in searching. Without
    these Boolean, Swish-e will assume you're anding the words together.

    Here are some examples:

        swish-e -w 'apples oranges'
        swish-e -w 'apples and oranges'  ( Same thing )

        swish-e -w 'apples or oranges'

        swish-e -w 'apples or oranges not juice' -f myIndex 

    retrieves first the files that contain both the words "apples" and
    "oranges"; then among those the ones that do not contain the word
    "juice"

    A few others to ponder:

        swish-e -w 'apples and oranges or pears'
        swish-e -w '(apples and oranges) or pears'  ( Same thing )
        swish-e -w 'apples and (oranges or pears)'  ( Not the same thing )

    Swish processes the query left to right.

    See SWISH-SEARCH for more information.

  Context Searching
    The "-t" option in the search command line allows you to search for
    words that exist only in specific HTML tags. Each character in the
    string you specify in the argument to this option represents a different
    tag in which the word is searched; that is you can use any combinations
    of the following characters:

        H search in all <HEAD> tags
        B search in the <BODY> tags
        t search in <TITLE> tags
        h is <H1> to <H6> (header) tags
        e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>)
        c is HTML comment tags (<!-- ... -->)

    For example:

        # Find only documents with the word "linux" in the <TITLE> tags.
        swish-e -w linux -t t

        # Find the word "apple" in titles or comments
        swish-e -w apple -t tc

  META Tags
    As mentioned above, Metanames are a way to define "fields" in your
    documents. You can use the Metanames in your queries to limit the search
    to just the words contained in that META name of your document. For
    example, you might have a META tagged field in your documents called
    "subjects" and then you can search your documents for the word "foo" but
    only return documents where "foo" is within the "subjects" META tag.

    Document *Properties* are somewhat related: Properties allow the content
    of a META tag in a source document to be stored within the index, and
    that text to be returned along with search results.

    META tags can have two formats in your documents.

        <META NAME="keyName" CONTENT="some Content">

    And in XML format

        <keyName>
            Some Content
        </keyName>

    If using libxml, you can optionally use a non-html tag as a metaname:

        <html>
        <body>
            Hello swish users!
            <keyName>
                this is meta data
            </keyName>.
        </body>

    This, of course, is invalid HTML.

    To continue with our sample Swish-e.conf file, add the following lines:

        # Define META tags
        MetaNames meta1 meta2 meta3

    Reindex to include the changes:

        swish-e -c swish-e.conf

    Now search, but this time limit your search to META tag "meta1":

        swish-e -w 'meta1=metatest1'

    Again, please see SWISH-RUN and SWISH-CONFIG for complete documentation
    of the various indexing and searching options.

  Spidering and Searching with a Web form.
    This example demonstrates how to spider a web site and setup the
    included CGI script to provide a web-based search page. This example
    uses Perl programs included in the Swish-e distribution: spider.pl will
    be used for reading files from the web server, and swish.cgi will
    provide the web search form and display results.

    As an example we will index the Apache Web Server documentation
    installed on the local computer at
    http://localhost/apache_docs/index.html

    1 Make a Working Directory
        Create a directory to store the Swish-e configuration and the
        Swish-e index.

            ~$ mkdir web_index
            ~$ cd web_index/
            ~/web_index$

    2 Create a Swish-e Configuration file
            ~/web_index$ cat swish.conf 
            # Swish-e config to index the Apache documentation
            #
            # Use spider.pl for indexing (location of spider.pl set at installation time)
            IndexDir spider.pl

            # Use spider.pl's default configuration and specify the URL to spider
            SwishProgParameters default http://localhost/apache_docs/index.html

            # Allow extra searching by title, path
            Metanames swishtitle swishdocpath

            # Set StoreDescription for each parser
            #  to display context with search results
            StoreDescription TXT* 10000
            StoreDescription HTML* <body> 10000

    3 Generate the Index
        Now run swish-e to create the index:

            ~/web_index$ swish-e -S prog -c swish.conf 

            Indexing Data Source: "External-Program"
            Indexing "spider.pl"
            /usr/local/lib/swish-e/spider.pl: Reading parameters from 'default'

            Summary for: http://localhost/apache_docs/index.html
                Duplicates:     4,188  (349.0/sec)
            Off-site links:       276  (23.0/sec)
                   Skipped:         1  (0.1/sec)
               Total Bytes: 2,090,125  (174177.1/sec)
                Total Docs:       147  (12.2/sec)
               Unique URLs:       149  (12.4/sec)
            Removing very common words...
            no words removed.
            Writing main index...
            Sorting words ...
            Sorting 7736 words alphabetically
            Writing header ...
            Writing index entries ...
              Writing word text: Complete
              Writing word hash: Complete
              Writing word data: Complete
            7736 unique words indexed.
            5 properties sorted.                                              
            147 files indexed.  2090125 total bytes.  200783 total words.
            Elapsed time: 00:00:13 CPU time: 00:00:02
            Indexing done!

        The above output is actually a mix of output from both swish-e and
        from spider.pl. Spider.pl reports the "Summary for:
        http://localhost/apache_docs/index.html".

        Also note that swish-e knows to find spider.pl at
        /usr/local/lib/swish-e/spider.pl. The script installation directory
        (called libexecdir) is set at configure time. You can see your
        setting by running swish-e -h:

            ~/web_index$ swish-e -h | grep libexecdir
             Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e

        This directory will be needed when setting up the CGI script in the
        next step.

        Finally, verify that the index can be searched from the command
        line:

            ~/web_index$ swish-e -w installing -m3
            # SWISH format: 2.4.0
            # Search words: installing
            # Removed stopwords: 
            # Number of hits: 17
            # Search time: 0.018 seconds
            # Run time: 0.050 seconds
            1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
            718 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
            680 http://localhost/apache_docs/windows.html "Using Apache with Microsoft Windows" 27165
            .

        Now try limiting the search to the title:

            ~/web_index$ swish-e -w swishtitle=installing -m3 
            # SWISH format: 2.3.5
            # Search words: swishtitle=installing
            # Removed stopwords: 
            # Number of hits: 2
            # Search time: 0.018 seconds
            # Run time: 0.048 seconds
            1000 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
            1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
            .

        Note that the above can also be done using the -t option:

            ~/web_index$ swish-e -w installing -m3 -tH

    4 Setup the CGI script
        Swish-e does not include a web server, therefore you must use your
        locally installed web server. Apache is highly recommended, of
        course.

        Locate your web server's CGI directory. This may be a cgi-bin
        directory in your home directory or a central cgi-bin directory
        setup by the web server administrator. Once located copy the
        swish.cgi script into the cgi-bin directory.

        Where CGI scripts can be located depends completely on the web
        server used and how it is configured. See your web server's
        documentation or your site's administrator for additional
        information.

        This example will use a site cgi-bin directory located at
        /usr/lib/cgi-bin. Copy the swish.cgi script into the cgi-bin
        directory. Again, we will need the location of the libexecdir
        directory:

            ~/web_index$ swish-e -h | grep libexecdir
             Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e

            ~/web_index$ cd /usr/lib/cgi-bin
            /usr/lib/cgi-bin$ su
            Password: 
            /usr/lib/cgi-bin# cp /usr/local/lib/swish-e/swish.cgi .

        If your operating system supports symbolic links, AND your web
        server allows programs to be symbolic links, then you may wish to
        create a link to the swish.cgi program instead.

            /usr/lib/cgi-bin# ln -s /usr/local/lib/swish-e/swish.cgi

        We need to tell the swish.cgi script where to look for the index
        created in the previous step. It's also recommended to enter the
        path to the swish-e binary, otherwise the swish.cgi script will look
        for the binary in the PATH, and that may change when running under
        the CGI environment.

        Here's the configuration file:

            /usr/lib/cgi-bin# cat .swishcgi.conf 
            return {
                title        => 'Search Apache Documentation',
                swish_binary => '/usr/local/bin/swish-e',
                swish_index  => '/home/moseley/web_index/index.swish-e',
            }

        Now, test the script from the command line as a normal user:

            /usr/lib/cgi-bin# exit
            exit

            /usr/lib/cgi-bin$  ./swish.cgi | head
            Content-Type: text/html; charset=ISO-8859-1

            <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
            <html>
                <head>
                   <title>
                      Search Apache Documentation
                   </title>
                </head>
                <body>

        Notice that the CGI script returns the HTTP header (Content-Type)
        and the body of the web page, just like a well behaved CGI scrip
        should do.

        Now test using the web server (this step depends on the location of
        your cgi-bin directory). This example uses the "GET" command that is
        part of the LWP Perl library, but any web browser can run this test.

            /usr/lib/cgi-bin$ GET http://localhost/cgi-bin/swish.cgi | head
            <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Tranitional//EN">
            <html>
                <head>
                   <title>
                      Search Apache Documentation
                   </title>
                </head>
                <body>
                    <h2>

        The script reports errors to stderr, so consult the web server's
        error log if problems occur. The message "Service currently
        unavailable" reported by running swish.cgi typically indicates a
        configuration error, and the exact problem will be listed in the web
        server's error log.

        Detailed instructions on using the swish.cgi script and debugging
        tips can be found by running:

            $ perldoc swish.cgi

        while in the cgi-bin directory where swish.cgi was copied.

        The spider program spider.pl also has a large number of
        configuration options.

        Documentation is also available in the directory
        $prefix/share/doc/swish-e or at http://swish-e.org.

        Note: Also check out the search.cgi script found at the same
        location as the swish.cgi script. This is more of a skeleton script
        for those that want to create a custom search script.

    Now you are ready to search.

Indexing Other Types of Documents - Filtering
    Swish-e can only index HTML, XML and text documents. In order to index
    other documents such as PDF or MS Word documents you must use a utility
    to convert or "filter" those documents.

    How documents are filtered with Swish-e has changed over time. This has
    resulting in a bit of confusion. It's also a somewhat complex process as
    different programs need to communicate with each other.

    You may wish to read the Swish-e FAQ question on filtering before
    continuing here. How Do I filter documents?

  Filtering Overview
    There's two ways to filter documents with Swish-e. Both are described in
    the SWISH-CONFIG man page. They are using the FileFilter directive and
    the SWISH::Filter perl module.

    The FileFilter directive is a general purpose method of filtering. It
    allows running of an external program for each document processed (based
    on file extension), and requires. The external programs open an input
    file, convert as needed, and write their output to standard output.

    Previous versions of Swish-e (before 2.4.0) used a collection of filter
    programs for converting files such as PDF or MS Word documents. The
    external programs call other program do to the work of filtering (e.g.
    pdftotext to extract the contents from PDF files). Although these filter
    programs are still included with the Swish-e distribution as examples,
    it is not recommended to use the SWISH::Filter method instead.

    One disadvantage of using FileFilter is that the filter is called once
    for every document that needs to be filtered. This can slow down the
    indexing process.

    The SWISH::Filter Perl module works very much like the old system and
    uses the same helper programs. But, it provides a single interface for
    filtering all types of documents. The advantage of SWISH::Filter is that
    it is built into the program used for spidering web sites (spider.pl),
    so all that's required is installing the filter programs that do the
    actual work of filtering (e.g. catdoc, xpdf). (The Windows binary
    includes some of the filter programs.)

    But, Swish-e will not use SWISH::Filter by default when using the file
    system method of indexing. To use SWISH::Filter when indexing by file
    system method (-S fs) you can use a FileFilter directive with the
    "swish_filter.pl" filter (which is just a program that uses
    SWISH::Filter), or use the -S prog method of indexing and use the
    DirTree.pl program for fetching documents. DirTree.pl is included with
    the Swish-e distribution and is designed to work with SWISH::Filter.
    Using DirTree.pl will likely be faster way to index since the
    SWISH::Filter set of modules do not need to be compiled for every
    document that needs to be filtered.

    See the contents of swish_filter.pl and DirTree.pl for specifics on
    their use.

  Filtering Examples
    The FileFilter directive can be used in your config file to convert
    documents based on their extension. This is the old way of filtering,
    but provides an easy way to add filters to swish-e.

    For example:

        FileFilter .pdf  pdftotext   "'%p' -"
        IndexContents TXT* .pdf

    will cause all .pdf files to be filtered through the pdftotext program
    (part of the Xpdf package), and to parse the resulting output from
    pdftotext with the text ("TXT") parser.

    The other ways to filter documents is to use a -S prog program and
    convert the documents before passing them onto Swish-e.

    For example, spider.pl makes use of the Perl module included with the
    Swish-e distribution called SWISH::Filter. SWISH::Filter is passed a
    document and the document's content type and then looks for modules and
    utilities to convert the document into one of the types that Swish-e can
    index.

    Swish-e comes ready to index PDF, MS Word, MP3 ID3 tags, and MS Excel
    file types. But these filters need extra modules or tools to do the
    actual conversion.

    For example, the Swish-e distribution includes a module called
    SWISH::Filter::Pdf2HTML that uses the pdftotext and pdfinfo utilities
    provided by the Xpdf package.

    This means that if you are using spider.pl to spider your web site and
    you wish to index PDF documents, all that is needed is to install the
    Xpdf package and Swish-e (with the help of spider.pl) will begin
    indexing your PDF files.

    Ok, so what does all that mean? For a very simple site you should be
    able to run this:

        $ /usr/local/lib/swish-e/spider.pl default http://localhost/ | swish-e -S prog -i stdin

    which is running the spider with default spider settings, indexing the
    Web server on localhost and piping its output into Swish-e using default
    indexing settings. Documents will be filtered automatically if you have
    the required helper applications installed.

    Most people will not want to just use the default settings (for one
    thing the spider will take a while because its default is to delay a few
    seconds between every request). So, read the documentation for spider.pl
    to learn how to use a spider config file. And also read SWISH-CONFIG to
    learn about what configuration options can be used with Swish-e.

    The SWISH::Filter documentation provides more details on filtering and
    hints for debugging problems when filtering.

Document Info
    $Id: INSTALL.pod,v 1.36.2.5 2004/01/30 19:47:17 whmoseley Exp $

    .

