Web Site Upload Instructions

Beginning May 2002 all user directories and web sites were migrated to new servers. Instructions for uploading files into web sites on these new servers are in this document.

FTP Upload Instructions

To upload files via ftp, connect via ftp to ftp.rahul.net and log in with your login name and password. (Don't use anonymous ftp -- it will not work for uploads. Also, if you have a custom domain such as www.example.com, do not connect to it for ftp uploads. You may reach the wrong directories by doing so.)

After logging in for ftp, if you get a directory listing from ftp, you will see the following subdirectories:

shell: this is your home directory
www: this is your web (html) directory
cgi-bin: this is your cgi-bin directory

(If you don't see these directories, ask your ftp client to move up one directory level and then look again.)

Not all of the directories will exist for each user. But if you have a web site, and if your web site is on one of our new servers, then you should see at least the 'www' directory. If you don't see this directory, it means that you either do not have a web site, or there is some other problem. If you have any questions, please contact support@rahul.net.

Then upload your html files into the 'www' subdirectory and optionally upload any cgi-bin programs into the 'cgi-bin' subdirectory. Note: Not all web sites use cgi-bin programs.

UNIX Shell Instructions

(Skip this section if you are uploading files via ftp.) If you are a UNIX shell user, then connect to any of our UNIX machines of your choice, using 'rlogin', 'telnet', or 'ssh'. We advise using 'ssh' for best security. Then, after logging in, use the following command:

     cd ..

This will put you into a directory just above your home directory. Now if you use the 'ls' command you will see the same three subdirectories mentioned above for ftp uploads. You may copy files into these directories as needed. Note that any files that you wish to make available via http should be made world-readable, and any new directories that you create for http access should be made world-search-able. Otherwise the http server will not be able to access them. To make a file 'xxx' world-readable:

     % chmod a+r xxx

To make a directory 'yyy' world-search-able:

     % chmod a+x yyy

Pathnames in URLs

If your web site URL is of the form 'http://www.example.com/', then:

Any file called 'xxx.html' that has been uploaded into your 'www' directory has this URL:
http://www.example.com/xxx.html
Any file called 'testcgi.sh' that has been uploaded into your 'cgi-bin' directory has this URL:
http://www.example.com/cgi-bin/testcgi.sh

If your web site URL is of the form 'http://www.rahul.net/USER/', then:

Any file called 'xxx.html' that has been uploaded into your 'www' directory has this URL:
http://www.rahul.net/USER/xxx.html
Any file called 'testcgi.sh' that has been uploaded into your 'cgi-bin' directory has this URL:
http://www.rahul.net/cgi-bin/USER/testcgi.sh

For testing cgi-bin execution, we normally pre-populate your 'cgi-bin' directory with two test programs, called 'testcgi.sh' and 'testcgi.pl'. Their respective URLs, depending on whether or not you have a direct domain URL, are:

http://www.example.com/cgi-bin/testcgi.sh

http://www.example.com/cgi-bin/testcgi.pl

http://www.rahul.net/cgi-bin/USER/testcgi.sh

http://www.rahul.net/cgi-bin/USER/testcgi.pl

'USER' is your username.

Operating System-Specific cgi-bin execution

After the migration, all html accesses are on the new servers, but cgi-bin programs were leftto continue to execute in the old SunOS environment on our older machines. You may, if you wish, ask us to configure your web site so your cgi-bin programs execute in the Linux environment on our newer machines. Regardless of where the cgi-bin programs run, all static content (e.g., html pages and graphics) are served from our new faster servers running Linux. Newly added web sites are configured to run cgi-bin programs only on the new servers. If you wish to transition to running cgi-bin programs in the Linux environment, please let us know by email.

(This section is now obsolete.

As a transition aid, you can cause any cgi-bin program to run in a specific environment with URLs like those shown below. Inserting a subdirectory name '/sunos' or '/linux' just before the '/cgi-bin' part forces the cgi-bin program to run in the specified environment.

Consider these URLs:

http://www.example.com/cgi-bin/testcgi.sh

http://www.rahul.net/cgi-bin/USER/testcgi.sh

http://www.example.com/sunos/cgi-bin/testcgi.sh

http://www.rahul.net/sunos/cgi-bin/USER/testcgi.sh

http://www.example.com/linux/cgi-bin/testcgi.sh

http://www.rahul.net/linux/cgi-bin/USER/testcgi.sh

If the architecture is not specified (as in the first two URLs above) then the default execution environment will be used. This default will remain as SunOS for old web sites, but will be Linux for new web sites. The default for your web site can be changed from SunOS to Linux if you so request. If the architecture is specified (as in the remaining URLs) then the specified execution environment will be used.

You do not need to create any subdirectory called 'sunos' or 'linux'. Even if you use the '/sunos/cgi-bin' string or the '/linux/cgi-bin' string to force cgi-bin execution in a specific environment, your cgi-bin programs will be kept in the same 'cgi-bin' directory as described earlier in this document.

Note: Not every web site is automatically configured to allow the '/sunos' and '/linux' prefixes. If you have problems using these, please send email to support@rahul.net and request that we enable this feature for your web site.

End of obsolete section.)

Anonymous ftp

Anonymous ftp downloads are supported only in a very limited way. Any user connecting to your ftp site via anonymous ftp will find himself in a .ftpdir subdirectory within your www directory. If you have not created such a subdirectory, then the user will not be able to find it and will see an error message. Note: You can if you wish make .ftpdir a symbolic link to any directory owned by you.

Anonymous ftp connections should be made to a URL like this:

ftp://ftp.rahul.net/USER

where USER is your username. This URL will lead the anonymous ftp user to your .ftpdir directory. The directory part, i.e., "/USER", is required.

If a user tries to access an ftp URL that specifies a deeper subdirectory, the attempt will succeed only if you have created a corresponding deeper subdirectory within your .ftpdir directory.

For example, suppose you have announced this ftp URL: ftp://ftp.rahul.net/USER/a/b/c/README. Then a user making an anonymous ftp connection to that URL will find himself retrieving the file .ftpdir/a/b/c/README under your www directory.

If the file does not exist, the user will not find it. If the file does exist, the user will be able to download only about 200 bytes of it. You may create a small file or 200 bytes or less announcing where the user should go to download via http.

Directory Moves

Here is quick summary of how directories are being moved during the migration.

~aftp/pub/USER/www moves to the new location of your www directory as described previously.
~aftp/pub/USER/www/cgi-bin moves to the new location of your cgi-bin directory as described previously.
Alternatively ~aftp/pub/USER/cgi-bin moves to the new location of your cgi-bin directory as described previously.
~aftp/pub/USER, if nonempty after the above moves, moves to a subdirectory called FTP under your new www directory.

Debugging Hints

Remote Client IP Address for CGI Use

Due to our web servers being accessed through a front-end proxy, the REMOTE_ADDR environment variable seen by your cgi-bin programs will contain a fixed local IP address and not the IP address of the remote web client. To get the IP address of the remote client, look in the environment variable HTTP_X_FORWARDED_FOR. This will contain a list of one or more IP addresses separated by comma or blank. The last IP address in this list is the IP address of the client connecting to our web server. If that client itself is a proxy server, then there will be additional IP addresses in the list. The first IP address in the list is the IP address of the most remote client, in case a chain of proxies is involved.

Quick Access to Web Server Logs

The following links are available to let you examine the web server logs while debugging your web site. Each link will give you the last few lines in each of the available server logs in the appropriate environment. Simply reload the URL as needed to see updated log output.

Using the Old Server

If you encounter any problems with your web site on our new servers, you can (in most cases) also access your web pages via an old server for comparison and debugging. Likewise, if you want to retrieve the last value of the old hit counter, you can do so by accessing your web page via an old server. Simply access your web page via a URL like this:

http://tango.rahul.net/USER/...

USER is your username. This URL format will work for most users. In some cases, if you had a very old web site that was configured in a nonstandard manner, the above URL will not work. In that case, we can upon request still provide you with an alternate URL that will still let you access your web pages via an old server.

Such legacy URLs are supported to ease the transition and should not be used on a permanent basis.

Full Access to Web Server Logs

If your web site gives you any problems, such as server errors, after the migration, check for error messages from the web server. Check both old and new servers to be sure. You will need to look at the server's error log if debugging cgi-bin programs, .htaccess files, and files using server-side includes. If you are not familiar with any of these terms, then you can safely ignore this section, as it's likely that you are not using these features and you will not need to check the server's error log.

The locations of all the web server log files are given below.

The normal log file is called access_log and the error log file is called error_log. These filenames are the same for both the old and new servers. Locations of log files are below.
Old server logs are on the machine tango.rahul.net in the directory /etc/LOCAL/httpd/logs.
New server logs are on the machine mauve.rahul.net in the directory /etc/httpd/logs.

The easiest way of looking at the server logs is to use the less or tail programs while logged into the UNIX shell. You will usually want to look near the end of the log file. After doing a cd into the appropriate directory, use commands such as these:

To get the complete web logs for one or more days conveniently, please see the section ``Web Log Generation'' later in this document.

tail -f error_log -- to follow the growing error log in real time. Abort with control C.

tail -f access_log -- to follow the growing normal log in real time. Abort with control C.

less -n ++G error_log -- to browse the error log near its end. Abort with q.

less -n ++G access_log -- to browse the normal log near its end. Abort with q.

We advise against trying to do ftp downloads of these log files, because they can be very large in size.

Server Differences

We don't expect any major differences to be noticeable between old and new servers. Any differences that are found will be listed in this section. So far the following differences have been encountered by users.

cgi-bin user. On the old web servers all cgi-bin programs were run as the user 'nobody'. On the new servers all cgi-bin programs will run as your own username. This mostly matters only if your cgi-bin programs are updating your files. On the old servers it was necessary to keep the files being updated world-writable. On the new servers it's not necessary to make files world-writable.
Pathnames. Be sure that all pathnames you use in any web files are official pathnames. The official pathname of your home directory can be printed, while logged into the UNIX shell, with the command "echo ~". All new pathnames are of the form /xx/home/USER where /xx is /ma, /mi, /mf, etc. If your official home directory pathname is /mi/home/joeuser/shell, then the official pathname of your web directory will be /mi/home/joeuser/www. Old pathnames of the form /files/home/ftp/pub/USER should not be used any more for web sites that have been migrated to a new server, because the new server will not be able to find such pathnames.
.htaccess file syntax: If you are using the AuthName directive, be sure that its argument is either a single word, or a string enclosed in single or double quotes. The old server permitted the argument to be a phrase; the new server does more strict checking. If you were using a line like this in your .htaccess file:
AuthName Joe's Web Service
then you will need to change it to:
AuthName "Joe's Web Service"
Hit counter. The new web server does not support the hit counter that was built into our old web servers. You may use any of the commonly-available hit counters that you can find on the Internet.

Locations of Common Programs

For use in cgi-bin programs, the pathnames given below will work on both the old (SunOS) and new (Linux) servers.

perl: /usr/local/bin/perl5.
sendmail: /usr/sbin/sendmail.

Web Log Generation: Automatic

Once a night a log of your web hits will be automatically saved in a file called wwwlog_all in your web directory.

Note: All web logs are always publicly readable. There is no way of making web logs private. Even if you could set the protections on a specific log file that you generate, this will not make the information private, because our web servers generate raw logs that are always publicly readable.

Optionally you can exert some control over the automatic generation of web logs, by creating a file called .wwwlog.rc and putting it in your top-level web directory. In this file, empty lines and lines beginning with the '#' character are ignored. Remaining lines are interpreted as follows.

days N: N is any whole number such as 1, 2, 3, etc.. It may be negative (see below.) If N is 1 or greater, then when the automatic log is generated, previous generations will be saved as files called wwwlog_all.1, wwwlog_all.2, and so on, for that many days of old logs. If N is 0, then no old logs will be saved -- only a fresh wwwlog_all will be generated nightly. If N is negative, then no log of web hits will be generated at all.
no4: If the no4 keyword is present, all web hits whose status code was in the range 400 to 499 will be omitted from the automatically-generated web logs. This is a good way of excluding random invalid hits from machines running Microsoft operating systems infected with various viruses, worms, and other software anomalies.

Note: The keywords diskspace, numreqs, analog, sendlog, and tsendlog, which were recognized in the past, are currently ignored. If/when they are re-enabled this document will be revised to indicate so.

If you find that the automatic web log wwwlog_all is not being generated for you, the two likely reasons for this are: (a) you do not have enough disk quota available to generate it, or (b) your top-level web directory is not writable by you. To check further, please see if you can manually generate a log file in your top-level web directory following the manual log-generation instructions below.

Web Log Generation: Manual

Note: All web logs are always publicly readable. There is no way of making web logs private.

Log into the UNIX shell on one of these machines: mauve.rahul.net or yellow.rahul.net.
To get your web site logs for the previous 1, 2, or 3 days, ending at midnight, use any one of the following commands:
db.get.log -s -E midnight -D 1 > log.txt
db.get.log -s -E midnight -D 2 > log.txt
db.get.log -s -E midnight -D 3 > log.txt
This will put the log into a file called log.txt. You may specify any filename of your choice.
To get only a summary, change the -s to -S like this:
db.get.log -S -E midnight -D 1 > log.txt
db.get.log -S -E midnight -D 2 > log.txt
db.get.log -S -E midnight -D 3 > log.txt
For other options, invoke the program to get a help display as follows:
db.get.log -h

To top howto page
To a2i communications home page