a2i-nojunk -- anti-junk-email system An anti-junk-email system called 'Nojunk(tm) filtering' is available at a2i which is intended to substantially decrease the amount of incoming junk email that will reach you. A number of different strategies are used to together provide an effective shield against unsolicited junk email. Please see the CHANGES section near the end of this document, which will describes changes occurring in the Nojunk system as we migrate to a Linux environment. USE POLICY By using, or continuing to use, the Nojunk facility, you are granting a2i communications permission to (a) examine and analyse email that is diagnosed by the Nojunk facility as probably being junk email, (b) submit such email to external services such as RSS (see http://www.mail-abuse.org/rss/ for details), and (c) save such email in our archives for future analysis or future submission to other external services. It is our policy that any such email that we submit to an external service (a) will be submitted only if a visual inspection verifies that it is indeed junk email, and (b) will be edited such that a reasonable attempt is made to hide the identify of the a2i customer for whom the junk email arrived. QUICK NOJUNK ENABLE/DISABLE INSTRUCTIONS (a) We have an easy web interface to our Nojunk facility. Point your web browser to the a2i "howto" web page at http://www.rahul.net/howto/ and follow the link to 'Nojunk(tm) anti-junk-mail filtering'. There is plenty of online help available. (b) Or use one of these commands at the UNIX shell: % nojunk 10 % nojunk 9 % nojunk 8 % nojunk 5 The higher the level the more aggressive the Nojunk filtering is. At level 10, the highest level, nearly all junk email will be blocked, but there may be some false hits, i.e., email trapped that is not junk email. Some sites that send both junk email and non-junk email will be blocked too. The number of false hits will be lower at level 9, and we expect it will be very low at level 8. If you wish to be really safe, enable Nojunk filtering at level 5. At this level we expect false hits will be extremely few (but more junk email will get through). Please read the rest of this document for suggestions about how to safely use Nojunk filtering at the highest level 10 without risk of losing any useful email. (c) To disable or minimize Nojunk filtering, use one of these commands at the UNIX shell: % nojunk 1 % nojunk 0 At level 0 no email will be trapped at all, i.e., ALL email, including all junk email, will get through to you. At level 1 only specific senders, and some domains and networks believed to be 'extremely rogue', are blocked. We expect almost no false hits, unless you are deliberately corresponding with somebody at one of these rogue sites. (d) From the UNIX shell, you may use the command % test.nojunk to test the Nojunk facility. This command will automatically send a series of test messages formatted to look like junk email. The bounces will come back to you so you may examine them and see what rejected messages look like. NOTE: To take advantage of Nojunk mail filtering you must have an account of one of these types: Premium, Advanced, Economy, Budget, or Drop-in. Nojunk filtering is not available for accounts of types Web-only. Nojunk filtering is available for POP mailboxes via the Web interface, which provides a substantial subset of functionality. NOJUNK FILTERING STRATEGIES Here is a summary of the various strategies available. Combinations of these may be used for best protection against junk email. A: A special address format that allows mail delivery only if the destination address is found in a To: or Cc: line. Otherwise the message bounces with an explanation. B: A global and/or user-specified list of search patterns. If a message header matches, the message bounces with an explanation. C: A special address format encoded based on the Subject: heading of a Usenet posting. Email replies to the posting are delivered normally if the Subject: heading is preserved. Otherwise the message bounces with an explanation. D: A special address format that expires after a specified time. See the section AGGRESSIVE NOJUNK FILTERING for some hints on how to use the highest level of Nojunk filtering while minimizing the risk of and harm from false hits. STRATEGY A Most junk email companies search Usenet and various portions of the Internet, such as web pages, to collect email addresses. Strategy A involves your providing a special email address in such places. This email address can be any of these: @nojunk.rahul.net @ether.rahul.net @network.rahul.net @spams.r.us.com @.boxmail.com @email.rahul.net <-- ultrafiltered -- see notes below where is your shell login name at a2i and is an imaginary machine name of your choice (containing alphabetic, numeric, and hyphen characters). For example, if you normally log in as joeuser, your Strategy A email address might be any of these: joeuser@nojunk.rahul.net joeuser@ether.rahul.net joeuser@network.rahul.net joeuser@email.rahul.net <-- ultrafiltered -- see notes below joeuser@spams.r.us.com joeuser@mercury.boxmail.com joeuser@joeuser.boxmail.com joeuser@spam-me.boxmail.com joeuser@no-spam-please.boxmail.com Good places to use this email address are in the From: line in Usenet postings and in web pages. When incoming mail arrives for any address of the forms described above, the Nojunk software at a2i checks any To: and Cc: lines in the incoming message. If the destination address is found in one of these, the message is delivered normally to you. If the destination address is not found in a To: or Cc: line, the Nojunk software assumes that a mailing list of some type was used, and bounces the message back to the sender with an explanation, and an invitation to resend the message directly to you. The use of a Strategy A email address in Usenet postings is likely to cut down your incoming junk email by about 50%, because junk emailers usually generate a mailing list and do not put the recipient's address on a To: or Cc: line. (However, please see the NOTE below.) If you wish, you can create a random host name, for use with the boxmail.com addresses, by using an address like joeuser@%RND%.boxmail.com in your .a2inewsfrom file. (See 'man a2i' for more information about the .a2inewsfrom file.) The string %RND% in your .a2inewsfrom file will be automatically replaced by a random integer before it is included in a Usenet posting. We don't necessarily advise you to do this, since each person sending junk email may now send you the same junk email many times, once for each distinct email address that you used. SPECIAL NOTE regarding 'email.rahul.net': In addition to the normal checking of To:/Cc: lines, addresses of the form @email.rahul.net are also ultrafiltered to block spam, by the use of a number of other aggressive filtering techniques. These should almost entirely eliminate incoming spam arriving for email.rahul.net addresses. (At the same time, the risk of legitimate mail not getting through is somewhat higher.) Techniques used for email.rahul.net include: - Strict checking of the format of mail headers. Spam generated by amateur spamware will be nearly eliminated by this. - Blocking of mail that is relayed through misconfigured machines, by consulting various black lists on the Internet. The majority of contemporary spam is relayed though such misconfigured machines, and such spam will be almost entirely eliminated due to the use of these black lists. (Also some legitimate email will be blocked.) - Strict checking of the host name of the machine transferring the mail to our machines. If the machine has no host name in DNS, no mail sent to any address in email.rahul.net will be accepted from that machine. WHERE TO USE STRATEGY A: You should publish your Strategy A email address only in noncritical places. It would be unwise to use a Strategy A email address in places where it would be used for sending email individually to you, for example, as a reply address in outgoing mail, or in .signature lines at the end of outgoing email, or on a business card. Also, it would not make sense to subscribe to a mailing list using a Strategy A email address, since all mail from the mailing list to you will bounce back. NOTE: Strategy A was very effective when it was first made available. Spammers have become much smarter since then, and their spam software sends large volumes of email with each message directly addressed to each recipient. This has made Strategy A (except for the email.rahul.net domain) much less useful. For Usenet postings you will find Strategy C much more effective. STRATEGY C Strategy C provides a very effective mechanism to prevent your reply address in Usenet postings from being used to send you junk email. It works like this: Each time you post to Usenet, a new email address is generated for you that encodes within it the first few characters of the Subject: heading of your Usenet posting. Replies to your postings that use the same subject heading will reach you normally. Email destined for your Strategy C address that uses any other subject heading will bounce back with an explanatory message. Using Strategy C from the UNIX shell: Create a .a2inewsfrom file in your home directory which contains your email address in one of these formats: @%SUBJ%.killspam.us.com @%SUBJ%.usenet.us.com @%HASH%.killspam.us.com @%HASH%.usenet.us.com c..@%SUBJ%.killspam.us.com c..@%SUBJ%.usenet.us.com c..@%HASH%.killspam.us.com c..@%HASH%.usenet.us.com where is your normal shell login name. The %SUBJ% or %HASH% portion of the above reply address will be automatically replaced, when you post to Usenet from the UNIX shell, by a Strategy C code. The encoding is slightly different for the two cases: %SUBJ% is replaced by a character string based on the subject, while %HASH% is replaced by a two-digit code. The use of %HASH% leads to shorter and more attractive-looking addresses. The use of the c.. prefix will cause your username to be encoded into an unrecognizable format. It will be decoded when replies arrive. This makes it even harder for your Strategy C address to be used for junk email. If you are using the 'tin' newsreader in the FreeBSD environment, you must also take additional steps. Please see the 'INSTRUCTIONS FOR NEWSREADING SOFTWARE' section. Examples of Strategy C addresses for a user 'joeuser': joeuser@%SUBJ%.killspam.us.com joeuser@%SUBJ%.usenet.us.com joeuser@%HASH%.killspam.us.com joeuser@%HASH%.usenet.us.com c..joeuser@%SUBJ%.killspam.us.com c..joeuser@%SUBJ%.usenet.us.com c..joeuser@%HASH%.killspam.us.com c..joeuser@%HASH%.usenet.us.com Using Strategy C from a SLIP/PPP connection: You must configure your News software to use one of the following formats for the address in your From: line in News postings: @xx.usenet.us.com %{}@xx.usenet.us.com where is your normal shell login name. For example, user joeuser would use one of these formats: joeuser@xx.usenet.us.com %{joeuser}@xx.usenet.us.com Now when you post News via the nntp servers at a2i. your From: address will be automatically transformed into a Strategy C address, usable only for replies to the specific posting provided the Subject is preserved. The first address format leaves your username unchanged and replaces the xx part with a code. The second one encodes your username too, making your email address quite unrecognizable. The same caveats apply to the use of Strategy C email addresses as for Strategy A. CANCELING USENET POSTINGS The use of various customized reply addresses may make it harder for you to cancel a Usenet posting you have made. We recommend that you never post anything to Usenet that you might want to take back later. Even if you successfully send a cancellation message, there is no guarantee that the posting will really be canceled everywhere. Many Usenet sites ignore and discard cancellation messages. If you really do want to cancel a posting, and are having trouble, please look at the section 'CANCELING POSTINGS WITH CUSTOMIZED FROM LINES' in the online manual 'man a2i-newsgroups'. STRATEGY D This allows you to use an email address format that expires at a certain time. The address format is: @.rahul.net where is your shell login name, is a three-letter abbreviation for a month, and is the last two digits of a year. (For 1999 use 99, for the year 2000 use 00< and for 2001 use 01.) After Decembr 1999, mail to dec99.rahul.net will bounce with an explanation. After January 2000 is over, mail to jan00.rahul.net will bounce with an explanation. And so on. For example, a user called 'joeuser' could use addresses like these: joeuser@may00.rahul.net -- valid through end of May 2000 joeuser@dec99.rahul.net -- valid through end of Dec 1999 Strategy D lets you use an email address that will exist for a while, but not forever. This minimizes the possibility of your email address being archived in places and sticking around indefinitely. Instead, just decide how long you want your email address to be valid for a given purpose. For example, if you need to supply your email address when registering on somebody's web form, and you want them to be able to send replies in the near future but not benefit from selling it to junk emailers, use an address that expires in about a month. To see what sort of error message will be seen after the address has expired, send email to yourself @dec96.rahul.net (which is an expired address). Strategy D addresses up to 24 months into the future may be used. If you use an address that is more than 24 months into the future, it's not guaranteed to be valid. (But test it and see -- if it's valid now, it will remain valid.) STRATEGY B Strategy B does not require you to use any special email address format. It relies on pattern matching to detect junk email. 1. Incoming mail headers (and optionally body) are checked against a list of search patterns. If a match is found, the incoming mail is rejected and bounces back. The bounced message includes a brief explanation, which includes a copy of the header line that was matched by one of the search patterns, so the sender has some idea of why the message bounced. Also, the bounced message tells the sender of a short passcode that he can include in the Subject: heading to get through our Nojunk facility. Thus, if an innocent sender's message was blocked by a false match, the sender can still resend including the passcode and will get through. A global list of patterns is maintained by a2i management. These are search patterns that we expect will usually match headers of unsolicited junk email, based solely on our subjective opinion. There is no guarantee that these patterns will only match unsolicited junk email, and there is no guarantee that they will match all unsolicited junk email. You may also maintain your own patterns. These may be used either instead of, or in addition to, the global list of patterns. The global list of patterns is kept in the file /local/scripts/nojunk.patterns See below for an explanation of the syntax of the contents of this file. From the UNIX shell, the command 'nojunk.stats' will print some information about the number and types of search patterns in this file. 2. From time to time a2i management will revise the global list. We may sometimes announce such changes in the local newsgroups. In most cases we will simply make such changes quietly. You may, of course, examine said file of patterns at any time to see what is in it. You may also, following instructions below, choose to maintain your own set of Nojunk patterns. 3. Strategy B Nojunk filtering is designed to be active only if 'nojunk' filtering is active for your account at a level higher than 0. Newer accounts have Nojunk filtering enabled by default at level 9. Older accounts, created before Nojunk filtering was introduced, now have Nojunk filtering enabled at level 1. In any case, you may adjust your nojunk filtering level with the 'nojunk' command, following the steps below. 4. If you are not using a .forward file to forward email, skip ahead to item 5. If you are using a .forward file, Nojunk filtering will be effective only if your .forward file causes mail delivery into your mailbox. If you are using a .forward file to process mail via a program such as procmail, you will need to take the following steps to enable Nojunk filtering. a. Don't invoke the external program from your .forward file. b. Instead, include a 'pipe' command in your .nojunk.patterns file that invokes the external program. (See section 9 below.) This tells the Nojunk facility that, after it has finished its work, it should invoke the program specified in the 'pipe' command to do additional filtering of your incoming email. 5. To activate or deactivate Strategy B Nojunk filtering, login into the UNIX shell and use the command 'nojunk' as follows: % nojunk n where n is an integer between 0 and 10, which specifies the level of Nojunk filtering. At level 0 no filtering is done, i.e., all email, including all junk email, will be allowed to get through to you. At level 10 Nojunk filtering is done with extreme prejudice, and there may be significant false hits. For more information about what the levels mean, please use the command 'nojunk -h' to get a help screen, and also look at the section 'QUICK NOJUNK ENABLE/DISABLE INSTRUCTIONS' near the beginning of this document. The 'nojunk' command affects only Strategy B Nojunk filtering. It does not affect Strategies A, C, and D. For upward compatibility the old command 'nojunk y' is interpreted to mean 'nojunk 9', and 'nojunk n' is interpreted to mean 'nojunk 0'. 6. WARNING WARNING WARNING: If you enable Strategy B Nojunk filtering and don't substitute your own patterns as described below, you are agreeing to accept the judgment of a2i communications about which mail should be filtered out, and you agree that we cannot guarantee that only unwanted mail will be filtered out. If you have any doubts, either enable Nojunk filtering at a low level (5 or lower), or set up your own patterns, or both. 7. How to test: Once Strategy B Nojunk filtering is enabled, incoming mail with the string ::nojunk:: in the subject heading will bounce back. However, the Nojunk facility does not block mail sent by you to yourself. So you will need to ask a friend to send you email with the string ::nojunk:: in the subject. The message should bounce back to that person with an 'Insufficient permission' error. You can also use the 'test.nojunk' command to test the general working of the Nojunk facility. This causes junk messages to be created and mailed to both you and a test user nojunk@rahul.net. You will see some bounced messages come back to you. 8. Log file: If a file called 'nojunk.log' exists in your home directory, some lines will be added to it recording selected headers from mail that is filtered out by Strategy B Nojunk filtering. If this log file does not already exist in your home directory, it won't be automatically created. Normally only selected message headers are logged. If you have a .nojunk.patterns file (see below) and it contains a line logheaders then the entire message headers will be logged. The line biglog in your .nojunk.patterns file will cause any nojunk.log file to grow indefinitely. Otherwise it will be automatically pruned when it grows to around 10,000 bytes. The line logbody will cause a complete copy of each rejected message to be saved in a file 'nojunk.messages' in your home directory. If this file does not exist, it will be created. This file will be in mailbox format, so you read its contents with any mail program. Optionally you may specify an alternative name for this log file: logbody logs/junk.email will save the complete copy of each rejected message in the file 'logs/junk.email' relative to your home directory. There is also a system log, accessible only to a2i support staff, in which we log some limited information about rejected messages for software debugging: Your username, date, sender of the rejected message, subject, and the pattern match or reason why the message was rejected. The line fakereject causes log entries to be created in nojunk.log and nojunk.messages exactly as if a message had been rejected, but no message is rejected due to Strategy B. If you are curious to know how much mail would be rejected at a certain Nojunk level, you may enable the fakereject feature for a while. This is especially useful if you want to find out whether enabling Nojunk filtering at level 10 will cause any significant number of false hits. Just enable the fakereject feature and also do 'nojunk 10'. Then watch your log files for a week or two. If you see only junk email logged, then you can have some confidence that Nojunk filtering at level 10 will not cause inconvenience to people who send you non-junk email. Note: The fakereject feature does not affect Strategies A, C, or D. A line of the form bodyscore n where n is a numeric (integer) value, tells the Nojunk facility to reject a message, based on searching strings in the body, if the score is n or higher. For example, the line bodyscore 20 means that messages with that score 20 or higher should be rejected. If no bodyscore line is found, the Nojunk facility rejects messages based on the following values: your Nojunk level rejection threshold ----------------- ------------------- 8 24 9 14 10 12 9. Maintaining your own Nojunk patterns: If a file called .nojunk.patterns exists in your home directory, patterns from it (instead of the global list) will be used to filter your incoming mail. If you want both the patterns in your .nojunk.patterns file as well as the patterns in the global list to be used, include a line in your .nojunk.patterns file like this: global If this is found in your .nojunk.patterns file, then first the patterns in your .nojunk.patterns file, then the patterns in the global patterns file, will be used to filter your mail. Any patterns in your .nojunk.patterns file will be effective in blocking incoming mail even if the sender includes the passcode (see section 1) in the Subject: heading. INVOKING AN EXTERNAL PROGRAM If you wish to have Strategy B Nojunk filtering active and also process mail through an external program, include a command like this in your .nojunk.patterns file: pipe = The 'pipe' command shown above may be anywhere in your .nojunk.patterns file. The specified command will be invoked after Nojunk filtering has been completed. If Nojunk filtering causes an incoming message to bounce, then the message will never reach the external program. If multiple pipe commands exist in your .nojunk.patterns file, only the last one will be effective and all others will be ignored. See section 11 below for more information about how to invoke the procmail program from your .nojunk.patterns file. FORWARDING TO A REMOTE ADDRESS You can do Nojunk filtering and forward accepted messages to one or more addresses by using one or more 'forward' lines. The syntax of a forward line is: forward =
or forward =
... If you use a forward line, then any pipe line will be ignored. Thus you may either invoke an external program, or forward your mail, but not both. If either a forward or a pipe line is found, then mail is not delivered locally into your mailbox. However, you may invoke procmail and from within procmail do any number of things, including forwarding mail to other addresses, invoking other programs, and delivering mail locally. Hints are below. SAVING A LOCAL COPY OF MAIL If you use the 'forward=' or 'pipe=' syntax, normal mail delivery into your local mailbox is not done (unlesss the program to which you are piping mail does it). To also have normal mail delivery done into your mailbox, use the line ccme in your .nojunk.patterns file. Example .nojunk.patterns files: # example 1: forward to a remote address, but also deliver # a copy into our local mailbox ccme forward = me@remote.domain.name # example 2: pipe through a program, but also deliver # a copy into our local mailbox ccme pipe = bin/my.filter.program If the program through which you are piping mail also delivers a copy into your mailbox (as is the default action of procmail), then the use of the 'ccme' line will result in your getting two copies of incoming messages. SYNTAX OF NOJUNK PATTERNS In your .nojunk.patterns file: Blank lines are ignored. Lines beginning with # are ignored. All other lines should have one of the following formats. If any specified pattern matches any header line in a message, the message is rejected. // Pattern is perl pattern. E.g.: /^From: .*\@roguesite\.com/ sh:// Pattern is shell wildcard pattern, with * and ? recognized, and ^ recognized only as first char to force a match at beginning of line. E.g.: sh:/^From: *@roguesite.com/ sh:/^Received: *smtp?.roguesite.com/ f:// Pattern is a fixed string with no wildcards. E.g.: f:/::nojunk::/ site:// Pattern is a site name or domain name. It will be converted to patterns matching From: and Received: lines, so mail will be excluded if it originates from or passes through that domain. E.g.: site:/roguesite.com/ from:// Pattern is an email address. It will be used to match From: lines. A line ending with backslash (\) is joined with the next line to form a single line, the backslash and any whitespace surrounding it being replaced by a single blank. Matches are case-insensitive, but any of the above may have the closing slash immediately be followed by 'c', which causes the match to become case-sensitive. For example, the line from:/joe@example.COM/c would exclude mail from joe@example.COM but would still allow mail from JOE@EXAMPLE.COM. If the closing slash of a pattern is followed by 's', a match on that pattern causes the incoming message to be discarded without bouncing. E.g. the pattern from:/joe@example.com/s would cause mail coming from joe@example.com to be silently discarded, without bouncing back to the sender. If the closing slash of a pattern is followed by 'b', any match causes the message to be accepted rather than rejected. Such a pattern is called a "bless" pattern because the match is said to bless the message. For example, suppose we have two patterns: site:/roguedomain.com/ from:/goodguy@roguedomain.com/b This will cause mail from goodguy@roguedomain.com to be accepted, even though all other mail from that domain is blocked. NOTE: Bless patterns affect only Strategy B Nojunk filtering. They will not cause mail to be accepted that would be blocked by Strategies A, C, or D. To minimize processing overhead, only up to 100 bless statements are accepted (any more will be ignored). You can, however, include multiple matches in a single bless statement. For example, to bless a number of different senders in one bless line, you can use a perl-syntax pattern. Be sure to precede special characters such as @ and . with a backslash so they will be matched directly and not treated in a special way. The following pattern will cause mail from any of the senders joeuser@example.com, sueuser@elsewhere.com, and me@whoami.org, to be accepted even if some other pattern match would have rejected it: /^From:.*(joeuser\@example\.com|sueuser\@elsewhere\.com|me\@whoami\.org)/b You could of course have listed these separately, provided you had no more than thirty bless patterns: from:/joeuser@example.com/b from:/sueuser@elsewhere.com/b from:/me@whoami.org/b Bless patterns need not be in any particular order in your .nojunk.patterns file. Even if they are interspersed with other patterns they will have the same effect. Meaningful combinations of 'c', 's', and 'b' may be used with the same pattern, e.g; from:/joe@example.com/cs from:/joe@example.com/sc from:/joe@example.com/bc The statement silentdrop makes all matches for all patterns cause the incoming message to be silently discarded without bouncing back. The statement nobody disables checking the message body for suspicious strings that are often found in junk email. These checks are normally done if your Nojunk filtering level (set by the 'nojunk' command, see above) is at 8 or higher. The higher the level, the more aggressive the Nojunk facility will be in rejecting mail based on suspicious strings in the body. If you encounter any significant false hits, the 'nobody' statement will disable checking message bodies. Statements such as level 1 level 5 level 9 set the Nojunk level of subsequent patterns in the file. If you have use the 'nojunk' command to set Nojunk filtering at a certain level, only those patterns will be effective in your .nojunk.patterns file that have a declared level of the same as, or less, than, your Nojunk level. If there is no 'level' command in your .nojunk.patterns file a default level of 1 is assumed. For example, suppose you have enabled Nojunk with the command 'nojunk 8' from the UNIX shell. This sets your Nojunk level to 8. Then if you have some lines in your .nojunk.patterns file preceded by a line 'level 9', these lines will be ignored, because the level of these lines (9) is higher than your Nojunk level (8). But lines in your .nojunk.patterns file that are preceded by a line 'level 1' will be effective, because the level of these lines (1) is lower than your Nojunk level (8). If you create your own .nojunk.patterns file, be sure to test by logging into bolero.rahul.net and sending mail to yourself with the command 'mailx -v '. This will verbosely cause mail delivery, and any syntax errors in your patterns will become obvious. NOTE: If there are any syntax errors in your .nojunk.patterns file, they may affect the correct processing of other parts of that file. In particular, if any bless line is incorrectly formatted, all bless patterns may end up being ignored. To minimize problems, whenever you use a perl pattern, be sure to escape all non-alphanumeric characters with a backslash. See the section on quoting of metacharacters that follows below QUOTING OF METACHARACTERS Metacharacters are characters that might have some special meaning. In perl patterns most non-alphanumeric characters are metacharacters. Whenever a character might have special meaning for perl, if you want to suppress this special meaning, precede the character with a backslash. Such quoting is done and required only in perl patterns. Examples: /joeuser@example\.com/b <- perl pattern, incorrect /joeuser\@example\.com/b <- correct, @ and . must be quoted sh:/joeuser\@example\.com/b <- shell pattern, incorrect sh:/joeuser@example.com/b <- correct, @ and . must not be quoted from:/joeuser\@example\.com/b <- sender address, incorrect from:/joeuser@example.com/b <- correct, @ and . must not be quoted 10. Using procmail on your own. If you wish, you can use procmail to do your own mail filtering, instead of using a2i's Strategy B Nojunk facility. For hints, take a look at this URL: http://www.panix.com/e-spam.html At the above URL you will find some hints about how to use procmail for filtering out junk email. Although the hints are for users local to that site, you should be able to adapt them for your own use. 11. GENERAL PROCMAIL HINTS: Here are some simple procmail hints that will be useful if you were previously using a .forward file. You can't do Nojunk filtering and also forward mail via a .forward file, but you can achieve equivalent results by using procmail recipes. a. Include the following line somewhere in your .nojunk.patterns file: pipe = /etc/LOCAL/bin/procmail3.11p4 -Yf- This tells the Nojunk facility that, after it has finished its work, it should feed the mail message (if it was accepted) to procmail. You will also create a file called .procmailrc in your home directory that contains the procmail commands you need, as described below. b. To forward all your incoming mail (after Nojunk has done its work) to some other email address, include the following lines in your .procmailrc file: :0 !
where
is the actual email address to which mail should be forwarded. For example, to forward all mail to joeuser@aol.com you would use: :0 !joeuser@aol.com You can also forward to multiple addresses on the same line, by giving a comma-separated list: :0 !joeuser@aol.com,sueuser@example.com,anne.onymous@elsewhere.net The above procmail lines will cause mail to be forwarded to the specified addresses without being locally saved. So your incoming mailbox at a2i will not receive a copy of incoming messages. c. To forward your incoming mail and also save a copy in your mailbox at a2i, use the 'c' suffix like this: :0c !joeuser@aol.com or like this: :0c !joeuser@aol.com,sueuser@example.com,anne.onymous@elsewhere.net IMPORTANT: The above lines should work correctly assuming your .procmailrc file contains only the example lines shown above and nothing else. If you have other things in your .procmailrc file, they will affect the way procmail does mail processing and may cause the effect of the lines shown above to be different than described here. You should in that case be sure to read the procmail documentation ('man procmail') and understand what you are doing. 12. AGGRESSIVE NOJUNK FILTERING At level 10 Nojunk filtering (set by the command 'nojunk 10') most junk email will be quite effectively killed, but false hits are also possible. To take advantage of level 10 Nojunk filtering, while minimizing the harm due to false hits, try Method A, B, or C below. Once you know what sort of false hits you are seeing, you can use bless patterns to accept email from specific senders. This should bring down the rate of false hits to very low or zero. (Note: Only Strategy B is affected by Nojunk level settings and by bless patterns.) Method A - Enable level 10 with the UNIX command 'nojunk 10'. - Create an empty log file with the command 'touch nojunk.log'. - Create a .nojunk.patterns file and in it put these lines: global fakereject - The 'fakereject' line above will cause the Nojunk facility to log all matches that would cause a message to be rejected, BUT no message will actually be rejected. Watch the logs for a couple of weeks. This will give you a good idea of how effective level 10 Nojunk filtering is, and how many false hits you are likely to get. If you are satisfied with the result, delete the 'fakereject' line from your .nojunk.patterns file. This will actually cause messages to be rejected. Note that rejected messages always cause the sender to get a bounced message that includes instructions for bypassing the Nojunk facility, so no sender who is sufficiently interested in sending email to you will be prevented from doing so. Method B - Enable level 10 with the UNIX command 'nojunk 10'. - Create an empty log file with the command 'touch nojunk.log'. - Create a .nojunk.patterns file and in it put these lines: global logbody - The 'logbody' line above will cause the Nojunk facility to save a complete copy of each rejected message in a file nojunk.messages in your home directory. This file will be in mailbox format, so you can read messages in it with any UNIX mail program. For example, you can use elm as follows: 'elm -f nojunk.messages'. Once a day, check the rejected messages this way. This way, any false hits still leave the message available to you, but your normal routine of reading email is not disrupted by junk email at unexpected times. Also, if you use the command 'silentdrop' in your .nojunk.patterns file, rejected messages will not bounce back to the sender. Method C This method is suitable for procmail-knowledgeable users. It works only for those account types that allow incoming mail to be fed to a program. - Enable level 10 with the UNIX command 'nojunk 10'. - Create an empty log file with the command 'touch nojunk.log'. - Create a .nojunk.patterns file and in it put these lines: global fakereject bodyscore 12 pipe = /etc/LOCAL/bin/procmail3.11p4 -Yf- (In place of bodyscore 12 you may use a higher or lower value, depending on how aggressive you wish to be. Lower values will cause more false hits.) - This will cause all mail to be piped to the procmail program. Due to the 'fakereject' keyword no messages will actually be rejected. However, each message as seen by procmail will include an 'X-Nojunk-Status' header, which is described below in section 15. - Now you may set up procmail filter lines as you wish, to route junk mail to separate folders under control of procmail, or selectively discard some junk messages while saving others. Sample (very simple) .procmailrc file for Method C: # rejected senders go into folder 'badguys' :0: * ^X-Nojunk-Status: RH Mail/badguys # all other rejects go into folder 'scams' :0: * ^X-Nojunk-Status: (RS|RB|RT) Mail/scams 13. IMPLEMENTATION NOTES We sometimes get inquires from people about how Nojunk is implemented here and how they might do something similar. Here is a brief description. Strategy A: Selected domains are set up with wildcard MX records, and all mail for those domains is initially delivered to a special user called 'nojunk'. The local mailer recognizes the 'nojunk' user, retrieves the intended recipient from the topmost Received: line in the message, checks to see whether or not the intended recipient is in a To: or Cc: line, and then either rejects it or recursively invokes sendmail to deliver it. The recursive invocation allows aliases to be resolved. For example, suppose mail arrives for joeuser@nojunk.rahul.net. This causes /bin/mail to be invoked with a recipient user being 'nojunk' but with joeuser@nojunk.rahul.net in the topmost Received: line. After the message is accepted, sendmail is invoked to send it to joeuser@rahul.net. If joeuser is an alias it will be resolved by sendmail and then the message will be delivered normally. Strategy B: Implemented as (a) a Nojunk server process that does pattern matching and (b) a replacement local mail delivery agent /bin/mail, both written using perl. When the Nojunk server starts it reads a database of various search patterns and compiles perl subroutines for pattern matching. The Nojunk server then goes into a loop accepting TCP/IP connections from the local mail delivery agent, reading message headers and bodies for analysis, and sending back a result classifying the message as junk email or non-junk email. For reliability and load balancing, multiple servers are used, and the local mailer will connect to any available server. The Nojunk server creates an in-memory hash table of all specific sender addresses that are believed to have been used in junk email, and can thus do look-ups very quickly and efficiently. When we see a steady stream of junk email from a specific sender at a site that is already blocked, we also add that specific sender to the Nojunk list. Then at runtime that sender will be quickly detected via a hash table look-up and the message will be rejected without further processing. Strategy C: For posting from the UNIX shell, we have a global 'inews' script written in perl through which all Newsreaders post. This script inserts an encoded address in the user's From line if needed, based on the first few characters of the Subject line. For posting via nntp, a revised nnrp server does the same thing. When incoming mail is received for a Strategy C address, it is handled by the same code that handles Strategy A addresses, via wildcard MX records and the 'nojunk' user. If the encoded user and Subject are together successfully decoded, sendmail is recursively invoked to deliver the message to the final user. Strategy D: Each address corresponds to a host name in the sendmail mailertable and sendmail.cf files. Expired host names resolve to an error indication. 14. CHECKING INDIVIDUAL MESSAGES MANUALLY We provide a command 'check.nojunk' that will let you manually check a message to see what Nojunk filtering would do with it. Save the message in a file, then feed it to check.nojunk, specifying the desired Nojunk level: % check.nojunk 8 < Message For more information invoke as 'check.nojunk -h' for help. 15. X-Nojunk-Status HEADER Each message processed by the Nojunk facility is delivered with a header line added to it, in this form: X-Nojunk-Status: where is a two-character code that tells you how the message was classified, and is a numeric value that provides more information. Here is a summary of all the possible values of . code meaning ---- ------- OK Message accepted. No junk mail detected. If your Nojunk level is 0, then all messages will include exactly this code. RB Reject body -- suspicious strings were found in the body of the message. The value of tells you how suspicious the strings were. The higher the score, the more likely it is that the message is junk email. Usually scores of 0-4 indicate that the message is not junk email, while scores higher than 12 indicate that it is probably junk email. The RB code appears only if the score based on suspicious strings exceeds a certain threshold, which you may set yourself with the 'bodyscore' command in your .nojunk.patterns file. For example, if you included 'bodyscore 20' in your .nojunk.patterns file, then the code RB will be seen only if a message scores 20 or higher. RH Reject header -- a header line matched one of the blocking patterns listed in either the global nojunk.patterns file or your private .nojunk.patterns file. RS Reject subject -- mismatch between encoded address and subject line, based on Strategy C. RT Reject because recipient was not found in a To: or Cc: line, based on Strategy A. The is currently always 0 for code OK, and is always 1 for codes RH, RS, and RT. 16. IF YOUR MAIL IS FORWARDED THROUGH SOME OTHER ACCOUNT Suppose you have an account at another ISP, and its email address is joeuser@otherisp.com. Most of your incoming mail is sent to that address, and then forwarded to your account at a2i. This has some consequences that you should know about. a. In some cases we block some spam sites from connecting to our machines. This can block much junk email. These sites will still be able to connect to your other ISP, and will be able to send junk email to your address there. The junk mail will be forwarded to your account at a2i, and might not be filtered out by the Nojunk system. So Nojunk filtering will not be as effective as it would have been had your mail come directly to your account at a2i. b. If you are being sent a large amount of junk email, and the junk email is trapped by our software, the junk email will appear to have been relayed through the hosts at your other ISP. This can cause us to accidentally diagnose your other ISP as the source of the junk email, since all the junk email will be delivered to our machines when your other ISP's machines connect to ours. As a result we might block your other ISP's machines from sending mail here, even though they are not at fault. To prevent this problem, we suggest that you create a file in your home directory called a2i.README and in it put a comment identifying where your mail is being forwarded from. E.g., your a2i.README file might say: Mail for joeuser@rahul.net is forwarded here from joeuser@otherisp.com. Should we observe a large volume of junk email for your account from a single source, we can then check for an a2i.README file in your home directory, and look to see if your mail is being forwarded from that source. 17. COMPLAINTS Before sending support@rahul.net any complaints about non-junk email being blocked, please consider the following questions carefully. a. Is your Nojunk level set at 9 or 10? At these high levels, the filtering is very aggressive, and significant amounts of non-junk email will also be filtered out. This should not prevent somebody who really wants to reach you from reaching you anyway, since the error bounce that he gets will tell him how to bypass the filtering. If this is not acceptable, you should lower your Nojunk level to 8 or less. Or you may use one of the safe strategies described above, which involve saving all junk email in a separate folder and not bouncing any of it. b. Is your Nojunk level 8 or lower? In that case, if you still notice any non-junk email being bounced, please see if this is caused by a domain in our Nojunk list that originates large amounts of junk email. If a certain source of mail sends almost all junk email, we may choose to block it, even if a tiny fraction of mail from that source is wanted by you. Your choices are usually to either turn off all Nojunk filtering, or add "bless" entries to allow mail from that domain. After considering these questions, if you still believe that we are unfairly blocking a source of email, please let us know. 18. INSTRUCTIONS FOR NEWSREADING SOFTWARE If you are using the 'tin' news reader from the FreeBSD environment, then the following special steps are needed to use Strategy C. a. Invoke tin, then type M to invoke the configuration menu. Hit the space bar to move to near the end of the configuration pages. Set the option 'Use builtin inews' to OFF. Exit tin. b. Go into the subdirectory .tin in your home directory, and use your favorite text editor to edit the file called 'headers'. To this file append the following line (without any leading blanks): From: FILLIN@rahul.net Now you are ready to use tin. It will use a From: line in postings as shown above. This From: line will be replaced, when you post something, by the contents of your .a2inewsfrom line with appropriate modifications as described above for Strategy C. 19. CHANGES -- last revised Sat Nov 2 02:10:33 PST 2002 As we migrate to a Linux environment some changes are taking place. These will be listed here as they become applicable. FROM LINE IN USENET POSTINGS: The updated newsreaders in the Linux environment need special handling to let you set your From: line in Usenet News postings. Please see instructions in the following document: /local/setup/linux/newsreaders.txt EXPLICIT NOJUNK STRATEGY B PROCESSING IN A LINUX ENVIRONMENT. You can force any mail to be processed via Strategy B in a Linux environment by forwarding it to USER@nj.rahul.net where USER is your username. This feature is for power users only. You are a power user if you understand these instructions enough to follow them without losing mail. This feature is made available for testing purposes, to let you see if your .nojunk.patterns file will be correctly processed in a Linux environment. You should not blindly forward all mail to USER@nj.rahul.net as this might result in an infinite loop. However you can forward mail to USER@nj.rahul.net as follows, from a .procmailrc file: :0 * !^X-SMTP-To: USER@nj\.rahul\.net !USER@nj.rahul.net The above lines will forward mail to USER@nj.rahul.net only if it has not already been processed on nj.rahul.net. Once mail reaches nj.rahul.net, your normal Nojunk level will not be checked. Instead, your Nojunk level will be taken from a file .nojunklevel in your home directory. You should therefore set a Nojunk level for use in the Linux environment by manually creating the file .nojunklevel in your home directory. It should contain exactly one line of text with a number between 0 and 10 and nothing else. To avoid Nojunk processing occurring twice, be sure to set your normal Nojunk level (in the SunOS environment) to 0. Here is a summary of the steps you should take: 1. Set your normal (SunOS) Nojunk level to 0. 2. Set your Linux Nojunk level to some value by creating a .nojunklevel file. 3. Add the forwarding lines into your .procmailrc file to cause mail to be forwarded to USER@nj.rahul.net. 4. Cause procmail to run in a SunOS or FreeBSD environmant by adding a line in your .forward or .forward.bsd file that looks like this: |"/etc/LOCAL/bin/procmail3.11p4 -Yf- #USER" This will cause procmail to initially run under SunOS or FreeBSD, forward your mail to Linux, and cause Nojunk processing and final delivery there. The path taken by mail will be: mail on SunOS or FreeBSD -> procmail on SunOS or FreeBSD -> USER@nj.rahul.net on mauve.rahul.net (Linux) -> Nojunk Strategy B processing based on level in .nojunklevel -> processing via 'pipe = ' or procmail delivery Please note that this should be done for testing purposes only, as the Nojunk environment under Linux is still being tested and is not yet considered to be of production quality. NOJUNK DOMAINS MIGRATED TO LINUX: The Nojunk domains mentioned in this document are being moved to be processed in a Linux environment. The schedule is as follows. Moved night of Friday, October 25: email.rahul.net network.rahul.net ether.rahul.net Moved night of Monday, October 28: nojunk.rahul.net spams.r.us.com *.boxmail.com *.killspam.us.com *.usenet.us.com When mail is processed in a Linux environment for any of the above domains, any 'pipe = ' line in your .nojunk.patterns file will be active. (The 'pipe = ' line was not active for these domains in the past.) Also, all mail processed in the Linux environment is always processed by procmail if you have any .procmailrc file in your home directory, even if no 'pipe = ' line is used to enable procmail processing. EARLY REJECTION OF STRATEGY C Mail arriving for Strategy C domains is rejected during SMTP if the Subject heading of the incoming message does not match the encoded sender address. The machine that tried to send us the spam will still bounce it back to the sender with an error message, and the error message will give the sender an email address at which you can be directly reached. Since the message is never saved to disk, and never enters our mail queues, it will also not be logged in any Nojunk log in your home directory. The purpose of this early rejection is to discard large volumes of spam quickly and efficiently without incurring any significant processing overhead and without clogging internal mail queues with bounces destined for forged sender addresses. Mail that is rejected during the SMTP transaction is never saved in a mail queue. Rejections are logged and may be viewed via the web at: http://www.rahul.net/spamlogs/ In the future we will also implement early rejection of Strategy A mail, if it does not include the recipient in a To: or Cc: header. == END ==