Short: Update to NNTP client for AS225 Author: Michael van Elst (mlelstv@serpens.rhein.de) Architecture: m68k-amigaos nntpget is a NNTP (Net News Transfer Protocol) client that gathers news articles from a remote NNTP server. It was written for AS225R2 but seems to work fine with AmiTCP and the faked socket.library done by Henning Schmiedehausen (also on Aminet). What would you do when you do not have a working NNTP daemon or do not want to run (and let it write to disk) all the time ? My solution is nntpget, a program that talks with a remote NNTP server using the 'newnews' and 'article' commands to query for new articles since a specific date and then to retrieve these articles. nntpget generates a valid batchfile suitable for 'rnews' or the like (i.e. including the #! rnews xxxx headers). Usage: > nntpget ? FROM/M,SUBSCRIPTIONS/K,ARTICLES/K,SINCE/K,DATESTAMP/K,TO/A/K,APPEND/S,NNTPSERVER/K/A,CPS/K/N,NBUFS/K,HISTORY/K,QUIET/S: FROM multiple strings denoting newsgroups or complete hierarchies like: FROM de.test alt.* news.announce SUBSCRIPTIONS file with list of newsgroups or hierarchies, one per line Characters after the first white space are ignored, so that you can use your active file. ARTICLES file with list of MsgIDs, one per line you can use all 3 sources at the same time. nntpget will first work on FROM, then on SUBSCRIPTIONS then on ARTICLES. SINCE date fetch only articles since given date in AmigaDOS format substitutes like 'monday' or 'yesterday' are valid meaning the last such weekday (at current time). So yesterday says to retrieve news from the last 24 hours. If you tell it the current day or 'today' this says to fetch all articles since this day at midnight. You can also specify a time meaning since today (or yesterday if the time would be in the future) at that time. DATESTAMP file fetch only articles newer than that file. When all goes well then the modification time of that file is updated to the _starting time_ of nntpget. This ensures that subsequent runs will not lose articles. The filesystem where the datestamp file is located must support the SetFileDate function (which is true for the standard ROM filesystem). If you do not use either the SINCE or the DATESTAMP parameter the newserver is told to retrieve articles since 1.1.1900 (i.e. since ever). Some servers may interpret this as 1.1.2000 instead and won't send anything for a while. Articles requested by the ARTICLES file are not filtered out. You will get all articles independent of age ! TO file generated batch file APPEND append to batch file NNTPSERVER the name or IP number of your NNTP server machine CPS num bandwidth limitation. nntpget tries to keep traffic below the given characters per second. This can only be an average since there are bursts of at least one TCP segment which are under control of the TCP protocol stack. The bandwidth is just limited for article transfers and not for the initial transfer of article MsgIDs but which is short. NBUFS num n-buffer NNTP protocol while fetching articles. nntpget will request up to num articles in advance. This should improve throughput if you receive lots of small articles. A good value for num is 5. A value of 1 means to fetch articles in lock-step with the server's answers (which is the default). There have been reports about problems with n-buffering and the INN (InterNetNews) daemon. Reason is that n- buffering can produce bulks of output. If the network is slow then INN erroneously assumes that the connection has been lost and terminates the session. HISTORY base If you want to update your news database from several servers or don't know what articles need to be fetched you can use this option to tell nntpget the basename of a dbz(1) compatible history database (i.e. without the .pag or .dir suffixes). nntpget will then match msgids against the history and just transfer articles that are not in your database. QUIET suppress verbose output messages. On high bandwidth connections this would degrade performance significantly. You will still see the average data transfer rate and number of articles transferred. nntpget uses 2 temporary files in T:. One to gather all MsgIDs, one for the current article (it cannot write directly to the batchfile since it doesn't know the size at that time which has to go to the #! rnews header). So please have enough space on that filesystem (usually T: is on RAM:). While running nntpget will show you (part of) the NNTP traffic on stdout as well as the average transfer rate. >> NEWS (0.95) << This version fixes some problems of the first version. When you used the CPS option and something modifies the clock (say when adjusting for DST) then nntpget measures a negative speed and would wait a very long time. Again with the CPS option it was impossible to stop nntpget while it was throttling the transfer making the previous problem even more nasty. When you abort nntpget it removes the temporary files including the article list but while it is running it did maintain an exclusive lock on the list file. Now it closes and reopens the list file so that you can examine (or copy) the list while the transfer is in progress. This helps when you want to abort and resume a transfer. Caveat: if you keep the list file open from a second process then nntpget cannot delete it. >> NEWS (0.101) << Some minor corrections to follow closer the NNTP specification. >> NEWS (0.115) << Added DATESTAMP, NBUFS and HISTORY options. Fixed two cases where a parameter was missing from an error message. The CPS and NBUFS options aren't that useful together. NBUFS is used to improve transfer rates which would be reduced again by CPS. You may still want it for maintaining a low average bandwidth usage together with a high peak bandwidth usage after a temporary network failure. >> NEWS (0.118) << Bugfixes >> NEWS (0.119) << Fixed a bug where nntpget would hang when you used n-buffering and a requested article couldn't be transferred. nntpget didn't count the failure responses and waited forever for acknowledgements after all articles were transferred. You could still abort nntpget with CTRL-C and the batchfile was intact. Added QUIET option, modified text output so that you can see the number of articles transferred and rejected. Michael van Elst