Download
slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
INN PowerPoint Presentation

INN

133 Views Download Presentation
Download Presentation

INN

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. INN InterNet News

  2. Introduction to INN (1) • Originally written by Rich Salz • ISC took over development in 1996 • http://www.isc.org/software/inn • INN is a complete Usenet system • innd – NNTP server, main program, inbound receiver • Exchange news articles with peers • innfeed: outbound, feed articles to peers • nnrpd – NNTP reader server • Separate readers and peers • For readers: readers.conf • For peers: incoming.conf、innfeed.conf、newsfeeds • A lot of components…

  3. Introduction to INN (2)

  4. Introduction to INN (3) • Main configuration • inn.conf • Components • innd – main daemon for peers (and users) • incoming.conf • innfeed – NNTP feeder • innfeed.conf、newsfeeds • nnrpd – for users • readers.conf • Newsgroups database • active、active.times、newsgroups、history • Overview database • buffindexed.conf、ovdb.conf、overview.fmt • Spool • storage.conf、cycbuff.conf

  5. Introduction to INN (4) • Specific functions • Article and history expiration • expire.ctl • Handling of control messages • control.ctl • Submission addresses for moderated groups • moderators • Cron jobs • news.daily • Summery of INN log files • innreports.conf • Others • …

  6. INN Installation

  7. Install INN (1) • In FreeBSD ports • news/inn: INN 2.4.6 • 2009-06-08 - INN 2.4.6 was made EOL by the release of 2.5.0. • 2010-Mar-25 - INN v2.5.2 , ISC is pleased to announce a new bug fixed version of INN 2.5 • How to install INN 2.5.2? • Tarball: ftp://ftp.isc.org/isc/inn/inn-2.5.2.tar.gz • Dependencies: “Requirements” in http://www.eyrie.org/~eagle/software/inn/ • C compiler gcc: built-in • Perl and MIME::Parser: lang/perl5.8+、mail/p5-MIME-Tools、graphics/p5-GD • GNU make: devel/gmake • GnuPG: security/gnupg • Berkeley DB: databases/db47+ • Yacc implementation: devel/bison

  8. Install INN (2) • http://www.eyrie.org/~eagle/software/inn/docs-2.5/checklist.html • http://www.eyrie.org/~eagle/software/inn/docs-2.5/install.html • Set up account ‘news’ • Set up the home directory for ‘news’ • # mkdir –p /home/news && chown news:news /home/news • You *CANNOT* chown -R news:newsafter installing. Some bins are SUID root. • Login as ‘news’ • su - news / sudo -u news … • rlogin –l news localhost • Enable rlogind via inetd, and set ~news/.rhosts • localhost username news:*:8:8::0:0:News Subsystem:/home/news:/bin/tcsh

  9. Install INN (3) • The source • Fetch the tarball and put it in /tmp by fetch or wget(ftp/wget) • Extract it: tar xzf inn-2.5.2.tar.gz • Configure • ./configure --help • --prefix=$HOME • --with-http-dir=$HOME/public_html • --enable-largefiles cannot be used with --enable-tagged-hash • --enable-keywords • --with-perl • --with-berkeleydb=/usr/local • You can copy some environment variables from ports/news/inn • make -n CONFIG_DONE_INN=1 /usr/ports/news/inn/work/.configure_done.inn._usr_local • CC, CFLAGS, CXXFLAGS, LDFLAGS

  10. Install INN (4) • After configure • Modify Makefile.global Please check the following files before running make, to ensure that everything was set correctly. Makefile.global include/config.h include/inn/options.h include/inn/paths.h innfeed/innfeed.h ## Berkeley DB support. If this support is configured, anything linking ## against libstorage also needs to link against DB_LDFLAGS and DB_LIBS. DB_CPPFLAGS = -I/usr/local/include/db47 DB_LDFLAGS = -L/usr/local/lib DB_LIBS = -ldb-4.7 –lz DBM_CPPFLAGS = -I/usr/local/include/db47 DBM_LIBS = -L/usr/local/lib -ldb-4.7

  11. Install INN (5) • Compile INN • $ gmake (make) • Install INN • Login as ‘root’ • # gmake (make) install • To upgrade INN • # gmake (make) update • upgrade binaries, scripts, and manual pages, but not config files • Be sure to configure INN with the same options that was used previously. • Finally • Create mail alias for usenet@ (which is defined in --with-news-master) • Set .tcshrc setenv PATH $HOME/bin:$PATH setenv MANPATH $HOME/share/man:`manpath`

  12. Install INN (6) • Set up syslog • In /etc/syslog.conf • $ cd ~/log && touch news.crit news.err news.notice • # cd /var/log && ln –s ~news/log news • # /etc/rc.d/syslogd reload • Log rotate • Done by news.daily, which is triggered by cron # uncomment these if you're running inn news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice /var/log/news/news.notice

  13. Install INN (7) • Set up cron jobs • crontab for ‘news’ • $ crontab –e • news.daily • If you're using any non-CNFS storage methods, add delayrm to the above option list for news.daily. • ~news/log/expire.rm • expirerm file • rnews • Processing spooled messages to the server again 0 3 * * * ~news/bin/news.dailyexpireoverlowmark 0 3 * * * ~news/bin/rnews -U

  14. Install INN (8) • Copy start-up script from news/inn/files/innd.sh.in • Put it in /usr/local/etc/rc.d/innd • Modify ‘%%VAR%%’s • %%PREFIX%%  ~news (/home/news、/usr/local/news、…) • %%DBDIR$$  ~news/db • %%NEWSBASE%%  ~news • # chmod 555 /usr/local/etc/rc.d/innd • Modify /etc/rc.conf • A lot of configurations remain… innd_enable=“YES”

  15. INN Databases ~news/db/

  16. active (1) • List the newsgroups carried by INN. • Maintained using ctlinnd or control messages • Edit it directly: active.times, overview database • Format • Each newsgroup is listed only once • Each line specifies one newsgroup • Four fields separated by a space: • <name> <high> <low> <status> • cs.test 0000003681 0000000285 y • name: junk、control* • high is the highest article number ever seen • low is not guaranteed to be accurate, and should only be taken to be a hint. • Normally updated nightly as part of the expire process • If the lowest article number is greater than the highest article number, then there are no articles in the newsgroup.

  17. active (2) • Format • status: • y: allow posting and feeding • m: moderated, all posting must be approved • n: allow feeding only • j: fed articles are filed in junk group • x: disallow posting and feeding • =foo.bar: articles are filed in the newsgroup foo.bar • Get a close to complete newsgroups database file • ftp://ftp.isc.org/pub/usenet/CONFIG/ • active、newsgroups • tw.bbs.* • Big5 newsgroups for tw.bbs.*

  18. active.times • Provide a chronological record of when newsgroups were created on the local server • Format • Three fields • <name> <time> <creator> • cs.test 1128713744 news@news.cs.nctu.edu.tw • time: the time it was created, expressed as Unix timestamp • date -r <time> • creator: plain text intended to describe the entity that created the newsgroup • This field should be encoded in UTF-8 • Usually the e-mail address of the creator

  19. newsgroups • Contain a list of newsgroups and its short description • It is not necessary that all the groups carried by the news server be listed in the newsgroups file • It is also not necessary that all the groups listed in the newsgroups file be carried by the news server • If you use ctlinnd newgroup to manually create a group, only the active file is updated • edit newsgroups file to add a short description for the created group • Format • Each line consists of two fields separated by at least one tab • <name>\t<description> • cs.computer-center [系計中公告] (Moderated) • Preferred format: The total line length should be at most 79 columns. The description should start with a capital and not be more than 55 characters long • using UTF-8 for non-ASCII characters in description is recommended

  20. history (1) • Keep a record of current and recently expired articles • Format • [<Hash>]\t<date>\t<token> • [E7516D3A69051BC9712D093D52A4C67B] 1274105567~-~1274105565 @0305414C543100000000000E92A20000025E@ • Each line corresponds to one article • Hash: ASCII representation of the hash of the Message-ID header • date: three sub-fields separated by a ‘~’, each contains Unix timestamp • 1st sub-field: arrival time of the article • 2nd sub-field: ‘Expires:’ header or ‘-’ • 3rd sub-field: ‘Date:’ header, record of the posting time • token: a token of the article. • This field is empty if the article has been expired • Use sm utility to retrieve the article

  21. history (2) • grephistory • Query the INN history database via Message-ID • Output the token if the article is still on the news server • By use of sm to retrieve the article • makehistory • Initialize or rebuild INN history database • Also can be used to rebuild the overview database • makehistory –FOx • Be sure to stop innd and delete or zero out the existing database

  22. INN Configurations ~news/etc/

  23. Article Storage Format • The supported storage formats • tradspool • Articles are stored as individual text files, named by the article number, and divided up into directories based on the newsgroups name • article 12345 in cs.test would be stored as cs/test/12345 • timehash • Articles are stored as individual text files, and divided into directories based on the arrival time. • timecaf • Similar to timehash, but multiple articles are put in the same file • cnfs • Articles are stored in pre-configured buffer files • Advantages and Disadvantages • http://www.eyrie.org/~eagle/software/inn/docs-2.5/install.html#S6 • INN can support all four at the same time

  24. Overview Storage Mechanism • Overview is for readers • Three overview mechanisms • tradindexed • Uses two files per newsgroup, one containing the overview data and one containing the index • Fast for readers, but slow to write to • buffindexed • Stores overview data and index information into buffers, which are preconfigured files defined in buffindexed.conf, similar to CNFS buffers • Slower for readers • ovdb • Stores overview data in a Berkeley DB database • Fast and robust, but may require more disk space

  25. Configration outline • Access controls • Peers: incoming.conf、newsfeeds、innfeed.conf • Readers: readers.conf • Storage • cycbuff.conf、storage.conf • Overview • ovdb.conf、buffindexed.conf、overview.fmt • Other controls • Expiration: expire.ctl • Moderators list: moderators • INN report: innreport.conf • Default subscriptions for a new client: subscriptions • …

  26. inn.conf (1) • The primary general configuration file for all INN programs • Most likely to need setting • organization: “College of Computer Science, NCTU” • It will be filled in the ‘Organization:’ header for posts • ovmethod: ovdb • The overview mechanism • General Settings • server: csnews2.cs.nctu.edu.tw • Default NNTP server for nnrpd, actsync, nntpget, getlist • Feed Configuration • pathalias: news.cs.nctu.edu.tw • News servers within a particular organization to add a common identity • Article Storage • extraoverviewadvertised: [ ] • Extraoverviewhidden: [ ] • See inn.conf(5)

  27. inn.conf (2) • Reading • noreader: true • For separately running innd and nnrpd • Posting • nntpauthsender: true • Add ‘Sender:’ header to local posts containing the identity assigned by readers.conf • Monitoring • doinnwatch: false • Disable the innwatch • Logging • docnfsstat: true • Start cnfsstat when innd is started • logcycles: 3 • How many old logs scanlogs keeps

  28. incoming.conf • Specify who are permitted to connect to your host and feed it articles • Connections from hosts not listed in this file will be rejected or be handed off to nnrpd and checked against the access restrictions in readers.conf • Three types of entries: key/value, peer, group streaming: true # streaming allowed by default max-connections: 8 # per feed groupcs { max-connections: 16 patterns: "*,@local*" peer CSBBS { hostname: "bbs.cs.nctu.edu.tw, 140.113.168.8" } peer FREEBSD { hostname: "freebsd.cs.nctu.edu.tw, 140.113.17.209" patterns: "!*,freebsd.*,mailing.*" } } group peering { max-connections: 16 patterns: "*,@cis.*,@cs.*,@csie.*,@local*" peer NCTUPEER { hostname: "news-peer.nctu.edu.tw, 140.113.250.119" max-connections: 8 } }

  29. newsfeeds (1) • Determine how incoming articles are redistributed to your peers • Organized as a series of feed entries • Each entry is composed of four fields separated by ‘:’ • 1st field: <name>/<exclude>,<exclude>,… • If you don't want a feed to receive articles from a certain source, then … • 2nd field: <newsgroups-pattern>/<distrbution-list> • wildmat pattern matching syntax • These are simple wildcard matches using the ‘*’ as the wildcard character • wildmat patterns can be specified in a comma-separated list • The last pattern in the line that matches the group name is used • Patterns beginning with ! mean to exclude groups matching that pattern • *,!comp.*,comp.os.* • wildmat patterns also support "poison" patterns (patterns starting with @) • “misc.*,@misc.bar” vs. “misc.*,!misc.bar” • For a article crossposted between misc.foo and misc.bar • For a article posted only to misc.bar

  30. newsfeeds (2) • 3rd field: comma-separated list of flags • http://www.eyrie.org/~eagle/software/inn/docs-2.5/newsfeeds.html#S2 • 4th field: multi-purpose parameter, meaning depends on the flags in 3rd field • Feeding peers via innfeed • Site settings ## Uncomment if you're using innfeed. ## Add "-y" as an option to innfeed to use the name of each feed as the ## name of the host to feed articles to; without "-y" an innfeed.conf ## file is needed. # innfeed funnel master. innfeed!\ :!*\ :Tc,Wnm*:/home/news/bin/innfeed-y news.uu.net/uunet\ :*,!junk,!control,!control.*/!foo\ :Tm:innfeed!

  31. newsfeeds (3) • Other examples • News overview used when “useoverchan: true” in inn.conf • CS Archive-SQLite • Mirror nctu.talk and tw.bbs.campus.nctu # News overview. overview!:*:Tc,WnteO:/news/bin/overchan archive_sqlite_cs!\ :!*,cs.*,@cs.cc.*\ :Tc,Wn:/news/bin/archive-sqlite.pl cs # mirror mirrorpost!/mirrorpost,netnews.cs.nctu.edu.tw\   :!*,nctu.talk,tw.bbs.campus.nctu\  :Tc,Wf:/news/bin/mirrorpost.pl nctu.talktw.bbs.campus.nctu

  32. innfeed.conf • Used to control the innfeed program • Three types of entries: key/value, peer, group (as incoming.conf) • peer <name> { # body} • <name> is the same as the site name in ‘newsfeeds’ • If you do not specify -y in ‘innfeed!’ entry in newsfeeds, then … • In newsfeeds • In innfeed.conf NCTUCSBBS/bbs.cs.nctu.edu.tw,BS2\ :!*,cs.*,tw.bbs.*,twbbs.*,taiwan.*,nctu.*,mailing.*, :Tm:innfeed! group BBS { max-connections: 2 streaming: false port-number: 7777 peer NCTUCSBBS { ip-name: bbs.cs.nctu.edu.tw } }

  33. readers.conf (1) • Specify access control for nnrpd • Two types of entries: parameter/value, configuration groups • <parameter>: <value> • No way to continue a line on the next line, and no way to have a value longer than about 8,180 characters • Configuration groups • auth: provides mechanisms to establish the identity of the user • access: given the user's identity, what that user is permitted to do • readers.conf is a two-step process • Put all of the auth groups first, and all of the access groups below, last match • A user identity, as established by an auth group, looks like an e-mail address "<username>@<domain>“ • When matching access groups the last matching one in the file is used to determine the user's permissions • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S4

  34. readers.conf (2) • auth group • Generate the user identity like • <UNKNOWN>@EVERYWHER • liuyh@NCTUCSCC • nnrpdauthsender • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S2 auth "EVERYWHERE" { hosts: "*" default: "<UNKNOWN>" default-domain: "EVERYWHERE" } auth "NCTUCSCC" { hosts: "bsd*.cs.nctu.edu.tw, linux*.cs.nctu.edu.tw, sun*.cs.nctu.edu.tw" default-domain: "NCTUCSCC" res: "ident -t" }

  35. readers.conf (3) • access group • newsgroupsvs. read + post • access: R(read)、P(Post)、I(IHAVE)、A(Approver)、N(NEWNEWS)、L(post to non-local posting newsgroups, [jnx] in active) • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S3 access "READING_LIMIT" { users: "*@*“ access: "R" read: "!*,cs.help,freebsd.*,gmane.*,mailing.*" } access "CS" { users: "*@NCTUCS, *@NCTUCSCC, *@NCTUCISCC, *@NCTUCSIECC“ access: "RP" newsgroups: "*,!control*,!cs.prsystem,!cs.cc.*,!csie.cc.*,!cis.sysadmin.*,!csie.sysadmin.*,!cis.computer-center.staff,!cis.syslog.*,!cis.bbs.admin.*,!junk" }

  36. cycbuff.conf • Define the cyclical buffers that make up the storage pools for CNFS • To use any sysbuff larger than 2GB: --enable-largefiles • Formats • cycbuffupdate:<interval> • refreshinterval:<interval> • cycbuff:<name>:<file>:<size> • No longer than 7 characters in <name>, and than 63 in <file> • <size> is the length in KB (2GB=2097152) • cycbuff:BIG80:/home/news/cycbuffs/BIG80:524288 cycbuff:BIG81:/home/news/cycbuffs/BIG81:524288 • dd if=/dev/zero of=/home/news/cycbuffs/BIG80 bs=1k count=524288 • metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>] • Mode: INTERLEAVE(default) or SEQUENTIAL • metacycbuff:BIG8:BIG80,BIG81

  37. storage.conf (1) • Contain the rules used to assign articles to different storage methods • Format • <methodname>: tradspool、timehash、timecaf、cnfs、trash • <storage_class>: a unique number in 0-255 to identify the storage • Used in expire.ctl, timehash, and timecaf • <options>: only for CNFS to specify the metacycbuff name method <methodname> { newsgroups: <wildmat> class: <storage_class> size: <minsize>[,<maxsize>] expires: <mintime>[,<maxtime>] options: <options> exactmatch: <bool> }

  38. storage.conf (2) • An example method tradspool { newsgroups: cs.cc.* class: 0 } method cnfs { newsgroups: cs.*, eecsep.*,nctu.* class: 1 size: 0,8191 options: NCTUCS } method cnfs { newsgroups: comp.*,humanities.*,misc.*,news.*,rec.*,sci.*,soc.*,talk.* class: 2 options: BIG8 }

  39. overview • ovdb.conf • Parameters for tuning ovdb, no need to modify • buffinedxed.conf • <index>:<filename>:<size> entries (similar to cycbuff) • <index> is unique in 0-65535, <size> is in KB • overview.fmt • No longer used by INN since v2.5 • Use extraoverviewadvertised and extraoverviewhidden in inn.conf

  40. expire.ctl (1) • Default configuration file for expire and expireover, define how long • History entries for expired or rejected articles are remembered • /remember/:<days> • Articles stored on the server are retained • <wildmat>:<flag>:<min>:<default>:<max> • if ‘groupbaseexpiry: true‘ in inn.conf (default) • Last match • <wildmat>: ! and @ not permitted, and only a single expression • <flag>: M(moderated)、U(unmoderated)、A(all)、X(delete from all groups) • <storage_class>:<min>:<default>:<max> • if ‘groupbaseexpiry: false‘ in inn.conf • <storage_class>: defined in storage.conf, ‘*’ to specify a default for all classes • <min>, <default>, <max>: decimal number of days, fraction is allowed or ‘never’ • <min>, <max>: come into play with an ‘Expire:’ header • <default>: used as the expiration period for most articles

  41. expire.ctl (2) • An example # Keep expired article history for 11 days, matching artcutoff plus one. /remember/:11 # Most articles stay for two weeks, ignoring Expires: headers. *:A:14:14:14 # Accept Expires: headers in moderated groups for up to a year and # retain moderated groups for a bit longer. *:M:1:30:365 # Keep local groups for a long time and local project groups forever. example.*:A:1:90:90 example.project.*:A:never:never:never

  42. moderators • Determine submission addresses for moderated newsgroups • Format • <pattern>:<address> • First match for <pattern> • More specific patterns should be listed before general patterns • <address>: a simple e-mail address • At most one ‘%s’ may occur and be replaced by the name of the newsgroup, with all ‘.’ in the name changed to ‘-’ ## CS moderated newsgroup cs.computer-center:newsTA@cs.nctu.edu.tw ## Public hierarchies with exceptions. ffm.*:%s@moderators.arcornews.de fj.*:%s@moderators.fj-news.org medlux.*:%s@news.medlux.ru ## Direct all other public hierarchies to the master moderator database. *:%s@moderators.isc.org

  43. innreport.conf • Configuration file for innreport • Default parameters • section default: • html true; • title "Daily Usenet report for mynews"; • footer "Local contact: usenet@mynews"; • html_css_url "innreport.css"; • A lot of options for report content and appearance

  44. Control Messages

  45. Control Messages (1) • Cancels are handled internally by INN • Others are processed by controlchan (newsfeeds) • The actions of controlchan are determined by control.ctl • ftp://ftp.isc.org/pub/usenet/CONFIG/control.ctl • newgroup、rmgroup、checkgroups • control* • control • control.cancel • control.checkgroups • control.newgroup • control.rmgroup

  46. Control Messages (2) • Authenticate control messages based on the ‘From:’ header • Obviously perilous and control messages are widely forged • Many hierarchies sign all of their control messages with PGP • controlchan knows how to do this (using pgpverify) without additional configuration • PGP key import • ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS • $ gpg --import PGPKEYS • $ ln -s ~/.gnupg ~/etc/pgp

  47. Control Messages (3) Path: csnews2!csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!ctu-peer!ctu-gate! news.nctu.edu.tw!newsfeed.berkeley.edu!ucberkeley!solaris.cc.vt.edu!news.vt.edu!guardian.oit.duke.edu!news.glorb.com!news2.glorb.com!usenet.stanford.edu!usenet-its.stanford.edu!bounce-back From: group-admin@isc.org Newsgroups: sci.physics.acoustics Subject: cmsg newgroup sci.physics.acoustics Control: newgroup sci.physics.acoustics Approved: group-admin@isc.org Date: Sat, 15 May 2010 13:00:01 -0700 Message-ID: <cmsg-20100515200001$2fac@isc.org> X-PGP-Sig: 2.6.3a Subject,Control,Message-ID,Date,From,Sender \011iQCVAwUBS+79QsJdOtO4janBAQGNsAP7BAj2Vl4LS2RoIZmYTfHc9GFg/nFve8Hj \011iOYqpW+WiF7pI6JaNzPaeS/Y3Dh7G9HRjjhuYbQ/+bTUHWeDItRFBbGVe3t+yXEv \011TbQ/NqWqMug/OKujGsOs9wy1HQKZBErDIfxCf+XD6JenRccrBPAZoCMdSFieztZe \011vFU9B4Az+ew= \011=d9AE Xref: csnews.cs.nctu.edu.tw control.newgroup:168581 For your newsgroups file: sci.physics.acoustics\011Topics in acoustics and vibrations.

  48. Control Messages (4) • control.ctl • Last match • Format • <type>:<from>:<newsgroups>:<action> • <type>: all, newgroup, rmgroup, checkgroups, … • <from>: match the ‘From:’ header • <action>: • doit、drop、log、mail、verify-pgp_userid • Encoding of newgroup and checkgroups control messages • /encoding/:*:nctu.*:big5 • /encoding/:*:tw.*:big5 • /encoding/:*:fido7.*:koi8-r • /encoding/:*:fido.*:utf-8 • /localencoding/:utf-8 • Checkgroups • /maxdocheckgroups/:*:*:10

  49. More on INN control and operations

  50. Start and Stop • Before starting, check the configurations • $ inncheck -a -v -f -pedantic -perm • /usr/local/etc/rc.d/innd • su -fm news -c "/home/news/bin/rc.news start" • su -fm news -c "/home/news/bin/rc.news stop" • If you run nnrpd separately • su -fm news -c "/home/news/bin/nnrpd –D" • su -fm news -c "killall nnrpd"