Freebees - Webpage from Tacacs+ user logs
Archiving user log files

These shell and perl files are designed to be called once a month to remove and archive all but last month's access records from the user's log file. This prevents the web page from becoming impossibly large.

The first line of the reduced log file has a special format and contains the accumulated bytes_in bytes_out and elapsed_time to date. The records removed are saved under the users ID in a temporary directory and then put into a tar file ready for permanent archiving in what ever way you choose.

The whole process is run from a shell file process-users-time-files.sh which calls a perl file process-users-time-files.pl to process each individual user's file. The shell file is designed to be run by cron anytime within the last week of the month or the first week of the month so it will suit either backing up on the 1st of the month or at the end of the month.

As with the other files presented here, this process assumes the last field in the user's usage file is the user's access type (acc_type=). Only the values in the INT (full internet) access type records are accumulated and carried forward. The Central West Web has three types of user, INT, CWE and CWW. Only the INT user pays by the hour.

Note the get_date subroutine in process-users-time-files.pl has been updated to allow for the year 2000.

File process-users-time-files.sh

This file is placed in the /usr/local/etc/system.perl directory with the following permissions
-rwx--x--- 1 root other 1086 Dec 1 12:42 process-users-time-files.sh

#!/bin/sh 
# process-users-time-files.sh
#
# Copyright(c) 1997 Forward Computing and Control Pty. Ltd.
# written by Matthew Ford
#
# This file processes all users time files and backups up the previous
# months useage.
# This leaves at least 1 months usage in their log
# The first line contains the accumlated usage from previous months
#
#

# dir of user's time logs
time_dir="/var/log/users-time-dir"

# dir of temp dir to store last months user log files before archiving
tmp_dir="/var/log/tmp_dir"

# dir to store tar file of last month's user log files in
tar_dir="/var/log"

# path and name of perl processing script
process_program="/usr/local/etc/system.perl/process-users-time-files.pl"

# clean out any files in tmp_dir
rm -r $tmp_dir/*

for userid in `ls $time_dir`
do
#   process each user's log file and put previous months access data in 
#    $tmp_dir$userid file
   $process_program $time_dir/$userid $tmp_dir/$userid
done

# tar up all the old user time logs
tar -cf - `find $tmp_dir/* -print` >$tar_dir/users-time-tar-log

# clean out any files in tmp_dir
rm -r $tmp_dir/*

# the general archive script will gzip this file and copy it to the backupdir

File process-users-time-files.pl

This file is placed in the /usr/local/etc/system.perl directory with the following permissions
-rwx--x--- 1 root other 8093 Dec 1 12:29 process-users-time-files.pl

Look here for a listing of the program (/cww/test/cgi-bin/sendmail) I use to mail error messages to the administrator (and check out the security note).

#!/usr/bin/perl -w
# process-users-time-files.pl
#
# Copyright(c) 1997 Forward Computing and Control Pty. Ltd.
# written by Matthew Ford
#
# process-users-time-files.pl
# Usage process-users-time-files.pl  path/userid  path/outputfile
#
#  Takes 2 args
#  first arg is userId time usage file including path
#  second arg is output filename (including path) for previous months usage
#
#
# overwrites userid file with new file with summary of accumulated
# time (and bytes_in, bytes_out) in first line and last month's useage removed
# 
# Output file contains last month's usage
#
# The month to archive is calculated by subtracting 6 weeks from today
# This means you can run this program in the last week or the first week
# of each month and the still leave a months usage in the user log

# This code assumes $[ is its default (0)  DO NOT CHANGE
#
# At the top of the new users useage file the accumulated values are written
# e.g after processing Oct this line was written at the top of the user's file
# Acc Nov  1 00:00:00  task_id=0000  addr=0.0.0.0  service=ppp   protocol=ip
# addr=0.0.0.0  bytes_in=6054017  bytes_out=65763877   paks_in=0
# paks_out=0  elapsed_time=137760  acc_type=INT^M

$ENV{'IFS'} = '' if $ENV{'IFS'} ;
$ENV{'SHELL'} = '/sbin/sh';
$ENV{'PATH'} = '/usr/sbin:/usr/bin:/usr/local/bin';

# name of user to get error messages by email.
$mailto = "someAdminUser\@your.domain.name"; # all mail to accounts

# path and filename of sendmail program
$mailprog = ' /cww/test/cgi-bin/sendmail ';

# path and filename of temporary file
$tmpfile = "/var/log/tmp_dir/tmp_file";  # temporary file


$webtime = 0; # start with no access time chargable
$webbytesin = 0; # no bytes sent
$webbytesout = 0; # no bytes downloaded

# Retrieve Date
&get_date;
# Calculate last month
&get_last_month;  # returns value in $last_month and $this_month
# Note that $this_month is actually $last_month + 1 month;

# convert to lower case
$last_month =~ tr/A-Z/a-z/;

if ($#ARGV != 1) { # not two args
   print "Usage: path/userid path/outputfile\n";
   die("\n");
}
$usertimefile = $ARGV[0];
$outfile = $ARGV[1];

# =======================
#  accumulate time used
# =====================

# open output file for this user
if ( !defined( open(OUT, ">$outfile")) ) {
   &mailerr("Could not open output file '$outfile'\n");
   die("\n");
}

# open temp file for this user
if ( !defined( open(TMP, ">$tmpfile")) ) {
   &mailerr("Could not open temporary output file '$tmpfile'\n");
   die("\n");
}

# open file for this user

$moreinput = 0;  # set to 1 if not EOF after removing all of last month's records

if ( defined( open(ACC, $usertimefile)) ) {
  while ($accrec = <ACC>) {
    $accrec =~ s/\n//g;   # remove all /r /n
    $accrec =~ s/\r//g;
    @accrec = split(/\t/,$accrec); # split on tabs
    $datetime = $accrec[0];

  # convert to lowercase
    $datetime =~ tr/A-Z/a-z/;
  # see if last month
    if (index($datetime,$last_month) >= 0) { # found one
        print OUT $accrec . "\r\n";  # include /r for PC use of files
     
        # find elapsedtime=  the second last item
  
       ($elapsedtime,$duration) = split(/=/,$accrec[$#accrec-1]);
       if ($elapsedtime !~ /^elapsed_time/) {
         $duration = 0;  # could not find elapsed time entry
       }
     # look for bytes_in and bytes_out files and get values
     # need to search here because TACAS+ is not consistant in the 
     # fields it puts in the log file
     ($accrec =~ /^(.*)\tbytes_in=([^\t]*)\tbytes_out=([^\t]*)\t(.*)/) 
          && ($bytesin = $2 ,  $bytesout = $3);
 
     @timearray = &calc_time($duration);
     $duration = $timearray[0];
      if ($accrec[$#accrec] eq "acc_type=INT") {
        $accesstype = "World Wide Web";
        $webtime = $webtime + $timearray[1]; # add these minutes
        $webbytesin += $bytesin;
        $webbytesout += $bytesout;
      } else {
        $accesstype = "Central West Web";
        # just ignore the access time and bytes in and out for 
        # non internet user.  Note this ignores World Wide Email users
        # but their mail useage can be picked up from the external mail log
      }
    } else {
      # finished with last month
      $moreinput = 1; # not end of input file
      last;
    }
  }
   close(OUT);

  # finished with last month
  # make new file for rest of usage log
  # first line is current accumulated usage

  if ($webtime) { # found one
     # print out accumulated total with next month's date
     print TMP "Acc $this_month  1 00:00:00 $this_year\ttask_id=0000\t";
     print TMP "addr=0.0.0.0\tservice=ppp\tprotocol=ip\taddr=0.0.0.0\t";
     print TMP "bytes_in=$webbytesin\tbytes_out=$webbytesout\t";
     print TMP "paks_in=0\tpaks_out=0\telapsed_time=". $webtime*60 ."\t";
     print TMP "acc_type=INT\r\n";
  }

  # now copy rest on input file to output
  if ($moreinput) {
     # write last line read
     print TMP $accrec . "\r\n";  # include /r for PC use of files 
     while ($accrec = <ACC>) { 
       $accrec =~ s/\n//g;   # remove all /r /n
       $accrec =~ s/\r//g;
        print TMP $accrec . "\r\n";  # include /r for PC use of files 
     }
  }
  close(TMP);
  close(ACC);
} else {
   &mailerr("Could not open input file '$usertimefile'\n");
   die("\n");
}

# finally copy tmpfile back over input usertimefile
system "cp " . $tmpfile . " " . $usertimefile;
if ($?) {
 &mailerr("Could not copy temporary file '$tmpfile' back to '$usertimefile'\n");
   die "\n";
}
 
# and remove temp file
system "rm " . $tmpfile;
if ($?) {
    &mailerr("Could not delete temporary file '$tmpfile'\n");
        die "\n";
}

exit;

#============================================================
sub calc_time {
   local($sec) = pop(@_);
   local($hrs,$mins,$minutes,$timestr,$timearray);

   # calc hours 
   $mins = ($sec / 60) - (($sec % 60) / 60);
   if ($sec % 60) { # add one 
      $mins = $mins + 1;
   }
   if ($mins < 1) {
      $mins = $mins + 1;
   }
   $minutes = $mins; # save for return below

   $hrs = ($mins / 60) - (($mins % 60)/60);
   $mins = $mins - ($hrs * 60);

   $mins =sprintf("%0.0d",$mins);
   if (length($mins) < 2) {
       $mins = " " . $mins;
   }
   $hrs = sprintf("%0.0d",$hrs);
   if (length($hrs) < 1) {
       $hrs = "0";
   }
   if (length($hrs) < 2) {
       $hrs = " " . $hrs;
   }


   $timestr = $hrs . " hrs  " . $mins . " mins";
   @timearray = ($timestr, $minutes);
   @timearray;
}

sub get_last_month {

   @months = ('Jan','Feb','Mar','Apr','May','Jun','Jul',
              'Aug','Sep','Oct','Nov','Dec');
   $time = time; #get current time
   $time = $time - 6*7*24*60*60;  # subtract 6 weeks
   # get month 6 weeks ago 
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($time);
   if ($year<90) {
      $year = "20$year";
   } else {
      $year = "19$year";
   }
   $last_month = $months[$mon];
   $mon += 1;
   if ($mon > 11)  { 
      $year = $year+1;
      $mon = 0;
   }
   $this_month = $months[$mon];  # note this assumes $[ is default 0
   $this_year = $year;
   $shortdate = $shortdate;
   $date = $date;
   $isdst = $isdst;
   $yday = $yday;
}



sub get_date {

   @days = ('Sunday','Monday','Tuesday','Wednesday','Thursday','Friday',
            'Saturday');
   @months = ('January','February','March','April','May','June','July',
              'August','September','October','November','December');

   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
   if ($hour < 10) { $hour = "0$hour"; }
   if ($min < 10) { $min = "0$min"; }
   if ($sec < 10) { $sec = "0$sec"; }

   # adjust for year 200
   if ($year<90) {
      $year = "20$year";
   } else {
      $year = "19$year";
   }

   $date = "$days[$wday], $months[$mon] $mday, $year at $hour\:$min\:$sec";
   $shortdate = "$months[$mon] $mday, $year";
   $date = $date;
   $isdst = $isdst;
   $yday = $yday;
}

sub mailerr {
  local($errmsg) = pop(@_);
  open(MAIL,"|".$mailprog) || die "Can't open mail program :$!\n";
  print MAIL "To: ". $mailto . "\n";
  print MAIL "From: webmaster\@cww.octec.org.au\n";
  print MAIL "Subject: Error Archiving User's Account";
  print MAIL " for $usertimefile\n";
  print MAIL "\n";
  print MAIL " While processing user's usage file '$usertimefile' at\n ";
  print MAIL "    " . $date . "\n";
  print MAIL $errmsg . "\n\n";
  close(MAIL);
}

Refer to Conditions of Use

Freebees - Webpage from Tacacs+ user logs Archiving user log files

File process-users-time-files.sh

File process-users-time-files.pl

Freebees - Webpage from Tacacs+ user logs
Archiving user log files