Scripty the Kid's: Talking With Daemons

Centralized Syslog With FreeBSD

2008-08-29T22:08:00.005-05:00

It is impossible to manage log files across a large multitude of servers. To assist with this syslog is able to send all messages to a single server. This makes keeping track and filtering much easier. It also reduces the time it takes to check the logs as only one login is needed. In this article only the use of the default syslog server which comes with FreeBSD will be covered. There are other ways of centralizing, viewing, and beautifying logs such as with syslog-ng and php-syslog-ng which will not be covered here.

In this example we'll use a fake environment with only the central syslog server and a single workstation for simplicity. Our small networks syslog server will be fbsd-syslog with an ip of 192.168.0.10 and the workstation will be called fbsd-wrk with an ip of 192.168.0.11.

First we must tell the central server to listen to fbsd-wrk. To do this we must append to /etc/syslog.conf . It is always a good idea to back up the original file to something like /etc/syslog.conf.bak in case things go wrong. Add the following to the end of /etc/syslog.conf on the central server (fbsd-syslog).

!*
+192.168.0.11
*.* /var/log/messages

This is simplicity at it's finest. It allows 192.168.0.11 access to the central server to write any file named *.* to /var/log/messages. By default each install sends logs to different locations. For example, by default FreeBSD has a line showing cron.* /var/log/cron . This sends all log files called cron.* to /var/log/cron. We could separate cron messages from our workstation (fbsd-wrk) by doing the following.

!*
+192.168.0.11
cron.* /var/log/cron.fbsd-wrk
*.* /var/log/messages

There are many more variations and applications other than cron that send messages. Any of these can be sent to a different location as above. A little Googling on the net will produce more details. Continuing with our simple example of receiving all messages and putting them on /var/log/messages, we now need to tell fbsd-wrk to send it's messages to fbsd-syslog. To do this fbsd-wrk must have it's /etc/syslog.conf file modified to read like the following.

*.* @192.168.0.10

That is all that should be present in fbsd-wrk's /etc/syslog.conf file. Now simply restart syslogd on both machines by running the following as root.

# /etc/rc.d/syslogd restart

To verify that this worked log into the central syslog server, fbsd-syslog in our example, and type the following.

# tail -f /var/log/messages

In another terminal log into the workstation, fbsd-wrk in our example, and type the following.

# logger hello world

Our "hello world" message should appear on the `tail` running on the central server. If the syslog server is receiving a lot of log entries, the log files may be turned over too frequently. This can be remedied by changing the "size" column entry in /etc/newsyslog.conf . That's it, enjoy.

Distributed Command Execution with Perl

2008-01-30T11:58:00.002-05:00

In modern workplaces, it is unreasonable to expect system administrators to manually run commands across vast numbers of servers. It is simply inneficient and unessesary. In order to resolve this many remote administration tools have been created. Tools vary from distributed shells to configuration management sofware. These tools may be overkill for smaller environments. To add to the multitude of distributed tools the following Perl script was created. Using the script is simple. First change the @bsdhosts array elements to the hostnames in your environment. Second, change the word "user" to a valid username that can ssh into the remote hosts. Finnally, change `pwd;ls` to whatever command needs to be performed on the remote hosts. To run multiple commands within one session, simply use a semicolon to seperate the commands as shown below. Passwords can be avoided by using `keygen`. There are many sites describing how to do this on the net. Here is one such location to get you started http://gentoo-wiki.com/SECURITY_SSH_without_a_password . In short, for a small environment where there is no need for a complex distributed solution a simple Perl script can step up to the task. Enjoy!

#!/usr/bin/perl

###########################################################
# Summary: An easily modified script to run remote commands.
#
# Last modified: 01/30/2008
#
# Author: Javier Prats
#
###########################################################

use warnings;
use diagnostics;
use strict;

our $counter;
our @bsdhosts=qw(hostname1 hostname2 hostname3 hostname4);

foreach $counter (@bsdhosts)
{
       my @command= ("/usr/bin/ssh user\@$counter pwd;ls");
       system(@command);
}

Checking Disk Space Remotely

2007-11-29T10:49:00.000-05:00

Recently I have been looking for ways to monitor disk space of remote servers. We have a Nagios server that has the NRPE plugin. This works fine on Linux, but I recieved an error on FreeBSD. Creating a script to do these checks seemed like it would be quicker than configuring the NRPE plugin to work. Before reinventing the wheel I Googled around and found a very helpful bash script here which can also be found below.
-

#!/bin/bash
ADMIN="me@somewher.com"
ALERT=70
ssh user@ip-address.com df -H  > /tmp/df.out
cat /tmp/df.out | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
partition=$(echo $output | awk '{ print $2 }' )
if [ $usep -ge $ALERT ]; then
echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
mail -s "Alert: Almost out of disk space $usep" $ADMIN
fi
done

Although, this does what I am attempting to accomplish, we do not have Bash on our FreeBSD servers. Below you can find my Ksh93 port of the script.

1 #!/usr/local/bin/ksh
2
3 ############################################################################################
4 #
5 # Summary:
6 # This script checks disk usage on remote hosts.
7 # Based on a script by Nixcraft posted on
8 # http://nixcraft.com/shell-scripting/3238-shell-script-check-disk-space-remote-systems.html
9 # which was modified and ported to Ksh by Javier Prats
10 #
11 # Author[s]: Nixcraft, Javier Prats
12 #
13 # Last Modified: 11/28/07
14 #
15 ############################################################################################
16
17 ADMIN="user@emailaddress.com"
18 ALERT=70
19
20 typeset -i usep
21 typeset -A hostnames
22 set -A hostnames hostname1 hostname2 hostname3 hostname4
23
24 for i in ${hostnames[@]}
25 do
26         print "checking $i";
27         ssh user@${i} df -H > ~/df.out
28         cat ~/df.out | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
29
30         do
31                 #echo $output
32                 usep=`echo $output | awk '{print $1}' | cut -d'%' -f1`
33                 partition=`echo $output | awk '{print $2}'`
34                 if (($usep >= $ALERT))
35                 then echo "Running out of space \"$partition ($usep%)\" on ${i} as on `date +%m/%d/%Y`"|mail -s "Alert: Almost out of disk space $usep" $ADMIN
36                 fi
37         done
38 done

A few things will need to be modified. First the "ADMIN" variable on line 17 needs to be a valid email address to recieve alerts on. Second, an array was added to the original script in order to be able to check multiple servers. Change "hostname1", "hostname2", etc on line 22 to valid hostnames you would like to check. Finally, this script uses ssh, so the user on line 27 must be modified to show a real username. On line 18 there is an "ALERT" variable. This sets the threshold for email alerts. By default email alerts are sent when disk usage is above 70%. This value can be changed to whatever is deemed reasonable. Enjoy.

Bugzilla on FreeBSD

2007-09-01T10:18:00.001-05:00

When asked to install Bugzilla at work, I wasn't concerned since FreeBSD has it in the ports tree. However, when it came to installing the necessary Perl modules I came accross a few issues. To install Bugzilla, the documentation at http://www.bugzilla.org/docs/2.22/html/index.html was being used. Three of the required modules were producing "/usr/bin/make --NOT-OK" errors. This result could not be replicated on all machines so it does not appear to be a problem with the modules themselves. I found workarounds for all three modules. Here is how to get them installed if you experience the same snags and are limited on time. First was the MIME::Parser module. MIME::Parser also had a "must use force to install" message in it's error. From the Perl CPAN shell, accessed by using `/usr/bin/perl -MCPAN -e shell`, simply type `make -f MIME::Parser`. Second was the Mail::Mailer module. After some research I found that this module is included in /usr/ports/mail/p5-Mail-Tools. I suspect there is some difference in modules as the port version installed without any complaints. Finnally was the Image::Magick module. ImageMagick's Makefile shows an option USE_PERL5=YES. In experimentation I installed ImageMagick without any other knobs. To ensure that this resolved the problem I ran Bugzilla's checksetup.pl again. That's all there was to the process. There wasn't enough time to actually research what the cause of these problems was. If anyone else has come across these please feel free to post your findings. Anyone else in a similar situation should find this to be an acceptable and fast workaround with a larger footprint being the only consequence.

Burning audio CD's in FreeBSD

2007-08-27T21:11:00.001-05:00

There are many places which show the process of extracting tracks from a CD and then burning the audio to a backup CD. Most of these directions however are very "hands on". Below you will find a way to automate the process with a simple Perl script. It is not very intelligent (I may work on this further later) but at least it saves a lot of typing. If your CDROM device is not "acd0" make the appropriate changes. Simply copy this script into a text file, save it, and make sure it's made executable. You will probably have to run this script as root.

#!/usr/local/bin/perl

##########################################
# Summary:
# This is a script to remove some of the
# redundancy and monotony of burning an
# audio CD in FreeBSD.
#
# Dependancies:
# Perl, dd, burncd
#
# Last modified: 08/27/2007
#
##########################################

use warnings;
use diagnostics;
use strict;

my @tracks;
my $counter;
my $extension=".cdr";

# Make sure the correct files are in /dev.
# Retaste the media.

system(`dd if=/dev/acd0 of=/dev/null count=1`);

# Create a list of all tracks in /dev
system(`ls /dev|grep acd0t > tracklist.tmp`);

# Assign all the tracks in the list to
# the @tracks array.
open(TRACKS, "tracklist.tmp");
chomp(@tracks=<TRACKS>);
close (TRACKS);
system(`rm tracklist.tmp`);

# Rip each track.
foreach $counter (@tracks)
{
system(`dd if=/dev/$counter of=$counter$extension bs=2352`);
}

# Prompt for a blank CD and burn it.
print "Please enter a blank CD and press enter.";
<STDIN>;
system(`burncd -f /dev/acd0 audio *.cdr fixate`);
system(`rm *.cdr`);

Using VNC

2007-08-10T07:51:00.001-05:00

Most of us work in hybrid environments. In most cases end users will have Windows machines, designers will be on OS X, and the servers will be running some form of *nix. Rdesktop is a very good solution if one just needs to RDP into a Windows box, but what if you need to get on an OS X machine or need to see Xorg on another machine. VNC is available for all three of these platforms. VNC performs its job well although slow.

I'll mainly be describing the FreeBSD configuration as this is a BSD site and FreeBSD it the flavor I'm most familiar with. This process should be very similar across all the *nix. Feel free to add OS specific instructions. I'll gladly post them. Use the relevant porting system or package manager in your case. For FreeBSD it is available in the ports tree. If the machine which will be installed on only needs the client, VNC can be compiled without it. From /usr/ports/net/vnc type `make -DWITHOUT_SERVER install clean` and only the client will be built. All the options can be seen in the MakeFile for ports.

If the computer is going to accept VNC clients the server needs to be started as the user that will be logging in. For example, if user guest01 wants to vnc into a FreeBSD VNC server from a Windows machine he/she must log into the FreeBSD machine as guest01 (or `su` as guest01) and run `vncserver`. VNCserver will ask for a password. Whatever you provide as a password is what will be used to authenticate the client. For simplicity I recommend using the same password that is used to login to the FreeBSD machine. VNCserver will then show the hostname followed by :1 . This shows what port VNCserver for user guest01 is running on. Now user guest01 can start VNCviewer from his/her windows machine, fill the host field with "hostname:1", type password that was set, and should connect.

When VNCviewer is started for the first time it defaults to the TWM window manager. Each user has a .vnc directory in his/her home directory containing a file called xstartup. This file is used similarly to .xinitrc. If one prefers to use Fluxbox for example, simply comment out or delete the line containing twm and add fluxbox&. This will cause VNCviewer to start in Fluxbox from now on. The file will look similar to the following.

#!/bin/sh

[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#twm &
fluxbox&

As sessions are started by different users the port numbers will increase. A lock file is created in /tmp for each session. This tells the vncserver what is available for users to use. Using the above example, there will be a file called /tmp/.X1-lock . These lock files remain and as users forget what port they were on and start new VNCserver instances port numbers and lock files can grow out of control. Investigate which sessions are not being used and then `kill` them. Running `ps aux|grep vnc` returns the vnc sessions, their owner, and what port it is running on. Finnally go into /tmp and remove the lock file for the relevant port.

VNC does not have good security. Although it is beyond the scope of this quick how-to, it is possible to tunnel VNC over SSH. This adds great encryption with a minimal hit on performance. Below are two good articles explaining how tunneling can be accomplished.

http://www.vnc.com/pipermail/vnc-list/2005-October/052697.html

http://www.oreillynet.com/cs/user/view/cs_msg/24540

Although VNC is not the smoothest or most secure way to graphically connect different operating systems it is one of the most compatible and easiest to use.

reCAPTCHA

2007-05-25T01:58:00.001-05:00

Recently, Slashdot featured an article about a new version of CAPTCHA called reCAPTCHA. CAPTCHA is the letters that are usually crossed out or distorted in some fashion on web pages. Usually the user is asked to re-enter these letters for validation purposes. CAPTCHA prevents bots and form completing software from accessing the site. Since these letters and numbers are usually images, as opposed to actual characters, bots cannot recognize them. Sites using CAPTCHA can now give back to the community by using reCAPTCHA. Instead of using random numbers and letters purely for authentication, reCAPTCHA uses text from scanned books which image recognition software did not validate. This means when a user validates he or she is actually contributing to the act of publishing one of these scanned texts to the web. This is a fantastic idea! Web sites continue to deny bots access while at the same time helping release new text to the community. Implementing reCAPTCHA onto a site is not a very complicated task. Users are required to enter a little more text then with CAPTCHA, but this is a trivial down side considering the benefits in my opinion. It would be great to see webmasters, developers, and admins contribute the small amount of time it would take to convert their sites to reCAPTCHA.

Tax Software Update

2007-04-12T07:16:00.000-05:00

I just stumbled across an article relevant to my "BSD Friendly Tax Software" post. Apparently TurboTax has more problems than a poor interface. slowness, and incompatibility. It seems it also allows users to look at other customer's returns. This is obviously a huge security concern and creates a threat of identity theft. For more information look at the nbc4.com article here.

BSD Friendly Tax Software

2007-04-06T02:35:00.001-05:00

Tax time is approaching it's frantic last minute rush. Over the last few years more companies have been offering there software online. This is a great change for BSD users who are usually neglected when it comes to software compatibility. Most of the web based services run in-browser making the OS irrelevant allowing users of superior operating systems to avoid the long waits at the post office and tax preparation offices. Here is a quick overview of some of the more popular web based tax preparation software.

My first look into tax preparation was very disappointing. TurboTax, one of the senior and most known pieces of tax software, complained about my operating system as most of these services do. Unlike the others though, it does not have an option to continue anyways. Why TurboTax does not allow me to continue at my own risk is incomprehensible. A large portion of potential users has been lost. I immediately dismissed this service due to it's frustrating lack of support. For those who may be able to use the software via Wine or some other similar method the only other guidance that I can give on this software is that if simple W2 forms are all that are being submitted it is free to use. If you require some of the extra features such as 1099 forms it is more expensive than some of the following options.

Next was TaxCut. TaxCut partnered with H&R Block a few years ago. Like most of these services Taxcut also complained about my operating system, but allowed me to continue. Despite, it's complaint there were no problems preparing my taxes. TaxCut is cheaper than TurboTax when it comes to the "Premium" versions, but for basic W2 filling, TurboTax and TaxAct are free. TaxCut is a little slow to load it's pages, but it's tolerable. Most fields in the forms have links to in depth explanations on what the options mean. There is also an option to submit questions to a "tax professional" if further assistance is needed.

My final review is of TaxAct. TaxAct is also free for the basic version, but it costs $9.95 for the "Deluxe" version. This is still cheap compared to TaxCut. It was good not to see a warning of my incompatible OS. After creating a user account TaxAct went right to buisness. TaxAct is the slowest of the three tax preparation options reviewed. It is slow to load it's pages and the interface is long and slow by design. Where TaxCut asks for several pieces of information per form, TaxAct only gathers three or four at a time. There is a relatively short timeout in TaxAct. Somewhere around ten minutes in my estimation. This is good for security, but a little annoying if you need a restroom or snack break during your taxes. Another weak point of TaxAct is it's little amount of assistance. Other than the aforementioned the other features are similar to it's competitors.

Overall I feel that the best piece of software is TaxCut due to it's interface, assistance, and compatibility. If value is what your looking for and you have a little patience TaxAct is not too far behind.

Firefox about:config

2007-03-26T02:22:00.000-05:00

In conversation at work yesterday, we were discussing the possibility of making some customizations to Firefox. This reminded me of the powerful, though well hidden, about:config file. It seems most users are not even aware of the existence of this file.

Firefox's about:config allows such changes as adjusting the amount of memory used for cache, enabling encryption with RSA authentication, and defining specific rules for the handling of cookies. Some of the setting available through about:config can be modified through the tools menu. There is usually a comment stating that changes can be made from the tools menu instead. In order to access about:config simply type about:config into the URL bar and hit enter. Doing so will bring up many pages worth of options. From about:config changes to networking, installation, browsing, saving, and security can be made, along with other things. There are even BiDi (bi-directional text) changes that can be made. For those users interested in appearance there are also options for customizing hyperlink colors, previously viewed colors, and the browser's background along with other options. Delving into all the available prefrences and the details of what each of them does is beyond the scope of this brief overview. Refer to mozillaZine for details on what each preference modifies.

Changes made via about:config, the tools menu, or by some extensions are kept in the prefs.js file. My prefs.js file is found in ~/.mozilla/firefox/6eawc2od.default/prefs.js though this may vary depending on your system. Although preference changes may also be made through the prefs.js file it is more prone to mistakes which have the potential to make Firefox unusable. This brings us to the usual warning that all files should be backed up before making changes. One advantage that the prefs.js file has over the other editing methods is the ease at which many profiles can be changed simultaneously simply by copying prefs.js for other users. An administrator can tighten Firefox's security settings via about:config then copy the prefs.js file across as many home directories and/or systems as desired.

Having so many options makes Firefox an even more appealing choice for web browsing. The inclusion of about:config and prefs.js gives powerusers the performance and security changes they crave and end users a vast amount of style configurations to try.

FreeBSD Latitude c510/c610

2007-03-20T23:47:00.000-05:00

Recently, I acquired a used Dell Latitude. It was easy to find documentation on Dell's site for the laptop, unfortunatly none of it was very usefull to get FreeBSD set up on it. Only the video and sound need some tinkering. Here is the relavant information for the two devices that were hard to find.

For sound the Latitude c510/c610 has an Intel based device. In order to get it to function AC97 compatibility is needed. This uses the snd_ich driver. For information on how to load the module reference the FreeBSD handbook . Video on this system comes from a 16mb ATI Radion Mobility, so select the appropriate choices when using xorgconfig.

These are really the only two pieces of information that were necessary and hard to find. Overall the Latitude c510/c610 works great out of the box. Hopefully this is of assistance to Latitude c510/c610 owners.

FIRST POST!

2007-03-19T05:43:00.000-05:00

Being that it is my own blog I would hope to be able to get first post. It seems to be the only way.