Category: Bash

Posted on

Connection problems to github.com

One of my colleagues had a small issue with pulling the latest from github for my site today. The error was:

ssh: connect to host github.com port 22: Connection refused
fatal: The remote end hung up unexpectedly

Of course the word “fatal” would freak out any non linux user. So a with a little searching the following is the solution.

Add the following to your ./.ssh/config:

host github.com
    hostname ssh.github.com
    port 443

This forces ssh over https for github. All fixed! Now I do not have to bother networking for a firewall rule.

Posted on

Add Your own Script to Default Run Levels in Ubuntu

This uses the old Sys V init.d scripts. Not upstart. I may write a post on upstart at a later date. So, you write a cool script that you want to run at startup. Place it in /etc/rc.d/init.d and make it executable. Next run the following command:

sudo update-rc.d my_cool_script defaults

This will set the script to run on all the default run levels.

Posted on

svnsync problems

I have a little cron job that syncs two svn repos. It runs every ten minutes. For some reason it stopped working so I killed the job. Now when I try to manually run the job, I get:

Failed to get lock on destination repos, currently held by...

Great, the sync is broken!
Here is how to clear it up. Use the following command to remove the lock:

svn propdel --revprop -r0 svn:sync-lock file:///path/to/the/repository

This is the sync script:

#!/bin/sh
svnsync synchronize file:///path/to/the/repository --username svnuser --password svnpassword
Posted on

Rsync Between Servers Automatically

Part 1 – Setting up SSH key authentication

First, we need to make sure the DESTSERVER has the ability to use key authentication enabled. Find your sshd configuration file (usually ‘/etc/ssh/sshd_config’) and enable the following options if they are not already set.

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

If you edit the file be sure to restart sshd afterwards.

# /etc/init.d/sshd restart

Next, on the SOURCESERVER we will create the public / private key pair to be used for authentication with the following command.

# ssh-keygen -t rsa
  • Note: Do not enter a passphrase for this, just hit enter when prompted.

This should create 2 files, a public key file and a private key file. The public key file (usually [homedir]/.ssh/id_rsa.pub) we will upload to the DESTSERVER. The private key file (usually [homedir]/.ssh/id_rsa) we will keep on the SOURCESERVER.

  • Be sure to keep this private key safe. With it anyone will be able to connect to the DESTSERVER that contains the public key.

Now we will plant the public key we created on to the DESTSERVER. Choose the user account which you will use to connect to on DESTSERVER, we’ll call this user ‘destuser’ for now.

In that account’s home directory, create a ‘.ssh’ subdirectory, and in that directory create a new text file called ‘authorized_keys’. If it already exists, great, use the existing file. Open the ‘authorized_keys’ file and paste in the contents of the public key you created in the previous step (id_rsa.pub). It should look something like the following

ssh-rsa <lots and lots of characters…> sourceuser@SOURCESERVER

Save the file and change the permissions to 600 for the file and 700 for the ‘.ssh’ directory.

Now to test that the keys are working. From the SOURCESERVER try logging in as normal using ssh to the DESTSERVER.

# ssh destuser@DESTSERVER

If all is working you should not be prompted for a password but instead connected directly to a shell on the DESTSERVER.

Part 2 – Creating the rsync script

 

Now for the rsync script.

#!/bin/sh

SOURCEPATH=’/var/svn/workingcopies/beta.app.jp/’
DESTPATH=’/var/www/lamp_root/wwwapps/app.jp’
DESTUSER=’updater’
LOGFILE=’/var/log/rsync_beta.log’

echo $’\n\n’ >> $LOGFILE
for SERVER in web01 web02
do
        rsync -av –rsh=ssh $SOURCEPATH $DESTUSER@$SERVER:$DESTPATH 2>&1 >> $LOGFILE
        echo “Completed at: `/bin/date`” >> $LOGFILE
done
Part 3 – Setting up the cron job

This is optional. I assume web updates will be done manually.

Assuming everything has worked so far all that’s left is to setup a cron job to run the script automatically at a predefined interval.

Use the ‘crontab’ command to create a new cron job.

>sudo crontab -e -u updater

This will open an editor where you can schedule the job.

Enter the following to have the script run once every hour

0 * * * * /var/scripts/rsync_betajp.sh
Posted on

Shell script to list current website IP

If your sites are in multiple geographical locations, this script will show where a site currently is being served from. It colors the output based on location.

#!/bin/bash
# where_are_you.sh
#do a dig against the company name servers and spit out the current IP of the sites.

array=( www.site.com www.site2.com www.site3.com www.site99.com )

for i in “${array[@]}”
do
dig @ns.server.company $i a | grep -v ‘;’ |grep $i | awk ‘{ if ( substr($5, 1, 8) == “x.x.x” ) printf “%-30s %s\n”, “\033[1;32;40m”$5,$1; else printf “%-30s %s\n”, “\033[1;34;40m”$5,$1 }’

done | sort -n
tput sgr0

Posted on

Deployment Script WIP

This is a work in progress…

#!/bin/sh
SOURCEPATH=’/var/git/working_copies/website/webroot/’
DESTPATH=’/var/www/website/webroot’
DESTUSER=’deploy’
LOGFILE=’/var/log/website_rsync.log’
SUBJECT=”website.com  Prod Updated:from MANAGE01SRV”
EMAIL=”me@me.com”
EMAILCC=”me@me.com“
echo “Job started at: `/bin/date`” > $LOGFILE
for SERVER in web01 web02 web03 web04
if $SERVER = “web01” then NSSRV = “wb01”
else NSSRV = “wb02”
do
ssh nsroot@netscaler ‘disable server site01_$NSSRV’
ssh $DESTUSER@$SERVER service apache2 stop >> $LOGFILE
rsync -avc –delete –exclude=”.git” –exclude=”var/” –exclude=”.gitignore” –rsh=ssh $SOURCEPATH $DESTUSER@$SERVER:$DESTPATH 2>&1 >> $LOGFILE
ssh $DESTUSER@$SERVER rm -rf $DESTPATH/var/cache/*
ssh $DESTUSER@$SERVER /var/scripts/fixperms.sh >> $LOGFILE
ssh $DESTUSER@$SERVER service apache2 start >> $LOGFILE
ssh nsroot@netscaler ‘enable server site01_$NSSRV’
done
echo “Job completed at: `/bin/date`” >> $LOGFILE
/usr/bin/mail -s “$SUBJECT” “$EMAIL” < $LOGFILE
Posted on

Analyzing Apache Log Files

List Client IP and Count Hits

awk -F'[ “]+’ ‘$7 == “/” { ipcount[$1]++ } END { for (i in ipcount) { printf “%15s – %d\n”, i, ipcount[i] } }’ access.log

 

List Client IP and RDNS

cat access.log | awk ‘{print $1}’ | logresolve