linux

How To Backup Your Website Through SSH Command Line

Tue, 21 Feb 2012 13:46:54 +0100

How To Backup Your Website Through SSH Command Line

Backing up your website or blog can be an expensive and arduous task, requiring a variety of plugins, or additional plans from your hosting provider – but it needn’t be really. If you have SSH access to your website host (generally you would need at least a virtual private server for this), then it’s easy to backup, restore and migrate your entire website with only a few commands. Let me show you how.

What is SSH Command Line?

SSH gives you the ability to talk directly to your web-server. It doesn’t give a pretty interface, or a nice GUI, just a straight-up powerful command line. This can be daunting to some people, but the sheer power, speed, and level of automation it provides can be an absolute life-saver and makes the process of migrating sites incredibly easy.

Most shared hosts unfortunately don’t allow SSH access to your account, at least not by default. If you’re hosting with GoDaddy, you can enable it though, so be sure to check first.

To log in via SSH, open up the Terminal in OS X (or get some free SSH software for Windows) and type in the following:

ssh username@yourdomain.com

You’ll be prompted for your password. If you’ve never used SSH before, you might be surprised when typing in your password doesn’t anything on screen. Don’t worry, that’s for security.

Once logged in, you’ll be presented with a command prompt, similar to the following:

-bash-3.2$

This means everything is fine, so go ahead and continue with these commands.

Start by taking a look around and trying to navigate to your web directory. Type:

ls

To ‘list’ the current files and folders.

cd directoryname

to change to a directory. In this case, I’m going to navigate to the httpdocs directory, which is the root of my web site (where all my wordpress files are stored). You can then ‘ls’ again, just to be sure.

At this point, we’re ready to begin the SSH backup process.

Backing up the Database:

Since the majority readers will be doing this with a WordPress install, you will most certainly have a database to backup in addition to any files stored on the site. First off, you’ll need 3 bits of information to backup your database, but all can be found within wp-config.php (if you’re running wordpress, that is):

Database name
Database user
Database password

Then, issue this simple command, being sure to replace the username, table name, and backup filename where neccessary:

mysqldump --add-drop-table -u username -p tablename > backupfilename.sql

Hit enter, and enter your password. Once it’s run, you can then issue another ‘ls’ command to check that the file has been output. Congratulations, this is all the information in your database as a single SQL file, ready to backup or import somewhere else.

Note: I’ve assumed that your database server is running on the same server on which you are hosting. On a GoDaddy host however, the MySQL database is actually stored remotely on a separate server to which you don’t have SSH access. In cases like these, you will need to access PHPMyAdmin via the hosting control panel, but that is out of the scope of this tutorial.

Backing Up Files:

Now that we have the database stored to single file on the server, we can go ahead and backup both that and your website files down to a single compressed backup file. To do this, we are going to issue one simple command. You need only replace yourbackupfilename with whatever you want it to be called.

tar -vcf yourbackupfilename.tar .

Let me break that down. Tar is a common linux compression format, similar to zip but more efficient. -vcf are simple some options that say “make a new archive, and tell me what you’re doing”. Next is the name of the file we want to create, and finally a single period mark tells it to include everything. We could have written * instead, but this would miss any hidden files such .htaccess which is essential for WordPress.

That’s it. Once that’s run, you will have a single .tar file consisting of every file on your site. You could log in via FTP at this point and download it, but let me show one final step that allows you to restore all these files.

Restoring Everything:

Let’s say the worst has happened, and something has gone horribly wrong with your site. You’ve got a tar file of everything that you backed up last week, so now you’d like to restore it to that. First off, log in via FTP and upload the backup file onto your server. Perhaps you’ve been storing them in a special directory. Either way, move the latest complete backup file into the root of your site, and we’ll begin.

Start by unpacking all the files, the reverse of what we did to back them up:

tar -vxf yourbackupfilename.tar

The crucial difference here is in the -vxf switch, which tells it to extract the files instead of creating a new backup. Also, there is no period on the end of the command this time.

The last step is to suck your database back in to where it was before. Make sure you have a blank database setup with the same password and tablename as before, or you’ll need to change your site configuration settings too. To suck the data back in, issue this command:

mysql -u username -p tablename < databasebackupfilename.sql

Next week: Automating Your Backups

That’s enough to get you started with doing SSH backups for now, then next I’ll show how to automate the task with a simple shell script and a CRON command. If you have some Amazon s3 storage space, I’ll even show you how you can automatically upload your backup files to a storage bucket once they’re done.

One last tip – when I first began to use the command line, this one really impressed me – try pressing the tab key when your typing in a long filename, and if the name is unique enough it will attempt to autocomplete the rest of the filename!

SSH Commands / Tricks - Best 25

Tue, 21 Feb 2012 13:43:45 +0100

25 Best SSH Commands / Tricks

1) Copy ssh keys to user@host to enable password-less ssh logins.

ssh-copy-id user@host

To generate the keys use the command ssh-keygen

2) Start a tunnel from some machine’s port 80 to your local post 2001

ssh -N -L2001:localhost:80 somemachine

Now you can acces the website by going to http://localhost:2001/

3) Output your microphone to a remote computer’s speaker

dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp

This will output the sound from your microphone port to the ssh target computer’s speaker port. The sound quality is very bad, so you will hear a lot of hissing.

4) Compare a remote file with a local file

ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

Useful for checking if there are differences between local and remote files.

5) Mount folder/filesystem through SSH

sshfs name@server:/path/to/folder /path/to/mount/point

Install SSHFS from http://fuse.sourceforge.net/sshfs.html
Will allow you to mount a folder security over a network.

6) SSH connection through host in the middle

ssh -t reachable_host ssh unreachable_host

Unreachable_host is unavailable from local network, but it’s available from reachable_host’s network. This command creates a connection to unreachable_host through “hidden” connection to reachable_host.

7) Copy from host1 to host2, through your host

ssh root@host1 “cd /somedir/tocopy/ && tar -cf – .” | ssh root@host2 “cd /samedir/tocopyto/ && tar -xf -”

Good if only you have access to host1 and host2, but they have no access to your host (so ncat won’t work) and they have no direct access to each other.

8) Run any GUI program remotely

ssh -fX <user>@<host> <program>

The SSH server configuration requires:

X11Forwarding yes # this is default in Debian

And it’s convenient too:

Compression delayed

9) Create a persistent connection to a machine

ssh -MNf <user>@<host>

Create a persistent SSH connection to the host in the background. Combine this with settings in your ~/.ssh/config:
Host host
ControlPath ~/.ssh/master-%r@%h:%p
ControlMaster no
All the SSH connections to the machine will then go through the persisten SSH socket. This is very useful if you are using SSH to synchronize files (using rsync/sftp/cvs/svn) on a regular basis because it won’t create a new socket each time to open an ssh connection.

10) Attach screen over ssh

ssh -t remote_host screen -r

Directly attach a remote screen session (saves a useless parent bash process)

11) Port Knocking!

knock <host> 3000 4000 5000 && ssh -p <port> user@host && knock <host> 5000 4000 3000

Knock on ports to open a port to a service (ssh for example) and knock again to close the port. You have to install knockd.
See example config file below.
[options]
logfile = /var/log/knockd.log
[openSSH]
sequence = 3000,4000,5000
seq_timeout = 5
command = /sbin/iptables -A INPUT -i eth0 -s %IP% -p tcp –dport 22 -j ACCEPT
tcpflags = syn
[closeSSH]
sequence = 5000,4000,3000
seq_timeout = 5
command = /sbin/iptables -D INPUT -i eth0 -s %IP% -p tcp –dport 22 -j ACCEPT
tcpflags = syn

12) Remove a line in a text file. Useful to fix

ssh-keygen -R <the_offending_host>

In this case it’s better do to use the dedicated tool

13) Run complex remote shell cmds over ssh, without escaping quotes

ssh host -l user $(<cmd.txt)

Much simpler method. More portable version: ssh host -l user “`cat cmd.txt`”

14) Copy a MySQL Database to a new Server via SSH with one command

mysqldump –add-drop-table –extended-insert –force –log-error=error.log -uUSER -pPASS OLD_DB_NAME | ssh -C user@newhost “mysql -uUSER -pPASS NEW_DB_NAME”

Dumps a MySQL database over a compressed SSH tunnel and uses it as input to mysql – i think that is the fastest and best way to migrate a DB to a new server!

15) Remove a line in a text file. Useful to fix “ssh host key change” warnings

sed -i 8d ~/.ssh/known_hosts

16) Copy your ssh public key to a server from a machine that doesn’t have ssh-copy-id

cat ~/.ssh/id_rsa.pub | ssh user@machine “mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys”

If you use Mac OS X or some other *nix variant that doesn’t come with ssh-copy-id, this one-liner will allow you to add your public key to a remote machine so you can subsequently ssh to that machine without a password.

17) Live ssh network throughput test

yes | pv | ssh $host “cat > /dev/null”

connects to host via ssh and displays the live transfer speed, directing all transferred data to /dev/null
needs pv installed
Debian: ‘apt-get install pv’
Fedora: ‘yum install pv’ (may need the ‘extras’ repository enabled)

18) How to establish a remote Gnu screen session that you can re-connect to

ssh -t user@some.domain.com /usr/bin/screen -xRR

Long before tabbed terminals existed, people have been using Gnu screen to open many shells in a single text terminal. Combined with ssh, it gives you the ability to have many open shells with a single remote connection using the above options. If you detach with “Ctrl-a d” or if the ssh session is accidentally terminated, all processes running in your remote shells remain undisturbed, ready for you to reconnect. Other useful screen commands are “Ctrl-a c” (open new shell) and “Ctrl-a a” (alternate between shells). Read this quick reference for more screen commands: http://aperiodic.net/screen/quick_reference

19) Resume scp of a big file

rsync –partial –progress –rsh=ssh $file_source $user@$host:$destination_file

It can resume a failed secure copy ( usefull when you transfer big files like db dumps through vpn ) using rsync.
It requires rsync installed in both hosts.
rsync –partial –progress –rsh=ssh $file_source $user@$host:$destination_file local -> remote
or
rsync –partial –progress –rsh=ssh $user@$host:$remote_file $destination_file remote -> local

20) Analyze traffic remotely over ssh w/ wireshark

ssh root@server.com ‘tshark -f “port !22″ -w -’ | wireshark -k -i -

This captures traffic on a remote machine with tshark, sends the raw pcap data over the ssh link, and displays it in wireshark. Hitting ctrl+C will stop the capture and unfortunately close your wireshark window. This can be worked-around by passing -c # to tshark to only capture a certain # of packets, or redirecting the data through a named pipe rather than piping directly from ssh to wireshark. I recommend filtering as much as you can in the tshark command to conserve bandwidth. tshark can be replaced with tcpdump thusly:
ssh root@example.com tcpdump -w – ‘port !22′ | wireshark -k -i -

21) Have an ssh session open forever

autossh -M50000 -t server.example.com ‘screen -raAd mysession’

Open a ssh session opened forever, great on laptops losing Internet connectivity when switching WIFI spots.

22) Harder, Faster, Stronger SSH clients

ssh -4 -C -c blowfish-cbc

We force IPv4, compress the stream, specify the cypher stream to be Blowfish. I suppose you could use aes256-ctr as well for cypher spec. I’m of course leaving out things like master control sessions and such as that may not be available on your shell although that would speed things up as well.

23) Throttle bandwidth with cstream

tar -cj /backup | cstream -t 777k | ssh host ‘tar -xj -C /backup’

this bzips a folder and transfers it over the network to “host” at 777k bit/s.
cstream can do a lot more, have a look http://www.cons.org/cracauer/cstream.html#usage
for example:
echo w00t, i’m 733+ | cstream -b1 -t2

24) Transfer SSH public key to another machine in one step

ssh-keygen; ssh-copy-id user@host; ssh user@host

This command sequence allows simple setup of (gasp!) password-less SSH logins. Be careful, as if you already have an SSH keypair in your ~/.ssh directory on the local machine, there is a possibility ssh-keygen may overwrite them. ssh-copy-id copies the public key to the remote host and appends it to the remote account’s ~/.ssh/authorized_keys file. When trying ssh, if you used no passphrase for your key, the remote shell appears soon after invoking ssh user@host.

25) Copy stdin to your X11 buffer

ssh user@host cat /path/to/some/file | xclip

Have you ever had to scp a file to your work machine in order to copy its contents to a mail? xclip can help you with that. It copies its stdin to the X11 buffer, so all you have to do is middle-click to paste the content of that looong file :)

Working with Permissions - 2

Thu, 16 Feb 2012 14:37:11 +0100

Permissions - Part II

Working with Permissions

Now that you should have the basics of file permissions down, this section includes some "tips and tricks" that I use to ensure that permissions are properly set on servers.

Home Directories

Most admins overlook the setting of permissions on user's home directories. I believe, this is because most admins do not have a good understanding of Unix Permissions (especially those coming from a Windows background). Because of this, most Linux Servers are deployed with the default permissions in place, which are the owner has full permissions and everyone else has read permissions. This may not be what you want (everyone has the ability to read everyone else's files), and you should seriously look at home directory permissions before deploying any servers.

To fully understand what you can accomplish with home directories I am going to give an example that I used when deploying SLES at a local high school:

First, in order to maintain some sort of organization, the school decided to separate out the home folders into the student's graduation year. So within the "/home" directory, there were directories "2008", "2009", "2010", "2011", etc. for all of the student's home directories. There were also directories called "teachers", "staff" and "admins" to hold all of the staff's home directories.

Using separate containing directories for the user's home directories proved to be quite successful not only for organization purposes, but also to ensure that correct file permissions were enabled. For every student's home directory we adjusted the permissions to have the group ownership of the "teachers" group and we ensured that the teachers could write to the folders. We also ensured that the other students had no access to other student's files. For instance, to set permissions we ran:

cd 2009
chgrp teachers * -R
chmod g+w * -R
chmod g+s * -R
chmod o-rwx * -R

This ensured that the teachers had write permissions to the student's files, while ensuring that any new file that was created by the student had the "teacher" group as the group owner (using the setgid bit). We did this for all of the student's folders as well as all of the staff's folders (except we changed the group ownership to "sadmins" (Super Admins) instead of "teachers" for the staff).

Note: We did run into one problem when implementing this. If a user logged into a Suse Linux workstation a permissions error appeared upon login. To fix this error we simply had to manually change the permissions of the ~/.dmrc file to 0644 to fix the error. (See the script below)

Scripting File Permissions

Once you have a deployment strategy for file permissions, you may want to create a shell script to apply those permissions. This alleviates manually running a group of commands after you create a new user or when the permissions are not correct (which occasionally happens when "copying" files).

Using the above example of the school, here is a shell script that is ran as a daily cron job to ensure the permissions are correct:

#!/bin/bash

# This script will reset permission's on home directories every night

cd /home

# First we will reset the permissions for the Student Folders.
# We store all of the students folders within their graduation year
# This makes it easy to sort the student's data

# The first loop changes into the student's graduation year directory

for grad_dirs in $(ls 20* -d); do
	cd $grad_dirs

# This next loop actually changes permission on all the student's files
#  and directories.

	for userlogin in $(ls); do
		cd $userlogin
		find -type d -exec chmod 2775 '{}' \;
		find -type f -exec chmod 664 '{}' \;
		install -m 0644 /var/defaults/.dmrc ./
		cd ..
		chown $userlogin $userlogin -R
	done

# Finally, while we are still in the grad year directory, we change group
#  ownership of all the files and directories to the teachers. This allows
#  them full access to the students files and folders.  Before we exit the
#  graduation year directory we also remove "others" access to the student's
#  directory - this ensures students can't access other students data.

	chgrp teachers * -R 
	chmod 2750 *
	cd ..
done

# Now we will do the same thing for the Teachers and Staff Accounts, the
#  difference will be the group owner will be sadmins (Super Admins) instead
#  of teachers.  This will allow Teachers and Staff to store private data
#  in their home directories, while allowing Administrators to monitor their
#  disk usage and ensure no copyright material is stored on school property.
for staff_accounts in staff teachers
do
        cd $staff_accounts
        # Run the same loop as the student dirs above
        for userlogin in $(ls); do
                cd $userlogin
                find -type d -exec chmod 2775 '{}' \;
                find -type f -exec chmod 664 '{}' \;
		install -m 0644 /var/defaults/.dmrc ./
                cd ..
                chown $userlogin $userlogin -R
        done

        chgrp sadmins * -R
        chmod 2750 *
        cd ..
done

As you can see in the above example it can be easy to ensure correct permissions are maintained on your shares. Another thing to note in the above example is the procedure used to change the permissions of items depending upon whether they are files or directories (since these two entities have different permissions - i.e. directories should be executable). So, if you copy files from a CD or another server, to ensure the permissions are "Unix Compatible" run something similar to:

cd directory
find -type d -exec chmod 755 '{}' \;
find -type f -exec chmod 644 '{}' \;

A Basic Network Share

Now I will put all this together and show how to create a network share that is fully accessed by members of the "company" group:

mkdir /srv/exports/share

chgrp company /srv/exports/share
chmod g+w /srv/exports/share
chmod o-rwx /srv/exports/share

cp /old/files/* /srv/exports/share -f -r
cd /srv/exports/share

chgrp company * -R
find -type d -exec chmod 2770 '{}' \;
find -type f -exec chmod 660 '{}' \;

Once that is done, if the share is accessed using NFS on Linux workstations, ensure the umask on those workstations is set to 002 or 007 (to ensure the "Group" has write access on new files), and if the share is accessed on Windows workstations, ensure the appropriate "masks" are set in the Samba Share definition.

Access Control Lists

Although basic file permissions will probably work for nearly all of your file share deployments, occasionally you might need the ability to "fine-tune" permissions for some network shares or files. You can accomplish this control by using Access Control Lists (ACLs) on your files and directories.

For this tutorial, I am not going to go into detail regarding ACLs. Although, it is easy to adjust and add ACLs from Linux using either the Konquerer file manager or the Nautilus file manager (with the eiciel plugin). You can even adjust ACLs using Windows Explorer (from remote Windows Workstations connecting through Samba). However, a word of warning is important: Not all GNU/Linux applications properly support ACLs (yet), if you deploy ACLs on Directories / Files accessed through NFS using SLED or another GNU/Linux Distribution, ensure that all of the apps that will access these shares properly support ACLs.

Finishing Up

Hopefully, after reading this you have a better understanding of Permissions within a GNU/Linux system. Once you develop a deployment plan, I highly recommend creating a script to ensure proper permissions are maintained on your servers. Linux file permissions may look too simplistic at first glance, but once you grasp the concept you will quickly see the ingenuity (and the power) behind them.

File & Directory Permissions on Linux

Thu, 16 Feb 2012 14:34:26 +0100

File & Directory Permissions on Linux

Special Bits

So far, you have been shown the basic manipulation of file and directory permissions. However, there are quite a few circumstances which these will not be enough for the situation. For instance, what if you have a few people that have different "Default Groups", but are members of another group and you need them to all have write access to certain files. Or what if you have some less knowledgeable users that may accidentally delete other peoples files if they are given write access to a directory.

To rectify these situations, Linux provides what are called "Special Bits" that you can apply to files and directories to change their default behaviour.

The setuid Bit

The setuid (set user id) bit allows you to specify which user a certain program is executed as. This is not really useful for sharing files on a network, but can be invaluable when you have an application that needs to run as another user (such as 'root') when launched. An example:

chown root /usr/bin/myprogram
chmod +x /usr/bin/myprogram
chmod +s /usr/bin/myprogram   (this sets the setuid bit)

In the above example, whenever anyone launches the /usr/bin/myprogram application, the program will have all of the rights of the "root" user regardless who actually launched the file. (Use with Caution)

On the command line, an "ls -l" output of a setuid bit file would be:

-rwsr-xr-x 1 root root 11159 2007-07-16 12:09 myprogram

The setgid Bit

The setgid (set group id) bit can be the cornerstone of good file permissions. This bit allows you to enforce what group ownership a directory (and all it's subdirectories and files) have. For instance if you set the setgid bit to "admins" on a directory, any file (and directory) created below that directory will also have the "admins" group ownership. This allows you to setup a shared network folder that is accessible by any member of the group, and any file below that directory will maintain that group ownership (regardless of the user's primary group). Here is an example:

mkdir accounts
chgrp acctns accounts
chmod g+w accounts
chmod g+s accounts  (this sets the setgid bit)
chmod o-rwx accounts

In this example, any member of the acctns group will be able to access and write to the accounts directory, and since the setgid bit is set, any new file or folder created will have "acctns" as it's group ownership. This allows you to create a shared network folder for a specified group without having to make it "world writable".

Note: Any directory created within a setgid set directory will also be "setgid".

On the command line, an "ls -l" output of a setgid bit directory would be:

drwxrws--- 2 root company 48 2007-07-16 12:12 share

The Sticky Bit

The "sticky" bit (also known as the "Save Text Attribute" bit) is set only on a directory and specifies that only the owner of a file can delete their own file within the directory regardless of the group or other's "writable" status. So, in the setgid example, if you did not want any member of the "accts" group to be able to delete a file (unless they "owned" the file), then you would set the sticky bit on the accounts directory:

chmod +t accounts

On the command line, an "ls -l" output of a "sticky" directory would be:

drwxrws--T 2 root company 48 2007-07-16 12:12 share

Setting Special Bits using Numeric Permissions

As with regular file permissions, you can also use numeric permissions to set these special bits (useful when creating scripts). When setting special bits using numeric permissions, you would increase the number of digits from 3 to 4 digits (for instance, instead of 664 you would use 0644), where the first digit would set the special attributes.

Similar to regular permissions, you need to add the digits together using the following chart:

0     no special bit is set
1     sticky bit is set
2     setgid bit is set
4     setuid bit is set

So, if you want the sticky and the setgid bits set on a directory (as well as group full control and others no access) you would use 3770. For example:

chmod 3770 directoryname
ls -l
drwxrws--T 2 root company 48 2007-07-16 12:12 share

20 things to plan for an IT Disaster Recovery

Fri, 16 Sep 2011 10:41:03 +0100

20 Things to Plan for an IT Disaster Recovery

Implementing a disaster recovery solution is dependent on three factors — 1) time 2) resources 3) dollar amount.

Most organization doesn’t even think about DR when the IT infrastructure and applications are running without any issues. Most of them think about DR only when something breaks that created a major negative impact on the business.

If you are a sysadmin, or someone who is responsible for keeping the IT running, you should be constantly working on disaster recovery. Whether your company allocated time and budget or not, you can still work on some aspects of DR.

The following is list of various items that you might want to consider while planning for a DR. This list is not comprehensive by any means, but it should give you enough ideas to get started on DR.

Resilient primary datacenter. Before you plan for a secondary remote datacenter, you should make sure all the components in your primary datacenter are highly redundant. Your focus should be to design your primary datacenter as resilient as possible that you should rebound quickly from most disasters (except the natural disasters) without having to ever use the secondary remote datacenter. For example, for your production database, have a physical standby database running on the same datacenter, configure dual NIC cards and HBA cards on all prod servers, configure multiple web servers with a load balancer, connect the server to two power circuits using the redundant power supply on the servers, etc.
Remote secondary datacenter. If you implement a resilient primary datacenter, your goal for using a redundant remote datacenter is primarily for natural disasters such as earthquakes, fires, floods, etc. While this might be very obvious, it is worth stating, as I’ve seen few companies do this, they have both the primary and secondary datacenter in the same city, which defeats the purpose of DR. If your primary datacenter is in california, setup a secondary datacenter somewhere in the east coast.
Replicate production components in DR datacenter. You don’t need to replicate all of the hardware and applications from primary to secondary datacenter. A sysadmin, or any technical person can quickly identify all critical hardware and software that needs to be replicated in the DR site, but you might need help from other departments to identify the applications that are critical for the business. You have to map the critical business functions to IT infrastructure components, and make sure all of those infrastructure components along with application and data are replicated to the DR site.
Storage plan. If you have some kind of SAN storage (or NAS storage) that supports the critical application in the primary DR, you need to have similar SAN storage (or NAS storage) on your DR site also. For performance reasons, the prod servers in the DR site should have the same spec as the prod servers in the primary site. However, for storage, if you have a high-end SAN storage from a big vendor on the primary site, consider implementing similar high-end SAN storage from a small vendor, which might cost you lot less money for the same configuration and similar performance.
Replicate data to DR site on an on-going basis. Syncing data between the primary and disaster datacenter is a critical aspect of a successful DR implementation. Once you’ve listed out all the applications that needs to be replicated to the DR site, you need to figure out how to sync the data between these two sites for all these applications. For example, you can replicate Oracle database at a block level using the replication technologies provided by the storage vendors, or you can use datagurad to replicate the data at a oracle level. Both has its own pros and cons. You have to analyze them carefully and choose the one that fits your budget and scope of the DR.
Replicate initial data using manual method. When you are setting up the DR site for the 1st time, you have to do decide how you’ll do the initial data sync. For example, if you are replicate a data warehouse database that is huge in size, you can’t copy the initial backup of the database over the network to the remote site, as it might hog the bandwidth. Instead, you can take a tape backup in the primary datacenter and use it in the secondary datacenter to setup the initial database. Once the initial setup is done, you can implement some form of automatic incremental sync between the sites.
RTO stands for Recovery Time Objective. Working with the management team, you identify the acceptable RTO for the business. For example, your organization may decide that the acceptable RTO is 8 hours. i.e After a disaster, within maximum of 8 hours all the critical applications should be fully operational at the DR site. RTO has a direct impact on how much dollar amount is spend in implementing the DR solution. For example, a RTO of 1 hour might need a very sophisticated DR solution that is way too expensive than what is required by 24 hour RTO.
RPO stands for Recovery Point Objective. Just like RTO, you should work with management to decide an acceptable RPO for the business. For example, your organization may decide that the acceptable RPO is 2 hours. i.e After a disaster, when you failover the service to secondary DR, 2 hours of data loss is acceptable to business. For example, if the disaster happens at 3 p.m, after you restore the system in the DR site, it will contain data from production only as of 1 p.m. So, you’ve lost data from 1 p.m – 3 p.m. In simple terms, if your RPO is 2 hours, your business should be willing to loose 2 hours worth of data during a disaster.
Automatic or manual failover? You have decide whether you want to automatically failover or manually failover after a disaster. In most cases, a manual intervention is acceptable, as you don’t want to failover automatically to a DR site based on some false signal. Keep in mind that once you failover, there is lot of work involved in coming back to the primary datacenter.
Network failover. I have seen DR plans that gives lot of focus to data replication, but less focus to the networking aspects of the DR. Working with your networking team, you need to identify all the network infrastructures that needs to be replicated. For example, DNA failover to make sure that your production URL is pointing to the DR site after the failover. If you have established VPN connections for your customers, identify how to failover the VPN connections. When you create/modify firewall rules (or anything security related) in the primary datacenter, you need to identify how those security policies can be replicated to the DR site on an ongoing basis.
Remote hand setup. You need to have appropriate plan for accessing the remote datacenter for debugging any issues. You can setup KVM in the DR site, to access the console of the hardware located in the DR site from your office. Or, you need to plan some form of manual remote hand services, where someone can go to the DR site physically and carry out your instructions.
Annual DR testing. Several organizations spend lot of time and money in setting up a DR site, only to figure out that it doesn’t really work as expected when they are in a real DR situation. Once a year, you should validate your current DR configurations to make sure the DR site works properly and meets the original objective. If everything is configured properly, you should be able to manually failover your critical applications from the primary site to the DR site, and let it run there for few days. This also helps you to see how the DR site handles real time load.
DR site as a QA platform. Instead of using DR site only for disaster situation, you can use them as a QA platform to do performance and load testing of your applications. This might be helpful, as you don’t need to invest in additional testing infrastructure in the primary data center. When you take this approach, you’ll sill be syncing data from primary site to the DR site on an on-going basis. However, whenever you do a load testing, you need to implement some additional solution where you take a checkpoint of the current state of the DR site, perform your QA testing, and then immediately rollback to the previous checkpoint and continue syncing the primary site data from there.
DR response plan. Once you’ve implemented the DR site, you need to have a clear DR plan on how you and your team will respond when there is a real disaster. Collaborate with various departments in your organization, and identify the key resources who will be part of the DR response team, and identify their specific role in the DR response plan. The DR response plan is a simple step-by-step instructions on what needs to be done when there is a DR, who will perform those tasks, and in what sequence those tasks will be performed.
Don’t have a DR site? Lot of organizations don’t have a DR site. If you are working for one of them, and you are responsible for critical applications and IT infrastructure, it is your responsibility to come up with a DR plan, and educate your top management on the importance of spending time and dollar on DR, and get their approval. Come-up with three different DR plans—one that costs $$$, one that costs $$, one that cost $. As we explained earlier, the DR plan can vary depending on various factors, and cost is one of them. After you’ve presented your detailed DR plan to your management, if they still don’t approve, at least you’ll feel good that you’ve given your best shot in coming up with a good DR plan.
When to declare a disaster? You need to clearly identify this ahead of time. You need to have a very clear written criteria on when you’ll switch to DR site. i.e What criteria triggers the DR failover criteria? When do you initiate a DR? At what point you declare its a DR, add start working on failing over to the DR site? The answer to these question should be clearly defined, and reviewed by every department in your organization, and finally these criteria should be approved by the top management. For some, when the production is down, because someone deleted something on production by mistake might not trigger a DR. For some organizations, they are probably better off restoring the data from the backup on the primary site itself. For other organizations, business cannot wait until they restore from backup, and they need to switch to DR site.
Backup, backup, and backup. Backups are very important factor in a DR plan. Like we mentioned earlier, your goal should be to never use the DR site, unless a real natural disaster happens. So, a strong backup strategy in your primary site is very critical. You should backup all your critical applications. When you backup your database, store the backup in four locations. Backups are pretty much useless if you don’t restore them on a test server to validate that they are working on an on-going basis.
Applying patches. When you apply OS patch, upgrade firmware, or perform any kind of configuration management to the hardware in the primary site, you need to have a strategy to do those on the DR site on an on-going basis. You don’t want to be in a situation, where the OS configuration on your primary site is different than the DR site.
Successful DR depends on lot of factors. Top management blessing, adequate budget allocation, involvement from all business divisions, strong DR plan, strong technical resources, fully tested DR implementation. Most importantly, a well defined DR scope that is in alignment with the business objective is very critical for a successful DR.
DR documentation. A proper DR planning requires lot of processes to be established. All these processes should be documented properly. For example, a document that explains the escalation process when a DR strikes. A technical documentation that explains what needs to be done to do the failover to DR site. A communication document that lists out all the team members involved in the DR, and what they are responsible for, and how they can be contacted during DR. A document for customer support team, who know what needs to be communicated to customer, and how to reach customers during a disaster. A DR testing document that lists everything that a QA team needs to test after the DR site is live, etc.

We’ve only scratched the surface of the Disaster Recovery. There is lot more to it than the above items. If you are a sysadmin, or someone who is responsible for your IT applications and infrastructure, and if don’t have a DR plan, consider this as a reminder to get something started for your DR strategy.

HTOP

Fri, 16 Sep 2011 10:38:20 +0100

HTOP:

htop is just like top, but on steroids.

Once you are used to htop, you’ll never go back to top again.

htop is a ncurses-based process viewer.

You can interact with the htop using mouse. You can scroll vertically to view the full process list, and scroll horizontally to view the full command line of the process.

This article explains 15 essential htop command examples.

Install Htop

top command is available on all Linux system by default.

To use htop, you need to install it first. Go to htop download page, and download the binaries that corresponds to your Linux distribution and install it.

If you prefer to install htop from souce, download the source code, and do the following. This will install htop under /usr/local/bin.

tar xvfz htop-0.9.tar.gz
cd htop-0.9
./configure
make
make install

1. htop Output

htop output consists of three sections 1) header 2) body and 3) footer.

Header displays the following three bars, and few vital system information. You can change any of these from the htop setup menu.

CPU Usage: Displays the %used in text at the end of the bar. The bar itself will show different colors. Low-priority in blue, normal in green, kernel in red.
Memory Usage
Swap Usage

Body displays the list of processes sorted by %CPU usage. Use arrow keys, page up, page down key to scoll the processes.

Footer displays htop menu commands.

2. Display CPU Averages in Header

htop by default displays a separate bar for every CPU (or core) on your system. On a multi core system, the list of individual CPU bar might be a distraction. Instead of multiple bars, you might want to display only one bar for CPU average usage.

Press F2 (or S) for setup menu -> Select “Meters” under setup -> Use right arrow to go to “Available Meters” column -> Use down arrow to select “CPU average” -> Press “F5″, which will add “CPU average” to the list under the “Left columns”.

Now, use left arrow to go to “Left columns” -> select “All CPUs” -> Press F9 to remove it from the “Left columns” -> Use down arrow, select “CPU” -> Press “F7″ to move it to the top -> Press to come out of the setup menu.

This will show only the CPU average in the bar as shown below, instead of showing all the individual cores.

3. Display Processes Sorted by any htop Output Column – Press F6 or >

By default htop command displays the processes sorted by CPU usage. Press F6 (or >), which will display the column names, select any one of the column, and press Enter.

Note: Instead sorting columns by going to sort menu, you can also press the sortcut key “P” to sort by CPU%, “M” to sort by MEM%, or “T” to sort by TIME%.

4. Kill a Process Without Exiting From htop – Press F9 or k

To kill a process, Select the process that needs to be killed from the list, and press F9 (or k), which will display the “Send signal” menu that lists all the available signals that you can send to the command.

If you want to do “kill -9″ on the selected process, select the “SIGKILL” from this menu, and press enter.

5. Renice a Unix Process Without Exiting From hTop – Press F7 or F8

Select the process that needs to be reniced from the list, and press F7 which will reduce thenice value of the process, press F8 which will increase the nice value of the process by one. If you press F8 three times, it sets the nice value of the process to 3.

6. Display Only Processes of a Single User – Press u

Press u, which will display “Show processes of” menu and list all the users who has some processes running. Select a user from this list, and press Enter, which will display only the processes that belongs to that particular user.

Note: You can also do “htop -u ramesh” from the command line, which will start htop and display only process that belong to the user ramesh.

7. Display Processes in Tree View – Press F5 or t

Tree view is expremely helpful to identify dependencies of a process. Press F5 (or t), which will display all the related processes in a tree view as shown below.

Note: In a tree view, you can selectively collapse (or expand) a specific process tree by pressing either “+” or “-” key.

8. Change the Output Refresh Interval

To change the refresh interval of the htop output, use the -d command line option. “htop -d x”. Where x is referred in thenths of seconds.

htop -d 10

9. Htop Setup Menu – Press F2 or S

The following are the 4 high-level menu items available in htop setup.

Meters – Change the behaviour of the meter bar in the header
Display Options – Customize some of the htop display options
Colors – Choose a color theme for the htop output
Columns – This is extermly helpful when you are debugging a problem. Choose what column needs to be displayed for the processes in the htop output. This has more than 60 columns that you can choose from.

10. Strace a Process from htop – Press s

You can attach the selected process to strace by pressing s. This will display live update of the strace output displaying the system calls invoked by the selected process as shown below.

11. Display lsof Output inside htop – Press l

Select a process from the htop output and press l, which will display all the open files by the selected process. This uses lsof.

12. Scroll Process list Hoziontally and Vertically

Unlike top, in htop you can scroll the process list both horizontally and vertically. Select a process and press down arrow key which will scroll the list vertically and show you more processes in the list. You can also press “Page Up” or “Page Down” key to scoll the process list.

Select a process and press right arrow key which will scroll vertically to display the full command line of the process.

13. Follow a Process in htop Output – Press F

Select a process, and press F, which will follow the process. Since the htop output is sorted based on a column (by default %CPU), a process will keep changing its position in the list.

Following a process in the htop output is helpful when you are monitoring a particular process.

14. Kill Multiple Processes using Tag – Press Space and F9

You can group multiple processes by tagging them. Select a process, and press “Space bar”, which will tag the selected process. Once a process is tagged, the color of that line will change indicating that it is tagged. Scoll the list and tag multiple processes, and press F9, which will display the kill menu, from where you can kill all the tagged process.

Note: To untag a specific process, highlight that process, and press space bar again. To untag all processes press U.

15. Use Shortcut Character Keys (Instead of Function Keys)

The htop footer displays only the function keys to invoke the menu items. I find it easier to remember the character keys (instead of function keys). Also, if you are using putty connection manager (or some other ssh client), where the function keys are mapped to other functionalies, you need to use the following character keys to interact with htop.

Shortcut Key	Function Key	Description
h	F1	Invoke htop Help
S	F2	Htop Setup Menu
/	F3	Search for a Process
I	F4	Invert Sort Order
t	F5	Tree View
>	F6	Sort by a column
[	F7	Nice - (change priority)
]	F8	Nice + (change priority)
k	F9	Kill a Process
q	F10	Quit htop

2 Easy Steps to Enable SSL/HTTP on Tomcat server

Fri, 16 Sep 2011 10:35:46 +0100

2 Easy Steps to Enable SSL / HTTPS on Tomcat Server

If you are running tomcat server that runs only on HTTP, follow the 2 easy steps mentioned below, to configure tomcat for SSL.

1. Create Keystore using Java keytool

First use the keytool to create a java keystore as shown below. Make sure to note down the password that you enter while creating the keystore.

# $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA
Enter keystore password:
Re-enter new password:
What is your first and last name?
 [Unknown]:  Ramesh Natarajan
What is the name of your organizational unit?
 [Unknown]:  Development
What is the name of your organization?
 [Unknown]:
What is the name of your City or Locality?
 [Unknown]:  Los Angeles
What is the name of your State or Province?
 [Unknown]:  CA
What is the two-letter country code for this unit?
 [Unknown]:  US
Is CN=Ramesh, OU=Development, O=Unknown, L=Los Angeles, ST=CA, C=US correct?
 [no]:  yes

Enter key password for
   (RETURN if same as keystore password):

This will create the .keystore file under the /root home directory as shown below.

# ls -l /root/.keystore
-rw-r--r-- 1 root root 1391 Apr  6 11:19 .keystore

2. Modify the server.xml file

Locate the conf/server.xml file located under the tomcat directory. If the Connector port=”8443″is commented out, you should uncomment it first. Please note that the comments in the server.xml file are enclosed in <!– and –> as shown below. You should remove the 1st and last line from the following code snippet.

# vi server.xml
   <!--
   <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
              maxThreads="150" scheme="https" secure="true"
              clientAuth="false" sslProtocol="TLS" />
   -->

Now, add the keystore information to the server.xml as shown below. Replace the your-key-password with the password you provided in the step 1 while creating the keystore.

# vi server.xml
   <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
              maxThreads="150" scheme="https" secure="true"
              keystoreFile="/root/.keystore" keystorePass="your-key-password"
              clientAuth="false" sslProtocol="TLS" />

Finally, restart the tomcat server and access the application using https://{your-ip-address}:8443/

SELinux Features

Mon, 12 Sep 2011 10:53:57 +0100

SELinux Features:

Restricts access by subjects (users and/or processes) to objects (files)
Provides Mandatory Access Controls (MACs)
MACs extend Discretionary Access Controls (DACs (Standard Linux Permissions))
Stores MAC permissions in extended attributes of file systems
SELinux provides a way to separate: users, processes (subjects), and objects, via labeling, and monitors/controls their interaction
SELinux is integrated into the Linux kernel
Implements sandboxes for subjects and objects
Default RH5 implementation creates sandboxes (domains) for 'targeted' daemons and one sandbox (unconfined_t) for everything else
SELinux is implemented/enabled by RHEL5, by default
Operates in the following modes:

a. Permissive - permission is always granted, but denials are logged in: /var/log/messages

b. Enforcing - strictly enforces 'targeted' policy rules

c. Disabled - Only DACs are applied

Operating modes can be applied upon startup or while the system is running

SELinux Config files & Tools:

1. sestatus - displays current SELinux status, including:

a. policy name 'targeted'

b. policy version '21'

c. Operating mode: 'enforcing|permissive|disabled'

2. /etc/sysconfig/selinux - primary startup|config file for SELinux

3. /etc/selinux/targeted - top-level container for the 'targeted' policy

4. setenforce = 0(permissive) 1(enforcing)

5. '-Z' can be applied to the following commands to obtain SELinux context info:

mv, cp, ls, ps, id

6. chcon -R -t type <file> - applies SELinux label to file/directory

Note: If files(objects) lose their SELinux context, there are multiple ways to relabel them:

1. 'touch /.autorelabel && reboot' - init will relable the system according to the 'targeted' policy

2. 'fixfiles' - use to relabel objects (files) while the system is running

Note: List of daemons protected by the 'targeted' SELinux policy:

1. apache(httpd)

2. dchpd

3. ntpd

4. named

5. syslogd

6. squid

7. snmpd

8. portmap

9. nscd

10. winbind

Note: The 'targeted' policy assigns ALL other subjects and objects to the 'unconfined_t' domain

Note: The default SELinux 'targeted' policy, using MACs, binds subject domains: i.e. 'httpd_t' to object types: i.e. 'httpd_config_t'

Note: SELinux MACs compound Linux DACs

Understanding TOP command ouput in Linux

Mon, 12 Sep 2011 10:39:02 +0100

Understanding TOP command ouput in Linux

How do I determine CPU and Memory utilization, based on running processes in Linux using TOP?

Top command provides a real-time look at what is happening with your system. Top produces so much output that a new user may get over whelmed with all that’s presented and what it means.

Let’s take a look at TOP one line at a time.

The first line in top:

top - 22:09:08 up 14 min, 1 user, load average: 0.21, 0.23, 0.30

“22:09:08″ is the current time; “up 14 min” shows how long the system has been up for; “1 user” how many users are logged in; “load average: 0.21, 0.23, 0.30″ the load average of the system (1minute, 5 minutes, 15 minutes).

Load average is an extensive topic and to understand its inner workings can be daunting. The simplest of definitions states that load average is the cpu utilization over a period of time. A load average of 1 means your cpu is being fully utilized and processes are not having to wait to use a CPU. A load average above 1 indicates that processes need to wait and your system will be less responsive. If your load average is consistently above 3 and your system is running slow you may want to upgrade to more CPU’s or a faster CPU.

The second line in top:

Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie

Shows the number of processes and their current state.

The third line in top:

Cpu(s): 9.5%us, 31.2%sy, 0.0%ni, 27.0%id, 7.6%wa, 1.0%hi, 23.7%si, 0.0%st

Shows CPU utilization details. “9.5%us” user processes are using 9.5%; “31.2%sy” system processes are using 31.2%; “27.0%id” percentage of available cpu; “7.6%wa” time CPU is waiting for IO.

When first analyzing the Cpu(s) line in top look at the %id to see how much cpu is available. If %id is low then focus on %us, %sy, and %wa to determine what is using the CPU.

The fourth and fifth lines in top:

Mem: 255592k total, 167568k used, 88024k free, 25068k buffers

Swap: 524280k total, 0k used, 524280k free, 85724k cached

Describes the memory usage. These numbers can be misleading. “255592k total” is total memory in the system; “167568K used” is the part of the RAM that currently contains information; “88024k free” is the part of RAM that contains no information; “25068K buffers and 85724k cached” is the buffered and cached data for IO.

So what is the actual amount of free RAM available for programs to use ?

The answer is: free + (buffers + cached)

88024k + (25068k + 85724k) = 198816k

How much RAM is being used by progams ?

The answer is: used – (buffers + cached)

167568k – (25068k + 85724k) = 56776k

The processes information:

Top will display the process using the most CPU usage in descending order. Lets describe each column that represents a process.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

3166 apache 15 0 29444 6112 1524 S 6.6 2.4 0:00.79 httpd

PID – process ID of the process

USER – User who is running the process

PR – The priority of the process

NI – Nice value of the process (higher value indicates lower priority)

VIRT – The total amount of virtual memory used

RES – Resident task size

SHR – Amount of shared memory used

S – State of the task. Values are S (sleeping), D (uninterruptible sleep), R (running), Z(zombies), or T (stopped or traced)

%CPU – Percentage of CPU used

%MEM – Percentage of Memory used

TIME+ – Total CPU time used

COMMAND – Command issued

Interacting with TOP

Now that we are able to understand the output from TOP lets learn how to change the way the output is displayed.

Just press the following key while running top and the output will be sorted in real time.

M – Sort by memory usage

P – Sort by CPU usage

T – Sort by cumulative time

z – Color display

k – Kill a process

q – quit

If we want to kill the process with PID 3161, then press “k” and a prompt will ask you for the PID number, and enter 3161.

Command Line Parameters with TOP

You can control what top displays by issuing parameters when you run top.

- d – Controls the delay between refreshes

- p – Specify the process by PID that you want to monitor

-n – Update the display this number of times and then exit

If we want to only monitor the http process with a PID of 3166

$ top -p 3166

If we want to change the delay between refreshes to 5 seconds

$ top -d 5

comparison of Ext3 and Ext4 File systems

Mon, 12 Sep 2011 09:07:05 +0100

Here is the quick facts and comparison of Ext3 and Ext4 File systems: Hope this helps..!

Features	Ext3	Ext4
Stands For	Ext3 stands for third extended file system.	Ext4 stands for fourth extended file system.
Introduced	It was introduced in 2001.	It was introduced in 2008.
Kernel Support	Supports from Linux Kernel 2.4.15	Supports from Linux Kernel 2.6.19
Maximum individual file size supported	Maximum individual file size can be from 16 GB to 2 TB	Maximum individual file size can be from 16 GB to 16 TB
Maximum file system size supported	Overall ext3 file system size can be from 2 TB to 32 TB	Overall maximum ext4 file system size is 1 EB (exabyte). 1 EB = 1024 PB (petabyte). 1 PB = 1024 TB (terabyte).
Maximum sub directories	Directory can contain a maximum of 32,000 subdirectories	Directory can contain a maximum of 64,000 subdirectories
Other Features	The main benefit of ext3 is that it allows journaling. Journaling has a dedicated area in the file system, where all the changes are tracked. When the system crashes, the possibility of file system corruption is less because of journaling. There are three types of journaling available in ext3 file system. Journal – Metadata and content are saved in the journal. Ordered – Only metadata is saved in the journal. Metadata are journaled only after writing the content to disk. This is the default. Writeback – Only metadata is saved in the journal. Metadata might be journaled either before or after the content is written to the disk.	In ext4, it supports journaling and also has the option of turning the journaling feature “off”. Several other new features are introduced in ext4: multi block allocation, delayed allocation, Journal checksum, fast fsck and etc. All you need to know is that these new features have improved the performance and reliability of the file system when compared to ext3. Supports huge individual file size and overall file system size.
	You convert ext2 to ext3 without having any data loss	You can also mount an existing ext3 fs as ext4 fs (without having to upgrade it).