History

« Previous - Version 25/33 (diff) - Next » - Current version
kuko, 17 March 2010 14:41

Backup service Deployment And Installation Guide

Preparation

  • Download the image here
  • Prepare the virtual machine with the following parameters:
    • Volume size: 3GB
    • RAM: 256 MB
    • Name: backup
  ./createvm backup 3G Backup-1.1_1.tgz 128 192.168.69.23 xenbr0
  • Boot the virtual machine:
  xm create -c /etc/xen/auto/backup
  • Login as root using the password passwd.root and change the default password.

Machine configuration

Two configuration options are available:
  • Centralised configuration: if the machine is going to be integrated in a ModularIT management structure.
  • Local configuration: if the machine is not going to be integrated in a ModularIT management structure.

Centralized configuration

On the Puppet server

  • The global parameters of the backup service are defined in the client group of the machine:
  ## Backup service variables
  # IP of the backup server
  $backup_ip = "192.168.69.23"
  • Create the entry of the ModularIT machine being installed:
  node "devel.backup" inherits "devel" {
        # First installation?
        $bootstrap = yes

        $modularit_name = "devel.backup" 
        $comment = "Development Backup" 
        include modularit 
  }

Client configuration (The machine being installed)

  • puppetd is restarted in debug mode to ensure that it updates correctly:
  /etc/init.d/puppet stop
  puppetd --debug --no-daemonize --runinterval 60 --fqdn NOMBRE_MODULARIT --server PUPPET_MODULARIT
  • The parameters NOMBRE_MODULARIT and PUPPET_MODULARIT will be provided by the service manager (register with the mailing list if you have questions regarding this).
  • The puppet is started as a service after it has completed several cycles without generating errors:
  /etc/init.d/puppet restart
  • PIFIA should be executed to ensure that notifications to Nagios are functioning correctly.
      /var/lib/pica/bin/scheduler Emergency
  /var/lib/pica/bin/scheduler Warning
  • Ensure that Nagios is not generating alerts that read OK. At this point critical AIDE alerts are expected.

Backups with Drobo

The Drobo creates a dynamic volume and needs to track space usage to adapt the redundancy level.

Initialize the Drobo volume: (WARNING: This steps delete ALL data on the drobo)

  drobom format ext3

You can also get the same result manually following these steps:

  • Initialize labels
      parted /dev/sdb mklabel gpt
  • Create partition
      parted /dev/sdb mkpart pri ext3 0 100%
  • Check partition table
      parted /dev/sdb print
  • Create ext3 filesystem
      mke2fs -j -i 262144 -L Drobo01 -m 0 -O sparse_super,^resize_inode /dev/sdb1

Encrypting data on the Drobo

As we said previously, in order to allow the Drobo to read the used storage space, we need to use filesystem encryption instead of block device encryption. For this purpose we use fuse-encs

  • Install fuse-encfs
      yum install fuse-encfs
  • Create a random password for encfs
      dd if=/dev/urandom count=1 | md5sum | cut -f1 -d" " > /etc/encfs-passwd
  • Create the directories:
      mkdir /dat/bck/.crypted
  mkdir /dat/bck/data
  • Create the encfs:
      cat /etc/encfs-passwd | encfs -S /dat/bck/.crypted /dat/bck/data

From now on, everything we write on /dat/bck/data will be encrypted and stored on /dat/bck/.crypted

Multiple encryption keys management

If you manage encrypted backups for many clients, you may need to use different random encryption keys. Managing all that keys can be a pain, so you may store the encryption password on the unencrypted portion of the encfs filesystes, and make sure you encrypt it using a strong key (i.e. using gpg). This way you can have a "master" key to access all client's passwords, and they won't have access to other client's backups.

  gpg -ca -o /dat/bck/encfs-passwd.asc /etc/encfs-passwd

Backups to USB disks in RAID1

Machine specific configuration

The RAID management is done from the machine (dom0) and exported as a virtual disk. Create the RAID configuration on the machine:

  • To check the disks:
fdisk -l
  • Assume the the backup disks are sdb and sdc.
  • Mark the partitions with type fd (Linux RAID):
  sfdisk --id /dev/sdb 1 fd
  sfdisk --id /dev/sdc 1 fd
  • Create the array:
  mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
  • Check it has been created:
  cat /proc/mdstat
  • A message similar to: md0 : should be returned.
_active raid1 sdc1[1] sdb1[0]_

this indicates that the array has been synchronized.

  • Configure /etc/mdadm.conf to ensure that the correct arrays are located:
  echo "DEVICE /dev/sd* /dev/hd*" > /etc/mdadm.conf
  mdadm -Q --examine --brief /dev/sd* >> /etc/mdadm.conf
  • The archive is not identified via its elements (sdb1 and sdc1) but by tag that describes it. This means that if the name of the drive changes (which is common for USB drives) the mdadm is still able to locate it.
  • Export ''/dev/md0'' to the virtual machine.
    • Add: phy:/dev/md0,hdc1,w to the disk configuration so that the array will be identified in the backup virtual machine as the partition: /dev/hdc1.
  • Reboot the backup virtual machine to check the configuration changes.

Follow the relevant section if data encryption is used.

  • Format the backup disk of the virtual machine:
  mkfs -t ext3 /dev/hdc1
  • In case the array is still synchronising.
 cat /proc/mdstat

the synchronization can take several hours (more than 4 for a 500GB usb disk)usb).

  • Uncomment (remove the # symbol) the disk in /etc/fstab and execute.
 mount -a

The backup machine has a samba that shares the backup directory. The default machine name for the BACKUP workgroup is BACKUP. By default only the backup user has access to the samba folder. Establish the samba password for the backup user:

  smbpasswd backup

Backup volume encryption

To encrypt the backup data:

  • Check the disk bocks and fill them with random data. Note: This process will overwrite the disk and can take in excess of 8 hours to complete on large capacity drives.
  badblocks -c 10240 -s -w -t random -v /dev/hdc1
  • Fill the backup volume with random data (this is optional if the previous step has been completed):
      dd if=/dev/urandom of=/dev/hdc1 bs=1M
  • Create the password/key. Note: It is not possible to recover or access stored data if this key is lost.
  dd if=/dev/random of=/etc/cryptsetup-key bs=1 count=256
  • Associate the device dm-crypt ,afterwards the encrypted device will be /dev/mapper/backup.
  cryptsetup create --key-file /etc/cryptsetup-key backup /dev/hdc1
  • Check the status of the encrypted data:
  cryptsetup status backup
  • Create the device filesystem. Do not use the journal option to avoid encrypting twice; once when the data is written, and once again when the journal is written:
  mkfs.ext3 -m 1 -O dir_index,filetype,sparse_super /dev/mapper/backup
  • Mount using:
  mount -t ext3 /dev/mapper/backup /dat/bck
  • Unmount using:
  unmount /dat/bck
  cryptsetup remove backup
  • The backup volume is not automatically mounted during the machine boot process if the backup volume is encrypted. To enable this the cryptbck script must be executed:
    • cryptbck mount: Make the encrypted volume active and mount it in /dat/bck
    • cryptbck umount: Unmount and deactivate the encrypted volume
    • cryptbck status: Shows the state of the encrypted volume

Creation and distribution of the backup keys

If neither Puppet nor PICA are used to manage the machines, a pair of ssh keys for the backup must be generated and distributed to every machine that a backup is required for. This is not necessary if puppet or PICA are being used. This is managed centrally.

  • Create a new pair of keys:
  modularit-backup-genkey
  • distribute the key to the remote machine (ldap for example):
  modularit-backup-sendkey ldap

Service preparation

By default, this is the history defined with /etc/dirvish/master.conf :

  # Caducidad copias
  expire-default:  +30 days #Las copias diarias se guarda durante 15 dias

  expire-rule:
        #MIN    HR      DOM     MON     DOW     STRFTIME_FMT
        *     *     *     *           su     +3 months  #Se guarda 3 meses la copia de todos los domingos
        *     *     1-7   *           su     +1 year    #Se guarda una año la copia del primer domingo del mes.
        *     *     1-7   1,4,7,10    su     never      #Se guarda siempre la copia del primer domingo de los meses de Enero, Marzo, Julio y Octubre

The configuration and initialization of the backups is required. Ensure that the Nagios backup notifications are functioning correctly. The backup copies will be made on the dom0 (the physical machine), not the service machine (domU). To allow secure backup copies of the virtual machine, a volume snapshot is done and mounted in a directory so the backup is done in that directory. Once the backup is done, the snapshot is unmounted and erased.

It is important to define a vault for each virtual machine volume as LVM volumes are being used. For example, if there is a samba machine in the volume /dev/sys/samba and its home is in the volume /dev/sys/samba_home, a vault has to be defined for each volume.

  • Go to the /dat/bck directory.
  • Create a directory for each backup (vault) with a subdirectory called dirvish. For example, to back up 3 services: ldap, samba and mail.
mkdir -p ldap/dirvish samba/dirvish mail/dirvish
  • In each dirvish subdirectory a default.conf archive must be created using the following syntax:
  pre-client: cd /root; /var/lib/pica/bin/DirvishSnapshot start ldap /snapshot
  post-client: cd /root; /var/lib/pica/bin/DirvishSnapshot stop ldap /snapshot
  client: hierro
  tree: /snapshot
Using this archive Dirvish is performing the following:
  • Backup the /snapshot directory of the base machine.
  • Before the coy has begun, run the script called DirvishSnapshot, this creates a snapshot (start parameter) of the ldap_volume (1st parameter) and mounts it in the _/snapshot directory (2nd parameter). It is important that the directory is the same that is used later on in the option tree. As the copy is done individually, the same directory (/snapshot) can be used for each backup.
  • When the copy has completed, run the same script but with the stop parameter, this will unmount and erase the snapshot. Add the line cd /root;. This is an important step, otherwise it is not possible to dismount nor erase the directory as the script will be executed from the snapshot directory.
  • To ensure that the backup copies are performed correctly, the first execution in each vault must be forced (init the vault):
dirvish --vault NOMBRE_VAULT --init
  • Ensure that the copy is has completed correctly and that there is no error archive in the dated subdirectory.
  • Once the backup (vaults) have been configured and initialized, add them to the RunAll option of the archive /etc/dirvish/master.conf so they will be executed automatically in the future.
  • Check in Nagios tat the backup report has been correctly received and it is in OK state.
  • The generation of the backup report is forced:
 /var/lib/pica/alarms/Warning/DirvishChk-picacaller_

Initializing AIDE

Once everything is functioning correctly, initialize the AIDE databases as described in the AIDE.