Xiaopei's DokuWiki

These are the good times in your life,
so put on a smile and it'll be alright

User Tools

Site Tools


it:linux:backup

备份

使用 unison 和 incron 同步两个服务器的目录

# 互相添加 server A 和 server B 的 user www 的 authorized_keys
 
## server A user www
# 编写 unison 同步脚本
$ vi /home/www/sync.sh
#!/bin/bash
# set paths / dirs
_paths="/tmp/test2"
 
# binary file name
_unison=/usr/bin/unison
 
# server names
# sync server1.cyberciti.com with rest of the server in cluster
_rserver="www@10.144.163.143"
 
# sync it
for r in ${_rserver}
do
	for p in ${_paths}
	do
         	${_unison} -batch -silent -contactquietly "${p}"  "ssh://${r}/${p}"
	done
done
 
 
# 编写 incron 配置,监控文件修改
$ incrontab
/tmp/test2 IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /home/www/sync.sh
 
 
## server B user www
# 编写让 server A 运行同步的脚本
$ vi /home/www/remote_sync.sh
#!/bin/bash
 
# 无论是哪边文件修改,统一从 10.144.84.152 发起同步
ssh www@10.144.84.152 /home/www/sync.sh
 
 
# 编写 incron 配置,监控文件修改
# 但目前会造成 unison 不必要的多运行一次
/tmp/test2 IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /home/www/remote_sync.sh

rsync

看看serverfault上的tag的问题是个好习惯 http://serverfault.com/questions/tagged/rsync?sort=votes

TO READ

使用推模式的备份逻辑

客户端 cron 执行 rsync 脚本, 推送备份到服务器端.

  1. /etc/lims2/rsync
    #!/bin/sh
    rsync -azv --delete --rsh=ssh /backups/ rsync@brandy.genee.cc:/home/rsync/us/
    # -a, archive mode, 适于备份
    # -z, compress
    # -v, verbose
    # --delete, 在服务器端删除客户端已删除的文件. archive mode 中不包含 --delete
    # --rsh=ssh, 使用 ssh 
    # TODO log
  2. 客户端在 root 下 ssh-keygen, 并 ssh-copy-id 到服务器端的备份专用用户 rsync 下
  3. rsync 需能增删文件, 应为普通用户, 即 shell 不能为 /bin/false, 但为安全考虑, 可修改 rsync 的 authorized_keys, 设置 no-pty,no-port-forwarding, 以防用此帐号登录或端口转发
  4. rsync 用户亦可和 autossh 用户合并
  5. crontab
    x y * * * /etc/lims2/rsync 2>&1
    # x/y 分别代表时/分, 应为某一时段内的随机值, 以防服务器端瞬时负载过大

SSH

若使用公钥 (pubkey, public key) 做 rsync 验证, 则 target 需对 source 的 pubkey 做限制:

  1. 编写 source 期望执行命令的检查脚本:
    validate-rsync
    #!/bin/sh
     
    case "$SSH_ORIGINAL_COMMAND" in
      *\&*)
        echo "Rejected"
        ;;
      *\(*)
        echo "Rejected"
        ;;
      *\{*)
        echo "Rejected"
        ;;
      *\;*)
        echo "Rejected"
        ;;
      *\<*)
        echo "Rejected"
        ;;
      *\`*)
        echo "Rejected"
        ;;
      *\|*)
        echo "Rejected"
        ;;
      rsync\ --server*)
        $SSH_ORIGINAL_COMMAND
        ;;
      *)
        echo "Rejected"
        ;;
    esac 
  2. /etc/ssh/sshd_config
    # 允许 root 以 key 登陆, 且登陆后只能运行 key 对应的 command
    PermitRootLogin forced-commands-only
  3. 增加 source 的 public, 并做限制
    authorized_keys
    command="/path/to/validate-rsync" ssh-rsa AAAAB3NzaC1...

参考

使用拉模式的备份逻辑

  1. 本地生成 ssh 密钥对
  2. 拷贝公钥到远程
  3. 远程 sshd 安全设置
    1. 修改 sshd_config, PermitRootLogin no
    2. 在公钥上添加限制, 如 from=“10.1.1.1”,command=“/home/remoteuser/cron/validate-rsync” ssh-dss AAAAB3Nza…
    3. 创建 validate-rsync, 并设置 x 权限
      #!/bin/bash
       
      case "$SSH_ORIGINAL_COMMAND" in
          *\&*)
          echo "Rejected"
          ;;
          *\(*)
      	echo "Rejected"
      	;;
          *\{*)
      	echo "Rejected"
      	;;
          *\;*)
      	echo "Rejected"
      	;;
          *\<*)
      	echo "Rejected"
      	;;
          *\`*)
      	echo "Rejected"
      	;;
          *\|*)
          echo "Rejected"
          ;;
          rsync\ --server*)
      	$SSH_ORIGINAL_COMMAND
      	;;
          *)
      	echo "Rejected"
      	;;
      esac 
  4. 测试命令
    $ rsync -avvvz -e "ssh -i /home/thisuser/cron/thishost-rsync-key" remoteuser@remotehost:/remote/dir /this/dir/
    # The more v's, the more verbose the output.
  5. 设置 cron
    #!/bin/sh
     
    RSYNC=/usr/bin/rsync
    SSH=/usr/bin/ssh
    KEY=/home/thisuser/cron/thishost-rsync-key
    RUSER=remoteuser
    RHOST=remotehost
    RPATH=/remote/dir
    LPATH=/this/dir/
     
    $RSYNC -az -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH $LPATH 
     
    # cron 规则可设置如下
    # 0 5 * * * /home/thisuser/cron/rsync-remotehost-backups 

参考

即使用 --archive ownership 还是没有了?

这种情况会发生在 rsync –rsh=ssh $LOCAL $REMOTE 中. 原因是 $REMOTE 无权限 chown, 可使用 rsync –rsh=ssh –rsync-path=“sudo rsync” $LOCAL $REMOTE 让 $REMOTE 用 sudo 解决.

Note of Linux Administration Handbook

rsync is a bit like a souped-up version of scp that is scrupulous about preserving links, modification times, and permissions.

As an example, the command:

rsync -gopt --password-file=/etc/rsync.pwd /etc/passwd lollipop::sysfiles

transfers the /etc/passwd file to the machine lollipop. The -gopt options preserve the permissions, ownerships, and modification times of the file. The double colon in lollipop::sysfiles makes rsync contact the remote rsync directly on port 873 instead of using ssh. The password stored in /etc/rsync.pwd authenticates the connection.

Although the password is encrypted for transmission across the network, the transferred files are not. If you use ssh as the transport:

rsync -gopt -e ssh /etc/passwd /etc/shadow lollipop:/etc

– note the single colon, the connection will be encrypted, but sshd will have to be configured not to require a password. Name your poison!

Linux rsync packages usually include a xinetd configuration for rsync. However, you must edit /etc/xinetd.d/rsync and change disable = yes to disable = no to actually enable the server.

Once you have enabled rsync, you need to set up a couple of config files to tell the rsync server how to behave. The main file is /etc/rsyncd.conf, which contains both global configuration parameters and a set of “modules,” each of which is a directory tree to export or import. A reasonable configuration for a module that you can push to (i.e., that will accept incoming file transfers initiated by the connecting client) looks something like this:

# sysfiles is just an arbitrary title for the particular module.
[sysfiles]
# This is the path you allow files to be pushed to. It could be just /.
path = /etc
# This is the file specifying the user/password pair to authenticate the module
secrets file = /etc/rsyncd.secrets
# Can be read only if you are pulling files
read only = false
# UID and GID under which the transfer will be done
uid = root
gid = root
# List of hosts that are allowed to connect
hosts allow = distribution_master_hostname

Whatever system you use, be careful not to overload your data server. If a lot of machines on the network try to access the server simultaneously (e.g., if everyone runs an update out of cron at the same time), you can cause an inadvertent denial of service attack. Large sites should keep this problem in mind and allow for staggering or randomization. A simple way to do this is to wrap cron jobs in a Perl script such as this:

#!/usr/bin/perl
sleep rand() * 600; # sleep between 0 and 600 seconds (i.e., 10 minutes)
system(command_to_copy_files_down);
it/linux/backup.txt · Last modified: 2015/01/21 15:46 by admin