Replicating base server -> target server (Digital Ocean/Fed/Centos)

[bruce]

Hi.

Kind of long ,but might be useful/helpful to others.. Feel free to
comment as you see fit!

I'm using this as a step/guide to recreate/replicate a smaller
droplet/vm from a larger vm within Digital Ocean.


The goal:
 Replicate/reproduce the users/processes/directories/files
  from a base server to a target server.
 To be able to then use the target server in place of the base/initial
  server
 The base server acts as a nfsClient server to a nfsServer

 Doing this to create a smaller vm/droplet from Digital Ocean (DO), and need
  to "replicate" a larger droplet/vm.
  DO doesn't provide a way to accomplish this!

::::---
This process doesn't deal/handle any iptable/firewall/selinux/dns
 issues....
 -need to figure out how to deal with these....

Process:
-analyse the base server to get all the existing users
-analyse the base server to get all the existing/running system processes
-analyse the base server to get all the existing/running 3rd party processes
-analyse the base server to get all the existing dirs/files
-analyse the base server to get all the existing sshkey data/files
-analyse the base server to get all the installed rpm/packages


-create process on the target to generate the user/group/passwd
 for the users on the base server
-create process on the target to generate the ssh key for the users
 to replicate the base users/sshkeys
-create process to copy all the dir/files from the base to the target,
 excluding a limited subset of dirs..
-create process to install on the target, all the installed packages from
 the base
-


Steps::
-on the target disable selinux for simplicity
 vi /etc/sysconfig/selinux
  set >>  SELINUX=disabled <<


-Create the list of users
-On the target, generate the users/passwd/groups. It appears this
 could/should be doable by copying the requisite files from the
 base->target with the associated perms..
 Decided to do it manually to ensure it matches..

-the users are/were:
 root/root_tmp/test_user

 -use useradd/passwd to generate the same users/passwd/group as well as
  the same uid/gid for the userID/groupID to match the base server

  --match the user/passwd/uid/gid so the target matches the base
  --if required, mod the uid/gid
    usermod -u xx test_user
    groupmod -g xx test_user

  --at the same time, change/match across the entire dir for any files
    to handle user/group owner (do this for all users)
    find / -group 500 -exec chgrp -h test_user {} \;
    find / -user 500 -exec chown -h test_user {} \;

-change to given user (root/root_tmp/test_user)
 set up the ssh keymkdir ~/.ssh
 chmod 700 ~/.ssh
 echo '' > ~/.ssh/authorized_keys
 chmod 600 ~/.ssh/authorized_keys

 chmod 755 ~ ~/.ssh

 update/cpy in the "authorized_keys" file the pub key from the
 base/user  "authorized_keys" file

at this point, the users are set, and the ssh keys are set..

----------------------------------
Copy the dirs/files....

The process runs on the base, copying/rsync from the base-> target handling
 the entire disk from the "/" top.. on down..

 The excluded list follows as well as as the rsync cmd..

The file "/etc/skipdirs.rsync" contains:

        /proc/*
        /sys/*
        /dev/*
        /media/*
        /var/log/*
        /var/log/journal/*
        **/.cache/google-chrome/***
        **/.ccache/***
        /BACKUPS/*
        /run/media/*
        /var/lib/nfs/*
        /usr/src/kernels/*
        /root/.cache/*
        /swapfile
        /bin/*

rsync --progress  -avAI --exclude-from=/etc/skipdirs.rsync / root@11.22.33.44:/
 (shout out to Rick!!)

I intentionally wanted to see all the files xfered as I ran/run the cmd..
-the cmd is run as "root" on the base, to ensure the process has
 complete access to all dirs/files..
-the "root" user on the target has access to top level as well..

RPM packages..
-To further ensure the target will mtach the base,
 on the base as "root" run
  rpm -qa > rpmlist.dat
  rsync --progress  -avAI /rpmlist.dat root@11.22.33.44:/

and on the target as root
  cat /rpmlist.dat | xargs yum -y install

followed by
 yum update

--end result is that all the packages on the target should match
  the base

::given that this is going to be a "client" to the nfsServer..
-make sure the nfs client utils are available.. they should already
 be but check anyway..

on the target as root

install any/all additional packages on the target::
yum install nfs-utils nfs-utils-lib

as root
sudo cat << EOF > /etc/yum.repos.d/google-chrome.repo
[google-chrome]
name=google-chrome - \$basearch
baseurl=http://dl.google.com/linux/chrome/rpm/stable/\$basearch
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
EOF

yum install -y  gcc xorg-x11-server-Xvfb google-chrome-stable
yum install xvfb x11-xkb-utils
yum install xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic

yum install vi
pip install pyvirtualdisplay
python -m pip install --upgrade pip setuptools wheel

easy_install simplejson
yum install libxml2-python

yum install epel-release
pip install -U selenium
pip install pyvirtualdisplay
yum install Xvfb libXfont Xorg


-finally.. install the "stuff" for the selenium/firefox/chrome testing..

wget
https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodri...
tar -x geckodriver -zf geckodriver-v0.19.1-linux64.tar.gz -O >
/usr/local/bin/geckodriver
chmod +x /usr/local/bin/geckodriver
rm geckodriver-v0.19.1-linux64.tar.gz
ln -s /path/to/file /path/to/symlink
ln -s /usr/local/bin/geckodriver /usr/bin/geckodriver

## Chromedriver
wget https://chromedriver.storage.googleapis.com/2.35/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo chmod +x chromedriver
sudo mv chromedriver /usr/local/bin/
rm chromedriver_linux64.zip
ln -s /usr/local/bin/chromedriver /usr/bin/chromedriver



finally.... on the base....
as root..
 vi /etc/ssh/sshd_config
#PermitRootLogin yes
PermitRootLogin no


ok....


if you can see something else that can be done that I might have
missed.. or that would be better.. shout it out.

At some point.. others on DO will be looking for something like this.

Ideally, it would be cool/good to have this in a kind of chef/puppet cookbook.

But that's above my pay grade for now!

thanks

[InvalidPath]

On Sun, Feb 11, 2018 at 1:51 PM, bruce <badouglas(a)gmail.com&gt; wrote:

>  Hi.
>
>  Kind of long ,but might be useful/helpful to others.. Feel free to
>  comment as you see fit!
>
>  I'm using this as a step/guide to recreate/replicate a smaller
>  droplet/vm from a larger vm within Digital Ocean.
>
>
>  The goal:
>   Replicate/reproduce the users/processes/directories/files
>    from a base server to a target server.
>   To be able to then use the target server in place of the base/initial
>    server
>   The base server acts as a nfsClient server to a nfsServer
>
>   Doing this to create a smaller vm/droplet from Digital Ocean (DO), and
>  need
>    to "replicate" a larger droplet/vm.
>    DO doesn't provide a way to accomplish this!
>
>  ::::---
>  This process doesn't deal/handle any iptable/firewall/selinux/dns
>   issues....
>   -need to figure out how to deal with these....
>
>  Process:
>  -analyse the base server to get all the existing users
>  -analyse the base server to get all the existing/running system processes
>  -analyse the base server to get all the existing/running 3rd party
>  processes
>  -analyse the base server to get all the existing dirs/files
>  -analyse the base server to get all the existing sshkey data/files
>  -analyse the base server to get all the installed rpm/packages
>
>
>  -create process on the target to generate the user/group/passwd
>   for the users on the base server
>  -create process on the target to generate the ssh key for the users
>   to replicate the base users/sshkeys
>  -create process to copy all the dir/files from the base to the target,
>   excluding a limited subset of dirs..
>  -create process to install on the target, all the installed packages from
>   the base
>  -
>
>
>  Steps::
>  -on the target disable selinux for simplicity
>   vi /etc/sysconfig/selinux
>    set >>  SELINUX=disabled <<
>
>
>  -Create the list of users
>  -On the target, generate the users/passwd/groups. It appears this
>   could/should be doable by copying the requisite files from the
>   base->target with the associated perms..
>   Decided to do it manually to ensure it matches..
>
>  -the users are/were:
>   root/root_tmp/test_user
>
>   -use useradd/passwd to generate the same users/passwd/group as well as
>    the same uid/gid for the userID/groupID to match the base server
>
>    --match the user/passwd/uid/gid so the target matches the base
>    --if required, mod the uid/gid
>      usermod -u xx test_user
>      groupmod -g xx test_user
>
>    --at the same time, change/match across the entire dir for any files
>      to handle user/group owner (do this for all users)
>      find / -group 500 -exec chgrp -h test_user {} \;
>      find / -user 500 -exec chown -h test_user {} \;
>
>  -change to given user (root/root_tmp/test_user)
>   set up the ssh keymkdir ~/.ssh
>   chmod 700 ~/.ssh
>   echo '' > ~/.ssh/authorized_keys
>   chmod 600 ~/.ssh/authorized_keys
>
>   chmod 755 ~ ~/.ssh
>
>   update/cpy in the "authorized_keys" file the pub key from the
>   base/user  "authorized_keys" file
>
>  at this point, the users are set, and the ssh keys are set..
>
>  ----------------------------------
>  Copy the dirs/files....
>
>  The process runs on the base, copying/rsync from the base-> target handling
>   the entire disk from the "/" top.. on down..
>
>   The excluded list follows as well as as the rsync cmd..
>
>  The file "/etc/skipdirs.rsync" contains:
>
>          /proc/*
>          /sys/*
>          /dev/*
>          /media/*
>          /var/log/*
>          /var/log/journal/*
>          **/.cache/google-chrome/***
>          **/.ccache/***
>          /BACKUPS/*
>          /run/media/*
>          /var/lib/nfs/*
>          /usr/src/kernels/*
>          /root/.cache/*
>          /swapfile
>          /bin/*
>
>  rsync --progress  -avAI --exclude-from=/etc/skipdirs.rsync /
>  root@11.22.33.44:/
>   (shout out to Rick!!)
>
>  I intentionally wanted to see all the files xfered as I ran/run the cmd..
>  -the cmd is run as "root" on the base, to ensure the process has
>   complete access to all dirs/files..
>  -the "root" user on the target has access to top level as well..
>
>  RPM packages..
>  -To further ensure the target will mtach the base,
>   on the base as "root" run
>    rpm -qa > rpmlist.dat
>    rsync --progress  -avAI /rpmlist.dat root@11.22.33.44:/
>
>  and on the target as root
>    cat /rpmlist.dat | xargs yum -y install
>
>  followed by
>   yum update
>
>  --end result is that all the packages on the target should match
>    the base
>
>  ::given that this is going to be a "client" to the nfsServer..
>  -make sure the nfs client utils are available.. they should already
>   be but check anyway..
>
>  on the target as root
>
>  install any/all additional packages on the target::
>  yum install nfs-utils nfs-utils-lib
>
>  as root
>  sudo cat << EOF > /etc/yum.repos.d/google-chrome.repo
>  [google-chrome]
>  name=google-chrome - \$basearch
>  baseurl=http://dl.google.com/linux/chrome/rpm/stable/\$basearch
>  enabled=1
>  gpgcheck=1
>  gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
>  EOF
>
>  yum install -y  gcc xorg-x11-server-Xvfb google-chrome-stable
>  yum install xvfb x11-xkb-utils
>  yum install xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic
>
>  yum install vi
>  pip install pyvirtualdisplay
>  python -m pip install --upgrade pip setuptools wheel
>
>  easy_install simplejson
>  yum install libxml2-python
>
>  yum install epel-release
>  pip install -U selenium
>  pip install pyvirtualdisplay
>  yum install Xvfb libXfont Xorg
>
>
>  -finally.. install the "stuff" for the selenium/firefox/chrome testing..
>
>  wget https://github.com/mozilla/geckodriver/releases/download/
>  v0.19.1/geckodriver-v0.19.1-linux64.tar.gz
>  tar -x geckodriver -zf geckodriver-v0.19.1-linux64.tar.gz -O >
>  /usr/local/bin/geckodriver
>  chmod +x /usr/local/bin/geckodriver
>  rm geckodriver-v0.19.1-linux64.tar.gz
>  ln -s /path/to/file /path/to/symlink
>  ln -s /usr/local/bin/geckodriver /usr/bin/geckodriver
>
>  ## Chromedriver
>  wget https://chromedriver.storage.googleapis.com/2.35/
>  chromedriver_linux64.zip
>  unzip chromedriver_linux64.zip
>  sudo chmod +x chromedriver
>  sudo mv chromedriver /usr/local/bin/
>  rm chromedriver_linux64.zip
>  ln -s /usr/local/bin/chromedriver /usr/bin/chromedriver
>
>
>
>  finally.... on the base....
>  as root..
>   vi /etc/ssh/sshd_config
>  #PermitRootLogin yes
>  PermitRootLogin no
>
>
>  ok....
>
>
>  if you can see something else that can be done that I might have
>  missed.. or that would be better.. shout it out.
>
>  At some point.. others on DO will be looking for something like this.
>
>  Ideally, it would be cool/good to have this in a kind of chef/puppet
>  cookbook.
>
>  But that's above my pay grade for now!
>
>  thanks
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  
I am keeping this for posterity.. thanks for such a well rounded
step-by-step.

[bruce]

Umm.... uh oh!

In testing out the process for replicating/copying a base server to a
smaller target server (droplet/vm) for Digital Ocean (DO), I've hit a
major snag.

It appears that copying "some" dirs/files screws up the target vm....
So, I'm trying to work my way through to see just what can be copied
without screwing up the target vm/droplet.

The files/dirs that I exclude are in the "skiprsync.dat" file. At the
moment, I'm doing an iterative process to see how much/what the skip
file should contain.

The process for now...
1- create target droplet/vm
2- modify the skipdirs.rsync file with the files/dirs to skip during the rsync
3- run the rsync to xfer files from the base to the target droplet..
4- reboot the target droplet to see if it can still be accessed/ssh
5- if not, repeat step 1...

thoughts/comments are appreciated.

skipdirs.rsync::
/proc/*
/sys/*
/dev/*
/media/*
/var/log/*
/var/log/journal/*
**/.cache/google-chrome/***
**/.ccache/***
/BACKUPS/*
/run/media/*
/var/lib/nfs/*
/usr/src/kernels/*
/root/.cache/*
/swapfile
/bin/*
/root
/home/*
/home/crawl_user
/home/root1
/boot/*
/mnt/*
/var/*
/usr/*
/run/*

/etc/*
/etc/fstab
/etc/exports
/etc/mtab
/etc/passwd
/etc/group
/etc/dbus-1/*
/etc/grub.d/*
/etc/iproute2/*
/etc/kernel/*
/etc/selinux/*
/etc/sysconfig/*
/etc/systemd/*
/etc/tuned/*
/etc/udev/*
/etc/wpa_suplicant/*
/etc/xdg/*

On Sun, Feb 11, 2018 at 4:35 PM, InvalidPath <invalid.path(a)gmail.com&gt; wrote:
>
>
> On Sun, Feb 11, 2018 at 1:51 PM, bruce <badouglas(a)gmail.com&gt; wrote:
>>
>> Hi.
>>
>> Kind of long ,but might be useful/helpful to others.. Feel free to
>> comment as you see fit!
>>
>> I'm using this as a step/guide to recreate/replicate a smaller
>> droplet/vm from a larger vm within Digital Ocean.
>>
>>
>> The goal:
>>  Replicate/reproduce the users/processes/directories/files
>>   from a base server to a target server.
>>  To be able to then use the target server in place of the base/initial
>>   server
>>  The base server acts as a nfsClient server to a nfsServer
>>
>>  Doing this to create a smaller vm/droplet from Digital Ocean (DO), and
>> need
>>   to "replicate" a larger droplet/vm.
>>   DO doesn't provide a way to accomplish this!
>>
>> ::::---
>> This process doesn't deal/handle any iptable/firewall/selinux/dns
>>  issues....
>>  -need to figure out how to deal with these....
>>
>> Process:
>> -analyse the base server to get all the existing users
>> -analyse the base server to get all the existing/running system processes
>> -analyse the base server to get all the existing/running 3rd party
>> processes
>> -analyse the base server to get all the existing dirs/files
>> -analyse the base server to get all the existing sshkey data/files
>> -analyse the base server to get all the installed rpm/packages
>>
>>
>> -create process on the target to generate the user/group/passwd
>>  for the users on the base server
>> -create process on the target to generate the ssh key for the users
>>  to replicate the base users/sshkeys
>> -create process to copy all the dir/files from the base to the target,
>>  excluding a limited subset of dirs..
>> -create process to install on the target, all the installed packages from
>>  the base
>> -
>>
>>
>> Steps::
>> -on the target disable selinux for simplicity
>>  vi /etc/sysconfig/selinux
>>   set >>  SELINUX=disabled <<
>>
>>
>> -Create the list of users
>> -On the target, generate the users/passwd/groups. It appears this
>>  could/should be doable by copying the requisite files from the
>>  base->target with the associated perms..
>>  Decided to do it manually to ensure it matches..
>>
>> -the users are/were:
>>  root/root_tmp/test_user
>>
>>  -use useradd/passwd to generate the same users/passwd/group as well as
>>   the same uid/gid for the userID/groupID to match the base server
>>
>>   --match the user/passwd/uid/gid so the target matches the base
>>   --if required, mod the uid/gid
>>     usermod -u xx test_user
>>     groupmod -g xx test_user
>>
>>   --at the same time, change/match across the entire dir for any files
>>     to handle user/group owner (do this for all users)
>>     find / -group 500 -exec chgrp -h test_user {} \;
>>     find / -user 500 -exec chown -h test_user {} \;
>>
>> -change to given user (root/root_tmp/test_user)
>>  set up the ssh keymkdir ~/.ssh
>>  chmod 700 ~/.ssh
>>  echo '' > ~/.ssh/authorized_keys
>>  chmod 600 ~/.ssh/authorized_keys
>>
>>  chmod 755 ~ ~/.ssh
>>
>>  update/cpy in the "authorized_keys" file the pub key from the
>>  base/user  "authorized_keys" file
>>
>> at this point, the users are set, and the ssh keys are set..
>>
>> ----------------------------------
>> Copy the dirs/files....
>>
>> The process runs on the base, copying/rsync from the base-> target
>> handling
>>  the entire disk from the "/" top.. on down..
>>
>>  The excluded list follows as well as as the rsync cmd..
>>
>> The file "/etc/skipdirs.rsync" contains:
>>
>>         /proc/*
>>         /sys/*
>>         /dev/*
>>         /media/*
>>         /var/log/*
>>         /var/log/journal/*
>>         **/.cache/google-chrome/***
>>         **/.ccache/***
>>         /BACKUPS/*
>>         /run/media/*
>>         /var/lib/nfs/*
>>         /usr/src/kernels/*
>>         /root/.cache/*
>>         /swapfile
>>         /bin/*
>>
>> rsync --progress  -avAI --exclude-from=/etc/skipdirs.rsync /
>> root@11.22.33.44:/
>>  (shout out to Rick!!)
>>
>> I intentionally wanted to see all the files xfered as I ran/run the cmd..
>> -the cmd is run as "root" on the base, to ensure the process has
>>  complete access to all dirs/files..
>> -the "root" user on the target has access to top level as well..
>>
>> RPM packages..
>> -To further ensure the target will mtach the base,
>>  on the base as "root" run
>>   rpm -qa > rpmlist.dat
>>   rsync --progress  -avAI /rpmlist.dat root@11.22.33.44:/
>>
>> and on the target as root
>>   cat /rpmlist.dat | xargs yum -y install
>>
>> followed by
>>  yum update
>>
>> --end result is that all the packages on the target should match
>>   the base
>>
>> ::given that this is going to be a "client" to the nfsServer..
>> -make sure the nfs client utils are available.. they should already
>>  be but check anyway..
>>
>> on the target as root
>>
>> install any/all additional packages on the target::
>> yum install nfs-utils nfs-utils-lib
>>
>> as root
>> sudo cat << EOF > /etc/yum.repos.d/google-chrome.repo
>> [google-chrome]
>> name=google-chrome - \$basearch
>> baseurl=http://dl.google.com/linux/chrome/rpm/stable/\$basearch
>> enabled=1
>> gpgcheck=1
>> gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
>> EOF
>>
>> yum install -y  gcc xorg-x11-server-Xvfb google-chrome-stable
>> yum install xvfb x11-xkb-utils
>> yum install xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic
>>
>> yum install vi
>> pip install pyvirtualdisplay
>> python -m pip install --upgrade pip setuptools wheel
>>
>> easy_install simplejson
>> yum install libxml2-python
>>
>> yum install epel-release
>> pip install -U selenium
>> pip install pyvirtualdisplay
>> yum install Xvfb libXfont Xorg
>>
>>
>> -finally.. install the "stuff" for the selenium/firefox/chrome
testing..
>>
>> wget
>>
https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodri...
>> tar -x geckodriver -zf geckodriver-v0.19.1-linux64.tar.gz -O >
>> /usr/local/bin/geckodriver
>> chmod +x /usr/local/bin/geckodriver
>> rm geckodriver-v0.19.1-linux64.tar.gz
>> ln -s /path/to/file /path/to/symlink
>> ln -s /usr/local/bin/geckodriver /usr/bin/geckodriver
>>
>> ## Chromedriver
>> wget
>> https://chromedriver.storage.googleapis.com/2.35/chromedriver_linux64.zip
>> unzip chromedriver_linux64.zip
>> sudo chmod +x chromedriver
>> sudo mv chromedriver /usr/local/bin/
>> rm chromedriver_linux64.zip
>> ln -s /usr/local/bin/chromedriver /usr/bin/chromedriver
>>
>>
>>
>> finally.... on the base....
>> as root..
>>  vi /etc/ssh/sshd_config
>> #PermitRootLogin yes
>> PermitRootLogin no
>>
>>
>> ok....
>>
>>
>> if you can see something else that can be done that I might have
>> missed.. or that would be better.. shout it out.
>>
>> At some point.. others on DO will be looking for something like this.
>>
>> Ideally, it would be cool/good to have this in a kind of chef/puppet
>> cookbook.
>>
>> But that's above my pay grade for now!
>>
>> thanks
>> _______________________________________________
>> users mailing list -- users(a)lists.fedoraproject.org
>> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>
>
> I am keeping this for posterity.. thanks for such a well rounded
> step-by-step.
>
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>