Vagrant :: SSH Inter-Connectivity of Multi Virtual Machines

Vagrant is one of the best example of Infrastructure as a Code (IAC) tools (VM based). It works based on the declarative configuration file which consists of requirements like OS, Apps, Users and Files etc…

By using Vagrant, we can reduce mundane tasks of downloading OS images, manual installation of OS, APPs, User Configuration and security etc… It saves a lot of time and efforts for the developers, Admins and as well as Architects. Vagrant is a cross-platform product and its free to use the community edition. Vagrant also has its own cloud where a thousands of OS with Apps images are uploaded by the active contributors. For more info and to download this great product please visit here. Please install Oracle Virtualbox which is one of the basic requirements to run the vagrant VMs.

Multi Machines: Is a type of Vagrant configuration where multiple machines can be build using a single configuration file. This is best suited for development where multiple VMs are required whether its a homogeneous/heterogeneous configuration. For example in a typical webapp development, a separate Web, DB, Middleware, Proxy servers along with client VMs required to match the production class environment.

Below Vagrant configuration file is a use-case for setting up ‘Ansible Practice Lab’. 6 nodes are build by vagrant running on CentOS 7. This LAB environment is build for the purpose of learning Ansible hands-on workshop to try out all the features offered by Ansible to automate configuration management and infrastructure automation. Ansible package is installed on node 1 and rest of the nodes are managed by ansible workstation – node1.

Later YUM is installed and configured by downloading EPEL repository across all 6 nodes using the global shell script. Post yum configuration, basic packages like wget, curl and sshpass are installed.

Most important requirement for Ansible to work is to enable SSH key based authentication between all 6 nodes. For this to work, a shell script ssh is written and added to configuration file which will be executed by vagrant during the build process. An interface with Private IP is configured across all nodes which is used to for the nodes inter-connectivity via SSH.

Here is the vagrant multi machines configuration file along with custom scripts to install packages and setup SSH key based authentication between the nodes

# Vagrant configuration file for multi machines with inter connectivity via SSH key based authentication
numnodes=6
baseip="192.168.10"

# global script
$global = <<SCRIPT

# Allow SSH to accept SSH password authentication. Find and replace if the line is commented out
sudo sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config

# Add Google DNS to access internet. 
echo "nameserver 8.8.8.8" | sudo tee -a  /etc/resolv.conf 

# Download and install  Centos 7 EPEL package to configure the YUM repository
sudo rpm -ivh https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
# Update yum
sudo yum update -y
# Install wget curl and sshpass
sudo yum install wget curl sshpass -y

# Exclude node* from host checking
cat > ~/.ssh/config <<EOF
Host node*
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null
EOF

# Populate /etc/hosts with the IP and node names 
for x in {11..#{10+numnodes}}; do
  grep #{baseip}.${x} /etc/hosts &>/dev/null || {
      echo #{baseip}.${x} node${x##?} | sudo tee -a /etc/hosts &>/dev/null
  }

done
yes y |ssh-keygen -f /home/vagrant/.ssh/id_rsa -t rsa -N ''
echo " **** SSH Key Pair created for node$c ****"

SCRIPT

# SSH configuration script
$ssh = <<SCRIPT1
numnodes=6

for (( c=1; c<$numnodes+1; c++ ))
do
    echo "$c"
    echo "node$c"
    if [ "$HOSTNAME" = "node1" ]; then
      echo "**** Install ansible on node1 ****"
      sudo yum install ansible -y
    fi
    # Skip the current host.
    if [ "$HOSTNAME" = "node$c" ]; then
        echo "node$c"
        continue
    fi

    # Copy the current host's id to each other host.
    # Asks for password.
    # create ssh key
    
    sshpass -p vagrant ssh-copy-id "node$c"
    echo "**** Copied public key to node$c ****"    
done

# Get the id's from each host.
for (( c=1; c<$numnodes+1; c++ ))
do
    # Skip the current host.
    if [ "$HOSTNAME" = "node$c" ]; then
        continue
    fi

    sshpass -p vagrant ssh "node$c" 'cat .ssh/id_rsa.pub' >> /home/vagrant/host-ids.pub
    echo "**** Copy id_rsa.pub contentes to host-ids.pub for host node$c ****"
done

for (( c=1; c<$numnodes+1; c++ ))
do
    # Skip the current host.
    if [ "$HOSTNAME" = "node$c" ]; then
        continue
    fi

    # Copy public keys to the nodes
    sshpass -p vagrant ssh-copy-id -f -i /home/vagrant/host-ids.pub "node$c"
    echo "**** Copy public keys to node$c ****"

done
# Set the permissions to config
sudo chmod 0600 /home/vagrant/.ssh/config
# Finally restart the SSHD daemon
sudo systemctl restart sshd
echo "**** End of the Multi Machine SSH Key based Auth configuration ****"

SCRIPT1

# Vagrant configuration
Vagrant.configure("2") do |config|
  # Execute global script
  config.vm.provision "shell", privileged: false, inline: $global
  prefix="node"
  #For each node run the config and apply settings
  (1..numnodes).each do |i|
    vm_name = "#{prefix}#{i}"
    config.vm.define vm_name do |node|
      node.vm.box = "centos/7"
      node.vm.hostname = vm_name
      ip="#{baseip}.#{10+i}"
      node.vm.network "private_network", ip: ip    
    end
    # Run the SSH configuration script
    config.vm.provision "ssh", type: "shell", privileged: false, inline: $ssh
  end
end

To execute the above configuration file, run the below commands

$vagrant up
$vagrant provision --provision-with ssh

Please note that the above example show vagrant user credentials by using sshpass -p option. If you want to secure use -f also read the sshpass documentation for more info. Many constants like EPEL repo URL, number of nodes, ssh key path etc.. need be customized according to your actual requirements.

To check the status of the nodes build by vagrant use the below command.

$vagrant status
Current machine states:

node1                     running (virtualbox)
node2                     running (virtualbox)
node3                     running (virtualbox)
node4                     running (virtualbox)
node5                     running (virtualbox)
node6                     running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

To login to the first node, and to SSH to other nodes use the below commands. Notice that there was no password prompts which SSHing between the nodes. By the way, Ansible is installed in node1 and ready to use. eth1 is the private network used for SSH inter-connectivity.

$vagrant ssh node1
Last login: Tue Jun 11 12:01:11 2019 from 192.168.10.12
[vagrant@node1 ~]$ssh node2
Warning: Permanently added 'node2,192.168.10.12' (ECDSA) to the list of known hosts.
Last login: Tue Jun 11 12:01:04 2019 from 192.168.10.11
[vagrant@node2 ~]$ssh node1
Warning: Permanently added 'node1,192.168.10.11' (ECDSA) to the list of known hosts.
Last login: Tue Jun 11 12:16:41 2019 from 10.0.2.2
[vagrant@node1 ~]$ssh node5
Warning: Permanently added 'node5,192.168.10.15' (ECDSA) to the list of known hosts.
[vagrant@node5 ~]$ssh node1
Warning: Permanently added 'node1,192.168.10.11' (ECDSA) to the list of known hosts.
Last login: Tue Jun 11 12:17:23 2019 from 192.168.10.12
[vagrant@node1 ~]$yum list ansible 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * epel: sg.fedora.ipserverone.com
 * extras: mirror.dhakacom.com
 * updates: mirrors.nhanhoa.com
Installed Packages
ansible.noarch                                             2.8.0-2.el7                                             @epel
[vagrant@node1 ~]$ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:26:10:60 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic eth0
       valid_lft 80872sec preferred_lft 80872sec
    inet6 fe80::5054:ff:fe26:1060/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:7b:8a:ef brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.11/24 brd 192.168.10.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7b:8aef/64 scope link
       valid_lft forever preferred_lft forever
[vagrant@node1 ~]$

Hope this use case help to understand how to install and configure multiple VMs at once with SSH inter-connectivity. Please leave your feedback if you found this blog useful and share suggestions in the below Comments section.

Image Courtesy: sumglobal.com

References:

https://www.vagrantup.com/docs/multi-machine/

https://www.vagrantup.com/docs/vagrantfile/

https://www.vagrantup.com/docs/provisioning/basic_usage.html

https://github.com/kikitux/vagrant-multimachine/edit/master/intrassh/Vagrantfile

Advertisements

Install and Configure Go/Golang on Raspberry Pi

Go/Golang is one of the hot programming languages while I’m typing this today.

Go is a programming language created at Google in 2009 by Robert Griesemer, Rob Pike and Ken Thompson. Go is a statically typed compiled, procedural language similar to C, with memory safety, garbage collection, structural typing, concurrency and other great features are bundled to make it better compared to other languages in the marketplace.

Docker, Kubernetes, Graphana, Hugo are some of the best apps written in Go. It has a robust set of libraries and app performance is better compared to other languages.

Today I’m starting my journey to learn Go/Golang and Google will be my mentor to install and setup Golang on my Raspbian OS/Raspberry Pi 3. To get the latest version use below steps instead of native package management tool such as apt.

Installation Steps:

  • Download the current stable version of Go available on the google’s official website. At the time of writing this tutorial, 1.12.4 is the stable version. Check the latest version here
cd ~ && curl -O https://dl.google.com/go/go1.12.4.linux-armv6l.tar.gz

Above command would change directory to your ‘Home’ directory and download Go compressed tar file using ‘Curl’

  • Extract compressed tar file and place it inside /usr/local directory. Please note root level access or sudo access required to perform this step.
sudo tar -C /usr/local -xzvf go1.12.4.linux-armv6l.tar.gz
  • Set Path variables to avoid typing complete path and in order to access binaries or libraries of Go by the Raspbian OS. Open ~/.profile, a hidden file located in your ‘Home’ directory. Use nano or vi or subl to edit the file (subl ~/.profile). Add below lines at the end of the file.
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
  • To take effect of the above changes made to the file ‘profile’, run the below command. The source command can be used to load any functions file into the current shell, script or a command prompt.
source ~/.profile
  • Create directory called ‘go’ in the ‘Home’ directory. All my codes are placed in the folder. Change the directory name as per choice but do change the GOPATH accordingly as mentioned above.
 mkdir $HOME/go 
  • Validate Go is working as expected or not by running below command
vb@pi:~ $  go version
go version go1.12.4 linux/arm
vb@pi:~ $

My first code in Go

Create a directory first_code in ‘go’ and write following content in the file and save it as first_code.go

mkdir -p $HOME/go/src/first_code 
{-p is used to create directory and its sub-directories at once}
package main

import "fmt"

func main() {
    fmt.Printf("My first code in Go Language!!!\n")
}

Now build and run the code. Change directory
to the first_code cd ~/go/src/first_code and run below commands

vb@pi:~/go/src/first_code $ go build
vb@pi:~/go/src/first_code $

vb@pi:~/go/src/first_code $ ./first_code
My first code in Go Language!!!
vb@pi:~/go/src/first_code $

Note:

Above steps are applicable to any Linux distributions just by changing the first step of downloading the compressed tar file. Change the architecture from arm6l to amd64 or as applicable to your hardware.

I am able to successfully setup GO on my Pi. Hope you’d also do the same and happy learning Go. If you have any issues or questions please mention them in the below comment section.

Docker’ize’ Python

What is Docker?

Docker is a computer program that performs operating-system-level virtualization, also known as “containerization”. It was first released in 2013 and is developed by Docker, Inc. source: Wikipedia

How Docker works?

Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools, and libraries. This guarantees that your application will always run the same and makes collaboration as simple as sharing a container image.
source: www.docker.com

Why Docker?

Docker unlocks the potential of your organization by giving developers and IT the freedom to build, manage and secure business-critical applications without the fear of technology or infrastructure lock-in. source: www.docker.com

Pycodestyle @ Sublime Text

Is your script/code looks dull and not in order? If yes, then its time to use pycodestyle!!

Pycodestyle (formerly known as PEP8) is a syntax and style checker tool for Python language. In this blog, I’ve documented step by step procedure to install this with Sublime Text.

Before we start, let us briefly understand more about linter. Lint or linter is a tool that analyzes source code to flag programming errors, bugs, stylistic errors. To know more please check out the wiki link

Pycodestyle is one of the linter plugins available for Sublime Text. To install this plugin, first, we need to install Sublime Linter plugin.

Prerequisites:

  1. Python3.x
  2. Sublime Text
  3. PIP

***This step by step procedure to apply to Debian and Debian based Linux Operation Systems.***

Steps to install pycodestyle:

  1. Install system package using pip or apt
$pip3 install pycodestyle
OR
$sudo apt install python3-pycodestyle
$which pycodestyle # shows the location where pycode style is

2. Install pycodestyle plugin in sublime text

a. Open package control (CTRL+SHIFT+P), type 'install package' and hit 'ENTER' # It takes some time to load repositories. 
b. Type 'sublimelinter-pycodestyle' and Click on the first option as shown below. After installation, a new tab window will open showing installed plugin information
C. To configure sublime-linter settings, goto Preferences > Package Settings > SublimeLinter > Settings
d. In the Sublime Linter settings, make sure @disable: false is set s shown in below screenshot. 
e. To test pycodestyle, create a new file and save it as test.py.
Start writing the code, Pycodestyle will automatically start showing the errors while you are typing in the console message.

That’s it!! Your code looks more disciplined of course after fixing the errors 🙂

In addition to pycodestyle, try pylint, pyflakes and pydocstyle plugins as shown above. These plugins improve Code Quality by following Python Coding Standards. Following best practices will make you a good scripter/developer and remember Quality Matters!!

References:

  1. https://pypi.org/project/pycodestyle/
  2. https://github.com/ergdev/SublimeLinter-pycodestyle
  3. http://pycodestyle.pycqa.org/en/latest/
  4. http://www.sublimelinter.com/en/stable/settings.html

So go ahead and use pycodestyle. Use the above references to deep dive and learn more about this plugin which is a ‘Must Have’ for the Sublime Text. Happy Pythonic way of coding!!

Python’s sh Library

Wish you all Happy, Prosperous & Pythonic New Year 2019!!

I thank everyone for stopping by my blog for 15K times in 2018 which is the highest so far! 

“Looking back at my life’s voyage, I can only say that it has been a good trip.” ~ Ginger Rogers

Similar to the above quote, I’d started my scripting voyage by writing shell, batch and Perl scripts in good old days. I miss some of the awesome, builtin & yet simple to use features available in shell. I felt using those features or builtins would make my life easier while writing scripts in Python…

Besides that, one main reason to look back at the past was while imparting the Python Workshop at my office. One of the participants raised concerns over Python’s builtin modules like OS and Subprocess. They mentioned we can accomplish with very simple steps in ‘Shell’ instead of using OS or Subprocess module. I had to accept the fact that sometimes Python is not so easy as we think!

After googling for a while I finally found this… Say hola to ‘sh‘ Module!

The ‘sh’ library provides simple and intuitive alternative to OS/System/Subprocess modules.

‘sh’ module simplifies Python’s ability to interact with native OS by calling shell commands. This would greatly helps in automation of routine tasks and running sequence of commands, parsing output as per requirement.

Python is a powerful language powered by those great developers who are continuously contributing to enable new features every day in day out. I would like to thank Andrew Moffat for the ‘sh’ library. This library is one of my favourites.

To install this library using PIP, run below command

pip3 install sh 

Please find sample scripts output. 

>>> import sh

>>> sh.uname()

Darwin

>>> sh.uptime()

 7:13  up 11 days, 23:29, 4 users, load averages: 1.51 1.65 1.88

>>> 

>>> sh.echo(‘Hello, This is shell\’s echo running from Python’)

Hello, This is shell's echo running from Python

>>> sh.who()

vinay    console  Dec 21 07:45 

vinay    ttys000  Dec 21 07:46 

vinay    ttys001  Dec 21 07:46 

vinay    ttys003  Jan  2 07:02 

>>>sh.df()

Filesystem    512-blocks     Used Available Capacity iused               ifree %iused  Mounted on

/dev/disk1s1   236568496 96124200 131754736    43% 1042911 9223372036853732896    0%   /

devfs                376      376         0   100%     653                   0  100%   /dev

/dev/disk1s4   236568496  6291496 131754736     5%       3 9223372036854775804    0%   /private/var/vm

map -hosts             0        0         0   100%       0                   0  100%   /net

map auto_home          0        0         0   100%       0                   0  100%   /home

>>> sh.ifconfig()

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>

inet 127.0.0.1 netmask 0xff000000 

inet6 ::1 prefixlen 128 

inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 

nd6 options=201<PERFORMNUD,DAD>

gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280

stf0: flags=0<> mtu 1280

XHC20: flags=0<> mtu 0

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether b8:e8:56:36:0b:58 

inet6 fe80::149c:1b88:578c:73f8%en0 prefixlen 64 secured scopeid 0x5 

inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304

ether 0a:e8:56:36:0b:58 

media: autoselect

status: inactive

awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484

ether 8a:0c:61:9e:64:9b 

inet6 fe80::880c:61ff:fe9e:649b%awdl0 prefixlen 64 scopeid 0x7 

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=60<TSO4,TSO6>

ether 72:00:00:20:b1:90 

media: autoselect <full-duplex>

status: inactive

en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=60<TSO4,TSO6>

ether 72:00:00:20:b1:91 

media: autoselect <full-duplex>

status: inactive

bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=63<RXCSUM,TXCSUM,TSO4,TSO6>

ether 72:00:00:20:b1:90 

Configuration:

id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0

maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200

root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0

ipfilter disabled flags 0x2

member: en1 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 8 priority 0 path cost 0

member: en2 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 9 priority 0 path cost 0

nd6 options=201<PERFORMNUD,DAD>

media: <unknown type>

status: inactive

utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000

inet6 fe80::bd0:9dea:48de:5645%utun0 prefixlen 64 scopeid 0xb 

nd6 options=201<PERFORMNUD,DAD>

>>> 

Thanks for stopping by, please share your comments and ideas to improve this blog. Keep watching for more Python libraries used for automation.

Telnet (NetCat) and Ping Script

Nowadays most of the Linux distros do not include Telnet. If needed we can install it using the usual methods.

Let’s embrace Netcat which is a best alternative to Telnet. Earlier I had written a Perl script which did Telnet and Ping checks across multiple hosts.

This script requires two input files ips.csv and ports.csv. As the name suggest all target host IP’s to be placed in ips.csv file and the ports to which to netcat is placed in ports.csv. Output is in HTML format and user receives an email with colour coded results.

Replace ips.csv & ports.csv file contents and change SMTP IP and email address.

Here is the Python script which replaces Telnet with Netcat.

SSH Keys – Password less authentication

I was trying to setup SSH Keys between two different flavors of Linux Host by following this Howto

I did run commands exactly as per mentioned in the how to; but it didn’t work. Error message as under

vin@CLIENT:~$ ssh vsa@192.0.0.10
vin@192.0.0.10’s password:
Last login: Mon May 22 10:45:03 2017 from 192.0.0.10
-bash: id: command not found
-bash: id: command not found
-bash: id: command not found
-bash: tty: command not found
-bash: uname: command not found

After googling and trail and error method finally found a fix.

Instead of Step 3 as per the above How to, use the below commands.

  1. Ensure on the server proper permissions set on the .ssh folder if not set permissions as

$chmod -R 775 .ssh (.ssh is located in user home directory e.g: /home/vin)

  1. Run this command to copy the Key file from client to server

$cat ~/.ssh/id_rsa.pub | ssh vin@192.0.0.10 ‘umask 0077; /bin/mkdir -p .ssh; /bin/cat >> .ssh/authorized_keys && echo “Done!”‘

After running above command, server 192.0.0.10 do not ask for password when user vin try to login.

vin@CLIENT:~$ ssh vin@192.0.0.10
Last login: Mon May 22 11:54:29 2017 from 192.0.0.10
[vin@SERVER ~]$