Most useful Tar command examples

In this post we will see how to use the tar command to compress and uncompress your files and directories. Many softwares are distributed by compressing them using tar so the examples below will guide you how you can uncompress or open such files. This command is also very useful when you have to archive or share huge directories or files. 

Create a tar 

In the below example we have a directory called cloudvedas which has multiple files. Let's tar this directory. With -v option you can see what all files it is adding in the tar ball.

-c: Create an archive.
-v: Verbose output shows you all the files being archived.
-f: Allows you to specify the filename of the archive.

cvterm#tar -cvf cloudvedas.tar cloudvedas
cloudvedas/
cloudvedas/cv1.sh
cloudvedas/cv2.sh
cloudvedas/cv3
cloudvedas/cv4
cloudvedas/cv5.py

So now we can see we have our tar file.

cvterm#ls -lrth
total 12K
drwxr-xr-x 1 sagu sagu 4.0K Jun 16 08:19 cloudvedas
-rw-r--r-- 1 sagu sagu  10K Jun 16 08:20 cloudvedas.tar
cvterm#

List or View contents of a tar

Now if you want to check all the files which are in the tar ball without doing untar/opening/uncompressing you can check it as below

 cvterm#tar --list -f cloudvedas.tar
cloudvedas/
cloudvedas/cv1.sh
cloudvedas/cv2.sh
cloudvedas/cv3
cloudvedas/cv4
cloudvedas/cv5.py
cvterm#

 Another way to see the contents of a tar ball without untar is using -tvf option like below. This will show the permissions also.

cvterm#tar -tvf cloudvedas.tar
drwxr-xr-x sagu/sagu         0 2021-06-16 08:19 cloudvedas/
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv1.sh
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv2.sh
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv3
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv4
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv5.py
cvterm#

Create tar.gz or .tgz which is tar and gzip in single command.

Now let's try to gzip the directory along with tar in a single command. We will do this with option -z .

-z: Compress the archive with gzip.

cvterm#tar -cvzf cloudvedas.tar.gz cloudvedas

We can see below a .tar.gz is created cloudvedas.tar.gz 

cvterm#ls -lrth
total 8.0K
drwxr-xr-x 1 sagu sagu 4.0K Jun 17 19:34 cloudvedas
-rw-r--r-- 1 sagu sagu 5.5K Jun 17 19:34 cloudvedas.tar.gz
cvterm#

We can also create .tgz which is same as .tar.gz 

cvterm#tar -cvzf cloudvedas.tgz cloudvedas

cvterm#ls -lrth
total 16K
drwxr-xr-x 1 sagu sagu 4.0K Jun 17 19:34 cloudvedas
-rw-r--r-- 1 sagu sagu 5.5K Jun 17 19:34 cloudvedas.tar.gz
-rw-r--r-- 1 sagu sagu 5.5K Jun 17 19:36 cloudvedas.tgz
cvterm#

Create a  tar.bz2 or tbz or tb2

The tar.bz2 or tbz or tb2 creates a file which is more compressed as compared to gz but these 3 extensions takes more time to compress and uncompress. If you see the file size listed below you can see clear difference.

cvterm#tar -cvjf cloudvedas.tar.tb2 cloudvedas

or

cvterm#tar -cvjf cloudvedas.tar.tbz cloudvedas

or

cvterm#tar -cvjf cloudvedas.tar.bz2 cloudvedas

cvterm#ls -lrth

total 5.1M

drwxr-xr-x 1 sagu sagu 4.0K Jun 17 19:34 cloudvedas

-rw-r--r-- 1 sagu sagu 5.5K Jun 17 19:34 cloudvedas.tar.gz

-rw-r--r-- 1 sagu sagu 5.5K Jun 17 19:36 cloudvedas.tgz

-rw-r--r-- 1 sagu sagu  332 Jun 17 19:37 cloudvedas.tar.tb2

-rw-r--r-- 1 sagu sagu  332 Jun 17 19:38 cloudvedas.tar.tbz

-rw-r--r-- 1 sagu sagu  332 Jun 17 19:38 cloudvedas.tar.bz2

-rw-r--r-- 1 sagu sagu 5.1M Jun 17 19:44 cloudvedas.tar

cvterm#

List / View the contents of .tar.gz .tgz .tar.bz2 .tar.tbz  .tar.tb2

You can simply view the contents of all these file types by same option of -tvf

cvterm#tar -tvf cloudvedas.tar.gz

or 

cvterm#tar -tvf cloudvedas.tgz

or

cvterm#tar -tvf cloudvedas.tar.tbz

or

cvterm#tar -tvf cloudvedas.tar.bz2

or

cvterm#tar -tvf cloudvedas.tar.tb2

Extract or Untar a file

To untar a file in the current directory do below

-f: It tells tar the name and path of the compressed file.
-x: Extract the files.
-v: Verbose output shows you all the files being extracted.

cvterm#tar -xvf cloudvedas.tar 

To untar a file in a different directory use option -C

cvterm#tar -xvf cloudvedas.tar -C /var/tmp/newdir 
cvterm#pwd
/var/tmp/newdir
cvterm#ls
cloudvedas
cvterm#

Now let's extract single file from tar

cvterm#tar -xvf cloudvedas.tar cloudvedas/file5.txt

Uncompress and untar .tar.gz or .tgz

cvterm#tar -xvf cloudvedas.tar.gz
or
cvterm#tar -xvf cloudvedas.tgz 

Uncompress and untar single file .tar.gz or .tgz

cvterm#tar -zxvf cloudvedas.tar.gz cloudvedas/file4.txt
cloudvedas/file4.txt
cvterm#tar -zxvf cloudvedas.tgz cloudvedas/file5.txt
cloudvedas/file5.txt

Uncompress and untar .tar.gz or .tgz in a different directory

cvterm#tar -xvf cloudvedas.tar.gz -C /home/sagu/newdir/
or
cvterm#tar -xvf cloudvedas.tgz -C /home/sagu/newdir/

Uncompress and untar tar.bz2, tar.tbz, tar.tb2 

cvterm#tar -xvf cloudvedas.tar.bz2
or
cvterm#tar -xvf cloudvedas.tar.tbz
or
cvterm#tar -xvf cloudvedas.tar.tb2

Extract multiple files from .tar

cvterm#tar -xvf cloudvedas.tar "cloudvedas/file.txt" "cloudvedas/file2.txt"
cloudvedas/file.txt
cloudvedas/file2.txt

Extract multiple files from tar.gz .tgz using -z option

-z: Decompress the archive using gzip

cvterm#tar -zxvf cloudvedas.tar.gz "cloudvedas/file3.txt" "cloudvedas/file4.txt"
or
cvterm#tar -zxvf cloudvedas.tgz "cloudvedas/file3.txt" "cloudvedas/file4.txt"

Extract multiple files from tar.bz2, tar.tbz, tar.tb2 using -j option

cvterm#tar -jxvf cloudvedas.tar.bz2 "cloudvedas/file5.txt" "cloudvedas/cv1.sh"
or
cvterm#tar -jxvf cloudvedas.tbz "cloudvedas/file5.txt" "cloudvedas/cv1.sh"
or
cvterm#tar -jxvf cloudvedas.tar.tb2 "cloudvedas/file5.txt" "cloudvedas/cv1.sh"

Extract group of files using --wildcard option

.tar

cvterm#tar -xvf cloudvedas.tar --wildcards '*.sh'

.tar.gz .tgz using -z option

cvterm#tar -zxvf cloudvedas.tar.gz --wildcards '*.sh'
cvterm#tar -zxvf cloudvedas.tgz --wildcards '*.sh'

tar.bz2, tar.tbz, tar.tb2 using -j option

cvterm#tar -jxvf cloudvedas.tar.bz2 --wildcards '*.txt'
cvterm#tar -jxvf cloudvedas.tar.tbz --wildcards '*.txt'
cvterm#tar -jxvf cloudvedas.tar.tb2 --wildcards '*.txt'

Add a new file or directory in .tar archive using -r option

cvterm#tar -rvf cloudvedas.tar cvnew1.sh   ##Adding a file
cvterm#tar -rvf cloudvedas.tar newdir1  ## Adding dir

Check below that now we have new file and dir added in our .tar

cvterm#tar -tvf cloudvedas.tar
drwxr-xr-x sagu/sagu         0 2021-06-17 19:34 cloudvedas/
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv1.sh
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv2.sh
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv3
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv4
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv5.py
drwxr-xr-x sagu/sagu         0 2021-06-17 19:17 cloudvedas/cvdir1/
drwxr-xr-x sagu/sagu         0 2021-06-17 19:17 cloudvedas/cvdir2/
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:33 cloudvedas/file.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file2.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file3.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file4.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file5.txt
-rw-r--r-- sagu/sagu         0 2021-06-18 19:27 cvnew1.sh
drwxr-xr-x sagu/sagu         0 2021-06-18 19:26 newdir1/
cvterm#

Note: You cannot add files or dir  to tar.gz, .tgz, tar.bz2, tar.tbz, tar.tb2 archives

Exclude specific file or directories while creating a tar

cvterm#tar -cf cloudvedasexclude.tar --exclude='cloudvedas/cvdir1' --exclude='cloudvedas/file2.txt' cloudvedas

As we can see below the file and directory we excluded are not in the tar.

cvterm#tar -tvf cloudvedasexclude.tar
drwxr-xr-x sagu/sagu         0 2021-06-18 19:25 cloudvedas/
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv1.sh
-rw-r--r-- sagu/sagu         0 2021-06-16 08:19 cloudvedas/cv2.sh
drwxr-xr-x sagu/sagu         0 2021-06-17 19:17 cloudvedas/cvdir2/
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:33 cloudvedas/file.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file3.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file4.txt
-rw-r--r-- sagu/sagu   1048576 2021-06-17 19:34 cloudvedas/file5.txt
cvterm#

Above we have seen some examples of frequently used tar command options. You can also use tar command to convert your vmware ova files to ovf . Check this post on how to convert ova to ovf  using tar.

There are lot more options in tar that you can see using tar --help in your terminal. Do let us know in the comment section if you want us to add any other examples.

Steps to create docker containers in your laptop

In this post we will see how to create a docker container of ubuntu on your windows laptop.

Pre-requisites:-

I am using “Docker for Windows” software to run dockers on my Windows 10 laptop. You can get “Docker for Windows” by clicking on this link .
If you have Windows 7 download Docker Toolbox for Windows with Virtualbox.

Ubuntu docker creation
Once you are done with docker installation let’s move ahead.

• In the windows command prompt or in “Docker Quickstart Terminal” execute below command. By default it will pull the latest image of ubuntu container available in repository.

C:\CloudVedas>docker run ubuntu

• If you need  specific version of Ubuntu you can mention the version name in command. Like below we are pulling Ubuntu 14.04 version. You can check all the available versions here .

C:\CloudVedas> docker run ubuntu:14.04

Unable to find image 'ubuntu:14.04' locally

14.04: Pulling from library/ubuntu

2e6e20c8e2e6: Pulling fs layer

30bb187ac3fc: Pulling fs layer

• Now let’s see the image we have downloaded .

C:\CloudVedas> docker image ls

REPOSITORY      TAG                 IMAGE ID            CREATED             SIZE

ubuntu                   14.04               6e4f1fe62ff1        4 months ago        197MB

• Let’s create a container with that image using the image id. Here we are using -t and -d option so that the container keeps on running in detached mode and we can login to it.

C:\CloudVedas>docker run -t -d 6e4f1fe62ff1

a8dee68d78026adb830edb04391af00ec8b7e1033e711fc640a1489ca54adc0a

• List the running containers using “docker container ls”

C:\CloudVedas>docker container ls
CONTAINER ID       IMAGE        COMMAND      CREATED         STATUS     PORTS       NAMES
a8dee68d7802        6e4f1fe62ff1   "/bin/bash"      About a minute ago   Up About a minute eager_johnson

We can see our container is created 5 minutes ago and is up. You can also identify the container using the container id. Note that the container ID is same as the first 12 digits of the string we got when we executed docker run in last step.

• Let’s get inside our container and check it.

C:\CloudVedas>docker exec -it  a8dee68d7802 /bin/bash

Once you are inside the Ubuntu container you can explore it. Let’s check the OS version.

root@a8dee68d7802:/# more /etc/os-release
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
root@a8dee68d7802:/#

• If you don’t see your container up, it may have stopped automatically. To check all the containers which are/were running, execute command

C:\CloudVedas>docker ps -a

You can also create your own customized container image which you can use to deploy more containers in your environment. Refer this post on how to create a container image.  If instead of Docker hub you are using AWS ECR, check the post on how to push/pull image to/from AWS ECR. 

If you want to try next level on docker you can user dockerfile or docker-compose files to install your application. Like in our other post we cover how you can install wordpress in docker.

Hope this post helped you. Do let us know if you have any query or you get stuck in any installation step. 

How to become a Google Cloud Certified Professional Cloud Architect

In this post I'll be giving tips on how to prepare for Google Cloud Certified Professional Cloud Architect exam. This will also be helpful for people who are currently working on other cloud platforms like AWS or Azure and looking for broadening their skills to Google Cloud Platform (GCP).

As many of you who are following this blog knows that I am already working on AWS and Azure. About a couple of years back we got heavily into Kubernetes. Being a curious techie when I started digging further about Kubernetes, I found that it was initially designed by Google. One thing led to another and I ended up exploring more about google cloud. In parallel, we started getting traction on multi-cloud strategy and GCP is also considered a good option with many features which are helpful for both startups and big Enterprises.

So, I decided to get more knowledge and expertise on Google Cloud. When I compared AWS with GCP I felt that most of the technologies are similar but obviously with different naming convention and some technical setting differences. Thus, if you have worked on AWS it won't be very difficult to grasp the Google Cloud as well. But even if you don't have background in other clouds then also learning about google cloud is not very difficult, you just need to spend extra time on the basics.

If, you are from AWS background you can get a good comparison of AWS and GCP services here and that's what I did as a starting point.

Next I went through the Udemy course Google Cloud (GCP) 2020: Zero to Cloud Architect. This course covers the GCP services in detail starting from basics. So, it is useful even for someone who is starting from scratch.

Since our company is a partner of Google so, I supplemented my preparation by enrolling in the online training labs of QWIKLABS. These labs are really helpful in getting you good hands-on practice on the various GCP services.

AWS background folks will find that the GCP services are not very different but it still has some differences for e.g. in AWS a subnet is restricted to an AZ but in GCP the subnet can span to multiple AZs in a region. You have to keep these subtle differences in mind when designing in GCP.

If we talk specifically about the certification exam it mainly focuses on below topics:-

• Managing and provisioning cloud infrastructure.
• Designing and planning a cloud solution which is scalable, resilient and cost effective.
• Security strengthening using IAM, Firewall, Security Groups etc.
• Analyzing and Monitoring the application behavior using GCP Operations Suite.
• Managing HA and DR.

The exam is in multiple choice questions format. You will also get 2-3 case studies and you have to select an answer which is most suitable considering the scenario mentioned in case study. 

You can choose to appear for an exam at the test center or go for an online proctored exam from home. Considering the Corona situation I appeared for an online proctored exam. You just have to follow the guidelines mentioned in the link I have shared above and with good internet connection it is pretty easy to appear for the exam from home.

Overall I found the exam to be very engaging covering wide ranging of topics.

If you have any queries regarding the exam preparation or GCP in general please post them in the comment section below. 

How to transfer files to and from EC2 instance

In our earlier post we showed how you can use Filezilla a GUI based solution to transfer files to an EC2 instance. But, in many companies installation of third party software like Filezilla is not allowed. 

So, in this post we will show you how you can transfer files to and from an EC2 linux instance using our old trustworthy friend SFTP. 

For those who don't know about sftp let us give you a gist of what it is.

SFTP is SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP)  thus it works on same port 22 as ssh. It's secure in comparison to ftp which works on port 21 and nowadays blocked because of security reasons. sftp is generally pre-installed on most linux versions including Amazon Linux.

Also, if you compare it with SCP , which supports only file transfers, the SFTP allows you to perform a range of operations on remote files and resume file transfers.

If you want to know more about SFTP please look at this sftp wiki page.  

Now let's see how we can use sftp to transfer files.

Pre-requisite for this are only two things both of which are pretty much standard requirement to access your EC2 linux instances.

1) ssh .pem key which you configured when you built the remote server where you want to connect.

2) Port 22 should be open to access the remote server. (In case you want to know if a port is open on a remote linux/unix server by CLI without using telnet check this post .)

Once you have checked that you have fulfilled the pre-requisites let's move to the next step.

Open your shell terminal, it can be GIT Bash installed on you local windows desktop or Linux shell terminal.

Inside the terminal you need to execute below command

sftp -o IdentityFile=identity_file ec2-user@10.xxx.xxx.xxx

where identity_file is you .pem key .

Your actual command will look like

sftp -o IdentityFile=cloudvedas.pem ec2-user@192.168.0.10

Let's check our directory in remote server

sftp>pwd

Remote working directory: /home/ec2-user

Let's go to /tmp

sftp> cd /tmp

Let's transfer a file from local machine to the remote server

sftp> PUT test123-file.sh

Now if you want to transfer a file from remote server to local machine 

sftp> GET remote123-file.sh

Note: PUT and GET commands are case sensitive and will work in uppercase only.

If you forgot what was the home directory on your local machine you can check that from sftp prompt

sftp>lpwd

Local working directory: /home/cloudvedas

If you want to change the directory in your local machine do this

sftp>lcd /tmp

Hope this simple tutorial is helpful for you. Do let us know in comments section if you have any queries or to share what methods you use to transfer file to EC2 instances.

AWS Subnet Calculator

This is a simple calculator to let you know how many IPv4 IPs you will get when you create a Subnet in AWS. 

AWS allows subnet mask only between /16 to /28 . Few IPs in each subnet are reserved for AWS internal usage . 

To calculate for e.g. in subnet 10.0.0.0/24 subnet mask is 24 so, enter 24 below to get available IPs in this subnet . 

Enter Subnet Mask

Disclaimer: Please note that this is not an offical AWS calculator. Please visit AWS VPC for more details.