Recover logical volumes data from deleted LVM partition

Have you ever deleted a logical volume by accident? Can you recover it looking into the backups? Well, the answer is YES. For those who are not familiar with Logical Volume Management (LVM) is a device mapper target that provides logical volume management for the Linux kernel.- WikipediaIt is an abstraction layer or software that has been placed on top of your hard drive for flexible manipulation of the disk space. Some of the articles published in the past on LVM are:

All test carried out on this blog post have been tested on a CentOS machine. Please don’t make a live test on a production server.

Image Credits: Redhat.com
Image Credits: Redhat.com

1. So, as you can see below I have an lv called lvopt which is from a vg called centos.

2. Same is mounted on the /opt

3. There are some data in that partition as well:

4. I created a directory inside the /opt directory

5. Now, let’s pretend to remove the lvm lvopt. Or say, someone did it by accident because it was unmounted. The command lvremove will be used here to remove the lv. Note: that the lv need to be unmounted.

6. If you make an lvs, lvdisplay or vgs or even mount again the partition, you cannot do it. The data is lost. But you can still recover it. This is because the lvm contains the archive of your lv inside the folder /etc/lvm/archive. But, you cannot read the files directly.

7. But you can still, interpret part of the files. Since we deleted the volume group called “centos”, we knew that it is referenced in the file centos_… The question that arises here is which file is relevant for you. Right? So to understand which archive you want to restore, you need to use the command vgcfgrestore –list <name of volume group>. Here is an example:

8.  If you observe carefully, each archive has been backup at a certain time. In my case, I deleted the LV on 18-Apr-2019 at 11:17:17 2019:

9. So, I want to restore from that last archive. You will need to copy the full patch of the vg file. In my case it is /etc/lvm/archive/centos_00004-1870674349.vg. The goal here is to restore the lv before this specific time, or simply restore back the lv before the command lvremove was fired. Here is the command:

10. If you launch the command lvs, you will notice the presence of the lv.

11. But, mounting back the lv won’t result in anything. This is because the lv is inactive. You can see it with the command lvscan. Please take note below that the lvopt is inactive.

12. To activate it you simply need to use the command lvchange.

13. Mount it back and you are done.

I believe this can be very useful especially when you have encountered a situation where someone deleted an lv. I hope you enjoy this blog post. Please share and comment below if you like it.

cyberstorm.mu team at Developers Conference Mauritius

A few weeks back, I registered myself to present the Ansible automation tool at the Developers Conference 2019 at Voila Hotel, Bagatelle Mauritius. The event is an initiative of Mauritius Software Craftsmanship Community – MSCC sponsored by several companies such as Mauritius Commercial Bank, SdWorx, Eventstore, Ceridian, Castille, etc. There were other members of cyberstorm.mu who also registered for their presentations: they are Codarren Velvindron, technical lead at Orange Business Services who spoke about “becoming an automation artist”, Loganaden Velvindron who spoke about “RedHat Enterprise Linux 8 and Derivatives have a new Firewall: NFTABLEs”, and Nathan Sunil Mangar who spoke about “Introduction to the STM32”. There was also a special event where Mukom Akong Tamon, head of capacity building for Africa region at Afrinic who spoke on “IPv6 deployment in Mauritius and Africa at large”. I presented myself as a member of cyberstorm.mu and DevOps Engineer at Orange Business Services and spoke on Ansible for beginners with some basic and advanced demos.

In the past, I have written several articles on Ansible:

  1. Getting started with Ansible deployment
  2. Some fun with Ansible Playbooks
  3. Configure your LVM via Ansible
  4. Some tips with Ansible Modules for managing OS and Applications
  5. An agentless servers inventory with Ansible and Ansible-CMDB
  6. Project Tabulogs: Linux Last logs on HTML table with Ansible

My presentation started with a basic introduction to Ansible following some brief examples and demos. I started with a brief introduction of myself. It looks like it was a mixed audience including, Students, Professional from the management and technical side, Engineers, and others. I brushed out quickly as to why we need Ansible in our daily life whether for home use or on production. Ansible is compatible with several Operating systems and one of the most interesting tools is the AWX which is an opensource product. Before getting started with Ansible, it is important to grasp some keywords. I introduced it as well as giving some examples using Playbooks. Ansible Ad-hoc commands were also used. The audience was asked to give some ideas about what they want to automate in the future. There were lots of pretty examples. I laid some emphasis on reading the docs and keep in touch with the version of Ansible one is using. Also gave some brief idea about Ansible-Galaxy, Ansible-doc, Ansible-pull, and Ansible-vault. To spice up your automation layout, it would be nice to use Jinja templates, verbosity for better visual comprehension. I also spoke about Ansible-CMDB, which is not a tool of Ansible. Some days back, I blogged on Ansible-CMDB which is pretty interesting to create an inventory. I also shed some ideas about how to modify the source code of Ansible-CMDB. Also, an example using an Ansible Playbook build up web apps.

Thanks, everyone for taking pictures and some recordings.

cyberstorm.mu @ DevConMru

Screen Shot 2019-04-16 at 8.01.34 PM
D4AvZJ-WsAY9anz.jpg large
D39IqdeX4AAJr5W
D38Pc7oU8AAF_6F
D4CEUKUWwAAXzU8.jpg large
56899687_10161551169545557_6969110695807811584_n
D4AvZJ-WsAY9anz.jpg large
D4AgYJAW4AIs76t
D4At0coW0AAjYww
D38bIioXoAAOJvj.jpg large
D38nQu_WAAAnZWp
D39CDaCXsAEX5W_
D4CEUKUWwAAXzU8.jpg large
D38Pc7oU8AAF_6F
D383Fg6X4AEJwGN
D4CD2reW4AAQxbJ
D38RrItUcAAE8Nu
D39IqdeX4AAJr5W
D39PRU1XsAAY5eE
Screen Shot 2019-04-16 at 8.01.34 PM
D38O7YPUIAIhRVG
D38j-R1WwAIagUi
D39NrSDWwAAV8yW
D389SnbX4AE-Dvk
D38LFHlU8AEWi_f
D38mJdsWwAAY925
D38lS6LXoAAmRL6
D38lRySXsAAShRB
D4CD2rfXsAEi889
56549524_1223529091139485_8901038877343481856_n
Screen Shot 2019-04-16 at 8.01.34 PM
56862360_2324788081102513_3470547932690776064_n
56980635_10157139028667365_5838586333253074944_n
56899687_10161551169545557_6969110695807811584_n
56842770_685911931837631_1427788979075284992_n
Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image...

After my session, I went to the Afrinic session on IPv6, where Mukom Akong Tamon was presenting on IPv6 where he brushed out on an introduction to IPv6 and the IPv6 format structure. Also, several examples of why it is important to migrate to IPv6. Loganaden Velvindron from Afrinic enlightened the audience about dual stack programming.

One of the important part where Mr. Mukom mentioned that there are still developers hard coding IP addresses in the code which is not a good practice.

There was another session by Loganaden Velvindron of Afrinic, who spoke on NFtables in RedHat 8. Mukom was also present there in the session. Loganaden explained about NFtables architecture and its advantages. Also explained how to submit patches and dual stack building with NFtables.

Codarren Velvindron, technical lead at Orange Business Services and member of cyberstorm.mu explain why automation is important. He took some example on the conference.mscc.mu website itself. Also gave some ideas using “Expect”. For those who are not familiar with “Expect”, it is a scripting programming language that talks with your interactive programs or script that require user interaction.

Nathan Sunil Mangar also presented on an introduction to the STM32 microcontroller. He also gave some hints to distinguish between fake and real microcontrollers on the market. Apart from the basic introduction, he went brushed out some examples on several projects and explain which one can is better. However, it also depends on the budget when choosing microcontrollers. He also showed how to use the tool of programming for the STM32 microcontroller. The documentation was also perused during the presentation. At the end of the presentation, there were several giveaways by Nathan Mangar including, fans, Microcontrollers, and a small light bulb made from STM32.

I also have the opportunity to meet with several staffs from the Mauritius Commercial Bank who asked for some hints and best practice on Ansible. Also had some conversations with other people in the tech industry such as Kushal Appadu, Senior Linux system Engineer at Linkbynet Indian Ocean. We discussed lengthily on new technologies. Some days back, I presented the technicalities of Automation as a DevOps Engineer at SupInfo university Mauritius under the umbrella of Orange Cloud for Business and Orange Business Service. I was glad to meet a few students of SupInfo at the DevCon 2019 who instantly recognized me and congratulated me for the Ansible session.

Speaker at SUPINFO under the umbrella of Orange Business Services

I sincerely believe there is still room for improvement at the Developers conference such as the website itself which needs some security improvements. Otherwise, a feature that could be added is to specify which session is for beginners, intermediate or advanced so that attendees can choose better. The rating mechanism which is not based on constructivism might discourage other speakers to come forward next time. But overall, it was a nice event. Someone from the media team filmed me for a one-minute video, hoping to see it on the net in the future. I also got a “Thank You” board for being a speaker by Vanessa Veeramootoo-Chellen, CTO at Extension Interactive and one of the organizers at the Developers conference who can be seen to be always working, busy and on the move during the event.

My trip to Pension Cargo, Bras-Panon – Reunion Island

I had a splendid time in Reunion Island this week. I stayed at “Pension Cargo” which is owned by Christian, a friend at Bras-Panon, Reunion island in the north-east of the neighbor island thirty minutes away from Roland-Garros, St-Denis Airport. I reached there on Wednesday at around midnight. Christian was waiting for me as I told him that I will reach there by Taxi. I was so tired already and went to sleep.

On the next day, I went for a casual meeting on cybersecurity at a Media-Tech center as well as meeting with another acquaintance who is in the medical field. At St-Marie Media-Tech center, I got the opportunity to discuss Ansible, Automation technologies, Linux and TLS 1.3 for the IETF hackathons by the cyberstorm.mu team. It’s true that in Mauritius there is considerable effort to be made to do better Developers conferences, but, I seized this opportunity to build up the circle on the neighbor island which is also advanced in software development and Automation Engineering. The day was an enriching and successful one, however, time-consuming it was, I have been able to achieve my goal. I reached quite late at “Pension Cargo” and was so tired.

Pension Cargo
Pension Cargo

On Friday, I went for a nature walk near Pension Cargo. That place is still in its natural state with lots of exotic fruits.

Exotic fruits on the beach near Pension Cargo
Exotic fruits on the beach near Pension Cargo

In case you are a fan of the nature walk, I’m sure you would like it. It was indeed a nice time meeting up with several Linux user groups to discuss avenues of collaboration between Reunion and Mauritius. However, there are much more to see in Reunion island.

Random picture taken near Pension Cargo

IMG_2608
IMG_2608
IMG_2606
IMG_2591
IMG_2591
IMG_2591
IMG_2591
pensioncargo
IMG_2607
IMG_2607
IMG_2568
IMG_2568
IMG_2568
IMG_2568
IMG_2568
IMG_2568
IMG_2594
IMG_2596
IMG_2596
IMG_2596
IMG_2593
IMG_2566
IMG_2584
IMG_2573
IMG_2573
IMG_2573
IMG_2592
IMG_2592
IMG_2592
IMG_2606
IMG_2606
IMG_2606
IMG_2606
IMG_2607
IMG_2607
IMG_2567
Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image... Loading image...

At “Pension Cargo”, you can never miss the Reunion beer made locally. I got it free too! This is something to never miss at all!

Beer made locally in Reunion island
Beer made locally in Reunion island

As regards to breakfast, lunch and dinner, its always nice and yummy whether it is chicken, duck, and seafood which is always accompanied by salads french style-made.

Lunch with chicken, Seafood, Potatoes and Salads
Lunch with chicken, Seafood, Potatoes, and Salads

In case, you are heading up to Reunion island or in transit, feel free to check out “Pension Cargo” which is always worth the price. As usual, I convinced Christian to join and create a Twitter account as a marketing strategy which worth for where it is located. Pension Cargo can seem to be easily booked on Booking.com and Airbnb. However, you can always contact the Christian and his family who are always there to welcome you with a smile and lots of beers and goodies.

The team at the bar
The team at the bar

I got the opportunity to visit Riviere des Roches which is located a few meters from where I live. I was told that fishermen build up walls on the river with the aim to narrow the passage and by using fishing nets to catch “bisik” fish known as the caviar of the Reunion island which is pretty expensive and delicious. Unfortunately, this can be a problem to the ecology of the island as depending on the curve being built with the wall and the increase of sea water levels might change drastically the width of the river. This is contributing to severe land erosion and inviting sharks close to the river. Obviously, some people in the vicinity are not happy at all. Its all a question of eating the famous caviar of Reunion island!

Walls built on the river
Walls built on the river

Walls built on the river to catch fish

IMG_2620
IMG_2621
IMG_2621
Loading image... Loading image... Loading image...

On the way to Mauritius, the weather was sunny and I seized the opportunity to make a video for my YouTube Channel. I already had so many landings from several countries. Why not add Mauritius to the playlist?

My Travels

Mauritius – United Kingdom – United States – United Kingdom – Mauritius

That’s my first time traveling abroad! Living in the US is pretty fun but expensive too especially in New York. I had the opportunity to visit NASA space shuttle and been to some military vessels. That experience is unforgettable. It’s not every day that anyone gets the opportunity to see a Space Ship in front of you. I visited several places in the US and learned how to travel which is one of the most important and basic things if you want to be able to do it on your own. From Climbing to the top of the Empire State Building in New York and back to the shooting range in Connecticut was a pretty nice experience. Shopping around is the largest shopping malls is yet another feelings.

Mauritius – Kenya – Ivory Coast – Senegal – Mali – South Africa – Mauritius

The trip to Dakar, Senegal was yet another experience of the West African vibes. Learning cultures of Dakar’s people is something that anyone should know. I got the opportunity to talk with many people in Dakar. They are very nice people and always polite. If you want to see Baobab trees, it can be seen almost everywhere even on the street of Dakar. Training and facilitating the African Internet Summit 2018 hackathon was my objective and I met not only people from the west of Africa but several other countries. I am glad to be able to meet many influential people who contribute to the Internet in Africa.

Mauritius – Hungary – Germany – Tunisia – Germany – Mauritius

Situated in the north of Africa, Tunisia had impressed me a lot especially at Yasmine Hammamet in Tunisia. I was staying at Diar Lemdina hotel and I should admit that the atmosphere was pretty calm. When it comes to tea, those people are good at making it alongside smoking Shish-a at the bar. Hard Alcoholic is a bit scarce there, but if you want to get drunk, they brew good wine which is just tasty. My aim in Tunisia as a speaker was to talk about DNS, DNSSEC and DNS Flag day. I also had the opportunity to meet more influential people contributing to the development of the Internet in Africa. We discussed a lot about Internet Policies in Africa. Besides, I had nice days in Frankfurt and Munich in Germany.

Mauritius –Reunion Island – Mauritius

Reunion island the neighbor island of Mauritius which is roughly less than 40 minutes. I should admit that the people of Reunion island are very good people and courteous towards me. I did not expect such a kind welcome from the people there. My goal was to participate in a conference at a media-tech sharing several technologies such as Ansible and other automation tools. I was also there for some personal meeting with a friend who is in the medical field. We discussed a lot about AI in the medical field.

 

Mauritius – Kenya – South-Africa – Kenya  – Mauritius

I was in Sandton, Johannesburg, South Africa. Sandton is very safe and so huge at the same time. It was a business trip. I seized the occasion to visit Nelson Mandela Square which is a very interesting place to hang around. You can also view a small video that I uploaded on my Youtube Channel at the Nelson Mandela Square. 

 

Mauritius – South Africa – USA

I landed in the United States on 08/08/2019 as a resident of the United States of America. It is all a new life and the start of a new adventure 🙂

Next flight can be anywhere around the world… Follow me here on my Twitter account @TheTunnelix

Linux Performance & Analysis – Strace and syscall

A quick look at the manual of Strace would show you an indication that the strace command is used to trace system calls and signals. The desciption part stipulates that “In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.”

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

Photo credits: Linuxintro.org
Photo credits: Linuxintro.org

However, there are much more than that to discover. Since strace uses ptrace which observe and control execution of another process and examination of memory and registers. In some way, strace can be dangerous because signal injection and suppression may occur. The debugging mechanism is dangerous as it pause the target process for syscalls to read the state – ptrace(PTRACE_restart, pid, 0, sig)

Proof of concept strace can be dangerous

From the example below we can see the time taken copied is much slower compared with a strace.

[[email protected] ~]# dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 0.38371 s, 1.

[[email protected] ~]# strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 16.9985 s, 36.1 kB/s
+++ exited with 0 +++
6 MB/s

The 12 main syscalls

There are 12 main syscalls worth learning to grasp output of strace

SyscallDescription
readread bytes from a file descriptor (file and socket)
writewrite bytes from a file descriptor (file and socket)
openopen a file (returns a descriptor)
closeclose the file descriptor
forkcreate a new process (current process is forked)
execexecute a new program
connectconnect to a network host
acceptaccept a network connection
statread files statistics
ioctlset IO properties and other functions
mmapmap a file to the process memory address space
brkextend the heap pointer

Strace output analysis

I will now take a strace example. I have created a file test in /tmp. You can check out the strace ouput at this link http://pastebin.com/zziCAwDz. Let’s analyse it.

We can noticed the following at the beginning

  1. execve(“/bin/ls”, [“ls”, “-l”, “/etc”], [/* 22 vars */]) = 0
  2. brk(0)                                  = 0x8ca8000
  3. mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7791000
  4. access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
  5. open(“/etc/ld.so.cache”, O_RDONLY)      = 3
  6. fstat64(3, {st_mode=S_IFREG|0644, st_size=25200, …}) = 0
  7. mmap2(NULL, 25200, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb778a000

The execve() variant is running /bin/ls then libraries are called in the variant followed by several libraries from /lib directory. After the file descriptor is close with the close() function, you will noticed at line 10 there is a open(“/etc/ld.so.cache”, O_RDONLY)  = 3 which means that whilst opening the /etc it has returned a value 3, a file descriptor for later use with other syscalls.

You will noticed that the content of /etc is being read, then for each file inside /etc it calls lstat() vand stat() variant. Two extended attribute varients are also called that are lgetxattr() and getxattr() and finally ls -l start printing out the results. But hey! Did you noticed that ls is running /etc/localtime on every output? stat64(“/etc/localtime”, {st_mode=S_IFREG|0644, st_size=239, …}) = 0 is being called each time!

Some strace commands

#Slow the target command and print details for each syscall: strace command

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

$Slow the target PID and print details for each syscall: strace -p PID

# Slow the target PID and any newly created child process, printing syscall details: strace -fp PID

# Slow the target PID and record syscalls, printing a summary: strace -cp PID

# Slow the target PID and trace open() syscalls only: strace -eopen -p PID

# Slow the target PID and trace open() and stat() syscalls only: strace -eopen,stat -p PID

# Slow the target PID and trace connect() and accept() syscalls only: strace -econnect,accept -p PID

# Slow the target command and see what other programs it launches (slow them too!): strace -qfeexecve command

# Slow the target PID and print time-since-epoch with (distorted) microsecond resolution: strace -ttt -p PID

# Slow the target PID and print syscall durations with (distorted) microsecond resolution: strace -T -p PID

From what we can understand is that if /etc/localtime is being run each time, it is consuming more resource and heavily interrupting the system. So, strace is based on rather simple syscalls, however, it can also cause heavy performance overhead.

I have created a new tag called Linux Performance. This article does not give a clear overview of strace in itself. Some more articles coming later on Linux performance, analysis and tuning.