Tag: Linux Performance

DevConMru – Backup in the cloud for the Paranoid by cyberstorm.mu

At Cyberstorm Mauritius we work on several projects and code for fun. One of the interesting projects we have look at is an application called Tarsnap which is used to perform a secure backup on the cloud. At Cyberstorm Mauritius, myself (@TheTunnelix) and Codarren (@Devildron) recently send codes to Tarsnap and same were approved. That’s really cool when someone’s code is approved and used worldwide by thousands of companies. Today, I have the privilege to speak on Tarsnap at the DevConMru 2016 which was held at Voila hotel, Bagatelle. On reaching there, I was impressed by the number of people already waiting inside the conference room who were curious about Tarsnap. Some were entrepreneurs whilst others were students. I should say around 30 people attended the conference. Since it was a Sunday at 11:30 am, the team did not hesitate to bring some beer to the little crowd present there. I was busy setting up my laptop for the presentation.

As usual, I like to get the attention of my audience before the presentation. My first slide showed the logo of Tarsnap upside down.

Screenshot from 2016-05-22 19-05-41

Everyone was turning their head and making the effort to read the content. And here we go. I noticed that they are all ready and curious about it.

Check out the Slide here. Please wait some minutes. It’s loading…

The basics of Tarsnap were explained. Tarsnap take streams of archive data and splits then into variable-length blocks. Those blocks are compared and any duplicate blocks are removed. Data de-duplication happens before its uploaded to the Tarsnap server. Tarsnap does not create Temporary files but instead create a cache file on the client. The cache file is the files that are being back up to the Tarsnap server. After deduplication, the data is then compressed, encrypted, signed and send to the Tarsnap server. I also explained that the archived are saved on an Amazon S3 with EC2 server to handle it. Another interesting point raised was the concept of Tarsnap which uses smart Rsync-like block oriented snapshot operations that upload only data which is charged to minimize transmission costs. One does not need to trust any vendor cryptographic claims and you have full access to the source codes which uses open-source libraries and industry-vetted protocols such as RSA, AES, and SHA.

Getting on to the other part of Tarsnap and Bandwidth, an emphasis was made on Tarsnap which synchronized blocks of data using a very intelligent algorithm. Nowadays, there are companies that still use tapes for backups. Imagine having so many tapes and when restoration time has arrived, this would take tremendous time. Tarsnap compresses, encrypts and cryptographically signs every byte you send to it. No knowledge of cryptographic protocols is required. At this point, I asked a question about volunteers who are thinking to look at the Tarsnap code. There were three persons who raised their hands. The importance of the key file was raised up as some companies secure their private key in a safe. Tarsnap also supports the division of responsibilities where an explanation was laid out where a particular key can only be used to create an archive and not delete them.

An analogy between google drive compared to Tarsnap was given. Many already understood the importance of Tarsnap compared to Google Drive. The concept of deduplication was explained using examples. For the network enthusiasts, I laid emphasis on the port 9279 which should not be blocked on the firewall as Tarsnap runs on the following port number. Coming to confidentiality, the matter was made clear enough to the audience how much the data is secured. If it happens someone lost the key there is no way of getting back the data. 

Tarsnap is not an open source product. However, their client code is open to learn, break and study. I laid emphasis on the reusable open source components that come with Tarsnap, for example, the Script KDF (Key derivation function). KDF derives one or more secret keys from a secret value such as a master key, a password or passphrase or using a pseudo-random function. The Kivaloo data store was briefly explained. Its a collection of utilities which together form a data store associating keys up to 255 bytes with a value up to 255 bytes. Writes are accepted until data has been synced. If A completed before B, B will see the results of A. The SPIPED secure pipe daemon which is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses so that one may connect to one address. 

I also explained to the audience the pricing mechanism which was perceived rather cheap for its security and data deduplication mechanisms. Tarsnap pricing works similarly as a prepaid utility-metered model. A deposit of $5 is needed. Many were amazed when I told them that the balance is a track to 18 decimal places. Prices are paid exactly what is consumed.

Other interesting features such as regular expression support and interesting kinds of stuff with the dry run features of Tarsnap was given. The concept of Tar command compared to Tarsnap was also explained. Commands, hints, and tricks explained.

At the end, i consider it really important to credit Colin, the author of Tarsnap and i have been strongly inspired by the work of Michael Lucas on Tarsnap. Indeed, another great achievement of Cyberstorm Mauritius at the DevConMru 2016.

Linux Performance & Analysis – Strace and syscall

A quick look at the manual of Strace would show you an indication that the strace command is used to trace system calls and signals. The desciption part stipulates that “In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.”

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

Photo credits: Linuxintro.org
Photo credits: Linuxintro.org

However, there are much more than that to discover. Since strace uses ptrace which observe and control execution of another process and examination of memory and registers. In some way, strace can be dangerous because signal injection and suppression may occur. The debugging mechanism is dangerous as it pause the target process for syscalls to read the state – ptrace(PTRACE_restart, pid, 0, sig)

Proof of concept strace can be dangerous

From the example below we can see the time taken copied is much slower compared with a strace.

[[email protected] ~]# dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 0.38371 s, 1.

[[email protected] ~]# strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 16.9985 s, 36.1 kB/s
+++ exited with 0 +++
6 MB/s

The 12 main syscalls

There are 12 main syscalls worth learning to grasp output of strace

readread bytes from a file descriptor (file and socket)
writewrite bytes from a file descriptor (file and socket)
openopen a file (returns a descriptor)
closeclose the file descriptor
forkcreate a new process (current process is forked)
execexecute a new program
connectconnect to a network host
acceptaccept a network connection
statread files statistics
ioctlset IO properties and other functions
mmapmap a file to the process memory address space
brkextend the heap pointer

Strace output analysis

I will now take a strace example. I have created a file test in /tmp. You can check out the strace ouput at this link http://pastebin.com/zziCAwDz. Let’s analyse it.

We can noticed the following at the beginning

  1. execve(“/bin/ls”, [“ls”, “-l”, “/etc”], [/* 22 vars */]) = 0
  2. brk(0)                                  = 0x8ca8000
  3. mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7791000
  4. access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
  5. open(“/etc/ld.so.cache”, O_RDONLY)      = 3
  6. fstat64(3, {st_mode=S_IFREG|0644, st_size=25200, …}) = 0
  7. mmap2(NULL, 25200, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb778a000

The execve() variant is running /bin/ls then libraries are called in the variant followed by several libraries from /lib directory. After the file descriptor is close with the close() function, you will noticed at line 10 there is a open(“/etc/ld.so.cache”, O_RDONLY)  = 3 which means that whilst opening the /etc it has returned a value 3, a file descriptor for later use with other syscalls.

You will noticed that the content of /etc is being read, then for each file inside /etc it calls lstat() vand stat() variant. Two extended attribute varients are also called that are lgetxattr() and getxattr() and finally ls -l start printing out the results. But hey! Did you noticed that ls is running /etc/localtime on every output? stat64(“/etc/localtime”, {st_mode=S_IFREG|0644, st_size=239, …}) = 0 is being called each time!

Some strace commands

#Slow the target command and print details for each syscall: strace command

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

$Slow the target PID and print details for each syscall: strace -p PID

# Slow the target PID and any newly created child process, printing syscall details: strace -fp PID

# Slow the target PID and record syscalls, printing a summary: strace -cp PID

# Slow the target PID and trace open() syscalls only: strace -eopen -p PID

# Slow the target PID and trace open() and stat() syscalls only: strace -eopen,stat -p PID

# Slow the target PID and trace connect() and accept() syscalls only: strace -econnect,accept -p PID

# Slow the target command and see what other programs it launches (slow them too!): strace -qfeexecve command

# Slow the target PID and print time-since-epoch with (distorted) microsecond resolution: strace -ttt -p PID

# Slow the target PID and print syscall durations with (distorted) microsecond resolution: strace -T -p PID

From what we can understand is that if /etc/localtime is being run each time, it is consuming more resource and heavily interrupting the system. So, strace is based on rather simple syscalls, however, it can also cause heavy performance overhead.

I have created a new tag called Linux Performance. This article does not give a clear overview of strace in itself. Some more articles coming later on Linux performance, analysis and tuning.

Repair your Kernel Panic with Dracut

If you have encountered a Kernel Panic which usually happens after a major change in the Linux System, you can follow these procedures to rebuild the Kernel files with Dracut tools.

  1. Boot the server on rescue mode or simply through a live CD or ISO.
  2. To boot the server on rescue mode login on the Vsphere Interface and look for a live CD. In case of Kernel Panic on your own machine, you can boot your machine with a live CD.
  3. Once booted, create a directory in the folder /mnt
    mkdir /mnt/sysimage
  4. Use fdisk –l to find where is the /boot. However, you can also create another directory in mnt to mount different partitions. [sysimage is just a name given]
  5. Mount the disk into sysimage with the aim to mount the boot file. In my case, the sda1 is the boot partition
    mount /dev/sda2 /mnt/sysimage
    mount/dev/sda1 /mnt/sysimage/boot
  6. Once the disks are mounted mount the proc/dev/ and sys folders. Here are the commands:
    mount - -bind /proc /mnt/sysimage/proc
    mount - -bind /dev /mnt/sysimage/dev
    mount - -bind/sys /mnt/sysimage/sys
  7. After the mount operations have been carried out, you need to access the directory by chrooting into it.
    chroot /mnt/sysimage
  8. Get into the directory sysimage 
  9. You can back up the /boot to another location and use the command Dracut to regenerate anew the file initramfs. An example is as follows: 
    dracut -f /boot/initramfs-2.6.32-358.el6.x86_64.img 2.6.32-358.el6.x86_64
  10. You can umount all partitions and /or simply reboot the machine.

Repair your Kernel Panic with Dracut 1


    • On Vcenter, you may need to boot go through the BIOS interface first before being able to boot through the ISO and force the BIOS screen to appear on your screen.
    • You may also use the Finnix ISO which is usually compatible with all UNIX system.
    • When firing the dracut command make sure you only paste the kernel version with the architecture. Do not use the file .img extension, otherwise, it won’t work – Step9
    • The last part ‘2.6.32-358.el6.x86_64’ is just the same version which needs to be regenerated. -Step9
    • To know which kernel version your machine is actually using, you need to get into the grub folder and look for the grub.conf. The first option is usually the kernel used by default.
    • Sometimes, you need to try with the same version of the OS, it may happen after you have boot your machine with a live CD, the ISO which you have used do not detect your disk or the data store. You may, for example, think the disk is not good or there is a problem in the SAN.
    • However, without doing a root cause analysis, you cannot be certain if by repairing the initrd the Kernel Panic might be the unique solution. There are circumstances where a mounted NFS is not the same version with the actual machine which can result in Kernel Panic. The Dracut solution is not a definite solution.
  • Always investigate on the Dmesg log if possible or the crash dump if same has been set up.