Tag: C

DevConMru – Backup in the cloud for the Paranoid by cyberstorm.mu

At Cyberstorm Mauritius we work on several projects and code for fun. One of the interesting projects we have look at is an application called Tarsnap which is used to perform a secure backup on the cloud. At Cyberstorm Mauritius, myself (@TheTunnelix) and Codarren (@Devildron) recently send codes to Tarsnap and same were approved. That’s really cool when someone’s code is approved and used worldwide by thousands of companies. Today, I have the privilege to speak on Tarsnap at the DevConMru 2016 which was held at Voila hotel, Bagatelle. On reaching there, I was impressed by the number of people already waiting inside the conference room who were curious about Tarsnap. Some were entrepreneurs whilst others were students. I should say around 30 people attended the conference. Since it was a Sunday at 11:30 am, the team did not hesitate to bring some beer to the little crowd present there. I was busy setting up my laptop for the presentation.

As usual, I like to get the attention of my audience before the presentation. My first slide showed the logo of Tarsnap upside down.

Screenshot from 2016-05-22 19-05-41

Everyone was turning their head and making the effort to read the content. And here we go. I noticed that they are all ready and curious about it.

Check out the Slide here. Please wait some minutes. It’s loading…

The basics of Tarsnap were explained. Tarsnap take streams of archive data and splits then into variable-length blocks. Those blocks are compared and any duplicate blocks are removed. Data de-duplication happens before its uploaded to the Tarsnap server. Tarsnap does not create Temporary files but instead create a cache file on the client. The cache file is the files that are being back up to the Tarsnap server. After deduplication, the data is then compressed, encrypted, signed and send to the Tarsnap server. I also explained that the archived are saved on an Amazon S3 with EC2 server to handle it. Another interesting point raised was the concept of Tarsnap which uses smart Rsync-like block oriented snapshot operations that upload only data which is charged to minimize transmission costs. One does not need to trust any vendor cryptographic claims and you have full access to the source codes which uses open-source libraries and industry-vetted protocols such as RSA, AES, and SHA.

Getting on to the other part of Tarsnap and Bandwidth, an emphasis was made on Tarsnap which synchronized blocks of data using a very intelligent algorithm. Nowadays, there are companies that still use tapes for backups. Imagine having so many tapes and when restoration time has arrived, this would take tremendous time. Tarsnap compresses, encrypts and cryptographically signs every byte you send to it. No knowledge of cryptographic protocols is required. At this point, I asked a question about volunteers who are thinking to look at the Tarsnap code. There were three persons who raised their hands. The importance of the key file was raised up as some companies secure their private key in a safe. Tarsnap also supports the division of responsibilities where an explanation was laid out where a particular key can only be used to create an archive and not delete them.

An analogy between google drive compared to Tarsnap was given. Many already understood the importance of Tarsnap compared to Google Drive. The concept of deduplication was explained using examples. For the network enthusiasts, I laid emphasis on the port 9279 which should not be blocked on the firewall as Tarsnap runs on the following port number. Coming to confidentiality, the matter was made clear enough to the audience how much the data is secured. If it happens someone lost the key there is no way of getting back the data. 

Tarsnap is not an open source product. However, their client code is open to learn, break and study. I laid emphasis on the reusable open source components that come with Tarsnap, for example, the Script KDF (Key derivation function). KDF derives one or more secret keys from a secret value such as a master key, a password or passphrase or using a pseudo-random function. The Kivaloo data store was briefly explained. Its a collection of utilities which together form a data store associating keys up to 255 bytes with a value up to 255 bytes. Writes are accepted until data has been synced. If A completed before B, B will see the results of A. The SPIPED secure pipe daemon which is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses so that one may connect to one address. 

I also explained to the audience the pricing mechanism which was perceived rather cheap for its security and data deduplication mechanisms. Tarsnap pricing works similarly as a prepaid utility-metered model. A deposit of $5 is needed. Many were amazed when I told them that the balance is a track to 18 decimal places. Prices are paid exactly what is consumed.

Other interesting features such as regular expression support and interesting kinds of stuff with the dry run features of Tarsnap was given. The concept of Tar command compared to Tarsnap was also explained. Commands, hints, and tricks explained.

At the end, i consider it really important to credit Colin, the author of Tarsnap and i have been strongly inspired by the work of Michael Lucas on Tarsnap. Indeed, another great achievement of Cyberstorm Mauritius at the DevConMru 2016.

Linux Performance & Analysis – Strace and syscall

A quick look at the manual of Strace would show you an indication that the strace command is used to trace system calls and signals. The desciption part stipulates that “In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.”

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

Photo credits: Linuxintro.org
Photo credits: Linuxintro.org

However, there are much more than that to discover. Since strace uses ptrace which observe and control execution of another process and examination of memory and registers. In some way, strace can be dangerous because signal injection and suppression may occur. The debugging mechanism is dangerous as it pause the target process for syscalls to read the state – ptrace(PTRACE_restart, pid, 0, sig)

Proof of concept strace can be dangerous

From the example below we can see the time taken copied is much slower compared with a strace.

[[email protected] ~]# dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 0.38371 s, 1.

[[email protected] ~]# strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 16.9985 s, 36.1 kB/s
+++ exited with 0 +++
6 MB/s

The 12 main syscalls

There are 12 main syscalls worth learning to grasp output of strace

Syscall Description
read read bytes from a file descriptor (file and socket)
write write bytes from a file descriptor (file and socket)
open open a file (returns a descriptor)
close close the file descriptor
fork create a new process (current process is forked)
exec execute a new program
connect connect to a network host
accept accept a network connection
stat read files statistics
ioctl set IO properties and other functions
mmap map a file to the process memory address space
brk extend the heap pointer

Strace output analysis

I will now take a strace example. I have created a file test in /tmp. You can check out the strace ouput at this link http://pastebin.com/zziCAwDz. Let’s analyse it.

We can noticed the following at the beginning

  1. execve(“/bin/ls”, [“ls”, “-l”, “/etc”], [/* 22 vars */]) = 0
  2. brk(0)                                  = 0x8ca8000
  3. mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7791000
  4. access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
  5. open(“/etc/ld.so.cache”, O_RDONLY)      = 3
  6. fstat64(3, {st_mode=S_IFREG|0644, st_size=25200, …}) = 0
  7. mmap2(NULL, 25200, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb778a000

The execve() variant is running /bin/ls then libraries are called in the variant followed by several libraries from /lib directory. After the file descriptor is close with the close() function, you will noticed at line 10 there is a open(“/etc/ld.so.cache”, O_RDONLY)  = 3 which means that whilst opening the /etc it has returned a value 3, a file descriptor for later use with other syscalls.

You will noticed that the content of /etc is being read, then for each file inside /etc it calls lstat() vand stat() variant. Two extended attribute varients are also called that are lgetxattr() and getxattr() and finally ls -l start printing out the results. But hey! Did you noticed that ls is running /etc/localtime on every output? stat64(“/etc/localtime”, {st_mode=S_IFREG|0644, st_size=239, …}) = 0 is being called each time!

Some strace commands

#Slow the target command and print details for each syscall: strace command

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

$Slow the target PID and print details for each syscall: strace -p PID

# Slow the target PID and any newly created child process, printing syscall details: strace -fp PID

# Slow the target PID and record syscalls, printing a summary: strace -cp PID

# Slow the target PID and trace open() syscalls only: strace -eopen -p PID

# Slow the target PID and trace open() and stat() syscalls only: strace -eopen,stat -p PID

# Slow the target PID and trace connect() and accept() syscalls only: strace -econnect,accept -p PID

# Slow the target command and see what other programs it launches (slow them too!): strace -qfeexecve command

# Slow the target PID and print time-since-epoch with (distorted) microsecond resolution: strace -ttt -p PID

# Slow the target PID and print syscall durations with (distorted) microsecond resolution: strace -T -p PID

From what we can understand is that if /etc/localtime is being run each time, it is consuming more resource and heavily interrupting the system. So, strace is based on rather simple syscalls, however, it can also cause heavy performance overhead.

I have created a new tag called Linux Performance. This article does not give a clear overview of strace in itself. Some more articles coming later on Linux performance, analysis and tuning.

CVE-2016-0777 – Are you still vulnerable to this OpenSSH Vulnerability?

I was quite surprised to notice that though, since the 14th of January 2016, the OpenSSH has released a patch to correct a major bug issue for those using the OpenSSH remote connectivity login tool many IT professionals have forgotten about that issue. The security vulnerability has been announced on the Common Vulnerabilities and Exposures (CVE) website since the middle of January, yet many systems and security administrators may have not missed this information.

Credits: openssh.com
Credits: openssh.com

As mentioned, the ssh bug is about “The resend_bytes function in roaming_common.c in the client in OpenSSH 5.x, 6.x, and 7.x before 7.1p2 allows remote servers to obtain sensitive information from process memory by requesting transmission of an entire buffer, as demonstrated by reading a private key. “ To be more explicit, what really happens is that an attacker can guess the client’s get_snd_buf_size() in order to control out_buf_size(). In order to exploit this vulnerability, the attacker will have to force the malloc() to return free()d and an uncleaned chunk of sensitive information. 

This block of code on the old roaming_common.c clearly demonstrate the issue. View full code here.

resend_bytes(int fd, u_int64_t *offset)
	size_t available, needed;

	if (out_start < out_last)
		available = out_last - out_start;
		available = out_buf_size;
	needed = write_bytes - *offset;
	debug3("resend_bytes: resend %lu bytes from %llu",
	    (unsigned long)needed, (unsigned long long)*offset);
	if (needed > available)
		fatal("Needed to resend more data than in the cache");
	if (out_last < needed) {
		int chunkend = needed - out_last;
		atomicio(vwrite, fd, out_buf + out_buf_size - chunkend,
		atomicio(vwrite, fd, out_buf, out_last);
	} else {
		atomicio(vwrite, fd, out_buf + (out_last - needed), needed);

You can perform some test on your machine or may be you are using an emulator such as MobaXterm to SSH on several servers by launching the following command

ssh whoami.filippo.io

You might encounter the following beautiful message:

         ***** WARNING ***** WARNING *****
You have roaming turned on. If you are using OpenSSH, that most likely means you are vulnerable to the CVE-2016-0777 information leak.

In case, this is true, you might want to secure yourself from this vulnerability by editing the /etc/ssh/ssh_config and pass the following parameter UseRoaming no . Of course, you will need to reload the ssh daemon and don’t forget to perform a new test!

Pointers and Dynamic Memory – Heap

Yesterday, I wrote an article on Pointers and Dynamic Memory – Stack and Stack Overflow of Memory and I received good feedback about the article. More articles are requested by the fellow fans of cyberstorm.mu and I proposed to blog about “Heap”. As I discussed in the previous article, there are limitations to stack. We have to see exactly how stack overflow happens, that is, if our “calls” stack grows beyond the reserved memory of the stack, the program will consequently crash.

There are actually limitations to stack especially when the memory is set aside and it does not grow during runtime. Consequently, the application cannot request more memory. Note that the allocation and deallocation of memory on the stack are governed by a set of rules, that is, when a function is called, it is pushed onto the stack and when it finishes it is popped out of the stack. Hence, it is quite impossible to manipulate the aim of a variable when it is on the stack. Another limitation is when there is a large data type. For example. an array sets as a local variable where we need to know the size of the array at the compiling time! To overcome this problem of a large variable into the memory we have “heap”. Let’s now have some fun with heap?

Compared to size in  Stack, an application’s heap is not fixed. Its size can vary during the whole lifetime of the application and there are no special rules applied during allocation and deallocation memory. Heap will grow its reserved memory as long as you do run out of memory in the whole system itself. That is why in the introduction article, the heap is described as a free pool of memory (Dynamic memory). To use Dynamic Memory there are four functions that need to be grasped, that is; Malloc, Calloc, realloc, and Free. I will get into more details of these terms in my next article.

Let’s see a C program to get into the heap structure:

int main()
 int a;
 int *p;
 p = (int*)malloc( sizeof (int));
 *p = 10;
 p = (int*)malloc( sizeof (int));
 *p = 20;
Follow the numbers with the explanation below
  1. A variable “a” is declared from the main() method. It goes into the stack as explained in the diagram.
  2. We store the integer on the heap by calling the malloc() function. The line int *p; followed by p = (int*) malloc(sizeof (int)); will specify how much memory to allocate on the heap in terms of bytes. We just say “malloc please pass as argument the size of the integer..” and the program said “ok I give you a block of memory of 4 bytes” – 4 bytes is the size of an integer. Malloc will then return a pointer to the starting address of this block.
  3. Malloc will return a void pointer to the starting address of the block. Let’s say the starting address of the 4 bytes is 100.
  4. Since we have an integer p which is a local variable to main, this is allocated in the stack. Note that at this stage p stores the address 100 in the heap.
  5. We can store something in the heap by removing the reference pointer p towards the heap. This done by using pointer p and add some value –  *p = 10;
  6. A pointer variable is also kept in the stack to point to the heap that is 100.
  7. At the line, p = (int*) malloc ( sizeof (int));  and *p=20 which is another call has been made to malloc and consequently one more byte of 4 bytes is allocated on the heap.
  8. Let’s say the address is 200 and the address called is stored in the variable p which will point to the address 200 from the stack. Note that the previous block of address 100 will still sit in the heap. Unnecessary consumptions of memory at this stage on the heap need to be clear out.
  9. This exactly where the line free (p); gets in to clear out. Number 3 in the heap block will be cleared out.

Anything allocated on the heap is not automatically deallocated when the functions get completed compared to the stack. It does not need to live the whole lifetime of the application. Programs which is coded with this fault can be used to explore vulnerabilities. We need to control anything that can be freed on the heap.

Pointers and Dynamic Memory – Stack and StackOverflow

From the article Pointers and Dynamic Memory – Part1, I made a little introduction of Pointers and a brief overview of Dynamic memory. In this article, we will get into some more details of Stack and StackOverflow in the context of Pointers and Dynamic memory.

Let’s see a simple C program which we can use to see what happens in memory when it executes

 int total;
 int square(int x)
     return x*x;
 int SquareSum( int x, int y)
      int z = square(x+y);
      return z;
 int main()
      int a=10, int b=20;
      total = SquareSum(a,b);
      printf(“Output = %d”, total);

From the program, we can see that there is a function that gives the Square of a number and another function that gives the square of the sum. In the Main method, the square of the sum is passed through the arguments a and b.

Like I mentioned in the previous article, Stack is reserved for function calls and local variables and Static means the global variables that are not stored inside the variables. From the C code, this can be interpreted under Stack and Static.

  1. When the main method is invoked, some amount of the memory is allocated to the Stack for execution of main() as described in the Yellow box.
  2. The total is a global variable so it gets into the Static section.
  3. All local variables, arguments and the information where the function main should return are stored within the stack frame. The size of the stack frame for a method is calculated when the program is compiling.
  4. Now, when main calls SquareSum method, then a stack frame is allocated for the call SquareSum.
  5. All the variables x, y, and z will sit in the same stack frame.
  6. The function sum of the square that is “square” will also be found in the Stack frame and it will have its own local variables.

When the program executes, the function at the top of the stack keeps on executing the rest are in a “pause mode” waiting for other function to return something before it resumes execution. When the whole program finishes executing, the stack and static frame get cleared. Remember, a global variable should only be set with the aim to use it for a whole lifetime of the program, otherwise, it will be a waste of memory. The global variable used in this program has been done deliberately to understand the concept.

Another interesting issue is that when the program starts, by default, the operating system will reserve some amount of space on the stack. When there are functions in a stack calling other functions until there is no memory available in the stack, its called “StackOverflow” which consequently crashes the whole program. For example, a bad program having bad recursion codes. In brief, there are limitations to stack.