Tag: linux

Linux Performance & Analysis – Strace and syscall

A quick look at the manual of Strace would show you an indication that the strace command is used to trace system calls and signals. The desciption part stipulates that “In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.”

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

Photo credits: Linuxintro.org
Photo credits: Linuxintro.org

However, there are much more than that to discover. Since strace uses ptrace which observe and control execution of another process and examination of memory and registers. In some way, strace can be dangerous because signal injection and suppression may occur. The debugging mechanism is dangerous as it pause the target process for syscalls to read the state – ptrace(PTRACE_restart, pid, 0, sig)

Proof of concept strace can be dangerous

From the example below we can see the time taken copied is much slower compared with a strace.

[[email protected] ~]# dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 0.38371 s, 1.

[[email protected] ~]# strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=600k
614400+0 records in
614400+0 records out
614400 bytes (614 kB) copied, 16.9985 s, 36.1 kB/s
+++ exited with 0 +++
6 MB/s

The 12 main syscalls

There are 12 main syscalls worth learning to grasp output of strace

SyscallDescription
readread bytes from a file descriptor (file and socket)
writewrite bytes from a file descriptor (file and socket)
openopen a file (returns a descriptor)
closeclose the file descriptor
forkcreate a new process (current process is forked)
execexecute a new program
connectconnect to a network host
acceptaccept a network connection
statread files statistics
ioctlset IO properties and other functions
mmapmap a file to the process memory address space
brkextend the heap pointer

Strace output analysis

I will now take a strace example. I have created a file test in /tmp. You can check out the strace ouput at this link http://pastebin.com/zziCAwDz. Let’s analyse it.

We can noticed the following at the beginning

  1. execve(“/bin/ls”, [“ls”, “-l”, “/etc”], [/* 22 vars */]) = 0
  2. brk(0)                                  = 0x8ca8000
  3. mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7791000
  4. access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
  5. open(“/etc/ld.so.cache”, O_RDONLY)      = 3
  6. fstat64(3, {st_mode=S_IFREG|0644, st_size=25200, …}) = 0
  7. mmap2(NULL, 25200, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb778a000

The execve() variant is running /bin/ls then libraries are called in the variant followed by several libraries from /lib directory. After the file descriptor is close with the close() function, you will noticed at line 10 there is a open(“/etc/ld.so.cache”, O_RDONLY)  = 3 which means that whilst opening the /etc it has returned a value 3, a file descriptor for later use with other syscalls.

You will noticed that the content of /etc is being read, then for each file inside /etc it calls lstat() vand stat() variant. Two extended attribute varients are also called that are lgetxattr() and getxattr() and finally ls -l start printing out the results. But hey! Did you noticed that ls is running /etc/localtime on every output? stat64(“/etc/localtime”, {st_mode=S_IFREG|0644, st_size=239, …}) = 0 is being called each time!

Some strace commands

#Slow the target command and print details for each syscall: strace command

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

$Slow the target PID and print details for each syscall: strace -p PID

# Slow the target PID and any newly created child process, printing syscall details: strace -fp PID

# Slow the target PID and record syscalls, printing a summary: strace -cp PID

# Slow the target PID and trace open() syscalls only: strace -eopen -p PID

# Slow the target PID and trace open() and stat() syscalls only: strace -eopen,stat -p PID

# Slow the target PID and trace connect() and accept() syscalls only: strace -econnect,accept -p PID

# Slow the target command and see what other programs it launches (slow them too!): strace -qfeexecve command

# Slow the target PID and print time-since-epoch with (distorted) microsecond resolution: strace -ttt -p PID

# Slow the target PID and print syscall durations with (distorted) microsecond resolution: strace -T -p PID

From what we can understand is that if /etc/localtime is being run each time, it is consuming more resource and heavily interrupting the system. So, strace is based on rather simple syscalls, however, it can also cause heavy performance overhead.

I have created a new tag called Linux Performance. This article does not give a clear overview of strace in itself. Some more articles coming later on Linux performance, analysis and tuning.

Hackathon in Mauritius – Operation SAD

We did it! The hackathon in Mauritius with title Operation SAD – Search and Destroy is a success. Who are the cyberstorm.mu winning team? Well, its a group of Linux developers residing in Mauritius who are happily improving the security of Linux An audit is first performed on the Linux platform and several security vulnerabilities are fixed. That is how the name “Search and Destroy” gets its name. It was proudly hosted by ISVTEC, a Linux firm operating in Mauritius.

Hackathon in Mauritius - Operation SAD 1

DAY 0 – A briefing was carried out. We started at around 09.30 hrs setting our laptops. I always feel the need for a second screen to save time. The briefing was carried quickly on a whiteboard. I am happy how all the necessary tools were provided at the ISVTEC conference room. More details on this link by LoganOperation SAD – DAY 0

DAY 1 – The hacking team were so fast. Everyone reached ISVTEC more early. It was so intense that I almost forgot lunchtime. All codes were submitted and some reviews were extremely fast. A whiteboard was of great help as soon as a project is completed we are moving to the other one. You can check out another article for Day 1 by Logan – Operation SAD – Day 1

A debrief session was carried out by Logan after the patches were submitted. I would sincerely thank Cyril and of course the staff of ISVTEC who has welcomed us since the very first day of the Hackathon. Indeed, an immense success from the winning team.

As announced some days back, cyberstorm Mauritius team will be at Flying DODO Bagatelle conference room as from 13:00 hrs to give a resume about the work done. We have the pleasure to hear from Avinash Meetoo, CEO of Knowledge7 for a superb speech.

Analysing an attack from WordPress Hello Dolly plugin

You might notice a heavy CPU usage consumption on your machine. Some may be a natural cause, for example, a known script being executed at a specific time whilst others may be due to a simple attack. Even if the attack is not a successful one, you may encounter a high CPU usage of your server which may eventually cause several kernels hangs or even cause other applications to be deprived of CPU usage. What I mean is that the goal of the attacker though it has not been reached, you may encounter a worse situation on your server.

Let’s see a brief analysis of an attack caused by a WordPress plugin known as “Hello Dolly”. The event started with a high CPU consumption on a server. Of course, by viewing the Htop or Atop processes, you can determine processes consuming more CPU.

Photo credits: Komodosec.com
Photo credits: Komodosec.com

1. Here is an idea of the Processes consuming more CPU by firing a simple ps command. The processes 829 and 4416 were the one consuming more CPU.

[[email protected]:/www/website.com/htdocs/wp-content]# ps aux|grep php

apache     829  6.2  0.5 351212 97492 ?        S    Oct05 127:36 php -q /tmp/tmp

apache    3459  0.3  0.3 416340 60080 ?        S    Oct06   1:43 php-fpm: pool www                                                                                            

apache    4416  7.2  0.5 336860 82656 ?        D    Oct05 146:43 php -q /tmp/tmp

apache    4753  0.2  0.3 420176 64048 ?        S    Oct06   1:20 php-fpm: pool www                                                                                            

root      7539  0.0  0.0 103248   868 pts/3    S+   06:55   0:00 grep php

2. We can notice that the process php -q /tmp/tmp emanate from a plugin on the server. For example, the PID 4416 corroborate with the lsof command.

[[email protected]:/www/website.com/htdocs/wp-content]# lsof plugins/

COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME

php       829 apache  cwd    DIR   0,20     4096 168820763 plugins

php      4416 apache  cwd    DIR   0,20     4096 168820763 plugins

bash    22664   root  cwd    DIR   0,20     4096 168820763 plugins

php     29199 apache  cwd    DIR   0,20     4096 168820763 plugins

php     29304 apache  cwd    DIR   0,20     4096 168820763 plugins

php     30153 apache  cwd    DIR   0,20     4096 168820763 plugins

3. If we make a strace -p of 2919 we can notice that it is trying to open the /etc/hosts file. 

[[email protected]:/www/website.com/htdocs/wp-content/plugins]# strace -p 29199

Process 29199 attached - interrupt to quit

socket(PF_NETLINK, SOCK_RAW, 0)         = -1 EMFILE (Too many open files)

open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = -1 EMFILE (Too many open files)

socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many open files)

socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many open files)

alarm(0)                                = 15

rt_sigaction(SIGALRM, {SIG_DFL, [], SA_RESTORER, 0x7f66381729a0}, NULL, 8) = 0

poll([{fd=3447, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)

4. Another interesting information to know which website or URL does the intrusion emanating from is by firing a lsof -p on the PID:

[[email protected]:/www/website.com/logs]# lsof -p 29304

COMMAND   PID   USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME

php     29304 apache  cwd    DIR       0,20     4096  168820763 /www/website.com/htdocs/wp-content/plugins 

php     29304 apache  rtd    DIR      253,0     4096          2 /

php     29304 apache  txt    REG      253,2  4105624      16544 /usr/bin/php

5. If we now try to analyze the log by sorting only the bot, we can find some “POST” being carried out which comes from IP 92.62.129.97 . At first glimpse, it looks like a google bot. Are we sure?

92.62.129.97 - - [05/Oct/2015:20:45:49 -0400] "POST /wp-content/plugins/index.php?cookie=1 HTTP/1.0" 200 13 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com

/bot.html)"

92.62.129.97 - - [05/Oct/2015:21:20:43 -0400] "POST /wp-content/plugins/index.php?cookie=1 HTTP/1.0" 200 13 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com

/bot.html)"

6. Who is 92.62.129.97 ?

92.62.129.97

GeoIP Country Edition: LT, Lithuania

GeoIP City Edition, Rev 1: LT, N/A, N/A, N/A, N/A, 56.000000, 24.000000, 0, 0

GeoIP ASNum Edition: AS42549 UAB Baltnetos komunikacijos

7. Did we notice that this IP is well reputed for attacks?

Check out this website https://cleantalk.org/blacklists/92.62.129.97. You would notice that there were attacks even on some Windows server reported by some people.

8. After more research, we can conclude that several WordPress users have encountered the same situation where the Hello Dolly plugin was causing a heavy load on their servers. After they have removed it, things have changed. Ref:

We can deduce how unknown plugins on WordPress can be dangerous if the codes are not properly audited by security experts. An analysis is very important before using such kind of plugins.

Note: This information might be incomplete in some sort as it may happen that the Hello Dolly was already compromised prior to the attack. The aim of the article is to get show an analysis methodology due to a high CPU consumption.

Converting a deb into rpm using alien on openSUSE

The alien command is used by almost all system administrators. You might come across situations where you may need to install a .deb package on an OpenSUSE machine. You will need to convert it to a .rpm prior to doing the installation. The alien command is simply a way to convert or install an alien binary package.

Photo credits: comicvine.com
Photo credits: comicvine.com

Installing Alien on OpenSUSE Leap

A general idea of how to install a .deb package on an OpenSUSE by converting it to a .rpm file can be done with the command alien. If you have freshly install OpenSUSE Leap, you might notice that command zypper install alien gives you the following error.

Screenshot from 2016-03-29 16-37-48

This can be solved easily as there is no repositories available. You can just jump on the Kamikaz Repo of the openSUSE factory. and fire the following commands :

zypper addrepo http://download.opensuse.org/repositories/home:KAMiKAZOW/openSUSE_Leap_42.1/home:KAMiKAZOW.repo
zypper refresh
zypper install alien

You  would have a result similar to this with all the dependencies installed.

Screenshot from 2016-03-29 16-43-42

You can finally launched the zypper install alien which will look similar to this.

Screenshot from 2016-03-29 16-44-24

Let’s now convert a .deb into a .rpm

I will take the example of the nmap tool. I have downloaded the nmap .deb file from the Ubuntu repo. You can choose your own deb file. This is the link to download the nmap from the Ubuntu repo.

wget http://mirrors.kernel.org/ubuntu/pool/main/n/nmap/nmap_7.01-2ubuntu1_amd64.deb

So to convert the file into a .rpm you need to launch the following command

alien --to-rpm <deb file name here>

Of course, on an openSUSE machine you would need the spec file. Here is an idea what kind of error you might came across.

Screenshot from 2016-03-29 16-56-07

Solving the error

The error “rpmbuild not found” clearly give a hint that the package rpmbuild is not found on the machine. Just install in with :

zypper install rpmbuild

Now that the rpmbuild package is installed with all the dependencies you can relaunch the command which in my case is

alien --to-rpm nmap_7.01-2ubuntu1_amd64.deb 

A nice message message where the package.rpm is generated will be prompted. I have just taken the nmap package as a example. It’s generally inadvisable to run alien on a machine having both RPMs and DEBs package because the two systems do not share installed-file database information. You can chose your own .deb file. Have funs with aliens.

 

Some basics of UEFI and BIOS

The BIOS (Basic Input Output Settings) and UEFI (Unified Firmware Extensible Interface) are two different types of firmware interface that enable the interaction between the mechanisms of your machine hardwSome basics of UEFI and BIOS 2are and its firmware. It is a type of “program” or “firmware” that initialize the boot process through your hardware. Your machine cannot boots up without this mechanism. Let’s see the main differences between BIOS and UEFI.


BIOS

When a machine boots up with BIOS, it reads the first boot sector in a hard disk followed by the normal boot process. One interesting thing is that BIOS basically runs a 16-bit (in real mode), 32-bit (protected mode) and 64 bit (Long mode) and during the boot up the BIOS is assigned 1MB of address space where it reads the MBR (Master Boot Record) file to understand the Machine partitions. You can activate the BIOS interface on VMware by clicking on VM -> Power -> Power on BIOS.

UEFI

So, UEFI is a sort of the next generation “firmware” replacing BIOS. It has been built with a better graphical interface since it performs in 32 bit or 64-bit mode. UEFI on the other hand when boots up uses the GPT (GUID Partition table); GUID is another abbreviation stands for Globally Unique IDs. With UEFI, you have the following advantages:

  • Managing disk of size more than 2.2 TB
  • Partition entries backup
  • Advanced network functionality
  • 64 Bit architecture support

In brief, UEFI boots up by loading several .efi files from a partition to the hard disk ESP (EFI system partition) which is a partition in itself.