On day 1, I woke up early in the morning and went outside for a morning walk. Everyone in Senegal says Bonjour to each other, irrespective of being a stranger. The people of Dakar seem very polite and relaxed in nature. Whilst walking on the coastal road of Novotel, I admired the beauty of several massive Baobab trees.
Back at the hotel, the breakfast was delicious with lots of fruits and cakes accompanied with juice. By the time, breakfast was over, it was already 0800 AM. I took a Panoramic view from the back of Novotel.
I had to get ready as I needed to travel from Novotel hotel to Radisson Blu where the hackathon preparation was going on. I met with Serge-Parfait GOMA instructor at the Hackathon together with Loganaden Velvindron from Afrinic. On Day 1, there were about 15 participants who already registered themselves. We had to prepare for the Hackathon as it needed to be carried out both in English and French.
Preparing for the Hackathon demands lots of time and trying to cover the maximum: from the basics until when the code need to be hacked. The project chosen was the NTP client. I created both slides in English and French.
Whilst I was preparing for the slides, Serge was busy setting up the Pidgin channel. We also tested the livestream. I brought a tripod for my Iphone 7 as it’s so easy for live YouTube video broadcast. We also checked out the hackathon room and carried out several tests. We were happy to be assisted by the guys from ISOC who were always there to help. We reviewed the code anew and discussed a little about the RFCs and Internet Drafts for that specific hackathon.
Time for dinner where I met fellows from Afrinic such as Duksh Koonjoobeeharry from Atlassian User group of Mauritius and Afrinic, Tamon Mookoom from Afrinic – That guy is an IPv6 ninja, Charles Eckel from CISCO who was also leading the hackathon on the network programmability track. I also met other persons from the ISOC team and Nishal Goburdhan a FreeBSD evangelist who gave me a FreeBSD sticker.
By the time, dinner was over, it was already late. I went to meet the ISOC and Afrinic guys who were still working hard to set up the hackathon room. I took a cab and headed directly to Novotel hotel.
Dakar offers much to see and do, but my goal this trip lies elsewhere: facilitating the #AISdakar 2018 NTP – Network Time Protocol hackathon under the banner of hackers.mu which has been planned days back at Radisson blu hotel. Network Time Protocol (NTP) packets, as specified by RFC 5905 [RFC5905], carry a great deal of information about the state of the NTP daemon which transmitted them. In the case of mode 4 packets (responses sent from server to client), as well as in broadcast (mode 5) and symmetric peering modes (mode 1/2), most of this information is essential for accurate and reliable time synchronizaton. However, in mode 3 packets (requests sent from client to server), most of these fields serve no purpose. Server implementations never need to inspect them, and they can achieve nothing by doing so. Populating these fields with accurate information is harmful to privacy of clients because it allows a passive observer to fingerprint clients and track them as they move across networks.
The trip from Mauritius to Senegal was lengthy but at the same time, I got to discover parts of Africa: from South Africa to Kenya, hitting Ivory Coast before reaching Dakar, Senegal. During our transit in Johannesburg, Logan and I discussed several aspects of the AIS hackathon 2018 over two large pizzas and beers. One of the main goal is to maintain clear and precise billingual communication in English and French. Our next objective was to make sure that the required level should be reached for the hackathon.
I did not know that the plane will land at Ivory Coast before heading towards Dakar. Gazing out of the plane offered us unique breath taking and impressive views of infrastructures, panorama of the land and landscapes.
Disembarking at Dakar International Airport, I was received by the driver who works for a well reputed company – Prestige. Logan was received by an other company. Whilst travelling to the hotel, that guy was displaying curiosity and was inquisitive about computer repairs. I gave him some tips such as YouTube tutorials and some helpful links.
After landing, I headed directly to and checked in Novotel Hotel in Dakar where I checked in. I received a warm welcome staff members. Tired after long hours of travelling, a nap was very much needed before anything else. The view from the hotel room was magnificent with a swimming pool and beach nearside.
By the time I woke up, it was already late at about 21:00 hrs, I went to Radisson blu and met Kevin Chege and other delegates at the Gala Dinner. The atmosphere was friendly, welcoming and promising.
Tunnelix.com is constantly retweeting the #AISdakar. It can be viewed here:
blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space. There are three major components: a kernel component, a utility to record the i/o trace information for the kernel to user space, and utilities to analyse and view the trace information. This man page describes blktrace, which records the i/o event trace information for a specific block device to a file.
Limitations and Solutions with blktrace
There are several limitations and solutions when using blktrace. We will focus mainly on its goal and how the blktrace tool can help us in our day to day task. Assuming that you want to debug any IO related issue, the first command will be probably an iostat. These utilities can be installed through a yum install sysstat blktrace. For example:
iostat -tkx -p 1 2
The limitation here with iostat is that it does not gave us which process is utilising which IO. To resolve this issue, we can use another utility such as iotop. Here is an example of the iotop output.
blktrace, iotop, blkparse, btt and btrace
Here iotop shows us exactly which process is consuming ‘which’ and ‘how’ much IO. But another problem with that solution is that it does not give detailed information. So, blktrace comes handy as it gives layer-wise information. How it does that? It sees what is going on exactly inside block I/O layer. When used correctly, it’s possible to generate events for all I/O request and monitor it from where it is evolving. Though it extracts data from the kernel, it is not an analysis tool and the interpretation of the data need to be carried out by you. However, you can feed the data to btt or blkparse to get the analysis done.
Before looking at blktrace, let’s check out the I/O architecture. Basically, at the userspace the user will write and read data. This is what the User Process at the user space. The user do not write directly to the disk. They first write to the VFS Page Cache and from which there are various I/O Scheduler and the Device Driver will interact with the disk to write the data.
The blktrace will normally capture events during the process. Here is a cheat sheet to understand blktrace event capture.
blkparse will parse and format events acquired from blktrace. If you do not want to run blkparse, btrace is a shortcut to generate data out of blktrace and blkparse. Finally we have btt which will analyze data from blktrace and generate time deltas for each layer.
Another tool to grasp before moving on with blktrace is debugfs which is a simple-to-use RAM-based file system specially designed for debugging purposes. It exists as a simple way for kernel developers to make information available to user space. Unlike /proc, which is only meant for information about a process, or sysfs, which has strict one-value-per-file rules, debugfs has no rules at all. Developers can put any information they want there. – lwn.
So the first thing to do is to mount the debugfs file system using the following command:
mount -t debugfs debugfs /sys/kernel/debug
The aim is to allow a kernel developer to make information available in user space. The output of the command below describe how to mount and verify same. You can use the command mount to test if same has been successful. Now that you have the debug file system, you can capture the events.
Diving into the commands
1.So you can use blktrace to trace out the I/O on the machine.
blktrace -d /dev/sda -o -|blkparse -i -
2. At the same time, on another console launch the following command to generate some I/O for testing purpose.
dd if=/dev/zero of=/mnt/test1 bs=1M count=1
From the blktrace console you will get an output which will end up as follows :
3. Same result can also be achieved using the btrace command. Apply the same principle as in part 2 once the command has been launched.
4. In part 1, 2 and 4 the blktrace commands were launched in such a way that it will run forever – without exiting. In this particular example, I will output the file name for analysis. Assume that you want to run the blktrace for 30 seconds, the command will be as follows:
blktrace -w 30 -d /dev/sda -o io-debugging
5. On another console, launch the following command:
7. You will notice at the directory /mnt the file will be created. To read it use the command blkparse.
blkparse io-debugging.blktrace.0 | less
8. Now let’s see a simple extract from the blkparse command:
8,001 0.0000000005686Q R 0 + 1024 [dd]8,000 0.000028926 0m N cfq5686S / alloced8,002 0.0000298695686G R 0 + 1024 [dd]8,003 0.0000345005686P N [dd]8,004 0.0000365095686I R 0 + 1024 [dd]8,000 0.000038209 0m N cfq5686S / insert_request8,000 0.000039472 0m N cfq5686S / add_to_rr
The first column shows the device major,minor tuple, the second column gives information about the CPU and it goes on with the sequence, the timestamps, PID of the process issuing the IO process. The 6th column shows the event type, e.g. ‘Q’ means IO handled by request queue code. Please refer to above diagram for more info. The 7th column is R for Read, W for Write, D for block, B for Barrier operation and finally the last one is the block number and a following + number is the number of blocks requested. The final field between the [ ] brackets is the process name of the process issuing the request. In our case, I am running the command dd.
9.The output can be also analyzed using btt command. You will get almost the same information.
btt -i io-debugging.blktrace.0
Some interesting information here is D2C means the amount of time the IO has been spending into the device whilst Q2C means the total time take as there might be different IO concurrent.
A graphical user interface to makes life easier
The Seekwatcher program generates graphs from blktrace runs to help visualize IO patterns and performance. It can plot multiple blktrace runs together, making it easy to compare the differences between different benchmark runs. You should install the seekwatcher package if you need to visualize detailed information about IO patterns.
The command to be used to generate a picture to for analysis is as follows where seek.png is the output of the png name given.
Lime is a Loadable Kernel Module (LKM) which allows for volatile memory acquisition from Linux and Linux-based devices, such as Android. This makes LiME unique as it is the first tool that allows for full memory captures on Android devices. It also minimises its interaction between user and kernel space processes during acquisition, which allows it to produce memory captures that are more forensically sound than those of other tools designed for Linux memory acquisition. – Lime. Volatility framework was released at Black Hat DC for analysis of memory during forensic investigations.
Analysing memory in Linux can be carried out using Lime which is a forensic tool to dump the memory. I am actually using CentOS 6 distribution installed on a Virtual Box to acquire memory. Normally before capturing the memory, the suspicious system’s architecture should be well known. May be you would need to compile Lime on the the suspicious machine itself if you do not know the architecture. Once you compile Lime, you would have a kernel loadable object which can be injected in the Linux Kernel itself.
Linux memory dump with Lime
1. You will first need to download Lime on the suspicious machine.
git clone https://github.com/504ensicsLabs/LiME
2. Do the compilation of Lime. Once it has been compiled, you will noticed the creation of the Lime loadable kernel object.
3. Now the kernel object have to be loaded into the kernel. Insert the kernel module. Then, define the location and format to save the memory image.
6. Now, we will create a Linux profile. We will also need to download the DwarfDump package. Once it is downloaded go to Tools -> Linux directory, then create the module.dwarf file.
yum install epel-release libdwarf-tools -y && make
7. To proceed further, the System.map file is important to build the profile. The System.map file contains the locations of all the functions active in the compiled kernel. You will notice it inside the /boot directory. It is also important to corroborate the version appended with the System.map file together the version and architecture of the kernel. In the example below, the version is 2.6.32-696.23.1.el6.x86_64.
8. Now, go to the root of the Volatility directory using cd ../../ since I assumed that you are in the linux directory. Then, create a zip file as follows:
zip volatility/plugins/overlays/linux/Centos6-2632.zip tools/linux/module.dwarf /boot/System.map-2.6.32-696.23.1.el6.x86_64
9. The volatility module has now been successfully created as indicated in part 8 for the particular version of the Linux and kernel version. Time to have fun with some Python script. You can view the profile created with the following command:
python vol.py --info | grep Linux
As you can see the profile LinuxCentos6-2632 profile has been created.
10. Volatile contains plugins to view details about the memory dump performed. To view the plugins or parsers, use the following command:
python vol.py --info | grep -i linux_
11. Now imagine that you want to see the processes running at the time of the memory dump. You will have to execute the vol.py script, specify the location of the memory dump, define the profile created and call the parser concerned.
I find the LiMEaid tools really interesting to remote executing of Lime. “LiMEaide is a python application designed to remotely dump RAM of a Linux client and create a volatility profile for later analysis on your local host. I hope that this will simplify Linux digital forensics in a remote environment. In order to use LiMEaide all you need to do is feed a remote Linux client IP address, sit back, and consume your favorite caffeinated beverage.” – LiMEaid
Linux architecture is very important when dealing with Lime. This is probably the first question that one would ask.
The kernel-headers package is a must to create the kernel loadable object.
Once a memory dump have been created, its important to take a hash value. It can be done using the command md5sum Linux64.mem
I would also consider to download the devel tools using yum groupinstall “Development Tools” -y
As good practice as indicated in part 8 when creating the zip file, use the proper convention when naming the file. In my case I used the OS version and the kernel version for future references.
Not all Parsers/Plugins will work with Volatile as same might not be compatible with the Linux system.
You can check out the Volatile wiki for more info about the Parsers.
“Webelieve in rough consensus andrunning code” – Just have a look at the IETF website, this is the motto that you would come across. This is why the IETF hackathons are so special during the year and hackers.mu team is proud to be the first team in Mauritius who does not only participate in such type of event but also lead the TLS working group. The IETF 101 hackathon was yet another challenge for the hackers.mu team. But, once you are in, the fun begins. Compared to the IETF 100 hackathon, hackers.mu team made an improvement in terms of lines of codes and focused on more projects. We participated remotely in projects such as TLS 1.3, DNS, and HTTP 451. A wiki was also created during that event.
We used Jabber to communicate for the IETF 101 hackathon. Other media such as Facebook was found out to be interesting. I should admit that on Friday and Saturday I went to sleep at 02.00 AM with just the testing part completed. At 23:00 hrs, Logan was asking everyone to go to sleep as we needed more energy on the next day. Selven was also working hard remotely to bring all members on track. What is more relieving is the team spirit where everyone was helping each other during that hackathon.
One of the interesting issues noticed is about TLS malformed traffic and such thing was able to be detected using Wireshark. Once the patches were ready and the testing part was working fine, we made a debrief at Flying Dodo beer brewing company at Bagatelle Mall and was ready submit patches to their respective projects. I was assigned the “Stunnel” project and a library in “Eclipse Paho”.
After the debriefing, Logan was getting ready for his remote presentation at the IETF. We all went through the slides that logan created and went back home happily to see the presentation live on YouTube.
Special thanks goes to the IETF Organising team for having us as Technology Champions! Nick Sullivan head of cryptography expert at CloudFlare, Charkes Eckel, Barry Leiba, Meetecho team, Cisco for sponsoring the event and the all members of the hackers.mu team which made this hackathon a success in the world history of Mauritius.
Other’s are also talking about the IETF 101 hackathon ?
“I had initially started a bit slow, as I was working on other projects in parallel. Everyone was already deeply immersed in their projects, we could see PRs and code merges flying right from the first day.” – Codarren Velvindron
“It seems that I am not the only one who feels that this hackathon was really addictive. we were hooked the moment we started working out on our tasks.” – Pirabarlen Cheenaramen
“Developers working with OpenSSL can finally start to work with TLS 1.3, thanks to the alpha version of OpenSSL 1.1.1 that landed yesterday.” – TheRegister
“I think that you guys have more better weather and more fun that we did” – Charles Eckel
“hackers.mu pioneering the internet! We made it to IETF 101 hackathon with our team members getting featured in front of thousands, followed by a round of applause by IETF members in London. Congratulations guys, we did it again!” – Yasir Auleear
“IETF Hackathons encourage developers to collaborate and develop utilities, ideas, sample code and solutions that show practical implementations of IETF standards. The IETF Hackathon in London on 17-18 March is poised to be the largest ever.” – IETF
In case you are asking yourself, “who are the hackers.mu ?” You can consider is as “a group of developers from Mauritius who loves to code and are passionate about information security.” More information at https://www.hackers.mu