Apart from the IETF hackathons, the hackers.mu team also focused on internal hackathon either remotely or on-site participation. Another remote hackathon was already in progress since Saturday the 16th of June 2018. It was named Operation JASK – Just a Single Keystroke. Announced publicly on Sunday the 17th of June 2018 after noticing that several Crypto currency mining tools were vulnerable to CVE-2018-12356. By the time, many members of the team were already mobilised even if it was a public holiday in Mauritius. The operation was named JASK – Just a Single Keystroke as the security issues is concerned with the hardening of a regular expression, in particular requiring [GNUPG:] to be at the beginning of a line (^\[GNUPG:\]). We had to fire a single keystroke at the right place to fix a single vulnerability.
Marcus Brinkmann, who is a free software activist explained “An issue was discovered in password-store.sh in pass in Simple Password Store 1.7 through 1.7.1. The signature verification routine parses the output of GnuPG with an incomplete regular expression, which allows remote attackers to spoof file signatures on configuration files and extensions scripts. Modifying the configuration file allows the attacker to inject additional encryption keys under their control, thereby disclosing passwords to the attacker. Modifying the extension scripts allows the attacker arbitrary code execution.”
However, simple the patch is, the attack aimed GnuPG signature verification process which is specific to pass the Simple Password Store. It can give the attacker access to passwords and remote code execution. On theRegister.co.uk – Pass gets a fail: Simple Password Store suffers GnuPG spoofing bug, Loganaden Velvindron core member of the hackers.mu explained “It’s hard to identify just how many downstream projects inherit a vulnerability like the one Brinkmann spotted, but the number of problem projects will likely be non-trivial because the GnuPG cryptography suite has applications beyond e-mail protection.”
Back to blogging after some days, I still recalled the moments in Dakar, Senegal for the Africa Internet Summit. By the time, so many days already elapsed, many have already blogged about the event and more pictures raining on the social media. Our camera, tripod and laptops were all ready. In case, you have missed Day 0 and Day 1, feel free to click on the links.
On day 2, myself with Logan and Serge made a brief introduction of the Network Time Protocol. Serge explained about the TCPDump and Wireshark tools that we can use to understand NTP traffic. We also made some demo about the NTP packets exchange between the client and the server. The algorithm behind was made clear, brief and concise as without which the hacking part would be difficult. Participants chose their projects for the hackathon. Some registered themselves for the Network Programmability track and the Intelligent Transportation Systems Projects. At the end of the day 2 we were already convinced about the hackathon would be a success. Myself, Logan and Charles decided to have a beer at a nearby restaurant.
Day 3 was the moment where everyone looked forward to hack into the code. The team spirit was here. Everyone was helping each other in their ‘parcours’. For the NTP hackathon, more and more participants start joining the team. Additional chairs and tables were needed. The best idea was to form into groups and this is where things change for the good. Patches start raining. Several tests were also carried out to confirm the code was running.
At the end of the hackathon, each group went to present their project and achievements. Their presentation slides can be viewed on the AIS wiki.
Some interesting links:
More information about the NTP hackathon is already uploaded on the AIS wiki.
The meeting statistics and report can be viewed here.
There is also a blog coverage by Charles from CISCO.
Dawit Bekele speech at the African Internet Summit 2019.
On the last day of the hackathon, Logan, Charles and myself made a video on the hackathon.
More and more pictures:
A big thank to the organisers and sponsors for doing a great job. Also congratulations to the participants for stepping ahead in the hackathon. Looking forward to see you soon in the growth and security of Internet in Africa.
On day 1, I woke up early in the morning and went outside for a morning walk. Everyone in Senegal says Bonjour to each other, irrespective of being a stranger. The people of Dakar seem very polite and relaxed in nature. Whilst walking on the coastal road of Novotel, I admired the beauty of several massive Baobab trees.
Back at the hotel, the breakfast was delicious with lots of fruits and cakes accompanied with juice. By the time, breakfast was over, it was already 0800 AM. I took a Panoramic view from the back of Novotel.
I had to get ready as I needed to travel from Novotel hotel to Radisson Blu where the hackathon preparation was going on. I met with Serge-Parfait GOMA instructor at the Hackathon together with Loganaden Velvindron from Afrinic. On Day 1, there were about 15 participants who already registered themselves. We had to prepare for the Hackathon as it needed to be carried out both in English and French.
Preparing for the Hackathon demands lots of time and trying to cover the maximum: from the basics until when the code need to be hacked. The project chosen was the NTP client. I created both slides in English and French.
Whilst I was preparing for the slides, Serge was busy setting up the Pidgin channel. We also tested the livestream. I brought a tripod for my Iphone 7 as it’s so easy for live YouTube video broadcast. We also checked out the hackathon room and carried out several tests. We were happy to be assisted by the guys from ISOC who were always there to help. We reviewed the code anew and discussed a little about the RFCs and Internet Drafts for that specific hackathon.
Time for dinner where I met fellows from Afrinic such as Duksh Koonjoobeeharry from Atlassian User group of Mauritius and Afrinic, Tamon Mookoom from Afrinic – That guy is an IPv6 ninja, Charles Eckel from CISCO who was also leading the hackathon on the network programmability track. I also met other persons from the ISOC team and Nishal Goburdhan a FreeBSD evangelist who gave me a FreeBSD sticker.
By the time, dinner was over, it was already late. I went to meet the ISOC and Afrinic guys who were still working hard to set up the hackathon room. I took a cab and headed directly to Novotel hotel.
Dakar offers much to see and do, but my goal this trip lies elsewhere: facilitating the #AISdakar 2018 NTP – Network Time Protocol hackathon under the banner of hackers.mu which has been planned days back at Radisson blu hotel. Network Time Protocol (NTP) packets, as specified by RFC 5905 [RFC5905], carry a great deal of information about the state of the NTP daemon which transmitted them. In the case of mode 4 packets (responses sent from server to client), as well as in broadcast (mode 5) and symmetric peering modes (mode 1/2), most of this information is essential for accurate and reliable time synchronizaton. However, in mode 3 packets (requests sent from client to server), most of these fields serve no purpose. Server implementations never need to inspect them, and they can achieve nothing by doing so. Populating these fields with accurate information is harmful to privacy of clients because it allows a passive observer to fingerprint clients and track them as they move across networks.
The trip from Mauritius to Senegal was lengthy but at the same time, I got to discover parts of Africa: from South Africa to Kenya, hitting Ivory Coast before reaching Dakar, Senegal. During our transit in Johannesburg, Logan and I discussed several aspects of the AIS hackathon 2018 over two large pizzas and beers. One of the main goal is to maintain clear and precise billingual communication in English and French. Our next objective was to make sure that the required level should be reached for the hackathon.
I did not know that the plane will land at Ivory Coast before heading towards Dakar. Gazing out of the plane offered us unique breath taking and impressive views of infrastructures, panorama of the land and landscapes.
Disembarking at Dakar International Airport, I was received by the driver who works for a well reputed company – Prestige. Logan was received by an other company. Whilst travelling to the hotel, that guy was displaying curiosity and was inquisitive about computer repairs. I gave him some tips such as YouTube tutorials and some helpful links.
After landing, I headed directly to and checked in Novotel Hotel in Dakar where I checked in. I received a warm welcome staff members. Tired after long hours of travelling, a nap was very much needed before anything else. The view from the hotel room was magnificent with a swimming pool and beach nearside.
By the time I woke up, it was already late at about 21:00 hrs, I went to Radisson blu and met Kevin Chege and other delegates at the Gala Dinner. The atmosphere was friendly, welcoming and promising.
Tunnelix.com is constantly retweeting the #AISdakar. It can be viewed here:
blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space. There are three major components: a kernel component, a utility to record the i/o trace information for the kernel to user space, and utilities to analyse and view the trace information. This man page describes blktrace, which records the i/o event trace information for a specific block device to a file.
Limitations and Solutions with blktrace
There are several limitations and solutions when using blktrace. We will focus mainly on its goal and how the blktrace tool can help us in our day to day task. Assuming that you want to debug any IO related issue, the first command will be probably an iostat. These utilities can be installed through a yum install sysstat blktrace. For example:
iostat -tkx -p 1 2
The limitation here with iostat is that it does not gave us which process is utilising which IO. To resolve this issue, we can use another utility such as iotop. Here is an example of the iotop output.
blktrace, iotop, blkparse, btt and btrace
Here iotop shows us exactly which process is consuming ‘which’ and ‘how’ much IO. But another problem with that solution is that it does not give detailed information. So, blktrace comes handy as it gives layer-wise information. How it does that? It sees what is going on exactly inside block I/O layer. When used correctly, it’s possible to generate events for all I/O request and monitor it from where it is evolving. Though it extracts data from the kernel, it is not an analysis tool and the interpretation of the data need to be carried out by you. However, you can feed the data to btt or blkparse to get the analysis done.
Before looking at blktrace, let’s check out the I/O architecture. Basically, at the userspace the user will write and read data. This is what the User Process at the user space. The user do not write directly to the disk. They first write to the VFS Page Cache and from which there are various I/O Scheduler and the Device Driver will interact with the disk to write the data.
The blktrace will normally capture events during the process. Here is a cheat sheet to understand blktrace event capture.
blkparse will parse and format events acquired from blktrace. If you do not want to run blkparse, btrace is a shortcut to generate data out of blktrace and blkparse. Finally we have btt which will analyze data from blktrace and generate time deltas for each layer.
Another tool to grasp before moving on with blktrace is debugfs which is a simple-to-use RAM-based file system specially designed for debugging purposes. It exists as a simple way for kernel developers to make information available to user space. Unlike /proc, which is only meant for information about a process, or sysfs, which has strict one-value-per-file rules, debugfs has no rules at all. Developers can put any information they want there. – lwn.
So the first thing to do is to mount the debugfs file system using the following command:
mount -t debugfs debugfs /sys/kernel/debug
The aim is to allow a kernel developer to make information available in user space. The output of the command below describe how to mount and verify same. You can use the command mount to test if same has been successful. Now that you have the debug file system, you can capture the events.
Diving into the commands
1.So you can use blktrace to trace out the I/O on the machine.
blktrace -d /dev/sda -o -|blkparse -i -
2. At the same time, on another console launch the following command to generate some I/O for testing purpose.
dd if=/dev/zero of=/mnt/test1 bs=1M count=1
From the blktrace console you will get an output which will end up as follows :
3. Same result can also be achieved using the btrace command. Apply the same principle as in part 2 once the command has been launched.
4. In part 1, 2 and 4 the blktrace commands were launched in such a way that it will run forever – without exiting. In this particular example, I will output the file name for analysis. Assume that you want to run the blktrace for 30 seconds, the command will be as follows:
blktrace -w 30 -d /dev/sda -o io-debugging
5. On another console, launch the following command:
7. You will notice at the directory /mnt the file will be created. To read it use the command blkparse.
blkparse io-debugging.blktrace.0 | less
8. Now let’s see a simple extract from the blkparse command:
8,001 0.0000000005686Q R 0 + 1024 [dd]8,000 0.000028926 0m N cfq5686S / alloced8,002 0.0000298695686G R 0 + 1024 [dd]8,003 0.0000345005686P N [dd]8,004 0.0000365095686I R 0 + 1024 [dd]8,000 0.000038209 0m N cfq5686S / insert_request8,000 0.000039472 0m N cfq5686S / add_to_rr
The first column shows the device major,minor tuple, the second column gives information about the CPU and it goes on with the sequence, the timestamps, PID of the process issuing the IO process. The 6th column shows the event type, e.g. ‘Q’ means IO handled by request queue code. Please refer to above diagram for more info. The 7th column is R for Read, W for Write, D for block, B for Barrier operation and finally the last one is the block number and a following + number is the number of blocks requested. The final field between the [ ] brackets is the process name of the process issuing the request. In our case, I am running the command dd.
9.The output can be also analyzed using btt command. You will get almost the same information.
btt -i io-debugging.blktrace.0
Some interesting information here is D2C means the amount of time the IO has been spending into the device whilst Q2C means the total time take as there might be different IO concurrent.
A graphical user interface to makes life easier
The Seekwatcher program generates graphs from blktrace runs to help visualize IO patterns and performance. It can plot multiple blktrace runs together, making it easy to compare the differences between different benchmark runs. You should install the seekwatcher package if you need to visualize detailed information about IO patterns.
The command to be used to generate a picture to for analysis is as follows where seek.png is the output of the png name given.