Author Archives: tunnel1234

Operation JASK – Just a Single Keystroke

Apart from the IETF hackathons, the hackers.mu team also focused on internal hackathon either remotely or on-site participation. Another remote hackathon was already in progress since Saturday the 16th of June 2018. It was named Operation JASK – Just a Single Keystroke. Announced publicly on Sunday the 17th of June 2018 after noticing that several Crypto currency mining tools were vulnerable to CVE-2018-12356. By the time, many members of the team were already mobilised even if it was a public holiday in Mauritius. The operation was named JASK – Just a Single Keystroke as the security issues is concerned with the hardening of a regular expression, in particular requiring [GNUPG:] to be at the beginning of a line (^\[GNUPG:\]). We had to fire a single keystroke at the right place to fix a single vulnerability.

Marcus Brinkmann, who is a free software activist explained “An issue was discovered in password-store.sh in pass in Simple Password Store 1.7 through 1.7.1. The signature verification routine parses the output of GnuPG with an incomplete regular expression, which allows remote attackers to spoof file signatures on configuration files and extensions scripts. Modifying the configuration file allows the attacker to inject additional encryption keys under their control, thereby disclosing passwords to the attacker. Modifying the extension scripts allows the attacker arbitrary code execution.” 

However, simple the patch is, the attack aimed GnuPG signature verification process which is specific to pass the Simple Password Store. It can give the attacker access to passwords and remote code execution. On theRegister.co.uk – Pass gets a fail: Simple Password Store suffers GnuPG spoofing bug, Loganaden Velvindron core member of the hackers.mu explained “It’s hard to identify just how many downstream projects inherit a vulnerability like the one Brinkmann spotted, but the number of problem projects will likely be non-trivial because the GnuPG cryptography suite has applications beyond e-mail protection.”

The hackers.mu usual suspects during Operation JASK hackathon are: Kifah Meeran, Loganaden Velvindron, Rahul Golam, Muzaffar Auhammud, Nigel Yong and myself (Nitin J Mutkawoa) all members from the hackers.mu. Some of the projects are Bitcoin, Litecoin, Dash, Bitcoin Gold, Monacoin, Binarium, Terracoin, SmartCash and many other crypto currency projects.

Hackers.mu is now looking forward for other hackathons. We are also inviting everyone to meet us at Flying Dodo Bagatelle conference room for the Security Disclosure Process event. Feel free to RSVP on meetup.com  and Facebook before attending.

Africa Internet Summit 2018 – Hackathon – Day 2 & 3

Back to blogging after some days, I still recalled the moments in Dakar, Senegal for the Africa Internet Summit. By the time, so many days already elapsed, many have already blogged about the event and more pictures raining on the social media. Our camera, tripod and laptops were all ready. In case, you have missed Day 0 and Day 1, feel free to click on the links.

On day 2, myself with Logan and Serge made a brief introduction of the Network Time Protocol. Serge explained about the TCPDump and Wireshark tools that we can use to understand NTP traffic. We also made some demo about the NTP packets exchange between the client and the server. The algorithm behind was made clear, brief and concise as without which the hacking part would be difficult. Participants chose their projects for the hackathon. Some registered themselves for the Network Programmability track and the Intelligent Transportation Systems Projects. At the end of the day 2 we were already convinced about the hackathon would be a success. Myself, Logan and Charles decided to have a beer at a nearby restaurant.

Day 3 was the moment where everyone looked forward to hack into the code. The team spirit was here. Everyone was helping each other in their ‘parcours’. For the NTP hackathon, more and more participants start joining the team. Additional chairs and tables were needed. The best idea was to form into groups and this is where things change for the good. Patches start raining. Several tests were also carried out to confirm the code was running.

At the end of the hackathon, each group went to present their project and achievements. Their presentation slides can be viewed on the AIS wiki.

Some interesting links:

  • More information about the NTP hackathon is already uploaded on the AIS wiki.
  • The meeting statistics and report can be viewed here.
  • There is also a blog coverage by Charles from CISCO.
  • Dawit Bekele speech at the African Internet Summit 2019.
  • Another Interesting article by Kevin Chege from Internet Society.

On the last day of the hackathon, Logan, Charles and myself made a video on the hackathon.

More and more pictures:

IMG_3295
56146117-666f-4daa-80c3-799ddbd8d6f5
IMG_3332
IMG_1080
8f360295-2549-4c3f-8fe6-aa85b70122e0
ef15ce44-a90c-4ad4-805d-ddcb48550c5e
ef15ce44-a90c-4ad4-805d-ddcb48550c5e
52940f96-4084-45e9-a66a-f4e0b534581c
a169c827-e81a-4baf-bbd8-117b42f728b3
IMG_3335

A big thank to the organisers and sponsors for doing a great job. Also congratulations to the participants for stepping ahead in the hackathon. Looking forward to see you soon in the growth and security of Internet in Africa.

Africa Internet Summit 2018 – Getting ready for the Hackathon – Day 1

On day 1, I woke up early in the morning and went outside for a morning walk. Everyone in Senegal says Bonjour to each other, irrespective of being a stranger. The people of Dakar seem very polite and relaxed in nature. Whilst walking on the coastal road of Novotel, I admired the beauty of several massive Baobab trees. 

Baobab tree coastal road of Novotel
Baobab tree coastal road of Novotel

Back at the hotel, the breakfast was delicious with lots of fruits and cakes accompanied with juice. By the time, breakfast was over, it was already 0800 AM. I took a Panoramic view from the back of Novotel.

Panorama picture coastal road of Novotel
Panorama picture coastal road of Novotel

I had to get ready as I needed to travel from Novotel hotel to Radisson Blu where the hackathon preparation was going on. I met with Serge-Parfait GOMA instructor at the Hackathon together with Loganaden Velvindron from Afrinic. On Day 1, there were about 15 participants who already registered themselves. We had to prepare for the Hackathon as it needed to be carried out both in English and French.

From Left to Right : Logan, Nitin and Serge

Preparing for the Hackathon demands lots of time and trying to cover the maximum: from the basics until when the code need to be hacked. The project chosen was the NTP client. I created both slides in English and French.

Whilst I was preparing for the slides, Serge was busy setting up the Pidgin channel. We also tested the livestream. I brought a tripod for my Iphone 7 as it’s so easy for live YouTube video broadcast. We also checked out the hackathon room and carried out several tests. We were happy to be assisted by the guys from ISOC who were always there to help. We reviewed the code anew and discussed a little about the RFCs and Internet Drafts for that specific hackathon.

Time for dinner where I met fellows from Afrinic such as Duksh Koonjoobeeharry from Atlassian User group of Mauritius and Afrinic, Tamon Mookoom from Afrinic – That guy is an IPv6 ninja, Charles Eckel from CISCO who was also leading the hackathon on the network programmability track. I also met other persons from the ISOC team and Nishal Goburdhan a FreeBSD evangelist who gave me a FreeBSD sticker.

Panorama view at the cocktail event
Panorama view at the cocktail event
 
Screen Shot 2018-05-15 at 10.28.00 AM
Screen Shot 2018-05-15 at 10.27.39 AM
Screen Shot 2018-05-15 at 10.27.14 AM
Screen Shot 2018-05-15 at 10.27.26 AM
Screen Shot 2018-05-15 at 10.27.50 AM
Screen Shot 2018-05-15 at 10.26.59 AM

By the time, dinner was over, it was already late. I went to meet the ISOC and Afrinic guys who were still working hard to set up the hackathon room. I took a cab and headed directly to Novotel hotel.

In case you missed Day 0, do check the article here

Tunnelix.com is constantly retweeting the #AISdakar. It can be viewed here:

Africa Internet Summit 2018 – My first day in Dakar Senegal – Day 0

Dakar offers much to see and do, but my goal this trip lies elsewhere: facilitating the #AISdakar 2018 NTP – Network Time Protocol hackathon under the banner of hackers.mu which has been planned days back at Radisson blu hotel. Network Time Protocol (NTP) packets, as specified by RFC 5905 [RFC5905], carry a great deal of information about the state of the NTP daemon which transmitted them. In the case of mode 4 packets (responses sent from server to client), as well as in broadcast (mode 5) and symmetric peering modes (mode 1/2), most of this information is essential for accurate and reliable time synchronizaton. However, in mode 3 packets (requests sent from client to server), most of these fields serve no purpose. Server implementations never need to inspect them, and they can achieve nothing by doing so. Populating these fields with accurate information is harmful to privacy of clients because it allows a passive observer to fingerprint clients and track them as they move across networks.

The trip from Mauritius to Senegal was lengthy but at the same time, I got to discover parts of Africa: from South Africa to Kenya, hitting Ivory Coast before reaching Dakar, Senegal. During our transit in Johannesburg, Logan and I discussed several aspects of the AIS hackathon 2018 over two large pizzas and beers. One of the main goal is to maintain clear and precise billingual communication in English and French. Our next objective was to make sure that the required level should be reached for the hackathon.

I did not know that the plane will land at Ivory Coast before heading towards Dakar. Gazing out of the plane offered us unique breath taking and impressive views of infrastructures, panorama of the land and landscapes.

 

Disembarking at Dakar International Airport, I was received by the driver who works for a well reputed company – Prestige. Logan was received by an other company. Whilst travelling to the hotel, that guy was displaying curiosity and was inquisitive about computer repairs. I gave him some tips such as YouTube tutorials and some helpful links.

After landing, I headed directly to and checked in Novotel Hotel in Dakar where I checked in. I received a warm welcome staff members. Tired after long hours of travelling, a nap was very much needed before anything else. The view from the hotel room was magnificent with a swimming pool and beach nearside.

 

By the time I woke up, it was already late at about 21:00 hrs, I went to Radisson blu and met Kevin Chege and other delegates at the Gala Dinner. The atmosphere was friendly, welcoming and promising.

Tunnelix.com is constantly retweeting the #AISdakar. It can be viewed here:

 

 

 Next article coming up soon..

Debugging disk issues with blktrace, blkparse, btrace and btt in Linux environment

blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space. There are three major components: a kernel component, a utility to record the i/o trace information for the kernel to user space, and utilities to analyse and view the trace information. This man page describes blktrace, which records the i/o event trace information for a specific block device to a file.

 

Photo credits : www.brendangregg.com
Photo credits : www.brendangregg.com

Limitations and Solutions with blktrace

There are several limitations and solutions when using blktrace. We will focus mainly on its goal and how the blktrace tool can help us in our day to day task. Assuming that you want to debug any IO related issue, the first command will be probably an iostat. These utilities can be installed through a yum install sysstat blktrace. For example:

iostat -tkx -p 1 2

The limitation here with iostat is that it does not gave us which process is utilising which IO.  To resolve this issue, we can use another utility such as iotop. Here is an example of the iotop output.

blktrace, iotop, blkparse, btt and btrace

Here iotop shows us exactly which process is consuming ‘which’ and ‘how’ much IO. But another problem with that solution is that it does not give detailed information. So, blktrace comes handy as it gives layer-wise information. How it does that? It sees what is going on exactly inside block I/O layer. When used correctly, it’s possible to generate events for all I/O request and monitor it from where it is evolving. Though it extracts data from the kernel, it is not an analysis tool and the interpretation of the data need to be carried out by you. However, you can feed the data to btt or blkparse to get the analysis done.

Before looking at blktrace, let’s check out the I/O architecture. Basically, at the userspace the user will write and read data. This is what the User Process at the user space. The user do not write directly to the disk. They first write to the VFS Page Cache and from which there are various I/O Scheduler and the Device Driver will interact with the disk to write the data.

Photo credits: msreekan.com
Photo credits: msreekan.com

The blktrace will normally capture events during the process. Here is a cheat sheet to understand blktrace event capture.

photo credits: programering.com
photo credits: programering.com

blkparse will parse and format events acquired from blktrace. If you do not want to run blkparse, btrace is a shortcut to generate data out of blktrace and blkparse. Finally we have btt which will analyze data from blktrace and generate time deltas for each layer.

Another tool to grasp before moving on with blktrace is debugfs which is a simple-to-use RAM-based file system specially designed for debugging purposes. It exists as a simple way for kernel developers to make information available to user space. Unlike /proc, which is only meant for information about a process, or sysfs, which has strict one-value-per-file rules, debugfs has no rules at all. Developers can put any information they want there.lwn.

So the first thing to do is to mount the debugfs file system using the following command:

mount -t debugfs debugfs /sys/kernel/debug

The aim is to allow a kernel developer to make information available in user space. The output of the command below describe how to mount and verify same. You can use the command mount to test if same has been successful. Now that you have the debug file system, you can capture the events.

Diving into the commands

1.So you can use blktrace to trace out the I/O on the machine.

blktrace -d /dev/sda -o -|blkparse -i -

2. At the same time, on another console launch the following command to generate some I/O for testing purpose.

dd if=/dev/zero of=/mnt/test1 bs=1M count=1

From the blktrace console you will get an output which will end up as follows :

CPU0 (8,0):
 Reads Queued:           2,       60KiB Writes Queued:       5,132,   20,524KiB
 Read Dispatches:        2,       60KiB Write Dispatches:       61,   20,524KiB
 Reads Requeued:         0 Writes Requeued:         0
 Reads Completed:        2,       60KiB Writes Completed:       63,   20,524KiB
 Read Merges:            0,        0KiB Write Merges:        5,070,   20,280KiB
 Read depth:             1         Write depth:             7
 IO unplugs:            14         Timer unplugs:           9
Throughput (R/W): 8KiB/s / 2,754KiB/s
Events (8,0): 21,234 entries
Skips: 166 forward (1,940,721 -  98.9%)

3. Same result can also be achieved using the btrace command. Apply the same principle as in part 2 once the command has been launched.

btrace /dev/sda

4. In part 1, 2 and 4 the blktrace commands were launched in such a way that it will run forever – without exiting. In this particular example,  I will output the file name for analysis. Assume that you want to run the blktrace for 30 seconds, the command will be as follows:

blktrace -w 30 -d /dev/sda -o io-debugging

5. On another console, launch the following command:

dd if=/dev/sda of=/dev/null bs=1M count=10 iflag=direct

6. Wait for 30 seconds to allow step 4 to be completed. I got the following results just after:

[[email protected] mnt]# blktrace -w 30 -d /dev/sda -o io-debugging
=== sda ===
  CPU  0:                  510 events,       24 KiB data
  Total:                   510 events (dropped 0),       24 KiB data

7. You will notice at the directory /mnt  the file will be created. To read it use the command blkparse.

blkparse io-debugging.blktrace.0 | less

8. Now let’s see a simple extract from the blkparse command:

8,0    0        1     0.000000000  5686  Q   R 0 + 1024 [dd]
8,0    0        0     0.000028926     0  m   N cfq5686S / alloced
8,0    0        2     0.000029869  5686  G   R 0 + 1024 [dd]
8,0    0        3     0.000034500  5686  P   N [dd]
8,0    0        4     0.000036509  5686  I   R 0 + 1024 [dd]
8,0    0        0     0.000038209     0  m   N cfq5686S / insert_request
8,0    0        0     0.000039472     0  m   N cfq5686S / add_to_rr

The first column shows the device major,minor tuple, the second column gives information about the CPU and it goes on with the sequence, the timestamps, PID of the process issuing the IO process. The 6th column shows the event type, e.g. ‘Q’ means IO handled by request queue code. Please refer to above diagram for more info. The 7th column is R for Read, W for Write, D for block, B for Barrier operation and finally the last one is the block number and a following + number is the number of blocks requested. The final field between the [ ] brackets is the process name of the process issuing the request. In our case, I am running the command dd.

9.The output can be also analyzed using btt command. You will get almost the same information.

btt -i io-debugging.blktrace.0

Some interesting information here is D2C means the amount of time the IO has been spending into the device whilst Q2C means the total time take as there might be different IO concurrent.

A graphical user interface to makes life easier

The Seekwatcher program generates graphs from blktrace runs to help visualize IO patterns and performance. It can plot multiple blktrace runs together, making it easy to compare the differences between different benchmark runs. You should install the seekwatcher package if you need to visualize detailed information about IO patterns.

The command to be used to generate a picture to for analysis is as follows where seek.png is the output of the png name given.

seekwatcher -t io-debugging.blktrace.0 -o seek.png

What is also interesting is that you can create a movie-like with seekwatch to view the graph.

seekwatcher -t io-debugging.blktrace.0 -o seekmoving.mpg --movie

Tips:

  • For the debugfs, you can also edit the /etc/fstab file to make the mount point permanently. 

  • By default, blktrace will capture all events. This can be limited with the argument -a.
  • In case you want to capture persistently for a long time or for a certain amount of time, then use the argument -w.
  • blktrace will also stored the extracted data in local directory with a format device.blktrace.cpu, for example sda1.blktrace.cpu.
  • At step 1 and 2, you will need to fire a CTRL +C to stop the blktrace.
  • As seen in part 2, you have created test test1 file, do delete it same may consume disk space on your machine.

  • On part 5, the size of the file created in the  /mnt will not exceeds more that 1M since same has been specified in the command.
  • At part 9, you will noticed several other information which can be helpful such as D2C and  Q2C.
    • Q2D latency – time from request submission to Device.
    • D2C latency – Device latency for processing request.
    • Q2C latency – total latency , Q2D + D2C = Q2C.
  • To be able to generate the movie, you will have to install mencoder with all its dependencies.