ESXi installation on my Dell Laptop and hands on VMware Labs

If you are thinking why i should install a bare metal hypervisor on a laptop, i assure you its just for educational and testing purpose only. I noticed that it was quite difficult for me to get this done. However, after some research it looks that my Dell Inspiron n5110 motherboard will not authorised me to install ESXi 6.x. Probably, it looks like there are some drivers missing or the motherboard does not support it.

Here is what my processors looks like from the configuration menu on VMware vSphere Center

Anyway, i have been able to inject some network drivers – VIB files into the ESXi5.0 which allowed me to install the ESXi 5.0 on the laptop. You can follow the instructions at the link how to make your unsupported NIC work with ESXi. Once installed, VMware will provide you with a two months free trial before you purchase the license.

Another way of messing around VMware Vsphere is to deploy a lab from labs.hol.vmware.com That’s so easy to deploy labs and access the VMware vSphere web client. All credentials will be available on the readme.txt file found on the desktop. Also a lab manual will be shown alongside whilst working on the environement labs.

I am sure this would help anyone to get into hands on lab quickly and it would be a nice start for beginners.


Out of Memory (OOM) in Linux machines

Since some months I have not been posting anything on my blog. I should admit that I was really busy. Recently, a friend asked me about the Out of Memory messages in Linux. How is it generated? What are the consequences? How can it be avoided in Linux machines? There is no specific answer to this as an investigation had to be carried out to have the Root Cause Analysis. Before getting into details about OOM, let’s be clear that whenever the Kernel is starved of memory, it will start killing processes. Many times, Linux administrators will experience this and one of the fastest way to get rid of it is by adding extra swap. However, this is not the definite way of solving the issue. A preliminary assessment needs to be carried out followed by an action plan, alongside, a rollback methodology.

If after killing some processes, the kernel cannot free up some memory, it might lead to a kernel panic, deadlocks in applications, kernel hungs, or several defunct processes in the machine. I know cases where the machine change run level mode. There are cases of kernel panic in virtual machines where the cluster is not healthy. In brief, OOM is a subsystem to kill one or more processes with the aim to free memory. In the article Linux kernel crash simulation using kdump, I gave an explanation how to activate Kdump to generate a vmcore for analysis. However, to send the debug messages during an out of memory error to the vmcore, the SYSCTL file need to be configured. I will be using a CentOS 7 machine to illustrate the OOM parameters and configurations.

1.To activate OOM debug in the vmcore file, set the parameter vm.panic_on_oom to 1 using the following command:

systctl -w vm.panic_on_oom=1

To verify if the configuration has been taken into consideration, you can do a sysctl -a | grep -i oom. It is not recommended to test such parameters in the production environment.

2. To find out which process the kernel is going to kill, the kernel will read a function in the kernel code called badness() . The badness() calculate a numeric value about how bad this task has been. To be precise, it works by accumulating some “points” for each process running on the machine and will return those processes to a function called select_bad_process() in the linux kernel. This will eventually start the OOM mechanism which will kill the processes. The “points” are stored in the /proc/<pid>/oom_score. For example, here, i have a server running JAVA.

As you can see, the process number is 2153. The oom_score is 21

3. There are lots of considerations that are taken into account when calculating the badness score. Some of the factors are the Virtual Memory size (VM size), the Priority of the Process (NICE value), the Total Runtime, the Running user and the /proc/<pid>/oom_adj. You can also set up the oom_score_adj value for any PID between -1000 to 1000. The lowest possible value, -1000, is equivalent to disabling OOM killing entirely for that task since it will always report a badness score of 0.

4. Let’s assume that you want to prevent a specific process from being killed.

echo -17 > /proc/$PID/oom_adj

5. If you know the process name of SSH Daemon and do not it from being killed, use the following command:

pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

6. To automate the sshd from being killed through a cron which will run each minute use the following:

* * * * * root pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done
7. Let's now simulate the OOM killer messages. Use the following command to start an out of memory event 
on the machine.
echo f > /proc/sysrq-trigger 

You will notice an OOM error message in the /var/log/messages.
As you can notice here, the PID 843 was calculated by the OOM killer before killing it. 
There is also the score number which is 4 in our case.


Before the 'Out of memory' error, there will be a call trace which will be sent by the kernel.



8. To monitor how the OOM killer is generating scores, you can use the dstat command. To install the dstat 
package on RPM based machine use: 
yum install dstat 

or for debian based distribution use:
apt-get install dstat

Dstat is used to generate resource statistics. To use dstat to monitor the score from OOM killer use:
dstat -top-oom


TIPS:

  • oom_score_adj is used in new linux kernel. The deprecated function is oom_adj in old Linux machine.
  • When disabling OOM killer under heavy memory pressure, it may cause the system to kernel panic.
  • Making a process immune is not a definite way of solving problem, for example, when using JAVA Application. Use a thread/heap dump to analyse the situation before making a process immune.
  • Dstat is now becoming an alternative for vmstat, netstat, iostat, ifstat and mpstat. For example, to monitor CPU in a program, use dstat -c –top-cpu -dn –top-mem
  • Testing in production environment should be avoided!

A trip to a Wind Farm at Plaine Des Roches

This Sunday the 16th of April, I came across an interesting location in the North-East at Plaine Des Roches, Mauritius where electricity is produced through Wind Farms. The company is Quadran which has invested in this environmental friendly interesting project. Quadran is the global actor in renewable energy encompassing hydroelectricity, solar energy, wind energy and biogas. It has 130 collaborators from 13 agencies and subsidiaries in France metropolitan and Outre-mer including Reunion Island.Quadran. 

The electricity is produced by means of kinetic energy from the wind. The wind turns the blades, which spin a shaft connected to a generator which makes electricity. At some moment, when there is not enough wind, a fuel-powered engine will use switch on automatically for some seconds to run the turbine after which, the wind will take over to turn the blades.

I believe that Mauritius which is looking forward for a more eco-friendly island should invest more in these type of project. This project which involves 11 wind turbines with a power production of 9, 35 MW will satisfy the energy consumption of approximately 10,150 people.

However, side effects of wind turbines are not false. According to some source, there are also reports of negative effects on radio and television reception in wind farm communities. Potential solutions include predictive interference modeling as a component of site selection. A 2007 report by the U.S. National Research Council noted that noise produced by wind turbines is generally not a major concern for humans beyond a half-mile or so. Low-frequency vibration and its effects on humans are not well understood and sensitivity to such vibration resulting from wind-turbine noise is highly variable among humans. – www.nap.edu


Hackers.mu attracted a massive crowd at the DevConMru 2017

This is yet another dazzling inspiration that hackers.mu brought into the mind of the audience today on the 1st of April 2017 at the DevConMru – Day 2. After the mesmerising speech at the DevConMru by Logan, this time Codarren Velvindron, core member of hackers.mu hit the conference room with so many attendees. Fast Coding Skills – A well chosen topic especially for the curious ones, beginners or professionals who want to remove the barrier between the code and them. Codarren started the presentation by giving some examples about the applications he ventured into, for example MariaDB.

The room was full with over fifty attendees. While some were sitting on the floor, others leaned up against the wall focussed on Codarren. I heard someone from the crowd murmuring “I want to be a hacker”.. 🙂

C8V2P3IXoAARtuF
C8V2P3HXUAAmYBf
C8VgQ35WAAAOPv9

Several analogies were brought to the attention of the audience such as the difficulties which one has to encounter whilst coding. Tips and tricks to get relief from these difficulties were offered; such as playing, breaking the huge task into parts and analysing the mini parts of each. Another way to understand how the code works is by “deleting” part it after a backup to know how it would behave in a different environment. Codarren also shared his experience about the IETF hackathon in which he participated.

Here is the Slide of Codarren at the DevConMru 2017

Fast Coding Skills by Codarren Velvindron on Scribd

At the end, we thanked Codarren for the job done. Members of hackers.mu kept on responding to people from the audience who were showing interest in coding. Some questions from the audience were about the challenges faced in the IETF hackathon as well as Codarren’s favourite programming language. “Talk is cheap, show me the code” – Linus Torvalds.


Hackers.mu mesmerising speech at the DevConMru 2017

The message was clear and direct at the DevConMru 2017. Painted with a humorist approach, Loganaden Velvindron #2 of hackers.mu bridged the gap between students who were mainly in the audience to reach their goal in the IT industry and Linux in Mauritius.

The DevConMru is a yearly event to bring together developers, beginners, students and professionals. The goal is to bring more craftsmen under the same roof. “Mauritius has been branded “Cyber Island” in the Indian Ocean… Opinions in those matters vary but with this conference we strive to improve the general attribution of our island. Mauritius has great political stability and economical advantages for foreign investors, and the most precious resource Mauritius has to offer is people’s knowledge. The ICT sector in Mauritius is growing since years and maturing as the fourth pillar of our economy. With its geographical position Mauritius is also welcome as a business and knowledge hub between Africa and Asia.”MSCC.

In today’s Mauritius IT industry, everyone wants to have a better standard of living. But how? How to build a successful IT industry? Are we moving in the right direction? Are foreign investors attracted by the quality of the Mauritius IT industry ? Logan did not miss those points to bring the audience on the track.

Photo Credits: Hackers.mu
Photo Credits: Hackers.mu

After giving a brief intro of the hackers.mu team, Logan explained the requirements and life cycles of IT companies and their profits as to whether they are in the same line of fresh IT graduates and professionals. A vivid example is by analyzing the statistics of Github accounts in Mauritius, the quality and quantity of code contribution compared with Singapore. Students were encouraged to publish their coding exercise on GitHubs, create a blog and take part in Google Code-in.

Indeed, hackers.mu work towards such goals, for example, participation and mentoring for the Google Code-in. Several hackathons were organised. Contributions in the real world applications such as Pfsense, OpenSSH, OpenSSL, OpenBSD, LibArchive, Firejails, Linux and others.. This list is long. An award was also received during the IETF 98 Hackathon.

Logan at the DevConMru 2017
Logan at the DevConMru 2017

The slide can be viewed here or on the Scriba website.

I was impressed how Avinash Meetoo, honorary member of the hackers.mu hacked the audience and shed some light to boost the students. Avinash mentioned himself about his passion for blogging and the importance of projecting his personality with the right vision.

At hackers.mu, we invite many to join us, but one have to work hard to attain a certain level of professionalism.  After the presentation, many came to congratulate us for the job done. We were around chatting with many sharing our work and job experience as well as the passion for coding. I once read a phrase in an old book as follows “You are what you eat”.  But things have change now because You are what You CODE!!