Category: Linux System

Securing MySQL traffic with Stunnel in a jail environment on CentOS

Stunnel is a program by Michal Trojnara that allows you to encrypt arbitrary TCP connections inside SSL. Stunnel can also allow you to secure non-SSL aware daemons and protocols (like POP, IMAP, LDAP, etc) by having Stunnel provide the encryption, requiring no changes to the daemon’s code. It is a proxy designed to add TLS encryption functionality to existing clients and servers without any changes in the programs’ code. Its architecture is optimized for security, portability, and scalability (including load-balancing), making it suitable for large deployments. – Stunnel.org

The concept that lies behind Stunnel is about the encryption methodology that is used when the client is sending a message to a server using a secure tunnel. In this article, we will focus on using MySQL alongside Stunnel. MariaDB Client will access the MariaDB server database using the Stunnel for more security and robustness.



Photo Credits: danscourses.com
Photo Credits: danscourses.com

I will demonstrate the installation and configuration using the CentOS distribution which is on my Virtual Box lab environment. I created two CentOS 7 virtual machines with hostname as stunnelserver and stunnelclient. We will tunnel the MySQL traffic via Stunnel. You can apply the same concept for SSH, Telnet, POP, IMAP or any TCP connection.

The two machines created are as follows:

  1. stunnelserver : 192.168.100.17 – Used as the Server
  2. stunnelclient : 192.168.100.18 – Used as the Client

Basic package installation and configuration on both servers

1. Install the Stunnel and OpenSSL package on both the client and the server.

yum install stunnel openssl -y

2. As we will be using Stunnel over MariaDB, you can use the MariaDB repository tools to get the links to download the repository. Make sure you have the MariaDB-client package installed on the stunnelclient which will be used as client to connect to the server. Also, install both packages on the stunnelserver. The commands to install the MariaDB packages are as follows:

sudo yum install MariaDB-server MariaDB-client

3. For more information about installations of MariaDB, Galera etc, refer to these links:

MariaDB Galera Clustering

MariaDB Master/Master installation

MariaDB Master/Slave installation

Configuration to be carried out on the stunnelserver (192.168.100.17)

 

4. Once you have all the packages installed, it’s time to create your privatekey.pem. Then, use the private key to create the certificate.pem. Whilst creating the certificate.pem, it will prompt you to enter some details. Feel free to fill it.

openssl genrsa -out privatekey.pem 2048

openssl req -new -x509 -days 365 -key privatekey.pem -out certificate.pem

5. Now comes the most interesting part to configure the stunnel.conf file by tunnelling it to the MySQL port on the stunnelserver. I observed that the package by default does not come with a stunnel.conf or even a Init script after installing it from the repository. So, you can create your own Init script. Here is my /etc/stunnel/stunnel.conf on the server:

chroot = /var/run/stunnel
setuid = stunnel
setguid = stunnel
pid /stunnel.pid
debug = 7
output = /stunnel.log
sslVersion = TLSv1
[mysql]
key = /etc/stunnel/privatekey.pem
cert = /etc/stunnel/certificate.pem
accept = 44323
connect = 127.0.0.1:3306

6. Position your privatekey.pem and certificate.pem at /etc/stunnel directory. Make sure you have the right permission (400) on the privatekey.pem.

7. Based upon the configuration in part 5, we will now create the /var/run/stunnel directory and assign it with user and group of stunnel:

useradd -G stunnel stunnel && mkdir /var/run/stunnel && chown stunnel:stunnel /var/run/stunnel

8.  The port 44323 is a non reserved port which I chose to tunnel the traffic from the client.

9. As we do not have the Init script by default in the package, start the service as follows:

stunnel /etc/stunnel/stunnel.conf

10. A netstat or ss command on the server will show the Stunnel listening on port 44323.

Configuration to be carried out on the stunnelclient (192.168.100.18)

11. Here is the stunnel.conf file on the stunnel client :

verify = 2
chroot = /var/run/stunnel
setuid = stunnel
setguid = stunnel
pid = /stunnel.pid
CAfile = /etc/stunnel/certificate.pem
client = yes
sslVersion = TLSv1
renegotiation = no
[mysql]
accept = 24
connect = 192.168.100.17:44323

12. Import the certificate.pem in the /etc/stunnel/ directory.

scp <user>@<ipofstunnelserver>:/etc/stunnel/certificate.pem

13. Based upon the configuration in part 11, we will now create the /var/run/stunnel directory and assign it with user and group of stunnel:

useradd -G stunnel stunnel && mkdir /var/run/stunnel && chown stunnel:stunnel /var/run/stunnel

14. You can now start the service on the client as follows:

stunnel /etc/stunnel/stunnel.conf

15. A netstat on the client will show the Stunnel listening on port 24.

16. You can now connect on the MySQL database from your client to your server through the tunnel. Example:

mysql -h 127.0.0.1 -u <Name of Database> -p -P 24



Tips:

  • When starting Stunnel, the log and the pid file will be created automatically inside the jail environment that is /var/run/stunnel.
  • You can also change the debug log level. Level is a one of the syslog level names or numbers emerged (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), or debug (7). All logs for the specified level and all levels numerically less than it will be shown. Use debug = debug or debug = 7 for greatest debugging output. The default is notice (5).
  • If you compile from source, you will have a free log rotate and Init scripts. Probably on CentOS, it’s not packaged with the script!
  • You can also verify if SSLv2 and SSLv3 have been disabled using openssl s_client -connect 127.0.0.1:44323 -ssl3 and try with -tls1 to compare.
  • For the purpose of testing, you might need to check your firewall rules and SELINUX parameters.
  • You don’t need MariaDB-Server package on the client.
  • Stunnel is running on a Jail environment. The logs and the PID described in part 5 and 11 will be found in /var/run/stunnel.
  • You can invoke stunnel from inetd. Inetd is the Unix ‘super server’ that allows you to launch a program (for example the telnet daemon) whenever a connection is established to a specified port. See the “Stunnel how’s to” for more information. The Stunnel manual can also be viewed here.

 


Out of Memory (OOM) in Linux machines

Since some months I have not been posting anything on my blog. I should admit that I was really busy. Recently, a friend asked me about the Out of Memory messages in Linux. How is it generated? What are the consequences? How can it be avoided in Linux machines? There is no specific answer to this as an investigation had to be carried out to have the Root Cause Analysis. Before getting into details about OOM, let’s be clear that whenever the Kernel is starved of memory, it will start killing processes. Many times, Linux administrators will experience this and one of the fastest way to get rid of it is by adding extra swap. However, this is not the definite way of solving the issue. A preliminary assessment needs to be carried out followed by an action plan, alongside, a rollback methodology.

If after killing some processes, the kernel cannot free up some memory, it might lead to a kernel panic, deadlocks in applications, kernel hungs, or several defunct processes in the machine. I know cases where the machine change run level mode. There are cases of kernel panic in virtual machines where the cluster is not healthy. In brief, OOM is a subsystem to kill one or more processes with the aim to free memory. In the article Linux kernel crash simulation using kdump, I gave an explanation how to activate Kdump to generate a vmcore for analysis. However, to send the debug messages during an out of memory error to the vmcore, the SYSCTL file need to be configured. I will be using a CentOS 7 machine to illustrate the OOM parameters and configurations.

1.To activate OOM debug in the vmcore file, set the parameter vm.panic_on_oom to 1 using the following command:

systctl -w vm.panic_on_oom=1

To verify if the configuration has been taken into consideration, you can do a sysctl -a | grep -i oom. It is not recommended to test such parameters in the production environment.

2. To find out which process the kernel is going to kill, the kernel will read a function in the kernel code called badness() . The badness() calculate a numeric value about how bad this task has been. To be precise, it works by accumulating some “points” for each process running on the machine and will return those processes to a function called select_bad_process() in the linux kernel. This will eventually start the OOM mechanism which will kill the processes. The “points” are stored in the /proc/<pid>/oom_score. For example, here, i have a server running JAVA.

As you can see, the process number is 2153. The oom_score is 21

3. There are lots of considerations that are taken into account when calculating the badness score. Some of the factors are the Virtual Memory size (VM size), the Priority of the Process (NICE value), the Total Runtime, the Running user and the /proc/<pid>/oom_adj. You can also set up the oom_score_adj value for any PID between -1000 to 1000. The lowest possible value, -1000, is equivalent to disabling OOM killing entirely for that task since it will always report a badness score of 0.

4. Let’s assume that you want to prevent a specific process from being killed.

echo -17 > /proc/$PID/oom_adj

5. If you know the process name of SSH Daemon and do not it from being killed, use the following command:

pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

6. To automate the sshd from being killed through a cron which will run each minute use the following:

* * * * * root pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done
7. Let's now simulate the OOM killer messages. Use the following command to start an out of memory event 
on the machine.
echo f > /proc/sysrq-trigger 

You will notice an OOM error message in the /var/log/messages.
As you can notice here, the PID 843 was calculated by the OOM killer before killing it. 
There is also the score number which is 4 in our case.


Before the 'Out of memory' error, there will be a call trace which will be sent by the kernel.



8. To monitor how the OOM killer is generating scores, you can use the dstat command. To install the dstat 
package on RPM based machine use: 
yum install dstat 

or for debian based distribution use:
apt-get install dstat

Dstat is used to generate resource statistics. To use dstat to monitor the score from OOM killer use:
dstat -top-oom


TIPS:

  • oom_score_adj is used in new linux kernel. The deprecated function is oom_adj in old Linux machine.
  • When disabling OOM killer under heavy memory pressure, it may cause the system to kernel panic.
  • Making a process immune is not a definite way of solving problem, for example, when using JAVA Application. Use a thread/heap dump to analyse the situation before making a process immune.
  • Dstat is now becoming an alternative for vmstat, netstat, iostat, ifstat and mpstat. For example, to monitor CPU in a program, use dstat -c –top-cpu -dn –top-mem
  • Testing in production environment should be avoided!

SAR command daily tips and tricks

As promised on Twitter days back, I would post some interesting tips and tricks using the SAR (System Activity Report) linux command. The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the count and interval parameters, writes information the specified number of times spaced at the specified intervals in seconds. If the interval parameter is set to zero, the sar command displays the average statistics for the time since the system was started.die.net 

Understanding SAR and its main configuration files

The SAR command is part of the sysstat package which is a multi-purpose analysis tool and it is useful to pin point specific issue related to CPU, Memory, I/O and Network. The command is really useful especially to plot the output on a graph for visual analysis and reporting. One example of such tool is GNUplot. To install GNUplot and SAR use the command yum install sysstat gnuplot. The configuration file of SAR is located at /etc/cron.d/sysstat and /etc/sysconfig/sysstat . If you would perform a rpm -ql sysstat | less , you would noticed that there are other binaries such as iostat,mpstat,pidstat etc.. that comes along with the package sysstat.

Difference between SA and SAR logs

In the directory /etc/cron.d/sysstat you would noticed that there is a cron which have been set up by default to run every ten minutes. The purpose is to write a log in the directory /var/log/sa . In this directory there are two type of files starting with sa and sar. SAR is the text file while SA is a binary. The sa file – Binary is updated every 10 minutes whereas the sar file is written at the end of the day. This parameter have been configured in the cron itself. By using the file command you would know which one is a binary or a text file.

To open the sa file, you need to use the command sar -f . Here is an example:

The /etc/sysconfig/sysstat file allows you to configure how long you want to keep the log, compression etc..

Some ways to use the SAR command

You can also visualize sar logs live using the sar command with the start and ending second. Let’s say you have run a command on the background or simply want to track the resource status for some seconds, you can use the command sar 1 3 Here is an example with the command

sar 1 3

Checking the load average, we apply the same principle but with the following command. The load average will also include load on each processor.

sar -q 1 3

To check for memory being consumed per seconds, use the following command

sar -r 1 3

To check number of memory coming in and out of the swap space, use the following command

sar -W 1 3

For the Disk I/O read write per seconds use the following command. Read/Write on disk also depend on the hardware

sar -b 1 3

For info about the CPU use the following

sar -u 1 3

To monitor the network activity in terms of packets in and out per interface received and compressed, use the following command

sar -n DEV 1 3

If the sar file has not been generated yet from the binary, you can use the following command, let’s say to convert it to a text in the /tmp directory.

sar -A -f  sa25 > /tmp/sar25

KSAR Graph with SAR logs

Now, in production environment, you need to to analyze for example at a specific time where memory or CPU was high. This can be done by means of a graph. I use the Ksar program. Ksar is a BSD-Licensed JAVA based application which create graph based on sar logs. You will need to install JAVA Runtime and launch the run.sh script to install the Ksar program. Once downloaded, just click on ‘Data’ and ‘Load from text file’. This is an example of swap usage

TIPS:

  • The SAR output by default is in 12 hour clock format. To make it become a 24 hour clock format edit the .bashrc file and insert the parameter alias sar=”LANG=C sar
  • GNUPlot is one another application to plot your information on a graph for better analysis.

What goes on behind the Network Time Protocol ?

Several times, I had discussions with friends on how NTP works! What is the logic behind NTP and its configurations? We noticed that there are several terms and calculations to grasp especially when it comes to debugging. Well, I decided to make a research on it and shed some ideas on NTP – Network Time Protocol. These recent days, we have noticed several vulnerabilities and attacks going on the NTP servers. NTP is a protocol designed to synchronize the clocks of computers over a network.

Photo credits: networktimefoundation.org
Photo credits: networktimefoundation.org

NTP is a utility where timestamps are used. Examples are logs, database replications, the time packets  exchanged in a network. NTP uses its own binary format and runs on port 123 UDP. RFC 1305 and RFC 2030 give detailed explanations of NTP.

The logic behind NTP

In brief, packets are exchanged between the NTP server and the client in the following order. It is to be noted that latency is an important issue when it comes to NTP:

  1. The NTP client will send a request with a timestamp.
  2. The NTP server will return the packet with 3 timestamps.
    • echo of the client timestamp
    • The timestamp of the received timestamp by the server
    • The timestamp response sent by the server.
  3. The client will then estimate the offset (the difference in timestamps between the client and the server)

A client may have several NTP servers configured, but will synchronize with only one NTP server. A server may also take some time to respond to a client depending when it is not busy. NTP packets are exchanged between 64 – 1024 seconds for each server. – This configurations are called “minpoll” and “maxpoll”

Some basic configuration on a CentOS 7 machine

The command timedatectl can be used to check if NTP is enabled or not

[[email protected] ~]# timedatectl | egrep -i ntp
     NTP enabled: yes
NTP synchronized: yes

You can also check if the service is running using the following command;

systemctl status chronyd.service

The configuration file is located by default at /etc/chrony.conf . You will notice that the NTP servers are configured by default in that file.

[[email protected] ~]# head -n 7 /etc/chrony.conf 

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

You can check if your machine are synchronised from the sources with the following command. The column Name/IP address are the location from where the time is being synchronized.

[[email protected] ~]# chronyc sources


210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ cpt-ntp.mweb.co.za            2   6   367    40    +29ms[  +29ms] +/-  151ms
^* cpto-afr-01.time.jpbe.de      2   7   377    43    +69ms[  +66ms] +/-  178ms
^+ ntp2.inx.net.za               2   6   377    43    +64ms[  +64ms] +/-  181ms
^+ ns.bitco.co.za                3   7   377   107    +79ms[  +77ms] +/-  199ms

A drift means a deviation. A drift happens when  the hardware clock is either fast or slow compared to the NTP server clock. The drift file contains 2 values. If it’s a positive number, it means the clock is fast from the NTP server whereas if it is a negative number, it means the clock is slow compared to the NTP server. Here i have a slow clock.

[[email protected] ~]# cat /var/lib/chrony/drift 

         -870.203668          1484.980507
The Maths - How the NTP client adjust its response from the NTP server

Now, we will get into the math behind the logic as discussed previously what happens when the NTP client request the time from the NTP server.

  1. NTP Client A send request to server X – Let’s assume A=100 where 100 is the time of the client
  2. NTP Server X received the request after some secs. – Let’s assume X= 150 where 150 is the time of the server.
  3. Being given that, the request from NTP client is not necessarily served immediately, there is lapse of time at this point. let’s assume that X is now 160
  4. We now have 3 values i.e; The time the client sent the request, the time (real time) the server received the request and the time the server want to respond back.
  5. Now the NTP client gets the request back at 120. This is because the NTP client has its own time.
  6. Client will now determine the time using the formulae B-A – (Y-X) which means 120-100-(160-150) = 10 seconds
  7. Client assumed that the time it took to get the response from server to client is 10/2 = 5 seconds. Assuming 5 seconds is the latency.
  8. Now the client adds 5 seconds to  the server time at the time it received the response which makes 160 +5 = 165 seconds.
  9. The client knows it needs to add 45 seconds to its clock. This is done by subtracting 165 – 120 = 45 seconds where 45 seconds is the difference between the client and the server clock to which the client will set forward its clock by 45 seconds. This indication will be given in the drift file in PPM – Parts Per Million.

TIPS:

  • If the iburst parameters are removed, communication between the server will 8 times faster.
  • You can also increase the verbosity of the command chronyc sources by adding the parameter -v to it and detailed explanation of the values will be given i.e; chronyc source -v
  • You can pick up different NTP servers from the NIST website and restart the NTP service (chronyd).
  • NTP was invented by David L. Mills in 1981 and it is based on Marzullo’s algorithm to get accurate time from several sources.
  • Timestamps of NTP are stored in seconds and it is 64 bit in size – 32 bit for number of seconds and 32 bit for fraction of seconds.
  • Number in drift file is measured in PPM – Parts per million
  • Offset – The difference in timestamps between the client and the server
  • Burst – The speed of communication between NTP client and NTP server will be 8 times more if “burst” is used.
  • The drift value is in PPM – Parts Per Million.
  • To convert into PPM is easy. Since we have 86,400 seconds in a day, therefore, 86,400 / 1,000,000 = 0.0864 PPM
  • If my drift file shows a value of Z  where Z = 30.3 simply do (30.3 x 0.0864) to get the drift file into milliseconds.

How i optimised my WordPress website ?

Some days back, I was brewing up some plans to optimise my website source codes, HTTP headers, latency and other security aspects. I had to carry out some analysis and research using some tools available on the internet. I should admit that, at first, it looked pretty simple, but it was not. For instance, I did not permit myself to directly modify the production environment. So, I had to migrate it on a pre-production environment. Page caching was yet another issue which could trick oneself after modifications.

Since my website is behind Cloudflare, which is already an advantage in terms of security, performance, reliability and insight, it does not mean that the website cannot be hacked. According to sucuri.net, websites using WordPress CMS are constantly being hacked. Of course, it depends on the mode of attack and the infection impact.

Photo credits: sucuri.net
Photo credits: sucuri.net

Migrating to TLS

Migrating a CMS which already has several articles posted can be an issue as the URLs are already recorded in the database as well as in the source code itself. Also, there were links on the website which were not pointed on HTTPS. After moving to the HTTPS version, errors such as “Mixed content” could be noticed when accessing the website. One of the interesting free feature of Cloudflare is that everyone can have a free SSL certificate issued by Comodo. You will have to generate your certificate and your private key from Cloudflare and point it on your Virtual Host.

Some corrections on WordPress source code needed to be added in the wp-config file as follows:

define('WP_HOME','https://tunnelix.com/');

define('WP_SITEURL','https://tunnelix.com/');

On top of that, there seemed to be lots of URLs on the database itself that needed corrections using the following commands:

update wp_options set option_value = replace(option_value, ‘http://www.tunnelix.com’, ‘https://www.tunnelix.com’) where option_name = ‘siteurl’;


update wp_posts set guid = replace(guid, 'http://www.tunnelix.com', 'https://www.tunnelix.com');

update wp_posts set post_content = replace(post_content, 'http://www.tunnelix.com', 'https://www.tunnelix.com');

However, there are some tricks to identify those non-HTTPS URLs by making a dump of the database and do a “Grep” in it, followed by a “Sed” to eliminate those unwanted parameters. Once the “Mixed Content” errors have been identified, I launched a scan on the Qualys SSL Labs website. The result was an “A+”. You can also use the Htbridge free SSL server test which is pretty fascinating especially to verify PCI DSS Compliance, HIPAA compliance, NIST guidelines and industry best practice in general. If all those criteria have been met, then you would score an “A+” rather than an “A” or worse a “F”.

Source code optimisation and Page speed verification

This can be verify using the GTmetrix tool available for free online. I noticed that my rank was a “C”. This was caused due to lack of minified HTML and CSS, and Image dimension. To handle the minify HTML errors, I enabled the plugin Minify HTML Markup on WordPress itself which corrected these errors. To tweak the Image dimension i downloaded the tool Optipng from Epel repository:

optipng.x86_64                  0.7.6-1.el6                        @epel        

For example, if you want to optimize a specific image, use the following command:

optipng -o2 Screen-Shot-2016-12-24-at-1.04.45-AM.png

Another verification was made on GTmetrix website and i noticed that the result was then an “A”

from GTMETRIX.COM

Tweaking the Web server HTTP headers

Htbridge will surely give you an overview of the web server security and will accompany you step by step to get a better result.

Of course, since the website is behind cloudflare,  it is limited to certain security tweaks such as Public-key-pins.The Public Key Pinning Extension for HTTP (HPKP) is a security feature that tells a web client to associate a specific cryptographic public key with a certain web server to decrease the risk of Man-in-the-Middle (MITM) attacks with forged certificates. I found an interesting article on https://raymii.org which explained how to activate the HPKP. 

Once you are in possession of your certificate and Private key, you can create the public key and a token will be received to activate the HPKP extension. The following commands can be used to get the token and the public key.

# openssl x509 -noout -in certificate.pem -pubkey | openssl asn1parse -noout -inform pem -out public.key;

# openssl dgst -sha256 -binary public.key | openssl enc -base64
 4vr+koFuogsfghGjgvpsqQIIikg5KowHTIGNQ5Prspc=

However, it looked that HPKP is not supported on Cloudflare. But, there are other issues such as HSTS. HTTP Strict Transport Security (HSTS, RFC 6797) is a web security policy technology designed to help secure HTTPS web servers against downgrade attacks. HSTS is a powerful technology which is not yet widely adopted. CloudFlare aims to change this. I enabled it as per recommendations by Cloudflare.

A curl on the url https://tunnelix.com now prompts the following headers :

No system is perfectly secure, but I believe that these modifications are worth to adventure around. I should say I was really impressed by free tools such as the Qualys SSL test, HTbridge free SSL and Web security test and Gtmetrix in terms of page speed.

 Hello Tunnelers, this is my first article for the year 2017, I seize this opportunity to wish my readers a Happy New Year 2017 and wish you all lots of prosperity. – TheTunnelix