Dare to do a brute force attack again!

Dare to do a SSH Bruteforce attack again and you are banned!! I have noticed that there are several DDOS SSH botnets attack these days on my server. Despite that i would prefer SSH to listen on port 22, i can imagine how many attempts can be made to breakthrough it. Though these attacks are very common, it can increase CPU consumption on your server and consequently the server can die. However, if you did not protect the server from malicious SSH remote connection, things can get pretty dangerous and the attacker can take over the machine.

fail2ban
Photo credits – fail2ban.org

Fail2Ban is one of the tools which you can installed on your machine to ban IPs that show malicious signs. However, today with the help of Kheshav, we have decided to find a solution to reveal all the IPs to the public. From the fail2Ban log we can find all IPs that that are being banned. The solution was an easy one.

1.Install Nodejs, npm package

yum install nodejs npm

2. Install frontail with the npm utility

npm install frontail -g

3. Now you can launch frontail on any port as a demon with the following command

frontail -p {port number here} -h {IP or Hostname here} {location of your log} -d

Afterwards, you have to include the IP, the port number and the location where you want the log to be streamed live.

Here are the banned IPs – US time attempting some brute force on tunnelix.com. You can also view the IPs on the right side widget of the blog. It might take some few seconds before loading.

There are several websites where you can report IPs for abuse as well as verification of precedent attacks. We are still brewing up some ideas to produce a better and well defined output of the log.


Simple Master-Slave replication on MariaDB

This article will explained all the necessary configurations to set up a simple Master-Slave replication on MariaDB. I have test it on CentOS7. More articles on MariaDB optimisation, Master-Master replication and Galera cluster are coming up soon. Well, i was introduced to MariaDB in the year 2014 by Joffrey Michaie of SkySQL at Flying Dodo video conference room, Bagatelle, Mauritius during a LUGM [Linux User Group of Mauritius] meet up. You can still view out the video on the LUGM Youtube Channel.

What is Mariadb ? “MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. Being a fork of a leading open source software system, it is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle. ” – Wikipedia. 

ice_logo-5dcea9e47b780ff52f75c3c3304d54827f56211e
Photo Credits – Mariadb.com

I have set up 2 Centos 7 Labs; one as Master and the other as Slave.

You will need to install MariaDB on both servers to start with.

Make sure port 3306 is opened to enable replication.

Since, the test is based on 2 Virtual Box labs, i have temporarily disabled iptables.

At the time i am writing this article, the latest version is 10.1.8-stable.

You can easily download MariaDB through the repository configuration tool

1.After having installed a fresh MariaDB on both virtual machines – Master / Slave, you will need to configure the root access to MySQL by using the following command. It will prompt you to enter a password which is a blank password by default. Then, you will have to enter a password for root which will prompt you to remove anonymous users, disallow root login, remove test database and access to it and reloading privilege tables. Just press ‘Y’ by following the command below:

mysql_secure_installation

2. After setting up the root user of MySQL, you can test it with a fake or no password to ensure the root password has been set up effectively. Next step, is to create a database with some tables in it or simply import a database. I created a table in the database called ’employees’ on the Master.

create database employees;
use employees;
create table profile (name char(30), age int(2), address varchar(40)) ;

3.  So, a database has been created on the Master. We will now edit the /etc/my.cnf on the master and under the [mysqld] enter the following:

server-id=1
log-bin = mysql-bin

4. Next step, is to create a user for replication purpose to the slave server (The command need to be launched on the master server). The ip 192.168.1.8 is the slave server. Once the grant replication is launched the user repliuser will be created automatically. Afterwards, launch the command flush privileges so that the server noticed these changes to load the grant tables into memory.

MariaDB [(none)]>

grant replication slave on *.* to [email protected]'192.168.1.8' identified by 'replipassword';

MariaDB [(none)]>

flush privileges;

5. At this point, if you launch a show master status; on the master server you will have a prompt “Empty set“. To enable the master, we need to restart MariaDB. As we want to replicate this master database on the slave server, we need to stop mysql to continue writing on the table.

MariaDB [(none)]>

flush tables with read lock;
systemctl restart mariadb

6. Connect back to MariaDB and launched a show master status\G You will need to find a result similar to this screenshot.

Screenshot from 2015-11-03 20:18:21

7. We now have a file name and a position. These two values will be use to set up the slave database. Do a mysql dump on the master database and import it to the slave server (I copied it from master to slave on the /home folder).

On the Master server:

[[email protected] ~]#

mysqldump -u root -p employees  > employees.sql

[[email protected] ~]#

scp employees.sql [email protected]:/home

8. Now, we log in  to the slave server. Create a database employees just as in the master server and import the database employees to the slave database server.

[[email protected] home]#

mysql -u root -p employees < employees.sql

9. Edit the /etc/my.cnf in the slave server and enter the following. Note the value of the server-id should be greater than that of the master server, otherwise it may not work. Then restart MariaDB on the slave server.

server-id = 2
systemctl restart mariadb

10. Right now, the slave server cannot identify the master slave to synchronize the database for replication.

On the master server:

unlock tables;

On the slave server:

MariaDB [(none)]>

change master to master_host='master', master_user='repliuser', master_password='replipassword', master_log_file='mysql-bin.000001', master_log_pos=313;

11. You can now start the slave replication. Afterward, you can type a show slave status\G;

start slave;
Screenshot from 2015-11-03 21:04:07
show slave status\G indicating replication on slave server

Tips:

  • To find the list of services for which specific ports can be opened, use the command firewall-cmd –get-services. Hence, you can add the mysql service with the command firewall-cmd –permanent –add-service-mysql. After adding the mysql service you will need to reload the firewall service with the command firewall-cmd reload. To verify if a certain rules have been loaded in the firewall use the command firewall-cmd –list-all
  • To check which database you have, you can fire the command show schemas;
  • At step 9, If you want to replicate a specific database on the slave server, you can use the following parameter in the /etc/my.cnf in the slave under [mysqld] use replicate-wild-do-table=employees.% where the .% means all tables under the database employees.
  • Since you will be on Virtual Box or Vmware, you may need to edit your /etc/hosts so that each server recognized each other as the Master_User and Slave_User respectively. In this case, in the /etc/hosts for the master enter the IP of the slave server followed by the hostname of the slave and vice versa. Test it with a ping. In my case, if i ping master from slave, it will answer promptly.

Analyzing vmcore with crash

In the article linux kernel crash simulation using kdump, i gave a brief idea as to how to generate a vmcore file during a crash or hangs. On this article, i will emphasize on the analysis of a vmcore which has been generated and the tool ‘crash’ which can be used for advance analysis. In a future article, i will elaborate on how to decode the detailed information given with the crash tool. Lets see how to use the crash utility first.

tux-logo

1.Download the package kernel-debuginfo and kernel-debuginfo-common. You will notice a vmlinux file has been created just after the installation under /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux

Screenshot from 2015-11-02 12:49:34

yum install kernel-debuginfo kernel-debuginfo-common -y

2. Now, we will launch the crash utility which can be used for live debugging. By default, it will give you the info from the available vmcore.

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686

3. However, you can specify a specific vmcore file with the following command by mentioning the location of the vmcore

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686 /var/crash/127.0.0.1-2015-10-30-00\:12\:34/vmcore

Screenshot from 2015-11-02 13:52:46

4. You will have several information related to the kernel as well as the most interesting stuff is what have cause the panic that is the warning message. In this case it is a “SysRq”. If you remember from the last article we had fired an echo c > /proc/sysrq-trigger. Under the state tab it also gave an indication of the task SYSRQ running.

5. We can also check the process running on the crash utility using the PID given.

Screenshot from 2015-11-02 14:03:396. Another interesting command is the bt which enable us to see execution history of the process

Screenshot from 2015-11-02 14:05:22

7. The sys command will give you an idea of the system. ps | grep “>” – will show you running processes during time of the crash. mount command will show you partitions mounted etc..  h command for the history.

Tips:

  • A good crash utility manual page can be found at people.redhat.com/anderson. Almost all info is available there.
  • To be able to dowload the kernel-debuginfo package, you will need to activate the repo located at /etc/yum.repos.d
  • The version of the kernel of the machine should corroborate with that of the kernel-debug-info otherwise it will not work.

Linux Kernel-4.3 Compilation from source

The Linux Kernel 4.3 has been released today, Monday the 2nd of November 2015. I have compiled it from source on a Virtual Box CentOS 7 minimum install virtual machine for some further testing. I have also use my same old configuration file. You can also view detailed packages and commits on the git repo. Here, is a brief idea how to compile it from source.

Linux_kernel_map
Linux Kernel Map – Photo credits Wikipedia

1. You will need to download all the pre-requirements if you are on a minimum install.

yum groupinstall "Development Tools"
yum install ncurses-devel bc hmaccalc zlib-devel elfutils-libelf-devel binutils-devel qt-devel

2.Download the wget tools to download the Kernel itself.

yum install wget

3.Download and untar the kernel directory

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.3.tar.gz
tar -xvzf linux-4.3.tar.gz

4. You will need to ensure that the decompressed directory is in the /usr/src/kernels directory. If you have untar it at a location other than this one move the linux-4.3 directory in the /usr/src/kernels

5. Choose your default kernel configuration options

make menuconfig

6. To use the old config file

make oldconfig

7. Compiling the kernel

make

8. Installing the kernel

make modules_install install

Tips:

  • Be sure to get rid of to many old kernels files in the /boot directory to do not get confused.
  • You can also use the command make olddefconfig to set the default values without prompting anew for configuration.
  • To set different boot options use the command sudo grub2-set-default 0 – 0 in this case is the default kernel.
  • The command make usually take lots of time. If you have 4 vCPU, you can use make as this: make -j 4 where j stands for jobs and 4 for all the 4 CPUs
  • uname -r allows you to find your kernel version. Example uname -r gives me 3.19.0-25-generic ; i.e the letter 3 is the major, number 19 is the minor (developmental stage) and 0 is the revision number

Linux Kernel crash simulation using Kdump

There are several reasons for a Linux Kernel Crash which may include hangs, hardware and software errors. We usually consider a “Kernel hangs” and a “Kernel crash” as just a ‘crash’. In fact, these are totally two different issues; a “hang” occurs due to a time consuming operation whilst a “crash” occurs instantaneously leading to a reboot. However, during the crash process prior to the reboot, the kernel will register “oops” messages.

In this article, i will lay emphasis on the installation of the tools for analyzing Linux Kernel crash. I will elaborate more on Linux Kernel errors in a future article. Right now, we will look at the installation of Kdump – Kernel dump, a Linux kernel dumping mechanism which uses a ‘kexec mechanism’ to enable us to collect a ‘dump’ of the Linux kernel called “vmcore” (virtual memory core). Whatever event occurred during the time of the crash is registered in the “vmcore” for future analysis.

tux-logo

“Kdump uses kexec to quickly boot to a dump-capture kernel whenever a dump of the system kernel’s memory needs to be taken (for example, when the system panics). The system kernel’s memory image is preserved across the reboot and is accessible to the dump-capture kernel.”Kernel.org

Follow the steps below:

1.On both CentOS 6/7, you will need to install the kexec package using the command yum install kexec-tools

2.vim /boot/grub/grub.conf and for the kernel you are actually running edit the parameter crashkernel = auto and replace it with crashkernel= 128M (I tested it on a virtual machine with 1024MB)

3.Start the Kdump service using the command service kdump start

4.Save this parameter and verify it using the command cat /proc/cmdline. Here is an screenshot how it should look

Screenshot from 2015-10-29 23:57:42

5.You would noticed that the Kdump have the following configuration files using the command rpm -qc kexec-tools

  • /etc/kdump.conf
  • /etc/rc.d/init.d/kdump
  • /etc/sysconfig/kdump
  • /etc/udev/rules.d/98-kexec.rules

6.You can also choose the location to save your vmcore. By default, it will be saved  in /var/crash/. However, if your /var directory is assigned to a different partition with low disk space, you can choose exactly where you want to generate your vmcore by modifying the parameter path /var/crash in the /etc/kdump.conf file.

7. After modification, you will need to restart the kdump service  using the command service kdump restart.

8. Now the last step is to crash the machine thus creating a vmcore. Use the command echo c > /proc/sysrq-trigger. You would noticed that this will take some time and the server will reboot by itself. A crash simulation has been done.

9. You will noticed now after the reboot that a vmcore file has been created in the /var/crash directory.

Screenshot from 2015-10-30 00:15:18

10. The size of the vmcore depends on the consequence of the crash. In this simulation its just 19M. It also depend on the kernel activity during the time of the crash.

Tips:

  • You can also specify crashkernel = auto on a 64 bit machine. However, you can calculate it as follows:
  • If your RAM is greater than 0 GB  and less than 2 GB use 128 MB
  • If your RAM is greater than 2 GB and less than 6 GB use 256 MB
  • If your RAM is greater than 6 GB and less than 8 GB use 512 MB and so on
  • You can also test with less than 128 MB, it may work but the reliability and consistency is cautioned
  • If the kdump service does not start after a fresh installation, you might need to reboot your machine.
  • Since you have allocate a portion of the memory to the kdump, you might need to reboot your machine again and test it with a free -m