Author Archives: Nitin J Mutkawoa

Anatomy of a simple dig result

The ‘dig’ (Domain Information Gropper) command is one of the tools which is frequently used to troubleshoot DNS and BIND configurations. Its main purpose is to perform DNS lookups and query DNS servers. Though the subject is vast, I decided to blog some DNS stuff under the ‘Bind and DNS tools’ category which I just created. I will keep on updating this article as I keep on finding interesting dig commands.


Screenshot from 2015-11-08 15:45:52

Let’s analyze the result from a simple dig google.com. You would have a result similar to this one (In green). By default, dig perform query A record when launched without any arguments.

1. I made a dig google.com on my Linux terminal

[[email protected] ~]# dig google.com

2. The header section starts here. Several files in /etc/ld.so.* is being read and the dig command will also launch a uname with the argument sys and node. The uname is already inbuilt in the code of the dig command. It then reads the /etc/resolv.conf


; <<>> DiG 9.9.4-RedHat-9.9.4-18.el7_1.5 <<>> google.com

3. The ;; global options: +cmd is referred to the default arguments sets by dig to use only the +cmd variables.  The opcode value is always static. The status is to inform us if any error occurred during the query. Each query is also associated with an id number ( ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35133).

The flags qr (query response), rd (recursion desired) and ra (recursion available) are also information retrieved from the DNS header. As per the IETF RFC1035, when a dig with the default arguments is performed it will flag the qr, rd, ra and when the bit is 1 it’s a response and 0 for a query. Therefore ‘qr’ appears as 1

The ANSWER:2 is the numbers of answers received in the Answer section, same for QUERY, AUTHORITY and ADDITIONAL.


 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35133
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

4. The “question section” is what you are querying for. In this case, a ‘dig’ has been done on the A record. An A record simply means Adress i.e; the address associated with the website. We have several DNS records types which I will elaborate in future articles.

;; QUESTION SECTION:
 ;google.com.            IN    A

5. The number 173 is the TTL (Time To Live), IN refers to Internet i.e the class in which it is. TTL  is a 32 bit signed integer which corresponds to the time interval a record can be cached before the information is again queried. A TTL zero is used for extremely volatile data.

To resume we read it as follows: Google.com has a 173 seconds Time To Live on the Internet with the IP Address 74.125.226.168


;; ANSWER SECTION:
 google.com.        173    IN    A    74.125.226.168
 google.com.        173    IN    A    74.125.226.162

6. This section acts like a stat section. Information is given about the time it takes to query. The server IP address i.e; 4.2.2.1 and the port number 53 which is associated with it. The date and finally the message size received is 204 bytes.

;; Query time: 22 msec
 ;; SERVER: 4.2.2.1#53(4.2.2.1)
 ;; WHEN: Sun Nov 08 01:19:23 EST 2015
 ;; MSG SIZE  rcvd: 204

More analysis can be performed by launching a strace in front of a dig command. The RFC 1035 is also of great help. You can also check out the Internet System Consortium (ISC) website for more details.

Tips:

    • dig -t MX google.com will show you in the list of MX records in the ‘Answer Section’
    • A  dig result is composed of only 5 parts i.e; Header, Question (question for the name server), Answer (resource records answering the question ), Authority (resource records pointing towards an authority) and Additional (resource records holding additional information).
    • To filter information from a default dig command you can use dig google.com +nocomments +noauthority +noadditional +nostats which will give you only the answer. With an additional +noanswer won’t give you anything.
    • However, the reverse way to filter dig results with a specific answer can be dig google.com +noall +answer will give you only the answer section.
    • Another interesting tip from Keith Wright is to use recursion during the name resolution using the command dig +trace google.com.


Simple Master-Master replication on MariaDB

Lets set up a simple Master-Master database replication on MariaDB. I have already posted an article on the setting up of Master-Slave database replication. You can test it on Virtual Box or Vmware labs for your own analysis and benchmark. Lets called both MariaDB Master servers as Master1 and Master2.


ice_logo-5dcea9e47b780ff52f75c3c3304d54827f56211e
Photo credits – Mariadb.org

1. Create the 2 Labs (test it on Centos7) and edit your /etc/hosts so that each server can ping each other via the hostname. On Master1, edit your /etc/my.cnf file and on the top enter the following parameter:


[mysqld]
server-id=1
log-bin=mysql-bin

2. On master1 SQL prompt, create a user for replication slave followed by flush privileges.

MariaDB [(none)]> grant replication slave on *.* to [email protected]'%' identified by 'replipassword';
MariaDB [(none)]> flush privileges;

3. Flush the tables for read-only access.


MariaDB [(none)]> flush tables with read lock;

4. At this point, if you do a show master status, you need to have a file with the mysql-bin incrementing number as well a  position number.

MariaDB [(none)]> show master status\G
File: mysql-bin.000001
Position: 612
Binlog_Do_DB:
Binlog_Ignore_DB:
1 row in set (0.00 sec)

5. Now, move on to the Master2 server, I assume you have already start MariaDB, setting up mysql_secure_installation, and edit the /etc/hosts file etc..

6. On Master2 edit your /etc/my.cnf file and on top enter the following parameter, after which you can restart the MariaDB service.


[mysqld]
server-id=2

7. Now on the Master1 server, do a dump of the database MySQL and import it to Master2, so that we make sure it starts and synchronize with the same conf. So, I did a dump of the MySQL database on master1 and SCP it to master2

[[email protected] yum.repos.d]# mysqldump -u root -p --database mysql > mysql.mysql
[[email protected] home]# scp mysql.mysql [email protected]:/home

8. On Master2 import the MySQL database


[[email protected] home]# mysql mysql -u root -p <  mysql.mysql 

9. Now, we will temporarily run Master2 as the slave.

MariaDB [(none)]> change master to master_host='master1', master_user='repli', master_password='replipassword', master_log_file='mysql-bin.000001', master_log_pos=612;
MariaDB [(none)]> start slave;

10. At this point, you would notice that the Slave is running correctly by verifying with a show slave status\G. Do check the Slave_IO_Running, Master_User, Log_Pos and the Slave_SQL_Running parameters.

11. Now that Master1 is the master and Master2 is the slave, we will perform the reverse order to reach our goal. At this level, you can unlock the tables which we did at step3. On master1 fire this command.

unlock table

12. On Master2 (which is actually a slave), re-edit my.cnf file and under the [mysqld] and server-id add the following parameter. It should look like this


[mysqld]
server-id=2
log-bin=mysql-bin

13. On Master2 restart MariaDB with the command systemctl restart mariadb and logging to Mysql and do a show master status. The result should be similar to this.

MariaDB [(none)]> show master status\G
File: mysql-bin.000001
Position: 313
Binlog_Do_DB:
Binlog_Ignore_DB: 

14. On Master1, a show slave status will prompt you an empty set. Now since we have dumped the MySQL database to Master2, we can now just run Master1 as the slave, i.e to repeat the process as in step 9. Launch this command on Master1 to create it as a slave. The File and Position should be taken from the Master2.

MariaDB [(none)]> change master to master_host='master2', master_user='repli', master_password='replipassword', master_log_file='mysql-bin.000001', master_log_pos=313;
MariaDB [(none)]> start slave;

15.  Test by creating database and tables on both servers, it will replicate on each other.

Tips:

    • Each time you edit my.cnf file, you need to restart MariaDB so that the conf take effect.
    • After having created the Master-Slave replication at step 10, you can check on both Master1/2 with a netstat -ntpla | egrep -i established.*mysql. You would notice the established connection and the port it’s listening.
    • The command host with the IP established will also confirm its from the specific host.
    • Further testing on Master1 can also be done with command as mysql -u root -p -e “show processlist” which will give you an indication of the state of the server.
  • The value of File and Position of Master1 when using the command show master status should correspond with the values of Master_Log_File and Read_Master_Log_Pos of Master2 with the command show slave status and vice versa.

Dare to do a brute force attack again!

Dare to do an SSH Bruteforce attack again and you are banned!! I have noticed that there are several DDOS SSH botnets attack these days on my server. Despite that I would prefer SSH to listen on port 22, I can imagine how many attempts can be made to break through it. Though these attacks are very common, it can increase CPU consumption on your server and consequently the server can die. However, if you did not protect the server from malicious SSH remote connection, things can get pretty dangerous and the attacker can take over the machine.


fail2ban
Photo credits – fail2ban.org

Fail2Ban is one of the tools which you can install on your machine to ban IPs that show malicious signs. However, today with the help of Kheshav, we have decided to find a solution to reveal all the IPs to the public. From the fail2Ban log, we can find all IPs that that are being banned. The solution was an easy one.



1.Install Nodejs, npm package

yum install nodejs npm

2. Install frontail with the npm utility

npm install frontail -g

3. Now you can launch frontail on any port as a demon with the following command

frontail -p {port number here} -h {IP or Hostname here} {location of your log} -d

Afterward, you have to include the IP, the port number and the location where you want the log to be streamed live.

Here are the banned IPs – US time attempting some brute force on tunnelix.com. You can also view the IPs on the right side widget of the blog. It might take some few seconds before loading.



There are several websites where you can report IPs for abuse as well as verification of precedent attacks. We are still brewing up some ideas to produce a better and well-defined output of the log.

Simple Master-Slave replication on MariaDB

This article will explain all the necessary configurations to set up a simple Master-Slave replication on MariaDB. I have tested it on CentOS7. More articles on MariaDB optimization, Master-Master replication, and Galera cluster are coming up soon. Well, I was introduced to MariaDB in the year 2014 by Joffrey Michaie of SkySQL at Flying Dodo video conference room, Bagatelle, Mauritius during a LUGM [Linux User Group of Mauritius] meet up. You can still view out the video on the LUGM Youtube Channel.


What is MariaDB? “MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. Being a fork of a leading open source software system, it is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle. ” – Wikipedia. 


ice_logo-5dcea9e47b780ff52f75c3c3304d54827f56211e
Photo Credits – Mariadb.com

I have set up 2 Centos 7 Labs; one as Master and the other as Slave.

You will need to install MariaDB on both servers to start with.

Make sure port 3306 is opened to enable replication.


Since the test is based on 2 Virtual Box labs, I have temporarily disabled IPtables.

At the time I am writing this article, the latest version is 10.1.8-stable.

You can easily download MariaDB through the repository configuration tool

1. After having installed a fresh MariaDB on both virtual machines – Master / Slave, you will need to configure the root access to MySQL by using the following command. It will prompt you to enter a password which is a blank password by default. Then, you will have to enter a password for root which will prompt you to remove anonymous users, disallow root login, remove the test database and access to it and reloading privilege tables. Just press ‘Y’ by following the command below:


mysql_secure_installation

2. After setting up the root user of MySQL, you can test it with a fake or no password to ensure the root password has been set up effectively. Next step, is to create a database with some tables in it or simply import a database. I created a table in the database called ’employees’ on the Master.

create database employees;
use employees;
create table profile (name char(30), age int(2), address varchar(40)) ;

3.  So, a database has been created on the Master. We will now edit the /etc/my.cnf on the master and under the [mysqld] enter the following:


server-id=1
log-bin = mysql-bin

4. Next step, is to create a user for replication purpose to the slave server (The command need to be launched on the master server). The IP 192.168.1.8 is the slave server. Once the grant replication is launched the user repliuser will be created automatically. Afterward, launch the command flush privileges so that the server noticed these changes to load the grant tables into memory.

MariaDB [(none)]>

grant replication slave on *.* to [email protected]'192.168.1.8' identified by 'replipassword';

MariaDB [(none)]>

flush privileges;

5. At this point, if you launch a show master status; on the master server you will have a prompt “Empty set“. To enable the master, we need to restart MariaDB. As we want to replicate this master database on the slave server, we need to stop mysql to continue writing on the table.

MariaDB [(none)]>

flush tables with read lock;
systemctl restart mariadb

6. Connect back to MariaDB and launched a show master status\G You will need to find a result similar to this screenshot.

Screenshot from 2015-11-03 20:18:21

7. We now have a file name and a position. These two values will be used to set up the slave database. Do a MySQL dump on the master database and import it to the slave server (I copied it from master to slave on the /home folder).

On the Master server:

[[email protected] ~]#


mysqldump -u root -p employees  > employees.sql

[[email protected] ~]#

scp employees.sql [email protected]:/home

8. Now, we log in to the slave server. Create database employees just as in the master server and import the database employees to the slave database server.

[[email protected] home]#

mysql -u root -p employees < employees.sql

9. Edit the /etc/my.cnf in the slave server and enter the following. Note the value of the server-id should be greater than that of the master server, otherwise, it may not work. Then restart MariaDB on the slave server.

server-id = 2
systemctl restart mariadb

10. Right now, the slave server cannot identify the master-slave to synchronize the database for replication.

On the master server:

unlock tables;

On the slave server:

MariaDB [(none)]>


change master to master_host='master', master_user='repliuser', master_password='replipassword', master_log_file='mysql-bin.000001', master_log_pos=313;

11. You can now start the slave replication. Afterward, you can type a show slave status\G;

start slave;
Screenshot from 2015-11-03 21:04:07
show slave status\G indicating replication on the slave server

Tips:

    • To find the list of services for which specific ports can be opened, use the command firewall-cmd –get-services. Hence, you can add the MySQL service with the command firewall-cmd –permanent –add-service-mysql. After adding the MySQL service you will need to reload the firewall service with the command firewall-cmd reload. To verify if certain rules have been loaded in the firewall use the command firewall-cmd –list-all
    • To check which database you have, you can fire the command show schemas;


  • At step 9, If you want to replicate a specific database on the slave server, you can use the following parameter in the /etc/my.cnf in the slave under [mysqld] use replicate-wild-do-table=employees.% where the .% means all tables under the database employees.
  • Since you will be on Virtual Box or Vmware, you may need to edit your /etc/hosts so that each server recognized each other as the Master_User and Slave_User respectively. In this case, in the /etc/hosts for the master enter the IP of the slave server followed by the hostname of the slave and vice versa. Test it with a ping. In my case, if I ping master from the slave, it will answer promptly.

Analyzing vmcore with crash

In the article Linux kernel crash simulation using kdump, I gave a brief idea as to how to generate a vmcore file during a crash or hangs. On this article, I will emphasize the analysis of a vmcore which has been generated and the tool ‘crash’ which can be used for advanced analysis. In a future article, I will elaborate on how to decode the detailed information given with the crash tool. Let’s see how to use the crash utility first.


tux-logo

1.Download the package kernel-debuginfo and kernel-debuginfo-common. You will notice a vmlinux file has been created just after the installation under /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux

Screenshot from 2015-11-02 12:49:34

yum install kernel-debuginfo kernel-debuginfo-common -y

2. Now, we will launch the crash utility which can be used for live debugging. By default, it will give you the info from the available vmcore.


crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686

3. However, you can specify a specific vmcore file with the following command by mentioning the location of the vmcore

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686 /var/crash/127.0.0.1-2015-10-30-00\:12\:34/vmcore

Screenshot from 2015-11-02 13:52:46

4. You will have several pieces of information related to the kernel as well as the most interesting stuff is what has caused the panic that is the warning message. In this case, it is a “SysRq”. If you remember from the last article we had fired an echo c > /proc/sysrq-trigger. Under the state tab, it also gave an indication of the task SYSRQ running.


5. We can also check the process running on the crash utility using the PID given.

Screenshot from 2015-11-02 14:03:396. Another interesting command is the bt which enable us to see execution history of the process

Screenshot from 2015-11-02 14:05:22

7. The sys command will give you an idea of the system. ps | grep “>” – will show you running processes during the time of the crash. mount command will show you partitions mounted etc.. 

Tips:


    • To be able to download the kernel-debuginfo package, you will need to activate the repo located at /etc/yum.repos.d
  • The version of the kernel of the machine should corroborate with that of the kernel-debug-info otherwise it will not work.