Project Tabulogs: It will give you the ability to display logs on an HTML tabular format page using Ansible Playbook. Some days back, I shed some ideas, how to use Ansible to create an agentless server inventory with an interesting HTML format. In this post, we will see how to present information on an HTML tabular format logging information from lastlog. You can apply the same principle for any logs and present it to an HTML template. One of the advantage with Ansible is that you can also launch shell commands which allows you to automate several tasks whether remote or local. Many times, I came across non-technical personnel who want to see logins and logouts of staffs on specific servers. The goal of this project is to present accurate and reliable information to non-technical persons in an IT organization. It’s also a way to perused logins and logouts of users more easily.
Limitations and Solutions
However, there is some limitation of presenting bulk information such as logs on an HTML page. Imagine having hundreds of servers which could amount more than 20,000 lines. This would make the page impossible to load and may crash on the browser. To remediate this situation, a database such as SQLite could be interesting. Otherwise, AJAX can be used to fetch page by page. Since I’m keeping all information in JSON format in a JavaScript page, I decided to make use of Pagination. A search button will also come handy for all columns and rows.
Now, the thing is how to report that information which keeps on changing? let’s say every month you want to have a report of users connected on your organization’s servers in JSON format that’s too in a JavaScript page. Here is where Ansible comes useful. I created a Playbook to build the JSON array itself and the JavaScript page using the Shell module based on Linux Awk commands. Of course, there are other tasks Ansible will perform such as fetching of the information remotely, updating the HTML page: Example the date the report was created. The directory can then be compressed and send easily to anyone. So, no database needed!!. Cool isn’t it? However, if you want to adopt this system for really huge logs of information, it might work but could be really slow, hence, the need of a database.
You can clone the repo from my GitHub repository.
HTML, AngularJS and JSON
I created a simple HTML page which will call the AngularJS to keep the information in a JSON array. The HTML page will be static. However, for your own use, if you want to add some more columns, feel free to edit the section below. It is located at TabuLogs/master/perimeter1/index.html
<tr class="tableheader">
<th>Hostname</th>
<th>Username</th>
<th>Login</th>
<th>TimeTaken</th>
<th>IPAddress</th>
<th>Perimeter</th>
<th>Application</th>
</tr>
</thead>
<tbody>
<tr ng-repeat="rec in SampleRecords | filter:q | startFrom:currentPage*pageSize | limitTo:pageSize">
<td>{{ rec.Hostname }}</td>
<td>{{ rec.Username }}</td>
<td>{{ rec.Login }}</td>
<td>{{ rec.TimeTaken }}</td>
<td>{{ rec.IPAddress }}</td>
<td>{{ rec.Perimeter }}</td>
<td>{{ rec.Application }}</td>
</tr>
Since I have used the lastlog from Linux, I called the page “Linux Login Management”. Otherwise, you can also filter any logs such as Apache or secure log. Some modifications will have to be carried out with the awk command or using Ansible Jinja filters.
You can also point a logo on the HTML page and kept in the images folder
<body ng-app="AngTable" ng-controller="Table">
<p></p>
<img src="images/logo1.png" align="right" alt=" " width="380" height="180">
<img src="images/logo2.png" alt="cyberstorm" align="top-left" width="380" height="180">
<p><h2>Login Management for my servers</p></h2>
The most interesting thing is about the data which is stored in a JSON array. In that particular example, we have information about the hostname, username, login, the time taken, IP Address, Application, and the Perimeter.
[
{"Hostname":"Apacheserver","Username":"tunnelix","Login":"Fri-Sep-28-15:11","TimeTaken":"15:11(00:00)","IPAddress":"192.168.0.1","Application":"Apache","Perimeter":"Production"},
];
The JSON array will be fetched by AngularJS to render the HTML table.
As mentioned previously, if you had tried to load all the JS page into your browser, same would crash, hence, to overcome this situation, pagination comes handy.
Messing around with Linux AWK command and the Ansible Playbook
The Linux AWK command is pretty powerful to concatenate all logs together. The more interesting is the conversion of the logs into JSON format. The playbook is located at TabuLogs/AnsiblePlaybook/AuditLoginLinux.yml
When the Playbook is launched we can see the first AWK will create a file with the hostname of the machine in /tmp and inside that file, the data is comprised of the Hostname, Username, Login, Time-Taken, and IP Address. To get back the logs from the previous month, I used Date=`date +%b -d ‘last month’`
Assuming you have rotated logs for /var/log/wtmp either weekly or monthly or any range, the loop is supposed to find logs for only last month. Now, in case you have kept wtmp logs for years, another method needs to be used to fetch log for the actual year.
Date=`date +%b -d 'last month'`; t=`hostname -s` ; for i in `ls /var/log/wtmp*` ; do last -f $i ; done | grep $Date | awk -v t="$t" '{print t, $1, $4 "-" $5 "-" $6 "-" $7, $9$10, $3 }' | egrep -v 'wtmp|reboot' > /tmp/$t
Once the logs have been created on the remote hosts, it is fetched by ansible and kept on the local server. Consequently, the remote files are destroyed.
We also need to merge all the data into one single file and also removing all blank lines, which is done using the following command:
awk 'NF' /Tabulogs/AnsibleWorkDesk/Servers/* | sed '/^\s*$/d' > /Tabulogs/AnsibleWorkDesk/AllInOne
Assuming that the file located at /Tabulogs/AnsibleWorkDesk/rhel7_application_perimeter contain information about the server, application, and perimeter, the following awk command will append the information to the table created remotely on the hosts.
awk 'FNR == NR {a[$3] = $1 " " $2; next} { if ($1 in a) { NF++; $NF = a[$1] }; print}' /Tabulogs/AnsibleWorkDesk/rhel7_application_perimeter /Tabulogs/AnsibleWorkDesk/AllInOne > /Tabulogs/AnsibleWorkDesk/AllInOneWithPerimeterAndApplication
Example of the format of the table is:
Nginx dev server2
Apache prod server1
Nginx dev server4
After adding all the data together on a simple table, the following AWK will convert each row into JSON format
awk '{print "{" "\"" "Hostname" "\"" ":" "\"" $1"\"" "," "\"" "Username" "\"" ":" "\"" $2"\"" "," "\"" "Login" "\"" ":" "\"" $3"\"" "," "\"" "TimeTaken" "\"" ":" "\"" $4"\"" ",""\"" "IPAddress" "\"" ":" "\"" $5"\"" "," "\"" "Application" "\"" ":" "\"" $6"\"" "," "\"" "Perimeter" "\"" ":" "\""$7"\"""}" "," }' /Tabulogs/AnsibleWorkDesk/AllInOneWithPerimeterAndApplication > /Tabulogs/AnsibleWorkDesk/perimeter1/table.js
It does not end here, we also need to add the JS codes to it. Using Lineinfile module of Ansible, writing at the beginning of the file is pretty easy.
lineinfile: dest=/Tabulogs/AnsibleWorkDesk/perimeter1/table.js line={{ item.javascript }} insertbefore=BOF
with_items:
- { javascript: "$scope.SampleRecords=[ " }
- { javascript: "$scope.q = ''; " }
- { javascript: "$scope.pageSize = 10; " }
- { javascript: "$scope.currentPage = 0; " }
- { javascript: "app.controller('Table', ['$scope', '$filter', function ($scope, $filter) { "}
- { javascript: "var app = angular.module(\"AngTable\", []); "}
Same method to write at the end of the file to create the table.js file.
lineinfile: dest=/Tabulogs/AnsibleWorkDesk/Perimeter1/table.js line={{ item.javascript }} insertafter=EOF
with_items:
- { javascript: " ]; " }
- { javascript: "$scope.getData = function () { " }
- { javascript: "return $filter('filter')($scope.SampleRecords, $scope.q) " }
- { javascript: " }; "}
- { javascript: "$scope.numberOfPages=function(){ "}
- { javascript: "return Math.ceil($scope.getData().length/$scope.pageSize); "}
- { javascript: "}; "}
- { javascript: "}]); "}
- { javascript: "app.filter('startFrom', function() { "}
- { javascript: " return function(input, start) { "}
- { javascript: "start = +start; "}
- { javascript: "return input.slice(start); "}
- { javascript: " } "}
- { javascript: "}); "}
After writing at the end of the file, the table.js is now completed.
Everything is assembled using the Ansible playbook. The Playbook has been tested for RedHat 7 servers. I believe it can also be adapted for other environments.
Final result
Here is an idea how will be the final result
Future Enhancements
I’m so excited to share this article though I know there are much more improvements that can be done such as:
- The awk command can be combined.
- Removal of backticks on the shell commands.
- shell command like rm -rf will be removed as well using the file module.
- Some deprecated HTML tags were used.
- Code sanitization.
My future goal is to improve the Ansible Playbook and the source code itself. Maybe someone else has a much better way of doing it. I would be glad to hear more. Please do comment below for more ideas. In case, you believed that some improvements can be done on the Playbook, feel free to send some commits on my GitHub repository or comment below.
As usual, a plan is needed. Consider reading the “tips” section below which might give you some hints.
TIPS:
- There is, however, some other limitations with last command. Some arguments will not be present if you are using old utils-linux packages. Consider updating the package to be able to filter the last command easily.
- If you can group your servers by some category or environment, it would be helpful.
- There will be other versions of the project Tabulogs to create better and fast Ansible playbook. Lets called this one version 1.0
