Background
This project was initially started back in 2012. Then, one day, my computer's hard drive crashed. I was lucky enough to be able to recover most of the files. After spending some time, I was able to get it to compile again. After that, I worked on it maybe once or twice a year. There is no doubt lots of improvements can be made to it. Lately, I've made some changes to the project and decided to share it now instead of letting it sit on the shelf for another couple of years. Hopefully, someone will find this project useful and continue to make enhancements to it.
Introduction
Like many other bloggers or website owners out there, most of us host our website on a shared server instead of dedicated server hosting due to the cost. The hosting company shall not be named here. Based on my own experiences, once in a while, some malicious folders and files/malware were being injected into the website. Every time I submitted a ticket about it, I would get a response like "Your computer was compromised", "you're using outdated software", "you need to change your password", etc. All kinds of this nonsense but the hosting company never took responsibility or did their due diligence. Some hosting companies even offered to clean up the vulnerability and monitor EACH site on the hosting for a yearly subscription. Imagine if you have more than one website. The cost will quickly add up due to someone else's negligence.
Since I refused to pay some company every year to monitor all my websites, why not build my own? The title says it all, "Poor Man's Web Monitoring Tools". If you're poor, then you need to work hard and put together all the available free tools yourself. It might look like a Frankenstein tool now, but I’m positive it will look better once we pour some more thought into it.
Shown below is a brief description of some of the components of the solution. In this article, I'll not go into every detail on how the solution was being implemented as I believe some more work is needed to optimize it. But I will share what each component will do and how to set it up.
- DownloadIISLog
- Download IIS Log files through FTP (WinSCP)
- Insert the log file data into the SQL Database using Log Parser
- SimpleMonitor.Web
- Web UI to display the IIS Log data using jqGrid
- Option to mark certain IP addresses "Banned"
- SimpleMonitor.BlockIpAddress
- Example of HTTP Module to block a banned IP address from accessing the Website
- SimpleMonitor.ScanFile
- Download latest files
- Compare Baseline vs latest files using WinMerge
- Send email notification of the results
DownloadIISLog
One of the responsibilities of this console application is to download the log files from the FTP Server using WinSCP. Please note that I only tested this on two different shared hosting environments. Here, what I would suggest is that if you get a permission denied error while accessing the log folder, first, log into your hosting account, navigate to the file manager, look for the logs folder, and assign read-only permission to the folder. The second option is to submit a ticket.
Shown in Listing 1 is the method to download the log files. The initial run will take a while, depending on the number of log files residing in the hosting log folder. The subsequent execution will be a lot faster because this method will download only files that were not previously downloaded. To achieve this, the method will first get the list of log file metadata from the server using the WinSCP ListDirectory method and store it in the directoryInfo object. Next, it will get the most recent log file name stored in the database. After that, the module will try to query the directoryInfo object for the LastWriteTime by the log file name. Finally, the module will download all the log files where the LastWriteTime is after the LastWriteTime that was determined previously.
static internal void WinScpGetLog() {
using(Session session = new Session()) {
// Connect
session.Open(GetWinScpSession());
string remotePath = ApplicationSetting.FtpRemoteFolder;
string localPath = ApplicationSetting.DownloadPath;
// Get list of files in the directory
RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);
// latest file name in the table by application name
var latestLogFileInDb = Path.GetFileName(unitOfWork.IISLogRepository.LatestFileName(ApplicationSetting.ApplicationName));
// get the date of the latest file from FTP
var logFileDate = directoryInfo.Files
.Where(w => w.Name.ToLower() == latestLogFileInDb.ToLower())
.Select(s => s.LastWriteTime).FirstOrDefault();
// Select the files not in database table
IEnumerable notInLogTable =
directoryInfo.Files
.Where(file => !file.IsDirectory && file.LastWriteTime > logFileDate).ToList();
//// Download the selected file
foreach(RemoteFileInfo fileInfo in notInLogTable) {
string localFilePath =
RemotePath.TranslateRemotePathToLocal(
fileInfo.FullName, remotePath, localPath);
string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);
// download
TransferOperationResult transferResult =
session.GetFiles(remoteFilePath, localFilePath);
}
}
}
Listing 2 shows the logic employed by the Microsoft Log Parser tool to query the log files and insert them into the database. The query used by the Log Parser in the CallLogParser()method is very straightforward. It will connect to the SQL database using the provided username and password. Then read all the log files in the folder and insert them into a table. The ApplicationNameparameter was added recently as an option to allow storing log files data for more than one website. If you have ten websites, you can duplicate this console application ten times and configure different ApplicationNamevalues in the configuration file.
static void CallLogParser()
{
ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.LogParserPath);
startInfo.WindowStyle = ProcessWindowStyle.Minimized;
Process.Start(startInfo);
startInfo.Arguments = string.Format("\"SELECT '{7}', *,1 INTO {0} FROM {1}{2}*.log\" -o:SQL -createTable:OFF -server:{3} -database:{4} -username:{5} -password:{6}",
ApplicationSetting.LogParserSqlTable, ApplicationSetting.DownloadPath, ApplicationSetting.LogFilePrefix, ApplicationSetting.LogParserSqlserver,
ApplicationSetting.LogParserSqlDatabase, ApplicationSetting.LogParserSqlUserName, ApplicationSetting.LogParserSqlPassword,
ApplicationSetting.ApplicationName);
Process.Start(startInfo);
}
Figure 1 shows the data imported into the database table through the CallLogParser()method.
![CallLogParser]()
DownloadIISLog - Configuration
Table 1 shows the descriptions of the settings in the app. config.
| Key |
Description |
| FtpHost |
The address of the server. Example: poormantool.arr or 123.123.123.123 |
| FtpRemoteFolder |
The path to the log files on the server. Example: /virtual folder/logs/W3SVC999/ |
| FTP username |
FTP account |
| password |
FTP password |
| FilePrefix |
The log file prefix, if any. |
| DownloadPath |
Location to store the downloaded logs file. Example: c:\temp\app1\download\ |
| LogPath |
LogPath The path to the log file for logging purposes like when the console runs, stops, errors, etc. Example: c:\temp\app1\log\log.txt |
| LogParserPath |
The path to the log parser. Example: C:\Program Files (x86)\Log Parser 2.2\LogParser.exe |
| LogParserSqlTable |
The table to use in the SQL database to store the log files data. Right now, it can only be "Syslog" unless you update the entity and code |
| ApplicationName |
The name of the application. Example: app1 |
| LogParserSqlserver |
The SQL server address |
| LogParserSqlDatabase |
The SQL server database |
| LogParserSqlUserName |
The SQL server username |
| LogParserSqlPassword |
The SQL server password |
| connectionStrings |
Replace the XXX with your SQL server information |
SimpleMonitor.Web
The purpose of this web application is to display the log file data. Now that we have the data in the database, is up to you, what technology you want to use to interface with the data. This web application was developed by using MVC 5, EntityFramework v6.0, jqGrid, Bootstrap v4.0, and jQuery v3.3.
Figure 2 shows how the Interface looks from the web browser. All the columns are sortable and filterable. The blocked column indicates if the IP address is blocked from accessing the site. The goal of the blocked Hit column is to show the number of times a blocked IP address attempted to access the website.
![Web browser]()
To block an IP address, click on the green icon. A confirmation dialog will appear. Click Continue. Refer to Figure 3. The application will insert the selected IP address into the box.BlockedIp table, then mark all the blocked IPs in the grid with a red ban circle icon.
![Update]()
To unblock an IP address, click on the red circle icon and continue button. Refer to Figure 4.
![IP Address]()
SimpleMonitor.Web - Configuration
Make sure to update the connection string on the web. config to mirror your SQL server environment.
SimpleMonitor.BlockIpAddress
The purpose of this module is to demonstrate how to utilize the data in dbo.BlockedIp table. This module will check if the requester IP address exists in the table, if yes, redirect the request to an error page and then increase the blocked hit count. Listing 3 shows how to register the HTTP modules in the web application web. config file. To test it, I'll add my IP address to the table and re-run the web application. Again, this is just an example, is up to you how you want to implement the detection and prevention control.
<system.web>
<httpModules>
<remove name="BlockIpHttpModules" />
<add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" />
</httpModules>
</system.web>
<system.webServer>
<modules>
<add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" preCondition="managedHandler" />
</modules>
</system.webServer>
SimpleMonitor.ScanFile
The main purpose of this console application is to compare the folders and files between the baseline and the production. Initially, the application will download and save the files to the destination folder using WinSCP. Then it will run the WinMerge command to compare the files with the baseline. Listing 4 shows the WinMerge command line to compare the files.
static void CompareFiles() {
var tempFileName = $"{Guid.NewGuid()}.html";
ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.WinMergePath);
startInfo.WindowStyle = ProcessWindowStyle.Minimized;
startInfo.Arguments = $" {ApplicationSetting.BaselineFilesPath} {ApplicationSetting.LatestFilesPath} -minimize " +
"-noninteractive -noprefs -cfg Settings/DirViewExpandSubdirs=1 -cfg Settings/DiffContextV2=2 " +
"-cfg ReportFiles/ReportType=2 -cfg ReportFiles/IncludeFileCmpReport=1 -cfg Settings/ShowIdentical=0 " +
$" -r -u -or {ApplicationSetting.FileCompareOutputPath}{tempFileName}";
var process = Process.Start(startInfo);
process.WaitForExit();
Email.SendEmail($"{ ApplicationSetting.FileCompareOutputPath}{ tempFileName}");
}
Table 2 shows the WinMerge parameters in listing 4 and their descriptions.
| Key |
Description |
| -minimize |
starts WinMerge as a minimized window. |
| -noninteractive |
Exit WinMerge after compare / report generation |
| -prefs |
Do not read/write setting information from the registry (use default value) |
| -cfg Settings/DirViewExpandSubdirs=1 |
Expand folder tree after comparison 0: do not expand the folder tree |
| -cfg Settings/DiffContextV2=2 |
0: shows only the different
1: shows the different and one line from above and below
2: shows the different and two different lines from above and below |
| -cfg ReportFiles/ReportType=2 |
Generate HTML-format report |
| -cfg ReportFiles/IncludeFileCmpReport=1 |
Include file comparison report in folder comparison report, 0: do not include |
| -cfg Settings/ShowIdentical=0 |
Do not show Identical files in the result |
| -r |
compares all files in all subfolders |
| -u |
prevents WinMerge from adding either path (left or right) to the Most Recently Used (MRU) list |
| -or |
Path to the output file |
At the end of the comparison, the CompareFiles()method will send a summary to the specified email address using Gmail. Figure 5 shows how the comparison summary looks.
![Compare]()
Figure 6 shows how the comparison details look when clicking on the Filename to drill down from the computer where the comparison reports are saved.
![HTML]()
SimpleMonitor.ScanFile – Configuration
Table 3 shows the descriptions of the settings in the app. config.
| Key |
Description |
| FtpHost |
The address of the server. Example: poormantool.arr or 123.123.123.123 |
| FtpRemoteFolder |
The path to the log files on the server. Example: /virtual folder/wwwroot/ |
| FTP username |
FTP account |
| FtpPassword |
FTP password |
| BaselineFilesPath |
The path to the application production files |
| LatestFilesPath |
The path to the latest files downloaded by using the WinSCP from the FTP Server |
| FileCompareOutputPath |
The path where the comparison results will be stored |
| WinMergePath |
The path to the WinMerge application |
| SmtpHost |
The email SMTP host |
| SmtpTo |
The recipient |
| SmtpPort |
The SMTP Port number |
| SmtpUserName |
The email account username/email |
| SmtpPassword |
The email account password |
| SmtpSubject |
The email subject |
Require software
Download and install the following software.
How to set up?
Execute the database_objects.sql script on an existing/new database instance. The script will create the following objects:
- [BlockedIp] table
- [iislog] table
- [vwIISLog] View
- [InsertUpdateBlockedIp] Stored Procedure
Conclusion
First of all, I would like to thank the new WinMerge owner for continuing to maintain the WinMerge software. Please keep in mind that the solution provided here is not a preventive control. Whatever you see in the logs and comparison results are after the fact. But you can create your own preventive control such as an IP blocker after reviewing and analyzing the information.
Here are other topics of interest that you can extract from this project.
- How to Use jqGrid with server-side paging, filtering, sorting
- How to use Log Parser to import IISLog into SQL database table
- How to use WinSCP to download files from the FTP server
- How to use WinMerge to compare files
- How to create and utilize HTTP Modules
- How to send email using Gmail
I hope someone will find this project useful. If you find any bugs or disagree with the contents or want to help improve this article, please drop me a line and I'll work with you to correct it. I would suggest visiting the demo site and exploring it in order to grasp the full concept because I might miss some important information in this article. Please contact me if you want to help improve this article.
History
- 08/12/2018 - Initial version
Download
Resources
Feedback/issue