mod_performance 0.4 – module of monitoring web server’s requests (compatible with Apache 2.0/2.2/2.4)

Source code here

Functionality

This is usual Apache module for Linux (CentOS, RedHat, Fedora, Debian, OpenSuse) which can: 1) accumulate requests resource usage; 2) gives ability to analize stored data. It takes such parameters of request as:

  • requested virtualhost;
  • requested file;
  • URI;
  • CPU usage %;
  • Memory usage %;
  • I/O usage;
  • execution time;

Collected info can be stored in SQLite, MySQL, PostgreSQL database or in text file. Module’s filter provide an opportunity to save only especial requests (for example only php scripts).

Module can work with such apache configurations:

  1. Apache mod_php (prefork, worker, event);
  2. Apache itk + mod_php;
  3. Apache mod_ruid2 + mod_php
  4. Apache + suphp (patched by mod_performance patch)
  5. Apache + php-fpm (patched by mod_performance patch)

How to build module

  1. disble selinux
  2. yum install httpd-devel apr-devel gd-devel sqlite3
  3. mkdir ~/tmp
  4. cd ~/tmp
  5. tar -xvf mod_performance.tar.gz –strip 1
  6. make
  7. cp ~/tmp/.libs/mod_performance.so /etc/httpd/modules/
  8. cp ~/tmp/mod_performance.conf /etc/httpd/conf.d/
  9. uncomment LoadModule performance_module modules/mod_performance.so
  10. mkdir -p /opt/performance/
  11. chown apache:apache /opt/performance/
  12. chmod 755 /opt/performance/
  13. uncomment in mod_performance.conf strings #PerformanceDB and #PerformanceSocket
  14. service httpd restart
  1. disable selinux
  2. apt-get install make apache2-prefork-dev libgd2-xpm-dev sqlite3
  3. mkdir ~/tmp
  4. cd ~/tmp
  5. tar -xvf mod_performance.tar.gz –strip 1
  6. make
  7. cp ~/tmp/.libs/mod_performance.so /usr/lib/apache2/modules/
  8. cp ~/tmp/mod_performance.conf /etc/apache2/mods-available/
  9. echo «LoadModule performance_module /usr/lib/apache2/modules/mod_performance.so» > /etc/apache2/mods-available/mod_performance.load
  10. a2enmod mod_performance
  11. /etc/init.d/apache2 restart
  12. mkdir -p /opt/performance/
  13. chown www-data:www-data /opt/performance/
  14. chmod 755 /opt/performance/
  15. uncomment in mod_performance.conf strings #PerformanceDB and #PerformanceSocket
  16. service apache2 restart
  1. disable selinux
  2. yast2 -i apache2-devel gd-devel libapr1-devel sqlite3 make
  3. mkdir ~/tmp
  4. cd ~/tmp
  5. tar -xvf mod_performance.tar.gz –strip 1
  6. make
  7. cp ~/tmp/.libs/mod_performance.so /usr/lib/apache2-prefork/
  8. cp ~/tmp/mod_performance.conf /etc/apache2/conf.d/
  9. uncomment LoadModule performance_module /usr/lib/apache2-prefork/mod_performance.so
  10. mkdir -p /opt/performance/
  11. chown wwwrun:wwwrun /opt/performance/
  12. chmod 755 /opt/performance/
  13. uncomment in mod_performance.conf strings #PerformanceDB and #PerformanceSocket
  14. service apache2 restart

Typical configuration for CentOS:

mod_performance.conf
LoadModule performance_module modules/mod_performance.so
 
<IfModule mod_performance.c>
 PerformanceEnabled On
 PerformanceUseCanonical On
 PerformanceScript .php
 PerformanceLogType SQLite
 PerformanceDB /opt/performance/perfdb
 PerformanceSocket /opt/performance/perfsock
 #Minimal script execution time, wich accounts by script as load
 PerformanceMinExecTime 50 SOFT
 <Location /performance-status>
  SetHandler performance-status
  Allow from 1.1.1.1
 </Location>
 
</IfModule>

PerformanceEnabled — this parameter activate the module PerformanceScript – this is filter(regexp) that sets which request should be saved to base. In example above will be stored only request for php files PerformanceUseCanonical – ignore the redirect handler generated by mod_rewrite. If this parameter will be off, you can get not script name, but – redirect: /index.php Next options sets which URL will display web interface of module PerformanceLogType – shows that data will be saved in SQLite database PerformanceDB and PerformanceSocket installs database path and socket path.

Web-interface

Web-interface available only when PerformanceLogType = SQLite, MySQL, Postgres.

it is listed on the figure 1:

1. Hide / Show – button “Show / Hide Filter” – is designed to collapse and expand the window filters.

2. Report – a drop-down list of available report (analysis of historical data)

3. Sort Field – number of the output field which will be sorted (does not apply to user reports)

4. Sort Type – type of sorting – ascending or descending (does not apply to user reports)

5. Period (days) – the interval in days from the current date for which the data will be analyzed

6. Period begin (Period End) – the exact date and time of the beginning and end of the period (if these values ​​are, the field Period (days) is ignored)

7-8-9. – Additional filter options analyzed data.

10. Link to download the latest version

11. Link to documentation of module

12. Sort icon on a page (v – descending, ^-increasing)

13. Column headers

14. Fixed line number column. Does not move while scrolling.

Built-in reports

  • Show output without analytics – shows collected data without analytics (only filtered and sorted);
  • Maximal% CPU – display only entries with a maximum value of% CPU (with the filter);
  • Maximal memory% – display only entries with a maximum value of% memory (with the filter);
  • Maximal execution request time – shows longest request;
  • Host requests statistics – print statistics requests for hosts, sorted in descending order (in% of the total with the filter);
  • Number of requests per domain – display statistics requests to hosts, sorted in descending order;
  • Average usage per host – take an average server load each host (the amount of% CPU, amount% MEMORY, the sum of the scripts, the average% CPU for the period, the average% of memory, the average execution time);

Custom reports

You can add own report. In source code of module available file – custom.tpl. This file expands the number of reports and it is an example of custom reports templates.

List of custom reports:

  1. Show site average usage in different time period – to show the average site’s load in different time periods
  2. Show most popular script by site – show most requested script
  3. Show CPU load range by site – show CPU load for each site
  4. Show Memory Usage range by site – show the distribution of memory for each site
  5. Show Max CPU load in different time periods by site – show the maximum load of the CPU in the different time periods for each site
  6. Show Max Memory Usage in different time periods by site – to show the maximum memory usage in different time periods for each site
  7. Show Max IO Usage in different time periods by site – IO show the maximum load in different time periods for each site

Template for custom report creation:

[Report name]
header=Fld 1|Fld 2|Fld 3;
ssql=request in SQLite syntax;
msql=request in MySQL syntax;
psql=request in PostgerSQL syntax;
sort=1,2;

Alias and table fields accordance:

  • id :ITEMNUMBER: – item identificator
  • dateadd :DATEADD: – date add
  • host :FORHOST: – requested host
  • uri :REQUESTURI: – URI
  • script :CALLEDSCRIPT: – requested script
  • cpu :CPUUSAGE: – CPU in %
  • memory :MEMUSAGEINPRCT: – memory in %
  • exc_time :EXECUTIONTIME: – execution time
  • cpu_sec :CPUUSAGEINSEC:
  • memory_mb :MEMUSAGEINMB: – memory in Mb
  • bytes_read :BYTESREAD:- read bytes
  • bytes_write :BYTESWRITE: – write bytes
  • hostnm :HOSTNAME: – request was on the server with name …

Filtee fileds:

  • :FILTER: – use web-interface filters
  • :PERIOD: – use web-interface period
  • performance – :TBL: – table alias

Select example:

select count(*) as t1, sum(:CPUUSAGE:)/count(*) as t2, sum(:MEMUSAGEINPRCT:)/count(*) as t3, sum(:BYTESREAD:+:BYTESWRITE:)/count(*) as t4,  :CALLEDSCRIPT:, :FORHOST: from :TBL: where :FILTER: and :PERIOD: group by :CALLEDSCRIPT:,:FORHOST:

Where statistics can be saved

Example 1. Save in SQLite. Most easy to install and tunning.

PerformanceDB /statistics/apache/perfdb
PerformanceLogType SQLite

Example 2. Save in MySQL. At first, you should create user and database.

mysql> create database perf;
mysql> CREATE USER ‘perf’@'localhost’ IDENTIFIED BY ‘perf’;
mysql> GRANT ALL PRIVILEGES ON *.* TO ‘perf’@'localhost’ WITH GRANT OPTION;

then, make such module options:

PerformanceLogType MySQL
PerformanceDbUserName perf
PerformanceDBPassword perf
PerformanceDBName perf

Example 3. save in PostgreSQL. t first, you should create user and database.

postgres=# CREATE USER perf WITH PASSWORD 'perf';
postgres=# CREATE DATABASE perf;
postgres=# GRANT ALL PRIVILEGES ON DATABASE perf to perf;

put in file /var/lib/pgsql/data/pg_hba.conf:

/var/lib/pgsql/data/pg_hba.conf
local all all trust
host all all 0.0.0.0/0 trust
host all all : : : 1/128 trust

then, make such module options:

PerformanceLogType Postgres
PerformanceDbUserName perf
PerformanceDBPassword perf
PerformanceDBName perf

Example 4. Save data in text log.

PerformanceLogType Log
PerformanceLog /statistics/apache/perf.log

By default data in log saved in format: %DATE%] from %HOST% (%URI%) script %SCRIPT%: cpu %CPU% (%CPUS%), memory %MEM% (%MEMMB%), execution time %EXCTIME%, IO: R — %BYTES_R% W — %BYTES_W%

for example:

[2011-06-05 19:28:28] from example.com (/index.php) script /var/www/example.com/index.php: cpu 0.093897 (0.010000), memory 0.558202 (5.597656), execution time 10.298639, IO: R — 104.000000 W — 248.000000
[2011-06-05 19:28:39] from example.com (/index2.php) script /var/www/example.com/index2.php: cpu 0.000000 (0.000000), memory 0.558202 (5.597656), execution time 10.159158, IO: R — 0.000000 W — 0.000000

Available format:

  • % DATE% – converted to start date of the request;
  • % CPU% – CPU usage in percent;
  • % MEM% – memory usage in percent;
  • % URI% – URI query
  • % HOST% – name of the virtual host to which the request was addressed;
  • % SCRIPT% – the name of the script;
  • % EXCTIME% – the duration of the script, in seconds;
  • % CPUS% – how many seconds the system is spent on this process in seconds;
  • % MEMMB% – use of memory in megabytes;
  • % BYTES_W% – kilobytes written;
  • % BYTES_R% – kilobytes read;
  • %% – To bring the percent sign.

Store the data in centralized storage

Configuring the module for use with a centralized repository:

1) in a centralized repository (MySQL, PostgreSQL) user database and table

MySQL: CREATE TABLE IF NOT EXISTS performance (id INT NOT NULL AUTO_INCREMENT, dateadd DATETIME, host VARCHAR (255), uri VARCHAR (512), script VARCHAR (512), cpu FLOAT (15,5), memory FLOAT (15,5 ), exc_time FLOAT (15,5), cpu_sec FLOAT (15,5), memory_mb FLOAT (15,5), bytes_read FLOAT (15,5), bytes_write FLOAT (15,5), hostnm CHAR (32), PRIMARY KEY (id))
PostgreSQL: create table performance (id SERIAL, dateadd timestamp, host varchar (255), uri varchar (512), script varchar (512), cpu float (4), memory float (4), exc_time float (4), cpu_sec float (4), memory_mb float (4), bytes_read float (4), bytes_write float (4), hostnm char (32), PRIMARY KEY (id))

2) on the server from which the data will be collected should to write in mod_performance.conf PerformanceLogType MySQL or Postgres PerformanceHostId HostName (enter a unique name for each host, such as its IP or Other) PerformanceDbUserName username PerformanceDBPassword userpassword PerformanceDBName dbname PerformanceDBHost hostname – IP address or host name of a centralized data repository