POM Definition

The procedure for both Eclipse artifacts is similar. In both, you have to insert a pom.xml and fill it with the Maven coordinate (group id, artifact id and version) and the packaging type. But there exists some rules about the content.

The packaging type is depended by what Eclipse artifact should be built:

The version specification in the POM is depended by the version specification in the manifest (in case of Eclipse plugin) and by the version specification in the feature.xml (in case of Eclipse feature) respectively:

The specification of the artifact id in the POM must be equal like  the bundle symbolic name in the manifest (in case of Eclipse plugin) and by the specification of the id in the feature.xml (in case of Eclipse feature) respectively. That’s all you need to define the pom.xml. Then call only mvn clean install and the Eclipse artifact is built by Maven Tycho.

Troubleshooting

I met two problems during the introduction of Maven Tycho for the building of Eclipse plugins. The first problem was that Maven Tycho sometimes throws a NullPointerException during the read-out of the manifest file. The reason is that a blank must be between the colon, that follows after the manifest header, and the command.

# Don't
Import-Package:com.library.*

# Do
Import-Package: com.library.*

The second problem was that Maven Tycho throws compiler error althought everything is alright in Eclipse. The analyse of the error message shows that the command for the Import-Package in the manifest is not completed.  A second analyse shows that package names of used Eclipse RCP artifacts are missing. After adding these missing package names in the manifest, Maven Tycho builds without error. But this also means that Eclipse does not generate the manifest osgi-compliantly.

In the first part I’ve described the basic usage and the installation with Puppet of Chris Jones’ TangledStrings plugin for vim. You may want to refer to this post before; it’s found here: Recognize Puppet Managed Files – Part 1.

Now this article is about a different way of installing the plugin with Puppet. The reason for this alternative is to make it possible also for non-root users to use the plugin. As per default this is not possible, since the TangledStrings plugin needs read access to the yaml file(s) below /var/lib/puppet/client_yaml/catalog. Here Puppet stores a yaml file which contains all managed resources. But this file will be only readable by user Puppet and root.

Additionally there are some hard-wired strings inside the plugin once it is installed. They contain user specific information. So we need to find a way to make those strings dynamic and fill them with the correct user information. I will describe both, accessing the yaml file as well as the dynamic generation of the user data here.

Accessing the Yaml data
This part is quite easy to accomplish. Some analysing turned out that these files/directories were preventing a normal user from accessing the file:

• /var/lib/puppet/client_yaml
• /var/lib/puppet/client_yaml/catalog
• /var/lib/puppet/client_yaml/catalog/your.host.here.yaml

I checked this by using the same command the plugin is using:

grep -h title /var/lib/puppet/client_yaml/catalog/*.yaml | awk '{ print $2 }' | grep '^/' 2>/dev/null

Finally I turned the manual chmod commands into an appropriate Puppet configuration:

  file {'/var/lib/puppet/client_yaml':
    ensure => directory,
    mode => 0755,
  }
  
  file {'/var/lib/puppet/client_yaml/catalog':
    ensure => directory,
    mode => 0755,
  }
  
  file {"/var/lib/puppet/client_yaml/catalog/${fqdn}.yaml":
    ensure => file,
    mode => 0644,
  }

Using an ERB template for configuring the plugin
In the first part of this post I simply copied the extracted plugin as a file resource to my Puppet agents. The structure of the installed plugin looks like this:

[root@somehost ~]# tree .vim
.vim
├── doc
│   ├── tags
│   └── tangledstrings.txt
├── plugin
│   ├── tangledstrings.vba
│   └── tangledstrings.vim
└── .VimballRecord

2 directories, 5 files
[root@somehost ~]#

If you look at the file .VimballRecord you will the see the user specific strings I mentioned:

tangledstrings.vba: call delete('/root/.vim/plugin/tangledstrings.vim')|call delete('/root/.vim/doc/tangledstrings.txt')

Obviously the user’s home directory is hard-coded there. In Puppet making a string dynamic is a straightforward issue. So I simply turned file .VimballRecord into an ERB template VimballRecord.erb. The content of this template looks like this:

tangledstrings.vba: call delete('<%= user_home %>/.vim/plugin/tangledstrings.vim')|call delete('<%= user_home %>/.vim/doc/tangledstrings.txt')

The user_home variable will be replaced by Puppet with the correct user’s home directory. Of course we need the appropriate configuration inside Puppet’s manifest file. I’m showing only the relevant parts here:

  $youruser_home="/home/youruser"
  
  configure_vim { "youruser": 
    group => "yourgroup",
    user_home => "${youruser_home}",
  }
  
  configure_vim { "root": 
    group => "root",
    user_home => "/root",
  }
  
  define configure_vim($user_home, $group) {
  
    file {"$user_home/.vim":
      ensure => directory,
      source => 'puppet:///modules/unix-common/vim',
      recurse => true,
      purge => false,
      group => "${group}",
      owner => "${name}",
      mode => 0755,
    }
  
    file {"$user_home/.vim/.VimballRecord":
      ensure => file,
      content => template('unix-common/VimballRecord.erb'),
      recurse => true,
      purge => false,
      group => "${group}",
      owner => "${name}",
      mode => 0755,
      force => true,
    }
  }

This way one will always have an individually configured user for the plugin. Beware that you need to install also the rest of the plugin’s files. I’ve described this in the first part of this post.

Verifying
If everything went well and you open a Puppet managed file, you will see a warning message like this one:

Warning message displayed when a Puppet managed file is opened.

Why Maven Tycho?

One requirement is that we can use Maven artifacts. So it seems obvious to search after a solution based on Maven.  After a search in the web, two Maven plugins are noticed:

• PDE Maven Plugin ([7]): This plugin builds the RCP artifacts with the help of a generated Ant script by Eclipse. But this generated Ant script is error-prone.
• Maven Tycho Plugin ([1]): Very young plugin to build RCP artifacts (current version 0.13.0), but the community is very active. It tries to build RCP artifacts  in the similiar way than Eclipse.

The Proceeding

During working the tutorials out the impression occurs that the main problem is to solve the dependency problem. So this is the first task for the introduction of Maven Tycho.

Solution of the dependency problem

In Maven the dependencies are declared in the POM file. In Eclipse the dependencies are declared by a target definition file. The goal is to merge these both worlds.  Maven Tycho enables Maven to build against a pre-defined target definition file.

The next question is where the dependecies come from.  Maven’s solution is the Maven Repository.  Eclipse has its own solution, P2 repository.  In Eclipse’s target definition file you can specify an URL to a P2 repository and Eclipse searchs after the needed dependencies, automatically.

A further requirement is that the Maven repository can still use. The reason is that our server artifacts are stored in a Maven repository and the communication between client and server is via Hessian, so some artifacts are needed by client and server. Furthermore, many third party libraries are only available in a Maven repository.

All examples in the tutorial work only with P2 repositories. But the introducing of a P2 repository is not an option, because then two repositories have to be administered.

First Approach – Define a Target Definition File with the Type  ‘Directory’

The idea of this approach is that you can define a folder with jars  (that are the dependencies)  in the target definition file. So it seems that the problem can be reduced to how I get the dependencies to this folder.

This problem can be solved by the Maven Dependency Plugin ([9]). It has the goal copy-dependencies. This goal copies all dependecies that are declared in the <dependencies>-element of the pom file to a declared output directory. The configuration of the dependency plugin could look like the following one:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-dependency-plugin</artifactId>
 <version>2.3</version>
 <executions>
  <execution>
   <id>copy-dependencies</id>
   <phase>process-resources</phase>
   <goals>
    <goal>copy-dependencies</goal>
   </goals>
   <configuration>
    <excludeTransitive>true</excludeTransitive>
    <outputDirectory>${project.build.directory}/plugins</outputDirectory>
   </configuration>
  </execution>
 </executions>
</plugin>

This solution works well in Eclipse, but not in with Maven’s mvn clean install, because Maven Tycho doesn’t support target type ‘Directory’, only the target type ‘Software Site’. So this solution doesn’t work for us.

Second Approach – A solution based on Maven’s ‘pom-first’ approach

With Maven Tycho, you can define that the RCP artifacts should be built against the dependencies that are defined in the POM (old-known Maven way).

<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>target-platform-configuration</artifactId>
<version>0.13.0</version>
<configuration>
<pomDependencies>consider</pomDependencies>
</configuration>
</plugin>

With Maven Tycho, the build works fine. But in Eclipse you cannot work in Eclipse-known way, because it has not the needed dependencies. I thought I could solve this problem with the Maven Eclipse Plugin and its goal eclipse:eclipse. This plugin generates all needed Eclipse files but the dependencies are generated in the Java Build Path named Referenced Library. But Eclipse needs the dependencies for the RCP plugin development in the Java Build Path named Plug-in Dependencies. Next try!

Third Approach – Nexus P2 Repository Plugin

A further research results that Nexus OSS should get a new plugin that can simulate the access to a Maven repository whether it would be a P2 repository (P2 proxy) (see [4]). This plugin is an experimental feature at the moment, so you can only use the SNAPSHOT version of this plugin. Unfortunately, the SNAPSHOT version (mid december 2011) that I used did not work with Nexus OSS version 1.9.2.2. But I think this would be a good idea for this problem.

Fourth Approach – Generation of a Local P2 Repository

A hint in the Tycho user mailing list (see [5]) helps me to find this approach. It exits a Maven Tycho plugin (see [6]), that can generate a P2 repository from a directory with a bunch of bundles.  In the first approach, we used the Maven Dependency Plugin to copy all needed dependencies in a directory. For this approach I combine these two plugins with two another plugins to build a local P2 repository.

By the Maven Dependency Plugin all dependencies are copied to a directory named plugins. The Tycho P2 Extra Plugin can generate from this directory a local P2 repository. Unfortunately, this plugin does not generate the needed feature.jar. But without this jar Eclipse does not recognize that we have here a P2 repository (an introduction to the P2 repository you can find in [10]).  So we have to build this jar.

We create a new Maven project with the packaging option jar. In this project, there is only one file: feature.xml. In this xml file the name of the bundles of the plugins  directory are stored. The next code snippet shows an example for this entry:

<plugin
id="org.eclipse.core.contenttype"
download-size="0"
install-size="0"
version="3.4.100.v20110423-0524"
unpack="false"/>

Then this new generated feature.jar are copied by the Maven Dependency Plugin to the directory features. Now with these two directories (features and plugins), the Tycho P2 Extra Plugin can generate a  local P2 repository that are recognized by Eclipse.

The next problem is that the target definition file accepts only absolute path for the targret type ‘Software Site’. But we want a target definition file that is independent from the platform. So we save a target definition file with a token for the location of the local P2 repository in src/main/resources. During the Maven build this token are replaced by absolute platform-dependent path to the location of the local P2 repository. This replacement is done by the Maven Replacer Plugin ([11]). The next code snippet shows the whole plugin configuration for the above described solution:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>process-resources</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<excludeTransitive>true</excludeTransitive>
<outputDirectory>${project.build.directory}/source/plugins</outputDirectory>
</configuration>
</execution>
<execution>
<id>copy-feature</id>
<phase>process-resources</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>skm.p2</groupId>
<artifactId>repository.feature</artifactId>
<version>0.0.1-SNAPSHOT</version>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>${project.build.directory}/source/features</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.5</version>
<executions>
<execution>
<id>copy-target-definition</id>
<phase>process-resources</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>${target.definition}</include>
</includes>
</resource>
</resources>
<outputDirectory>${project.build.directory}</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.google.code.maven-replacer-plugin</groupId>
<artifactId>maven-replacer-plugin</artifactId>
<version>1.4.0</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>replace</goal>
</goals>
</execution>
</executions>
<configuration>
<ignoreMissingFile>false</ignoreMissingFile>
<file>target/${target.definition}</file>
<regex>false</regex>
<replacements>
<replacement>
<token>$repository.location$</token>
<value>${project.build.directory}/repository/</value>
</replacement>
<replacement>
<token>\</token>
<value>/</value>
</replacement>
</replacements>
</configuration>
</plugin>
<plugin>
<groupId>org.eclipse.tycho.extras</groupId>
<artifactId>tycho-p2-extras-plugin</artifactId>
<version>0.13.0</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>publish-features-and-bundles</goal>
</goals>
</execution>
</executions>
<configuration>
<compress>false</compress>
</configuration>
</plugin>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-p2-repository-plugin</artifactId>
<version>0.13.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>archive-repository</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>attach-artifacts</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>target/${target.definition}</file>
<type>target</type>
<classifier>indigo</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>

In Eclipse, this generated target definition file works fine, but in Maven Tycho it does not work:  Before Maven Tycho builds the local P2 repository and the target definition file, it tries to build the RCP plugins against the not yet existing target definition file and the build failed.

Fivth Approach and Final Solution – Combination of Dependency in the POM and Target Definition

We know Maven works with in the POM defined dependencies, perfectly and Eclipse works with the target platform, perfectly. So the idea is to combine this two approaches, so that every tool can work in its perfect way. So I combine the first both approachs that are described in the above chapters.

All dependencies that are used by all RCP plugins are defined in the root project POM, so that every sub module that defines one RCP plugin is built against these dependencies and one sub module is for the building of the target definition file (this build is described in the first approach). Maven Tycho builds the RCP artifacts in the known Maven way and in Eclipse the RCP artifacts are developed with the generated target definition file.

Links

[1] Maven Tycho Home Page
[2] Tutorial by Matthias Holmqvist – Good for the first introduction
[3] Tutorial with example (in German)
[4] Nexus P2 Repository Plugin (Experimental Feature)
[5] Post on Tycho User Mailing List – How to build local P2 Repository
[6] Introduction to the Tycho-P2-Extra-Plugin
[7] PDE Maven Plugin
[8] Hessian Protocol
[9] Maven Dependency Plugin
[10] Introduction to the format of  P2 repositoy
[11] Maven Replacer Plugin

Once a host which has been integrated into Puppet along with stored configurations is removed again from Puppet, it has to be removed from the MySQL database which holds the data of the stored configurations as well. There’s no automatic deletion.

This is not a big issue, but it might be interesting for those not familiar with MySQL or for those just want to have a quick solution.

Login to MySQL

[root@somehost ~]# mysql -u puppet -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g. 
Your MySQL connection id is 38160
Server version: 5.0.77 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Switch the database

mysql> use puppet;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> 

Get the host’s ID.

mysql> select id,name from hosts where name like 'myhost%';
+----+--------------------------+
| id | name                     |   
+----+--------------------------+
| 40 | myhost.somedomain.net | 
+----+--------------------------+
1 row in set (0.00 sec)

mysql>

Delete the host’s resources using its ID.

mysql> delete from resources where host_id=40;
Query OK, 331 rows affected (0.14 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> 

Delete the host.

mysql> delete from hosts where id=40;
Query OK, 1 row affected (0.02 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> 

Leave MySQL

mysql> quit
Bye
[root@somehost ~]#

That’s it.

On Puppet managed systems it might be interesting and sometimes even important to know, if a file is managed by Puppet, because manual changes will be overridden next time Puppet executes. There are some exceptions from this behavior, e.g. files below /var/spool/cron still allow manual changes. In any case, it would be nice to know if the file is managed by Puppet. Of course it’s good practice to have a header in every file from Puppet, but this may not be possible in every case.

I’m only using vim on my machines and discovered a very nice plugin. It’s called Tangledstrings by Chris Jones and displays a warning message once a file is opened or saved. However, it does of course not prevent from saving the file.

This plugin is both simple and awesome! So I’d like to share some of my experiences and the way I rolled out the plugin on my Puppet boxes.

Once the plugin is downloaded – it’s just a simple file – open it with vim:

[root@somehost ~]# vim tangledstrings.vba

:so %

Then you may quit vim or continue working as usual. However, looking at the directory structure below ~/.vim brings up these files:

[root@somehost ~]# tree .vim
.vim
├── doc
├── tags
│ └── tangledstrings.txt
├── plugin
│ ├── tangledstrings.vba
│ └── tangledstrings.vim
└── .VimballRecord

For rolling out with Puppet, I simply copied this directory structure to the files directory of my unix-common module. A very common file resource then takes care of having everything in place on my machines:

file {"/root/.vim":
    ensure => directory,
    source => 'puppet:///modules/unix-common/vim',
    recurse => true,
    purge => false,
    group => "root",
    owner => "root",
    mode => 0755,
  }

Verifying
If everything went well and you open a Puppet managed file, you will see a warning message like this one:

Warning message displayed when a Puppet managed file is opened.

Credits

• Thanks to Chris Jones who provided this plugin! He gives a short introduction on the plugin on his blog here.
• Download the plugin from its launchpad site here.

By the way: VIM turned 20 this year (2011)! Congratulations!

Ever been concerned about the health of your Puppet agents? You know they are executing, but you don’t know if execution is successful? I’d like to show how to achieve this by monitoring the syslog of the Puppet machines. In case any errors occur during Puppet’s execution, the appropriate error messages will be logged to – most probably – /var/log/messages. So the only thing left to do is to scan this file for fishy patterns.

The best tool known to me is the NRPE plugin check_logfiles. It’s a powerful Perl plugin with many configuration options. I will not give an introduction on this plugin, but you will find the plugin’s website useful. Instead I will simply show two examples of how I am monitoring my Puppet agents and my Puppet master machines.

For my agents, I have this command configuration inside my nrpe.cfg:

command[check_puppet_errors_agent]=/usr/lib64/nagios/plugins/check_logfiles -config /usr/lib64/nagios/plugins/check_puppet_errors_agent.conf

About the same for the masters:

command[check_puppet_errors_master]=/usr/lib64/nagios/plugins/check_logfiles -config /usr/lib64/nagios/plugins/check_puppet_errors_master.conf

As you can see, all the configuration stuff is done within the configuration file. Let’s start with the agent’s configuration check_puppet_errors_agent.conf:

$scriptpath = '/usr/bin';
$prescript = 'sudo';
$prescriptparams = 'setfacl -m u:$CL_USERNAME$:r-- /var/log/messages*';
@searches = ({
  tag => 'puppet-failing',
  rotation => 'loglog0log1',
  logfile => '/var/log/messages',
  criticalpatterns => [
     '.*Too\ many\ open\ files.*',
       '.*Could\ not\ retrieve\ catalog.*',
       '.*Retrieved\ certificate\ does\ not\ match\ private\ key.*',],
  okpatterns => ['.*puppetd.*Finished\ catalog\ run\ in.*'],
  options => 'noperfdata',
});

• Line 1: Define the path to scripts being executed.
• Line 2: The script to be executed before the actual check takes place.
• Line 3: The parameters handed over to the prescript. Basically what is happening here is to grant access (setfacl) to all files matching /var/log/messages to the user this plugin is executed with. In my case this is nrpe. Beware that $CL_USERNAME$ is replaced with the actual user name. You may want to check your nrpe.cfg.
• Line 6: The log rotation method. This plugin is also capable of scanning rotated logs.
• Line 7: The actual log file to scan.
• Lines 8-11: These are the patterns which are used for indicating failed Puppet runs. Basically this is the sum of all error messages I detected manually while setting up all my agents. Of course one may add or remove patterns as desired.
• Line 12: One run may fail, the next run may be OK again. In this case we don’t want to have any critical errors any more. That’s what the OK pattern is for.
• Line 13: Disable performance data. (Admittedly I never had a closer look what performance data would be actually generated here.)

The Puppet master is configured pretty much the same way. However there are a lot more error patterns which may occur on a master, so I will show only the patterns here:

  criticalpatterns => [
     '.*Too\ many\ open\ files.*',
       '.*Could\ not\ retrieve\ catalog.*',
       '.*Syntax\ error\ at.*of\ file.*',
       '.*Mysql\:\:Error.*',
       '.*MySQL\ server\ has\ gone\ away.*',
       '.*Could\ not\ read\ YAML\ data.*',
       '.*Could\ not\ run\ Puppet\ configuration\ client.*',
       '.*Parameter\ group\ failed.*Invalid\ group\ name.*'],

It is very important to allow user nrpe executing the setfacl command along with sudo. That’s why I have this line inside my /etc/sudoers:

nrpe    ALL=NOPASSWD:/usr/bin/setfacl

Of course all the configuration files as well as the check_logfiles plugin are maintained by Puppet itself. Everything is done using stored configurations, so every new Puppet agent attached to the system will establish its own monitoring. By the time of writing this, I had these checks running for several months now and it gives me a very reliable overview of my agent’s health as soon as I have a look at Icinga’s front end.

Related topics

Some other posts I made covering Puppet monitoring might also be interesting:

• Monitoring Puppet – Part 1 shows how to keep the clients alive and how to check proper execution using the master’s /var/log/puppet/masterhttp.log file.
• Monitoring Puppet – Part 2 describes how to monitor the execution intervals of a Puppet agent.

As mentioned in one of my previous posts here, there are some more possibilities of observing the health of a Puppet agent. This one shows how I observe the state of an agent using its file /var/lib/puppet/state/state.yaml.

Every time the Puppet agent executes, this file is being regenerated. By simply checking the age of the file, one may get a hint if the agent has executed within a certain time. The NRPE plugin used for this check is the check_file_age plugin.

The help screen gives all relevant information for configuring the plugin:

[root@somehost plugins]# ./check_file_age --help
check_file_age v1.4.15 (nagios-plugins 1.4.15)
...
Usage:
  check_file_age [-w ] [-c ] [-W ] [-C ] -f
  check_file_age [-h | --help]
  check_file_age [-V | --version]

    File must be no more than this many seconds old (default: warn 240 secs, crit 600)
    File must be at least this many bytes long (default: crit 0 bytes)
...
[root@somehost plugins]#

I wanted to have a warning issued once the client did not run for more than 35 minutes (2100 secs) and a critical error as soon as execution time was more than 65 minutes (3900 secs) ago. To achieve this, I added this line to the NRPE configuration file nrpe.cfg:

command[check_puppet_client_state_file]=/usr/lib64/nagios/plugins/check_file_age -w 2100 -c 3900 -f /var/lib/puppet/state/state.yaml

This check is then applied to every agent with an appropriate stored configuration manifest in Puppet:

   @@nagios_service { "check_puppet_client_state_file_${hostname}":
      use => "generic-service",
      host_name => "$fqdn",
      service_description => "${prefix}Puppet client state file",
      check_command => "check_nrpe!check_puppet_client_state_file",
      }

Beware that this check only indicates if the agent has executed. It does not detect any errors while executing. For detecting errors, you may want to have a look at this post I made.

Related topics

I got several other posts covering the monitoring of Puppet which might be interesting:

1. Monitoring Puppet – Part 1 shows how to keep the clients alive and how to check proper execution using the master’s /var/log/puppet/masterhttp.log file.
2. Monitoring Puppet – Part 3 describes how to scan syslog for Puppet errors.

If you think this tip is useful and maybe if you applied it to your Puppet system, I’d be glad if you share your experiences and give some feedback. Thanks!

Ever wondered why switching the NRPE daemon to debug mode has no effect in any log files at all? I did, a lot of times. You may have a configuration something like this in your nrpe.cfg file:

log_facility=daemon
pid_file=/var/run/nrpe.pid
server_port=5666
nrpe_user=nrpe
nrpe_group=nrpe
allowed_hosts=some.ip.address.here
debug=1

Line 1 indicates it should log everything to the syslog daemon. Depending on your system, this will most likely be the file /var/log/messages or something else depending on your log daemon. Line 7 should switch the NRPE daemon to debug mode and issue some more information than the usual standard output – but it does not.

My clients observed by Icinga were all running on RHEL6. So I decided to have a closer look at the syslog configuration. It turned out that rsyslog was running on my boxes. The configuration file for this log daemon is /etc/rsyslog.conf. Digging into the man pages and some ‘googling’ I found out that something like this was missing inside this file:

*.debug    /var/log/debug

After adding this line and restarting the daemon with

[root@somehost ~]# service rsyslog restart

Even though this might be a little help when trying to find errors when executing remote Icinga/Nagios checks, this may not be sufficient. Maybe you will also have a look at another post I made which covers a very special case here which covers a problem with failing remote checks caused by SELinux.

Ever had an Unable to read output error when executing an Icinga/Nagios remote check using NRPE? I’m pretty sure you had since everyone working with NRPE will run into this error sooner or later. Most solutions are trivial (e.g. wrong file permissions) and I don’t want to repeat the few thousand posts already covering this issue here.

However in a very particular case I had today, I already had eliminated all the usual suspects and still was not able to execute the check from my Icinga host. The check that failed was a Perl file which is automatically rolled out by Puppet along with some other check plugins. On some hosts it worked, on some hosts not and I was not able to figure out what the actual cause was since everything was identical on all hosts. Besides the Unable to read output error message there was no additional information nowhere. Even not when turning on debugging for NRPE.

More or less accidentally I had a look at the SELinux configuration on that box:

[root@somehost plugins]# getenforce 
Enforcing
[root@somehost plugins]# 

[root@somehost plugins]# setenforce 0
[root@somehost plugins]# getenforce 
Permissive
[root@somehost plugins]#

Of course I did not want to have SELinux in permissive mode on that box so I made some investigations. First, I checked the SELinux settings on the failing file:

[root@somehost plugins]# ls -Z check_mem.pl
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       check_mem.pl
[root@somehost plugins]# 

Then I checked the settings of a file which was installed along with NRPE’s RPM:

[root@somehost plugins]# ls -Z check_swap 
-rwxr-xr-x. root root system_u:object_r:nagios_system_plugin_exec_t:s0 check_swap
[root@somehost plugins]# 

Obviously the SELinux type was different. So I changed the settings of the failing plugin and made them identical with the above shown check_swap:

[root@somehost plugins]# chcon system_u:object_r:nagios_system_plugin_exec_t:s0 check_mem.pl

The last step was to have everything configured in Puppet. I never made SELinux stuff in Puppet before. But it turned out to be quite easy. Just use the arguments of the chcon command shown above and add the appropriate attributes to the file resource responsible for the check plugin like this:

  file { "/usr/lib64/nagios/plugins/check_mem.pl":
    owner => "root",
    group => "root",
    mode => "0755" ,
    replace => true,
    source => "puppet:///modules/icinga/check_mem.pl",
    require => Package["nagios-plugins-nrpe"],
    seluser => "system_u",
    selrole => "object_r",
    seltype => "nagios_system_plugin_exec_t",
    selrange => "s0",
  }

Just encountered that the MySQL database which is used for storing the Puppet reports for the Dashboard has grown very large. The regular backup dumps I’ve created had more than 2GB of size. So I thought it’s time to have a look if everything is going well and if I can get rid of some old records. No, I had no cron job so far for deleting old records as recommended by Puppet’s documentation. However this cron job is now in place and I’d like to share some experiences I’ve made while analysing the database and while deleting old records manually.

The Analysis
First I wanted to find out if it is really Dashboard’s database which is responsible for the huge size of the dumps. So I logged in as root to MySQL and had a look:

mysql> SELECT table_schema "Data Base Name", SUM( data_length + index_length) / 1024 / 1024 "Data Base Size in MB" FROM information_schema.TABLES GROUP BY table_schema;
+----------------------+----------------------+
| Data Base Name       | Data Base Size in MB |
+----------------------+----------------------+
| dashboard_production |        2878.34375000 | 
| information_schema   |           0.00390625 | 
| mysql                |           0.52497292 | 
| puppet               |          26.93750000 | 
+----------------------+----------------------+
4 rows in set (1.30 sec)

mysql>

Obviously no doubt that dashboard_production alone creates 2.8GB of data. Digging a little deeper showed the table which holds the most records:

mysql> SELECT TABLE_NAME, table_rows, data_length, index_length,  round(((data_length + index_length) / 1024 / 1024),2) "Size in MB" FROM information_schema.TABLES WHERE table_schema = "dashboard_production";
+------------------------------+------------+-------------+--------------+------------+
| TABLE_NAME                   | table_rows | data_length | index_length | Size in MB |
+------------------------------+------------+-------------+--------------+------------+
| metrics                      |    1010174 |    51986432 |     14172160 |      63.09 | 
| node_class_memberships       |          0 |       16384 |            0 |       0.02 | 
| node_classes                 |          0 |       16384 |            0 |       0.02 | 
| node_group_class_memberships |          0 |       16384 |            0 |       0.02 | 
| node_group_edges             |          0 |       16384 |            0 |       0.02 | 
| node_group_memberships       |          0 |       16384 |            0 |       0.02 | 
| node_groups                  |          0 |       16384 |            0 |       0.02 | 
| nodes                        |         39 |       16384 |            0 |       0.02 | 
| old_reports                  |          0 |       16384 |            0 |       0.02 | 
| parameters                   |          0 |       16384 |            0 |       0.02 | 
| report_logs                  |    1112049 |   330104832 |     15220736 |     329.33 | 
| reports                      |      20866 |     2637824 |      2048000 |       4.47 | 
| resource_events              |     882791 |   107610112 |     13123584 |     115.14 | 
| resource_statuses            |    7925550 |  2369781760 |    109723648 |    2364.64 | 
| schema_migrations            |         45 |       16384 |            0 |       0.02 | 
| timeline_events              |      44091 |     3686400 |            0 |       3.52 | 
+------------------------------+------------+-------------+--------------+------------+
16 rows in set (1.07 sec)

mysql>

Table resource_statuses held almost 8 Million records producing some 2.3GB of data. So I had a look at Puppet’s documentation which says to issue a rake command to drop old records:

rake RAILS_ENV=production reports:prune upto=1 unit=mon

This command will delete all records older than one month. So I gave it a try, but:

[root@phost puppet-dashboard]# rake RAILS_ENV=production reports:prune upto=1 unit=wk
(in /usr/share/puppet-dashboard)
Deleting reports before 2011-11-24 13:37 UTC...
Deleted 66 reports.
Deleting resource_statuses records before 2011-11-24 13:37 UTC...
rake aborted!
Mysql::Error: The total number of locks exceeds the lock table size: DELETE FROM `resource_statuses` WHERE (time < '2011-11-24 13:37:32') 

(See full trace by running task with --trace)
[root@phost puppet-dashboard]# 

Seemed, MySQL had not enough buffer to lock all records. So I increased the buffer pool of the InnoDB by adding the appropriate configuration to my.cnf and restarted MySQL:

innodb_buffer_pool_size=1024M

Those 1024M might be too much, I’m not quite sure. However, it worked. After some estimated 15 minutes the task was finished. But after checking the table sizes again, it turned out that resource_statuses still had the same size of 2.3GB. Some research in Puppet’s user list showed up some posts with this problem. One advice was to add a patch to file /usr/share/puppet-dashboard/lib/tasks/prune_reports.rake. That’s the file which is executed by the above rake command and does the actual deletion of the table records. That’s the patch:

--- prune_reports.rake	2011-09-08 12:42:54.433928840 -0700
+++ prune_reports.rake-new	2011-09-08 12:44:01.928583735 -0700
@@ -52,5 +52,11 @@
     puts "Deleting reports before #{cutoff}..."
     deleted_count = Report.delete_all(['time < ?', cutoff])
     puts "Deleted #{deleted_count} reports."
+
+    # inserted to clean up resource_statuses table simultaneously
+    puts "Deleting resource_statuses records before #{cutoff}..."
+    deleted_count = ResourceStatus.delete_all(['time < ?', cutoff])
+    puts "Deleted #{deleted_count} resource_status records."

Obviously only the table reports is deleted by this script, but this table only makes up some 4.5MB. So the solution of this patch is to add the table resource_statuses to this script.

Once again I triggered the rake command after applying the patch and after execution the table size really had decreased:

mysql> SELECT TABLE_NAME, table_rows, data_length, index_length,  round(((data_length + index_length) / 1024 / 1024),2) "Size in MB" FROM information_schema.TABLES WHERE table_schema = "dashboard_production";
+------------------------------+------------+-------------+--------------+------------+
| TABLE_NAME                   | table_rows | data_length | index_length | Size in MB |
+------------------------------+------------+-------------+--------------+------------+
...
| report_logs                  |     971812 |   331153408 |     15220736 |     330.33 | 
| reports                      |      21109 |     2637824 |      2031616 |       4.45 | 
| resource_events              |     899663 |   107610112 |     13123584 |     115.14 | 
| resource_statuses            |    3331699 |  1037041664 |     45694976 |    1032.58 | 
...

So the solution was found. Now I wanted to have everything managed by Puppet. First I created a bash script which would be executed by a cron job:

#!/bin/bash
DASHBOARD_DIR=/usr/share/puppet-dashboard

cd ${DASHBOARD_DIR}
rake RAILS_ENV=production reports:prune upto=1 unit=wk
rake RAILS_ENV=production db:raw:optimize

This script calls the rake command I executed manually before along with another command that optimizes the database. Though Puppet’s documentation recommends to optimize the database only once a month, I have this script is running once every night. And, as you can see, I now delete all records older than just one week which is perfectly OK for our system.

The Puppet configuration which is responsible for executing the cron job and having the patched file in place is this:

  $purge_file="purgeDashboardDatabase.sh"
  $prune_reports_file="/usr/share/puppet-dashboard/lib/tasks/prune_reports.rake"

  file{"${prune_reports_file}":
    ensure => file,
    mode => 0644,
    source => "puppet:///modules/mysql/${prune_reports_file}",
    replace => true,
    before => "${bin_dir}/${purge_file}",
  }
  
  file{"${bin_dir}/${purge_file}":
    ensure => file,
    mode => 0755,
    source => "puppet:///modules/mysql/${purge_file}",
    replace => true,
    require => File["${bin_dir}"],
  }
  
  cron { purge-dashboard-database:
    command => "${bin_dir}/${purge_file}",
    ensure => present,
    user => "root",
    hour => 2,
    minute => 0, 
    require => File["${bin_dir}/${purge_file}"],
  }

finally{}
Beware that this solution covers Puppet 2.6.6 along with its Ruby libraries. The rake script might be enhanced in newer versions. So you may want to check if this is necessary for you once you run into the same problem.

Credits
Thanks to Mkyong’s blog post who provided the SQL statements for analysing the database and tables. And of course thanks to Puppet’s user list which is always a very good resource if something does not work as expected.