Sidebar

iTop Extensions

Data collector for CheckMK

with Workaround-Release for Compatibility with CheckMK 2.0

name:
Data collector for CheckMK (with Workaround-Release for Compatibility with CheckMK 2.0)
version:
22.3.0/17.4.0
itop-version-min:
2.7
language:
en

Short Description

  • Collects Server- and PC-Inventory with Brands, Models, OS-Families and -Versions aswell as Softwarepackets
  • Uses the iTop SynchroDataSource Feature to synchronize the data with iTop

Versions and Downloads

  • 17.4.0: Use this version, if you are running CheckMK 1.x. Ignore the last chapter of this manual. Download
  • 22.3.0: Use this version, if you are running CheckMK 2.x. Do not ignore the last chapter of this manual. It is mandatory to create compatibility with CheckMK 2.x. Download

Restrictions and Limitations

  • This collector needs (write-)access to the CheckMK Inventory Files. This implies the collector is running on a machine, which as access to the file system where CheckMK stores its data. You can aswell instruct CheckMK or the underlying server to copy/clone the Inventory Files somewhere else and let the collector do its work there.
  • The distinction on whether a file describes a server or a PC depends on your configuration of the collector with regards to filenamepatterns you can specify (more info below).
This version of the collector does not yet support CheckMK 2.0 natively. It is a temporary workaround until a new release of the collector will be available implementing the new interface CheckMK 2.0 has introduced to exchange data. Please do mind the last chapter of this manual for more details.

Prerequisites

  • PHP Version 5.3.0
  • Read access on CheckMK's Inventory Files
  • Access to the iTop REST service

Overview on how to implement the collector at your site

  • Download one of the two files suiting your needs
  • Extract the archive to your filesystem
  • Modify the file conf/params.local.xml with respect to your iTop instance (you can use conf/params.distrib.xml as a template)
  • Create the file collectors/params.distrib.xml (you can use the file collectors/params.template.xml as a template) and modify the configuration according to your needs
The files README and README_checkmk within the collector contain more documentation.

Configuration

Main parammeters

Modify the following parameters in conf/params.local.xml according to your needs:

<itop_url>https://localhost/itop/</itop_url>
<itop_login>admin</itop_login>
<itop_password>admin</itop_password>
Parameter Description Sample value
itop_url URL to the iTop Application https://localhost/itop/
itop_login User name for connecting to iTop. Must have admin rights for executing the data synchro and must have the REST-User profile, if you are using the restricted REST API feature of iTop (which you absolutely should). admin
itop_password Password for the iTop account.

The following parameters must be set in conf/params.local.xml aswell.

<default_org_id>Demo</default_org_id>
<check_mk_dir>/var/lib/check_mk/inventory</check_mk_dir>
<type_mapping type="array">
  <pattern>/.*pc.*/PC</pattern>
  <pattern>/.*srv.*/Server</pattern>
</type_mapping>
<import_software_packages>yes</import_software_packages>
Parameter Description Sample value
default_org_id The name of the iTop organization under which inventory items should be stored. Demo
check_mk_dir The directory where the collector should check for inventory data files. /var/lib/check_mk/inventory/
type_mapping The mapping table which specifies how hosts should be categorized based on their host name in check_mk. Takes patterns in the format /pattern/replacement, where pattern is a regex matched against the host name and replacement must either be Server or PC. Host names which match no pattern will be ignored. <pattern>/.*/Server</pattern> - catch-all pattern which categorizes all hosts as servers
import_software_packages determine whether the software packages should be imported or not (yes or no) yes

Optional Parameters

The following optional boolean parameters might be set according to your needs in conf/params.local.xml aswell:

<use_network_hostname>false</use_network_hostname>
<skip_gz>true</skip_gz>
<skip_dot>true</skip_dot>
Parameter Description Sample value
use_network_hostname Use the host name used on the network by a given inventory machine instead of using the host name stored it is stored under in check_mk. Default is false. false
skip_gz When looking for inventory files, skip .gz counterparts if they are found. Default is true. true
skip_dot Skip any dotfiles encountered in the inventory file directory. Default is true. true

Standardisierte Felder

It might be of interest for your, to specify default values for certain fields, which otherwise won't be filled due to the data delivered by CheckMK. You can do so by editing collectors/params.distrib.xml as follows. (And you can overwrite those defaults in conf/params.local.xml.)

<default_fields type="hash">
  <field_name>field value</field_name>
  <field_name2>another field value</field_name2>
</default_fields>
<default_fields_sw type="hash">
  <status>active</status>
</default_fields_sw>

Default fields for servers an PCs can be specified in <default_fields>. Default fields for software packages can be specified in <default_fields_sw>

This is especially useful for iTop fields like “status” or “business criticality”.

Collecting Data

The data delivered by CheckMK do not always fit the iTop Data Structures (which you might have modified). A mapping table is used to convert data acccording to your datamodel. This collector uses this feature for Brands, Models, OS-Families and OS-Versions.

Default Mapping Tables are located in collectors/params.distrib.xml. You can overwrite those via conf/params.local.xml. The file collectors/params.template.xml contains more examples and more information.

Example:

<osfamily_mapping type="array">
  <pattern>/.*linux.*/Linux</pattern>
  <pattern>/.*windows.*/Windows</pattern>
  <pattern>/.*mac os.*/Mac OS</pattern>
  <pattern>/.*solaris.*/Solaris</pattern>
  <pattern>/.*bsd.*/BSD</pattern>
  <pattern>/.*/$1$s</pattern>
</osfamily_mapping>

This example considers Linux, Windows, Mac, Solars and BDS OS-Families normalizing their naming. With <pattern>/.*/$1$s</pattern> you are instructing the collector to keep all non-fitting names to be preserved as-is. Always keep this rule, but always keep this rule the last one.

Usage

The first time you are running the collector, it is recommended to run the collector as follows:

php exec.php --configure_only

By running the collector with the parameter “–configure_only” you are instructing it to only connect to iTop and create a new SynchroDataSource as specified by the collector.

To only collect data (but not synchronize with iTop), use the following command:

php exec.php --collect_only

aus.

This instructs the collector to collect the data delivered by CheckMK, transform and output the transformed data into the folder data/ into csv-files. This might be useful to manually check the transformed data for missing or wrongly transformed data.

Finally, if all the data is transformed correctly, you now can run

php exec.php --synchro_only

to synchronize the data with iTop for the first time, but you should run –collect directly before that command again.

If everything works, you can use the command

php exec.php

to do all of the above with one command.

More information on how and which runtime parameters for the collector can be used you can find at itop-data-collector-base

Data Collection Reference

Servers and PCs

Server and PCs contain the same types of data.

Field in iTop check_mk inventory entry
name* The check_mk host name, which is the name of the check_mk inventory data file, or networking host name (see Configuration above)
org_id* Taken from default_org_id in configuration
cpu hardware → cpu → model
ram hardware → memory → total_ram_usable (in MiB)
osfamily_id Mapped using data in either software→os→type or software→os→name using osfamily_mapping pattern list
osversion_id Mapped from software→os→name using osversion_mapping pattern list
serial hardware → system → serial
brand_id Mapped from hardware→system→vendor using brand_mapping pattern list
model_id Mapped from hardware→system→family

Software

If activated Software will be collected and synchronized as objects in the iTop Software Catalogue.

Field in iTop check_mk inventory entry
name* software → packages[] → name
type Mapped from software → packages[] → name using sw_type_mapping pattern list
vendor* Mapped from software → packages[] → package_type using softwarevendor_mapping pattern list
version* software → packages[] → version

SoftwareInstances

All Softwareinstanzen like PCSoftware, WebServer, DBServer, Middleware oder OtherSoftware contain the same set of fields to be synchronized.

Field in iTop check_mk inventory entry
name* Concatenation of software → packages[] → name and check_mk host name
org_id* Taken from default_org_id in configuration
system_id* check_mk host name, which is the name of the check_mk inventory data file, or networking host name
software_id Link to software catalog entry (see above) <br> Concatenation of software → packages[] → name and software → packages[] → version

Workaround for running the Collector with CheckMK 2.x

This release is a workaroundrelease to temporarely create compatibilty with how CheckMK 2.x now stores data in Inventory Files.

In this chapter you will learn how to make the collector compatible with CheckMK 2.x

Why does this collector not work out-of-the-box with CheckMK 2.x?

CheckMK 1.x used Python 2. For that reason, the Inventory Files contain serialized objects in Python2 Object Notation (P2ON). The collector compatible with CheckMK 1.x contains a custom parser to handle P2ON.

With the release of CheckMK 2.x they moved to Python 3. Thus, the Inventory Files now contain P3ON. The custom parser for P2ON is not able to parse those files on number of occassions (e. g. special Unicode chars).

This is the reason, ITOMIG has (temporarely) implemented a workaround to still be able to use the collector with CheckMK 2.x.

How is the workaround working (Overview)?

The workaround is based on letting Python 3 do the work to change the format of the Inventory Files from P3ON to valid, simple JSON, which PHP/the collector is able to hanle easily.

How to apply the workaround?

Assumptions

It is assumed, that

  • you have read and understood the manual above (how the collector basically works) and that you have configured it according to your needs,
  • you are using CheckMK 2.x,
  • there is Python 3 installed on the machine, where the collector will be residing,

For the following example we furthermore assume, that

  • the Inventory Files of CheckMK are saved in /var/lib/checkmk/inventory_files/,
  • the collector resides in the folder /etc/itop/collectors/collector.checkmk/ and
  • the Cronjob for running the collector is configured to run once every day at 10:23 a.m.

Initial configuration

  • Inside the file <collectorpath>/workaround/pon2json.py modify the values of the two top variables according to your needs, e. g.:
path_original = "/var/lib/checkmk/inventory_files/"
path_output = "/etc/itop/collectors/collector.checkmk/json_files/"
  • Pay attention, the folder “json_files” really exists and is writable for the cronuser.
  • Edit the file <collectorpath>/collectors/params.distrib.xml like so: <check_mk_dir>/etc/itop/collectors/collector.checkmk/json_files/</check_mk_dir>, such that the collector is configured to read the JSON-Files, which are already transformed.

Manual Tests

  • Move inside the folder workaround folder inisde the collector and on the commandline and run the following command python3 pon2json.py. The expected output of the script looks like this:
Inputfolder: /var/lib/checkmk/inventory_files/
Files found:
['server-01', 'pc-01', 'example.txt']
Files to ignore:
['example.txt']
Files to convert:
{'server-01', 'pc-01'}
server-01 converted...
pc-01 converted...
Outputfolder: /etc/itop/collectors/collector.checkmk/json_files/
Done
  • Now check, there really are transformed JSON-Files written to the folder “json_files” and that the content looks reasonable.
  • Afterwards move inside the root-folder of the collector and run php exec.php --collect_only.
  • Check for new/updated csv-files in the folder /data/ of your collector and that the content looks reasonable.
  • Finally, run a complete synchronisation by running the command php exec.php from the root folder of your collector. In iTop, check, if Models, Brands, Server, PCs and so on are created according to your configuration and the Inventory Files.

Configuration the Cronjobs

  • Create a new Cronjob additional to the Cronjob already existing for running the collector.
    • The content should look something like this: 17 10 * * * /usr/bin/python3 /etc/itop/collectors/collector.checkmk/workaround/pon2json.py >> /var/www/html/itop/log/checkmk-konverter.log 2>&1
    • With this Cronjob you are making sure, that the actual collector always works on freshly transformed Inventory Files, because the collector runs 5 minutes after the transformation from P3ON to JSON.

Other notes

  • Pay attention for “Traceback”-Messages inside the converter log. If they appear an error happened during the transformation from P3ON to JSON.
  • The converter assumes, all applicable Inventory Filenames in the Inventory Folder do not contain any dots. E. g., they should be named “pc-01” and son on. Files with dots in the filename are ignored. Other folders inside the Inventory Folder aswell.
extensions/itomig/check_mk-collector.txt · Last modified: 2022/07/25 09:29 by vdumas
Back to top
Contact us