vRealize Log Insight Data Sets using custom filters

IMPORTANT NOTE: All log file examples are not real and exact PCF log files. Since I do not have a running PCF environment while writing this post, I have used some fake examples and modified them to meet my needs.

Let’s consider following use case.

You are responsible for operating a Pivotal Cloud Foundry (PCF) environment.

PCF is sending all log messages for all spaces, orgs and apps etc. to one central vRealize Log Insight cluster.

Your developers using the PCF services would like to access their log messages.

Now, you could just grant read access to all developers but doing that you would allow every developer see all messages received by vRLI, not only PCF related.

The first and pretty easy solution is to leverage the static tag already being used by the PCF Content Pack.

The installation instruction of the content pack says:

Log Insight Setup:

1.      Obtain a Virtual IP from your IT department that will be under the same Subnet as your Log Insight environment.

2.    Follow the `Procedure` defined in the following link: http://pubs.vmware.com/log-insight-33/topic/com.vmware.log-insight.administration.doc/GUID-72CB469A-F276-438D-BC93-77853E2CC0B9.html

3.    When prompted for `a list of static tags (key=value)`, enter the following:

          

            product=pcf

This static tag can be used for creation of an appropriated data set, which will contain only logs from your PCF environment.

If you run this settings in interactive analytics you will get only those log messages which are tagged with "product=pcf":

But what if you want your developers to access only logs which belongs to their PCF org or/and space?

Reading the documentation, you could come up with the idea of using extracted fields:

"Use the first drop-down menu to select a field defined within vRealize Log Insight to filter on. 

For example, hostname. 

The list contains all defined fields that are available statically, in content packs, and in custom content."

But if you try to use your own extracted fields in a data set you will notice that this kind of fields are not available in a data set configuration.

The solution I have used with my last customer was to configure the vRLI agent on the PCF syslog server to set static fields "dynamically" via RegEx expression. These static fields based on app or space IDs can be used in data set filters. 

NOTE: This is just an example (fake messages):

[filelog|pcf-syslog]

directory=/var/log/pcf_sample

include=*.log

tags={"tkopton_tag01":"testing-orgs"}

parser=pcforg-parser

enabled=yes

[parser|pcforg-parser]

base_parser=regex

format=cf_org_id=(?<tkopton_tag02>\S+\b)

debug=no

next_parser=pcfspace-parser

[parser|pcfspace-parser]

base_parser=regex

format=cf_space_id="(?<tkopton_tag03>\S+\b)

debug=no

next_parser=syslog_parser

 Now, you can use those fields as filters in data sets, like:

And the test using "Run in Interactive Analytics":

IMPORTANT NOTE: Tested on a PCF Linux syslog server, I do not think, this will work on a Windows system;-)