topic Data Gateway Splunk Output - No host in Product Forum

Data Gateway Splunk Output - No host

ZachF — Thu, 08 Oct 2020 13:28:00 GMT

We have integrated data gateway with Splunk, however, we are noticing the output is being sent and the Blue Prism server that sends it is not being identified as the host. The Splunk HEC API allows for additional fields to be sent with the API request, one being host. Without the host field in the request, Splunk will default to the HEC aggregator host. This causes a problem if you have multiple servers writing to Splunk because you can't identify where it is coming from.

Is there a way to customize the output

It looks like what is being sent is as a payload to Splunk is

{"event":
{"EventType": 1 ...}
}

What should be sent is:

{"host": <<ServerHostName>>,
"event":{
{"EventType": 1...}
}

I have seen in the configuration guide that you can filter the output, but I haven't seen anything specifying you can customize the actual JSON being sent to the Splunk or HTTP servers.

------------------------------
Zach F
------------------------------

RE: Data Gateway Splunk Output - No host

ZachF — Thu, 08 Oct 2020 16:39:00 GMT

I was able to solve this. Posting the solution in case it helps anyone else out.

The logstash output can be configured. An update to the mapping can add the hostname, whether static or variable.

mapping => ["host", "${COMPUTERNAME}","source", "bp:datagateway, "event", "%{event}"]

This example will use the environment variable (ComputerName) in windows and sets it as host. A static value could have been used instead

mapping => ["host", "example.com", "source", "bp:datagateway, "event", "%{event}"]

This will format the outbound JSON message

{"host": <<ServerHostName>>, <-- variable (COMPUTERNAME) or static value
"source": "bp:datagateway",
"event":{
{"EventType": 1...}
}

Additional values can be sent using the mapping, these are just an example.

------------------------------
Zach F
------------------------------

RE: Data Gateway Splunk Output - No host

charliekovacs — Thu, 08 Oct 2020 18:34:00 GMT

Zach,

Glad to hear you have a working solution! I want to echo a couple of points for others that may be also be looking at similar problems/solutions.

The engine behind Data Gateways is Logstash, so the Data Gateways configuration file is really a Logstash config. That means the Logstash documentation is a good resource for expanding what Data Gateways can do.

Two important points:
1. The config file can reference environment variables (i.e. computer name, ports, folder paths, etc.):
https://www.elastic.co/guide/en/logstash/current/environment-variables.html

2. Blue Prism event data is written in JSON format. For each output type (file, database, HTTP, Splunk), you can edit the output section in the config file to add more info in the JSON body than what is included by default. In Zach's example, it's the mapping parameter in the HTTP Output Plugin that can be added onto:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-mapping

------------------------------
Charles Kovacs
Developer Consultant
Blue Prism
America/Chicago
------------------------------