Scripted Inputs and Splunk

Josh Porto, Senior Consultant, Cyber Engineering, Coalfire

Splunk is an extremely versatile tool when dealing with data:

- Monitor files?  Check!

- Listen in on an open port?  Check!

- Monitor the file system? Performance monitor?  HTTP Event Collector?

- Check, check aaaaand check!

But what if the data you want to ingest does not have a method listed above? Say, something like a database or a security tool’s API? Scripted inputs are the solution! Splunk can even employ a variety of scripts to include (but not limited to) PowerShell, shell scripts, and Python. Besides working around data sources, which do not use log files and cannot send via TCP or UDP, the advantages abound and include:

  • Structure (or restructure) data for easy ingestion and manipulation
    • Examples: Key value pairs and/or Common Information Model (CIM) compliance
  • Discard unnecessary data before it hits your indexers and your license
  • Utilize a tools API

So how do we go about implementing this magical data ingestion method? As with most methods there are a few caveats, but these can be overcome with a little Splunk-fu and knowledge!

For this example, we’ll be using a Python script. First, let’s test our script using Splunk’s Python version (as opposed to Linux’s Python version). This will verify that any dependencies the script uses are available. (TIP: Copy any missing dependencies to /opt/splunk/lib/python2.7/site-packages/):

$SPLUNK_HOME/bin/splunk cmd python <SCRIPT_NAME>.py

We can validate the output is the data we want in a format we expected. Now we’ll copy the script to whichever app we want this script associated with:


Next, lets add a stanza to the inputs.conf of the same app:

disabled = 0
host = <HOSTNAME>
#Hostname will dynamically resolve to the hostname of the host the script is executed on if blank
index = <INDEX>
interval = 86400
#How often execution script occur in seconds if an integer or set to cron job
sourcetype = <SOURCETYPE>

If the script is set to an integer, the script will execute on boot and the interval timer will begin. If Splunk is restarted inside that interval, the script will run again on boot and the timer will restart. Additionally, if the script has not finished executing before the timer runs out, Splunk will wait for the instance to complete before executing another instance.

Run a search, and behold the glory that unfolds before you! At this point, we can start to treat the data like we would any other new data source, configure the props.conf and transforms.conf, create reports and dashboards, and generally be more awesome.

That’s all there is to it!

As always, Splunk has boundless additional detailed documentation, which I could not include without writing a novel!
Josh Porto


Josh Porto — Senior Consultant, Cyber Engineering, Coalfire

Recent Posts

Post Topics



Accounting Agency AICPA Assessment assessments ASV audit AWS AWS Certified Cloud Practitioner AWS Certs AWS Summit bitcoin Black Hat Black Hat 2017 blockchain Blueborne Breach BSides BSidesLV Burp BYOD California Consumer Privacy Act careers CCPA Chertoff CISO cloud CMMC CoalfireOne Compliance Covid-19 credit cards C-Store Culture Cyber cyber attacks Cyber Engineering cyber incident Cyber Risk cyber threats cyberchrime cyberinsurance cybersecurity danger Dangers Data DDoS DevOps DevSecOps DFARS DFARS 7012 diacap diarmf Digital Forensics DoD DRG DSS e-banking Education encryption engineering ePHI Equifax Europe EU-US Privacy Shield federal FedRAMP financial services FISMA Foglight forensics Gartner Report GDPR Google Cloud NEXT '18 government GRC hack hacker hacking Halloween Health Healthcare heartbleed Higher Education HIMSS HIPAA HITECH HITRUST HITRUST CSF Horror Incident Response interview IoT ISO IT JAB JSON keylogging Kubernetes Vulnerability labs LAN law firms leadership legal legislation merchant mobile NESA News NH-ISAC NIST NIST 800-171 NIST SP 800-171 NotPetya NRF NYCCR O365 OCR of P2PE PA DSS PA-DSS password passwords Payments PCI PCI DSS penetration Penetration Testing pentesting Petya/NotPetya PHI Phishing Phising policy POODLE PowerShell Presidential Executive Order Privacy program Ransomware Retail Risk RSA RSA 2019 Safe Harbor Scanning Scans scary security security. SOC SOC 2 social social engineering Spectre Splunk Spooky Spraying Attack SSAE State Stories Story test Testing theft Virtualization Visa vulnerability Vulnerability management web Wifi women XSS