Skip to main content
search

Documentation: Splunk Admin Ninja

Technical Add-on for Splunk built by Splunk admins to manage track & audit Splunk instances

ENQUIRE NOW

Overview

The Admin Ninja App is a Search Head App of Splunk, which is a companion-app to the Admin Ninja TA – built by Splunk admins for Splunk admins – that will greatly assist any Splunk admin in managing, tracking & auditing their wide array of Splunk instances.

The TA pulls data from each Splunk instance you install it on about the Splunk instance itself. This data includes everything from configs and settings to users, messages and more. The reason this is needed is because Splunk doesn’t log everything about itself, and the _audit index could use with more detailed auditing.
All of this culminates in a Splunk admin wanting more trackability and information about the Splunk architecture that they manage, which becomes increasingly difficult as any environment gets larger and larger.

What is the SH App used for?

The app is used to aggregate and organise the data ingested by the Admin Ninja TA on your environment’s Search Head. This is useful for keeping track of your entire environment from a glance – similar to a Monitoring Console, however this will account for additional info that isn’t covered by a regular Splunk install.

How does the TA get this information?

The TA retrieves information about the Splunk instance it’s running on by pinging the REST API of that instance, and ingesting the results in JSON format.
When used in a distributed manner, this provides extremely efficient tracking of key settings & provides auditability of what was changed and when.

Documentation

Install

To install either the SH App or TA:

For Splunk Enterprise:

For Splunk Enterprise, it is recommended to install as much as possible through the Deployment Server (for HF’s), Deployer (for SH’s) and Cluster Master (for Indexers) to allow better management.

Manual Install

  1. Download the app from Splunkbase
  2. Navigate to the Splunk instance you wish to install on.
  3. Navigate to Apps > Manage Apps > Install App from File
  4. Upload the app package, and restart Splunk.

    Via Deployment Server

  5. Download the app from Splunkbase
  6. Configure according to the ‘Setup & Config’ section of this document
  7. Unpack the app package and extract to your Deployment Server’s $SPLUNK_HOME/etc/deployment-apps/
  8. Configure a serverclass to push the app out as per the Splunk Documentation.

    Via Search Head Deployer

  9. Download the app from Splunkbase
  10. Configure according to the ‘Setup & Config’ section of this document
  11. Unpack the app package and extract to your Deployer’s $SPLUNK_HOME/etc/shcluster/apps
  12. Push the app out as per the Splunk Documentation.

    Via Cluster Master

  13. Download the app from Splunkbase
  14. Configure according to the ‘Setup & Config’ section of this document
  15. Unpack the app package and extract to your Cluster Master’s $SPLUNK_HOME/etc/manager-apps/
  16. Push the app out as per the Splunk Documentation.

    For Splunk Cloud:

    If on Splunk Cloud (Classic):

  17. Raise a request to your Splunk Cloud Support team to install the apps with app ID’s 6665 & 6664.
  18. Wait as the Splunk Cloud Support team installs the apps for you.

    If on Splunk Cloud (Victoria):

  19. Navigate to the Splunk Cloud instance you wish to install on
  20. Navigate to Apps > Manage Apps > Browse more Apps
  21. Search for ‘Admin Ninja’, and click install on both the SH App and TA.

Setup & Config

Search Head App:

After installation, there are a few setup actions we need to take:
Configure Macros

  1. Navigate to Settings > Advanced Search > Macros
  2. Filter by app: “Admin Ninja App”
  3. Edit the below macros:
    • ninja_index: This should be configured to match the index you plan on ingesting Admin Ninja data to.
      • Default: index=main
      • Example: index=adminindex
    • ninja_summary: This should be configured to match the index you plan on summary indexing Admin Ninja data to (Don’t include ‘index=’).
      • Default: summary,
      • Example: adminindex_summary
    • ninja_blacklist_apps: This should be configured to exclude apps you don’t want to see in the App Support or Splunk Apps dashboards.
      • Default:
        NOT name IN("splunk_essentials_8_2", "splunk_essentials_9", "splunk_ingest_actions", "splunk_enterprise_on_docker")
        
      • Example:
        NOT name IN("splunk_essentials_8_2", "splunk_essentials_9", "splunk_ingest_actions", "splunk_enterprise_on_docker", "TA_admin_ninja", "*_hf")
        
    • ninja_blacklist_messages: This should be configured if you are receiving common Splunk UI Messages that you want to filter out, e.g. “File integrity checks found XX files that did not match the system-provided manifest.”

Configure Alerts & Reports

  1. Navigate to Settings > Searches, reports and alerts
  2. Filter by app: “Admin Ninja App”
  3. Review & add appropriate alert actions to the alerts (E.g. Add email action to alerts for Splunk Admin team)

Technical Add-on:

After installation, setup is super simple:

  1. Determine what inputs you need on your given Splunk Enterprise install. Refer to the ‘Data Description’ section of this document to learn more.
  2. Once you’ve determined what inputs you need, set them up through either the UI or backend under $SPLUNK_HOME/etc/apps/TA_admin_ninja/local/inputs.conf (Example has been provided below).
    1. Through the UI: Navigate to Settings > Data Inputs.
      • Ideally, make sure you do this under the TA_admin_ninja context by changing your url to: http(s)://$SPLUNK_HOST/en-GB/manager/TA_admin_ninja/datainputstats. This will ensure inputs are written to the correct inputs.conf under the TA_admin_ninja app.
    2. For each of the inputs you want to set-up, find the input name in the list of data inputs, click on it, and fill out the input details, including:
      • Name: Unique name of the input
      • Maximum Entries: Number of results to return, this should ALWAYS be 0 unless testing.
      • Interval: Frequency in seconds for inputs to run (defaults have been provided and will be pre-loaded on a clean input).
      • Index: Index to ingest data to (default is ‘index=default’).

Example inputs.conf

[ninja_apps://apps]
index = adminindex
interval = 3600
maximum_entries = 0

[ninja_authtokens://auth tokens]
index = adminindex
interval = 3600
maximum_entries = 0

[ninja_configdir://config directory]
index = adminindex
interval = 3600
maximum_entries = 0

[ninja_dashboards://dashboards]
index = adminindex
interval = 3600
maximum_entries = 0

[ninja_datamodels://datamodels]
index = adminindex
interval = 86400
maximum_entries = 0

# App Support Input - run on Search Head
[ninja_appsupport://appsuppooooort]
apps_lookup = admin_ninja_unique_apps.csv
interval = 86400
lookup_located_app = admin_ninja_app
sourcetype = ninja:apps:support
index = main
disabled = 0

Splunkbase Input

Other than the default set-up, you can also set-up an input from a Search Head (that has access to indexers with the Admin Ninja data) to query Splunkbase for the latest Support & Compatibility info.

Pre-req’s
  1. Ensure you’re bringing in ninja apps data from all your Enterprise or Cloud instances
  2. Ensure the savedsearch “App Support lookup gen” is enabled and successfully filling out the lookup ninja_app_support, from the SH app admin_ninja_app.
  3. Ensure the Search Head has connectivity to https://splunkbase.splunk.com/.
    • In order to do this for Splunk Cloud, you can refer to the docs to open up an outgoing rule, e.g.:
      curl --request POST \
      --url https://admin.splunk.com/<stackname>/adminconfig/v2/access/outbound-ports \
      --header 'Authorization: Bearer <token>' \
      --header 'Content-Type: application/json' \
      --data '{
      "outboundPorts": [{"subnets": ["3.90.75.149/32", "52.86.171.41/32", "44.194.123.38/32"], "port": 443}],
      "reason": "Opening connection to splunkbase.splunk.com"
      }'
      
  4. Otherwise, Proxy support is under consideration for a future release!
Enabling
  1. Navigate to Settings > Data Inputs > Admin Ninja: App Support Details
  2. If using the default lookup from the admin_ninja_app, specify:
    • Unique App List Lookup: admin_ninja_unique_apps.csv
    • App Lookup is located in: admin_ninja_app
  3. Give the input a name, and click Save.

Data Description

Below is a list of recommendations for what inputs should go on what servers.

  • Note! The License Pools input is NOT supported on Splunk Cloud.
Input Name Script Sourcetype Recommended Server Types
Admin Ninja: Apps ninja_apps.py ninja:apps All Splunk Instances
Admin Ninja: Authentication Tokens ninja_authtokens.py ninja:authtokens All Splunk Instances
Admin Ninja: KV Store Status ninja_kvstatus.py ninja:kvstore All Splunk Instances
Admin Ninja: Messages ninja_messages.py ninja:messages All Splunk Instances
Admin Ninja: Disk Partitions ninja_partitions.py ninja:partitions All Splunk Instances
Admin Ninja: Roles ninja_roles.py ninja:roles All Splunk Instances
Admin Ninja: Server Info ninja_server_info.py ninja:serverinfo All Splunk Instances
Admin Ninja: Users ninja_users.py ninja:authentication All Splunk Instances
Admin Ninja: Deployment Server Apps ninja_ds_apps.py ninja:dsapps Deployment Server
Admin Ninja: Deployment Clients ninja_ds_clients.py ninja:dsclients Deployment Server
Admin Ninja: DS Serverclasses ninja_ds_serverclasses.py ninja:dsserverclasses Deployment Server
Admin Ninja: HEC Tokens ninja_hec_tokens.py ninja:httpinputs Heavy Forwarders/Indexers (depending on HEC input endpoint location)
Admin Ninja: Indexes ninja_indexes.py ninja:indexes Indexers
Admin Ninja: License Pools ninja_license_pools.py ninja:licenserpools License Master
Admin Ninja: Config Directory ninja_configdir.py ninja:configdirectory Search Heads
Admin Ninja: Dashboards ninja_dashboards.py ninja:dashboards Search Heads
Admin Ninja: Datamodels ninja_datamodels.py ninja:datamodel Search Heads
Admin Ninja: Eventtypes ninja_eventtypes.py ninja:eventtypes Search Heads
Admin Ninja: Lookups ninja_lookups.py ninja:lookups Search Heads
Admin Ninja: Macros ninja_macros.py ninja:macros Search Heads
Admin Ninja: Calculated Fields ninja_calcfields.py ninja:calcfields Search Heads
Admin Ninja: Savedsearches ninja_savedsearches.py ninka:savedsearches Search Heads

Troubleshooting

Log Files

The TA writes its logs to both of the below files:

  • $SPLUNK_HOME/var/log/splunk/splunkd.log
  • $SPLUNK_HOME/var/log/admin_ninja/$scriptname$.log

The most common cause of data not being ingested is that there is simply no data to retrieve from the endpoint.
This will be highlighted within the admin_ninja logs as “No data to retrieve! Likely no config item exists in Splunk for this endpoint type.

Searching

The best Splunk searches to run to troubleshoot the TA (if no data is being onboarded) are:

  • Search _internal splunkd.log:
    index=_internal host=$host$ sourcetype=splunkd TA_admin_ninja (ERROR OR WARN OR FATAL)
  • Search TA-generated logs:
    index=_internal host=$host$ sourcetype=admin_ninja:log source=*$scriptname$.log

Known Issues

  1. There is a known issue where the Admin Ninja: Disk Partitions endpoint doesn’t work on the initial restart of a Splunk Instance (with that input already configured). It takes some time for the endpoint to calculate partition size after restarting Splunkd, so after 5-10 minutes the endpoint should work again.
  2. Inputs don’t work on Search Heads in a ‘Classic’ (non-victoria) Splunk Cloud environment.

Upgrading

In order to upgrade, please follow the exact same steps found in under the Install section of this document (however, you will need to check ‘update’ when uploading the app, or asking Splunk Support to do it for you). Special notes and provisions for upgrading from version-to-version will be listed below.

  • 1.0.0 -> 1.0.1: No special requirements.

Release Notes

Admin Ninja App

v1.0.2

Fixed:

  • MAD-49: Fixed an issue to do with the App Support lookup generating search not retrieving the latest result per app
  • MAD-56: Fixed the percentages shown for disk availability panel on the Architecture page
  • MAD-57: Fixed an issue where more than 1 deployment server wouldn’t show up in the DS dashboard drop-down input.

Changes:

  • MAD-58: Added details to the architecture table on the Architecture dashboard to include useful information such as CPU cores and memory.

v1.0.1

Fixed:
  • MAD-34: Auth Tokens dashboard: limit argument was being used that caused some issues on particular versions of Splunk.
  • MAD-35: Dashboard panels for mount point info are referring to incorrect field names
  • MAD-36: Fixed bad use of export = \<value\> in default.meta
  • Fixed README to refer to new documentation page
  • Fixed bad syntax on the Architecture OS input list bug
  • MAD-42: App permissions were read-all (by default)
  • MAD-40: Removed old item in Navigation XML that was causing users to be shown and directed to any other dashboard called indexes.xml
New:
  • MAD-18: MAJOR FEATURE! Added functionality to scan splunkbase API for installed apps compatability & latest version info, under “Admin Ninja : AppSupport”.
  • MAD-11: Added Calculated Fields to KO Content Search dashboard.
  • MAD-39: Removed the Configs & Objects dashboard – has been replaced by HEC Tokens dashboards
  • Added two blacklist macros, ninja_blacklist_apps and ninja_blacklist_messages to filter out annoying or unneeded results, such as apps dedicated to having an outputs.conf, etc.
  • Added alert for license expiry
  • Added Unique count panels in Apps & Users dashboards
Changes:
  • Changed some applicable index-time-field filters to use tstats for dynamic results.
  • Added more detailed descriptions to each dashboard
  • MAD-41: Made several small changes and minor bugfixes to dashboards
  • Identified panels requiring *_configtracker* index as requiring v9.0.0+
  • Apps Dashboard: Changed “Percent of Splunkbase Apps that require update” to use the new ninja_app_support lookup as it’s far more accurate.
  • Significant changes to Architecture & Health including changing single panels that state environment health.

Admin Ninja TA

v1.0.1

Fixed:
  • MAD-27: Savedsearches input doesn’t work on Splunk Cloud.
New:
  • MAD-18: MAJOR FEATURE! Added functionality to scan splunkbase API for installed apps compatability & latest version info, under “Admin Ninja : AppSupport”.
  • MAD-11: Added input for Calculated Fields.
  • Added input for Splunk Licenses, pulling license file info from your Splunk LM.
  • MAD-37: Enable a default value of 0 for ‘maximum entries’ input argument.
Changes:
  • Changed Apps input to filter out ‘core’ apps when making the API call.
  • Improved how requests were being made against the API.
  • Changed an event where no results were found in the endpoint to log to the sourcetype admin_ninja:log rather than splunkd.
Close Menu