Technical Add-on for Splunk built by Splunk admins to manage track & audit Splunk instances
Overview
The Admin Ninja App is a Search Head App of Splunk, which is a companion-app to the Admin Ninja TA – built by Splunk admins for Splunk admins – that will greatly assist any Splunk admin in managing, tracking & auditing their wide array of Splunk instances.
The TA pulls data from each Splunk instance you install it on about the Splunk instance itself. This data includes everything from configs and settings to users, messages and more. The reason this is needed is because Splunk doesn’t log everything about itself, and the _audit index could use with more detailed auditing.
All of this culminates in a Splunk admin wanting more trackability and information about the Splunk architecture that they manage, which becomes increasingly difficult as any environment gets larger and larger.
What is the SH App used for?
The app is used to aggregate and organise the data ingested by the Admin Ninja TA on your environment’s Search Head. This is useful for keeping track of your entire environment from a glance – similar to a Monitoring Console, however this will account for additional info that isn’t covered by a regular Splunk install.
How does the TA get this information?
The TA retrieves information about the Splunk instance it’s running on by pinging the REST API of that instance, and ingesting the results in JSON format.
When used in a distributed manner, this provides extremely efficient tracking of key settings & provides auditability of what was changed and when.
Documentation
Install
To install either the SH App or TA:
For Splunk Enterprise:
For Splunk Enterprise, it is recommended to install as much as possible through the Deployment Server (for HF’s), Deployer (for SH’s) and Cluster Master (for Indexers) to allow better management.
Manual Install
- Download the app from Splunkbase
- Navigate to the Splunk instance you wish to install on.
- Navigate to Apps > Manage Apps > Install App from File
- Upload the app package, and restart Splunk.
Via Deployment Server
- Download the app from Splunkbase
- Configure according to the ‘Setup & Config’ section of this document
- Unpack the app package and extract to your Deployment Server’s $SPLUNK_HOME/etc/deployment-apps/
- Configure a serverclass to push the app out as per the Splunk Documentation.
Via Search Head Deployer
- Download the app from Splunkbase
- Configure according to the ‘Setup & Config’ section of this document
- Unpack the app package and extract to your Deployer’s $SPLUNK_HOME/etc/shcluster/apps
- Push the app out as per the Splunk Documentation.
Via Cluster Master
- Download the app from Splunkbase
- Configure according to the ‘Setup & Config’ section of this document
- Unpack the app package and extract to your Cluster Master’s $SPLUNK_HOME/etc/manager-apps/
- Push the app out as per the Splunk Documentation.
For Splunk Cloud:
If on Splunk Cloud (Classic):
- Raise a request to your Splunk Cloud Support team to install the apps with app ID’s 6665 & 6664.
- Wait as the Splunk Cloud Support team installs the apps for you.
If on Splunk Cloud (Victoria):
- Navigate to the Splunk Cloud instance you wish to install on
- Navigate to Apps > Manage Apps > Browse more Apps
- Search for ‘Admin Ninja’, and click install on both the SH App and TA.
Setup & Config
Search Head App:
After installation, there are a few setup actions we need to take:
Configure Macros
- Navigate to Settings > Advanced Search > Macros
- Filter by app: “Admin Ninja App”
- Edit the below macros:
- ninja_index: This should be configured to match the index you plan on ingesting Admin Ninja data to.
- Default:
index=main
- Example:
index=adminindex
- Default:
- ninja_summary: This should be configured to match the index you plan on summary indexing Admin Ninja data to (Don’t include ‘index=’).
- Default:
summary
, - Example:
adminindex_summary
- Default:
- ninja_blacklist_apps: This should be configured to exclude apps you don’t want to see in the App Support or Splunk Apps dashboards.
- Default:
NOT name IN("splunk_essentials_8_2", "splunk_essentials_9", "splunk_ingest_actions", "splunk_enterprise_on_docker")
- Example:
NOT name IN("splunk_essentials_8_2", "splunk_essentials_9", "splunk_ingest_actions", "splunk_enterprise_on_docker", "TA_admin_ninja", "*_hf")
- Default:
- ninja_blacklist_messages: This should be configured if you are receiving common Splunk UI Messages that you want to filter out, e.g. “File integrity checks found XX files that did not match the system-provided manifest.”
- ninja_index: This should be configured to match the index you plan on ingesting Admin Ninja data to.
Configure Alerts & Reports
- Navigate to Settings > Searches, reports and alerts
- Filter by app: “Admin Ninja App”
- Review & add appropriate alert actions to the alerts (E.g. Add email action to alerts for Splunk Admin team)
Technical Add-on:
After installation, setup is super simple:
- Determine what inputs you need on your given Splunk Enterprise install. Refer to the ‘Data Description’ section of this document to learn more.
- Once you’ve determined what inputs you need, set them up through either the UI or backend under $SPLUNK_HOME/etc/apps/TA_admin_ninja/local/inputs.conf (Example has been provided below).
- Through the UI: Navigate to Settings > Data Inputs.
- Ideally, make sure you do this under the TA_admin_ninja context by changing your url to: http(s)://$SPLUNK_HOST/en-GB/manager/TA_admin_ninja/datainputstats. This will ensure inputs are written to the correct inputs.conf under the TA_admin_ninja app.
- For each of the inputs you want to set-up, find the input name in the list of data inputs, click on it, and fill out the input details, including:
- Name: Unique name of the input
- Maximum Entries: Number of results to return, this should ALWAYS be 0 unless testing.
- Interval: Frequency in seconds for inputs to run (defaults have been provided and will be pre-loaded on a clean input).
- Index: Index to ingest data to (default is ‘index=default’).
- Through the UI: Navigate to Settings > Data Inputs.
Example inputs.conf
[ninja_apps://apps]
index = adminindex
interval = 3600
maximum_entries = 0
[ninja_authtokens://auth tokens]
index = adminindex
interval = 3600
maximum_entries = 0
[ninja_configdir://config directory]
index = adminindex
interval = 3600
maximum_entries = 0
[ninja_dashboards://dashboards]
index = adminindex
interval = 3600
maximum_entries = 0
[ninja_datamodels://datamodels]
index = adminindex
interval = 86400
maximum_entries = 0
# App Support Input - run on Search Head
[ninja_appsupport://appsuppooooort]
apps_lookup = admin_ninja_unique_apps.csv
interval = 86400
lookup_located_app = admin_ninja_app
sourcetype = ninja:apps:support
index = main
disabled = 0
Splunkbase Input
Other than the default set-up, you can also set-up an input from a Search Head (that has access to indexers with the Admin Ninja data) to query Splunkbase for the latest Support & Compatibility info.
Pre-req’s
- Ensure you’re bringing in ninja apps data from all your Enterprise or Cloud instances
- Ensure the savedsearch “App Support lookup gen” is enabled and successfully filling out the lookup ninja_app_support, from the SH app admin_ninja_app.
- Ensure the Search Head has connectivity to https://splunkbase.splunk.com/.
- In order to do this for Splunk Cloud, you can refer to the docs to open up an outgoing rule, e.g.:
curl --request POST \ --url https://admin.splunk.com/<stackname>/adminconfig/v2/access/outbound-ports \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "outboundPorts": [{"subnets": ["3.90.75.149/32", "52.86.171.41/32", "44.194.123.38/32"], "port": 443}], "reason": "Opening connection to splunkbase.splunk.com" }'
- In order to do this for Splunk Cloud, you can refer to the docs to open up an outgoing rule, e.g.:
- Otherwise, Proxy support is under consideration for a future release!
Enabling
- Navigate to Settings > Data Inputs > Admin Ninja: App Support Details
- If using the default lookup from the admin_ninja_app, specify:
- Unique App List Lookup:
admin_ninja_unique_apps.csv
- App Lookup is located in:
admin_ninja_app
- Unique App List Lookup:
- Give the input a name, and click Save.
Data Description
Below is a list of recommendations for what inputs should go on what servers.
- Note! The License Pools input is NOT supported on Splunk Cloud.
Input Name | Script | Sourcetype | Recommended Server Types |
---|---|---|---|
Admin Ninja: Apps | ninja_apps.py | ninja:apps | All Splunk Instances |
Admin Ninja: Authentication Tokens | ninja_authtokens.py | ninja:authtokens | All Splunk Instances |
Admin Ninja: KV Store Status | ninja_kvstatus.py | ninja:kvstore | All Splunk Instances |
Admin Ninja: Messages | ninja_messages.py | ninja:messages | All Splunk Instances |
Admin Ninja: Disk Partitions | ninja_partitions.py | ninja:partitions | All Splunk Instances |
Admin Ninja: Roles | ninja_roles.py | ninja:roles | All Splunk Instances |
Admin Ninja: Server Info | ninja_server_info.py | ninja:serverinfo | All Splunk Instances |
Admin Ninja: Users | ninja_users.py | ninja:authentication | All Splunk Instances |
Admin Ninja: Deployment Server Apps | ninja_ds_apps.py | ninja:dsapps | Deployment Server |
Admin Ninja: Deployment Clients | ninja_ds_clients.py | ninja:dsclients | Deployment Server |
Admin Ninja: DS Serverclasses | ninja_ds_serverclasses.py | ninja:dsserverclasses | Deployment Server |
Admin Ninja: HEC Tokens | ninja_hec_tokens.py | ninja:httpinputs | Heavy Forwarders/Indexers (depending on HEC input endpoint location) |
Admin Ninja: Indexes | ninja_indexes.py | ninja:indexes | Indexers |
Admin Ninja: License Pools | ninja_license_pools.py | ninja:licenserpools | License Master |
Admin Ninja: Config Directory | ninja_configdir.py | ninja:configdirectory | Search Heads |
Admin Ninja: Dashboards | ninja_dashboards.py | ninja:dashboards | Search Heads |
Admin Ninja: Datamodels | ninja_datamodels.py | ninja:datamodel | Search Heads |
Admin Ninja: Eventtypes | ninja_eventtypes.py | ninja:eventtypes | Search Heads |
Admin Ninja: Lookups | ninja_lookups.py | ninja:lookups | Search Heads |
Admin Ninja: Macros | ninja_macros.py | ninja:macros | Search Heads |
Admin Ninja: Calculated Fields | ninja_calcfields.py | ninja:calcfields | Search Heads |
Admin Ninja: Savedsearches | ninja_savedsearches.py | ninka:savedsearches | Search Heads |
Troubleshooting
Log Files
The TA writes its logs to both of the below files:
- $SPLUNK_HOME/var/log/splunk/splunkd.log
- $SPLUNK_HOME/var/log/admin_ninja/$scriptname$.log
The most common cause of data not being ingested is that there is simply no data to retrieve from the endpoint.
This will be highlighted within the admin_ninja logs as “No data to retrieve! Likely no config item exists in Splunk for this endpoint type.”
Searching
The best Splunk searches to run to troubleshoot the TA (if no data is being onboarded) are:
- Search _internal splunkd.log:
index=_internal host=$host$ sourcetype=splunkd TA_admin_ninja (ERROR OR WARN OR FATAL)
- Search TA-generated logs:
index=_internal host=$host$ sourcetype=admin_ninja:log source=*$scriptname$.log
Known Issues
- There is a known issue where the Admin Ninja: Disk Partitions endpoint doesn’t work on the initial restart of a Splunk Instance (with that input already configured). It takes some time for the endpoint to calculate partition size after restarting Splunkd, so after 5-10 minutes the endpoint should work again.
- Inputs don’t work on Search Heads in a ‘Classic’ (non-victoria) Splunk Cloud environment.
Upgrading
In order to upgrade, please follow the exact same steps found in under the Install section of this document (however, you will need to check ‘update’ when uploading the app, or asking Splunk Support to do it for you). Special notes and provisions for upgrading from version-to-version will be listed below.
- 1.0.0 -> 1.0.1: No special requirements.
Release Notes
Admin Ninja App
v1.0.2
Fixed:
- MAD-49: Fixed an issue to do with the App Support lookup generating search not retrieving the latest result per app
- MAD-56: Fixed the percentages shown for disk availability panel on the Architecture page
- MAD-57: Fixed an issue where more than 1 deployment server wouldn’t show up in the DS dashboard drop-down input.
Changes:
- MAD-58: Added details to the architecture table on the Architecture dashboard to include useful information such as CPU cores and memory.
v1.0.1
Fixed:
- MAD-34: Auth Tokens dashboard: limit argument was being used that caused some issues on particular versions of Splunk.
- MAD-35: Dashboard panels for mount point info are referring to incorrect field names
- MAD-36: Fixed bad use of export = \<value\> in default.meta
- Fixed README to refer to new documentation page
- Fixed bad syntax on the Architecture OS input list bug
- MAD-42: App permissions were read-all (by default)
- MAD-40: Removed old item in Navigation XML that was causing users to be shown and directed to any other dashboard called indexes.xml
New:
- MAD-18: MAJOR FEATURE! Added functionality to scan splunkbase API for installed apps compatability & latest version info, under “Admin Ninja : AppSupport”.
- MAD-11: Added Calculated Fields to KO Content Search dashboard.
- MAD-39: Removed the Configs & Objects dashboard – has been replaced by HEC Tokens dashboards
- Added two blacklist macros, ninja_blacklist_apps and ninja_blacklist_messages to filter out annoying or unneeded results, such as apps dedicated to having an outputs.conf, etc.
- Added alert for license expiry
- Added Unique count panels in Apps & Users dashboards
Changes:
- Changed some applicable index-time-field filters to use tstats for dynamic results.
- Added more detailed descriptions to each dashboard
- MAD-41: Made several small changes and minor bugfixes to dashboards
- Identified panels requiring *_configtracker* index as requiring v9.0.0+
- Apps Dashboard: Changed “Percent of Splunkbase Apps that require update” to use the new ninja_app_support lookup as it’s far more accurate.
- Significant changes to Architecture & Health including changing single panels that state environment health.
Admin Ninja TA
v1.0.1
Fixed:
- MAD-27: Savedsearches input doesn’t work on Splunk Cloud.
New:
- MAD-18: MAJOR FEATURE! Added functionality to scan splunkbase API for installed apps compatability & latest version info, under “Admin Ninja : AppSupport”.
- MAD-11: Added input for Calculated Fields.
- Added input for Splunk Licenses, pulling license file info from your Splunk LM.
- MAD-37: Enable a default value of 0 for ‘maximum entries’ input argument.
Changes:
- Changed Apps input to filter out ‘core’ apps when making the API call.
- Improved how requests were being made against the API.
- Changed an event where no results were found in the endpoint to log to the sourcetype admin_ninja:log rather than splunkd.