Building a Multi-node Vagrant Splunk Environment.
In my previous blog post, I focused on creating a multi-node development environment. As a Senior Consultant at Avocado, I recommend creating several environments prior to production for clients in the early stages of the DevOps journey. This was to assist DevOps/ Splunk engineers developing on their local workstation or laptop. I went through the process of creating a basic lab with four tiers for a Splunk architecture (including management, searching, indexing and forwarding tier). In this part, I will explore the integration of the Ansible playbooks. This will assist you with on the fly provisioning of the virtual machine, creating a provisioning server with Ansible installed. This will produce roles in which to provision the servers.
Update the guests.yml file
Firstly let’s update the guests.yml file which should include our provisioning server. This server is called “mgmt”. Remember this is not a Splunk server. It is a provisioning server to be used if you don’t want to install Ansible on your host machine or if your host runs on Windows. I’ve added a shell script to install Ansible on CentOS 7 and another script to set the bash_profile prompt, however mileage may vary using other vagrant boxes. The following shell script is to be executed in the guests.yml file.
In addition, I have included a code block to process the scripts located in the options of guests.yml. under the guest provisioning block.
The ansible_install.sh file looks like the following, it’s a quick and dirty shell installer for Ansible.
So let’s bring up our mgmt box. As in the previous article “vagrant up mgmt” will execute this. Ensure the shell provisioner is running the Ansible install process.
[sourcecode language="plain"]✔ ~/Workspace/multinode-lab [master|✚ 2…41]
09:24 $ vagrant up mgmt
Bringing machine 'mgmt' up with 'virtualbox' provider...
==> mgmt: Importing base box 'centos-7.2-64-base'...
==> mgmt: Matching MAC address for NAT networking...
==> mgmt: Setting the name of the VM: mgmt.dev.avocado.lab
==> mgmt: Clearing any previously set network interfaces...
==> mgmt: [landrush] virtualbox requires an additional private network; adding it
==> mgmt: Preparing network interfaces based on configuration...
mgmt: Adapter 1: nat
mgmt: Adapter 2: hostonly
==> mgmt: Forwarding ports...
mgmt: 22 (guest) => 2222 (host) (adapter 1)
==> mgmt: Running 'pre-boot' VM customizations...
==> mgmt: Booting VM...
==> mgmt: Waiting for machine to boot. This may take a few minutes...
mgmt: SSH address: 127.0.0.1:2222
mgmt: SSH username: vagrant
mgmt: SSH auth method: private key
mgmt: Warning: Remote connection disconnect. Retrying...
==> mgmt: Machine booted and ready!
==> mgmt: Checking for guest additions in VM...
==> mgmt: Setting hostname...
==> mgmt: Configuring and enabling network interfaces...
==> mgmt: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
[landrush] Using enp0s8 (172.28.128.9)
==> mgmt: [landrush] adding machine entry: mgmt.cluster.avocado.lab => 172.28.128.9
[landrush] Using enp0s8 (172.28.128.9)
[landrush] Host DNS resolver config looks good.
==> mgmt: Mounting shared folders...
mgmt: /vagrant => /Users/moleary/Workspace/multinode-lab
==> mgmt: Running provisioner: shell...
mgmt: Running: /var/folders/98/jyl3ljxd3tzg42l38h4t55g40000gn/T/vagrant-shell20160923-16782-1qe75xu.sh
==> mgmt: Loaded plugins: fastestmirror
==> mgmt: Loading mirror speeds from cached hostfile
==> mgmt: * base: centos.mirror.crucial.com.au
==> mgmt: * extras: centos.mirror.crucial.com.au
==> mgmt: * updates: centos.mirror.serversaustralia.com.au
==> mgmt: No package python-pip available.
==> mgmt: No package python-httplib2 available.
==> mgmt: No package python-keyczar available.
==> mgmt: Package git-1.8.3.1-6.el7_2.1.x86_64 already installed and latest version
==> mgmt: Package wget-1.14-10.el7_0.1.x86_64 already installed and latest version
==> mgmt: Package curl-7.29.0-25.el7.centos.x86_64 already installed and latest version
==> mgmt: Package nano-2.3.1-10.el7.x86_64 already installed and latest version
==> mgmt: Resolving Dependencies
==> mgmt: --> Running transaction check
==> mgmt: ---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed
==> mgmt: ---> Package openssl-devel.x86_64 1:1.0.1e-51.el7_2.5 will be installed
==> mgmt: --> Processing Dependency: zlib-devel(x86-64) for package: 1:openssl-devel-1.0.1e-51.el7_2.5.x86_64
==> mgmt: --> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.1e-51.el7_2.5.x86_64
==> mgmt: ---> Package python-devel.x86_64 0:2.7.5-39.el7_2 will be installed
==> mgmt: ---> Package python-jinja2.noarch 0:2.7.2-2.el7 will be installed
==> mgmt: --> Processing Dependency: python-babel >= 0.8 for package: python-jinja2-2.7.2-2.el7.noarch
==> mgmt: --> Processing Dependency: python-markupsafe for package: python-jinja2-2.7.2-2.el7.noarch
==> mgmt: ---> Package python-paramiko.noarch 0:1.12.4-1.el7.centos will be installed
==> mgmt: --> Processing Dependency: python-crypto >= 2.1 for package: python-paramiko-1.12.4-1.el7.centos.noarch
==> mgmt: --> Processing Dependency: python-ecdsa for package: python-paramiko-1.12.4-1.el7.centos.noarch
==> mgmt: --> Running transaction check
==> mgmt: ---> Package krb5-devel.x86_64 0:1.13.2-12.el7_2 will be installed
==> mgmt: --> Processing Dependency: libverto-devel for package: krb5-devel-1.13.2-12.el7_2.x86_64
==> mgmt: --> Processing Dependency: libselinux-devel for package: krb5-devel-1.13.2-12.el7_2.x86_64
==> mgmt: --> Processing Dependency: libcom_err-devel for package: krb5-devel-1.13.2-12.el7_2.x86_64
==> mgmt: --> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.13.2-12.el7_2.x86_64
==> mgmt: ---> Package python-babel.noarch 0:0.9.6-8.el7 will be installed
==> mgmt: ---> Package python-crypto.x86_64 0:2.6.1-1.el7.centos will be installed
==> mgmt: ---> Package python-ecdsa.noarch 0:0.11-3.el7.centos will be installed
==> mgmt: --> Processing Dependency: python-six for package: python-ecdsa-0.11-3.el7.centos.noarch
==> mgmt: ---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed
==> mgmt: ---> Package zlib-devel.x86_64 0:1.2.7-15.el7 will be installed
==> mgmt: --> Running transaction check
==> mgmt: ---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed
==> mgmt: ---> Package libcom_err-devel.x86_64 0:1.42.9-7.el7 will be installed
==> mgmt: ---> Package libselinux-devel.x86_64 0:2.2.2-6.el7 will be installed
==> mgmt: --> Processing Dependency: libsepol-devel >= 2.1.9-1 for package: libselinux-devel-2.2.2-6.el7.x86_64
==> mgmt: --> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.2.2-6.el7.x86_64
==> mgmt: --> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.2.2-6.el7.x86_64
==> mgmt: ---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed
==> mgmt: ---> Package python-six.noarch 0:1.9.0-2.el7 will be installed
==> mgmt: --> Running transaction check
==> mgmt: ---> Package libsepol-devel.x86_64 0:2.1.9-3.el7 will be installed
==> mgmt: ---> Package pcre-devel.x86_64 0:8.32-15.el7_2.1 will be installed
==> mgmt: --> Finished Dependency Resolution
==> mgmt:
==> mgmt: Dependencies Resolved
==> mgmt:
==> mgmt: ================================================================================
==> mgmt: Package Arch Version Repository Size
==> mgmt: ================================================================================
==> mgmt: Installing:
==> mgmt: PyYAML x86_64 3.10-11.el7 base 153 k
==> mgmt: openssl-devel x86_64 1:1.0.1e-51.el7_2.5 updates 1.2 M
==> mgmt: python-devel x86_64 2.7.5-39.el7_2 updates 392 k
==> mgmt: python-jinja2 noarch 2.7.2-2.el7 base 515 k
==> mgmt: python-paramiko noarch 1.12.4-1.el7.centos extras 900 k
==> mgmt: Installing for dependencies:
==> mgmt: keyutils-libs-devel x86_64 1.5.8-3.el7 base 37 k
==> mgmt: krb5-devel x86_64 1.13.2-12.el7_2 updates 649 k
==> mgmt: libcom_err-devel x86_64 1.42.9-7.el7 base 30 k
==> mgmt: libselinux-devel x86_64 2.2.2-6.el7 base 174 k
==> mgmt: libsepol-devel x86_64 2.1.9-3.el7 base 71 k
==> mgmt: libverto-devel x86_64 0.2.5-4.el7 base 12 k
==> mgmt: pcre-devel x86_64 8.32-15.el7_2.1 updates 479 k
==> mgmt: python-babel noarch 0.9.6-8.el7 base 1.4 M
==> mgmt: python-crypto x86_64 2.6.1-1.el7.centos extras 470 k
==> mgmt: python-ecdsa noarch 0.11-3.el7.centos extras 69 k
==> mgmt: python-markupsafe x86_64 0.11-10.el7 base 25 k
==> mgmt: python-six noarch 1.9.0-2.el7 base 29 k
==> mgmt: zlib-devel x86_64 1.2.7-15.el7 base 50 k
==> mgmt:
==> mgmt: Transaction Summary
==> mgmt: ================================================================================
==> mgmt: Install 5 Packages (+13 Dependent packages)
==> mgmt:
==> mgmt: Total download size: 6.5 M
==> mgmt: Installed size: 23 M
==> mgmt: Downloading packages:
==> mgmt: --------------------------------------------------------------------------------
==> mgmt: Total 4.2 MB/s | 6.5 MB 00:01
==> mgmt: Running transaction check
==> mgmt: Running transaction test
==> mgmt: Transaction test succeeded
==> mgmt: Running transaction
==> mgmt: Installing : libcom_err-devel-1.42.9-7.el7.x86_64 1/18
==> mgmt:
==> mgmt: Installing : python-six-1.9.0-2.el7.noarch 2/18
==> mgmt:
==> mgmt: Installing : python-ecdsa-0.11-3.el7.centos.noarch 3/18
==> mgmt:
==> mgmt: Installing : libsepol-devel-2.1.9-3.el7.x86_64 4/18
==> mgmt:
==> mgmt: Installing : pcre-devel-8.32-15.el7_2.1.x86_64 5/18
==> mgmt:
==> mgmt: Installing : libselinux-devel-2.2.2-6.el7.x86_64 6/18
==> mgmt:
==> mgmt: Installing : python-crypto-2.6.1-1.el7.centos.x86_64 7/18
==> mgmt:
==> mgmt: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 8/18
==> mgmt:
==> mgmt: Installing : python-babel-0.9.6-8.el7.noarch 9/18
==> mgmt:
==> mgmt: Installing : python-markupsafe-0.11-10.el7.x86_64 10/18
==> mgmt:
==> mgmt: Installing : libverto-devel-0.2.5-4.el7.x86_64 11/18
==> mgmt:
==> mgmt: Installing : krb5-devel-1.13.2-12.el7_2.x86_64 12/18
==> mgmt:
==> mgmt: Installing : zlib-devel-1.2.7-15.el7.x86_64 13/18
==> mgmt:
==> mgmt: Installing : 1:openssl-devel-1.0.1e-51.el7_2.5.x86_64 14/18
==> mgmt:
==> mgmt: Installing : python-jinja2-2.7.2-2.el7.noarch 15/18
==> mgmt:
==> mgmt: Installing : python-paramiko-1.12.4-1.el7.centos.noarch 16/18
==> mgmt:
==> mgmt: Installing : PyYAML-3.10-11.el7.x86_64 17/18
==> mgmt:
==> mgmt: Installing : python-devel-2.7.5-39.el7_2.x86_64 18/18
==> mgmt:
==> mgmt: Verifying : python-devel-2.7.5-39.el7_2.x86_64 1/18
==> mgmt:
==> mgmt: Verifying : zlib-devel-1.2.7-15.el7.x86_64 2/18
==> mgmt:
==> mgmt: Verifying : python-jinja2-2.7.2-2.el7.noarch 3/18
==> mgmt:
==> mgmt: Verifying : libselinux-devel-2.2.2-6.el7.x86_64 4/18
==> mgmt:
==> mgmt: Verifying : libverto-devel-0.2.5-4.el7.x86_64 5/18
==> mgmt:
==> mgmt: Verifying : python-markupsafe-0.11-10.el7.x86_64 6/18
==> mgmt:
==> mgmt: Verifying : python-paramiko-1.12.4-1.el7.centos.noarch 7/18
==> mgmt:
==> mgmt: Verifying : python-babel-0.9.6-8.el7.noarch 8/18
==> mgmt:
==> mgmt: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 9/18
==> mgmt:
==> mgmt: Verifying : python-crypto-2.6.1-1.el7.centos.x86_64 10/18
==> mgmt:
==> mgmt: Verifying : pcre-devel-8.32-15.el7_2.1.x86_64 11/18
==> mgmt:
==> mgmt: Verifying : PyYAML-3.10-11.el7.x86_64 12/18
==> mgmt:
==> mgmt: Verifying : libsepol-devel-2.1.9-3.el7.x86_64 13/18
==> mgmt:
==> mgmt: Verifying : 1:openssl-devel-1.0.1e-51.el7_2.5.x86_64 14/18
==> mgmt:
==> mgmt: Verifying : python-six-1.9.0-2.el7.noarch 15/18
==> mgmt:
==> mgmt: Verifying : libcom_err-devel-1.42.9-7.el7.x86_64 16/18
==> mgmt:
==> mgmt: Verifying : python-ecdsa-0.11-3.el7.centos.noarch 17/18
==> mgmt:
==> mgmt: Verifying : krb5-devel-1.13.2-12.el7_2.x86_64 18/18
==> mgmt:
==> mgmt:
==> mgmt: Installed:
==> mgmt: PyYAML.x86_64 0:3.10-11.el7
==> mgmt: openssl-devel.x86_64 1:1.0.1e-51.el7_2.5
==> mgmt: python-devel.x86_64 0:2.7.5-39.el7_2
==> mgmt: python-jinja2.noarch 0:2.7.2-2.el7
==> mgmt: python-paramiko.noarch 0:1.12.4-1.el7.centos
==> mgmt:
==> mgmt: Dependency Installed:
==> mgmt: keyutils-libs-devel.x86_64 0:1.5.8-3.el7
==> mgmt: krb5-devel.x86_64 0:1.13.2-12.el7_2
==> mgmt: libcom_err-devel.x86_64 0:1.42.9-7.el7
==> mgmt: libselinux-devel.x86_64 0:2.2.2-6.el7
==> mgmt: libsepol-devel.x86_64 0:2.1.9-3.el7
==> mgmt: libverto-devel.x86_64 0:0.2.5-4.el7
==> mgmt: pcre-devel.x86_64 0:8.32-15.el7_2.1
==> mgmt: python-babel.noarch 0:0.9.6-8.el7
==> mgmt: python-crypto.x86_64 0:2.6.1-1.el7.centos
==> mgmt: python-ecdsa.noarch 0:0.11-3.el7.centos
==> mgmt: python-markupsafe.x86_64 0:0.11-10.el7
==> mgmt: python-six.noarch 0:1.9.0-2.el7
==> mgmt: zlib-devel.x86_64 0:1.2.7-15.el7
==> mgmt:
==> mgmt: Complete!
==> mgmt: Collecting pip
==> mgmt: Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
==> mgmt: Collecting setuptools
==> mgmt: Downloading setuptools-27.3.0-py2.py3-none-any.whl (467kB)
==> mgmt: Collecting wheel
==> mgmt: Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
==> mgmt: Installing collected packages: pip, setuptools, wheel
==> mgmt: Successfully installed pip-8.1.2 setuptools-27.3.0 wheel-0.29.0
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): paramiko in /usr/lib/python2.7/site-packages
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): PyYAML in /usr/lib64/python2.7/site-packages
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): Jinja2 in /usr/lib/python2.7/site-packages
==> mgmt: Collecting httplib2
==> mgmt: Downloading httplib2-0.9.2.zip (210kB)
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/site-packages
==> mgmt: Collecting ansible
==> mgmt: Downloading ansible-2.1.1.0.tar.gz (1.9MB)
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): pycrypto!=2.4,>=2.1 in /usr/lib64/python2.7/site-packages (from paramiko)
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): ecdsa in /usr/lib/python2.7/site-packages (from paramiko)
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): markupsafe in /usr/lib64/python2.7/site-packages (from Jinja2)
==> mgmt: Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/site-packages (from ansible)
==> mgmt: Building wheels for collected packages: httplib2, ansible
==> mgmt: Running setup.py bdist_wheel for httplib2: started
==> mgmt: Running setup.py bdist_wheel for httplib2: finished with status 'done'
==> mgmt: Stored in directory: /root/.cache/pip/wheels/c7/67/60/e0be8ccfc1e08f8ff1f50d99ea5378e204580ea77b0169fb55
==> mgmt: Running setup.py bdist_wheel for ansible: started
==> mgmt: Running setup.py bdist_wheel for ansible: finished with status 'done'
==> mgmt: Stored in directory: /root/.cache/pip/wheels/bf/19/86/b154874da73a5a70f342ab4426b01cd18301013350bd309c1f
==> mgmt: Successfully built httplib2 ansible
==> mgmt: Installing collected packages: httplib2, ansible
==> mgmt: Successfully installed ansible-2.1.1.0 httplib2-0.9.2
==> mgmt: Running provisioner: shell...
mgmt: Running: /var/folders/98/jyl3ljxd3tzg42l38h4t55g40000gn/T/vagrant-shell20160923-16782-1akvnns.sh
✔ ~/Workspace/multinode-lab [master|✚ 2…41]
[/sourcecode]
From your host machine, login into the ‘mgmt’ node using “vagrant ssh mgmt”. Confirm that ansible is installed by running the “ansible –version” command.
To source the environment variables we want to configure – there is a script under the root directory called conf_ansible_env.sh. This is used to set the ANSIBLE_ENV and ANSIBLE_CONFIG variables. To see a list of parameters that can be specified, run the script with no parameters identified.
[sourcecode language="plain"]10:17 $ ./conf_ansible_env.sh
Set variables with the following flags
-e {{ ENVIRONMENT }}
-p {{ VAGRANT_PROVIDER }}
Example Usage: source conf_ansible_env.sh -e dev -p parallels
Current settings are:
ANSIBLE_ENV: cluster
ANSIBLE_CONFIG: /Users/moleary/Workspace/multinode-lab/ansible/vars/cluster/ansible.cfg
VAGRANT_DEFAULT_PROVIDER:[/sourcecode]
In addition, to get a list of valid parameters, use the ‘list’ command when specifying the environment or provider (see below for an example). The default environment and provider is ‘cluster’ and ‘vagrant’ respectively.
[sourcecode language="plain"]✔ ~/Workspace/multinode-lab [master|✚ 2] 11:06 $ ./conf_ansible_env.sh -e list Ansible Environment is current set to: cluster Selectable vagrant environments are - cluster - single ✔ ~/Workspace/multinode-lab [master|✚ 2] 11:06 $ ./conf_ansible_env.sh -p list Current Default Vagrant provider is not set Selectable vagrant providers are - virtualbox - parallels[/sourcecode]
Currently, two environments have been displayed in the output above. In a later blog I will expand these.
Ansible folder
Next we will add an Ansible folder. This folder will contain all the automation for our servers including roles, variables and playbooks. Ansible is described as self-documenting. The tasks in the roles and playbooks should tell you what the automation code is doing. Under the Ansible folder, I’ve created two subdirectories, “vars” and “roles”.
The “vars” folder contains all variables for the specific environment. In this case our environment is the “cluster” folder. I have also created an environment called “single” for the Splunk environment which has one combined search head / indexer (a common architecture for the sub 100 GB/day).
The “roles” contains the following for the configuration of our Splunk cluster:
- Base – The base role for common configurations across the infrastructure, for example setting up a Splunk repository
- Certificates – The certificate role is used to generate SSL certificates and configuration required to enable secure forwarding with SSL compress
- Splunk – A common Splunk role for installing a Splunk base
- Cluster Master – The Splunk role to create the cluster master
- Indexer – The Splunk role to create the indexers
- Search Head – The Splunk role to create the search heads
- Universal_Forwarder – The Splunk roles to install the universal forwarder agent and configure deployment server
- Local – A local host role that will download the Splunk binaries from their website
In addition to the roles above, I have created a repo directory which will be “rsynced” or shared (via virtualbox guest additions) to each individual server. This will be used to serve the binaries for splunk and the splunk universal forwarder as a custom yum repository. I have also added the splunk_repo.yml under the base role to create the necessary files for the repository to work. This will allow the standard “yum install splunk –y” to work as well.
We need to ensure that we have connectivity with the other nodes using the “ansible -m ping all” command. This command should be run from the root of the project folder (which is /vagrant when using the mgmt box). If everything is green then we have connectivity via Ansible (see the screenshot below for reference).
Running the “run_main_play.sh” from the root of the project folder will configure the available nodes and output the urls for the web interfaces. I have included a standard playbook run below for reference.
[sourcecode language="plain"]
[vagrant@mgmt|11:19:57| /vagrant] $ ./run_main_play.sh
Running Playbook: ansible-playbook playbook.yml --extra-vars vars_location=cluster
Using /vagrant/ansible/vars/cluster/ansible.cfg as config file
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [local : download splunk file to repo location] ***************************
ok: [localhost] => {"changed": false, "dest": "repo/splunk-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm", "gid": 1000, "group": "vagrant", "mode": "0644", "msg": "file already exists", "owner": "vagrant", "size": 222237528, "state": "file", "uid": 1000, "url": "https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.5.0&product=splunk&filename=splunk-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm&wget=true"}
TASK [local : download splunkforwarder file to repo location] ******************
ok: [localhost] => {"changed": false, "dest": "repo/splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm", "gid": 1000, "group": "vagrant", "mode": "0644", "msg": "file already exists", "owner": "vagrant", "size": 19651506, "state": "file", "uid": 1000, "url": "https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.5.0&product=universalforwarder&filename=splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm&wget=true"}
TASK [local : ensure vagrant key permission] ***********************************
changed: [localhost] => {"changed": true, "gid": 1000, "group": "vagrant", "mode": "0600", "owner": "vagrant", "path": "ansible/vars/cluster/keys/insecure_private_key", "size": 1675, "state": "file", "uid": 1000}
TASK [local : install base packages] *******************************************
ok: [localhost] => (item=[u'nano', u'createrepo']) => {"changed": false, "item": ["nano", "createrepo"], "msg": "", "rc": 0, "results": ["nano-2.3.1-10.el7.x86_64 providing nano is already installed", "createrepo-0.9.9-25.el7_2.noarch providing createrepo is already installed"]}
TASK [local : register repodata] ***********************************************
ok: [localhost] => {"changed": false, "stat": {"atime": 1475498286.0, "checksum": "af07e5655c2e20e2e00357c1f099271b2eafbd93", "ctime": 1475497745.0, "dev": 35, "exists": true, "gid": 1000, "gr_name": "vagrant", "inode": 69, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "md5": "f5821a10edd90882fa37001542ea01ce", "mode": "0644", "mtime": 1475497745.0, "nlink": 1, "path": "repo/repodata/repomd.xml", "pw_name": "vagrant", "rgrp": true, "roth": true, "rusr": true, "size": 2975, "uid": 1000, "wgrp": false, "woth": false, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}}
TASK [local : register repodata mtime] *****************************************
ok: [localhost] => {
"msg": "1475497745"
}
TASK [local : register current time] *******************************************
ok: [localhost] => {
"msg": "1475454004"
}
TASK [local : create splunk repo] **********************************************
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional check failed", "skipped": true}
PLAY [standalone] **************************************************************
skipping: no hosts matched
PLAY [cluster_master] **********************************************************
TASK [setup] *******************************************************************
ok: [clm01]
TASK [base : register splunk repo] *********************************************
ok: [clm01] => {"changed": false, "stat": {"exists": false}}
TASK [base : add a local splunk repository] ************************************
changed: [clm01] => {"changed": true, "repo": "splunk-repo", "state": "present"}
TASK [splunk : install base packages] ******************************************
changed: [clm01] => (item=[u'splunk']) => {"changed": true, "item": ["splunk"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.crucial.com.au\n * extras: centos.uberglobalmirror.com\n * updates: centos.uberglobalmirror.com\nResolving Dependencies\n--> Running transaction check\n---> Package splunk.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunk x86_64 6.5.0-59c8927def0f splunk-repo 212 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 212 M\nInstalled size: 506 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunk-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunk-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunk.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
TASK [splunk : accept splunk license] ******************************************
changed: [clm01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:02.073424", "end": "2016-10-04 11:22:31.951091", "rc": 0, "start": "2016-10-04 11:22:29.877667", "stderr": "Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.\nGenerating RSA private key, 1024 bit long modulus\n.......++++++\n........++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nGenerating RSA private key, 1024 bit long modulus\n.................++++++\n.........++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nMoving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
TASK [splunk : create splunk .ui_login file] ***********************************
changed: [clm01] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunk/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540552.44-51742252874333/source", "state": "file", "uid": 1001}
TASK [splunk : start the splunk service] ***************************************
ok: [clm01] => {"changed": false, "enabled": true, "name": "splunk"}
TASK [certificates : creates inputs application default folder] ****************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/master-apps/secure_inputs/default", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : creates inputs application local folder] ******************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/master-apps/secure_inputs/local", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : creates inputs application certs folder] ******************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/master-apps/secure_inputs/certs", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : generate cakey] *******************************************
changed: [clm01] => {"changed": true, "cmd": ["openssl", "req", "-newkey", "rsa:1024", "-passout", "pass:password", "-subj", "/countryName=AU/stateOrProvinceName=NSW/localityName=Sydney/organizationName=ams@avocado.com.au/commonName=avocadoCA/", "-sha1", "-keyout", "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cakey.pem", "-out", "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_careq.pem"], "delta": "0:00:00.031784", "end": "2016-10-04 11:22:35.310123", "rc": 0, "start": "2016-10-04 11:22:35.278339", "stderr": "Generating a 1024 bit RSA private key\n......++++++\n.......++++++\nwriting new private key to '/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cakey.pem'\n-----", "stdout": "", "stdout_lines": [], "warnings": []}
TASK [certificates : generate cacert] ******************************************
changed: [clm01] => {"changed": true, "cmd": ["openssl", "x509", "-req", "-in", "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_careq.pem", "-passin", "pass:password", "-sha1", "-extensions", "v3_ca", "-signkey", "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cakey.pem", "-out", "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cacert.pem", "-days", "3650"], "delta": "0:00:00.034192", "end": "2016-10-04 11:22:35.785412", "rc": 0, "start": "2016-10-04 11:22:35.751220", "stderr": "Signature ok\nsubject=/C=AU/ST=NSW/L=Sydney/O=ams@avocado.com.au/CN=avocadoCA\nGetting Private key", "stdout": "", "stdout_lines": [], "warnings": []}
TASK [certificates : create private ca] ****************************************
changed: [clm01] => {"changed": true, "cmd": "cat /opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cacert.pem /opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cakey.pem > /opt/splunk/etc/master-apps/secure_inputs/certs/avocado_ca.pem", "delta": "0:00:00.015028", "end": "2016-10-04 11:22:36.301188", "rc": 0, "start": "2016-10-04 11:22:36.286160", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
TASK [certificates : create private ca] ****************************************
changed: [clm01] => {"changed": true, "cmd": "cp /opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cacert.pem /opt/splunk/etc/master-apps/secure_inputs/certs/cacert.pem", "delta": "0:00:00.006267", "end": "2016-10-04 11:22:36.843646", "rc": 0, "start": "2016-10-04 11:22:36.837379", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
TASK [certificates : create private ca] ****************************************
changed: [clm01] => {"changed": true, "cmd": "cp /opt/splunk/etc/master-apps/secure_inputs/certs/avocado_ca.pem /opt/splunk/etc/master-apps/secure_inputs/certs/ca.pem", "delta": "0:00:00.007561", "end": "2016-10-04 11:22:37.425731", "rc": 0, "start": "2016-10-04 11:22:37.418170", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
TASK [certificates : generate server certificate] ******************************
changed: [clm01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "createssl", "server-cert", "-d", "/opt/splunk/etc/master-apps/secure_inputs/certs/", "-n", "avocado_server", "-c", "avocadoCA"], "delta": "0:00:00.055693", "end": "2016-10-04 11:22:38.019271", "rc": 0, "start": "2016-10-04 11:22:37.963578", "stderr": "\n* Create certificate avocado_server.pem signed by the root CA.\n* Store the avocado_server.pem key file locally with your client/server application.\n* Enter a secret passphrase when requested.\n* The passphrase is used to access avocado_server.pem in your application.\n* Enter the application's hostname as the Common Name when requested.\n* Enter the root CA passphrase (Getting CA Private Key) to sign the keyfile.\n* The keyfile will expire after one year or sooner if the root CA expires.\n\nGenerating a 1024 bit RSA private key\n....................................++++++\n.........++++++\nwriting new private key to 'avocado_serverkey.pem'\n-----\nSignature ok\nsubject=/CN=avocadoCA/O=SplunkUser\nGetting CA Private Key", "stdout": "subject= /CN=avocadoCA/O=SplunkUser\nissuer= /C=AU/ST=NSW/L=Sydney/O=ams@avocado.com.au/CN=avocadoCA\nnotBefore=Oct 4 00:22:38 2016 GMT\nnotAfter=Oct 4 00:22:38 2019 GMT", "stdout_lines": ["subject= /CN=avocadoCA/O=SplunkUser", "issuer= /C=AU/ST=NSW/L=Sydney/O=ams@avocado.com.au/CN=avocadoCA", "notBefore=Oct 4 00:22:38 2016 GMT", "notAfter=Oct 4 00:22:38 2019 GMT"], "warnings": []}
TASK [certificates : deployment-apps - create local folder] ********************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/deployment-apps/secure_outputs/local", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : deployment-apps - create default folder] ******************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/deployment-apps/secure_outputs/default", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : deployment-apps - create certs folder] ********************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/deployment-apps/secure_outputs/certs", "size": 6, "state": "directory", "uid": 1001}
TASK [certificates : copy certs ca certificate] ********************************
changed: [clm01] => {"changed": true, "checksum": "d358f9264889dc60048b63c08e1954a852b76a7a", "dest": "/opt/splunk/etc/deployment-apps/secure_outputs/certs/avocado_cacert.pem", "gid": 1001, "group": "splunk", "md5sum": "3c8bb1e4c8d82e081ff4661650b5c5b2", "mode": "0600", "owner": "splunk", "size": 847, "src": "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_cacert.pem", "state": "file", "uid": 1001}
TASK [certificates : copy server certificate] **********************************
changed: [clm01] => {"changed": true, "checksum": "ebf4a244d22d1de11c47d07acafeeaf95cfbe975", "dest": "/opt/splunk/etc/deployment-apps/secure_outputs/certs/avocado_server.pem", "gid": 1001, "group": "splunk", "md5sum": "0069435278e6f5c2db1a58553b2ec2a1", "mode": "0600", "owner": "splunk", "size": 2653, "src": "/opt/splunk/etc/master-apps/secure_inputs/certs/avocado_server.pem", "state": "file", "uid": 1001}
TASK [certificates : create forwarder outputs - outputs.conf] ******************
changed: [clm01] => {"changed": true, "checksum": "be96824241becd7c5d927f4ce030c29898e02638", "dest": "/opt/splunk/etc/deployment-apps/secure_outputs/local/outputs.conf", "gid": 1001, "group": "splunk", "md5sum": "9cb8630ef2aa5b6b3d198c1807566d7d", "mode": "0600", "owner": "splunk", "size": 331, "src": "/tmp/ansible-tmp-1475540560.61-280074172851078/source", "state": "file", "uid": 1001}
TASK [certificates : create tcp inputs - inputs.conf] **************************
changed: [clm01] => {"changed": true, "checksum": "ce22755a6de884211cfcee99a527c46c888b8653", "dest": "/opt/splunk/etc/master-apps/secure_inputs/local/inputs.conf", "gid": 1001, "group": "splunk", "md5sum": "d1d063044bc404f5ac3fb98652739a71", "mode": "0600", "owner": "splunk", "size": 246, "src": "/tmp/ansible-tmp-1475540561.52-104303611872718/source", "state": "file", "uid": 1001}
TASK [certificates : create forwarder outputs - app.conf] **********************
changed: [clm01] => {"changed": true, "checksum": "73bc37be4f0c38d60cd1106a6c5b6249760cc79f", "dest": "/opt/splunk/etc/deployment-apps/secure_outputs/local/app.conf", "gid": 1001, "group": "splunk", "md5sum": "5b057fef5a57f6ad0038315a28ae67a5", "mode": "0644", "owner": "splunk", "size": 81, "src": "/tmp/ansible-tmp-1475540562.29-272685057316110/source", "state": "file", "uid": 1001}
TASK [certificates : create serverclass - serverclass.conf] ********************
changed: [clm01] => {"changed": true, "checksum": "32f13582052a23c1d5f592edab7f4180e6442c68", "dest": "/opt/splunk/etc/system/local/serverclass.conf", "gid": 1001, "group": "splunk", "md5sum": "7631cf7f396ae35dd7800a4441de99c4", "mode": "0644", "owner": "splunk", "size": 155, "src": "/tmp/ansible-tmp-1475540563.08-247826531728114/source", "state": "file", "uid": 1001}
TASK [cluster_master : create the cluster master directory] ********************
changed: [clm01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/apps/cluster_config/local", "size": 6, "state": "directory", "uid": 1001}
TASK [cluster_master : create server.conf from template] ***********************
changed: [clm01] => {"changed": true, "checksum": "9ae7358f3bc95a3ecf0beb9d24c66dea6df1155f", "dest": "/opt/splunk/etc/apps/cluster_config/local/server.conf", "gid": 1001, "group": "splunk", "md5sum": "13a252e8569b3f17f6923f9312faa74d", "mode": "0600", "owner": "splunk", "size": 223, "src": "/tmp/ansible-tmp-1475540564.48-2073762586893/source", "state": "file", "uid": 1001}
RUNNING HANDLER [splunk : splunk restart] **************************************
changed: [clm01] => {"changed": true, "name": "splunk", "state": "started"}
RUNNING HANDLER [splunk : display splunk url] **********************************
ok: [clm01] => {
"msg": "Splunk host clm01 is available @ http://clm01.cluster.avocado.lab:8000"
}
RUNNING HANDLER [certificates : apply cluster bundle] **************************
changed: [clm01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "apply", "cluster-bundle", "-auth", "admin:changeme"], "delta": "0:00:00.970083", "end": "2016-10-04 11:23:03.811567", "rc": 0, "start": "2016-10-04 11:23:02.841484", "stderr": "No new bundle will be pushed. The master and peers already have this bundle with bundle id = EA8D7A8169ED63C074E8E880DBD51081", "stdout": "\nEncountered some errors while applying the bundle.", "stdout_lines": ["", "Encountered some errors while applying the bundle."], "warnings": []}
RUNNING HANDLER [certificates : reload deploy-server] **************************
changed: [clm01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "reload", "deploy-server", "-auth", "admin:changeme"], "delta": "0:00:01.106605", "end": "2016-10-04 11:23:05.625477", "rc": 0, "start": "2016-10-04 11:23:04.518872", "stderr": "", "stdout": "Reloading serverclass(es).", "stdout_lines": ["Reloading serverclass(es)."], "warnings": []}
PLAY [indexer] *****************************************************************
TASK [setup] *******************************************************************
ok: [idx01]
ok: [idx02]
TASK [base : register splunk repo] *********************************************
ok: [idx01] => {"changed": false, "stat": {"exists": false}}
ok: [idx02] => {"changed": false, "stat": {"exists": false}}
TASK [base : add a local splunk repository] ************************************
changed: [idx01] => {"changed": true, "repo": "splunk-repo", "state": "present"}
changed: [idx02] => {"changed": true, "repo": "splunk-repo", "state": "present"}
TASK [splunk : install base packages] ******************************************
changed: [idx01] => (item=[u'splunk']) => {"changed": true, "item": ["splunk"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.crucial.com.au\n * extras: mirror.as24220.net\n * updates: mirror.overthewire.com.au\nResolving Dependencies\n--> Running transaction check\n---> Package splunk.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunk x86_64 6.5.0-59c8927def0f splunk-repo 212 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 212 M\nInstalled size: 506 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunk-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunk-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunk.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
changed: [idx02] => (item=[u'splunk']) => {"changed": true, "item": ["splunk"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.digitalpacific.com.au\n * extras: centos.mirror.digitalpacific.com.au\n * updates: centos.mirror.digitalpacific.com.au\nResolving Dependencies\n--> Running transaction check\n---> Package splunk.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunk x86_64 6.5.0-59c8927def0f splunk-repo 212 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 212 M\nInstalled size: 506 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunk-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunk-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunk.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
TASK [splunk : accept splunk license] ******************************************
changed: [idx01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:02.471908", "end": "2016-10-04 11:24:41.659922", "rc": 0, "start": "2016-10-04 11:24:39.188014", "stderr": "Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.\nGenerating RSA private key, 1024 bit long modulus\n..................++++++\n..............++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nGenerating RSA private key, 1024 bit long modulus\n.......++++++\n...................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nMoving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
changed: [idx02] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:02.617275", "end": "2016-10-04 11:24:41.792160", "rc": 0, "start": "2016-10-04 11:24:39.174885", "stderr": "Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.\nGenerating RSA private key, 1024 bit long modulus\n.............................++++++\n..................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nGenerating RSA private key, 1024 bit long modulus\n..............++++++\n...............................................................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nMoving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
TASK [splunk : create splunk .ui_login file] ***********************************
changed: [idx01] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunk/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540682.35-228492877190264/source", "state": "file", "uid": 1001}
changed: [idx02] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunk/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540682.47-262440568887541/source", "state": "file", "uid": 1001}
TASK [splunk : start the splunk service] ***************************************
ok: [idx01] => {"changed": false, "enabled": true, "name": "splunk"}
ok: [idx02] => {"changed": false, "enabled": true, "name": "splunk"}
TASK [indexer : create the cluster master directory] ***************************
changed: [idx01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/apps/cluster_config/local", "size": 6, "state": "directory", "uid": 1001}
changed: [idx02] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/apps/cluster_config/local", "size": 6, "state": "directory", "uid": 1001}
TASK [indexer : create server.conf from template] ******************************
changed: [idx01] => {"changed": true, "checksum": "0211e672dbb59565e96125d969ba86eda6b72afe", "dest": "/opt/splunk/etc/apps/cluster_config/local/server.conf", "gid": 1001, "group": "splunk", "md5sum": "8d9a1078a84d63b2bff2c318ce01d3b3", "mode": "0600", "owner": "splunk", "size": 166, "src": "/tmp/ansible-tmp-1475540684.06-83996154303155/source", "state": "file", "uid": 1001}
changed: [idx02] => {"changed": true, "checksum": "0211e672dbb59565e96125d969ba86eda6b72afe", "dest": "/opt/splunk/etc/apps/cluster_config/local/server.conf", "gid": 1001, "group": "splunk", "md5sum": "8d9a1078a84d63b2bff2c318ce01d3b3", "mode": "0600", "owner": "splunk", "size": 166, "src": "/tmp/ansible-tmp-1475540684.08-12230920895407/source", "state": "file", "uid": 1001}
RUNNING HANDLER [splunk : splunk restart] **************************************
changed: [idx01] => {"changed": true, "name": "splunk", "state": "started"}
changed: [idx02] => {"changed": true, "name": "splunk", "state": "started"}
RUNNING HANDLER [splunk : display splunk url] **********************************
ok: [idx01] => {
"msg": "Splunk host idx01 is available @ http://idx01.cluster.avocado.lab:8000"
}
ok: [idx02] => {
"msg": "Splunk host idx02 is available @ http://idx02.cluster.avocado.lab:8000"
}
PLAY [search_head] *************************************************************
TASK [setup] *******************************************************************
ok: [shd01]
ok: [shd02]
TASK [base : register splunk repo] *********************************************
ok: [shd01] => {"changed": false, "stat": {"exists": false}}
ok: [shd02] => {"changed": false, "stat": {"exists": false}}
TASK [base : add a local splunk repository] ************************************
changed: [shd01] => {"changed": true, "repo": "splunk-repo", "state": "present"}
changed: [shd02] => {"changed": true, "repo": "splunk-repo", "state": "present"}
TASK [splunk : install base packages] ******************************************
changed: [shd02] => (item=[u'splunk']) => {"changed": true, "item": ["splunk"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.crucial.com.au\n * extras: mirror.ventraip.net.au\n * updates: mirror.ventraip.net.au\nResolving Dependencies\n--> Running transaction check\n---> Package splunk.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunk x86_64 6.5.0-59c8927def0f splunk-repo 212 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 212 M\nInstalled size: 506 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunk-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunk-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunk.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
changed: [shd01] => (item=[u'splunk']) => {"changed": true, "item": ["splunk"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.digitalpacific.com.au\n * extras: centos.mirror.digitalpacific.com.au\n * updates: centos.mirror.digitalpacific.com.au\nResolving Dependencies\n--> Running transaction check\n---> Package splunk.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunk x86_64 6.5.0-59c8927def0f splunk-repo 212 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 212 M\nInstalled size: 506 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunk-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunk-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunk.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
TASK [splunk : accept splunk license] ******************************************
changed: [shd01] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:03.049325", "end": "2016-10-04 11:26:52.695604", "rc": 0, "start": "2016-10-04 11:26:49.646279", "stderr": "Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.\nGenerating RSA private key, 1024 bit long modulus\n......................................++++++\n....++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nGenerating RSA private key, 1024 bit long modulus\n........++++++\n.......................................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nMoving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
changed: [shd02] => {"changed": true, "cmd": ["/opt/splunk/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:03.024069", "end": "2016-10-04 11:26:52.753667", "rc": 0, "start": "2016-10-04 11:26:49.729598", "stderr": "Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.\nGenerating RSA private key, 1024 bit long modulus\n...................++++++\n......................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nGenerating RSA private key, 1024 bit long modulus\n...............................................++++++\n.........................................++++++\nunable to write 'random state'\ne is 65537 (0x10001)\nwriting RSA key\n\nMoving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
TASK [splunk : create splunk .ui_login file] ***********************************
changed: [shd01] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunk/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540813.44-76300626625863/source", "state": "file", "uid": 1001}
changed: [shd02] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunk/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540813.49-149962141723613/source", "state": "file", "uid": 1001}
TASK [splunk : start the splunk service] ***************************************
ok: [shd01] => {"changed": false, "enabled": true, "name": "splunk"}
ok: [shd02] => {"changed": false, "enabled": true, "name": "splunk"}
TASK [search_head : create the cluster master directory] ***********************
changed: [shd01] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/apps/cluster_config/local", "size": 6, "state": "directory", "uid": 1001}
changed: [shd02] => {"changed": true, "gid": 1001, "group": "splunk", "mode": "0755", "owner": "splunk", "path": "/opt/splunk/etc/apps/cluster_config/local", "size": 6, "state": "directory", "uid": 1001}
TASK [search_head : create server.conf from template] **************************
changed: [shd01] => {"changed": true, "checksum": "00f4c5953a093275c854c07a02a872001a2e7f71", "dest": "/opt/splunk/etc/apps/cluster_config/local/server.conf", "gid": 1001, "group": "splunk", "md5sum": "c1b1f8cad37ea60f63790cc12a8e3128", "mode": "0600", "owner": "splunk", "size": 144, "src": "/tmp/ansible-tmp-1475540815.83-247325310928993/source", "state": "file", "uid": 1001}
changed: [shd02] => {"changed": true, "checksum": "00f4c5953a093275c854c07a02a872001a2e7f71", "dest": "/opt/splunk/etc/apps/cluster_config/local/server.conf", "gid": 1001, "group": "splunk", "md5sum": "c1b1f8cad37ea60f63790cc12a8e3128", "mode": "0600", "owner": "splunk", "size": 144, "src": "/tmp/ansible-tmp-1475540815.85-212679712130086/source", "state": "file", "uid": 1001}
RUNNING HANDLER [splunk : splunk restart] **************************************
changed: [shd01] => {"changed": true, "name": "splunk", "state": "started"}
changed: [shd02] => {"changed": true, "name": "splunk", "state": "started"}
RUNNING HANDLER [splunk : display splunk url] **********************************
ok: [shd01] => {
"msg": "Splunk host shd01 is available @ http://shd01.cluster.avocado.lab:8000"
}
ok: [shd02] => {
"msg": "Splunk host shd02 is available @ http://shd02.cluster.avocado.lab:8000"
}
PLAY [universal_forwarder] *****************************************************
TASK [setup] *******************************************************************
ok: [ufw01]
TASK [base : register splunk repo] *********************************************
ok: [ufw01] => {"changed": false, "stat": {"exists": false}}
TASK [base : add a local splunk repository] ************************************
changed: [ufw01] => {"changed": true, "repo": "splunk-repo", "state": "present"}
TASK [universal_forwarder : install base packages] *****************************
changed: [ufw01] => (item=[u'splunkforwarder']) => {"changed": true, "item": ["splunkforwarder"], "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.crucial.com.au\n * extras: centos.mirror.serversaustralia.com.au\n * updates: centos.mirror.serversaustralia.com.au\nResolving Dependencies\n--> Running transaction check\n---> Package splunkforwarder.x86_64 0:6.5.0-59c8927def0f will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n splunkforwarder x86_64 6.5.0-59c8927def0f splunk-repo 19 M\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 19 M\nInstalled size: 48 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : splunkforwarder-6.5.0-59c8927def0f.x86_64 1/1 \ncomplete\n Verifying : splunkforwarder-6.5.0-59c8927def0f.x86_64 1/1 \n\nInstalled:\n splunkforwarder.x86_64 0:6.5.0-59c8927def0f \n\nComplete!\n"]}
TASK [universal_forwarder : accept splunk license] *****************************
changed: [ufw01] => {"changed": true, "cmd": ["/opt/splunkforwarder/bin/splunk", "enable", "boot-start", "--accept-license", "-user", "splunk"], "delta": "0:00:00.535988", "end": "2016-10-04 11:28:27.425451", "rc": 0, "start": "2016-10-04 11:28:26.889463", "stderr": "", "stdout": "\nThis appears to be your first time running this version of Splunk.\nInit script installed at /etc/init.d/splunk.\nInit script is configured to run at boot.", "stdout_lines": ["", "This appears to be your first time running this version of Splunk.", "Init script installed at /etc/init.d/splunk.", "Init script is configured to run at boot."], "warnings": []}
TASK [universal_forwarder : create splunk .ui_login file] **********************
changed: [ufw01] => {"changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/opt/splunkforwarder/etc/.ui_login", "gid": 1001, "group": "splunk", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "splunk", "size": 0, "src": "/tmp/ansible-tmp-1475540907.91-8888942851973/source", "state": "file", "uid": 1001}
TASK [universal_forwarder : create forwarder deploymentclient configuration - deploymentclient.conf] ***
changed: [ufw01] => {"changed": true, "checksum": "c92df3aa337edae28bb7cd13a62b1751ae18285d", "dest": "/opt/splunkforwarder/etc/system/local/deploymentclient.conf", "gid": 1001, "group": "splunk", "md5sum": "36d00b8f2ec24221c4b4c13dddfb0f54", "mode": "0644", "owner": "splunk", "size": 97, "src": "/tmp/ansible-tmp-1475540908.38-228775467556006/source", "state": "file", "uid": 1001}
TASK [universal_forwarder : start the splunk service] **************************
ok: [ufw01] => {"changed": false, "enabled": true, "name": "splunk"}
RUNNING HANDLER [universal_forwarder : splunk restart] *************************
changed: [ufw01] => {"changed": true, "name": "splunk", "state": "started"}
RUNNING HANDLER [universal_forwarder : display splunk url] *********************
ok: [ufw01] => {
"msg": "Splunk host ufw01 is available @ http://ufw01.cluster.avocado.lab:8000"
}
PLAY RECAP *********************************************************************
clm01 : ok=31 changed=27 unreachable=0 failed=0
idx01 : ok=11 changed=7 unreachable=0 failed=0
idx02 : ok=11 changed=7 unreachable=0 failed=0
localhost : ok=8 changed=1 unreachable=0 failed=0
shd01 : ok=11 changed=7 unreachable=0 failed=0
shd02 : ok=11 changed=7 unreachable=0 failed=0
ufw01 : ok=10 changed=6 unreachable=0 failed=0
[vagrant@mgmt|11:28:31| /vagrant] $[/sourcecode]
I hope these posts have assisted you in the basic architecture. As part of Avocado’s daily consulting practice, Avocado engineers have an in depth experience of working with clients to build multiple environments. If you would like to discuss this further please get in touch through hello@avocado.com.au
