Setting Up Splunk Free Edition on Ubuntu Server 10.04, and Securing it With an Apache Proxy and iptables

Whether you're managing one device or 1,000, Splunk (http://www.splunk.com/) is a useful product as it allows you to aggregate and search diagnostic information from a variety of systems.  At CalPOP we use it as a central syslog server, allowing us to view the logs of our several hundred Cisco and Juniper switches and other infrastructure elements in one central place, search for specific events, and build reports and dashboards to track performance.   If you're operating on a small scale, you can use the Free Edition of Splunk, which allows you to index up to 500 MB of data per day.   The Free Edition will likely cover you until your environment reaches enterprise-scale (think hundreds or thousands of servers), at which time Splunk will be more than happy to take your money.

The Free Edition of Splunk has one irritating drawback, however: it lacks any form of built in user account management or authentication.  We will (partially) address that shortcoming in the course of this tutorial.

First, download Splunk from their website, and upload it to the home directory your server.  If you're running Ubuntu, which is what we use for our infrastructure within CalPOP, you can use a .deb package.  There is also an .rpm for distros like CentOS and Fedora, and a tarball for everyone else.   Once you've uploaded it, if you're on Ubuntu (and presumably Debian, although I've yet to try it on that much-venerated distribution), run this command:

dpkg install splunk*.deb

Next, start Splunk: 

/opt/splunk/bin/splunk start

Now, enter http://your-server:8000 (where your-server is obviously the hostname or IP address of your box), and you'll enter the web interface.   You might well poke around for a bit.  After you've had your fill, and converted the license to the Free Edition (see the Splunk installation documentation for instructions on how to do this, by default, it runs as a 30 day trial of the paid version), log out again.   When you return to the site, you might notice, to your alarm, that it lets you straight in, without prompting for so much as a password to keep curious visitors away.   This naturally may pose some concern for anyone who wants to run Splunk on a world-accessible box; you certainly do not want just anyone to be able to search through your syslogs, or other diagnostics data indexed by Splunk. 

Fortunately, we can protect Splunk with an Apache proxy.

First, we need to install Apache, if is not installed already:

apt-get install apache2

Next, we must enable certain proxy modules:

a2enmod proxy

a2enmod proxy_html

Now, create a site for it in /etc/apache2/sites-enabled (you will want to point a domain name at the server; here I'll use splunk.example.com). 

cat > /etc/apache2/splunk.example.com

Paste in the following:

<virtualhost *:80>
ServerName splunk.example.com

ProxyPass / http://127.0.0.1:8000/
        ProxyPassReverse / http://127.0.0.1:8000/
</virtualhost>

<proxy http://127.0.0.1:8000/*>
        Order deny,allow
        Deny from all
        Allow from all
 AuthName "splunk.example.com"
        AuthType Basic
        AuthUserFile /var/www/.htpasswd
        Require valid-user
</proxy>

Go CTRL+D to save the file, and then check that its ok.

cat /etc/apache2/splunk.example.com

You should see the exact text you just pasted in above.

Now let's cd to /var/www/ and run the htpasswd command to create a password:

htpasswd -c .htpasswd username

Where username is of course, the username you wish to create.  It will prompt you to enter a password twice, which will not echo, just like using passwd(1).

With this configuration now behind us, it's time to fire up the web server:

service apache2 start

Now, go to http://your-server in a web browser (omit the :8000) and you'll be prompted for a user name and password.  Enter the ones you just created, et voila; Splunk awaits.  Of course, do not be deluded into thinking this is secure; cretinous characters of cyberspace can make an end-run around this new authentication system of yours by going directly to port 8000 in their browser.   Concordantly, we must block it in iptables:

iptables -A INPUT -s 127.0.0.1/32 -p tcp -m tcp --dport 8000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 8000 -j DROP

The first rule is neccessary to allow Apache to access Splunk for purposes of providing a proxy; the second rule serves to keep everyone else out, forcing them to authenticate on port 80 (rather than accessing Splunk directly on 8000).   Be sure to save these firewall rules so they persist, in the event your box is rebooted.

Note that this does not provide you with any user separation; even if you define multiple user accounts via .htpasswd, they will all have the same level of authentication once they login.   If you require fine-grained permissions, you will need to license the paid version of Splunk.