Logstash & Graphite in few words

Logstash is a tool for managing your logs. It helps you take logs and other event data from your systems and move it into a central place. logstash is a community driven open source project and completely free. You can find support on the mailing list and on IRC.


Logstash is not a Saas solution and you have to host it. It requires Java run-time for its components, it uses Lucene on the back-end which is written in JRuby. There is no dependencies in JRuby and Lucene since those are shipped with Logstash code.

 

Required service components: logstash - open source log management

1. Redis instance server
2. Log shippers servers
3. ElasticSearch servers:

 

Logstash requires specific configuration for each component in the system such as log shippers, log parsers and etc ... this makes Logstash very flexible at the same time complicated due to required configurations.
There are Chef cookbooks and Puppet modules available for automated installation: the logstash community cookbook - logstash cookbook


Assessment:

High customization, complex architecture to maintain,

requires hardware to host

requires administration : There are admin positions for Logstash system

Logstash has plugin for Graphite which is a log graphing tool

 

 

What is Graphite?

Graphite is a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested in graphing, and send it to Graphite's processing backend,carbon, which stores the data in Graphite's specialized database. The data can then be visualized through graphite's web interfaces.

 

http://graphite.wikidot.com/faq#toc1

Splunk Storm

Splunk Storm is a cloud-based service that turns machine data into valuable insights. Machine data is generated by web sites, applications, servers, networks, mobile devices, and the like. Splunk Storm consumes machine data and allows users to search and visualize it to monitor and analyze everything from customer clickstreams and transactions to network activity to call records.

Features

1- Multiple log support

Ruby,Java,IIS,apache, syslog and etc

2- Sending logs

1- Upload files
2- Stream data over network
3- Log from application to Splunk REST api
4- tail files using Splunk forwarders

3- Search and trobleshoot

1. Powerful data search: Simple query options as well as custom query language which is very powerful at the same time intuitive and easy to understand

4- Visualize data
5- Share

1. dashboards and reports
2. Export data

6- Pricing

• Free
  • Store up to 1 GB
  • Access to all features
  • Community support
  • No expiring trials
  • No hidden charges
  • No credit card required
• Monnthly plan
  • 2GB $20.00, 5GB $30.00,20GB $80.00
  • Guaranteed response time for reported issues
  • Phone support
    

Splunk Manual

Few useful links that would apply to Galen usage of Splunk Storm:

1. About Logging:
   

   • http://docs.splunk.com/Documentation/Storm/Storm/User/Aboutlogging
   • http://docs.splunk.com/Documentation/Storm/latest/User/Examples
   • http://docs.splunk.com/Documentation/Storm/latest/User/Bestpractices
2. About forwarding data to Storm
   
   • http://docs.splunk.com/Documentation/Storm/latest/User/AboutforwardingdatatoStorm
3. Set up a universal forwarder on *nix:
   
   • http://docs.splunk.com/Documentation/Storm/Storm/User/Setupauniversalforwarderonnix