Logstash & Graphite in few words

Logstash is a tool for managing your logs. It helps you take logs and other event data from your systems and move it into a central place. logstash is a community driven open source project and completely free. You can find support on the mailing list and on IRC.


Logstash is not a Saas solution and you have to host it. It requires Java run-time for its components, it uses Lucene on the back-end which is written in JRuby. There is no dependencies in JRuby and Lucene since those are shipped with Logstash code.

 

Required service components: logstash - open source log management

1. Redis instance server
2. Log shippers servers
3. ElasticSearch servers:

 

Logstash requires specific configuration for each component in the system such as log shippers, log parsers and etc ... this makes Logstash very flexible at the same time complicated due to required configurations.
There are Chef cookbooks and Puppet modules available for automated installation: the logstash community cookbook - logstash cookbook


Assessment:

High customization, complex architecture to maintain,

requires hardware to host

requires administration : There are admin positions for Logstash system

Logstash has plugin for Graphite which is a log graphing tool

 

 

What is Graphite?

Graphite is a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested in graphing, and send it to Graphite's processing backend,carbon, which stores the data in Graphite's specialized database. The data can then be visualized through graphite's web interfaces.

 

http://graphite.wikidot.com/faq#toc1

Splunk Storm

Splunk Storm is a cloud-based service that turns machine data into valuable insights. Machine data is generated by web sites, applications, servers, networks, mobile devices, and the like. Splunk Storm consumes machine data and allows users to search and visualize it to monitor and analyze everything from customer clickstreams and transactions to network activity to call records.

Features

1- Multiple log support

Ruby,Java,IIS,apache, syslog and etc

2- Sending logs

1- Upload files
2- Stream data over network
3- Log from application to Splunk REST api
4- tail files using Splunk forwarders

3- Search and trobleshoot

1. Powerful data search: Simple query options as well as custom query language which is very powerful at the same time intuitive and easy to understand

4- Visualize data
5- Share

1. dashboards and reports
2. Export data

6- Pricing

• Free
  • Store up to 1 GB
  • Access to all features
  • Community support
  • No expiring trials
  • No hidden charges
  • No credit card required
• Monnthly plan
  • 2GB $20.00, 5GB $30.00,20GB $80.00
  • Guaranteed response time for reported issues
  • Phone support
    

Splunk Manual

Few useful links that would apply to Galen usage of Splunk Storm:

1. About Logging:
   

   • http://docs.splunk.com/Documentation/Storm/Storm/User/Aboutlogging
   • http://docs.splunk.com/Documentation/Storm/latest/User/Examples
   • http://docs.splunk.com/Documentation/Storm/latest/User/Bestpractices
2. About forwarding data to Storm
   
   • http://docs.splunk.com/Documentation/Storm/latest/User/AboutforwardingdatatoStorm
3. Set up a universal forwarder on *nix:
   
   • http://docs.splunk.com/Documentation/Storm/Storm/User/Setupauniversalforwarderonnix

Notes on Kanban

Below are my notes on Kanban.

What is Kanban?

It's a Japanese word, it literally means "signboard" or "billboard," it is a concept related to lean and just-in-time (JIT) production. According to its creator, Taiichi Ohno, Kanban is one means through which JIT is achieved, he introduced this model for Toyota in mid 1920's. 
Kanban in software engineering is a flavor of Scrum methodology, in better words it is considered a leaner flavor of Scrum.

Why Kanban?

Implementing it can result in:

- Improved quality of work

- Faster turnaround of work requests

- Identification and elimination of bottlenecks

- Reduction of time work spends in queues

- Improved teamwork

- Reduction of wasted effort

How Kanban works?

Two most important Principles of Kanban are to:

       1- Visualize your work: Providing a means of visualizing work, allows you and others know what are you doing, What should I be working on right now?

       2- Limit work in process(WIP), which in turn:

1. reduces waste

2. improves quality

3. improves flow:  in Kanban Work processes should be optimized to maximize flow not capacity utilization! 100% utilization kills flow! An example of maximized capacity is a 100% CPU, which we all know it's not an good situation to be at.

                                

                                Little's Law shows that response time improves directly with reduction in WIP

               

                                Little's law Arrival rate=throughput

                                Length of queue=Arrival rate * average wait time

                                wait time= Length of Queue/Arrival rate

                                cycle time= WIP/throughput

                                Cycle time starts when work begins on the request and end when the item is ready for  delivery.

                                Lead time starts at feature request and ends when it is delivered

               

Team Kanban 

 Recipe for Success 

1- Focus on Quality

2- Reduce Work-in-Process 

3- Deliver often

1. Reduced WIP, shortens Lead Time
2. Shorter Lead Times allow for more frequent releases
3. Frequent releases build trust!!! which is very important!

4- Balance Demand Against Throughput  

1.      Not accepting new work at a rate that exceeds the rate that team produces. This will reveal bottle-necks activities in the process. 
2.       You need slack  in order to improve or Kaizen , 100% utilization kills flow

5- Prioritize

6- Attack source of variability to improve predictability

  

References:

http://leanandkanban.wordpress.com/

http://www.youtube.com/watch?v=5_lUo8u_FcI&feature=plcp 

http://jamesshore.com/Blog/Kanban-Systems.html

http://www.pluralsight.com/training/Courses/TableOfContents/kanban-fundamentals

Code Camp - USC Campus October 2011

Code Camp: http://www.socalcodecamp.com/

Sessions: http://www.socalcodecamp.com/session.aspx?sid=d5f8077c-0ecb-404e-9440-7310f4b4484f

Oct 15 - 8:45 AM

From Delegates, To Lamdas And Expression Trees: Your Guide To Writing Elegant Code In C# .NET 4.0

Chander Dhall

Room: SLH-102

Notes:

Oct 15 - 10:00 AM

Algorithms and patterns in C#

Chander Dhall

Room: SLH-102

Notes:

SOLID principles

http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod

http://www.hanselman.com/blog/HanselminutesPodcast145SOLIDPrinciplesWithUncleBobRobertCMartin.aspx

Single Responsibility Principle:

"There should never be more than one reason for a class to change." — Robert Martin, SRP paper linked from The Principles of OOD

Open Closed Principle
Liskov Substitution Principle
Interface Segregation Principle
Dependency Inversion Principle

Oct 15 - 12:15 PM

Get Started with HTML5

David Pallmann

Room: SLH-200

Notes:

HTML5 is the lingua franca on World Wide Web

Cross browser testing:

http://www.multibrowserviewer.com/

Adobe Browser Lab https://browserlab.adobe.com/en-us/index.html#

HTML5 Example sites:

David's HTML5 website: http://neudesic.blob.core.windows.net/azuredesignpatterns/index.html

http://www.apple.com/html5/

http://www.pirateslovedaisies.com/

http://www.html5-games.info/shooting-games/js-wars.html

https://www.lucidchart.com

http://alteredqualia.com/canvasmol/

http://html5.labs.ap.org/

Google body browser

Suggested Blogs:

http://www.alistapart.com/

Oct 15 - 1:30 PM

Building Native Mobile Apps with HTML5 and jQuery

Jon Flanders

Room: SLH-200

Notes:

CSS3:

@media: media query is very powerful feature of CSS3, it queries screen and view from browser.

example: @media: screen and max : after

How to make your web application/site to act like a native application on mobile devices:

1- Add special size images to be used in "Add to home screen" feature of iPhone and etc ...

2- Make all links compatible to app style

3- Use native shell if needed

Oct 15 - 2:45 PM

Mobile web development with HTML 5 and jQuery Mobile

Robert Altland

Room: SLH-200

Notes:

- Use Visual Studio Mobile project MVC4

- Design choices:

1- One size fits all, this is MVC4.0 out of the box

2- Single page mobile app

- JQuery mobile site http://jquerymobile.com/ has great amount of examples and documentations

- Testing Tools:

The ones provided by hardware manufacturer (IOS,Android and Windows)

- Visual studio projects new project on MVC 4.0:

1- Internet Application: One size fit all application

2- Mobile application: this is for one page mobile app

Oct 15 - 4:00 PM

A lap around ASP.NET MVC 4

Jon Galloway  

Blog: http://weblogs.asp.net/jgalloway/

Podcast: http://herdingcode.com/

Room: ZHS-163

Notes:

- MVC 4.0 Developer preview is available now.

- Modernizer.js is part of MVC 4.0

- Adaptive rendering

- Targeting mobile, by just creating a file with .mobile extension. layout.mobile.cshtml

- Custom Modes

How to configure Windows Server 2008 to host WCF services in IIS 7/WAS

Configuring Windows 2008 server for hosting WCF services in IIS7.0 and WAS

This document is focused on hosting a WCF service on Windows 2008 service Windows Activation Server (WAS). This service is utilizing tcp net as binding protocol.


Below are the steps:

• Enable Network Discovery
• Create a folder for your service code in root (c:\ MyService)
• Add Roles and features:

1. Add Web Service Role (IIS) to server, it will require to add WAS feature. (Figure 1)
2. Add Application Server Role, this will require to install WAS feature. (Figure 2)

 Figure 1)  Adding Application/Web Server Roles 

 Figure 2) Adding Application Server Role

Figure 3) Adding WAS Application Server Roles

Figure 4) Choosing Authentication Certificate for SSL Encryption

• Select all items under Security
• Go to Features Expand .NET framework 4.0 Features, check WCF Activation box.(done in previous section)
• Add Network Load Balancing Feature (See Figure 5) There will be another posting to cover setting up NLB.

Figure 5) NLB installation

• WCF Setup: (for more information go to: http://msdn.microsoft.com/en-us/library/ms751402.aspx)

1. Register .NET Framework (use Framework64 folder if you are on a  64 bit machine)
1. "%WINDIR%\Microsoft.Net\Framework\v4.0.30319\aspnet_regiis" –i –enable
2. Register WCF   (use Framework64 folder if you are on a  64 bit machine)
1.  "%WINDIR%\Microsoft.Net\Framework\v4.0.30319\ServiceModelReg.exe" -r

• Virtual Directory Setup (IIS7.0): (for more information to go to: http://msdn.microsoft.com/en-us/library/ms751432.aspx)

1. Create Application/virtual directory
2. Set additional virtual directory properties
3. Set security:
4. Add your designated service account myServiceAccount  to below User Groups
1. Performance Log Users
2. IIS_IUSRS

• Firewall setup: http://msdn.microsoft.com/en-us/library/ms751530.aspx

       Option 3: Enable a port range in advance. Start the Windows Firewall Control Panel applet

 and enable ports 80, 443, 8000-8003 and 9000, which are used by the samples. You can find more detailed instructions in the following procedure. This option is less secure than the others because it allows any program to use   these ports, not just the samples

1. Chose option 3 from above link.       
2. Create new Inbound, Outbound rules (named WCF-WF-TCP-Ports)

• Edit site bindings: Run below commands
  
  

  Command1: %windir%\system32\inetsrv\appcmd.exe set site "Default Web Site" -+bindings.[protocol='net.tcp',bindingInformation='808:*']

  
  
  
  
               Command2:%windir%\system32\inetsrv\appcmd.exe set app "Default Web      Site/yourServiceName" /enabledProtocols:http,net.tcp

Use a Meta Data Explorer to explore addresses:

Example client URI for MyService:

Client URI: net.tcp://servername.domain/MyService/MyService.svc

Meta Data URI: net.tcp://localhost/MyService/MyService.svc/mex

• install MSMQ service if needed http://msdn.microsoft.com/en-us/library/aa967729.aspx

Related Links:

1. http://msdn.microsoft.com/en-us/library/ms752218.aspx
2. http://msdn.microsoft.com/en-us/library/ms751530.aspx
3. http://msdn.microsoft.com/en-us/library/ms733768.aspx
4. http://msdn.microsoft.com/en-us/library/ms751432.aspx
5. http://msdn.microsoft.com/en-us/library/ms751527.aspx

Patterns and Practices Symposium - Summary

It's been a while since I've blogged and it's been a while since I've been to this event but still I wanted to blog about it. Back in October 2010 I attended my first Patterns& Practices Symposium (PnP) in Redmond, Seattle. Link to: Patterns & Practices Symposium 2010
It was one of the best tech gatherings that I’ve ever attended and I have been to PDC, DevConnection and TechEd! Great crowd of attendees as well as awesome speakers and sessions
Anyway, below is a summary/abstract of the sessions and concepts that were discussed.


Sessions

Enterprise Library Extending
Parallel Programming
Cloud Computing
Windows Phone 7
OData – sesame data browser

Programming principles/Concepts

- Fluent Interface
- Inversion of Control
- DI – Dependency Injection (Unity, Spring or Castle Windsor)
- AOP – logging, exception handling
- CQRS - Command/Query Responsibility Separation
- M-V-VM
- Object Mother

SDLC Processes

Agile + TDD:

- Incremental improvement is the key to clean the old code and avoid massive redesign.
- Boy Scout rule, leave the camp cleaner than when you got it!
- In every check-in improve slightly.
- The only way to go fast is the go well (write good code pays off)
- Code Clean!
- Suite of tests gives you confidence to make changes improve/re-factored.
- Tests are used as documentation.
- Test needs to be designed well, decoupled.
- Manual test is expensive it grows.


TDD+ Validation+ Dependency injection


TDD Side Notes

TDD is the key for high quality code, it guarantees less bugs, it takes a little longer in beginning iterations but it pays off by the end of the project. It enforces developers (junior and senior) to follow the guaranteed path of TDD for quality code.

TDD requires development tool like re-sharper!

Continuous Integration Process:

CI Builds + Runs Test + Runs Code Coverage (95% for BLL, 85% for UI) + Code Analysis

Patterns of Healthy Teams using Visual Studio and TFS

- Turn on Code analysis at CI level, create your custom rule set and store it in TFS to be accessible
- Peer reviews catch about 60% of defects (Barry Boehm)

Options:

         1- Peer review
         2- Pair program
        3- Do micro-reviews, Make check-in notes required, encourages peers to


- Build Notification


- Layer Diagram

         - Document code for new developers
         - Validate Architecture
         - Generate Dependencies
        - Set Validation architecture to true at Project properties, which will generate error at CI

Flicker deploys code every 30 minutes
Gmail pushes new features are deployed in 2 days!

Tools

- Visual Studio Layer Diagram
- Visual Studio scrum templates – TFS installation required
- Instant Feature Builder VSIX
- Re-sharper
- Visual studio productivity tool

Summary

Focus of the conference was Agile + TDD + V-M-VM + Repository + Dependency Injection in Silverlight and WPF.