Sisyphus: an event log data-mining toolkit

Sisyphus: snapshots

Sisyphus | Download

Sweb - sisyphus web interface

Here are a few slides from my presentation at SC06.

Sweb (above) is now the primary interface to sisyphus. The below stuff are basically historical snapshots.

Interactive review of automatically generated message templates (eg, regular expressions):

Logfiles (top) and terms therein (right) ranked by "interestingness" (eg using information-retrieval log.entropy term weighting):

The center plot in the above image shows a very simple "term-document" matrix of a set of logfiles. Sisyphus provides tools to convert your logs into matrices.

Logs automatically colorized according to term "interestingness". For example, a a point in the above "Doc Magnitude" plot pops up a colorized version of that log file ("interesting" (eg, "informational") terms are red, uninteresting are blue (eg, as determined by their distribution, not just their raw count)):

Explore the rate of "perpetual anomaly" in your logs (eg, at what rate do new terms and/or message templates appear? message traffic bursts do not always contain new information!):

Explore the term frequency distribution of terms (or message templates) in your logs (eg SLCT and Loghound are based on the property that that most terms occur very few times):

Export logs to other data-mining tools such as VxInsight:

The above image is a 2000-nodehour logset viewed in VxInsight. The log.entropy-weighted term-doc matrix has been rank-reduced via SVD decomposition ("latent semantic analysis") to rank 200. The below image is the same dataset, but reduced to rank 10 (maintains 80% of total variance). You can see how rank-reducing can further differentiate content clusters (eg, green dots clustered in above, but spread across multiple clusters in the below). Fun stuff, but hard to get quantitative mesures of effectiveness from (which is a project goal so I've not continued using VxInsight much).