Sisyphus: an event log data-mining toolkit

Sisyphus - a log data-mining toolkit

DEMOS | Defining Supercomputer RAS

Sisyphus is an easy way to find important stuff in logs. Yes it provides fast index-based searches when you know what to look for, but its distinguishing feature is that it helps you find anomalies when you don't know what to look for. It ranks log files and colorizes words based on information theory (answering "what is the strangest logfile, and why?"). It provides useful file and word statistics in interactive tabular and plot formats. It uses Vaarandi's excellent loghound program to generate message templates for you (which you can then use as regular expressions in monitoring programs if you wish). Web and command line interfaces are included.

Sisyphus has been developed specifically for use with supercomputer syslogs, based on the premise that similar computers correctly executing similar workload should produce similar logs (thus, anomalies warrant investigation). But it is general enough to be used on other log types (if your logs are not syslog, you will have to tweak arguments and/or provide a parser).

Latest Release: 1.7 on 2008-08-28 (readme, changelog, download). License: LGPL.

Publications:

Alert Detection in System Logs. A. J. Oliner, A. Aiken, and J. Stearley. In Proceedings of the International Conference on Data Mining (ICDM), Pisa, Italy, 2008. [pdf]
Bad Words: Finding Faults in Spirit's Syslogs. J. Stearley and A. J. Oliner. In Workshop on Resiliency in High-Performance Computing (Resilience-2008), Lyon, France, 2008. [pdf] [slides]
What Supercomputers Say: A Study of Five System Logs. A. J. Oliner and J. Stearley. In Proceedings of the International Conference on Dependable Systems and Networks (DSN), Edinburgh, UK, 2007. [pdf] [slides]
Towards Informatic Analysis of Syslogs. J. Stearley. IEEE Conference on Cluster Computing, Sepetember 2004. [pdf] [slides] [data]

Mailing Lists: See sisyphus-announce for release announcements, and sisyphus for general help and discussion. NOTE: the list archives are not viewable outside Sandia - SORRY!!!.

Contact: Email Jon Stearley <jrstear@sandia.gov> for further information.