Log Insight is an end-to-end solution for IT data management and analytics. It ingests large quantities of heterogeneous data at very high rates from local or remote/cloud sources and it automatically understands the intrinsic characteristics of semi-structured data. Such data are produced in vast amounts in the form of application logs, infrastructure (physical and virtual) logs, system stats, source code files, etc.. Log Insight then indexes the data and performs analytics and modeling on all the important metrics that are extracted from the data. Users interact through an intuitive UI tailored specifically for plowing through huge amounts of data quickly.

Unlike current generation log management solutions, Log Insight has been built to perform analytics from the ground up. It can ingest any IT artifact and provides answers (problem diagnosis, forensics, predictive modeling) through analytics as opposed to merely offering just search and simple aggregation. Here’s why:

• Smart Parsing: State of the art statistical machine learning-based parsing of diverse data sources (logs, configuration files, performance data, etc.) and unknown data formats. Essential to the usability and scalability of Log Insight.
• Advanced Indexing: Log Insight can utilize any field/data type information that exist in the data. Our indexing technology can offer real time searching for even the most complex queries..
• Data Mining and Proactive Analytics: Discover patterns in the data and monitor/model any metric/combination from the data to identify outliers and abnormalities that can reveal problems
• Distributed, Map-Reduce framework that scales up and down: Log Insight employes a Map-Reduct framework for scaling horizontally to very large data sets while maintaining easy of use for any size deployment
• Intuitive Querying and Analytics UI: Easy to aggregate data and perform analytical queries

Mining, analyzing and searching IT and Engineering data is a big challenge because of its sheer volume, semi-structured and dynamic nature and variability of users’ needs.

Beyond Search

The nature of IT data means there is some structure, whether a time series, event patterns, numeric fields, or other, that keyword search engines simply cannot capture. IT data search solutions can collect and aggregate data across different systems, but their emphasis is strictly operational. Search-based solutions cannot automateproblem diagnosis or perform sophisticated analytics. System administrators become smarter, but broader decision-making needs are unmet.

Beyond Traditional and NoSQL Databases

While there are structures in IT and Engineering data, they are sparse and change dynamically. This makes them a poor fit for traditional database systems where heavy ETL is required to load the data. The size and growth of these data and their append-only, semi-structure nature makes traditional DBMSes and data warehousing solutions a terrible fit.

More recent approaches to data management, such as NoSQL stores and large-scale distributed computing frameworks are far from being complete or enterprise-ready. They scale to large data sets but are typically optimized for throughput and not latency. They do not have the critical features necessary to deliver real-time data management and analytics:

• Data parsing and structure understanding
• Smart indexing, optimized for time-series data
• Domain specific query language
• Intuitive user interface
• Control and configuration for data size and resources
• Scale-up and scale-down capabilities

Log Insight provides sophisticated, real-time analytics by examining the entirety of users' data. It collects, ingests and analyzes any IT artifact providing real time answers to problems and deriving important insights about products, services and user behavior.

Real-time Problem Troubleshooting

Log Insight models and monitors any metric that derives from your data. It automatically figures out thresholds and points out errors and abnormalities as soon as they occur, leading to proactive troubleshooting of problems before they disrupt your system. Furthermore, every event, metric and signal is captured in real-time. Users can quickly generate aggregate views of the data and drill down to specific events or errors without having to jump between systems and applications. For example, while viewing an important system metric, e.g. number of HTTP errors per minute, users can drill down to a specific time-range of events and examine the errors with a single click.

Cloud Management

Cloud and Virtualization offers many benefits, but make monitoring far more complex. In the cloud world, whether you are a service provider with customer SLAs, a company managing your private cloud or a developer of an application deployed on a Platform/Infrastructure-as-a-Service environment, disruptions directly impact business. So, questions as to whether a performance issue is due to the network, storage, virtual machine manager, physical machines, virtual machines, middleware, or the apps running on them must be answered instantly and accurately by the monitoring system.

Log Insight can collect data from the whole stack in real time eliminating the risk of losing parts of it due to the ephemeral nature of the infrastructure. It can correlate the data on one or multiple dimensions (e.g. time or request id) providing a coherent view across the stack. Furthermore, it can point out abnormal events or signals automatically, enabling users to identify the root cause of problems quickly. Log Insight harvests performance and metric data from existing IT monitoring and performance management solutions and determines the complex relationships between these data.

Application Analytics

Application logs are the most valuable source of insights among the different data sources. They contain vast amounts of user interaction and system behavior information. Their format/structure is often custom and the needs of the users vary from simple monitoring of metrics to very sophisticated multidimensional analytics.

Log Insight offers solutions to all of these challenges. It can parse and understand unknown formats because of its statistical machine learning-based parsing. Any field that exists in the data is extracted and can be used to derive useful metrics. Application developers can utilize Log Insight's correlation and aggregation capabilities as well as interactive analytics functionality to get a good understanding of their system and build custom dashboards to follow the most important metrics.

Security Forensics

Effective Security Forensics requires the ability to collect all the relevant data in real time, retain them for as long as needed (often months), and correlate them across any dimension (e.g. actions of a user across different systems). System administrators need the ability to enforce policies, such as limiting access to sensitive data or limiting the number of failed login attempts. Furthermore, automatic identification of attacks is important

Log Insight offers real-time monitoring of policy violations and low-latency querying for interactive security investigation. It efficiently stores and archives data for long-term retention. Its analytics capabilities help users identify abnormal behavior and outliers that might reveal security attacks.