Trends In Database Log Management By Anton Chuvakin

Trends in Database Log Management Anton Chuvakin, Ph.D. WRITTEN: 2007 DISCLAIMER: Security is a rapidly changing field of human endeavor. Threats we face literally change every day; moreover, many security professionals consider the rate of change to be accelerating. On top of that, to be able to stay in touch with such ever-changing reality, one has to evolve with the space as well. Thus, even though I hope that this document will be useful for to my readers, please keep in mind that is was possibly written years ago. Also, keep in mind that some of the URL might have gone 404, please Google around. Introduction: Why Database Logs Fall In The Realm of Database Security Buried deep within enterprise IT infrastructures, databases can be said to hold the “crown jewels” of an organization. Unfortunately, database security is often lacking, leaving sensitive, business-critical information such as customer data, financial details, and more, vulnerable to hackers. Dept of VA, TJ Maxx, TD Ameritrade – these are just a few of the many organizations that have driven the media wild over data security breaches in the last year. It is common that database administrators (DBAs) are assigned the task of database security, but this is an issue that should be of utmost importance to any business that wants to stay in business. TJ Maxx reported at least 45.7 million credit and debit card numbers stolen over a period of several years, costing the company an estimated $168 million [or whatever other large random number ]. Proper security measures may not have stopped the initial hack-in, but perpetual data theft could have been avoided through careful log collection and analysis. This article will not only discuss the importance, challenges and benefits to database logging, but will also offer a few forward-looking trends to managing your database logs. About Logs and Database Logging Databases are now becoming one of the most voluminous log generators in the enterprise – rivaling firewalls for the top spot. Most databases (ie: Oracle, Microsoft SQL Server, IBM DB2, MySQL, etc.) will log system starts, stops and restarts by default, but database logging isn’t merely about “keeping the system running,” particularly when your databases contain sensitive, private information. Security and compliance requirements must therefore be considered when configuring your database and managing your logs. In fact, regulations such as

PCI, HIPAA, and FISMA all mandate log monitoring, with SOX strongly recommending it as a best practice. Database logging thereby becomes an essential (and required) component of database security – and it makes sense to not only focus on “keeping the bad guys out,” but also take a “what’s going on in here?” approach. After all, you may not know who the “bad guys” are. Logs can provide a continuous fingerprint of everything that happens in your IT systems and with your data and will point you to the “who, what, when, where” information of any breach – whether the malicious behavior comes from outside hackers, a disgruntled employee, or another source. Database security is a task often assigned to DBAs, not because they’re security experts, but because they know the in’s and out’s of databases. If configured properly, databases may be logging overwhelming amounts of files, perhaps up to gigabytes of data per day. Typical database log events may include: • • • • • • •

User logins and logouts Database system starts, stops and restarts Various system failures and errors User privilege changes Database structure (metadata) changes Most other DBA actions Select or all database data access (if configured to be so)

As we know, hackers are always looking for new ways to break through security barriers to access your sensitive information and all preventative security measures fail at some point. Thus, since you are not able to guard against every malicious hacker, logs will at least allow you to detect such security breaches as well as actually figure out how it was done during the incident investigation. At a minimal level, logs must be collected and archived, but log analysis does make the data significantly more useful. In more explicit terms, log monitoring and management should include: • • • • •

Collection: Gathering log data where it is being generated via an agent or remotely Transfer: Securely transmitting log data to a central server for analysis and storage Alerts: Issuing real-time alerts to database administrators if needed Reporting and Analysis: Providing reports and analytics based on log data Storage: Securely storing logs as long as prescribed by your retention policy and then, just as safely, destroying them

The above examples for managing your log data will help you keep tabs on the activities occurring in your business. Regularly collecting log data is a best practice for incident response and can save you during crunch time after a server crash, data theft, or surprise visit by your friendly auditor. Alternatively, if

someone is downloading an entire table or changing a database schema while being logged on from a remote connection, a real-time alert will catch your attention. Further, reports may help you track and analyze login failures and successes, or after hours access, to better evaluate insider privilege abuse. In other words, database logs can help you catch unusual behavior before a problem gets out of hand and into headline news. Database Log Management: Where to Begin If you are just beginning to set up a method for managing your database log data, be ready for a large volume of log records as well as issues pertaining to log availability and log format complexity. Log formats can be verbose and obscure, particularly in cases where a single message spans multiple lines, making it difficult to extract useful, actionable information via automated tools. Other challenges to database logging include decreased performance and storage restrictions. Unlike other situations where logging has a minimal impact on system performance, database audit logging slows down the database, sometimes significantly. High-performance databases are built to provide thousands of data transactions per second, logging all of these presents a challenge to system IO as well as CPU and disk storage resources. Since many regulations specify a 3-12 month period for log retention, plus a longer period for log retention on tape or another dedicated storage tool, database logging is typically getting a bad rap among DBAs already spread thin for time and resources. Because the difficulties associated with database log management can seem overwhelming, it’s best to take things one step at a time. Start slowly and build up your system from there. You’ll want to collect logs from multiple servers at one central location to facilitate incident analysis and response. This will also help prevent loss of log data during routine log rotations. To gain insight into “What’s going on” internally, conduct periodic reviews of DBA activity logs – you can then keep tabs on people entrusted with sensitive and/or private information such as customer data or product inventory information. When beginning to organize and manage database log data, also keep in mind that manual log analysis can cost a lot in terms of time, human resources, etc. Popular database solutions such as Oracle, Microsoft SQL Server, MySQL, and more, tend to offer various basic logging options, but none comprehensive enough to really capture a continuous feed of database activity. By contrast, an automated log management tool will not only free up DBA time for other important database performance and security tasks, but can also be more reliable and efficient than manually managing log data. Further, with an automated LMI tool, you can schedule log collection to occur at off hours when other database service operations, such as backups, are happening so that database performance is undeterred during the workday.

Trends in Database Log Management Database logging often presents a new frontier for many security practitioners – one that must be conquered. Given that historically many databases were running without any data access and data change logging disabled (as by default), the key trend is that this is finally beginning to change. Why is it happening? There are two main drivers for this trend: PCI DSS compliance requirements that apply to those who handle credit card data and the proliferation of data breaches and data loss discussed above. The cost of data loss investigations in the absence of detailed access logs is absurdly high! What would happen next? As more people enable logging, the challenge of “what to do with all that data?” will emerge. Handling log storage and controlling log retention so that logs will be there when needed for the investigations will be the next trend. After that, database log analysis will become all the rage. Analyzing logs for anomalies, suspicious user activities, unsafe administrator actions, privilege abuse as well as good old hacking will require deployment of log management tools with database-specific intelligence. Further, logging guidance from IT “best practices” such as ITIL and ISO will become the norm and we will reach the database logging nirvana, when logging is enabled because “it is the right thing” and not only due to compliance pressures or the latest data breach. Log collection and automated analysis will become the norm as new a log management and intelligence (LMI) technologies simplify managing the log flood as well as “making sense” of logs. So, enabling logging is a good start, and taking small progressive steps towards more in-depth log analysis can be greatly enhanced by deploying LMI platform. LMI tools are becoming increasingly advanced and are now typically able to take a deep dive in to log analysis and management. A good log management solution will not only automate log data analysis, but also the whole lifecycle of log management. There is also an increasing trend in logging across the entire IT infrastructure – firewalls, servers, network devices, applications, operating systems, other sources of log data all produce logs! – all of which can be managed by an LMI platform. In other words, jump on the logging bandwagon with your other system administrators and balance your IT infrastructure with a log management system that will work across multiple database servers, across various database types, and with all other log producers in your organization. You will not only improve performance and business continuity, but also be able to put database log data in

the context of other organizational log data to correlate IT activities with events occurring in your business.

Conclusion Database log management is becoming a “best practice” for database security – you should be aware of who is accessing or changing your data, when they are accessing it, and where they are accessing it. Luckily, you can combine database log management with other similar projects (such as firewall or Unix server syslog management) and use a single automated LMI platform to enable efficient and reliable log collection, reporting analysis, and retention. As long as you grow your LMI deployment in phases, rather than trying to cover all logs on day one, you will be on the path to overall greater IT security within your organization. ABOUT THE AUTHOR: This is an updated author bio, added to the paper at the time of reposting in 2009. Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in the industry. In addition, Anton teaches classes and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.