Sikich’s digital forensics and incident response team investigates a variety of different breach situations, including those involving ransomware events, ecommerce site compromises, account takeovers and internal employee threats. While the tools, techniques and findings related to each cyber breach situation vary greatly, there is one common theme that Sikich finds in almost every investigation: a lack of log data limits our ability to determine key factors like the root cause, timeline or extent of the compromise.
This might be surprising to many people, because at first blush it seems like most computing systems create a lot of log data. Unfortunately, there are some key areas where these logs usually fall short in relation to a cyber breach event.
Windows Event Logs
In about 90% of the ransomware events that Sikich investigates, the domain controller Windows security event log is set to the default size and, as a result, retains only a few days to a week of log data. Similar results are found on other Windows servers. These security event logs contain key information about the who, when and where for logins and failed login attempts, and are key to understanding an attacker’s activities leading up to a ransomware attack. However, most attackers spend days or weeks within a network before launching ransomware, and any logs for these activities are lost long before an investigation begins.
In addition to retention of Windows event logs, oftentimes the content of Windows event logs can be an issue. Again, with default settings, Windows may log that a PowerShell script was run, but custom log settings are needed for the log to record what the script did.
Organizations should take the time to tune their Windows security event logs and other event logs to make sure they are logging important actions and retaining enough history (180 days or more) to allow for a complete investigation if a breach occurs.
Linux Operating System Logs
Linux servers are commonly used for ecommerce websites. Similar to Windows, oftentimes Linux server default settings can cause important logs to be lost. Most Linux systems log data to the /var/log directory, and may only retain logs for a week or a month to preserve disk space. While ransomware attacks generally happen within a few weeks of an attacker gaining initial access, ecommerce site breaches, such as the installation of card-skimming malware, can continue for many months before any detection.
Organizations should make certain that ecommerce web servers retain at least a year of key operating system logs, such as for logins, remote access and actions performed by administrators.
Web Server Logs
When an ecommerce breach occurs, web server access logs are a critical source of information. They can show brute-force password guessing activities and SQL injection attacks in progress. They can help identify web shell backdoors on the site. However, web server access logs are also typically very large, logging an event every time a web page, image or script is loaded. As such, a lot of web server administrators configure limited retention of web server logs to preserve disk space. When we go to investigate an ecommerce breach, we may only have a few months’ worth of web logs, even though we’re seeing web shells installed a year or more ago.
Another common web server log challenge stems from the increasing use of cloud-based content delivery and web application firewall (WAF) systems, such as Cloudflare. With these types of services, website visitors send their web requests to the cloud WAF, and then the cloud WAF sends the request on to the real web server. However, with the default configuration in this situation, the web server will log the IP address of the cloud WAF, not the IP address of the website visitor, in the web access logs. As a result, instead of being able to find one suspicious web request and tie it to other attacker activity based on the source IP address, the investigator sees a log where all requests come from the same source.
Organizations should make sure that web server logs are retained for at least a year, and ideally for two years if storage resources are available. Organizations should also make sure that web logs record the true source IP address of requests.
Firewall and VPN Logs
Unlike servers, firewall and VPN appliances often do not have traditional hard drives with space available for storing logs. As a result, very little logging may be retained on firewall and VPN devices, and these logs may be lost during reboots. However, in investigating a security incident, it is often desirable to be able to look into firewall traffic history to find suspicious IP addresses, to look into firewall traffic patterns from servers that could indicate bulk theft of data, and to research the history of VPN login events going back weeks or even months.
Organizations should configure logging systems for firewall and VPN devices to record sessions, traffic patterns and successful and failed login events, and make certain that this log data is retained for 180 days or longer.
Cloud Service Logs
Today’s cyber breach investigations increasingly rely on cloud-service provider logs. Attacks against electronic payments often begin with unauthorized access to employee mailboxes, which are more and more frequently hosted in the cloud. Cloud service logs vary by provider, but you can’t count on the data you want being available unless you take steps ahead of time to check your cloud logging configurations. For example, Microsoft Outlook 365 by default today enables logging for new site setups, but that was not always so. In addition, certain log features, such as message tracking logs or log retention beyond 90 days, may not be in place unless an organization takes steps to enable those logs.
Organizations should review their log retention policies for cloud services to make sure that all important events are being logged and that logs retain at least 180 days of log data.
Taking just a bit of time now before a cyber breach to check and configure data logs can help organizations avoid the hardships caused by the lack of this key information at the time of a breach. Should you need any assistance in improving your logging practices, please don’t hesitate to reach out to our team.
