LogDNA Aims to Make Logging More Predictive with Machine Learning – InApps 2022

LogDNA, one of the newest entrants into the already crowded market for infrastructure log management systems, aims to help companies take a proactive approach to server logging by applying machine learning to detect IT problems.

The company’s founders, serial entrepreneurs Chris Nguyen and Lee Liu, were part of the Winter 2015 Y Combinator class but found their project e-commerce company, PushMarket, wasn’t gaining the traction they expected.

Looking to salvage what they could from the work they’d done, they decided to pivot and instead focus on the log management technology they had custom built for their project.

“After talking to potential customers, we realized that we had built an internal tool that was more powerful than our original product,” Nguyen said.

Conceding that they wouldn’t have everything they wanted to be built, to begin with, their San Francisco-based company released its initial product in February and has gained support with startup customers including T-shirt site Teespring, testing firm Rainforest QA, subscription marketplace Cratejoy, and construction equipment rental site Equipmentshare.com.

“What we’re really building is this predictive intelligence layer,” Nguyen said. “Logging today is all reactive. The reason people interact with their log is because something’s broken and we know that at some point in time, we could apply machine learning to say, ‘Heads up! Here’s what’s going to break and we see that through your log data.’ ”

Infusion of Cash

The co-founders previously created location-based recruitment website JobLoft, dating site cupid.com and social commerce site TeamSave, a partner with eBay Classifieds in Canada and the UK. Their new company recently raised $1.3 million from Initialized Capital and Skype co-founder Jaan Tallinn.

The duo built their algorithm on top of Elasticsearch to make search faster. It runs on AWS EC2. They tout their user interface as “the Apple of logging.”

“Our customers have told us they wanted faster search, a longer search retention, a better interface/experience and a more cost-effective solution than the competitors,” Nguyen said. “DevOps engineers wish they had a solution that allowed them to see infrastructure problems before they arise or troubleshoot issues much faster — problems like proactively seeing a DDOS attack, server outages and identity root causes behind code issues. That’s something we’re building.”

The initial offering has focused on aggregation, search, tail and alerting. The predictive intelligence layer will be ready for beta in Q4, according to Liu, and will include a graphing feature to visualize problems.

LogDNA allows users to aggregate, search and filter from all hosts and apps. It offers automatic parsing of fields from common log formats, such as weblogs, Mongo, Postgres and JSON.

It offers a live-streaming tail using a web interface or command line interface (CLI).

“We handle parsing by determining what type file it is based on the log data,” Liu said. “We can determine if it’s a Nginx, Apache, Ruby, Mongo, Redis, etc. dynamically. Once we know the file type, we can parse out different fields so we can aggregate and/or visualize the data. We are building our visualization tool as we speak.”

Having found themselves trying to reduce their storage costs in the past, they wanted to give customers more flexibility in storing larger quantities of data. LogDNA has a free tier at 50 gigabytes a month up to a one terabyte a month plan for $999/month.

The company took some heat in a Hacker News thread from February about offering encryption of data in transit, but not for data at rest. Liu said that’s since been added.

The company plans to open source its technology in the future as a means to add language libraries and other features.

Nguyen concedes it’s a very crowded market, and they don’t have all their envisioned features in place, but they decided to get an offering out there and build upon it.

“We’re building more intelligent logging, but it’s a long-term play,” he said. “There are players like Splunk in the market focused on enterprise, and we’re focused on a niche of SMBs. If you look at competitors like PaperTrail and LogEntries, they’ve been acquired, so I don’t know how much feature enhancement they’re going to do. We want to be that next player in the next generation in this space,” he said.

While the company plans to build out features, such as support for Docker, myriad competitors are also undertaking the same objective, including Rapid7, which bought out Logentries. Elastic.co, Graylog, Loggly, Papertrail and Sumo Logic are all Docker logging partners.

Standing out

Machine learning is the magic term that almost all the up-and-coming log analysis tools are trying to hang their hat on, according to Fixate IO analyst Chris Riley.

“I would say Sumo Logic, Splunk, LogEntries [and others] have it as a feature whereas Anodot and MoogSoft are making it the key element of everything that they do,” he said.

So far, however, most machine learning capabilities have been merely smart statistics, he said.

This company’s focus so far has been on basic features – more storage, better search — that won’t be true differentiators, he said.

“The problem they’re going to have is they’re competing with open source tools like the ELK stack and the others that allow you to do log analysis. The ELK stack has become tremendously popular – and that’s all free,” he said. “A pretty user interface is not what wins with developers.”

While all vendors want to be seen as the blanket logging solution, that means becoming the “DevOps hub” or single pane of glass.

“But I think this is far from reality,” he said. “I have yet to encounter a company who uses just one logging solution. And just because of the structure of most teams, this will always be the case.”

This crowded field makes it really difficult for new players to stand out.

“If I were a new vendor in this space, my approach would be very specific, very niche,” Riley said. “Maybe a specific development stack that they could address better than anybody else, such as Ruby applications or, dare I say the word, containers, though there’s a lot there already. It might be a specific vertical that has special requirements, such as financial services. In financial services, relating application operations to transactions is a big problem. It would have to be very niche and very focused. Otherwise, they’re just participating in the noise of better logging.”

Feature Image: “Tree rings” by Sheila Miguez, licensed under CC BY-SA 2.0.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.