GitLab’s Meltano Shows Why Open Source Is the ‘Only’ Future – InApps Technology 2022

GitLab sponsored this post.

GitLab’s open source Meltano data pipeline management project could turn out to be a failure. It could also become the industry-standard platform for managing and orchestrating data pipelines. Regardless, while the outcome will likely be somewhere in between, Meltano serves as a case study of open source development challenges and why learning from failure can eventually pay handsome dividends.

Meltano began as a way to solve a common DevOps problem associated with a lack of in-house expertise to manage data pipelines. Many organizations could certainly use a data platform to pool and analyze data from different sources. These sources might range from Google Analytics or Google Ads to CRM, but organizations often lack the resources to pay for such a high-powered system. Meltano’s creators intend for it to serve as a way to offer DevOps teams the extracting, loading and transforming capabilities of ETLs.

“Many commercial tools, such as those that enable data integration, are not actually accessible to a significant part of the market, meaning that there are a lot of companies out there that could be doing something with their data but they are not actually doing so because of the financial barrier they can’t overcome,” said Douwe Maan, Meltano’s lead at GitLab. “So what we hope to do is level the playing field and commoditize data integration tooling by building a really great open source version.”

But this is more than just an open source side project for GitLab. Meltano is seen as fulfilling a huge need, similar to how GitLab offered an open source alternative as a Git repository before its explosion in popularity.

“The Meltano project is our test case for open source in the data space. We’re trying to see if we can replicate GitLab’s story in a different market that really has lacked a competitive open source solution to a problem every company has,“ said Maan, who was one of GitLab’s first employees during its startup stage. “When GitLab was created, it was seen as an open source alternative that was so useful that it could actually compete with tools companies were currently paying for.”

It will likely be a couple of years before GitLab is able to commercialize services for Meltano’s adoption by organizations. Similarly,  there was a gap of at least a year between the time GitLab was created as an open source project until it became apparent that there might be “a business opportunity around  GitLab,” said Maan.

“That’s when the whole journey started, of course, when GitLab became open core, with GitLab still contributing the majority of the code to the open source version, while offering an enterprise edition and 24/7 support,” said Maan.

However, Meltano has not been without its challenges. The project’s headcount from GitLab, for example, has been reduced because “success would take longer than we anticipated,” said Sid Sijbrandij, co-founder and CEO at GitLab, during a InApps Technology Makers podcast.

However, “the premise of it that there should be an open source, ETL framework to load data. That is still correct,” said Sijbrandij. “So, we’ll keep investing until we’re convinced our premise was wrong in the first place.”

GitLab plans to back the project for as long as it remains viable. Sijbrandij offered the comparison of how Amazon has continued to back projects based on a long-term vision of their viability.

“Amazon is famous for investing in initiatives a bit longer than the competition and bringing them to fruition and being very patient. And I think it’s essential that as long as the assumptions don’t change, you keep investing,” said Sijbrandij. “You don’t give up but sometimes you have to change how you do it. The premise that there should be an open source ETL framework to load data is still correct, so we’ll keep investing until we’re convinced our premise was wrong in the first place.”

The project is yet another example of why the adoption of open source is no longer the de facto model for software development of the future. Its adoption is now proliferating across most software segments.

“Open source is the logical outcome for most of the software that is broadly used,” said Sijbrandij. “In the past, you had commercial software and it was ‘take it or leave it.’ Open source is going to give people a lot more freedom, lower prices for software, more freedom to modify, and less vendor lock-in — I think that’s a beautiful thing.”

Join GitLab Virtual Commit Aug 26 and hear speakers from the U.S. Air Force, the U.S. Army, GNOME Foundation, State Farm, Northwestern Mutual, Google and more about problems solved, cultures changed, and release times halved. Register for free today.

Amazon Web Services is a sponsor of InApps Technology

Feature image via Pixabay.

At this time, InApps Technology does not allow comments directly on this website. We invite all readers who wish to discuss a story to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email: [email protected].