Elyra’s Jupyter AI Pipelines Now Support Custom Components – InApps is an article under the topic Software Development Many of you are most interested in today !! Today, let’s InApps.net learn Elyra’s Jupyter AI Pipelines Now Support Custom Components – InApps in today’s post !

Read more about Elyra’s Jupyter AI Pipelines Now Support Custom Components – InApps at Wikipedia

You can find content about Elyra’s Jupyter AI Pipelines Now Support Custom Components – InApps from the Wikipedia website

Elyra, the artificial intelligence (AI) toolkit first released by IBM in early 2020, helps data scientists with the often difficult process of building AI pipelines. As they wrote in the tool’s introductory post, “Building an AI pipeline for a model is hard. Breaking down and modularizing a pipeline is harder.” A data pipeline can include a number of steps, some relying on others, and creating this pipeline can lie outside the core skills needed for data science. Elyra solves this by offering a visual interface that turns creating and altering data pipelines into a familiar experience.

Patrick Titzler, a developer advocate at the Center for Open-Source Data and AI Technologies at IBM, explained that Elyra lets users assemble basic building blocks — Jupyter notebooks, Python scripts, and R scripts — into a pipeline that lets them perform tasks in sequence, in parallel, or otherwise.

“If you go through a machine learning workflow, you might have to load the data, analyze the data, cleanse it, then build the model, train the model, tune the model. And then you might have to go back when the results don’t really meet your expectations,” said Titzler. “With a pipeline editor, you can create those pipelines using simple drag and drop and then configure the nodes in that pipeline. So it speeds up your development because you don’t have to write any custom code to run all of those components or nodes in the pipeline. Plus, it enables people to actually do these things without necessarily having a deep domain expertise.”

Read More:   Ruby Creator Yukihiro Matsumoto on the Challenges of Updating a Programming Language – InApps 2022

This has been arguably the project’s most important feature, but the building blocks remained limited to those three types. With the recently released Elyra 3.3, however, users can create pipelines using custom components, a feature that Titzler wrote was “a major milestone on our roadmap.”

Previously, Elyra users could string together their own Jupyter notebooks or scripts, but they didn’t have access to external components, such as those available in Kubeflow Pipelines or Apache Airflow, the two platforms for running pipelines currently supported by Elyra. For example, the following image shows the components that are now available with Kubeflow Pipelines in Elyra, which includes things like creating a dataset volume or counting rows.

Another example of a set of components that can be added to Elyra with these changes is the Machine Learning Exchange, which provides an open source Data and AI assets catalog and execution engine for Kubeflow Pipelines. Titzler also points to the Component Library for AI, Machine Learning, ETL, and Data Science (CLAIMED) as an example. CLAIMED is a set of Jupyter notebooks that implement tasks such as data loading, data transformation, or model training, and can be used in Elyra as of this last release. CLAIMED can now be used by simply cloning the CLAIMED repository, and then the pipelines in that repository can then be opened in the pipeline editor for immediate use.

Titzler cautions that custom components differ from other components in Elyra in a few ways. First, they are runtime specific and often use runtime-specific mechanisms to exchange data with other components, instead of the S3-compatible storage used by generic components. They also need to be managed separately and are black boxes. While the Visual Pipeline Editor can expose their input and output, it does not have access to the functionality itself, necessarily.

Currently, Elyra only has support for local execution, Kubeflow Pipelines, and Apache Airflow, in terms of pipeline orchestration, but Titzler says that the community at large is in the process of adding others. He says that he has heard of interest in both Ray and Argo, but that any movement in those directions currently depends on the efforts of the community.

Read More:   A Data-Aware Shell for Faster Distributed Text Processing – InApps 2022

Looking forward, Titzler says that the project has “a big list of wishes that have come from various sources” but that improvements to the visual editor and increasing usability were among the current aspirations.


Rate this post
As a Senior Tech Enthusiast, I bring a decade of experience to the realm of tech writing, blending deep industry knowledge with a passion for storytelling. With expertise in software development to emerging tech trends like AI and IoT—my articles not only inform but also inspire. My journey in tech writing has been marked by a commitment to accuracy, clarity, and engaging storytelling, making me a trusted voice in the tech community.

Let’s create the next big thing together!

Coming together is a beginning. Keeping together is progress. Working together is success.

Let’s talk

Get a custom Proposal

Please fill in your information and your need to get a suitable solution.

    You need to enter your email to download

      Success. Downloading...