Please enable JavaScript to view the comments powered by Disqus. Do Data Scientists Like Being Inclined Towards DevOps?




Do Data Scientists Like Being Inclined Towards DevOps?



Last updated 19/03/2024

Do Data Scientists Like Being Inclined Towards DevOps?

DevOps and Data Science are two of the hottest fields in tech, but they provide very different purposes. DevOps concentrates on collaboration, automation and integration to enhance the software delivery and Information Technology operations. Data Science uses statistical and analytical methods to extract insights from the data. 

In the unlikely event that you were going to lay up a production-level machine learning pipeline, the data science work would have a position at the beginning when it came to planning and model preparation.

An ordinary pipeline will eventually shift from information science to foundation errands, usually at the perfect moment to take models to creation. This is, intuitively, where the data science team transfers responsibility to another team member, such as DevOps.

Although this isn’t normally the condition, to an ever-increasing extent, researchers are also being approached to deal with conveying models to generations. Based on the algorithmia, the lion’s share of data scientists report investing over 25% of their energy in model sending alone. Specifically, you can check this with the help of taking the gender at what number of information researchers' work postings incorporate things, including Kubernetes, Docker, and EC2 under important knowledge. 

Why data scientists shouldn’t have to handle model serving?

One of the straightforward answers to this query is that model serving is the foundation issue, not the data science issue. You can see this with the help of simply looking at the stacks used for each:

There are obviously a few data scientists who like DevOps and can work cross-practically, yet they are uncommon. The truth here is that it gets easy to state the cover between data science and DevOps is now and again overestimated.  

To flip the facts, is it okay to expect the DevOps designer to have the option to plan another model engineering or to have a huge amount of involvement in hyper parameter running? There are likely DevOps Engineers who have those data science aptitudes, and everything is learnable. Still, it's odd to consider those obligations the space of your DevOps Groups.

Data scientists more than likely do not get the field to stress over auto-scaling or composing Kubernetes shows, so for what reason do businesses cause them to do it?

Businesses are neglecting their infrastructure

Different businesses have fundamental misunderstandings and need to understand how complex model serving is. The attitude is often just wrapping a model in Flask in good enough for now.

The reality is serving the models at any scale, which includes resolving some infrastructure challenges. You can take the following example:

  • How do you update the models in production automatically without having any downtime?
  • How do you effectively auto-scale the 5 GB model, which runs on GPUs?
  • How do you monitor and debug production deployments?
  • How do you do all of these practices without running up a massive cloud bill?

So, in the current scenario, it's essential to become reasonable. Machine Learning framework is gaining new ideas and practices. Two years ago, Uber had just revealed Michelangelo, their cutting-edge internal machine learning foundation. The ML framework's playbook is currently being produced from a variety of perspectives. Nevertheless, there are still many examples where an association can separate the concerns of DevOps and information science without requiring the kind of design resources found in an Uber.

How to separate Data Science and DevOps?

Cortex was designed to delineate data science from DevOps and to automate all of the infrastructure code they were writing. Since open sourcing, they have worked with the data science teams who have adopted and implemented it, and their experiences have informed our approach.

It is crucial to understand the distinct roles and responsibilities of Data Science and DevOps in order to separate them effectively. Data Science primarily focuses on extracting insights from data through statistical analysis and machine learning algorithms, while DevOps is centred on software development, deployment, and operations.

Here are some key steps to separate Data Science and DevOps:

  • Clear Role Definition:To prevent duplication and guarantee that each team is aware of its particular tasks, clearly define the roles of data scientists and DevOps engineers.
  • Collaborative Environment:Promote an atmosphere in which DevOps engineers and data scientists can keep their respective competencies while working together productively when necessary.
  • Integrated Goals:Make sure that the goals of the two teams working on a project are aligned so that there are shared goals rather than conflicting ones.
  • Agile Methods:To improve productivity and flexibility, integrate agile methods into Data Science and DevOps operations.
  • Cross-Functional Teams:To foster cooperation and mutual understanding, encourage the development of cross-functional teams comprising data scientists, DevOps engineers, and data engineers.

They conceptualize the handoffs among the data science, DevOps and product engineering with the easy, abstract architecture they refer to as the Model-API-Client:

  • Model:A trained model with some predict () function that engineers can use without needing data science expertise.
  • API:The infrastructure layer that takes the trained model and deploys it as the web service. Cortex was developed to automate this layer.
  • Client:The actual application which interacts with the web service deployed in the API layer.

Data scientists train and export a model during the model phase. In order to generate and filter model predictions, they also write the predict () method.

After that, they turn this model over to the API phase, when the DevOps function takes full accountability for it. The model is just a Python function to the DevOps function; it has to be transformed into a microservice, containerized, and deployed.

After the model-microservice becomes online, developers can query it just like they would any other API. The model is merely another online service in their eyes.

While there are other ways to divide the responsibilities of data science and engineering, the Model-API-Client design shows that it is possible to do so without creating costly end-to-end platforms or adding extra overhead. Creating distinct handoff points between ML pipeline tasks allows data scientists to focus on what they do best, which is data science.


In conclusion, the intersection between Data Science and DevOps or the Devops culture for data science poses both challenges and opportunities for organizations navigating the complexities of machine learning pipeline deployment.

While there is a growing trend towards blurring the lines between these domains, it's essential to recognize and respect the unique skill sets and responsibilities of each. Businesses often need to pay more attention to the complexities of model serving, assuming that wrapping a model in Flask is sufficient.

However, deploying and scaling models at the production level requires addressing infrastructure challenges such as automated updates, auto-scaling, monitoring, and cost management.

Topic Related Post

Securing the Pipeline: Integrating Security into Your SRE Practices
Ready for the Next Level? Top DevSecOps Skills to Master Before 2025
SRE in FinTech: Challenges and Opportunities

About Author

NovelVista Learning Solutions is a professionally managed training organization with specialization in certification courses. The core management team consists of highly qualified professionals with vast industry experience. NovelVista is an Accredited Training Organization (ATO) to conduct all levels of ITIL Courses. We also conduct training on DevOps, AWS Solution Architect associate, Prince2, MSP, CSM, Cloud Computing, Apache Hadoop, Six Sigma, ISO 20000/27000 & Agile Methodologies.



* Your personal details are for internal use only and will remain confidential.


Upcoming Events


Every Weekend


Every Weekend


Every Weekend


Every Weekend

Topic Related

Take Simple Quiz and Get Discount Upto 50%

Popular Certifications

AWS Solution Architect Associates
SIAM Professional Training & Certification
ITIL® 4 Foundation Certification
DevOps Foundation By DOI
Certified DevOps Developer
PRINCE2® Foundation & Practitioner
ITIL® 4 Managing Professional Course
Certified DevOps Engineer
DevOps Practitioner + Agile Scrum Master
ISO Lead Auditor Combo Certification
Microsoft Azure Administrator AZ-104
Digital Transformation Officer
Certified Full Stack Data Scientist
Microsoft Azure DevOps Engineer
OCM Foundation
SRE Practitioner
Professional Scrum Product Owner II (PSPO II) Certification
Certified Associate in Project Management (CAPM)
Practitioner Certified In Business Analysis
Certified Blockchain Professional Program
Certified Cyber Security Foundation
Post Graduate Program in Project Management
Certified Data Science Professional
Certified PMO Professional
AWS Certified Cloud Practitioner (CLF-C01)
Certified Scrum Product Owners
Professional Scrum Product Owner-II
Professional Scrum Product Owner (PSPO) Training-I
GSDC Agile Scrum Master
ITIL® 4 Certification Scheme
Agile Project Management
FinOps Certified Practitioner certification
ITSM Foundation: ISO/IEC 20000:2011
Certified Design Thinking Professional
Certified Data Science Professional Certification
Generative AI Certification
Generative AI in Software Development
Generative AI in Business
Generative AI in Cybersecurity
Generative AI for HR and L&D
Generative AI in Finance and Banking
Generative AI in Marketing
Generative AI in Retail
Generative AI in Risk & Compliance
ISO 27001 Certification & Training in the Philippines
Generative AI in Project Management
Prompt Engineering Certification
SRE Certification Course
Devsecops Practitioner Certification
AIOPS Foundation Certification
ISO 9001:2015 Lead Auditor Training and Certification
ITIL4 Specialist Monitor Support and Fulfil Certification
SRE Foundation and Practitioner Combo
Generative AI webinar
Leadership Excellence Webinar
Certificate Of Global Leadership Excellence
SRE Webinar
ISO 27701 Lead Auditor Certification