Let’s say you are in a customer care center, and you would like to know the probability distribution of the number of calls per minute, or in other words, you want to answer the question: what is the probability of receiving zero, one, two, … etc., calls per minute? You need this distribution in order […]
The post Method of Moments Estimation with Python Code appeared first on Towards Data Science.

The basic principle of Large Language Models (LLMs) is very simple: to predict the next word (or token) in a sequence of words based on statistical patterns in their training data. However, this seemingly simple capability turns out to be incredibly sophisticated when it can do a number of amazing tasks such as text summarization, […]
The post How to Measure the Reliability of a Large Language Model’s Response appeared first on Towards Data Science.

Introduction Developers work on applications that are supposed to be deployed on some server in order to allow anyone to use those. Typically in the machine where these apps live, developers set up environment variables that allow the app to run. These variables can be API keys of external services, URL of your database and […]
The post Manage Environment Variables with Pydantic appeared first on Towards Data Science.

Python has grown to dominate data science, and its package Pandas has become the go-to tool for data analysis. It is great for tabular data and supports data files of up to 1GB if you have a large RAM. Within these size limits, it is also good with time-series data because it comes with some […]
The post Pandas Can’t Handle This: How ArcticDB Powers Massive Datasets appeared first on Towards Data Science.

It’s been more than 15 years since I finished my master’s degree, but I’m still haunted by the hair-pulling frustration of managing my of R scripts. As a (recovering) perfectionist, I named each script very systematically by date (think: ancova_DDMMYYYY.r). A system I just *knew* was better than _v1, _v2, _final and its frenemies. Right? Trouble was, every time I wanted to […]
The post Branching Out: 4 Git Workflows for Collaborating on ML appeared first on Towards Data Science.

Decision tree algorithms have always fascinated me. They are easy to implement and achieve good results on various classification and regression tasks. Combined with boosting, decision trees are still state-of-the-art in many applications. Frameworks such as sklearn, lightgbm, xgboost and catboost have done a very good job until today. However, in the past few months, […]
The post Build a Decision Tree in Polars from Scratch appeared first on Towards Data Science.

Virtualization makes it possible to run multiple virtual machines (VMs) on a single piece of physical hardware. These VMs behave like independent computers, but share the same physical computing power. A computer within a computer, so to speak. Many cloud services rely on virtualization. But other technologies, such as containerization and serverless computing, have become […]
The post Virtualization & Containers for Data Science Newbies appeared first on Towards Data Science.

Bubble charts elegantly compress large amounts of information into a single visualization, with bubble size adding a third dimension. However, comparing “before” and “after” states is often crucial. To address this, we propose adding a transition between these states, creating an intuitive user experience. Since we couldn’t find a ready-made solution, we developed our own. […]
The post 4-Dimensional Data Visualization: Time in Bubble Charts appeared first on Towards Data Science.