Motivation

Consider you’re an e-commerce platform aiming to enhance recommendation personalization. Your data resides in S3.

To refine recommendations, you plan to retrain recommendation models using fresh customer interaction data whenever a new file is added to S3. But how …

Motivation

Data scientists often work in iterative and exploratory environments. Thus, there is often a focus on rapid results rather than creating maintainable or scalable code.

However, data scientists must avoid writing poor code for the following reasons:

1. Reduced code

…

Motivation

Functions are essential in a data science project because they make the code more modular, reusable, readable, and testable. However, writing a messy function that tries to do too much can introduce maintenance hurdles and diminish the code’s readability.…

Motivation

Data drift occurs when the distribution of input features in the production environment differs from the training data, leading to potential inaccuracies and decreased model performance.

To mitigate the impact of data drift on model performance, we can design …

Why Logging Is Better Than Print?

Compared to print, logging provides more control over what information is captured in the logs and how it is formatted.

With logging, you can log different levels (debug, info, warning, error) and selectively enable/disable …

Motivation

Imagine you are trying to find all URLs inside a text. Each of these URLs must:

• Start with either http:// or https:// or the domain of the URL
• End with either .com or .org

You might end up writing …

Why Choose Git for Your Data Science Project?

Git is a version control system widely used in software development, but is it the right choice for your data science project? Absolutely.

Here are some reasons why Git is invaluable for …

Motivation

It is important to structure your data science project based on a certain standard so that your teammates can easily maintain and modify your project.

But what kind of standard should you follow? Wouldn’t it be convenient if you …

Motivation

In the beginning stages of your data science project, using dependency management tools like pip or conda may be sufficient.

However, as the data science project expands, the number of dependencies also increases. This can make it difficult to …

Motivation

dbt (data build tool) is a powerful tool for data transformation within the data warehouse.

https://mathdatasimplified.com/2023/05/08/build-an-efficient-data-pipeline-is-dbt-the-key/

However, it does come with some limitations, including the following:

1. Lack of Output Preview: With dbt core, it is not possible to preview

…