Virtual Columns in Vaex
When we create a new column by adding expression to a DataFrame, Virtual columns are created. These columns are just like regular columns but occupy no memory and just stores the expression that defines them. This makes the task very fast and reduces the wastage of RAM. And Vaex makes no distinction between regular or virtual columns.
Python3
%time df_vaex['new_col'] = df_vaex['column3']**2df_vaex.mean(df_vaex['new_col']) |
Output:
Introduction to Vaex in Python
Working on Big Data has become very common today, So we require some libraries which can facilitate us to work on big data from our systems (i.e., desktops, laptops) with instantaneous execution of Code and low memory usage.
Vaex is a Python library which helps us achieve that and makes working with large datasets super easy. It is especially for lazy Out-of-Core DataFrames (similar to Pandas). It can visualize, explore, perform computations on big tabular datasets swiftly and with minimal memory usage.