What is Handling Large Datasets in Pandas?

In this article, we will learn Handling Large Datasets in Pandas,This free Python tutorial for complete beginners will help you learn Python from scratch.

Handling Large Datasets in Pandas

Working with database using Pandas

How To Save Python Scripts In Linux Via The Terminal?

Pandas is a robust Python data manipulation package that is frequently used for jobs involving data analysis and modification. However, standard Pandas procedures can become resource-intensive and inefficient when working with huge datasets. We’ll look at methods in this post for efficiently managing big datasets in Pandas Python applications.

Problem with Handling Large Datasets

Pandas is a great tool when working with tiny datasets, usually ranging from two to three gigabytes. For datasets bigger than this threshold, using Pandas is not recommended. This is because, should the dataset size surpass the available RAM, Pandas loads the full dataset into memory before processing. Memory problems can occur even with smaller datasets since preprocessing and modification creates duplicates of the DataFrame.
Despite these drawbacks, by using particular methods, Pandas may be used to manage bigger datasets in Python. Let’s explore these techniques, which let you use Pandas to analyze millions of records and efficiently manage huge datasets in Python.

How to handle Large Datasets in Python?

Use Efficient Datatypes: Utilize more memory-efficient data types (e.g., int32 instead of int64, float32 instead of float64) to reduce memory usage.
Load Less Data: Use the use-cols parameter in pd.read_csv() to load only the necessary columns, reducing memory consumption.
Sampling: For exploratory data analysis or testing, consider working with a sample of the dataset instead of the entire dataset.
Chunking: Use the chunksize parameter in pd.read_csv() to read the dataset in smaller chunks, processing each chunk iteratively.
Optimizing Pandas dtypes: Use the astype method to convert columns to more memory-efficient types after loading the data, if appropriate.
Parallelizing Pandas with Dask: Use Dask, a parallel computing library, to scale Pandas workflows to larger-than-memory datasets by leveraging parallel processing.

Using Efficient Data Types:

Reducing memory utilization in Pandas requires the use of efficient data types. For instance, if precision allows, you can use float32 or even float16 in instead of the standard float64 dtype. Similar to this, if the data range permits, integer columns can be downcast to smaller integer types like int8, int16, or int32.
Benefits: Significantly lessens memory footprint, particularly for big datasets.
Implementation: When reading data, you can use functions like pd.read_csv() or pd.read_sql() to specify the dtype parameter. Furthermore, existing columns can be changed to more memory-efficient types using the astype() method.

Python3

import pandas as pd
 
# Define the size of the dataset
num_rows = 1000000  # 1 million rows
 
# Example DataFrame with inefficient datatypes
data = {'A': [1, 2, 3, 4],
        'B': [5.0, 6.0, 7.0, 8.0]}
df = pd.DataFrame(data)
 
# Replicate the DataFrame to create a larger dataset
df_large = pd.concat([df] * (num_rows // len(df)), ignore_index=True)
 
# Check memory usage before conversion
print("Memory usage before conversion:")
print(df_large.memory_usage().sum())
 
# Convert to more memory-efficient datatypes
df_large['A'] = pd.to_numeric(df_large['A'], downcast='integer')
df_large['B'] = pd.to_numeric(df_large['B'], downcast='float')
 
# Check memory usage after conversion
print("Memory usage after conversion:")
print(df_large.memory_usage().sum())

Output:

Memory usage before conversion:
16000128
Memory usage after conversion:
5000128

Load Less Data

Overview: This technique entails loading only the relevant columns from the dataset. This is especially helpful when working with datasets that have a lot of columns or when analysis just requires a portion of the data.
Benefits: Enhances processing effectiveness and uses less memory.
Implementation: To select which columns to load, use the usecols parameter in routines such as pd.read_csv().

Python3

import pandas as pd
 
# Create sample DataFrame
data = {'A': range(1000),
        'B': range(1000), 
        'C': range(1000), 
        'D': range(1000)}
 
# Load only specific columns
df = pd.DataFrame(data)
df_subset = df[['A', 'D']]
print('Specific Columns of the DataFrame')
print(df_subset.head())

Output:

Specific Columns of the DataFrame
   A  D
0  0  0
1  1  1
2  2  2
3  3  3
4  4  4

Sampling:

Sampling is the process of choosing a random selection of the dataset’s data for examination. This can be used to quickly analyze the dataset, explore it, or create models using a representative sample of the data.
Benefits: Makes analysis and experimentation faster, especially when working with big datasets.
Implementation: To randomly select rows or columns from the DataFrame, use Pandas’ sample() method.

Python3

import pandas as pd
 
# Create sample DataFrame
data = {'A': range(1000),
        'B': range(1000), 
        'C': range(1000), 
        'D': range(1000)}
 
# Sample 10% of the dataset
df = pd.DataFrame(data)
df_sample = df.sample(frac=0.1, random_state=42)
print(df_sample.head())

Output:

       A    B    C    D
521  521  521  521  521
737  737  737  737  737
740  740  740  740  740
660  660  660  660  660
411  411  411  411  411

Chunking:

Rather than loading the complete dataset into memory at once, chunking entails processing the dataset in smaller, more manageable parts. When working with datasets that are too big to fit in memory, this is quite helpful.
Benefits: Processes huge datasets on devices with limited memory and uses less memory.
Implementation: To specify the number of rows to read at a time, use the chunksize argument in routines such as pd.read_csv().

Python3

import pandas as pd
 
# Create sample DataFrame
data = {'A': range(10000),
        'B': range(10000)}
 
# Process data in chunks
chunk_size = 1000
for chunk in pd.DataFrame(data).groupby(pd.DataFrame(data).index // chunk_size):
    print(chunk)

Output:

(0,        A    B
0      0    0
1      1    1
2      2    2
3      3    3
4      4    4
..   ...  ...
995  995  995
996  996  996
997  997  997
998  998  998
999  999  999

[1000 rows x 2 columns])
(1,          A     B
1000  1000  1000
1001  1001  1001
1002  1002  1002
1003  1003  1003
1004  1004  1004
...    ...   ...
1995  1995  1995
1996  1996  1996
1997  1997  1997
1998  1998  1998
1999  1999  1999

[1000 rows x 2 columns])
(2,          A     B
2000  2000  2000
2001  2001  2001
2002  2002  2002
2003  2003  2003
2004  2004  2004
...    ...   ...
2995  2995  2995
2996  2996  2996
2997  2997  2997
2998  2998  2998
2999  2999  2999

[1000 rows x 2 columns])
(3,          A     B
3000  3000  3000
3001  3001  3001
3002  3002  3002
3003  3003  3003
3004  3004  3004
...    ...   ...
3995  3995  3995
3996  3996  3996
3997  3997  3997
3998  3998  3998
3999  3999  3999

Optimising Pandas dtypes:

Described as: Finding columns with data types that are not as efficient as possible and changing them to ones that are would save more memory. Performance can be greatly enhanced and memory utilization can be much decreased.
Benefits: Increases processing speed and minimizes memory footprint.
Implementation: To convert columns to more efficient data types, use the astype() method. To convert columns to datetime or numeric types, respectively, use functions such as pd.to_datetime() or pd.to_numeric().

Python3

import pandas as pd
 
# Create sample DataFrame
data = {'date_column': ['2022-01-01', '2022-01-02', '2022-01-03'],
        'numeric_column': [1.234, 2.345, 3.456]}
 
df = pd.DataFrame(data)
 
# Convert inefficient dtypes
df['date_column'] = pd.to_datetime(df['date_column'])
df['numeric_column'] = pd.to_numeric(df['numeric_column'], downcast='float')
 
print(df.dtypes)

date_column       datetime64[ns]
numeric_column           float32
dtype: object

Parallelising Pandas with Dask:

Dask is a package for parallel computing that works well with Pandas and offers parallelized operations for big datasets. Your Pandas workflows can be scaled across many cores or even distributed clusters with its help.
Advantages: Allows Pandas operations to be executed in parallel, greatly reducing processing times for huge datasets.
Implementation: To execute parallelized operations on sizable datasets, use Dask data structures like dask.DataFrame and dask.array. Dask facilitates the smooth transfer of current codebases to parallel execution by supporting the majority of the well-known Pandas APIs.

Python3

import dask.dataframe as dd
import pandas as pd
 
# Create sample DataFrame
data = {'A': range(10000),
        'B': range(10000)}
 
df = pd.DataFrame(data)
 
# Load data using Dask
ddf = dd.from_pandas(df, npartitions=4)
 
# Perform parallelized operations
result = ddf.groupby('A').mean().compute()
print(result)

           B
A           
0        0.0
1        1.0
2        2.0
3        3.0
4        4.0
...      ...
9995  9995.0
9996  9996.0
9997  9997.0
9998  9998.0
9999  9999.0
[10000 rows x 1 columns]

Tags:

#Python Programs

Working with database using Pandas

How To Save Python Scripts In Linux Via The Terminal?