A Rising Library Beating Pandas in Performance


By Ezz El Din Abdullah, Data Platform Engineer

Header image


pandas was initially released in 2008 written in Python, Cython, and C. Today, we’re comparing the performance of this well-known library with pypolars, a rising DataFrame library written in Rust. We compare the two while sorting and concatenating a 25Mil-record data and also when joining two CSVs.


Downloading Reddit Usernames data

Let’s first download a CSV file that contains ~26 million reddit usernames from Kaggle: https://www.kaggle.com/colinmorris/reddit-usernames

And let’s form another CSV file that we will use, you can create it with your favorite text editor or through the command line:

$ cat >> fake_users.csv



Now, let’s compare the sorting algorithm of the two:


$ python sort_with_pandas.py 
Time:  34.35690135599998


$  python sort_with_pypolars.py 
Time:  23.965840922999973

~24 seconds meaning pypolars here is 1.4x faster than pandas



We’ll see now how each will perform while concatenating two data frames and stacking them into one data frame


18 seconds taken by pandas

$ python concat_with_pandas.py 
Time:  18.12720185799992


Here  pypolars is 1.2x faster

$ python concat_with_pypolars.py 
Time: 15.001723207000055



Downloading COVID Tracking data

Downloading data from COVID Tracking Project from this command:

$ curl -LO https://covidtracking.com/data/download/all-states-history.csv

will get you the latest data of the coronavirus spread across all the US in this all-states-history.csv file

Downloading states data

This is a CSV file indicating the abbreviations of each state since we need it to join with the previous CSV which has only the abbreviations listed in state column. Let’s get this data from this command:

$ curl -LO https://gist.githubusercontent.com/afomi/8824ddb02a68cf15151a804d4d0dc3b7/raw/5f1cfabf2e65c5661a9ed12af27953ae4032b136/states.csv

This will output states.csv, the file that has two columns: State and Abbreviation


Let’s use csvcut to filter out this resulted joined_pd.csv file:

$ csvcut -c date,state,State joined_pd.csv | head | csvlook 
|       date | state | State       |
| ---------- | ----- | ----------- |
| 2020-11-16 | AK    | ALASKA      |
| 2020-11-16 | AL    | ALABAMA     |
| 2020-11-16 | AR    | ARKANSAS    |
| 2020-11-16 | AS    |             |
| 2020-11-16 | AZ    | ARIZONA     |
| 2020-11-16 | CA    | CALIFORNIA  |
| 2020-11-16 | CO    | COLORADO    |
| 2020-11-16 | CT    | CONNECTICUT |
| 2020-11-16 | DC    |             |

Looks like the join is working and it’s left join. If you’re curious why the associated State values of AS and DC are nulls, that’s because there are no abbreviations existing in the states.csv file itself. If you look into the Abbreviation column, you’ll find no values for AS nor DC.

Here no AS abbreviations:

$ grep AS states.csv 

and here no values for DC:

P.S. if it’s not clear what csvcut is used for; we have some tutorials on csvkit (the command-line utility that contains csvcut and some other useful command-line tools for cleaning, processing, and analyzing CSVs).


$ python join_with_pypolars.py 
Time:  0.41163978699978543

Let’s see now how the joined data frame looks like:

$ csvcut -c date,state,State joined_pl.csv | head | csvlook 
|       date | state | State       |
| ---------- | ----- | ----------- |
| 2020-11-16 | AK    | ALASKA      |
| 2020-11-16 | AL    | ALABAMA     |
| 2020-11-16 | AR    | ARKANSAS    |
| 2020-11-16 | AZ    | ARIZONA     |
| 2020-11-16 | CA    | CALIFORNIA  |
| 2020-11-16 | CO    | COLORADO    |
| 2020-11-16 | CT    | CONNECTICUT |
| 2020-11-16 | DE    | DELAWARE    |
| 2020-11-16 | FL    | FLORIDA     |

so it looks like here pypolars missed the null values for the column it joined on, but that’s because the default join is inner join unlike pandas’ default join which is left join. To get the same result as pandas you need to change line 8 to:

df_all_states_history.join(df_states, left_on=”state”, right_on=”Abbreviation”, how=”left”).to_csv(“joined_pl.csv”)

which on my machine got ~317ms meaning here:

pypolars is 3x faster in left join


Final thoughts

In the end, we’ve found how performant pypolars is compared to pandas.  Of course, pandas is more mature since it’s been 12 years now and the community is still investing in it, but if more collaborations are made on pypolars; this rising library will rock!

You might find these tutorials useful:

Take care, will see you in the next tutorials 🙂


Click here to get fresh content to your inbox


Bio: Ezz El Din Abdullah (ezzeddinabdullah.com) is a Data Platform Engineer at Affectiva.

Original. Reposted with permission.


Read More …


Write a comment