Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem to be meant for either pure text or pure numerical data.

There are those that can handle data with the dual data types, but those are generally not flexible for wrangling. For example, for pickle you have to load the entire thing into memory if you want to wrangle anything. You can just append directly to the disk like you can with hdf5, which can be very helpful for huge datasets which can not be all loaded into memory?

Also, any alternatives to Pandas for wrangling Huge datasets? Sometimes you can't load all the data into Pandas without causing a memory crash.


Is your question about storing, writing, or reading/processing huge data? I'm not an expert in this topic, but I know a couple of possible ways to handle huge datasets:

  1. If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in the article Why and How to Use Pandas with Large Data, section 1. Read CSV file data in chunk size. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.

  2. If the data is really big, probably it's better to store and process it by multiple computers. A specific tool depends on what you want to do with the data, for some "plain" tasks online DBMSs like Google BigQuery can be used, otherwise you need something more complex, like Apache Hadoop or a custom system.

So, as always, it is a matter of your opportunities and your aims of dealing with the data.

| improve this answer | |
  • $\begingroup$ +1 Usually we ask for clarification in a comment before answering, though. This prevents you from wasting your time if you happen to have guessed wrong. $\endgroup$ – Philip Raeisghasem May 3 '19 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.