How to train with parquet files?

[MikkelWorkF]

Hello

How do I train mxnet with parquet files?
I have the training data stored in a bunch of parquet files(hundreds) and they cannot fit in memory (2TB+). Until now we have been able to not deal with the issue, because we could handle the training data in memory (we ran on 728GB memory sagemaker instances, but that is no longer sufficient)
We have been looking a long time for solutions, but nothing seems to be working. We are considering switching to PyTorch as that can handle a petastorm reader, which should work with parquet files. However, we feel like there has to be some solution we are not seeing.

[TristonC]

@szha Do you have someone to help with this question?

[szha]

If you are already familiar with petastorm, you can use the plain python reader of it and wrap the data into mxnet arrays using mx.nd.array

[MikkelWorkF]

I can use the make_batch_reader, but this does not give me random access(getitem), however I can get around that with a sequential sampler. I am still unable to provide a length though.