Is koalas still being worked on? or is the project on pause at the moment?

[Trodenn]

As written in the title above, I want to know a bit more on what is the current status of the koalas package? the project my team is working is looking for to transition to spark, but since most of the people have pandas experience, koalas seemed the perfect thing to fit in. However, other than the simple data manipulations, I have noticed that there are still big differences where sometime what works in pandas does not work in koalas instead.

Is it a good idea to continue using koalas? or would it be better to convert towards Spark's Pandas package?

[HyukjinKwon]

It's encouraged to migrate to PySpark itself since PySpark has it now. Releases might happen per security issues or critical issues. Otherwise, nothing much will be updated.

[Trodenn]

Thank you very much for the fast reply. I will see to switching to pyspark pandas. I have tried it before but somehow I always have some difficulties with importing the package for the pandas package. I mainly work on data bricks, do you happen to know what to do when it does not recognise the pandas package that I want to import? Sincerely, Pierre Nanquette

…

On 5 Jan 2023 at 03:46 -0500, Hyukjin Kwon ***@***.***>, wrote: It's encouraged to migrate to PySpark itself since PySpark has it now. Releases might happen per security issues or critical issues. Otherwise, nothing much will be updated. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>