How to disable statistics in version 1.13.1?

[felipepessoto]

Describe the usage question you have. Please include as many useful details as possible.

I found these two PRs to disable statistics, but they are available only in 1.15+ 
#2989
#3056

Is there any other way to disable statistics in 1.13.1?

Component(s)

Core

[wgtmac]

Thanks for reporting this! You're right, those mentioned commits are not ported to legacy branches. Could you use the version 1.15.0?

[felipepessoto]

I can't because I'm using Spark 3.5.

I was wondering if possible to somehow disable stats in 1.13.1, even if it is more complex than just setting a flag.

[wgtmac]

I agree with you that it seems impossible to disable stats in the past. Therefore I made those PRs to fix them. Perhaps you may want to set parquet.statistics.truncate.length and parquet.columnindex.truncate.length to 0 to disable stats of BYTE_ARRAY type. These flags can be found at https://github.com/apache/parquet-java/tree/master/parquet-hadoop

[felipepessoto]

Thanks. I've set parquet.statistics.truncate.length to 1. The parquet.columnindex.truncate.length I wasn't sure how it works so I am not setting it, but using the parquet.statistics.truncate.length also worked for byte array