library(nanoparquet)
library(pillar)
udf <- system.file("extdata/userdata1.parquet", package = "nanoparquet")The nanoparquet R package
Name: Gábor Csárdi Affiliation: Software Engineer at Posit, PBC
Outline
- Read and write Parquet files with nanoparquet
- Parquet features
- About nanoparquet
- The Parquet file format
- Future plans
1 Read and write Parquet files with nanoparquet
On GitHub: https://github.com/r-lib/nanoparquet/
Installation:
install.packages("nanoparquet")Use the example file that comes with the package:
Before reading the file, let’s look at its metadata:
read_parquet_info(udf)# A data frame: 1 × 7
file_name num_cols num_rows num_row_groups file_size parquet_version
<chr> <int> <dbl> <int> <dbl> <int>
1 /data/user-homes/a… 13 1000 1 73217 1
# ℹ 1 more variable: created_by <chr>
read_parquet_schema(udf)# A data frame: 14 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 /data/user-hom… sche… <NA> <NA> NA <NA> <NA>
2 /data/user-hom… regi… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
3 /data/user-hom… id integ… INT32 NA REQUIRED INT_32
4 /data/user-hom… firs… chara… BYTE… NA OPTIONAL UTF8
5 /data/user-hom… last… chara… BYTE… NA REQUIRED UTF8
6 /data/user-hom… email factor BYTE… NA OPTIONAL UTF8
7 /data/user-hom… gend… chara… BYTE… NA OPTIONAL UTF8
8 /data/user-hom… ip_a… chara… BYTE… NA REQUIRED UTF8
9 /data/user-hom… cc chara… BYTE… NA OPTIONAL UTF8
10 /data/user-hom… coun… chara… BYTE… NA REQUIRED UTF8
11 /data/user-hom… birt… Date INT32 NA OPTIONAL DATE
12 /data/user-hom… sala… double DOUB… NA OPTIONAL <NA>
13 /data/user-hom… title chara… BYTE… NA OPTIONAL UTF8
14 /data/user-hom… comm… chara… BYTE… NA OPTIONAL UTF8
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>Read the file:
ud1 <- read_parquet(udf)
ud1# A data frame: 1,000 × 13
registration id first_name last_name email gender ip_address cc
<dttm> <int> <chr> <chr> <chr> <fct> <chr> <chr>
1 2016-02-03 07:55:29 1 Amanda Jordan ajord… Female 1.197.201… 6759…
2 2016-02-03 17:04:03 2 Albert Freeman afree… Male 218.111.1… <NA>
3 2016-02-03 01:09:31 3 Evelyn Morgan emorg… Female 7.161.136… 6767…
4 2016-02-03 00:36:21 4 Denise Riley drile… Female 140.35.10… 3576…
5 2016-02-03 05:05:31 5 Carlos Burns cburn… <NA> 169.113.2… 5602…
6 2016-02-03 07:22:34 6 Kathryn White kwhit… Female 195.131.8… 3583…
7 2016-02-03 08:33:08 7 Samuel Holmes sholm… Male 232.234.8… 3582…
8 2016-02-03 06:47:06 8 Harry Howell hhowe… Male 91.235.51… <NA>
9 2016-02-03 03:52:53 9 Jose Foster jfost… Male 132.31.53… <NA>
10 2016-02-03 18:29:47 10 Emily Stewart estew… Female 143.28.25… 3574…
# ℹ 990 more rows
# ℹ 5 more variables: country <chr>, birthdate <date>, salary <dbl>,
# title <chr>, comments <chr>To show write_parquet(), we’ll use the flights data in the nycflights13 package:
library(nycflights13)
flights# A tibble: 336,776 × 19
year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
<int> <int> <int> <int> <int> <dbl> <int> <int>
1 2013 1 1 517 515 2 830 819
2 2013 1 1 533 529 4 850 830
3 2013 1 1 542 540 2 923 850
4 2013 1 1 544 545 -1 1004 1022
5 2013 1 1 554 600 -6 812 837
6 2013 1 1 554 558 -4 740 728
7 2013 1 1 555 600 -5 913 854
8 2013 1 1 557 600 -3 709 723
9 2013 1 1 557 600 -3 838 846
10 2013 1 1 558 600 -2 753 745
# ℹ 336,766 more rows
# ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
# tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
# hour <dbl>, minute <dbl>, time_hour <dttm>First we check how columns of flights will be mapped to Parquet types:
infer_parquet_schema(flights)# A data frame: 19 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 <NA> year integ… INT32 NA REQUIRED INT_32
2 <NA> month integ… INT32 NA REQUIRED INT_32
3 <NA> day integ… INT32 NA REQUIRED INT_32
4 <NA> dep_time integ… INT32 NA OPTIONAL INT_32
5 <NA> sched_dep_… integ… INT32 NA REQUIRED INT_32
6 <NA> dep_delay double DOUB… NA OPTIONAL <NA>
7 <NA> arr_time integ… INT32 NA OPTIONAL INT_32
8 <NA> sched_arr_… integ… INT32 NA REQUIRED INT_32
9 <NA> arr_delay double DOUB… NA OPTIONAL <NA>
10 <NA> carrier chara… BYTE… NA REQUIRED UTF8
11 <NA> flight integ… INT32 NA REQUIRED INT_32
12 <NA> tailnum chara… BYTE… NA OPTIONAL UTF8
13 <NA> origin chara… BYTE… NA REQUIRED UTF8
14 <NA> dest chara… BYTE… NA REQUIRED UTF8
15 <NA> air_time double DOUB… NA OPTIONAL <NA>
16 <NA> distance double DOUB… NA REQUIRED <NA>
17 <NA> hour double DOUB… NA REQUIRED <NA>
18 <NA> minute double DOUB… NA REQUIRED <NA>
19 <NA> time_hour POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>This looks fine, so we go ahead and write out the file. By default it will be Snappy-compressed, and many columns will be dictionary encoded.
write_parquet(flights, "flights.parquet")Check the schema of the file we created:
read_parquet_schema("flights.parquet")# A data frame: 20 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 flights.parquet sche… <NA> <NA> NA <NA> <NA>
2 flights.parquet year integ… INT32 NA REQUIRED INT_32
3 flights.parquet month integ… INT32 NA REQUIRED INT_32
4 flights.parquet day integ… INT32 NA REQUIRED INT_32
5 flights.parquet dep_… integ… INT32 NA OPTIONAL INT_32
6 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
7 flights.parquet dep_… double DOUB… NA OPTIONAL <NA>
8 flights.parquet arr_… integ… INT32 NA OPTIONAL INT_32
9 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
10 flights.parquet arr_… double DOUB… NA OPTIONAL <NA>
11 flights.parquet carr… chara… BYTE… NA REQUIRED UTF8
12 flights.parquet flig… integ… INT32 NA REQUIRED INT_32
13 flights.parquet tail… chara… BYTE… NA OPTIONAL UTF8
14 flights.parquet orig… chara… BYTE… NA REQUIRED UTF8
15 flights.parquet dest chara… BYTE… NA REQUIRED UTF8
16 flights.parquet air_… double DOUB… NA OPTIONAL <NA>
17 flights.parquet dist… double DOUB… NA REQUIRED <NA>
18 flights.parquet hour double DOUB… NA REQUIRED <NA>
19 flights.parquet minu… double DOUB… NA REQUIRED <NA>
20 flights.parquet time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>Or, to see the full metadata of the Parquet file:
read_parquet_metadata("flights.parquet")$file_meta_data
# A data frame: 1 × 5
file_name version num_rows key_value_metadata created_by
<chr> <int> <dbl> <I<list>> <chr>
1 flights.parquet 1 336776 <tbl [1 × 2]> https://github.com/gaborc…
$schema
# A data frame: 20 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 flights.parquet sche… <NA> <NA> NA <NA> <NA>
2 flights.parquet year integ… INT32 NA REQUIRED INT_32
3 flights.parquet month integ… INT32 NA REQUIRED INT_32
4 flights.parquet day integ… INT32 NA REQUIRED INT_32
5 flights.parquet dep_… integ… INT32 NA OPTIONAL INT_32
6 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
7 flights.parquet dep_… double DOUB… NA OPTIONAL <NA>
8 flights.parquet arr_… integ… INT32 NA OPTIONAL INT_32
9 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
10 flights.parquet arr_… double DOUB… NA OPTIONAL <NA>
11 flights.parquet carr… chara… BYTE… NA REQUIRED UTF8
12 flights.parquet flig… integ… INT32 NA REQUIRED INT_32
13 flights.parquet tail… chara… BYTE… NA OPTIONAL UTF8
14 flights.parquet orig… chara… BYTE… NA REQUIRED UTF8
15 flights.parquet dest chara… BYTE… NA REQUIRED UTF8
16 flights.parquet air_… double DOUB… NA OPTIONAL <NA>
17 flights.parquet dist… double DOUB… NA REQUIRED <NA>
18 flights.parquet hour double DOUB… NA REQUIRED <NA>
19 flights.parquet minu… double DOUB… NA REQUIRED <NA>
20 flights.parquet time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>
$row_groups
# A data frame: 1 × 7
file_name id total_byte_size num_rows file_offset total_compressed_size
<chr> <int> <dbl> <dbl> <dbl> <dbl>
1 flights.parq… 0 5684711 336776 NA NA
# ℹ 1 more variable: ordinal <int>
$column_chunks
# A data frame: 19 × 24
file_name row_group column file_path file_offset offset_index_offset
<chr> <int> <int> <chr> <dbl> <dbl>
1 flights.parquet 0 0 <NA> 23 NA
2 flights.parquet 0 1 <NA> 111 NA
3 flights.parquet 0 2 <NA> 323 NA
4 flights.parquet 0 3 <NA> 6738 NA
5 flights.parquet 0 4 <NA> 459931 NA
6 flights.parquet 0 5 <NA> 879534 NA
7 flights.parquet 0 6 <NA> 1292320 NA
8 flights.parquet 0 7 <NA> 1751079 NA
9 flights.parquet 0 8 <NA> 2216547 NA
10 flights.parquet 0 9 <NA> 2630778 NA
11 flights.parquet 0 10 <NA> 2813520 NA
12 flights.parquet 0 11 <NA> 3340805 NA
13 flights.parquet 0 12 <NA> 3844052 NA
14 flights.parquet 0 13 <NA> 3929450 NA
15 flights.parquet 0 14 <NA> 4226239 NA
16 flights.parquet 0 15 <NA> 4600386 NA
17 flights.parquet 0 16 <NA> 4937278 NA
18 flights.parquet 0 17 <NA> 5078965 NA
19 flights.parquet 0 18 <NA> 5378539 NA
# ℹ 18 more variables: offset_index_length <int>, column_index_offset <dbl>,
# column_index_length <int>, type <chr>, encodings <I<list>>,
# path_in_schema <I<list>>, codec <chr>, num_values <dbl>,
# total_uncompressed_size <dbl>, total_compressed_size <dbl>,
# data_page_offset <dbl>, index_page_offset <dbl>,
# dictionary_page_offset <dbl>, null_count <dbl>, min_value <I<list>>,
# max_value <I<list>>, is_min_value_exact <lgl>, is_max_value_exact <lgl>The columns chunk information also tells you whether a column chunk is dictionary encoded, its encoding, its size, etc.
cc <- read_parquet_metadata("flights.parquet")$column_chunks
cc[, c("column", "encodings", "dictionary_page_offset")]# A data frame: 19 × 3
column encodings dictionary_page_offset
<int> <I<list>> <dbl>
1 0 <chr [2]> 4
2 1 <chr [2]> 48
3 2 <chr [2]> 181
4 3 <chr [3]> 1445
5 4 <chr [2]> 455826
6 5 <chr [3]> 877389
7 6 <chr [3]> 1286655
8 7 <chr [2]> 1746406
9 8 <chr [3]> 2214210
10 9 <chr [2]> 2630682
11 10 <chr [2]> 2798121
12 11 <chr [3]> 3318738
13 12 <chr [2]> 3844016
14 13 <chr [2]> 3928888
15 14 <chr [3]> 4224174
16 15 <chr [2]> 4599437
17 16 <chr [2]> 4937172
18 17 <chr [2]> 5078689
19 18 <chr [2]> 5330993
cc[["encodings"]][1:5][[1]]
[1] "PLAIN" "RLE_DICTIONARY"
[[2]]
[1] "PLAIN" "RLE_DICTIONARY"
[[3]]
[1] "PLAIN" "RLE_DICTIONARY"
[[4]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[5]]
[1] "PLAIN" "RLE_DICTIONARY"2 Parquet features
Well supported
- R, Python, Rust, Java, Go, etc.
- Apache Arrow
- DuckDB
- R: Arrow, DuckDB, Polars, duckplyr
- Python: Arrow, DuckDB, Polars, fastparquet
- Positron data viewer (demo!)
Performant
- Columnar data storage. Columns or chunks of columns can be read efficiently.
- Several efficient encodings to keep data files small.
- Compression. Multiple types of compression in the same file.
- Skip columns and/or rows when only a subset of the data is needed.
- Designed for flexibility.
- Designed for parallel processing. Row groups, column chunks and pages can be processed (encoded/decoded and compressed/uncompressed) in parallel.
- Easy subsetting without reading the full file. (E.g. download only the required parts of the file from a URL.)
- Schema evolution: add and remove columns without re-encoding or even re-writing the existing data.
Rich data types
Low level (primitive) data types, encoded efficiently.
High level (logical) data types on top of this: UTF-8 strings, time stamps, JSON strings, enumeration type (factor), decimal numbers with arbitrary scale and precision, etc.
Missing values
- Parquet has built-in missing data support. Missing data is stored efficiently.
3 About nanoparquet
Why we created nanoparquet?
- Parquet tools are typically used for larger, out of memory data sets.
- Perception: Parquet is only for large data.
- We wanted to have a smaller tool that has no dependencies and is easy to install.
- Facilitate adoption of Parquet for smaller data sets, especially for teams that share data between multiple environments, e.g. R, Python, Java, etc.
nanoparquet features
- Completely dependency free. Compiles into an R package that is less than 1MB, in less than a minute.
- Read and write flat (i.e. non-nested) Parquet files.
- Can read most Parquet data types.
- Can write many R data types, including factors and temporal types to Parquet.
- Can read a subset of columns from a Parquet file.
- Can append a data frame to a Parquet file without first reading and then rewriting the whole file.
- Supports Snappy, Gzip and Zstd compression.
- Competitive with other tools in terms of speed, memory use and file size.
nanoparquet benchmarks
nanoparquet limitations
- Only flat tables, no
LISTorMAP, i.e. nested columns are not supported. - Some newer Parquet types are not supported:
GEOMETRY,GEOGRAPHY,VARIANT. - Cannot read a subset of the rows.
- Reading files from URLs is not supported.
- nanoparquet always reads the data (or the selected subset of it) into memory. It does not work with out-of-memory data in Parquet files like Apache Arrow and DuckDB does.
- No concurrency, both
read_parquet()andwrite_parquet()are single-threaded. - No encryption.
- Some compression codecs are not supported:
LZO,BROTLI,LZ4. - No checksum support. nanoparquet does not check or write checksums.
- No Bloom filter support.
- Cannot write some encodings. (It can read all Parquet encodings, though!)
4 The Parquet file format
Columnar data storage
Data is stored column-wise, so whole columns (or large chunks of columns) are easy to read quickly. Columnar storage allows better compression, fast operations on subsets of columns, and easy ways of removing columns or adding new columns to a data file.
┌───────────────────────────┐
│ HEADER (4 bytes) │
├───────────────────────────┤
│ COLUMN 1 │
├───────────────────────────┤
│ COLUMN 2 │
├───────────────────────────┤
···
├───────────────────────────┤
│ COLUMN n │
├───────────────────────────┤
│ METADATA │
├───────────────────────────┤
│ METADATA LENGTH (4 bytes) │
├───────────────────────────┤
│ FOOTER (4 bytes) │
└───────────────────────────┘Row groups
A horizontal partitioning of the data. Contains one column chunk for each column in the dataset.
┌─────────────────────────────────┐
│ HEADER (4 bytes) │
├──────────────┬──────────────────┤
│ ROW GROUP 1 │ COLUMN CHUNK 1 │
│ │ ··· │
│ │ COLUMN CHUNK n │
├──────────────┴──────────────────┤
···
├──────────────┬──────────────────┤
│ ROW GROUP m │ COLUMN CHUNK 1 │
│ │ ··· │
│ │ COLUMN CHUNK n │
├──────────────┴──────────────────┤
│ METADATA │
├─────────────────────────────────┤
│ METADATA LENGTH (4 bytes) │
├─────────────────────────────────┤
│ FOOTER (4 bytes) │
└─────────────────────────────────┘Metadata
┌────────────────────────────────────────┐
│ File metadata: │
│ - version │
│ - schema │
│ - key-value metadata │
├──────────────┬─────────────────────────┤
│ Row group 1 │ Column chunk 1 metadata │
│ metadata │ ··· │
│ │ Column chunk n metadata │
├──────────────┴─────────────────────────┤
···
├──────────────┬─────────────────────────┤
│ Row group m │ Column chunk 1 metadata │
│ metadata │ ··· │
│ │ Column chunk n metadata │
└──────────────┴─────────────────────────┘Column chunk metadata
┌───────────────────────────┐
│ Encoding │
│ Compression codec │
│ Offset of first data page │
│ Key-value metadata │
│ ··· │
└───────────────────────────┘Detailed file format
(Image from https://github.com/apache/parquet-format.)
Parquet data types
https://nanoparquet.r-lib.org/reference/nanoparquet-types.html
Primitive types
- BOOLEAN
- INT32
- INT64
- INT96 (deprecated)
- FLOAT
- DOUBLE
- BYTE ARRAY
- FIXED LENGTH BYTE ARRAY
Logical types
- STRING (BYTE ARRAY)
- ENUM (BYTE ARRAY)
- UUID (FIXED LENGTH BYTE ARRAY)
- INT(8 | 16 | 32, signed | unsigned) etc. (INT32 or INT64)
- DECIMAL (scale, precision) (INT32, INT64, BYTE ARRAY or FIXED LENGTH BYTE ARRAY)
- FLOAT16 (FIXED LENGTH BYTE ARRAY)
- DATE (INT32)
- TIME (millis | micros | nanos) (INT32, INT64)
- TIMESTAMP (UTC, millis | micros | nanos) (INT64)
- INTERVAL
- JSON
- BSON
- VARIANT
- GEOMETRY
- GEOGRAPHY
- LIST
- MAP
- UNKNOWN
nanoparquet example
E.g. the flights table’s carrier column was written as a string (STRING), because that’s the default for character columns:
read_parquet_schema("flights.parquet")# A data frame: 20 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 flights.parquet sche… <NA> <NA> NA <NA> <NA>
2 flights.parquet year integ… INT32 NA REQUIRED INT_32
3 flights.parquet month integ… INT32 NA REQUIRED INT_32
4 flights.parquet day integ… INT32 NA REQUIRED INT_32
5 flights.parquet dep_… integ… INT32 NA OPTIONAL INT_32
6 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
7 flights.parquet dep_… double DOUB… NA OPTIONAL <NA>
8 flights.parquet arr_… integ… INT32 NA OPTIONAL INT_32
9 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
10 flights.parquet arr_… double DOUB… NA OPTIONAL <NA>
11 flights.parquet carr… chara… BYTE… NA REQUIRED UTF8
12 flights.parquet flig… integ… INT32 NA REQUIRED INT_32
13 flights.parquet tail… chara… BYTE… NA OPTIONAL UTF8
14 flights.parquet orig… chara… BYTE… NA REQUIRED UTF8
15 flights.parquet dest chara… BYTE… NA REQUIRED UTF8
16 flights.parquet air_… double DOUB… NA OPTIONAL <NA>
17 flights.parquet dist… double DOUB… NA REQUIRED <NA>
18 flights.parquet hour double DOUB… NA REQUIRED <NA>
19 flights.parquet minu… double DOUB… NA REQUIRED <NA>
20 flights.parquet time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>If you want to write it as an ENUM, you need to customize write_parquet() :
write_parquet(flights, "flights2.parquet", schema = parquet_schema(carrier = "ENUM"))Double check:
read_parquet_schema("flights2.parquet")# A data frame: 20 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 flights2.parqu… sche… <NA> <NA> NA <NA> <NA>
2 flights2.parqu… year integ… INT32 NA REQUIRED INT_32
3 flights2.parqu… month integ… INT32 NA REQUIRED INT_32
4 flights2.parqu… day integ… INT32 NA REQUIRED INT_32
5 flights2.parqu… dep_… integ… INT32 NA OPTIONAL INT_32
6 flights2.parqu… sche… integ… INT32 NA REQUIRED INT_32
7 flights2.parqu… dep_… double DOUB… NA OPTIONAL <NA>
8 flights2.parqu… arr_… integ… INT32 NA OPTIONAL INT_32
9 flights2.parqu… sche… integ… INT32 NA REQUIRED INT_32
10 flights2.parqu… arr_… double DOUB… NA OPTIONAL <NA>
11 flights2.parqu… carr… chara… BYTE… NA REQUIRED ENUM
12 flights2.parqu… flig… integ… INT32 NA REQUIRED INT_32
13 flights2.parqu… tail… chara… BYTE… NA OPTIONAL UTF8
14 flights2.parqu… orig… chara… BYTE… NA REQUIRED UTF8
15 flights2.parqu… dest chara… BYTE… NA REQUIRED UTF8
16 flights2.parqu… air_… double DOUB… NA OPTIONAL <NA>
17 flights2.parqu… dist… double DOUB… NA REQUIRED <NA>
18 flights2.parqu… hour double DOUB… NA REQUIRED <NA>
19 flights2.parqu… minu… double DOUB… NA REQUIRED <NA>
20 flights2.parqu… time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>Encodings
The data of a column chunk is stored in pages, using one of the possible data encodings (not a complete list):
- Plain encoding. Dump values of the column to the data page back to back. This encoding makes sense if there are no repeated values in a column, and the range of values is also very wide.
- Dictionary encoding. Pages within a column chunk may be dictionary encoded. This is handy if there are many repetitions of a handful of possible values, especially if the values are lengthy. E.g. a factor column. The first page is a special page that defines the dictionary for this column chunk and the subsequent pages contain dictionary indices.
- RLE-BP: run-length encoding + bit packing. Data, or more commonly, repetition and definition levels, or dictionary indices and be run-length-encoded. This is handy for repeated values. RLE encoding is actually a hybrid run length encoding with bit packing, so it is also efficient when only a small subset of possible values are used in the data, or the dictionary indices.
- Delta encoding. Encode the data or the dictionary indices as differences to an initial value. E.g. it can encode a natural sequence of numbers very efficiently.
Encodings are very important to read and write data (space- and time-) efficiently. nanoparquet::write_parquet() chooses an encoding automatically, but you can override this.
nanoparquet example
By default most columns in flights are dictionary encoded, because nanoparquet detected repetition:
read_parquet_schema("flights.parquet")# A data frame: 20 × 12
file_name name r_type type type_length repetition_type converted_type
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 flights.parquet sche… <NA> <NA> NA <NA> <NA>
2 flights.parquet year integ… INT32 NA REQUIRED INT_32
3 flights.parquet month integ… INT32 NA REQUIRED INT_32
4 flights.parquet day integ… INT32 NA REQUIRED INT_32
5 flights.parquet dep_… integ… INT32 NA OPTIONAL INT_32
6 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
7 flights.parquet dep_… double DOUB… NA OPTIONAL <NA>
8 flights.parquet arr_… integ… INT32 NA OPTIONAL INT_32
9 flights.parquet sche… integ… INT32 NA REQUIRED INT_32
10 flights.parquet arr_… double DOUB… NA OPTIONAL <NA>
11 flights.parquet carr… chara… BYTE… NA REQUIRED UTF8
12 flights.parquet flig… integ… INT32 NA REQUIRED INT_32
13 flights.parquet tail… chara… BYTE… NA OPTIONAL UTF8
14 flights.parquet orig… chara… BYTE… NA REQUIRED UTF8
15 flights.parquet dest chara… BYTE… NA REQUIRED UTF8
16 flights.parquet air_… double DOUB… NA OPTIONAL <NA>
17 flights.parquet dist… double DOUB… NA REQUIRED <NA>
18 flights.parquet hour double DOUB… NA REQUIRED <NA>
19 flights.parquet minu… double DOUB… NA REQUIRED <NA>
20 flights.parquet time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC…
# ℹ 5 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
# precision <int>, field_id <int>
fc <- read_parquet_metadata("flights.parquet")$column_chunks
fc# A data frame: 19 × 24
file_name row_group column file_path file_offset offset_index_offset
<chr> <int> <int> <chr> <dbl> <dbl>
1 flights.parquet 0 0 <NA> 23 NA
2 flights.parquet 0 1 <NA> 111 NA
3 flights.parquet 0 2 <NA> 323 NA
4 flights.parquet 0 3 <NA> 6738 NA
5 flights.parquet 0 4 <NA> 459931 NA
6 flights.parquet 0 5 <NA> 879534 NA
7 flights.parquet 0 6 <NA> 1292320 NA
8 flights.parquet 0 7 <NA> 1751079 NA
9 flights.parquet 0 8 <NA> 2216547 NA
10 flights.parquet 0 9 <NA> 2630778 NA
11 flights.parquet 0 10 <NA> 2813520 NA
12 flights.parquet 0 11 <NA> 3340805 NA
13 flights.parquet 0 12 <NA> 3844052 NA
14 flights.parquet 0 13 <NA> 3929450 NA
15 flights.parquet 0 14 <NA> 4226239 NA
16 flights.parquet 0 15 <NA> 4600386 NA
17 flights.parquet 0 16 <NA> 4937278 NA
18 flights.parquet 0 17 <NA> 5078965 NA
19 flights.parquet 0 18 <NA> 5378539 NA
# ℹ 18 more variables: offset_index_length <int>, column_index_offset <dbl>,
# column_index_length <int>, type <chr>, encodings <I<list>>,
# path_in_schema <I<list>>, codec <chr>, num_values <dbl>,
# total_uncompressed_size <dbl>, total_compressed_size <dbl>,
# data_page_offset <dbl>, index_page_offset <dbl>,
# dictionary_page_offset <dbl>, null_count <dbl>, min_value <I<list>>,
# max_value <I<list>>, is_min_value_exact <lgl>, is_max_value_exact <lgl>
fc$encodings[[1]]
[1] "PLAIN" "RLE_DICTIONARY"
[[2]]
[1] "PLAIN" "RLE_DICTIONARY"
[[3]]
[1] "PLAIN" "RLE_DICTIONARY"
[[4]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[5]]
[1] "PLAIN" "RLE_DICTIONARY"
[[6]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[7]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[8]]
[1] "PLAIN" "RLE_DICTIONARY"
[[9]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[10]]
[1] "PLAIN" "RLE_DICTIONARY"
[[11]]
[1] "PLAIN" "RLE_DICTIONARY"
[[12]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[13]]
[1] "PLAIN" "RLE_DICTIONARY"
[[14]]
[1] "PLAIN" "RLE_DICTIONARY"
[[15]]
[1] "RLE" "PLAIN" "RLE_DICTIONARY"
[[16]]
[1] "PLAIN" "RLE_DICTIONARY"
[[17]]
[1] "PLAIN" "RLE_DICTIONARY"
[[18]]
[1] "PLAIN" "RLE_DICTIONARY"
[[19]]
[1] "PLAIN" "RLE_DICTIONARY"We can force write_parquet() to write them in PLAIN encoding. (Most often you should not do this.)
write_parquet(flights, "flights3.parquet", encoding = "PLAIN")
fs::file_info(c("flights.parquet", "flights3.parquet"))# A tibble: 2 × 18
path type size permissions modification_time user group device_id
<fs::path> <fct> <fs::> <fs::perms> <dttm> <chr> <chr> <dbl>
1 …ts.parquet file 5.42M rw-r--r-- 2026-03-12 18:00:18 andr… data… 2.35e9
2 …s3.parquet file 11.4M rw-r--r-- 2026-03-12 18:00:19 andr… data… 2.35e9
# ℹ 10 more variables: hard_links <dbl>, special_device_id <dbl>, inode <dbl>,
# block_size <dbl>, blocks <dbl>, flags <int>, generation <dbl>,
# access_time <dttm>, change_time <dttm>, birth_time <dttm>PLAIN encoding might also take longer to read:
bench::mark(
dict = read_parquet("flights.parquet"),
plain = read_parquet("flights3.parquet")
)# A tibble: 2 × 6
expression min median `itr/sec` mem_alloc `gc/sec`
<bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl>
1 dict 28.3ms 37.1ms 27.0 50.1MB 162.
2 plain 66.7ms 73.3ms 13.6 50.1MB 40.95 Future plans
Remote files
Support reading Parquet files over HTTP.
Support reading Parquet metadata over HTTP.
Subsetting rows
- Support reading a subset of rows into the memory.
Parallel Parquet reader and writer
- Make
read_parquet()andwrite_parquet()faster by processing row groups, column chunks, pages in parallel.
Schema evolution
- Support adding and removing columns efficiently.
Nested types
- Support reading and writing nested Parquet files, with
LISTandMAPcolumns.
The missing bits
Other new data types:
VARIANT,GEOMETRY,GEOGRAPHY.Add missing compression algorithms.
Add missing encodings to
write_parquet().Support encryption.
Checksumming.
Support using and writing Bloom filters.
ALTREP
- Support lazy-loading parts of Parquet files.
Manual type mapping
Allow specifying the Parquet -> R type mappings manually in
read_parquet().Already supported for
write_parquet().
Multi-file support
- Support Hive partitioning: split up a data set into multiple files, based on partition keys.
nanoparquet as a C++ library
nanoparquet Python package
nanoparquet for WebAssembly
6 Links
- nanoparquet on GitHub: https://github.com/r-lib/nanoparquet/
- Parquet file format: https://github.com/apache/parquet-format
- Apache Arrow: https://arrow.apache.org/
- DuckDB: https://duckdb.org/
- Polars: https://pola.rs/
- duckplyr: https://duckplyr.tidyverse.org/