How to make small fast grib2 databases
Introduction
The old joke is that an engineer tells his client that he
can design a gizmo that is fast, cheap and reliable, and
the client gets to choose two. The same principle applies
to many projects including making a grib-2 database. Here,
the desired qualities are small file size, speed for the users
and effort required to set up the database. For small databases,
you probably want to minimize the set-up effort. For larger
databases, all three factors come into play. Here are some
hints for minimizing size/speed-of-use. They all require some
set-up effort.
Making a smaller database
-
Have unneeded fields?
Use wgrib2 to remove records
- Smaller grids take less space
Use wgrib2 or some other program to make regional subsets
- Data stored at higher than needed precision?
Use wgrib2 to reduce the precision
- Better compression available?
Use wgrib2 or cnvgrib
- use submessages
Use wgrib2
Faster to read
- Jpeg2000 compression is makes small files but very slow to read.
I am finding that complex packing with spatial differencing is a
good trade off between speed and size. If the field has undefined values
then complex packing is smaller than Jpeg2000 and faster to read.
- Decode routines that have been parallelized are faster. Wgrib2 has
parallel packing and complex packing without bitmaps routines.
- Smaller grids are faster
- Reorder the records to maximize sequential access
Use sort and wgrib2. This is important for optical media.
- Use an index/inventory.
- Too many submessages can use up available memory.
Minimizing Effort in Setting up the Database
- Save grib data as provided
- Writing grib encoding software?
IEEE Grib-2 is the easiest format to write. It's produces
big files and is not compatible with some grib-2 decoders.
You can use wgrib2 to convert from ieee grib2 to a more common packing.
Effects of Compression on Sample Files
45x91 grid, 5 fields
simple complex1 complex2 complex3 jpeg
size 21884 16030 13623 13917 11514 bytes
read 0.001 0.001 0.001 0.001 0.012 seconds user time
write 0.002 0.006 0.006 0.006 0.017 seconds user time
note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.
4320x2160 high resolution SST field
simple complex1 complex2 complex3 jpeg
size 11054 4076 2365 2280 3252 K bytes
read 0.38 0.41 0.42 0.42 2.66 seconds user time
write 0.78 2.06 1.76 1.68 4.20 seconds user time
note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.
The commands to change the packing:
wgrib2 in.grb -set_grib_type X -grib_out out.grb
X=simple, complex1, complex2, complex3, jpeg
|