Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

 
About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 
HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2:how-to-make-fast-small-grib2-files
 

How to make small fast grib2 databases

Introduction

The old joke is that an engineer tells his client that he can design a gizmo that is fast, cheap and reliable, and the client gets to choose two. The same principle applies to many projects including making a grib-2 database. Here, the desired qualities are small file size, speed for the users and effort required to set up the database. For small databases, you probably want to minimize the set-up effort. For larger databases, all three factors come into play. Here are some hints for minimizing size/speed-of-use. They all require some set-up effort.

Making a smaller database

  1. Have unneeded fields?
    Use wgrib2 to remove records
  2. Smaller grids take less space
    Use wgrib2 or some other program to make regional subsets
  3. Data stored at higher than needed precision?
    Use wgrib2 to reduce the precision
  4. Better compression available?
    Use wgrib2 or cnvgrib
  5. use submessages
    Use wgrib2

Faster to read

  1. Jpeg2000 compression is makes small files but very slow to read.
    I am finding that complex packing with spatial differencing is a
    good trade off between speed and size. If the field has undefined values
    then complex packing is smaller than Jpeg2000 and faster to read.
  2. Decode routines that have been parallelized are faster. Wgrib2 has
    parallel packing and complex packing without bitmaps routines.
  3. Smaller grids are faster
  4. Reorder the records to maximize sequential access
    Use sort and wgrib2. This is important for optical media.
  5. Use an index/inventory.
  6. Too many submessages can use up available memory.

Minimizing Effort in Setting up the Database

  1. Save grib data as provided
  2. Writing grib encoding software?
    IEEE Grib-2 is the easiest format to write. It's produces big files and is not compatible with some grib-2 decoders.
    You can use wgrib2 to convert from ieee grib2 to a more common packing.

Effects of Compression on Sample Files


45x91 grid, 5 fields

         simple complex1 complex2 complex3    jpeg
size      21884    16030    13623    13917   11514  bytes
read      0.001    0.001    0.001    0.001   0.012  seconds user time
write     0.002    0.006    0.006    0.006   0.017  seconds user time

note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.

4320x2160 high resolution SST field

         simple complex1 complex2 complex3    jpeg
size      11054    4076      2365     2280    3252  K bytes
read       0.38    0.41      0.42     0.42    2.66  seconds user time
write      0.78    2.06      1.76     1.68    4.20  seconds user time

note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.

The commands to change the packing:

wgrib2 in.grb -set_grib_type X -grib_out out.grb
X=simple, complex1, complex2, complex3, jpeg


NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5830 University Research Court
College Park, Maryland 20740
Climate Prediction Center Web Team
Page last modified: Aug 7, 2009
Disclaimer Privacy Policy