Surface ¶

Surface

Data directory structure ¶

The Surface data is organized as xml files every 15 minutes

> ls -la /g3/tianwh/share/conobs/rec_RSURF_20190*
-rw-r--r-- 1 tianwh obs 11482070 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060100_g_06.dat
-rw-r--r-- 1 tianwh obs 11654320 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060106_g_06.dat
-rw-r--r-- 1 tianwh obs 11637320 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060112_g_06.dat
-rw-r--r-- 1 tianwh obs 11630570 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060118_g_06.dat
-rw-r--r-- 1 tianwh obs 11549820 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060200_g_06.dat
-rw-r--r-- 1 tianwh obs 11672570 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060206_g_06.dat
-rw-r--r-- 1 tianwh obs 11687820 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060212_g_06.dat
-rw-r--r-- 1 tianwh obs 11619320 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060218_g_06.dat
-rw-r--r-- 1 tianwh obs 11566570 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060300_g_06.dat
-rw-r--r-- 1 tianwh obs 11562820 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060306_g_06.dat
-rw-r--r-- 1 tianwh obs 11619320 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060312_g_06.dat
-rw-r--r-- 1 tianwh obs 11468820 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060318_g_06.dat
-rw-r--r-- 1 tianwh obs 11398070 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060400_g_06.dat
-rw-r--r-- 1 tianwh obs 11605820 9月  25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat
......

Data format ¶

CIMISS format data is looks like

RSURF 2019 6 1 0 45928 6 9 9 00081 81.00 -5.53 -36.87 19.00 1.00 100.00

1013.20 23.05 21.35 347.00 0.00 999999.00 999999.00 999999.00 1011.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00

00081 81.00 -5.53 -36.87 19.00 1.00 0.00: 1012.80 24.15 21.75 345.00 0.10 999999.00 999999.00 999999.00 1010.60 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00081 81.00 -5.53 -36.87 19.00 1.00 200.00: 1012.90 22.25 21.25 333.00 0.00 999999.00 999999.00 999999.00 1010.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 0.00: 1016.00 20.75 14.25 289.00 1.80 999999.00 999999.00 999999.00 882.20 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 100.00: 1016.30 20.25 14.35 306.00 1.50 999999.00 999999.00 999999.00 882.30 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 31.00 2100.00: 1015.10 21.35 13.95 288.00 3.90 999999.00 999999.00 999999.00 881.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 200.00: 1016.30 20.45 14.35 283.00 2.50 999999.00 999999.00 999999.00 882.40 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
01001 1001.00 70.94 -8.67 10.00 31.00 2200.00: 1015.80 -0.40 -1.80 341.00 10.70 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 200.00: 1014.60 -0.40 -1.97 337.00 10.30 999999.00 999999.00 999999.00 1013.40 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 100.00: 1015.00 -0.40 -2.06 340.00 11.60 999999.00 999999.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 31.00 2300.00: 1015.20 -0.40 -2.23 341.00 10.40 999999.00 999999.00 999999.00 1014.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 31.00 2100.00: 1015.80 -0.20 -1.89 340.00 13.60 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 0.00: 1015.00 -0.20 -1.58 344.00 12.10 999999.00 0.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 100.00: 1020.50 -1.90 -2.89 109.00 6.80 999999.00 999999.00 999999.00 1019.70 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2200.00: 1020.80 -0.20 -2.30 89.00 6.90 999999.00 999999.00 999999.00 1020.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2300.00: 1020.70 -1.20 -2.59 111.00 6.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2100.00: 1020.70 0.00 -1.96 85.00 7.20 999999.00 999999.00 999999.00 1019.90 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 0.00: 1020.70 -1.90 -3.02 119.00 7.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 200.00: 1020.30 -1.70 -2.96 111.00 8.60 999999.00 999999.00 999999.00 1019.50 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 1.00 100.00: 1014.70 -1.30 -2.83 110.00 13.70 999999.00 999999.00 999999.00 1013.30 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2300.00: 1014.40 1.50 -3.16 62.00 7.60 999999.00 999999.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2200.00: 1014.00 1.40 -3.26 59.00 12.50 999999.00 999999.00 999999.00 1012.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2100.00: 1013.50 1.40 -3.45 57.00 13.10 999999.00 999999.00 999999.00 1012.10 0.00 1.00 1.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 1.00 0.00: 1014.40 1.10 -2.99 53.00 9.80 999999.00 0.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00

Data extractor ¶

A python code is used to extract the desired information from this CIMISS file:

> cd /g5/nwpc_ep8/obsproc/nwprod/decoders
> rm cimiss_data_surface
> ./read_nmc_cimiss_rsurf_rgwst.py -f  /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat

If you want to batch process number of CIMISS files, you can use following command:

> # This command will find all CIMISS files and prcessing the file one by one
> rm cimiss_data_surface
> decod_nmc_cimiss_rsing.py -d /g3/tianwh/share/conobs

The information we want to extract from CIMISS is.

import fortranformat as ff
output_line = ff.FortranRecordWriter('(I8, 3f10.2, 5A8, 2f10.2, 2I8, f10.2)')
output.write(output_line.write([id, lat, lon, elv, obstime.strftime("%Y"), obstime.strftime("%m"), \
        obstime.strftime("%d"), obstime.strftime("%H"), obstime.strftime("%M"), \
        p, T, round(rh), round(dd), ff]) + "\n")

the content of cimiss_data_surface is:

> less cimiss_data_surface
  56076 33.85 102.6333 3429 2018 9 1 16 0 674.7 10.8 97 149 3.4
  674002 40.6644 109.6661 1052 2018 9 1 16 0 890.2 15 95 2 1.3
  735904 31.0231 118.4314 9 2018 9 1 16 0 999999 26.4 999999 158 1.7
  868552 34.55 109.2975 357 2018 9 1 16 0 965.8 24.9 74 356 0.9
  718262 22.8133 112.4264 49 2018 9 1 16 0 999999 25.4 999999 225 0
  708128 26.8458 116.7783 343 2018 9 1 16 0 970.6 22.9 96 176 1.1
  737152 30.7728 117.3278 28.3 2018 9 1 16 0 1005.9 26.3 97 135 0.6
  756060 29.1036 119.8056 58 2018 9 1 16 0 999999 26.5 999999 210 1.1
  804130 27.4056 110.4781 200 2018 9 1 16 0 999999 23.1 999999 999999 999999
  672002 43.2972 116.1158 1284 2018 9 1 16 0 860 13.9 94 217 7.6
  663689 40.2278 119.22 300 2018 9 1 16 0 999999 19.2 999999 999999 999999
  752628 30.1492 121.1961 5 2018 9 1 16 0 999999 28.1 999999 237 1.6
  860410 34.9769 108.5269 1193 2018 9 1 16 0 999999 19.8 999999 999999 999999
  868283 34.2217 108.5681 394.5 2018 9 1 16 0 959.8 26.9 64 302 1
  754019 29.9078 120.4844 59 2018 9 1 16 0 999999 24.6 999999 324 0
  878120 36.0133 108.1183 1129 2018 9 1 16 0 876.3 17.5 99 329 1.4
  751675 29.6344 119.5692 95 2018 9 1 16 0 999999 24.7 999999 333 0

Decoder source code ¶

Source code directory:

> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/sorc

Subroutines to decode surface data
- lsdcod.f

注解

The cimiss.tbl and decod_WMO.Res40.headers are not used, although they are required as arguments and read in.
The path and file name of cimiss_data_surface file are hard coded in the subroutines.

4. Compile the code

> make

Decode and convert to BUFR format ¶

1. enter into the exec directory

> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/exec
> ls -la
total 2028
drwxr-xr-x. 3 zwtd zwtd     211 Dec 20 22:45 .
drwxr-xr-x. 6 zwtd zwtd      82 Dec 20 22:42 ..
lrwxrwxrwx. 1 zwtd zwtd      34 Dec  9 23:48 bufrtab.000 -> ../../decod_shared/fix/bufrtab.000
-rw-r--r--. 1 zwtd zwtd 1121683 Dec  9 23:48 cimiss.tbl
lrwxrwxrwx. 1 zwtd zwtd      31 Dec  9 23:48 decod_WMO.Res40.headers -> ../parm/decod_WMO.Res40.headers
-rwxr-xr-x. 1 zwtd zwtd  940696 Dec 20 00:24 decod_dccimiss
-rw-r--r--. 1 zwtd zwtd     462 Dec 20 22:45 decod_dccimiss.log
-rwxr-xr-x. 1 zwtd zwtd     484 Dec 20 00:30 run.ksh
-rwxr-xr-x. 1 zwtd zwtd     776 Dec 20 22:44 run_dccimisssurf.py
drwxr-xr-x. 2 zwtd zwtd      58 Dec 20 22:45 tmp

we provide a script to run the decoder in batch mode:

> ./run_dccimisssurf.py -s 2018121600 -e 2018121700 -i 1

注解

given the starting datetime and ending datetime, it iterates all cycles (every 1 hours)
the units of interval is hour (-i)
this script call run.ksh

3. run the decoder script

> run.ksh

> cat run.ksh
!/bin/bash
export DBNBUFRT=120
export TRANJB=/g5/nwpc_ep8/obsproc/nwprod/ush/tranjb
export tank_dir=/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003
export DBNROOT=`pwd`
rm tmp/*
rm decod_dccimiss.log
./decod_dccimiss -d decod_dccimiss.log -b 240 -c $1 bufrtab.000 cimiss.tbl decod_WMO.Res40.headers
ls -la tmp/*

BUFR_FILES=$(echo tmp/BUFR*)
echo ${BUFR_FILES}

for file in ${BUFR_FILES}
do
  ${TRANJB} ${tank_dir} ${file}
done

注解

-c $1 : Set the current time (201809011600) used to calculate the time departures of the obs. data.
-b 240 : Number of hours to decode prior to “current” time (default)
The observations with date/time between current time - 240 hours and current time + 3 are kept.

4. The generated BUFR format file will be saved at

> ls -la tmp
-rw-r--r--  1 xinzhang  staff  4199744 Sep 21 18:45 tmp/BUFR.0.cimiss.1.6436.1536097072.8

Transfer bufr data to BUFR Tanks ¶

put data in BUFR tanks:

> /g5/nwpc_ep8/obsproc/nwprod/ush/tranjb /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 tmp/BUFR.0.cimiss.1.6436.1536097072.8

> ls -al /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx01
-rw-r--r-- 1 vagrant vagrant 4235328 Sep 27 04:42 /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001

注解

if environmental variable SCREEN=ON :
- Define Run Time is the system time when the tranjb is running.
- Only observations with date/time between Run Time - 10 days and Run Time + 12 hours are kept.
for retrospective run, set SCREEN=OFF
/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/yyyymmdd/bmmm/xxsss (where mmm is WMO BUFR message type and xxx is local BUFR message subtype)
000.001 (in dump group mnemonic adpsfc): Surface synoptic fixed land reports
BUFR format
Arranged by UTC day and continuously grow throughout the day, if you run decoders many time, the content of the file will grow
No QC (other than rudimentary checks inside decoders)
No duplicate checking

Interested users can use utility debufr to check the content of the bufr file:

> /g5/nwpc_ep8/obsproc/nwprod/util/exec/debufr /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001

the output is in debufr.out.

Surface¶

Data directory structure¶

Data format¶

Data extractor¶

Decoder source code¶

Decode and convert to BUFR format¶

Transfer bufr data to BUFR Tanks¶