Aircraft¶
Data directory structure¶
The Aircraft data is organized as CIMISS files every 6 hours :
> ls -la /g3/tianwh/share/conobs/rec_RSING_*:
-rw-r--r-- 1 tianwh obs 34351730 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060100_g_06.dat
-rw-r--r-- 1 tianwh obs 16639350 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060106_g_06.dat
-rw-r--r-- 1 tianwh obs 25511010 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060112_g_06.dat
-rw-r--r-- 1 tianwh obs 31514910 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060118_g_06.dat
-rw-r--r-- 1 tianwh obs 29063650 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060200_g_06.dat
-rw-r--r-- 1 tianwh obs 14376670 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060206_g_06.dat
-rw-r--r-- 1 tianwh obs 24181990 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060212_g_06.dat
-rw-r--r-- 1 tianwh obs 32740190 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060218_g_06.dat
-rw-r--r-- 1 tianwh obs 32471950 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060300_g_06.dat
-rw-r--r-- 1 tianwh obs 20091890 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060306_g_06.dat
-rw-r--r-- 1 tianwh obs 25002950 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060312_g_06.dat
-rw-r--r-- 1 tianwh obs 32896430 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060318_g_06.dat
-rw-r--r-- 1 tianwh obs 34783490 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060400_g_06.dat
-rw-r--r-- 1 tianwh obs 21610470 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060406_g_06.dat
-rw-r--r-- 1 tianwh obs 29883770 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060412_g_06.dat
-rw-r--r-- 1 tianwh obs 34280610 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060418_g_06.dat
-rw-r--r-- 1 tianwh obs 35147210 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060500_g_06.dat
Data format¶
CIMISS format data is looks like:
RSING 2019 6 1 0 245369 5 4 4
00HQVXZA 44.49 150.37 1.00 241.00 9.00
11247.00 -43.95 222.00 38.60 9.00 9.00 9.00 9.00
00HQVXZA 46.01 152.68 1.00 221.00 9.00
11217.00 -47.95 216.00 38.60 9.00 9.00 9.00 9.00
00HQVXZA 47.45 155.04 1.00 201.00 9.00
10638.00 -48.95 229.00 45.30 9.00 9.00 9.00 9.00
00HQVXZA 53.18 167.60 1.00 37.00 3.00
10368.00 -48.45 267.00 13.40 9.00 9.00 9.00 9.00
00HQVXZA 56.68 178.56 31.00 2340.00 3.00
10368.00 -50.45 345.00 44.20 9.00 9.00 9.00 9.00
00HQVXZA 49.34 158.56 1.00 135.00 3.00
10368.00 -47.45 232.00 25.20 9.00 9.00 9.00 9.00
00HQVXZA 58.23 -172.53 31.00 2301.00 3.00
10367.00 -49.95 7.00 11.80 9.00 9.00 9.00 9.00
00HQVXZA 51.98 164.39 1.00 56.00 3.00
10366.00 -50.15 207.00 23.10 9.00 9.00 9.00 9.00
00HQVXZA 52.59 165.94 1.00 47.00 3.00
10366.00 -49.45 227.00 20.60 9.00 9.00 9.00 9.00
00HQVXZA 50.69 161.37 1.00 116.00 3.00
10366.00 -47.95 274.00 25.70 9.00 9.00 9.00 9.00
00HQVXZA 50.02 159.94 1.00 125.00 3.00
10364.00 -47.65 249.00 24.20 9.00 9.00 9.00 9.00
00HQVXZA 57.15 -179.26 31.00 2330.00 3.00
10364.00 -46.95 358.00 29.80 9.00 9.00 9.00 9.00
00HQVXZA 54.93 171.98 1.00 12.00 9.00
10363.00 -48.95 320.00 32.90 9.00 9.00 9.00 9.00
00HQVXZA 59.00 -160.01 31.00 2212.00 9.00
10363.00 -44.95 83.00 6.20 9.00 9.00 9.00 9.00
00HQVXZA 53.75 169.23 1.00 28.00 9.00
10363.00 -48.95 287.00 18.50 9.00 9.00 9.00 9.00
00HQVXZA 57.55 -177.04 31.00 2320.00 3.00
10362.00 -47.95 1.00 20.60 9.00 9.00 9.00 9.00
00HQVXZA 51.34 162.86 1.00 106.00 3.00
10362.00 -47.65 259.00 14.40 9.00 9.00 9.00 9.00
00HQVXZA 54.28 170.43 1.00 21.00 9.00
10333.00 -47.95 314.00 23.10 9.00 9.00 9.00 9.00
00HQVXZA 55.56 174.12 1.00 1.00 9.00
10333.00 -50.95 328.00 34.00 9.00 9.00 9.00 9.00
00HQVXZA 58.77 -167.53 31.00 2241.00 9.00
10333.00 -50.95 39.00 12.90 9.00 9.00 9.00 9.00
00HQVXZA 57.53 -177.16 31.00 2321.00 9.00
10333.00 -46.95 5.00 21.10 9.00 9.00 9.00 9.00
00HQVXZA 50.32 160.57 1.00 121.00 9.00
10333.00 -46.95 261.00 21.60 9.00 9.00 9.00 9.00
Data extractor¶
A python code is used to extract the desired information from this CIMISS file:
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders
> rm cimiss_amdar_data
> ./read_nmc_cimiss_rsing.py -f /g3/tianwh/share/conobs/rec_RSING_2019060100_g_06.dat
If you want to batch process number of CIMISS files, you can use following command:
> # This command will find all CIMISS files and prcessing the file one by one
> decod_nmc_cimiss_rsing.py -d /g3/tianwh/share/conobs
The information we want to extract from CIMISS is:
import fortranformat as ff
output_line = ff.FortranRecordWriter('(A8, 5A8, 4f10.2, I8, f10.2)')
output.write(output_line.write([call_name, obstime.strftime("%Y"), obstime.strftime("%m"), \
obstime.strftime("%d"), obstime.strftime("%H"), obstime.strftime("%M"), lat, lon, phi, \
T, dd, ff]) + "\n")
the content of micaps_amdar_data is:
> less micaps_amdar_data
01ZXL5JA 2019 06 30 14 38 34.61 -109.08 11884.00 -56.45 277 7.20
01ZXL5JA 2019 06 30 14 32 33.93 -109.51 11883.00 -55.25 277 7.70
01ZXL5JA 2019 06 30 14 50 35.94 -108.19 11882.00 -56.95 295 9.30
01ZXL5JA 2019 06 30 14 59 37.01 -107.83 11882.00 -57.15 282 11.80
01ZXL5JA 2019 06 30 14 56 36.64 -107.90 11882.00 -56.95 280 10.30
01ZXL5JA 2019 06 30 14 41 34.94 -108.86 11881.00 -56.25 273 9.80
01ZXL5JA 2019 06 30 14 47 35.61 -108.41 11881.00 -56.45 282 9.30
01ZXL5JA 2019 06 30 14 44 35.28 -108.64 11881.00 -56.45 278 9.30
01ZXL5JA 2019 06 30 14 53 36.28 -107.99 11881.00 -56.65 286 9.80
01ZXL5JA 2019 06 30 14 35 34.27 -109.29 11880.00 -56.45 273 6.70
01ZXL5JA 2019 06 30 14 29 33.59 -109.72 11468.00 -52.75 287 6.70
01ZXL5JA 2019 06 30 14 26 33.26 -109.94 9933.00 -39.75 158 1.00
01ZXL5JA 2019 06 30 14 23 32.90 -110.16 8393.00 -27.25 232 3.10
01ZXL5JA 2019 06 30 14 19 32.55 -110.38 6529.00 -12.65 265 0.50
01ZXL5JA 2019 06 30 14 19 32.51 -110.40 6358.00 -11.75 305 1.50
01ZXL5JA 2019 06 30 14 19 32.48 -110.42 6135.00 -10.45 310 1.50
01ZXL5JA 2019 06 30 14 18 32.45 -110.44 5902.00 -8.75 334 1.50
01ZXL5JA 2019 06 30 14 18 32.42 -110.46 5637.00 -7.45 336 3.10
01ZXL5JA 2019 06 30 14 18 32.39 -110.47 5379.00 -5.75 340 2.10
01ZXL5JA 2019 06 30 14 17 32.35 -110.49 5122.00 -3.75 15 2.10
01ZXL5JA 2019 06 30 14 17 32.32 -110.51 4757.00 -0.75 47 3.10
01ZXL5JA 2019 06 30 14 17 32.29 -110.53 4485.00 1.55 34 1.50
01ZXL5JA 2019 06 30 14 16 32.26 -110.55 4214.00 3.55 24 3.10
01ZXL5JA 2019 06 30 14 16 32.24 -110.58 3946.00 5.85 31 3.10
01ZXL5JA 2019 06 30 14 16 32.22 -110.62 3665.00 8.85 34 3.60
01ZXL5JA 2019 06 30 14 15 32.21 -110.65 3390.00 11.05 17 3.60
01ZXL5JA 2019 06 30 14 15 32.19 -110.68 3270.00 11.55 28 1.50
01ZXL5JA 2019 06 30 14 15 32.17 -110.71 3111.00 13.85 65 2.10
01ZXL5JA 2019 06 30 14 14 32.15 -110.73 2809.00 16.35 157 1.50
01ZXL5JA 2019 06 30 14 14 32.14 -110.76 2511.00 18.85 126 4.60
01ZXL5JA 2019 06 30 14 14 32.12 -110.79 2160.00 22.05 131 4.10
01ZXL5JA 2019 06 30 14 13 32.11 -110.81 1942.00 23.55 131 3.10
01ZXL5JA 2019 06 30 14 13 32.10 -110.83 1797.00 24.25 103 2.10
01ZXL5JA 2019 06 30 14 13 32.08 -110.85 1606.00 25.85 108 2.60
01ZXL5JA 2019 06 30 14 13 32.08 -110.86 1576.00 26.05 110 3.10
01ZXL5JA 2019 06 30 14 13 32.08 -110.86 1527.00 26.35 111 3.60
01ZXL5JA 2019 06 30 14 12 32.08 -110.87 1487.00 27.05 117 3.60
01ZXL5JA 2019 06 30 14 12 32.08 -110.88 1445.00 27.35 127 4.10
01ZXL5JA 2019 06 30 14 12 32.08 -110.88 1397.00 27.55 133 4.60
01ZXL5JA 2019 06 30 14 12 32.08 -110.89 1351.00 28.05 132 4.60
01ZXL5JA 2019 06 30 14 12 32.08 -110.89 1305.00 28.35 124 4.10
01ZXL5JA 2019 06 30 14 12 32.08 -110.90 1260.00 28.35 109 4.10
01ZXL5JA 2019 06 30 14 12 32.09 -110.90 1220.00 28.85 108 4.10
01ZXL5JA 2019 06 30 14 12 32.09 -110.91 1171.00 29.25 103 3.60
01ZXL5JA 2019 06 30 14 12 32.09 -110.91 1104.00 29.55 103 3.10
01ZXL5JA 2019 06 30 14 12 32.10 -110.92 1023.00 28.85 119 2.10
01ZXL5JA 2019 06 30 14 11 32.10 -110.92 944.00 28.85 153 1.00
01ZXL5JA 2019 06 30 14 11 32.10 -110.92 866.00 29.25 104 0.00
01ZXL5JA 2019 06 30 14 11 32.11 -110.93 802.00 31.55 0 0.00
020UFPRA 2019 06 30 14 53 35.03 -96.72 11582.00 -52.15 24 6.70
020UFPRA 2019 06 30 14 38 35.03 -94.32 11552.00 -52.55 30 14.40
020UFPRA 2019 06 30 14 23 35.01 -91.94 11552.00 -53.05 66 2.10
020UFPRA 2019 06 30 10 45 30.08 -83.79 11552.00 -53.75 233 12.90
020UFPRA 2019 06 30 14 08 34.09 -89.61 10973.00 -50.05 107 2.60
020UFPRA 2019 06 30 13 53 34.04 -87.33 10942.00 -49.55 161 6.70
020UFPRA 2019 06 30 10 30 29.02 -82.58 10668.00 -46.25 249 4.60
Decoder source code¶
- Source code directory:
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dcmicapsamdar/sorc
Subroutines to decode Aircraft data
afdcod.f
注解
- The
pirep.tbl,airep.tblare not used, although they are required as arguments and read in. - The path and file name of
cmicaps_amdar_datafile are hard coded in the subroutines.
- Compile the code
> make
Decode and convert to BUFR format¶
- enter into the exec directory
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dcmicapsamdar/exec
> ls -la
total 1944
drwxr-xr-x 9 xinzhang staff 288 Sep 25 21:24 .
drwxr-xr-x 5 xinzhang staff 160 Sep 21 18:45 ..
lrwxr-xr-x 1 xinzhang staff 25 Sep 21 18:45 airep.tbl -> ../dictionaries/airep.tbl
lrwxr-xr-x 1 xinzhang staff 34 Sep 21 18:45 bufrtab.004 -> ../../decod_shared/fix/bufrtab.004
-rwxr-xr-x 1 xinzhang staff 985984 Sep 21 18:45 decod_dcmicapsamdar
lrwxr-xr-x 1 xinzhang staff 25 Sep 21 18:45 pirep.tbl -> ../dictionaries/pirep.tbl
-rwxr-xr-x 1 xinzhang staff 410 Sep 21 18:45 run.ksh
drwxr-xr-x 3 xinzhang staff 96 Sep 21 18:45 tmp
- we provide a script to run the decoder in batch mode:
> ./run_dcmicapsamdar.py -s 2018121612 -e 2018121700 -i 1
注解
- given the starting datetime and ending datetime, it iterates all cycles (every 1 hours)
- the units of interval is hour (-i)
- this script call run.ksh
- run the decoder script
> run.ksh
> cat run.ksh
#!/bin/bash
export DBNBUFRT=120
export TRANJB=/g5/nwpc_ep8/obsproc/nwprod/ush/tranjb
export tank_dir=/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003
export SCREEN="OFF"
export DBNROOT=`pwd`
rm tmp/*
rm decod_dcmicapsadmar.log
./decod_dcmicapsadmar -v 4 -d decod_dcmicapsadmar.log -b 240 -c $1 pirep.tbl airep.tbl bufrtab.004
ls -la tmp/*
BUFR_FILES=$(echo tmp/BUFR*)
echo ${BUFR_FILES}
for file in ${BUFR_FILES}
do
${TRANJB} ${tank_dir} ${file}
done
注解
- -c $1 : Set the current time (201809120200) used to calculate the time departures of the obs. data.
- -b 240 : Number of hours to decode prior to “current” time (default)
- The observations with date/time between current time - 240 hours and current time + 3 are kept.
- The generated BUFR format file will be saved at
> ls -la tmp/BUFR.0.aircraft.1.1933.1537419287.73
-rw-r--r-- 1 xinzhang staff 1806552 Sep 21 18:45 tmp/BUFR.0.aircraft.1.1933.1537419287.73
Transfer bufr data to BUFR Tanks¶
- put data in BUFR tanks:
> /g5/nwpc_ep8/obsproc/nwprod/ush/tranjb /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 tmp/BUFR.0.aircraft.1.1933.1537419287.73
> ls -al /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003
-rw-r--r-- 1 vagrant vagrant 1828720 Sep 19 22:54 /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003
注解
- if environmental variable SCREEN=ON :
- Define Run Time is the system time when the tranjb is running.
- Only observations with date/time between Run Time - 10 days and Run Time + 12 hours are kept.
for retrospective run, set SCREEN=OFF
/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/yyyymmdd/bmmm/xxsss (where mmm is WMO BUFR message type and xxx is local BUFR message subtype)
004.003 (in dump group mnemonic aircft): AMDAR format aircraft data from ASDAR/ACARS reporting systems
BUFR format
Arranged by UTC day and continuously grow throughout the day, if you run decoders many time, the content of the file will grow
No QC (other than rudimentary checks inside decoders)
No duplicate checking
Interested users can use utility debufr to check the content of the bufr file:
> /g5/nwpc_ep8/obsproc/nwprod/util/exec/debufr /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003the output is in
debufr.out.