A number of high-quality climate datasets have been developed for monitoring long-term trends and variability in the Australian climate. In particular, homogenised datasets of annual temperature (Torok and Nicholls 1996), daily temperature (Trewin 1999) and rainfall (Lavery et al. 1992, 1997) have been produced using a variety of quality control and data correction techniques. Where possible each station record in these datasets has been corrected for discontinuities caused by changes such as site location, exposure, instrumentation or observation practice. Potential discontinuities have been identified using graphical examinations and a number of different statistical tests. Intensive studies of station history information have also been undertaken to find supporting evidence for potential discontinuities.

Generally these high-quality datasets have been developed as a research project within the Bureau of Meteorology Research Centre and then transferred to the National Climate Centre (NCC) to maintain and use operationally. The datasets are used routinely to calculate Australian mean climate variables published in climate monitoring summaries and media statements. They are currently providing valuable input to Australia's State of the Environment report (Nicholls et al. 2000) and have also been used recently for research studies intended to provide input for the IPCC Third Assessment Report (Collins and Della-Marta 1999; Collins et al. 2000; Manton et al. 2000).

Torok and Nicholls (1996) developed a set of Australian high-quality historical maximum and minimum temperature records corrected for discontinuities at the annual timescale. For each candidate record a reference series was developed using the median of interannual temperature difference series from highly-correlated neighbouring stations. The difference between the candidate and reference series was tested for change points using the technique of Easterling and Peterson (1995). Visual examination of the annual mean diurnal temperature range was also used to detect discontinuities. An extensive search of available station documentation was undertaken to identify supporting evidence for potential discontinuities in each candidate temperature series. A subjective decision was made on which discontinuities should be corrected, with an average of six to seven corrections per record.

A common feature of many long Australian temperature records is a discontinuity associated with a change in instrument shelter to the current standard Stevenson screen. This occurred throughout Australia in the early 1900s and unfortunately change dates are not well documented at many sites (Nicholls et al. 1996). Consequently the high-quality annual temperature records of Torok and Nicholls (1996) are generally confined to the post-1910 period. 224 temperature records were reconstructed to an acceptable standard, 181 of which were identified as being non-urban.

Each year the annual mean temperature is calculated for all non-urban high-quality temperature records. These are then spatially averaged using a Thiessen polygon technique (Lavery et al. 1992) to compute Australia's annual mean temperature. Urban stations are not included to avoid the artificial warming associated with increasing population. The Australian annual mean temperature is routinely provided to the public via annual climate summaries, a media statement and the internet.

The annual mean temperature series for Australia shows a general rise throughout the 20th century, with most warming occurring since 1950 (Fig. 1). Since the development of the dataset some of the observation stations have closed. Also, data losses result in some of the remaining stations having insufficient data to be included in the analysis for some years. In recent years about 130 non-urban stations from the original 181 have had enough data to be included in the annual analysis.

Applying an adjustment used to create a homogeneous annual mean temperature series to all daily data during a year is not an appropriate method of producing a homogeneous daily data series. The magnitude of the discontinuity will vary at different times of the year and it is not known how an adjustment for the mean should be translated into an adjustment for the more extreme values in the distribution. Consequently, an Australian high-quality daily temperature dataset has been developed by Trewin (1999).

The first step in the development of the dataset was the detection and removal of gross single-day errors. Methods used include checks for internal consistency, such as ensuring that daily maximum temperatures were greater than the minimum temperature on the same day and that the maximum was greater than, and the minimum less than, any available fixed-hour observations on the same day. Each station's daily maximum and minimum temperatures were also checked against a feasible range for the location as well as against a number of neighbouring stations with similar climates. The validity of some suspect data was even checked against the synoptic situation at the time of observation. Any data deemed to be erroneous was removed from the dataset. Any data accumulated over several days were also removed at this stage.

After the gross errors had been removed inhomogeneities in the daily temperature series were identified and adjusted for. This was done by comparing each candidate series with a reference series calculated from a weighted mean of highly-correlated neighbouring records. A two-phased regression model based on that used by Solow (1987) and Easterling and Peterson (1995) was used to detect discontinuities in the differences between the candidate and reference series. Potential discontinuities were then checked visually. Available metadata was also examined for supporting evidence of an inhomogeneity. Any discontinuity determined to be artificial was corrected using the method of Trewin and Trevitt (1996). This involved matching the frequency distribution of daily maximum and minimum temperatures on either side of the inhomogeneity, allowing for different magnitudes of discontinuity across the distribution.

Rather than making adjustments in mean temperatures, the daily temperature records were adjusted for discontinuities at the 5, 10, ..., 90, 95 percentile levels. Consequently temperatures at the higher end of a station record's distribution could have been adjusted for different magnitudes of discontinuity than those at the middle or lower end of the distribution. This makes the dataset particularly useful for examining changes in the occurrence of extreme temperature events, such as numbers of hot and cold days per year (Fig. 2).

The final step in the production of the dataset involved the spatial analysis of the adjusted temperature data to identify stations whose data differed substantially from those of its neighbours. Daily temperature anomalies at each station were interpolated onto a 1 x 1 degree grid using the Barnes successive correction technique (Koch et al. 1983) and the subsequent plots examined for "bullseyes".

There are 99 non-urban stations in the Australian high-quality daily temperature dataset, with a further four located at major cities. Most of these stations have been chosen from the Bureau of Meteorology's Reference Climate Station (RCS) network, a set of stations chosen for the length, continuity and quality of their climate records and their likelihood of remaining open in the foreseeable future. Consequently the high-quality temperature stations have a relatively high chance of long-term survival.

Limited Australian daily data has been digitised prior to 1957. Consequently, most corrected daily temperature records are only available from this time. However, a major project within the Bureau of Meteorology aims to digitise hourly and daily data prior to 1957 at 50 key locations throughout the country. This extra data should enable some of the records in the dataset to be extended backwards in time.

As well as being useful for studies of temperature extremes (Collins et al. 2000), the high-quality temperature dataset has been used for a number of other applications. It has been used as the base dataset for an operational seasonal temperature prediction scheme (Jones 1998) and is currently being used for routine monitoring of Australian and regional mean temperatures at the monthly and seasonal timescale.

Lavery et al. (1992) used available station history information and a variety of graphical and statistical tests to select high-quality rainfall stations for monitoring long-term changes in Australian rainfall. Rainfall records were not adjusted for discontinuities but were rejected if suspected of having an inhomogeneity or data quality problem.

The reliability of the observers at candidate stations was investigated by determining the number of missed observations and by examining frequency histograms of daily rainfall amounts for evidence of rounding errors or non-reporting of small rainfall amounts. The cube root of precipitation amounts is often distributed normally so plots of the cube root of rainfall amounts against the cumulative percentage frequency were also used to look for similar evidence. Any record found to have problems with observer error was rejected.

The exposure and location of rainfall sites was also examined with available station history information. Stations with any changes that might have had an impact on the record were rejected. Most stations in Australia have used standard raingauges since they opened and few problems associated with changes in instrumentation were encountered.

Rainfall records that passed these quality checks were then tested for homogeneity using a method based on that of Craddock (1981). This involved calculating a cumulative deviation statistic for each annual rainfall series to identify years with excessive drift from the expected climate. Metadata was then examined to determine whether the change was likely to be artificial or not.

Out of an initial list of around 2100 candidate stations only 191 rainfall records passed all quality and homogeneity tests to be included in the high-quality rainfall dataset of Lavery et al. (1992). However, this network did not provide sufficient coverage of the country to allow the calculation of reliable spatial averages. Consequently, additional stations were introduced to the network by including shorter duration records and composite records of two or more neighbouring stations (Lavery et al. 1997).

Stations records were only composited if they had an overlap of at least 20 years and a high correlation (r > 0.7) between annual totals. In compositing records a conversion factor was used to match the earlier series to the later series. This factor was calculated from the average difference between annual totals for all years during the overlap period.

These additions resulted in a total of 379 records in the high-quality rainfall dataset. Station closures have resulted in a gradual decline in available stations with only about 310 high-quality rainfall stations remaining open. This network is used to calculate Australia's annual mean rainfall total (Fig. 3) and has been used in research studies examining trends in intense rainfall events (Hennessy et al. 1999).

Obviously the production of high-quality research datasets would be a lot easier if all climate data were recorded accurately in the first place. There are currently several projects underway within the Bureau of Meteorology to improve data quality. Most Australian climate data is now collected electronically, mainly from Automatic Weather Systems (AWSs) and Electronic Field Books (EFBs). Options are currently being considered to minimise data losses from AWSs, such as reconfiguring their communication protocols and more efficient methods of recovering data from the on-site AWS memory. The next generation of AWS software will also include a data filter to reduce the number "data spikes" currently being reported.

Many observation sites are now using EFBs to submit observations, removing the requirement for a hardcopy fieldbook. This has created new demands on traditional quality control procedures as often no hardcopy record exists when suspect data are checked. EFBs incorporate some internal quality control by automatically checking whether input data are within feasible bounds. However, further software modifications are required to remove unforeseen problems, often associated with user error.

The needs of data homogeneity are recognised by the Bureau of Meteorology's requirement for at least two years of overlap observations between old and new sites at important climate stations. This requirement cannot always be met for reasons beyond the Bureau's control, but where possible parallel observation programs are often continued for longer than the minimum two years. The Bureau's Physics Laboratory also undertakes instrument comparisons to test for potential impacts on the climate record before any new equipment is approved for use in the observation network. These procedures require strong links between the National Climate Centre and the Observation Branch which are maintained through a number of committee groups.

It is much easier to recover missed observations or check suspect data close to the observation time, rather than months or even decades later. It is also more appropriate for people with local knowledge about the observation site to make any subjective decision about the validity of an observation. Consequently the Bureau of Meteorology is currently trying to restructure data quality control procedures to be as close as possible to observation time and location.

The National Climate Centre and the Bureau's Observation section are developing a Quality Monitoring System (QMS) which will provide a range of improved statistical and graphical tools to help detect and correct problem climate data, such as RainQC (Fig. 4). The QMS will also provide performance information on observations networks and systems to help improve their management and ultimately the data provided by them. However, not only are improved data monitoring and correction tools necessary but a change in the culture of the organisation is also required, so that even operational weather forecasters see themselves as important in the quality control process.

The production of high-quality datasets is dependent on reliable station history information or "metadata". In recent years the Bureau of Meteorology\'s Observation section has developed SitesDb - a relational database to store metadata concerning site location, instrumentation and observation program. Much of this data is entered by the Bureau's inspectors and Regional Observation Managers in near-real time and consequently is very reliable information. Prior to this, such information has been recorded in paper history files. Eventually the information contained in these historical files will be included in SitesDb allowing researchers greater access to metadata.
The National Climate Centre is also scanning the historical station files to allow access to digital images of site diagrams and photographs. However, more effort is required to compile other important metadata such as historical changes in observation practices.

The development of high-quality datasets has been crucial to the investigation of long-term climate trends in Australia. These datasets have been developed using techniques appropriate to the observation network distribution. The numbers of high-quality observation stations has declined over recent years. However, improved data quality, greater access to metadata and periods of overlap observations should allow these high-quality datasets to be supplemented with additional homogeneous climate series.

Collins, D.A. and Della-Marta, P.M. 1999. Annual climate summary 1998: Australia's warmest year on record. Aust. Met. Mag., 48, 273-283.
Collins, D.A, Della-Marta, P.M., Plummer, N. and Trewin, B.C. 2000. Trends in annual frequencies of extreme temperature events in Australia. (in print) Aust. Met. Mag.
Craddock, J.M. 1981. Monitoring a changing climatic series. J. Climatol., 1, 333-343.
Easterling, D.R. and Peterson, T.C. 1995. A new method for detecting undocumented discontinuities in climatological time series. Int. J. Climatol., 15, 369-377.
Hennessy, K.J., Suppiah, R. and Page, C.M. 1999. Australian rainfall changes, 1910-1995. Aust. Met. Mag., 48, 1-13.
Jones, D.A. 1998. The prediction of Australian land surface temperatures using near global sea surface temperature patterns. BMRC Res. Rep. No.70, Bur. Met., Australia.
Koch, S.E., DesJardins, M. and Kocin, P.J. 1983. An interactive Barnes objective map analysis scheme for use with satellite and conventional data. J. Clim. and Appl. Met., 22, 1487-1503.
Lavery, B., Kariko, A. and Nicholls, N. 1992. A historical rainfall data set for Australia. Aust. Met. Mag., 40, 33-39.
Lavery, B., Joung, G. and Nicholls, N. 1997. An extended high-quality historical rainfall dataset for Australia. Aust. Met. Mag., 46, 27-38.
Manton, M.J., Della-Marta, P.M., Haylock, M.R., Hennessy, K.J., Nicholls, N., Chambers, L.E., Collins, D.A., Daw, G., Finet, A., Gunawan, D., Inape, K., Isobe, H., Kestin, T.S., Lefale, P., Leyu, C.H., Lwin, T., Maitrepierre, L., Ouprasitwong, N., Page, C.M., Pahalad, J., Plummer, N., Salinger, M.J., Suppiah, R., Tran, V.L., Trewin, B., Tibig, I. and Yee, D. 2000. Trends in extreme daily rainfall and temperature in southeast Asia and the South Pacific: 1961-1998. (in print) Int. J. Climatol.
Nicholls, N., Tapp, R., Burrows, K. and Richards, D. 1996. Historical thermometer exposures in Australia. Int. J. Climatol., 16, 705-710.
Nicholls, N., Trewin, B. and Haylock, M. 2000. Climate Extremes Indicators for State of the Environment Monitoring. State of the Environment Technical Paper Series (The Atmosphere), Department of the Environment and Heritage, Canberra, Australia, 20pp.
Solow, A. 1987. Testing for climatic change: an application of the two-phase regression model. J. Climate Appl. Meteorol., 26, 1401-1405.
Torok, S.J. and Nicholls, N. 1996. A historical annual temperature dataset for Australia. Aust. Met. Mag., 45, 251-260.
Trewin, B.C. and Trevitt, A.C.F. 1996. The development of composite temperature records. Int. J. Climatol., 16, 1227-1242.
Trewin, B.C. 1999. The development of a high-quality daily temperature data set for Australia, and implications for the observed frequency of extreme temperatures. Proc. Sixth Nat. AMOS Conf., Canberra, A.C.T., Aust., 87p.