This study concerns a Ka-band solid-state transmitter cloud radar, made in China, which can operate in three different work modes, with different pulse widths, and coherent and incoherent integration numbers, to meet ...This study concerns a Ka-band solid-state transmitter cloud radar, made in China, which can operate in three different work modes, with different pulse widths, and coherent and incoherent integration numbers, to meet the requirements for cloud remote sensing over the Tibetan Plateau. Specifically, the design of the three operational modes of the radar（i.e., boundary mode M1, cirrus mode M2, and precipitation mode M3） is introduced. Also, a cloud radar data merging algorithm for the three modes is proposed. Using one month＇s continuous measurements during summertime at Naqu on the Tibetan Plateau,we analyzed the consistency between the cloud radar measurements of the three modes. The number of occurrences of radar detections of hydrometeors and the percentage contributions of the different modes＇ data to the merged data were estimated.The performance of the merging algorithm was evaluated. The results indicated that the minimum detectable reflectivity for each mode was consistent with theoretical results. Merged data provided measurements with a minimum reflectivity of -35 dBZ at the height of 5 km, and obtained information above the height of 0.2 km. Measurements of radial velocity by the three operational modes agreed very well, and systematic errors in measurements of reflectivity were less than 2 dB. However,large discrepancies existed in the measurements of the linear depolarization ratio taken from the different operational modes.The percentage of radar detections of hydrometeors in mid- and high-level clouds increased by 60% through application of pulse compression techniques. In conclusion, the merged data are appropriate for cloud and precipitation studies over the Tibetan Plateau.展开更多
Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in...Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in high scalability mode, but due to the lack of effective design, there are amounts of computing redundancy in the process of data cleaning, which results in lower performance. In this research, we found that some tasks often are carried out multiple times on same input files, or require same operation results in the process of data cleaning. For this problem, we proposed a new optimization technique that is based on task merge. By merging simple or redundancy computations on same input files, the number of the loop computation in MapReduce can be reduced greatly. The experiment shows, by this means, the overall system runtime is significantly reduced, which proves that the process of data cleaning is optimized. In this paper, we optimized several modules of data cleaning such as entity identification, inconsistent data restoration, and missing value filling. Experimental results show that the proposed method in this paper can increase efficiency for grain big data cleaning.展开更多
With the rapid change in the Arctic sea ice,a large number of sea ice observations have been collected in recent years,and it is expected that an even larger number of such observations will emerge in the coming years...With the rapid change in the Arctic sea ice,a large number of sea ice observations have been collected in recent years,and it is expected that an even larger number of such observations will emerge in the coming years.To make the best use of these observations,in this paper we develop a multi-sensor optimal data merging(MODM)method to merge any number of different sea ice observations.Since such merged data are independent on model forecast,they are valid for model initialization and model validation.Based on the maximum likelihood estimation theory,we prove that any model assimilated with the merged data is equivalent to assimilating the original multi-sensor data.This greatly facilitates sea ice data assimilation,particularly for operational forecast with limited computational resources.We apply the MODM method to merge sea ice concentration(SIC)and sea ice thickness(SIT),respectively,in the Arctic.For SIC merging,the Special Sensor Microwave Imager/Sounder(SSMIS)and Advanced Microwave Scanning Radiometer 2(AMSR2)data are merged together with the Norwegian Ice Service ice chart.This substantially reduces the uncertainties at the ice edge and in the coastal areas.For SIT merging,the daily Soil Moisture and Ocean Salinity(SMOS)data is merged with the weekly-mean merged CryoSat-2 and SMOS(CS2SMOS)data.This generates a new daily CS2SMOS SIT data with better spatial coverage for the whole Arctic.展开更多
Wave energy resources are abundant in both offshore and nearshore areas of the China＇s seas. A reliable assessment of the wave energy resources must be performed before they can be exploited. First, for a water depth...Wave energy resources are abundant in both offshore and nearshore areas of the China＇s seas. A reliable assessment of the wave energy resources must be performed before they can be exploited. First, for a water depth in offshore waters of China, a parameterized wave power density model that considers the effects of the water depth is introduced to improve the calculating accuracy of the wave power density. Second, wave heights and wind speeds on the surface of the China＇s seas are retrieved from an AVISO multi-satellite altim-eter data set for the period from 2009 to 2013. Three mean wave period inversion models are developed and used to calculate the wave energy period. Third, a practical application value for developing the wave energy is analyzed based on buoy data. Finally, the wave power density is then calculated using the wave field data. Using the distribution of wave power density, the energy level frequency, the time variability indexes, the to-tal wave energy and the distribution of total wave energy density according to a wave state, the offshore wave energy in the China＇s seas is assessed. The results show that the areas of abundant and stable wave energy are primarily located in the north-central part of the South China Sea, the Luzon Strait, southeast of Taiwan in the China＇s seas; the wave power density values in these areas are approximately 14.0–18.5 kW/m. The wave energy in the China’s seas presents obvious seasonal variations and optimal seasons for a wave energy utilization are in winter and autumn. Except for very coastal waters, in other sea areas in the China＇s seas, the energy is primarily from the wave state with 0.5 m≤Hs≤4 m, 4 s≤Te≤10 s whereHs is a significant wave height andTe is an energy period; within this wave state, the wave energy accounts for 80% above of the total wave energy. This characteristic is advantageous to designing wave energy convertors （WECs）. The practical application value of the wave energy is higher which can be as an effective supplement for an energy con-sumption in some areas. The above results are consistent with the wave model which indicates fully that this new microwave remote sensing method altimeter is effective and feasible for the wave energy assessment.展开更多
Understanding the ocean＇s role in the global carbon cycle and its response to environmental change requires a high spatio-temporal resolution of observation.Merging ocean color data from multiple sources is an effect...Understanding the ocean＇s role in the global carbon cycle and its response to environmental change requires a high spatio-temporal resolution of observation.Merging ocean color data from multiple sources is an effective way to alleviate the limitation of individual ocean color sensors（e.g.,swath width and gaps,cloudy or rainy weather,and sun glint） and to improve the temporal and spatial coverage.Since the missions of Sea-Viewing Wide Field-of-View Sensor（Sea Wi FS） and Medium-spectral Resolution Imaging Spectrometer（MERIS） ended on December 11,2010 and May 9,2012,respectively,the number of available ocean color sensors has declined,reducing the benefits of the merged ocean color data with respect to the spatial and temporal coverage.In present work,Medium Resolution Spectral Imager（MERSI）/FY-3 of China is added in merged processing and a new dataset of global ocean chlorophyll a（Chl a） concentration（2000–2015） is generated from the remote sensing reflectance（Rrs（λ）） observations of MERIS,Moderate-resolution imaging spectra-radiometer（MODIS）-AQUA,Visible infrared Imaging Radiometer（VIIRS） and MERSI.These data resources are first merged into unified remote sensing reflectance data,and then Chl a concentration data are inversed using the combined Chl a algorithm of color index-based algorithm（CIA） and OC3.The merged data products show major improvements in spatial and temporal coverage from the addition of MERSI.The average daily coverage of merged products is approximately 24% of the global ocean and increases by approximately 9% when MERSI data are added in the merging process.Sampling frequency（temporal coverage） is greatly improved by combining MERSI data,with the median sampling frequency increasing from 15.6%（57 d/a） to 29.9%（109 d/a）.The merged Chl a products herein were validated by in situ measurements and comparing them with the merged products using the same approach except for omitting MERSI and Glob Colour and MEa SUREs merged data.Correlation and relative error between the new merged Chl a products and in situ observation are stable relative to the results of the merged products without the addition of MERSI.Time series of the Chl a concentration anomalies are similar to the merged products without adding MERSI and single sensors.The new merged products agree within approximately 10% of the merged Chl a product from Glob Colour and MEa SUREs.展开更多
Using 20 years （1993-2012） of merged data recorded by contemporary multi-altimeter missions, a variety of sea-level variability modes are recovered in the South China Sea employing three- dimensional harmonic extrac...Using 20 years （1993-2012） of merged data recorded by contemporary multi-altimeter missions, a variety of sea-level variability modes are recovered in the South China Sea employing three- dimensional harmonic extraction. In terms of the long-term variation, the South China Sea is estimated to have a rising sea-level linear trend of 5.39 mm/a over these 20 years. Among the modes extracted, the seven most statistically significant periodic or quasi-periodic modes are identified as principal modes. The geographical distributions of the magnitudes and phases of the modes are displayed. In terms of intra- annual and annual regimes, two principal modes with strict semiannual and annual periods are found, with the annual variability having the largest amplitudes among the seven modes. For interannual and decadal regimes, five principal modes at approximately 18, 21, 23, 28, and 112 months are found with the most mode- active region being to the east of Vietnam. For the phase distributions, a series of amphidromes are observed as twins, termed ＂amphidrome twins＂, comprising rotating dipole systems. The stability of periodic modes is investigated employing joint spatiotemporal analysis of latitude/longitude sections. Results show that all periodic modes are robust, revealing the richness and complexity of sea-level modes in the South China Sea.展开更多
This paper presented a rule merging and simplifying method and an improved analysis deviation algorithm. The fuzzy equivalence theory avoids the rigid way (either this or that) of traditional equivalence theory. Durin...This paper presented a rule merging and simplifying method and an improved analysis deviation algorithm. The fuzzy equivalence theory avoids the rigid way (either this or that) of traditional equivalence theory. During a data cleaning process task, some rules exist such as included/being included relations with each other. The equivalence degree of the being-included rule is smaller than that of the including rule, so a rule merging and simplifying method is introduced to reduce the total computing time. And this kind of relation will affect the deviation of fuzzy equivalence degree. An improved analysis deviation algorithm that omits the influence of the included rules' equivalence degree was also presented. Normally the duplicate records are logged in a file, and users have to check and verify them one by one. It's time-cost. The proposed algorithm can save users' labor during duplicate records checking. Finally, an experiment was presented which demonstrates the possibility of the rule.展开更多
Sea surface salinity(SSS)is an essential variable of ocean dynamics and climate research.The Soil Moisture and Ocean Salinity(SMOS),Aquarius,and Soil Moisture Active Passive(SMAP)satellite missions all provide SSS mea...Sea surface salinity(SSS)is an essential variable of ocean dynamics and climate research.The Soil Moisture and Ocean Salinity(SMOS),Aquarius,and Soil Moisture Active Passive(SMAP)satellite missions all provide SSS measurements.The European Space Agency(ESA)Climate Change Initiative Sea Surface Salinity(CCI-SSS)project merged these three satellite SSS data to produce CCI L4SSS products.We validated the accuracy of the four satellite products(CCI,SMOS,Aquarius,and SMAP)using in-situ gridded data and Argo floats in the South China Sea(SCS).Compared with in-situ gridded data,it shows that the CCI achieved the best performance(RMSD:0.365)on monthly time scales.The RMSD of SMOS,Aquarius,and SMAP(SMOS:0.389;Aquarius:0.409;SMAP:0.391)are close,and the SMOS takes a slight advantage in contrast with Aquarius and SMAP.Large discrepancies can be found near the coastline and in the shelf seas.Meanwhile,CCI with lower RMSD(0.295)perform better than single satellite data(SMOS:0.517;SMAP:0.297)on weekly time scales compared with Argo floats.Overall,the merged CCI have the smallest RMSD among the four satellite products in the SCS on both weekly time scales and monthly time scales,which illustrates the improved accuracy of merged CCI compared with the individual satellite data.展开更多
Extracting and mining social networks information from massive Web data is of both theoretical and practical significance. However, one of definite features of this task was a large scale data processing, which remain...Extracting and mining social networks information from massive Web data is of both theoretical and practical significance. However, one of definite features of this task was a large scale data processing, which remained to be a great challenge that would be addressed. MapReduce is a kind of distributed programming model. Just through the implementation of map and reduce those two functions, the distributed tasks can work well. Nevertheless, this model does not directly support heterogeneous datasets processing, while heterogeneous datasets are common in Web. This article proposes a new framework which improves original MapReduce framework into a new one called Map-Reduce-Merge. It adds merge phase that can efficiently solve the problems of heterogeneous data processing. At the same time, some works of optimization and improvement are done based on the features of Web data.展开更多
基金funded by the National Sciences Foundation of China(Grant No.91337103)the China Meteorological Administration Special Public Welfare Research Fund(Grant No.GYHY201406001)
文摘This study concerns a Ka-band solid-state transmitter cloud radar, made in China, which can operate in three different work modes, with different pulse widths, and coherent and incoherent integration numbers, to meet the requirements for cloud remote sensing over the Tibetan Plateau. Specifically, the design of the three operational modes of the radar（i.e., boundary mode M1, cirrus mode M2, and precipitation mode M3） is introduced. Also, a cloud radar data merging algorithm for the three modes is proposed. Using one month＇s continuous measurements during summertime at Naqu on the Tibetan Plateau,we analyzed the consistency between the cloud radar measurements of the three modes. The number of occurrences of radar detections of hydrometeors and the percentage contributions of the different modes＇ data to the merged data were estimated.The performance of the merging algorithm was evaluated. The results indicated that the minimum detectable reflectivity for each mode was consistent with theoretical results. Merged data provided measurements with a minimum reflectivity of -35 dBZ at the height of 5 km, and obtained information above the height of 0.2 km. Measurements of radial velocity by the three operational modes agreed very well, and systematic errors in measurements of reflectivity were less than 2 dB. However,large discrepancies existed in the measurements of the linear depolarization ratio taken from the different operational modes.The percentage of radar detections of hydrometeors in mid- and high-level clouds increased by 60% through application of pulse compression techniques. In conclusion, the merged data are appropriate for cloud and precipitation studies over the Tibetan Plateau.
文摘Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in high scalability mode, but due to the lack of effective design, there are amounts of computing redundancy in the process of data cleaning, which results in lower performance. In this research, we found that some tasks often are carried out multiple times on same input files, or require same operation results in the process of data cleaning. For this problem, we proposed a new optimization technique that is based on task merge. By merging simple or redundancy computations on same input files, the number of the loop computation in MapReduce can be reduced greatly. The experiment shows, by this means, the overall system runtime is significantly reduced, which proves that the process of data cleaning is optimized. In this paper, we optimized several modules of data cleaning such as entity identification, inconsistent data restoration, and missing value filling. Experimental results show that the proposed method in this paper can increase efficiency for grain big data cleaning.
基金EUMETSAT,Norwegian Ice Service,University of Bremen,University of Hamburg,and Alfred Wegener Institute are gratefully acknowledged for providing the dataWe thank two anonymous reviewers for their helpful commentsThis study was supported by the Norwegian Research Council through the SPARSE project(Grant no.254765)and CIRFA project(Grant no.237906).
文摘With the rapid change in the Arctic sea ice,a large number of sea ice observations have been collected in recent years,and it is expected that an even larger number of such observations will emerge in the coming years.To make the best use of these observations,in this paper we develop a multi-sensor optimal data merging(MODM)method to merge any number of different sea ice observations.Since such merged data are independent on model forecast,they are valid for model initialization and model validation.Based on the maximum likelihood estimation theory,we prove that any model assimilated with the merged data is equivalent to assimilating the original multi-sensor data.This greatly facilitates sea ice data assimilation,particularly for operational forecast with limited computational resources.We apply the MODM method to merge sea ice concentration(SIC)and sea ice thickness(SIT),respectively,in the Arctic.For SIC merging,the Special Sensor Microwave Imager/Sounder(SSMIS)and Advanced Microwave Scanning Radiometer 2(AMSR2)data are merged together with the Norwegian Ice Service ice chart.This substantially reduces the uncertainties at the ice edge and in the coastal areas.For SIT merging,the daily Soil Moisture and Ocean Salinity(SMOS)data is merged with the weekly-mean merged CryoSat-2 and SMOS(CS2SMOS)data.This generates a new daily CS2SMOS SIT data with better spatial coverage for the whole Arctic.
基金The Ocean Renewable Energy Special Fund Project of the State Oceanic Administration of China under contract No.GHME2011ZC07the Dragon Ⅲ Project of the European Space Agency and Ministry of Science and Technology of China under contract No.10412
文摘Wave energy resources are abundant in both offshore and nearshore areas of the China＇s seas. A reliable assessment of the wave energy resources must be performed before they can be exploited. First, for a water depth in offshore waters of China, a parameterized wave power density model that considers the effects of the water depth is introduced to improve the calculating accuracy of the wave power density. Second, wave heights and wind speeds on the surface of the China＇s seas are retrieved from an AVISO multi-satellite altim-eter data set for the period from 2009 to 2013. Three mean wave period inversion models are developed and used to calculate the wave energy period. Third, a practical application value for developing the wave energy is analyzed based on buoy data. Finally, the wave power density is then calculated using the wave field data. Using the distribution of wave power density, the energy level frequency, the time variability indexes, the to-tal wave energy and the distribution of total wave energy density according to a wave state, the offshore wave energy in the China＇s seas is assessed. The results show that the areas of abundant and stable wave energy are primarily located in the north-central part of the South China Sea, the Luzon Strait, southeast of Taiwan in the China＇s seas; the wave power density values in these areas are approximately 14.0–18.5 kW/m. The wave energy in the China’s seas presents obvious seasonal variations and optimal seasons for a wave energy utilization are in winter and autumn. Except for very coastal waters, in other sea areas in the China＇s seas, the energy is primarily from the wave state with 0.5 m≤Hs≤4 m, 4 s≤Te≤10 s whereHs is a significant wave height andTe is an energy period; within this wave state, the wave energy accounts for 80% above of the total wave energy. This characteristic is advantageous to designing wave energy convertors （WECs）. The practical application value of the wave energy is higher which can be as an effective supplement for an energy con-sumption in some areas. The above results are consistent with the wave model which indicates fully that this new microwave remote sensing method altimeter is effective and feasible for the wave energy assessment.
基金The National Key R&D Program of China under contract No.2016YFA0600102the National Natural Science Foundation of China under contract Nos 41506203,41476159,41506204,41606197,41471303 and 41706209the Cooperation Project of FIO and KOIST under contract No.PI-2017-03
文摘Understanding the ocean＇s role in the global carbon cycle and its response to environmental change requires a high spatio-temporal resolution of observation.Merging ocean color data from multiple sources is an effective way to alleviate the limitation of individual ocean color sensors（e.g.,swath width and gaps,cloudy or rainy weather,and sun glint） and to improve the temporal and spatial coverage.Since the missions of Sea-Viewing Wide Field-of-View Sensor（Sea Wi FS） and Medium-spectral Resolution Imaging Spectrometer（MERIS） ended on December 11,2010 and May 9,2012,respectively,the number of available ocean color sensors has declined,reducing the benefits of the merged ocean color data with respect to the spatial and temporal coverage.In present work,Medium Resolution Spectral Imager（MERSI）/FY-3 of China is added in merged processing and a new dataset of global ocean chlorophyll a（Chl a） concentration（2000–2015） is generated from the remote sensing reflectance（Rrs（λ）） observations of MERIS,Moderate-resolution imaging spectra-radiometer（MODIS）-AQUA,Visible infrared Imaging Radiometer（VIIRS） and MERSI.These data resources are first merged into unified remote sensing reflectance data,and then Chl a concentration data are inversed using the combined Chl a algorithm of color index-based algorithm（CIA） and OC3.The merged data products show major improvements in spatial and temporal coverage from the addition of MERSI.The average daily coverage of merged products is approximately 24% of the global ocean and increases by approximately 9% when MERSI data are added in the merging process.Sampling frequency（temporal coverage） is greatly improved by combining MERSI data,with the median sampling frequency increasing from 15.6%（57 d/a） to 29.9%（109 d/a）.The merged Chl a products herein were validated by in situ measurements and comparing them with the merged products using the same approach except for omitting MERSI and Glob Colour and MEa SUREs merged data.Correlation and relative error between the new merged Chl a products and in situ observation are stable relative to the results of the merged products without the addition of MERSI.Time series of the Chl a concentration anomalies are similar to the merged products without adding MERSI and single sensors.The new merged products agree within approximately 10% of the merged Chl a product from Glob Colour and MEa SUREs.
基金Supported by the National Natural Science Foundation of China(Nos.41331172,U1406404)the National High Technology Research and Development Program of China(863 Program)(No.2013AA09A505)
文摘Using 20 years （1993-2012） of merged data recorded by contemporary multi-altimeter missions, a variety of sea-level variability modes are recovered in the South China Sea employing three- dimensional harmonic extraction. In terms of the long-term variation, the South China Sea is estimated to have a rising sea-level linear trend of 5.39 mm/a over these 20 years. Among the modes extracted, the seven most statistically significant periodic or quasi-periodic modes are identified as principal modes. The geographical distributions of the magnitudes and phases of the modes are displayed. In terms of intra- annual and annual regimes, two principal modes with strict semiannual and annual periods are found, with the annual variability having the largest amplitudes among the seven modes. For interannual and decadal regimes, five principal modes at approximately 18, 21, 23, 28, and 112 months are found with the most mode- active region being to the east of Vietnam. For the phase distributions, a series of amphidromes are observed as twins, termed ＂amphidrome twins＂, comprising rotating dipole systems. The stability of periodic modes is investigated employing joint spatiotemporal analysis of latitude/longitude sections. Results show that all periodic modes are robust, revealing the richness and complexity of sea-level modes in the South China Sea.
文摘This paper presented a rule merging and simplifying method and an improved analysis deviation algorithm. The fuzzy equivalence theory avoids the rigid way (either this or that) of traditional equivalence theory. During a data cleaning process task, some rules exist such as included/being included relations with each other. The equivalence degree of the being-included rule is smaller than that of the including rule, so a rule merging and simplifying method is introduced to reduce the total computing time. And this kind of relation will affect the deviation of fuzzy equivalence degree. An improved analysis deviation algorithm that omits the influence of the included rules' equivalence degree was also presented. Normally the duplicate records are logged in a file, and users have to check and verify them one by one. It's time-cost. The proposed algorithm can save users' labor during duplicate records checking. Finally, an experiment was presented which demonstrates the possibility of the rule.
基金Supported by the National Natural Science Foundation of China(No.42075149)。
文摘Sea surface salinity(SSS)is an essential variable of ocean dynamics and climate research.The Soil Moisture and Ocean Salinity(SMOS),Aquarius,and Soil Moisture Active Passive(SMAP)satellite missions all provide SSS measurements.The European Space Agency(ESA)Climate Change Initiative Sea Surface Salinity(CCI-SSS)project merged these three satellite SSS data to produce CCI L4SSS products.We validated the accuracy of the four satellite products(CCI,SMOS,Aquarius,and SMAP)using in-situ gridded data and Argo floats in the South China Sea(SCS).Compared with in-situ gridded data,it shows that the CCI achieved the best performance(RMSD:0.365)on monthly time scales.The RMSD of SMOS,Aquarius,and SMAP(SMOS:0.389;Aquarius:0.409;SMAP:0.391)are close,and the SMOS takes a slight advantage in contrast with Aquarius and SMAP.Large discrepancies can be found near the coastline and in the shelf seas.Meanwhile,CCI with lower RMSD(0.295)perform better than single satellite data(SMOS:0.517;SMAP:0.297)on weekly time scales compared with Argo floats.Overall,the merged CCI have the smallest RMSD among the four satellite products in the SCS on both weekly time scales and monthly time scales,which illustrates the improved accuracy of merged CCI compared with the individual satellite data.
文摘Extracting and mining social networks information from massive Web data is of both theoretical and practical significance. However, one of definite features of this task was a large scale data processing, which remained to be a great challenge that would be addressed. MapReduce is a kind of distributed programming model. Just through the implementation of map and reduce those two functions, the distributed tasks can work well. Nevertheless, this model does not directly support heterogeneous datasets processing, while heterogeneous datasets are common in Web. This article proposes a new framework which improves original MapReduce framework into a new one called Map-Reduce-Merge. It adds merge phase that can efficiently solve the problems of heterogeneous data processing. At the same time, some works of optimization and improvement are done based on the features of Web data.
