Container-based virtualization is increasingly popular in cloud computing due to its efficiency and flexibility.Isolation is a fundamental property of containers and weak isolation could cause significant performance ...Container-based virtualization is increasingly popular in cloud computing due to its efficiency and flexibility.Isolation is a fundamental property of containers and weak isolation could cause significant performance degradation and security vulnerability.However,existing works have almost not discussed the isolation problems of system log which is critical for monitoring and maintenance of containerized applications.In this paper,we present a detailed isolation analysis of system log in current container environment.First,we find several system log isolation problems which can cause significant impacts on system usability,security,and efficiency.For example,system log accidentally exposes information of host and co-resident containers to one container,causing information leakage.Second,we reveal that the root cause of these isolation problems is that containers share the global log configuration,the same log storage,and the global log view.To address these problems,we design and implement a system named private logs(POGs).POGs provides each container with its own log configuration and stores logs individually for each container,avoiding log configuration and storage sharing,respectively.In addition,POGs enables private log view to help distinguish which container the logs belong to.The experimental results show that POGs can effectively enhance system log isolation for containers with negligible performance overhead.展开更多
As computer data grows exponentially,detecting anomalies within system logs has become increasingly important.Current research on log anomaly detection largely depends on log templates derived from log parsing.Word em...As computer data grows exponentially,detecting anomalies within system logs has become increasingly important.Current research on log anomaly detection largely depends on log templates derived from log parsing.Word embedding is utilized to extract information from these templates.However,this method neglects a portion of the content within the logs and confronts the challenge of data imbalance among various log template types after parsing.Currently,specialized research on data imbalance across log template categories remains scarce.A dual-attention-based log anomaly detection model(LogDA),which leveraged data imbalance,was proposed to address these issues in the work.The LogDA model initially utilized a pre-trained model to extract semantic embedding from log templates.Besides,the similarity between embedding was calculated to discern the relationships among the various templates.Then,a Transformer model with a dual-attention mechanism was constructed to capture positional information and global dependencies.Compared to multiple baseline experiments across three public datasets,the proposed approach could improve precision,recall,and F1 scores.展开更多
One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable...One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.展开更多
Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to bui...Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.展开更多
This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test eq...This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test equipment, we have used principal and subordinate structures, and interconnected the host computer and the front-end machine via Ethernet. The front-end machine is based on the ARM7 （advanced reduced instruction set computing （RISC） machine） technique, the processor of which runs an embedded operating system, namely, uClinux OS. We have analyzed the system telecommunication, human-machine interface circuit, transmitter mandrel interface circuit, receiver mandrel interface circuit, and board-level test interface circuit. The software used in the system consists of the embedded front-computer software and the host application software. We have explained in detail the flow chart of the boot loader in the embedded front-computer software. The host application software is composed of four application subroutines, which match with the functional modules of the system hardware. A net communication program based on the server^client mode is implemented by means of socket programming and multi-thread programming. Test results indicate that the data transmission rate of the system is higher than 1 MB/s, which completely meets the current requirements of the data transmission rate between the tool system and the wireline telemetry device. Application of the debugging system, which includes multiple level test methods, shows that the proposed system can fully meet the test requirements of MPAL at various levels.展开更多
In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing ...In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing work in the same area. This implies that the data logger and monitoring system is good and can be used to monitor solar energy variables even at the comfort of our homes. We fitted a model to the generated data and found that the meteorological variables considered accounted for 99.88% of the power output in the rainy seasons while 0.12% of the variation was not explained due to other factors. Solar panels inclined at an angle of 5° (Tilt) and facing South Pole perform optimally.展开更多
Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is in...Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is introduced to promote efficiency.However,most existing Transformer-based log anomaly detection methods convert unstructured log messages into structured templates by log parsing,which introduces parsing errors.They only extract simple semantic feature,which ignores other features,and are generally supervised,relying on the amount of labeled data.To overcome the limitations of existing methods,this paper proposes a novel unsupervised log anomaly detection method based on multi-feature(UMFLog).UMFLog includes two sub-models to consider two kinds of features:semantic feature and statistical feature,respectively.UMFLog applies the log original content with detailed parameters instead of templates or template IDs to avoid log parsing errors.In the first sub-model,UMFLog uses Bidirectional Encoder Representations from Transformers(BERT)instead of random initialization to extract effective semantic feature,and an unsupervised hypersphere-based Transformer model to learn compact log sequence representations and obtain anomaly candidates.In the second sub-model,UMFLog exploits a statistical feature-based Variational Autoencoder(VAE)about word occurrence times to identify the final anomaly from anomaly candidates.Extensive experiments and evaluations are conducted on three real public log datasets.The results show that UMFLog significantly improves F1-scores compared to the state-of-the-art(SOTA)methods because of the multi-feature.展开更多
基金supported by the National Key R&D Program(2022YFB4500704)the National Natural Science Foundation of China(Grant No.62032008).
文摘Container-based virtualization is increasingly popular in cloud computing due to its efficiency and flexibility.Isolation is a fundamental property of containers and weak isolation could cause significant performance degradation and security vulnerability.However,existing works have almost not discussed the isolation problems of system log which is critical for monitoring and maintenance of containerized applications.In this paper,we present a detailed isolation analysis of system log in current container environment.First,we find several system log isolation problems which can cause significant impacts on system usability,security,and efficiency.For example,system log accidentally exposes information of host and co-resident containers to one container,causing information leakage.Second,we reveal that the root cause of these isolation problems is that containers share the global log configuration,the same log storage,and the global log view.To address these problems,we design and implement a system named private logs(POGs).POGs provides each container with its own log configuration and stores logs individually for each container,avoiding log configuration and storage sharing,respectively.In addition,POGs enables private log view to help distinguish which container the logs belong to.The experimental results show that POGs can effectively enhance system log isolation for containers with negligible performance overhead.
基金funded by the Hainan Provincial Natural Science Foundation Project(Grant No.622RC675)the National Natural Science Foundation of China(Grant No.62262019).
文摘As computer data grows exponentially,detecting anomalies within system logs has become increasingly important.Current research on log anomaly detection largely depends on log templates derived from log parsing.Word embedding is utilized to extract information from these templates.However,this method neglects a portion of the content within the logs and confronts the challenge of data imbalance among various log template types after parsing.Currently,specialized research on data imbalance across log template categories remains scarce.A dual-attention-based log anomaly detection model(LogDA),which leveraged data imbalance,was proposed to address these issues in the work.The LogDA model initially utilized a pre-trained model to extract semantic embedding from log templates.Besides,the similarity between embedding was calculated to discern the relationships among the various templates.Then,a Transformer model with a dual-attention mechanism was constructed to capture positional information and global dependencies.Compared to multiple baseline experiments across three public datasets,the proposed approach could improve precision,recall,and F1 scores.
基金ZTE Industry-University-Institute Cooperation Funds under Grant No.20200492.
文摘One particular challenge for large‑scale software systems is anomaly detection.System logs are a straightforward and common source of information for anomaly detection.Existing log‑based anomaly detectors are unusable in real‑world industrial systems due to high false‑positive rates.In this paper,we incorporate human feedback to adjust the detection model structure to reduce false positives.We apply our approach to two industrial large‑scale systems.Results have shown that our approach performs much better than state‑of‑the-art works with 50%higher accuracy.Besides,human feedback can reduce more than 70%of false positives and greatly improve detection precision.
文摘Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.
基金supported by National Science Foundation of China (61102102, 11134011, 11204380 and 11374371)Major National Science and Technology Projects (2011ZX05020-002)+2 种基金PetroChina Innovation Foundation (2014D-5006-0307)Science and Technology Project of CNPC (2014A-3912 and 2011B-4001)the Foundation of China University of Petroleum (KYJJ2012-05-07)
文摘This paper presents a debugging system for multi-pole array acoustic logging （MPAL） tools. The debugging system proposed in this study can debug the MPAL tool system, sub-system and local electronics. In the test equipment, we have used principal and subordinate structures, and interconnected the host computer and the front-end machine via Ethernet. The front-end machine is based on the ARM7 （advanced reduced instruction set computing （RISC） machine） technique, the processor of which runs an embedded operating system, namely, uClinux OS. We have analyzed the system telecommunication, human-machine interface circuit, transmitter mandrel interface circuit, receiver mandrel interface circuit, and board-level test interface circuit. The software used in the system consists of the embedded front-computer software and the host application software. We have explained in detail the flow chart of the boot loader in the embedded front-computer software. The host application software is composed of four application subroutines, which match with the functional modules of the system hardware. A net communication program based on the server^client mode is implemented by means of socket programming and multi-thread programming. Test results indicate that the data transmission rate of the system is higher than 1 MB/s, which completely meets the current requirements of the data transmission rate between the tool system and the wireline telemetry device. Application of the debugging system, which includes multiple level test methods, shows that the proposed system can fully meet the test requirements of MPAL at various levels.
文摘In this paper we have developed a data logging and monitoring system, we validated the system by comparing the result from it with the existing one and found that the system performs slightly better than the existing work in the same area. This implies that the data logger and monitoring system is good and can be used to monitor solar energy variables even at the comfort of our homes. We fitted a model to the generated data and found that the meteorological variables considered accounted for 99.88% of the power output in the rainy seasons while 0.12% of the variation was not explained due to other factors. Solar panels inclined at an angle of 5° (Tilt) and facing South Pole perform optimally.
基金supported in part by the National Natural Science Foundation of China under Grant 62272062the Scientific Research Fund of Hunan Provincial Transportation Department(No.202143)the Open Fund ofKey Laboratory of Safety Control of Bridge Engineering,Ministry of Education(Changsha University of Science Technology)under Grant 21KB07.
文摘Log anomaly detection is an important paradigm for system troubleshooting.Existing log anomaly detection based on Long Short-Term Memory(LSTM)networks is time-consuming to handle long sequences.Transformer model is introduced to promote efficiency.However,most existing Transformer-based log anomaly detection methods convert unstructured log messages into structured templates by log parsing,which introduces parsing errors.They only extract simple semantic feature,which ignores other features,and are generally supervised,relying on the amount of labeled data.To overcome the limitations of existing methods,this paper proposes a novel unsupervised log anomaly detection method based on multi-feature(UMFLog).UMFLog includes two sub-models to consider two kinds of features:semantic feature and statistical feature,respectively.UMFLog applies the log original content with detailed parameters instead of templates or template IDs to avoid log parsing errors.In the first sub-model,UMFLog uses Bidirectional Encoder Representations from Transformers(BERT)instead of random initialization to extract effective semantic feature,and an unsupervised hypersphere-based Transformer model to learn compact log sequence representations and obtain anomaly candidates.In the second sub-model,UMFLog exploits a statistical feature-based Variational Autoencoder(VAE)about word occurrence times to identify the final anomaly from anomaly candidates.Extensive experiments and evaluations are conducted on three real public log datasets.The results show that UMFLog significantly improves F1-scores compared to the state-of-the-art(SOTA)methods because of the multi-feature.
