Big data technology for computer intrusion detection

1 Introduction

With the rapid development of computer, communication, and network technology, information network has become one of the important technologies in the process of sustainable development of countries, enterprises, and groups [1]. Considering the important impact of information network on various fields, people know that the issue of network security has become an important issue affecting the long-term interests and development of the country. In order to protect network data security, people have studied various security measures ranging from static security measures to effective security measures. The openness of the Internet makes the browsing and retrieval of data very fast. While enjoying the convenience of the Internet, people must also suffer the harm brought by network attacks. The information on the internet is complex, rich, and diverse, and the network cannot distinguish whether the information it carries is normal information or vicious attack. In daily life, there have been many incidents of data leakage caused by network attacks, and malicious network attacks have caused very serious economic losses [2].

As a new technology form, intrusion detection technology has more and more application in various technical fields and has become a research hotspot [3]. Intrusion detection is defined as a collection of behaviors that an unauthorized user illegally logs into a data server to steal and destroy data, or an authorized user performs illegal operations within the scope of authorized authority, exceeds the user’s authority, and endangers or threatens the confidentiality, consistency, and integrity of network resources. Intrusion detection is to identify a large number of attack patterns by analyzing user behavior characteristics, and then raise an alarm and take security protection measures. Intrusion detection system (IDS) is an application system used to deal with abnormal attacks in the network [4]. The traditional IDS mainly relies on expert system. By manually analyzing the data records and extracting the existing attack patterns, the attack patterns are stored according to certain rules. The stored attack patterns can be configured in the IDS to facilitate the data monitoring of the accepted network requests. Relatively speaking, the intrusion detection mechanism based on expert system is lagging behind, because it is configured in the system after manual analysis, and it is unable to identify new attack patterns. In addition, with the continuous increase in data volume, it is difficult to adapt to the demand by relying on manual pattern discovery. Intrusion detection depends on the observation of user behavior. Normal user behavior and abnormal user behavior can be distinguished. For example, normal users’ access to the server has a certain frequency and time interval. If the access frequency is too high, the current user behavior can be considered as malicious access behavior.

The research on intrusion detection technology is not only helpful to effectively solve the feature extraction of massive and rapidly changing network security data in scientific theory, but also helpful to the quantitative research of network security assessment, the research and judgment of network security situation under the condition of randomness and uncertainty of network attacks, and solve the scientific problems of network security perception measurement and integrated decision-making under the big data environment. It also helps to make key breakthroughs in the characteristics of network security big data and the core technologies such as feature extraction technology, network abnormal behavior classification and detection technology, information fusion technology, network security situation assessment technology, and network security situation prediction technology of multi-source heterogeneous network security big data in the big data environment. This helps identify network security risks as early as possible and prevent them in the future. It also provides guidelines for developing network security strategies and early warning of network security emergencies in big data environments. In short, intrusion detection technology has theoretical value, innovation, practice, and vision, and can help to solve problems such as understanding network security threat elements and their relationships in big data environments, extracting network security features, identifying attacks, implementing network security evaluation, and building prediction models, thereby providing theoretical basis and application technology for network security research [5].

2 Literature review

The network produces fast, large, and varied data, generating 2.5 exabytes of data every day [6]. Facebook has 1.65 billion monthly users and 1 billion active users; Wechat has more than 1 billion users and 700 million active users. With the gathering of big data in the network and the rapid development of big data analysis, the huge value hidden behind the data is gradually revealed. Network operators can take full advantage of big data to improve network performance and increase network revenue [7]. For large data networks, traditional data analysis has the following disadvantages.

1. Traditional data analysis methods are often associated with structured data, and many data sources are often unstructured.

2. The use of data analysis is usually limited to a department or business unit, and the final evaluation is based on the regional perspective rather than the general world perspective.

3. Modern technology focuses on data transfer and pays little attention to data processing, making real-time decision making impossible. Big data analytics can use data mining technology to extract more detailed information than traditional data analysis, integrating disparate data that is scattered across in the network, and explore the relationship between different elements of the network, which plays an important role in network planning and optimization, traffic prediction, monitoring, network security, and other issues. For example, processing data related to customers is often spread across multiple companies. Big data analytics can collect scattered data, understand consumer behavior and preferences from multiple perspectives, and thus paint the whole consumer picture [8]. Another important aspect of big data analysis is real-time functionality, which allows employees to monitor their locations in real-time and make personal decisions. As the internet evolves, many applications (such as networking, Internet of Things, smart grid, etc.) put more demands, such as flexible, fast, secure, smart, etc., on existing networks. To meet these requirements, through network integration and distributed computing of big data, a network platform based on big data can be created to enhance understanding of the network and awareness of data. However, in the use of large-scale technology, there are some problems such as data security and data failure. Such problems create a big gap in the creation of business ideas, increase the cost of production and work of enterprises, and cause a loss in production and work. Threats to the security of information may cause damage to human life and property [9]. Big data can be used for good, but criminals will use it as a tool for crime. In order to prevent negative impact on people, it is necessary to continuously strengthen the use of big data in the management of computer network data.

At present, some scholars have combed and summarized some applications of big data in the network field. Tomi and Todorovic summarized the research work of big data in wireless networks from four aspects: application layer, network layer, data transmission layer, and data layer [10]. Gupta and Lakhwani reviewed the research work of big data analysis in wireless communication technology, mainly including wireless spectrum control, network planning and optimization, and wireless resource management [11]. Wulandari et al. introduced advanced persistent threat attack detection based on big data analysis [12]. Qu and Wang summarized the role of big data in network security and intelligence analysis. In general, only a few foreign scholars have conducted an overall study on the application of big data technology in the network [13].

In order to support domestic research in these areas, it is very important to review the research on the development of big data in the network. In this article, big data for computer intrusion detection. Algorithms for clustering, classification, and association rules in data are used only to identify network attack patterns, and quickly learn and extract network attack characteristics.

Data mining technology, a very important tool in the era of big data, can extract rules and patterns from large amounts of data through examination analysis. Therefore, these tools are gradually replacing professional methods of research and are increasingly being used.

3 Research methods

3.1 Big data technology

3.1.1 Concept of big data

Big data is a data type that contains many records and has a large type. It can be created and changed very quickly, and is difficult to classify and process well. From the perspective of resources, big data are a new resource that reflects the new perspective of resources. From a technological perspective, big data represent a new generation of data management and analysis technologies. Conceptually, big data defines a new way of thinking, which brings new meaning to data processing. Its characteristics and processing process are shown in Figures 1 and 2.

Figure 1

Characteristics of big data.

Figure 2

Big data processing mode.

3.1.2 Classification of network big data

In view of the problems that technology enterprises, researchers, and data analysts have, there are different classifications of network big data due to their different concerns. Table 1 provides relevant summaries to help researchers better understand the profound connotation of network big data.

Table 1

Classification of network big data

Classification of angle	Category name
Data in the access network and core network	Data in IT systems
	Telecommunication operator
	Data in carrier internet applications
Potential applications	Stream record data
	Network performance data
	Mobile terminal data
	Additional data
Network social ecology	Raw wireless big data
	Derivative wireless big data
	Developing wireless big data
Application field	Cellular networks, Wi-Fi hotspots
	Smart phone D2D, smart grid
	Wireless sensor networks, Internet of Things

3.2 Data mining

3.2.1 Definition of data mining

Data mining is the process of finding important information and knowledge hidden in large, incomplete, and random data that people did not know before. This specification states that the original document should be large, objective, functional, and sound. What is extracted is the knowledge that the user wants to know and understand, to use. Data mining is a data processing technology, whose main meaning is to extract data features, transform data types, and analyze hidden values in data. It can provide data reference for important decisions of managers.

3.2.2 Tasks and functions of data mining

At present, the main research topics of data mining include the theoretical foundation, data storage, discovery algorithm, efficient and quantitative exchange model, machine vision, storage and reuse of knowledge discovery, knowledge representation, knowledge discovery in semi-structured and unstructured data, and web-based data mining. The most common types of knowledge found from the data are shown in Table 2.

Table 2

Classification knowledge of data mining

Classification	Concept description
Related knowledge	Refers to the knowledge that one event is interdependent or related to another event or other events. If two or more of the attributes are interdependent, the attribute value of one of them can be predicted based on the attribute bits associated with it.
General knowledge	Generalized knowledge refers to the general description knowledge of attribute characteristics. According to the attribute representation of data, similar universal and higher level knowledge are discovered, which can reflect the common attribute characteristics of similar things. It is a process of extracting, summarizing, and abstracting data.
Predictive knowledge	The main idea of predictive knowledge is to use correlation algorithm to predict future data based on past and current data, which is a kind of related knowledge with time as its attribute. Time series prediction methods are commonly used, including statistical methods, neural networks, and machine learning.
Classification of knowledge	Classification knowledge represents the common attribute knowledge of the same kind of things and the difference attribute knowledge of different things. The classical classification method is a decision tree classification force one method, which is a guided classification method and builds decision trees from sample sets.
Deviant type knowledge	Deviant knowledge is a kind of descriptive knowledge of differences and extreme special cases, reflecting the abnormal phenomenon that things deviate from the conventional class or standard, such as the description of special cases outside the standard class, the description of the difference between the actual observed value and the predicted bit of the system, etc.

Based on the above research contents, data mining is mainly divided into five types of functions, as shown in Figure 3.

Figure 3

Functions of data mining.

3.2.3 Process of data mining

Data mining refers to the process of successfully extracting significant, unknown, and complete information from data that can provide a clear basis for decision-making and support knowledge. The basic structure of the data are shown in Figures 4 and 5.

Figure 4

The process of data mining.

Figure 5

Schematic diagram of the overall structure of data mining system.

3.3 Intrusion detection technology

3.3.1 Principles of intrusion detection technology

Under normal circumstances, the computer network IDS detection data is generated when the normal operation of computer network of data objects, reflects the intrusion characteristics of external attackers to achieve efficient detection results [14]. According to the specific timing of computer network intrusion detection, it can be divided into real-time intrusion detection and post-intrusion detection. The working principle of network access is shown in Figure 6.

Figure 6

Working principle diagram of network IDS.

3.3.2 IDS framework

Computer network IDS makes use of the standard format of some security events alarm in computer network team work. On the premise of regulating the use of computer network terms, IDS provides more possibilities for the information diversity of security incidents output by computer network IDS. The whole framework model of IDS is shown in Figure 7.

Figure 7

IDS framework.

3.3.3 Network intrusion detection technology

Network intrusion detection technology is a security defense technology to identify and deal with the attacks or malicious acts launched by computers and networks. The advantages and disadvantages of intrusion detection technology based on machine learning are analyzed in Table 3. The typical representative of supervised learning is classification method, including Bayesian classification, support vector machine, decision tree, K-nearest neighbor classification method, etc. In solving classification problems, supervised learning establishes the mapping relationship between data and their corresponding labels, and uses labeled data to construct classifiers. The typical representative of unsupervised learning is clustering method, etc. [15].

Table 3

Intrusion detection techniques based on machine learning

Algorithm name	Supervised/unsupervised	Advantages	Disadvantages
Bayesian classification incremental training	Supervision	It is suitable for small-scale data, the classification effect is relatively stable, and it needs to know the prior probability	The classification effect is poor for those with many feature attributes and large correlation.
Support vector machine	Supervision	Suitable for small sample linear/nonlinear classification, with good learning ability	It is not suitable for large-scale training samples, and the operation time is long when the number of samples is large
Decision tree	Supervision	It is easy to understand and implement, and has a good classification or prediction effect for large data sources	For chronological data, a lot of preprocessing work is needed, and the error rate is high when there are many categories
K-nearest neighbor	Supervision	The classification is simple. It can solve multiple classification problems and is insensitive to outliers	High computational complexity and low efficiency
Clustering method	Supervision	Easy to understand, simple, high efficiency, low computational complexity	It is affected by the number of clusters and initial distribution

3.4 Data mining algorithm in IDS

IDS identifies and monitors data streams in real time. When abnormal data are detected in the data stream, it will report the data. The monitoring is based on the preset mode, which shows that a new attack mode comes from the real-time analysis of real-time data. The algorithms commonly used in intrusion detection of big data technology include clustering algorithm, classification algorithm, and association rule mining algorithm.

3.4.1 Clustering algorithm

The amount of IDS data is usually very large. Intrusion detection depends on attack modes that require labeled data samples. The original data of IDS is mostly unlabeled data. The clustering algorithm in data mining can complete the partitioning of unlabeled data well and keep the data purity well.

Clustering algorithm is a non-supervisory algorithm, and commonly used clustering algorithms include K-means clustering algorithm, hierarchical clustering algorithm, and density-based clustering algorithm [16]. K-means clustering algorithm has gained a large number of applications because its theory and implementation are easy to understand. The algorithm steps of K-means clustering algorithm is expressed in Table 4.

Table 4

K-means clustering algorithm flow table

Input:

1. Preprocessed data points p 1 , p 2 , ⋯ , p n

2. Data center cluster K

1. K points are randomly selected from N data points, and the selected K points are used as the initial center points of clustering.

2. For the data points, the distance from each data point to the center point is calculated, and the data points are divided into cluster C i ( 1 ⩽ i , ⩽ k ) with the nearest distance;

3. Calculate the sum d i of mean square deviation from each point of cluster C i ( 1 ⩽ i , ⩽ k ) to cluster center, sum d i corresponding to each cluster C i , and judge whether convergent;

4. If the current algorithm converges, output each cluster C i ; otherwise, adjust the center point of each cluster C i and proceed to Step 2.

Output: after the completion of the clustering of a cluster, C 1 , C 2 , ⋯ , C k

In Step 2 of the algorithm process, the distance from the data point to the center point is calculated using the European distance, as shown in equation (1).

(1) d ( d i , c j ) = ∑ t ( w t i − w t j ) 2 .

In equation (1), the data point and the center point are represented in the form of vectors, and each dimension of the vector is defined as a feature of the point, representing the feature weight of dimension T. The purpose of K-means clustering algorithm is to maximize the distance between clusters and minimize the distance within clusters, as shown in equation (2).

(2) min ∑ j 1 K × ∑ i d ( d i , c j ) , max ∑ j 1 K × ∑ i d ( c i , c j ) .

K-means clustering algorithm has been applied on a large scale because of the simplicity of clustering. In actual clustering analysis, data are aggregated into normal data and abnormal data according to the collected data, and different clustering clusters are marked as normal mode and abnormal attack mode. In actual intrusion detection, the corresponding clustering cluster is divided according to the data condition to carry out attack pattern recognition [17].

3.4.2 Classification algorithm

Since the goal of intrusion detection is to detect unusual information that is not found in many documents, the need to distinguish and classify real information is a problem that classification systems must solve. According to the different characteristics of data, the classification algorithm makes a distinction between the variables corresponding to the new data, and finally divides the new data into similar groups.

The classification algorithm is more suitable for the classification effect of nonlinear model, and the classification algorithms commonly used in classification include support vector machine, decision tree algorithm, neural network algorithm, etc. Because of its complex network structure, neural network can fit and self-learn many latent features, so it has good effect in classification algorithm [18]. Taking neural network as an example, neural networks use three layers in the network: input layer, hidden layer, and output layer. A hidden layer itself can consist of multiple hidden layers. The input method is the first input data, and the hidden method is used to explore the properties of the input method. After developing many implicit methods, the features are learned more and more. When the scale of the feature is sufficient, it can be identified by the output layer. The classical neural network structure is shown in Figure 8.

Figure 8

Structure diagram of neural network.

The neural network transmits the data to the hidden layer in the input layer, and the hidden layer transmits the data backward after the data transformation through the excitation function. Finally, the output is carried out in the output layer, and the difference between the output value of the output layer and the target value of the actual sample is calculated. The difference is propagated backward through gradient descent. During backward propagation, gradient attenuation is carried out in turn according to the path law of the sample, so as to update the weight value of the entire network path, as shown in Figure 9.

Figure 9

Data transformation of neural network excitation function.

Compared with other linear models, the biggest feature of neural network is the nonlinear transformation of data, which can be mapped through the excitation function so as to realize the distributable representation of data. Common excitation functions such as Sigmoid, Tanh, and ReLU are adopted in this essay. The classical Sigmoid function is shown in equation (3).

(3) f ( h ( x ) ) = 1 1 + e − h ( x ) .

In equation (3), it can be seen that the pair is nonlinearly transformed as the cumulative sum of the input data, as shown in equation (4).

(4) h ( x ) = ∑ i x i × w i .

Neural network can fully learn the characteristics of data through complex network structure, so it has a very obvious effect on distinguishing normal data from abnormal data. However, the complex network structure has very high requirements on the whole algorithm model, and the time cost will be very large. The continuous expansion of parameters is easy to lead to the problem of overfitting, which are inevitable problems of neural networks.

3.4.3 Association rule mining algorithm

Association rule algorithm is the earliest technology applied to network intrusion detection. It mainly discovers malicious attacks and related attack patterns by analyzing data correlation. Association rule algorithm can well discover the correlation between data [19,20]. The data format of association rules is shown in Table 5. Association rules define a data sample as a transaction, and the sample set can be defined as a transaction set. The purpose of association rules is to mine the rules in all transactions. In Table 5, in transactions T1 and T3, A, B, and E appear simultaneously in the transaction, which indicates that if events A and B occur, the support degree of the subsequent event E is 2. Association rules define the hidden rules in the data through evaluation indexes such as support degree and execution degree.

Table 5

Association rule data format table

T1	A, B, D, E, F
T2	D, F, G, K, I
T3	A, B, E, J
T4	C, D, F, G
T5	F, G

Apriori algorithm is the commonly used algorithm in association rules. The algorithm flow of Apriori is as follows: Suppose I = ( A , B , C ⋯ Z ) is A set of items, given a transaction database, each transaction is a non-null version, that is, each transaction corresponds to a TID identifier. Federal policy support in D is the proportion of X and Y in the exchange in D, i.e., the outcome, while trust is the proportion of the two having the same Y in the exchange of X, i.e., the custom. If the user reaches the minimum support and confidence level, the relevant policy is considered valid.

3.5 IDS system architecture

As a network IDS, IDS needs to monitor abnormal data in real time. The system design of IDS should include four aspects: real-time data streaming access part, real-time data processing and analysis part, data background offline mode analysis part, and data output part, all of which are connected together to form a complete set of IDS.

The real-time data streaming access part mainly collects real-time data in the system. The real-time data processing and analysis part analyzes the real-time data online and outputs the results of online analysis to the end users for control. The offline mode analysis part collects data in large quantities for pattern mining, as shown in Figure 10.

Figure 10

IDS system structure diagram.

4 Result analysis

In order to test the effectiveness of the IDS, in this article, the historical command operation record of the user server is collected as the test data, and the daily command of the user server is used as the normal data. At the same time, it randomly forges anomalies as intrusion commands, and constantly adjusts the proportion of intrusion commands to compare the accuracy of clustering algorithm, classification algorithm, and association rules in intrusion detection, as shown in Figure 11.

Figure 11

Comparison of detection accuracy of clustering algorithm, classification algorithm, and association rule algorithm.

In Figure 11, the abscissa represents the proportion of abnormal commands and the ordinate represents the accuracy. Figure 11 shows that the recognition rate of the classification algorithm is better than the joint method and the association rule, and the accuracy of 90% can still control the growth of abnormal command. In contrast, the recognition accuracy of the clustering algorithm is low, and the robustness is poor, which also proves that the clustering algorithm itself is too noisy. As a compromise between classification algorithm and clustering algorithm, the accuracy of association rule algorithm is basically kept above 75%.

5 Conclusion

This study proposes a big data technology for computer intrusion detection. The main content is to build a network intrusion detection model based on big data technology to automatically identify the attack patterns in the network by using clustering, classification, and association rule algorithms in data mining, and quickly learn and extract the characteristic forms of network attacks. Finally, by comparing the accuracy of clustering algorithm, classification algorithm, and association rules in intrusion detection, it is concluded that data mining technology based on big data is helpful to improve the effect of intrusion detection. It is proved that the big data technology oriented to computer intrusion detection can effectively improve the ability of computer intrusion detection. In particular, the classification algorithm has high accuracy and can be selectively used in practical applications.