Elasticsearch: What is machine learning?

Machine learning definition

Machine learning (ML) is a branch of artificial intelligence (AI) that focuses on using data and algorithms to imitate the way humans learn and gradually improve accuracy over time. Computer scientist and AI innovator Arthur Samuel first defined it in the 1950s as "a field of study that gives computers the ability to learn without being explicitly programmed."

Machine learning involves feeding large amounts of data into computer algorithms so that they can learn to recognize patterns and relationships in that data set. The algorithm then starts making its own predictions or decisions based on the analysis. As algorithms receive new data, they continue to refine their choices and improve their performance, just as a person gets better at an activity through practice.

What are the four types of machine learning?

The four types of machine learning aresupervised machine learning, unsupervised machine learning. reinforcement learning and semi-supervised learning,

Supervised machine learning is the most common type of machine learning. In a supervised learning model, an algorithm learns from a labeled training data set and improves its accuracy over time. It aims to build a model that can correctly predict a target variable when receiving new data that has never been seen before. An example is humans labeling and inputting images of roses and other flowers. When the algorithm receives a new, unlabeled image of a rose, it can correctly identify a rose.
Unsupervised machine learning is when algorithms search for patterns in unlabeled data with no target variable. The goal is to find patterns and relationships in data that humans may not yet recognize, such as detecting anomalies in logs, traces, and metrics to uncover system issues and security threats.
Semi-supervised learning is a hybrid of supervised and unsupervised machine learning. In semi-supervised learning, the algorithm is trained on both labeled and unlabeled data. It first learns from a small set of labeled data to make predictions or decisions based on the available information. It then uses larger unlabeled data sets to refine its predictions or decisions by finding patterns and relationships in the data.
Reinforcement learning means that the algorithm obtains feedback in the form of rewards or punishments for its behavior and learns through repeated trials. Some examples include: training an AI agent to play video games, where it is positively rewarded for improving levels and penalized for failing; optimizing supply chains, where agents are rewarded for minimizing costs and maximizing delivery speed; or recommendations A system where agents recommend products or content and are rewarded for purchases and clicks.

How does machine learning work?

Machine learning can work in different ways. You can apply a trained machine learning model to new data or train a new model from scratch.

Applying a trained machine learning model to new data is typically a faster and less resource-intensive process. Instead of developing parameters through training, you use the parameters of the model to make predictions about input data, a process called inference. You also don't need to evaluate its performance since it is already evaluated during the training phase. However, it does require you to carefully prepare the input data to ensure that it is in the same format as the data used to train the model.

Training a new machine learning model involves the following steps:

data collection

Start by selecting your dataset. Data can come from a variety of sources, such as system logs, metrics, and traces. In addition to logs and metrics, several other types of time series data are important in machine learning training, including:

Financial market data, such as stock prices, interest rates, and foreign exchange rates. This data is often used to build predictive models for trading and investing purposes.
Traffic time series data, such as traffic volumes, speeds, and travel times. This can be used to optimize routes and reduce traffic congestion.
Product usage data, such as website traffic and social media engagement. This helps companies understand customer behavior and identify areas for improvement.

Whatever data you use, it should be relevant to the problem you are trying to solve and should be representative of the population you want to make predictions or decisions about.

Data preprocessing

After you collect the data, you need to preprocess it to make it usable by machine learning algorithms. This sometimes involves labeling the data, or assigning a specific category or value to each data point in the dataset, which allows the machine learning model to learn patterns and make predictions.

Additionally, it may involve removing missing values, converting time series data into a more compact format by applying aggregation, and scaling the data to ensure that all features have similar ranges. Having large amounts of labeled training data is a requirement for deep neural networks, such as large language models (LLMs). With a classic supervised model, you don't have to deal with that much.

Feature selection

Some methods require you to select features that the model will use. Essentially, you have to identify the variables or attributes that are most relevant to the problem you want to solve. Correlation is the basic method of identifying features. For further optimization, many machine learning frameworks provide and support automatic feature selection methods.

Selection

Now that you have selected your features, you need to choose a machine learning model that is a good fit for the problem you are trying to solve. Some options include regression models, decision trees, and neural networks. (See "Machine Learning Techniques and Algorithms" below.)

train

After selecting a model, you need to train it using the collected and preprocessed data. Training is the process by which an algorithm learns to recognize patterns and relationships in data and encode them into model parameters. To achieve optimal performance, training is an iterative process. This can include tuning model hyperparameters and improving data processing and feature selection.

test

Now that the model has been trained, you need to test it on new data that has not been seen before and compare its performance with other models. You select the best performing model and evaluate its performance based on separate test data. Only previously unused data will give you a good estimate of your model's performance after deployment.

Model deployment

Once you are satisfied with the performance of your model, you can deploy it in a production environment where it can make predictions or decisions in real time. This may involve integrating the model with other systems or software applications. Machine learning frameworks integrated with popular cloud computing providers make it easy to deploy models to the cloud.

Monitor and update

After you deploy your model, you'll want to monitor its performance and update it periodically as new data becomes available or the problem you're trying to solve changes over time. This might mean retraining the model with new data, adjusting its parameters, or choosing a different machine learning algorithm entirely.

Why is machine learning important?

Machine learning is important because it learns to perform complex tasks using examples without having to write specialized algorithms. Compared to traditional algorithmic approaches, machine learning enables you to automate more, improve customer experience, and create innovative applications that were previously unfeasible. Moreover, machine learning models can iteratively improve themselves as they are used! Examples include:

Anticipate trends to improve business decisions
Personalized recommendations increase revenue and customer satisfaction
Automatically monitor complex applications and IT infrastructure
Identify spam and discover security vulnerabilities

Machine learning techniques and algorithms

There are many machine learning techniques and algorithms available. The method you choose will depend on the problem you are trying to solve and the characteristics of your data. Here's a quick overview of some of the more common ones: Linear regression is used when the goal is to predict a continuous variable.

Linear regression (linear regression) assumes a linear relationship between the input variable and the target variable. One example is predicting home prices as a linear combination of square footage, location, number of bedrooms, and other characteristics.
Logistic regression (logistic regression) is used for binary classification problems where the goal is to predict a yes/no outcome. Logistic regression estimates the probability of a target variable based on a linear model of the input variables. One example is predicting whether a loan application will be approved based on the applicant's credit score and other financial data.
Decision trees (decision trrees) follow a tree model that maps decisions to possible consequences. Each decision (rule) represents a test of an input variable, and multiple rules can be applied consecutively following a tree model. It divides the data into subsets using the most important features at each node of the tree. For example, decision trees can be used to identify potential customers for marketing campaigns based on demographics and interests.
Random Forest (random forests) combine multiple decision trees to improve prediction accuracy. Each decision tree is trained on a random subset of the training data and a subset of the input variables. Random forests are more accurate than individual decision trees and can handle complex datasets or missing data better, but they can become quite large and require more memory when used in inference.
Boosted decision trees (boosted decsion trees) trains a series of decision trees, each of which is an improvement over the previous one. The boosting process takes data points that were misclassified by previous iterations of the decision tree and retrains a new decision tree to improve the classification of these previously misclassified points. The popular XGBoost Python package implements this algorithm.
Support vector machines (support vector machines) strive to find a hyperplane that best separates one class of data points from another. It does this by minimizing the "gap" between classes. A support vector refers to a small number of observations that identify the location of a separating hyperplane, which is defined by three points. The standard SVM algorithm only works for binary classification. Multi-class problems are reduced to a series of binary problems.
Neural network (neural network) is inspired by the structure and function of the human brain. They consist of layers of interconnected nodes that can learn to recognize patterns in the data by adjusting the strength of connections between nodes.
Clustering algorithms (clustering algorithms) are used to group data points into clusters based on their similarity. They can be used for tasks such as customer segmentation and anomaly detection. It is particularly useful for image segmentation and processing.

What are the advantages of machine learning?

The advantages of machine learning are many. It can help your team reach higher performance levels in the following categories:

Automation: Cognitive tasks that are challenging for humans due to repetitiveness or objective difficulty can be automated through machine learning. Examples include monitoring complex network systems, identifying suspicious activity in complex systems, and predicting when equipment will require maintenance.
Customer Experience: Machine learning models provide intelligence that improves user experience. For search-driven applications, capturing intent and preferences allows you to deliver more relevant and personalized results. Users can search and find their meaning.
Innovation: Machine learning solves complex problems that cannot be solved by specialized algorithms. For example, searching unstructured data including images or sounds, optimizing traffic patterns and improving public transportation systems, and diagnosing health conditions.

Learn how machine learning is taking search to the next level

Machine learning use cases

Here are some subcategories of machine learning and their use cases:

Sentiment analysis (sentiment analysis) is the process of using natural language processing to analyze text data and determine whether its overall sentiment is positive, negative, or neutral. It's useful for businesses seeking customer feedback, as it can analyze various data sources (such as tweets on Twitter, Facebook comments, and product reviews) to measure customer opinion and satisfaction.
Anomaly detection (anomaly detection) is the process of using algorithms to identify unusual patterns or outliers in data that may indicate a problem. Anomaly detection is used to monitor IT infrastructure, online applications, and networks and identify activities that indicate potential security vulnerabilities or could cause network outages. Anomaly detection is also used to detect fraudulent banking transactions. Learn more about AIOps .
Image recognition (image recognition) Analyzing images and identifying objects, faces, or other features in the images. It has a variety of applications in addition to common tools like Google Image Search. For example, it can be used in agriculture to monitor crop health and identify pests or diseases. Self-driving cars, medical imaging, surveillance systems, and augmented reality games all use image recognition.
Predictive analytics (predicative analytics) Analyzing historical data and identifying patterns that can be used to predict future events or trends. This can help businesses optimize operations, forecast demand or identify potential risks or opportunities. Some examples include product demand forecasts, transportation delays, and how long manufacturing equipment can operate safely.

Learn more about predictive maintenance

What are the disadvantages of machine learning?

Disadvantages of machine learning include:

Reliance on high-quality training data: If the data is biased or incomplete, the model may also be biased or inaccurate.
Cost: Training models and preprocessing data can be costly. That being said, it's still less costly than writing a specialized algorithm to do the same task, and most likely won't be as accurate.
Lack of explainability: Most machine learning models, such as deep neural networks, lack transparency into how they operate. Often referred to as “black box” models, this makes it challenging to understand how the model makes decisions.
Expertise: There are many models to choose from. Without a designated data science team, organizations may struggle to tune hyperparameters for optimal performance. The complexity of training, especially of transformers, embeddings, and large language models, can also be a barrier to adoption.

Best practices for machine learning

Some best practices for machine learning include:

Make sure your data is clean, organized and complete.
Choose the right method for your current problem and data.
Use techniques to prevent overfitting, where a model performs well on training data but performs poorly on new data.
Evaluate the performance of your model by testing it on completely unseen data. The performance you measure while developing and optimizing a model is not a good predictor of its performance in production.
Tweaking a model's settings for optimal performance is called hyperparameter tuning.
In addition to standard model accuracy, a number of metrics are selected to evaluate model performance in real-world applications and business problems.
Keep detailed records to ensure others can understand and replicate your work.
Keep your model up to date to ensure it continues to perform well on new data.

Get started with Elastic machine learning

Elastic Machine Learning builds on the strengths of ourscalable Elasticsearch platform. By working with observability, security and < You get value out of the box with the integration of a i=7>search solutionsthat use models that require less training to get up and running. With Elastic, you can gather new insights and deliver revolutionary experiences to internal users and customers, all with reliability at scale.

Learn how to:

Get data from hundreds of sources and apply machine learning and natural language processing where the data lives with built-in integrations.

Apply machine learning in the way that works best for you. Get out-of-the-box value from pre-configured models based on your use case: pre-configured models for automated monitoring and threat hunting, pre-trained models and transformers for implementing NLP tasks such as sentiment analysis or question-and-answer interactions , andElastic Learned Sparse Encoder™implement semantic search with one click. Or, if your use case requires optimization and custom models, train a supervised model using your data. Elastic gives you the flexibility to apply an approach that suits your use case and matches your level of expertise!