ELK Stack users to extract data not through the API with one of the most common cases, but this is very useful for some cases. For example, using the REST API package its database services for commercial developers might be interested in this analysis purposes intelligence data. In today's article, we'll describe how to use ELK stack to a Service API for analysis.
For whatever reason, ELK Stack provides a simple way to integrate with the API. One of these methods is Logstash HTTP poller input plug. In the following example, I will use this plug-in to get some weather data via the public API disclosure.
To perform these steps and build their own pipeline to the HTTP API ELK analysis, you need to set the following:
- Elastic: Rookie Start Guide : install their own Elasticsearch and Kibana
- How to install Elastic stack Logstash : install their own Logstash
- In https://openweathermap.org/api register an account link. You can https://openweathermap.org/guide find the registration way and at the https://home.openweathermap.org/api_keys find their API keys.
Prepare data
As mentioned above, we use OpenWeatherMap API to collect the current weather data. The data set includes statistical information on existing weather conditions around the world gathered from 40,000 weather stations.
The data itself can JSON, XML or HTML formats, but the default format is JSON, which is very suitable for our use case, because we are using Elasticsearch index data. There are several ways to call data - you can, City ID, geographic coordinates or zip code to make the call provided by the city name (we will use the method here). You can also call multiple cities within a specific coordinate.
The following example calls the weather statistics in a bounding box that contains latitude and longitude coordinates:
http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10&appid=YourAppKeyIn the above, BBOX defined as [lon-left, lat-bottom , lon-right, lat-top, zoom], i.e. it is represented by a rectangular area. In the above appid, we must fill their own App Key before they can work.
API returns data provides us with all the statistical information needed to measure the current weather conditions, including temperature, humidity, atmospheric pressure, wind speed and direction.
This is an example:
{"cod":200,"calctime":0.00256473,"cnt":15,"list":[{"id":2563191,"dt":1581912559,"name":"Birkirkara","coord":{"Lon":14.46,"Lat":35.9},"main":{"temp":8,"feels_like":6.94,"temp_min":8,"temp_max":8,"pressure":1028,"humidity":93},"visibility":10000,"wind":{"speed":0.5},"rain":null,"snow":null,"clouds":{"today":1},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210247,"dt":1581912558,"name":"Tripoli","coord":{"Lon":13.19,"Lat":32.88},"main":{"temp":14.62,"feels_like":12.78,"temp_min":14.62,"temp_max":14.62,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":59},"wind":{"speed":1.53,"deg":42},"rain":null,"snow":null,"clouds":{"today":12},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]},{"id":2216885,"dt":1581912558,"name":"Zawiya","coord":{"Lon":12.73,"Lat":32.75},"main":{"temp":10.33,"feels_like":8.14,"temp_min":10.33,"temp_max":10.33,"pressure":1027,"sea_level":1027,"grnd_level":1015,"humidity":77},"wind":{"speed":1.96,"deg":87},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2212771,"dt":1581912558,"name":"Sabratah","coord":{"Lon":12.49,"Lat":32.79},"main":{"temp":14.59,"feels_like":12.27,"temp_min":14.59,"temp_max":14.59,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":57},"wind":{"speed":2.05,"deg":57},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2215163,"dt":1581912558,"name":"Masallatah","coord":{"Lon":14,"Lat":32.62},"main":{"temp":8.75,"feels_like":6.93,"temp_min":8.75,"temp_max":8.75,"pressure":1027,"sea_level":1027,"grnd_level":995,"humidity":77},"wind":{"speed":0.98,"deg":309},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2219905,"dt":1581912558,"name":"Al Khums","coord":{"Lon":14.26,"Lat":32.65},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2208425,"dt":1581912558,"name":"Zuwarah","coord":{"Lon":12.08,"Lat":32.93},"main":{"temp":12.65,"feels_like":10.83,"temp_min":12.65,"temp_max":12.65,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":64},"wind":{"speed":1.3,"deg":63},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210221,"dt":1581912558,"name":"Tarhuna","coord":{"Lon":13.63,"Lat":32.44},"main":{"temp":7.58,"feels_like":6.11,"temp_min":7.58,"temp_max":7.58,"pressure":1027,"sea_level":1027,"grnd_level":992,"humidity":85},"wind":{"speed":0.56,"deg":142},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208485,"dt":1581912558,"name":"Zlitan","coord":{"Lon":14.57,"Lat":32.47},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2217362,"dt":1581912558,"name":"Gharyan","coord":{"Lon":13.02,"Lat":32.17},"main":{"temp":5.59,"feels_like":4.01,"temp_min":5.59,"temp_max":5.59,"pressure":1027,"sea_level":1027,"grnd_level":940,"humidity":90},"wind":{"speed":0.4,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523693,"dt":1581912495,"name":"Pozzallo","coord":{"Lon":14.85,"Lat":36.73},"main":{"temp":5.56,"feels_like":2.34,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2524119,"dt":1581912495,"name":"Modica","coord":{"Lon":14.77,"Lat":36.85},"main":{"temp":5.73,"feels_like":2.54,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208791,"dt":1581912558,"name":"Yafran","coord":{"Lon":12.53,"Lat":32.06},"main":{"temp":7.01,"feels_like":5.31,"temp_min":7.01,"temp_max":7.01,"pressure":1027,"sea_level":1027,"grnd_level":959,"humidity":84},"wind":{"speed":0.68,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523581,"dt":1581912494,"name":"Rosolini","coord":{"Lon":14.95,"Lat":36.82},"main":{"temp":5.32,"feels_like":2.06,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523650,"dt":1581912495,"name":"Ragusa","coord":{"Lon":14.72,"Lat":36.93},"main":{"temp":5.52,"feels_like":2.29,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":22},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]}]}
Configuration Logstash
Method ELK introduced into the stack by using the data returned by the API Logstash http poller input plug-in that can be defined from the endpoint URL of the polymerization, and the decoded transport data. We first create a configuration file named logstash_openweathermap.conf name.
Logstash input
Http_poller input section defines an input plug - URL endpoints polling request times out, based CRON schedule (every 5 minutes) and codec (JSON) to be used. metadata_target setting is optional, and adds some fields related to the response to the polling performance.
input {
http_poller {
urls => {
url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
}
request_timeout => 60
schedule => { every => "5m"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}We want to include the following query parameters after calling the API URL inserted here:
- q = London, uk- asked to return to London weather data
- APPID = - OpenWeatherMap API key
- units = metrics - to convert to Celsius units format
Shown above is the url of micro-services interfaces . The results it returns are:
{
"coord": {
"lon": -0.13,
"lat": 51.51
},
"weather": [
{
"id": 802,
"main": "Clouds",
"description": "scattered clouds",
"icon": "03n"
}
],
"base": "stations",
"main": {
"temp": 6.7,
"feels_like": 0.46,
"temp_min": 5,
"temp_max": 8.33,
"pressure": 1006,
"humidity": 65
},
"visibility": 10000,
"wind": {
"speed": 6.2,
"deg": 230
},
"clouds": {
"all": 40
},
"dt": 1581916140,
"sys": {
"type": 1,
"id": 1414,
"country": "GB",
"sunrise": 1581923520,
"sunset": 1581959845
},
"timezone": 0,
"id": 2643743,
"name": "London",
"cod": 200
}You can use https://jsonformatter.org/json-viewer to format the results returned by the API.
Logstash filter
Since the use of the API response JSON, Logstash therefore require little or no analytical treatment. Therefore, we can now filter section blank.
filter {}Logstash output
The following part of this very straightforward:
output {
elasticsearch {
index => "openweather"
document_type => "_doc"
hosts => "localhost:9200"
}
}In the above, we put the data into localhost: 9200's Elasticsearch, and the index name is openweather.
Integrated the above configuration, our ultimate longstash_openweather.conf as follows:
input {
http_poller {
urls => {
url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
}
request_timeout => 60
schedule => { every => "1m"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter {}
output {
stdout {
codec => rubydebug
}
elasticsearch {
index => "openweather"
document_type => "_doc"
hosts => "localhost:9200"
}
}
Start Logstash. We run Logstash in the installation directory of logstash:
./bin/logstash -f ~/data/logstash_openweather.confYou will see the use of new Elasticsearch index weather data created. Define a new index model with Kibana to start analyzing:
analyze data
We called index pattern openweather by creating. You can then view by Discover:
Use openweather index pattern, we can do it our dashboard:
