Upgrading Graylog2 to Graylog (sounds confusing, I know).
We are going to upgrade Graylog 1.3.x that we set up some time ago to Graylog 2.0.x. Official upgrade instructions can be found here: http://docs.graylog.org/en/2.0/pages/upgrade.html.
The reason for upgrading to Graylog 2.0.x and not to Graylog 2.1.x is that Graylog 2.0.x still uses separate listeners for the REST API and the web interface by default. Beginning with Graylog 2.1.0 it is possible to run both the REST API and the web interface on the same host/port-combination and this is now the default. Upgrade to Graylog 2.0.x is a major change on its own, therefore we want to keep things as less complicated as possible. We will upgrade to 2.1.x afterwards.
Software
Software used in this article:
- CentOS 6.8 minimal
- Elasticsearch 1.7.6 -> will be upgraded to Elasticsearch 2.4.2
- Graylog server 1.3.4 -> will be upgraded to Graylog server 2.0.3
- Graylog web 1.3.4 -> will be removed
- Java 1.8.0 openjdk -> no change
- MongoDB 3.2.11 -> no change
SELinux is set to enforcing mode. Firewall is on and is configured to allow incoming traffic on TCP ports 9000 and 12900. Graylog server’s internal IP is 10.8.143.13.
MongoDB
Graylog 2.x requires MongoDB 2.4 or newer. We are on 3.2 already, therefore no change is required.
# mongo --version MongoDB shell version: 3.2.11
Elasticsearch 2.x
Graylog 2.0.0 and higher requires Elasticsearch 2.x.
Elasticsearch Migration Plugin
The first thing to do is to use theElasticsearch Migration Plugin to help to highlight some potential pitfalls before an existing Elasticsearch 1.x cluster is upgraded to Elasticsearch 2.x. Do that before attempting an upgrade!
# find / -name plugin -type f /usr/share/elasticsearch/bin/plugin
# cd /usr/share/elasticsearch/
# ./bin/plugin -i migration -u https://github.com/elastic/elasticsearch-migration/releases/download/v1.18/elasticsearch-migration-1.18.zip
Run the plugin: http://localhost:9200/_plugin/migration
In our case no issues were found, we can move forward.
Remove the plugin when done, as it won’t be compatible with Elasticsearch 2.x:
# ./bin/plugin remove migration
Elasticsearch Configuration Changes
Open Elasticsearch configuration file /etc/elasticsearch/elasticsearch.yml for editing and set the network.host value to a non-localhost address. Disable script.disable_dynamic as it is a deprecated setting. Make sure that you do those before starting Elasticsearch 2.x.
These are our settings for references, taken from the file /etc/elasticsearch/elasticsearch.yml:
cluster.name: graylog2 network.host: 10.8.143.13 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["10.8.143.13:9300"]
Note the existing cluster name, graylog2, we don’t change it.
The content of the file /etc/sysconfig/elasticsearch:
ES_HEAP_SIZE=4g ES_STARTUP_SLEEP_TIME=5 MAX_OPEN_FILES=65536
Elasticsearch related security limits /etc/security/limits.conf:
elasticsearch soft nofile 65536 elasticsearch hard nofile 65536 elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited
Index Range Types
This step needs to be performed before the update to Elasticsearch 2.x!
# curl -XGET localhost:9200/_all/_mapping/index_range; echo
{}
If the output is {} then we are not affected and can skip this step. Otherwise, we would need to delete the index_range type, Graylog does not use it anymore.
Graylog Index Template
Delete the index template:
# curl -X DELETE http://localhost:9200/_template/graylog-internal
{"acknowledged":true}
Graylog will automatically create the new index template on the next startup.
Index Retention and Rotation Settings
In 2.0.x the index rotation and retention settings have been moved from the Graylog server config file to the database and are now configurable via the web interface.
When we upgrade from a 1.x version and have modified any rotation/retention settings, we need to ensure that we keep our old settings in the config file so the migration process will add those old settings to the database. Otherwise the retention process will use the default settings and might remove a lot of indices.
Yum Repository
Edit the file /etc/yum.repos.d/elasticsearch.repo to include the new Elasticsearch 2.x repository:
[elasticsearch-1.7] name=Elasticsearch repository for 1.7.x packages baseurl=http://packages.elastic.co/elasticsearch/1.7/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=0 [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=https://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1
Install Elasticsearch:
# yum clean all # yum install elasticsearch # service elasticsearch restart
Check for any obvious issues:
# tail -f /var/log/elasticsearch/graylog2.log
# curl -X GET 'http://10.8.143.13:9200/'
{
"name" : "Ororo Munroe",
"cluster_name" : "graylog2",
"cluster_uuid" : "59OCRIl5SOu8I61teQLpjQ",
"version" : {
"number" : "2.4.2",
"build_hash" : "161c65a337d4b422ac0c805f284565cf2014bb84",
"build_timestamp" : "2016-11-17T11:51:03Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
# curl -X GET 'http://10.8.143.13:9200/_cluster/health?pretty'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 20,
"active_shards" : 20,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
In case your Elasticsearch cluster status is red, be advised that it takes several minutes for it to become green.
Graylog 2.0.x
Remove the old Graylog 1.3 repository and install the new one for 2.0.
# yum remove graylog-1.3-repository-el6 # rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.0-repository_latest.rpm
Also remove the Graylog web package as it’s not compatible with Graylog 2.0.x.
# yum remove graylog-web
Install Graylog (this will upgrade to the version 2.0.x):
# yum clean all # yum install graylog-server
Once finished, open the file /etc/graylog/server/server.conf for editing and put/modify the following settings:
is_master = true node_id_file = /etc/graylog/server/node-id password_secret = <passwd> root_password_sha2 = <passwd> rest_listen_uri = http://10.8.143.13:12900/ rest_transport_uri = http://10.8.143.13:12900/ web_listen_uri = http://10.8.143.13:9000/ elasticsearch_index_prefix = graylog2 elasticsearch_cluster_name = graylog2 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = 10.8.143.13:9300 elasticsearch_cluster_discovery_timeout = 10000 mongodb_useauth = false mongodb_uri = mongodb://localhost/graylog2
Note that we don’t change the cluster name nor the DB from graylog2 to graylog. Restart the service:
# service graylog-server restart
Check for issues:
# tail -f /var/log/graylog-server/server.log
# curl -X GET 'http://10.8.143.13:12900/system/cluster/node'
{"cluster_id":"6ddf78b8-3fae-43aa-a3d2-8cdf7ddca900","node_id":"604e0a3a-891b-47ef-a08e-66c68e604b82","type":"server","transport_address":"http://10.8.143.13:12900/","last_seen":"2016-12-12T18:20:27.000Z","short_node_id":"604e0a3a","hostname":"vm-graylog","is_master":true}
All done, we should be able to log into Graylog WebUI on http://10.8.143.13:9000/.
Upgrade Graylog 2.0.x to 2.1.x
Once everything is working as expected and we are happy with the upgrade to the version 2.0.x, we can upgrade to 2.1.x. We need to add a new repository:
# rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm
Upgrade Graylog. This will keep our existing configuration.
# yum clean all # yum install graylog-server
Restart the service:
# service graylog-server restart
Optionally, to run the REST API and the web interface on the same port, we will need the following:
rest_listen_uri = http://10.8.143.13:9000/api/ web_enable = true web_listen_uri = http://10.8.143.13:9000/ web_endpoint_uri = http://10.8.143.13:9000/api/ rest_transport_uri = http://10.8.143.13:9000/api/
More info about configuring Graylog 2.1.x web interface can be found on this page: http://docs.graylog.org/en/2.1/pages/configuration/web_interface.html.