1. 技术文章
  2. datastax Opscenter can't add nodes, “Error provisioning cluster: Request ID is invalid” ,

datastax Opscenter can't add nodes, “Error provisioning cluster: Request ID is invalid” ,

由 廉价感情. 提交于 2020-01-11 07:19:40

问题


Update 2

There was a bug in Opscenter not matching dsc22 configuration with cassandra community version, this solved one problem.

Update

After reading the opscenter log again I think there actually something wrong with the 4 authentication fields or some ssh configuration, but I still don't know what exactly should be done, The field says "Local node credentials (sudo) private key (optional)

the scenario is as following:

  1. I installed 4 nodes with vagrant and ansible where each has dsc22,opscenter (redundant I know),datastax-agent,cassandra-tool, oracle java 8
  2. configuration below
  3. nodetool status, everything is good they all see each other
  4. I create a keyspace, it replicates to all the nodes just fine
  5. on my host machine I open the datastax using the forwarded port from node02 for example.
  6. First time I see two choices add existing cluster or manage existing one When I try to manage existing cluster > add 192.168.50.3 I get the following:
  7. So I try using 127.0.0.1, it works just fine but I only see this machine local cassandra node only.
  8. So I try to add nodes from inside when I get a dialog (I think this is important) it has credentials fields i add admin,admin for repository, and also admin,admin for local username, I actually don't know what to put in these 4 fields, whether these are created or they're actually preconfigured somewhere else
  9. So after I add some node to the data center with RAC info etc, I get the "Error provisioning cluster: Request ID is invalid"

I have no clue where the problem is, the only unknown step I did was the credentials thing (repository username/pass, local username/pass) when I add nodes from inside. But why I can't do the manage existing from the beginning where I only get in when I use 127.0.0.1 as ip.

So here's the datastax-agent:

Starting DataStax agent monitor datastax_agent_monitor.
 INFO [main] 2015-08-24 22:39:59,506 Loading conf files: /var/lib/datastax-agent/conf/address.yaml
  INFO [main] 2015-08-24 22:39:59,657 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_60
  INFO [main] 2015-08-24 22:39:59,657 DataStax Agent version: 5.2.0
  INFO [main] 2015-08-24 22:39:59,732 Default config values: {:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings", :restore_req_update_period 60, :my_$
  INFO [main] 2015-08-24 22:39:59,740 Waiting for the config from OpsCenter
  INFO [main] 2015-08-24 22:39:59,752 Starting Stomp
  INFO [main] 2015-08-24 22:39:59,752 Starting up agent communcation with OpsCenter.
  INFO [main] 2015-08-24 22:39:59,753 Reconnecting to a backup OpsCenter instance
  INFO [main] 2015-08-24 22:39:59,756 SSL communication is disabled
  INFO [main] 2015-08-24 22:39:59,757 Creating stomp connection to 192.168.50.3:61620
  INFO [async-dispatch-1] 2015-08-24 22:39:59,756 Using 127.0.0.1 as the cassandra broadcast address
  INFO [async-dispatch-1] 2015-08-24 22:39:59,762 New JMX connection (127.0.0.1:7199)
  INFO [StompConnection receiver] 2015-08-24 22:39:59,787 Reconnecting in 0s.
  INFO [main] 2015-08-24 22:39:59,791 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
  INFO [StompConnection receiver] 2015-08-24 22:39:59,872 Connected to 192.168.50.3:61620
  INFO [StompConnection receiver] 2015-08-24 22:40:00,200 Got new config from OpsCenter [note values in address.yaml override those from OpsCenter]: {:cassandra_port 904$
  INFO [StompConnection receiver] 2015-08-24 22:40:00,224 Starting up agent collection.
  INFO [StompConnection receiver] 2015-08-24 22:40:00,225 New JMX connection (127.0.0.1:7199)
  INFO [Jetty] 2015-08-24 22:40:00,347 Jetty server started
  INFO [StompConnection receiver] 2015-08-24 22:40:00,452 agent RPC address is  127.0.0.1
  INFO [async-dispatch-1] 2015-08-24 22:40:00,454 cassandra RPC address is  nil
  INFO [StompConnection receiver] 2015-08-24 22:40:00,471 Starting OS metric collectors (Linux)
  INFO [StompConnection receiver] 2015-08-24 22:40:00,516 Starting Cassandra JMX metric collectors
  INFO [install-location-finder] 2015-08-24 22:40:00,614 New JMX connection (127.0.0.1:7199)
  INFO [StompConnection receiver] 2015-08-24 22:40:00,639 New JMX connection (127.0.0.1:7199)
  INFO [StompConnection receiver] 2015-08-24 22:40:00,793 New JMX connection (127.0.0.1:7199)
  INFO [clojure-agent-send-off-pool-0] 2015-08-24 22:40:02,094 Attempting to load stored metric values.

Here's full opscenter log http://pastebin.com/fXT2vkFR The following is section from it:

 2015-08-24 23:13:38+0000 [Test_Cluster]  WARN: Ignoring scheduled job with type=best-practice, which is only supported with DataStax Enterprise.
2015-08-24 23:13:38+0000 [Test_Cluster]  INFO: Done loading persisted scheduled job descriptions
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Using 192.168.50.4 as the RPC address for node 127.0.0.1
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {'search': None, 'jobtracker': None, 'tasktracker': None, 'spark': {u'master': None, u'version': None, u'worker': None}, 'dse': None, 'cassandra': u'2.2.0'}
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {u'search': None, u'jobtracker': None, u'tasktracker': None, u'spark': {u'master': None, u'version': None, u'worker': None}, u'dse': None, u'cassandra': u'2.2.0'}
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Node 127.0.0.1 changed its mode to normal
2015-08-24 23:13:40+0000 [Test_Cluster]  INFO: Done loading persisted alert rules
2015-08-24 23:13:41+0000 [Test_Cluster]  INFO: OpsCenter starting up.
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Using 192.168.50.2 as the RPC address for node 127.0.0.1
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {'search': None, 'jobtracker': None, 'tasktracker': None, 'spark': {u'master': None, u'version': None, u'worker': None}, 'dse': None, 'cassandra': u'2.2.0'}
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {u'search': None, u'jobtracker': None, u'tasktracker': None, u'spark': {u'master': None, u'version': None, u'worker': None}, u'dse': None, u'cassandra': u'2.2.0'}
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:42+0000 [Test_Cluster]  INFO: Node 127.0.0.1 changed its mode to normal
2015-08-24 23:13:42+0000 []  INFO: Starting to update agents' configuration
2015-08-24 23:13:47+0000 [Test_Cluster]  INFO: Using 192.168.50.5 as the RPC address for node 127.0.0.1
2015-08-24 23:13:48+0000 [Test_Cluster]  INFO: Using 192.168.50.4 as the RPC address for node 127.0.0.1
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Using 192.168.50.3 as the RPC address for node 127.0.0.1
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {'search': None, 'jobtracker': None, 'tasktracker': None, 'spark': {u'master': None, u'version': None, u'worker': None}, 'dse': None, 'cassandra': u'2.2.0'}
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Node <Node 127.0.0.1='-6574032654670847999'> changed version to {u'search': None, u'jobtracker': None, u'tasktracker': None, u'spark': {u'master': None, u'version': None, u'worker': None}, u'dse': None, u'cassandra': u'2.2.0'}
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Processing spark version {u'master': None, u'version': None, u'worker': None}
2015-08-24 23:13:49+0000 [Test_Cluster]  INFO: Node 127.0.0.1 changed its mode to normal
2015-08-24 23:13:58+0000 [Test_Cluster]  INFO: Using 192.168.50.3 as the RPC address for node 127.0.0.1
2015-08-24 23:13:58+0000 [Test_Cluster]  INFO: Using 192.168.50.2 as the RPC address for node 127.0.0.1
2015-08-24 23:14:22+0000 []  INFO: Testing SSH connectivity to 192.168.50.4
2015-08-24 23:14:23+0000 []  INFO: Testing SSH login to 192.168.50.4
2015-08-24 23:14:29+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:14:29+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:14:41+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:14:41+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:14:52+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:14:52+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:15:03+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:15:03+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:15:14+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:15:14+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:15:26+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:15:26+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:15:38+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:15:38+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:15:50+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:15:50+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:16:01+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

2015-08-24 23:16:01+0000 []  INFO: Sleeping before retrying ssh login.
2015-08-24 23:16:13+0000 [] There was a problem verifying an ssh login on 192.168.50.4
    Traceback (most recent call last):
    Failure: opscenterd.SecureShell.SshFailed: ssh to u'192.168.50.4' failed

Configuration

I'm using vagrant to create my vms with

... a section from the vagrantfile
config.vm.define "node02" do |node|
    node.vm.host_name = "node02"
    node.vm.network :forwarded_port, guest: 8888, host: 3023
    node.vm.network "private_network", ip: "192.168.50.2", virtualbox__intnet: "intnet"
end
...

A section from the cassandra.yaml in each node

- seeds: "192.168.50.xx, 192.168.50.xx, ... rest of nodes"

The address.yaml in each node

# couple of nodes that have opscenter
# The following hosts line is commented out because when I use it the datastax-agent doesn't connect to any nodes, so I guess the default is 127.0.0.1 which works fine
# hosts: ["192.168.50.xx","192.168.50.xx"]

local_interface: 127.0.0.1

# opscenter ip
stomp_interface: 192.168.50.xx  
# this nodeXX ip
agent_rpc_broadcast_address: 192.168.50.xx

A section from cassandra-env.sh, note I don't use any authentication for jmx

LOCAL_JMX=NO

if [ "$LOCAL_JMX" = "yes" ]; then
  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"
else
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

nodetool status:

--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  192.168.50.2  982.74 KB  256          ?       a35..  RAC1
UN  192.168.50.3  679.05 KB  256          ?       e6c..  RAC1
UN  192.168.50.4  912.1 KB   256          ?       634..  RAC1
UN  192.168.50.5  939.55 KB  256          ?       0a...  RAC1

来源：https://stackoverflow.com/questions/32193091/datastax-opscenter-cant-add-nodes-error-provisioning-cluster-request-id-is-i

标签
cassandra
datastax
opscenter
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!