Cassandra update fails silently with several nodes

问题

I have following situation.

There is a CQL table (Cassandra 2.0.12)

CREATE TABLE article (
  version timeuuid,
  id timeuuid,
  active boolean,
  contentbody text,
  contentformat text,
  createdat text,
  entitytype text,
  externalsources list<text>,
  geolat double,
  geolong double,
  lastcomments list<text>,
  lastmodifiedat text,
  lstmodbyuserid text,
  lstmodbyusername text,
  previewimage text,
  publishedatarticle text static,
  publishedatver text,
  status text,
  subcategory text,
  subtitle text,
  title text,
  userid text static,
  username text static,
  PRIMARY KEY ((version), id)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

and work with it with datastax-java-driver (cassandra-driver-core 2.1.1)

When cluster contains 3 nodes data update, like

UPDATE article SET title='updated title2',subtitle=null,status='draft',subCategory='{"id":"a6b68330-2ef5-4267-98c5-cd793edbb1a8","name":"sub cat name","color":"blue","parentCategory":{"id":"prim_cat_id","name":"prim cat name","color":"blue"}}',contentBody='someOtherBody',contentFormat='someOtherFormat',geoLat=138782.34,geoLong=138782.34,lastModifiedAt='2015-03-02 11:14:57',publishedAtArticle=null,publishedAtVer=null,lstModByUserId='e264fb2c-2485-488a-965f-765d139be9ea',lstModByUsername='reg1 user',externalSources=[],previewImage='{"width":1,"height":2,"duration":32,"original":"orig string","thumbs":{"prefix":"prefix str","ext":"jpg","sizes":["size1","size2"]}}' WHERE version=2480d891-c0cd-11e4-a691-df79ef55172c AND id=2480d890-c0cd-11e4-a691-df79ef55172c;

doesn't work in about half of cases silently (no errors I see in Cassandra logs, nothing suspicious in traces, no failure answer or exception, I can see that it didn't succeed just by SELECT). In case of cluster from one node, it always works.

Could you help me with some direction in case investigation?

回答1:

Since you mentioned that your nodes times aren't in sync, you may have a rare, but still possible condition.

If time is not in sync between the nodes, it may lead to some unpredictable results during the updates/inserts.

Usually when the write comes, the one with the latest timestamp wins. If one of your nodes has a time way behind, then when it becomes a coordinator, it stamps all of your records with its timestamps and therefore decides that this update is too old since there are already other updates with newer timestamp (from the nodes with accurate time). And therefore discards the update.

来源：https://stackoverflow.com/questions/28812339/cassandra-update-fails-silently-with-several-nodes

标签

cassandra

datastax-java-driver