high-availability

namenode ha failover time

只愿长相守 提交于 2019-12-10 09:45:50
问题 Namenode HA (NFS, QJM) is available in hadoop 2.x (HDFS-1623). It provides fast failover for Namenode, but I can't find any description on how long does it take to recover from a failure. Can any one tell me? Thanks for your answer.As the matter of fact,I want to know the time between the transformation of two nodes(active namenode and standby namenode).can you tell me how long? 回答1: Here are some qualified examples of times for failover with a standby NameNode: A 60 node cluster with 6

How to hand-over a TCP listening socket with minimal downtime?

蹲街弑〆低调 提交于 2019-12-09 16:08:17
问题 While this question is tagged EventMachine, generic BSD-socket solutions in any language are much appreciated too. Some background: I have an application listening on a TCP socket. It is started and shut down with a regular System V style init script. My problem is that it needs some time to start up before it is ready to service the TCP socket. It's not too long, perhaps only 5 seconds, but that's 5 seconds too long when a restart needs to be performed during a workday. It's also crucial

Design Patterns (or techniques) for Scalability

可紊 提交于 2019-12-09 04:03:43
问题 What design patterns or techniques have you used that are specifically geared toward scalability ? Patterns such as the Flyweight pattern seem to me to be a specialized version of the Factory Pattern, to promote high scalability or when working within memory or storage constraints. What others have you used? (Denormalization of Databases, etc.) Do you find that the rules change when high availability or scalability is your primary goal? Possible situations are: Mobile devices with more

Netty High Availability Cluster

别来无恙 提交于 2019-12-06 10:20:59
Wondering if Netty has any examples of how I can create a high availability application whereby the netty client will use a backup server in case of live server failure. If you want to make the client and server highly available and to manage the connections state by your code with ease, Have a look on Akka Remote Actor API which is using Netty for underlying communication . There is no example of this. But I think its quite straight-forward. You need to have a pool of different "channels" that are connected to remote hosts. The do something like this: channel.write(msg).addListener() { public

Scaling database in a microservice architecture

£可爱£侵袭症+ 提交于 2019-12-06 08:25:05
Lets say we have 5 microservices (μS) and each microservice requires one database with a few tables. Each of these services needs to be able to scale including the database queries. To do that we have a few options that I know of: 1. 1-μS; 1 instance; 1 DB instance Don't scale horizontally, just vertically. 2. 1-μS; N instances; 1 DB instance Every instance of a service gets to read and write to the same database. Simple way to scale if the service processing is the bottleneck. 3. 1-μS; N instances; HA DB cluster Every instance of a service gets to read and write to the same HA database. Good

Managing the migration of breaking database changes to a database shared by old version of the same application

亡梦爱人 提交于 2019-12-06 07:36:23
问题 One of my goals is to be able to deploy a new version of a web application that runs side by side the old version. The catch is that everything shares a database. A database that in the new version tends to include significant refactoring to database tables. I would like to be rollout the new version of the application to users over time and to be able to switch them back to the old version if I need to. Oren had a good post setting up the issue, but it ended with: "We are still in somewhat

rabbitmq HA cluster

好久不见. 提交于 2019-12-06 00:22:14
I am wanting to setup RabbitMQ as a two (or more) node cluster with HA. Use case: a client producer app (C#.NET) knows that the cluster has two nodes and publishes to the cluster. Various consumer apps (also C#.NET) connect to the cluster and get all messages generated by the producer. So long as at least one node is up and running the producer and consumers will all continue to work without error. Supposing nodes A and B are running and B dies for a while, then gets restarted, then a while later A dies, the clients all continue to function without receiving an error since at all times at

namenode ha failover time

大兔子大兔子 提交于 2019-12-05 22:42:03
Namenode HA (NFS, QJM) is available in hadoop 2.x (HDFS-1623). It provides fast failover for Namenode, but I can't find any description on how long does it take to recover from a failure. Can any one tell me? Thanks for your answer.As the matter of fact,I want to know the time between the transformation of two nodes(active namenode and standby namenode).can you tell me how long? Here are some qualified examples of times for failover with a standby NameNode: A 60 node cluster with 6 million blocks using 300TB raw storage, and 100K files: 30 seconds. Hence total failover time ranges from 1-3

Spark Config Files

血红的双手。 提交于 2019-12-05 21:26:18
Can someone share with me the spark-env.sh and spark-default.conf file content which needs to be set to run Spark on YARN as client or cluster ? Also should i store spark_assembly jar to the HDFS and then create a environment variable in the ~/.bashrc file ? I am unable to start my spark-shell with --master yarn-client command. Please help !! Update **Daemon in NN1 :** 2945 JournalNode 3137 DFSZKFailoverController 6385 Jps 3338 NodeManager 22730 QuorumPeerMain 2747 DataNode 3228 ResourceManager 2636 NameNode **Daemon in NN2** 19620 Jps 3894 QuorumPeerMain 16966 ResourceManager 16808

AMQP/RabbitMQ - Process messages sequentially

怎甘沉沦 提交于 2019-12-05 19:24:41
I have one direct exchange. There is also one queue, bound to this exchange. I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing. The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time. Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS