Eureka结合网关实现多集群分片
问题描述
目前EurekaClient客户端只能往一个EurekaServer集群进行注册发现。而一个EurekaServer集群受制于硬件配置的高低，所能承受的最大服务注册实例数一般在4000至8000个不等。当一群服务的实例数超过了一个EurekaServer集群所能承受的最大实例数时，EurekaServer集群就不能正常运行，会导致大量请求超时，影响所有服务的注册和发现。
解决思路
为了解决这个问题，可以引入一个注册网关，将这一群服务按照某种规则分别注册到不同的EurekaServer集群中，再由注册网关通过调用Eureka API获取所有集群中的注册列表，将这些注册列表全部返回给EurekaClient客户端，这样每一个客户端都能拿到所有EurekaServer集群里的服务实例的注册列表。
具体实现
逻辑图
实现步骤说明
1、 通过修改Eureka Client源码，在请求的注册中心URL里添加项目名；
2、 注册网关通过识别请求中的项目名，将注册的请求转发到对应的集群中；
3、 注册网关通过定时任务每3秒调用Eureka API，从所有的集群中获取全量的注册列表和增量的注册列表分别存入本地缓存；
4、 当客户端发送获取全量注册列表或者是获取增量注册列表请求时，注册网关将本地缓存中存放的所有集群的未解压的全量注册列表或增量注册列表信息封装处理后返回给客户端；
5、 客户端接收到从注册网关返回的全量或增量注册列表后，经过解压合并处理转换成Applications对象存入本地注册列表缓存。
Eureka Client源码调整
将从GitHub上下载的Eureka源码spring-cloud-netflix-2.2.0.RELEASE打开，修改里面的子模块spring-cloud-netflix-eureka-client。
找到类org.springframework.cloud.netflix.eureka.EurekaClientConfigBean.java，修改getEurekaServerServiceUrls方法，将配置文件中的project.name参数加入到eurekaServerUrl中，修改的示例如下：
@Override
public List<String> getEurekaServerServiceUrls(String myZone) {
String serviceUrls = this.serviceUrl.get(myZone);
if (serviceUrls == null || serviceUrls.isEmpty()) {
serviceUrls = this.serviceUrl.get(DEFAULT_ZONE);
}
if (!StringUtils.isEmpty(serviceUrls)) {
final String[] serviceUrlsSplit = StringUtils.commaDelimitedListToStringArray(serviceUrls);
List<String> eurekaServiceUrls = new ArrayList<>(serviceUrlsSplit.length);
for (String eurekaServiceUrl : serviceUrlsSplit) {
if (!endsWithSlash(eurekaServiceUrl)) {
eurekaServiceUrl += "/";
}
// ### 根据配置判断是否是需要连接注册网关 TOP ###
String isEurekaServer = this.environment.getProperty("is.use.eureka.gateway", "true");
if ("true".equalsIgnoreCase(isEurekaServer.toLowerCase())) {
/*
* 微服务客户端连注册中心需要经过注册网关转发
* 连注册网关的URL=http://gatewayIp:port/{project.name}/eureka/
*/
String projectName = this.environment.getProperty("project.name", "default");
String str = "/" + projectName + "/eureka/";
eurekaServiceUrl = eurekaServiceUrl.trim().replace("/eureka/", str);
System.out.println("EurekaGatewayUrl=" + eurekaServiceUrl);
} else {
System.out.println("EurekaServiceUrl=" + eurekaServiceUrl);
}
// ### END ###
eurekaServiceUrls.add(eurekaServiceUrl);
}
return eurekaServiceUrls;
}
return new ArrayList<>();
}
注册网关添加路由配置
使用Spring Cloud Gateway做为注册网关，在注册网关中添加对客户端请求的路由设置，将项目APP-1发来的注册发现请求转发到eurekaServer1，将APP-2发来的注册发现请求转发到eurekaServer2，配置示例如下：
spring.cloud.gateway.routes.0.id = app-1
spring.cloud.gateway.routes.0.uri = lb://app-1
spring.cloud.gateway.routes.0.predicates.0 = Path=/**/APP-1/eureka/**
spring.cloud.gateway.routes.0.filters.0 = StripPrefix=3
spring.cloud.gateway.routes.0.filters.1 = EurekaResponse
spring.cloud.gateway.routes.1.id = app-2
spring.cloud.gateway.routes.1.uri = lb://app-2
spring.cloud.gateway.routes.1.predicates.0 = Path=/**/APP-2/eureka/**
spring.cloud.gateway.routes.1.filters.0 = StripPrefix=3
spring.cloud.gateway.routes.1.filters.1 = EurekaResponse
app-1.ribbon.listOfServers = http://eurekaServer1Ip:8001
app-2.ribbon.listOfServers = http://eurekaServer2Ip:8002
注册网关定时获取注册列表
注册网关通过定时任务，每3秒去拉取eurekaServer1和eurekaServer2两个集群的全量注册列表，存入本地缓存。
@Slf4j
@Component
@EnableScheduling
public class EurekaAppsTask implements SchedulingConfigurer {
@Autowired
private ConfigBean configBean;
@Override
public void configureTasks(ScheduledTaskRegistrar scheduledTaskRegistrar) {
scheduledTaskRegistrar.addTriggerTask(() -> eurekaAppsInfo(),
triggerContext -> {
String cron = configBean.getEurekainstanceCron();
if (StringUtils.isEmpty(cron)) {
cron = "*/3 * * * * ?";
}
return new CronTrigger(cron).nextExecutionTime(triggerContext);
});
}
// 全量拉取分片注册中心服务列表
private void eurekaAppsInfo() {
List<String> allSubDomainId = configBean.getAllSubDomainId();
allSubDomainId.stream().filter(subDomainId -> !CollectionUtils.isEmpty(allSubDomainId))
.forEach(subDomainId -> {
String oneDomainPath = configBean.getOneDomainPath(subDomainId);
if (StringUtils.isEmpty(oneDomainPath)) return;
String url = "http://" + oneDomainPath + "/eureka/apps";
long startTime = System.currentTimeMillis();
RestTemplate restTemplate = new RestTemplate();
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.set("Accept-Encoding", "gzip");
HttpEntity<MultiValueMap<String, String>> entity = new HttpEntity<>(httpHeaders);
byte[] value = new byte[0];
try {
ResponseEntity<byte[]> exchange = restTemplate.exchange(url, HttpMethod.GET, entity, byte[].class);
value = exchange.getBody();
ServerData.allInstanceServersMap.put(subDomainId, value);
} catch (Exception e) {
log.error(e.getMessage(),e);
}
log.warn("获取分片 {} ,耗时：{} ms", url, System.currentTimeMillis() - startTime);
});
}
}
注册网关通过定时任务，每3秒去拉取eurekaServer1和eurekaServer2两个集群的增量注册列表，存入本地缓存。
@Slf4j
@Component
@EnableScheduling
public class EurekaDeltaAppsTask implements SchedulingConfigurer {
@Autowired
private ConfigBean configBean;
@Override
public void configureTasks(ScheduledTaskRegistrar scheduledTaskRegistrar) {
scheduledTaskRegistrar.addTriggerTask(() -> eurekaDeltaAppsInfo(),
triggerContext -> {
String cron = configBean.getEurekainstanceCron();
if (StringUtils.isEmpty(cron)) {
cron = "*/3 * * * * ?";
}
return new CronTrigger(cron).nextExecutionTime(triggerContext);
});
}
// 增量拉取分片注册中心服务列表
private void eurekaDeltaAppsInfo() {
List<String> allSubDomainId = configBean.getAllSubDomainId();
allSubDomainId.stream().filter(subDomainId -> !CollectionUtils.isEmpty(allSubDomainId))
.forEach(subDomainId -> {
String oneDomainPath = configBean.getOneDomainPath(subDomainId);
if (StringUtils.isEmpty(oneDomainPath)) return;
String url = "http://" + oneDomainPath + "/eureka/apps/delta";
long startTime = System.currentTimeMillis();
RestTemplate restTemplate = new RestTemplate();
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.set("Accept-Encoding", "gzip");
HttpEntity<MultiValueMap<String, String>> entity = new HttpEntity<>(httpHeaders);
byte[] value = new byte[0];
try {
ResponseEntity<byte[]> exchange = restTemplate.exchange(url, HttpMethod.GET, entity, byte[].class);
value = exchange.getBody();
ServerData.deltaInstanceServersMap.put(subDomainId, value);
} catch (Exception e) {
log.error(e.getMessage(),e);
}
log.warn("获取分片 {} ,耗时：{} ms", url, System.currentTimeMillis() - startTime);
});
}
}
注册网关返回所有集群注册列表
注册网关通过注册分片过滤器，将所有集群的注册列表封装到Map对象中，最后转成不压缩的JSON报文返回给客户端。
@Slf4j
public class EurekaResponseGatewayFilter implements GatewayFilter, Ordered {
private static final String START_TIME = "startTime-EurekaResponse";
@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
String path = request.getPath().value();
HttpMethod method = request.getMethod();
HttpHeaders headers = request.getHeaders();
exchange.getAttributes().put(START_TIME,System.currentTimeMillis());
//获取服务列表
if (method.matches(HttpMethod.GET.name()) && path.indexOf("/eureka/") != -1) {
String domain = "";
String domainName = "";
ConcurrentHashMap<String, String> domainProjectRelation = ApplicationContextBeanUtils.getBean(ConfigBean.class).getDomainProjectRelation();
if (!domainProjectRelation.isEmpty()) {
for (Map.Entry<String, String> entry : domainProjectRelation.entrySet()) {
String projectName = entry.getKey();
String domainId = entry.getValue();
if (path.indexOf(projectName) != -1) {
domain = domainId;
domainName = path.split("\\/")[2];
break;
}
}
}
if (StringUtils.isEmpty(domain)) {
return chain.filter(exchange);
}
List<String> subDomainIds = ApplicationContextBeanUtils.getBean(ConfigBean.class).getSubDomainId(domainName);
if (subDomainIds == null || subDomainIds.isEmpty()) {
return chain.filter(exchange);
}
Iterator<String> iterator = subDomainIds.iterator();
while (iterator.hasNext()) {
String obj = iterator.next();
if (domain.equals(obj)) {
iterator.remove();
}
}
return wrapEurekaBody(exchange, path, chain, headers, domain, domainName, subDomainIds);
}
return chain.filter(exchange).then(Mono.fromRunnable(() -> {
Long startTime = exchange.getAttribute(START_TIME);
if (startTime != null) {
Long executeTime = System.currentTimeMillis() - startTime;
log.info("全量或增量获取服务列表 {} 耗时： {} ms",exchange.getRequest().getURI().getRawPath(),executeTime);
if (executeTime > 3*1000) {
log.info("全量或增量获取服务列表 {} 耗时： {} ms ,超过3s",exchange.getRequest().getURI().getRawPath(),executeTime);
}
}
}));
}
@Override
public int getOrder() {
return -2;
}
public Mono<Void> wrapEurekaBody(ServerWebExchange exchange, String path, GatewayFilterChain chain, HttpHeaders headers, String domain, String domainName, List<String> subDomainIds) {
ServerHttpResponse originalResponse = exchange.getResponse();
HttpStatus statusCode = originalResponse.getStatusCode();
if (statusCode == HttpStatus.OK) {
ServerHttpResponseDecorator decoratedResponse = new ServerHttpResponseDecorator(originalResponse) {
@Override
public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
Flux<? extends DataBuffer> fluxBody = Flux.from(body);
if (getStatusCode().equals(HttpStatus.OK) && body instanceof Flux) {
return super.writeWith(fluxBody.buffer().map(dataBuffers -> {
DataBufferFactory dataBufferFactory = new DefaultDataBufferFactory();
DataBuffer join = dataBufferFactory.join(dataBuffers);
byte[] content = new byte[join.readableByteCount()];
join.read(content);
DataBufferUtils.release(join);
HashMap<String, byte[]> eurekaApps = getEurekaApps(subDomainIds, path, headers);
eurekaApps.put(domain, content);
byte[] bytes;
JSONObject jsonObject1 = new JSONObject();
jsonObject1.put("applicationsMap", JSONObject.toJSONString(eurekaApps));
jsonObject1.put("apps__hashcode", "apps__hashcode");
jsonObject1.put("versions__delta", 1);
JSONObject jsonObject2 = new JSONObject();
jsonObject2.put("applications", jsonObject1);
bytes = JSONObject.toJSONBytes(jsonObject2);
//content = GZipUtils.gZip(bytes);
originalResponse.getHeaders().setContentLength(bytes.length);
originalResponse.getHeaders().set("shard", "true");
originalResponse.getHeaders().remove("Content-Encoding");
eurekaApps.clear();
jsonObject1.clear();
jsonObject2.clear();
return bufferFactory().wrap(bytes);
}));
}
return super.writeWith(body).then(Mono.fromRunnable(() -> {
Long startTime = exchange.getAttribute(START_TIME);
if (startTime != null) {
Long executeTime = System.currentTimeMillis() - startTime;
log.info("处理完decoratedResponse {} 耗时： {} ms",exchange.getRequest().getURI().getRawPath(),executeTime);
if (executeTime > 3*1000) {
log.info("处理完decoratedResponse {} 耗时： {} ms ,超过3s",exchange.getRequest().getURI().getRawPath(),executeTime);
}
}
}));
}
};
return chain.filter(exchange.mutate().response(decoratedResponse).build()).then(Mono.fromRunnable(() -> {
Long startTime = exchange.getAttribute(START_TIME);
if (startTime != null) {
Long executeTime = System.currentTimeMillis() - startTime;
log.info("所有分片返回 {} 耗时： {} ms",exchange.getRequest().getURI().getRawPath(),executeTime);
if (executeTime > 3*1000) {
log.info("所有分片返回 {} 耗时： {} ms ,超过3s",exchange.getRequest().getURI().getRawPath(),executeTime);
}
}
}));
}
return chain.filter(exchange).then(Mono.fromRunnable(() -> {
Long startTime = exchange.getAttribute(START_TIME);
if (startTime != null) {
Long executeTime = System.currentTimeMillis() - startTime;
log.info("获取增量或者全量服务列表失败 {} 耗时： {} ms",exchange.getRequest().getURI().getRawPath(),executeTime);
if (executeTime > 3*1000) {
log.info("获取增量或者全量服务列表失败 {} 耗时： {} ms ,超过3s",exchange.getRequest().getURI().getRawPath(),executeTime);
}
}
}));
}
public HashMap<String, byte[]> getEurekaApps(List<String> domains, String path, HttpHeaders headers) {
HashMap<String, byte[]> result = new HashMap<>();
domains.forEach(domain -> {
if (path.indexOf("/eureka/apps") != -1 && path.indexOf("/eureka/apps/delta") == -1) {
result.put(domain,ServerData.allInstanceServersMap.get(domain));
}
if (path.indexOf("/eureka/apps/delta") != -1){
result.put(domain,ServerData.deltaInstanceServersMap.get(domain));
}
});
return result;
}
}
Eureka Client客户端处理注册列表
将从GitHub上下载的Eureka源码eureka-1.9.13打开，修改里面的子模块eureka-client。
找到类com.netflix.discovery.shared.transport.jersey. AbstractJerseyEurekaHttpClient.java，修改getApplicationsInternal方法，将获取的注册网关返回的JSON报文进行解压合并处理，修改的示例如下：
private EurekaHttpResponse<Applications> getApplicationsInternal(String urlPath, String[] regions) {
ClientResponse response = null;
String regionsParamValue = null;
// ### 新增 TOP ###
InputStream inputStream = null;
InputStreamReader inputStreamReader = null;
BufferedReader bufferedReader = null;
ByteArrayInputStream byteArrayInputStream = null;
// ### 新增 END ###
try {
WebResource webResource = jerseyClient.resource(serviceUrl).path(urlPath);
if (regions != null && regions.length > 0) {
regionsParamValue = StringUtil.join(regions);
webResource = webResource.queryParam("regions", regionsParamValue);
}
Builder requestBuilder = webResource.getRequestBuilder();
addExtraHeaders(requestBuilder);
response = requestBuilder.accept(MediaType.APPLICATION_JSON_TYPE).get(ClientResponse.class);
// ### 处理请求体中的压缩报文 TOP ###
// 只有是经过注册网关拼接的报文才需要特殊处理（多注册中心分片的注册列表MAP拼接）
if ("TRUE".equalsIgnoreCase(response.getHeaders().getFirst("shard"))) {
if (logger.isInfoEnabled()) {
logger.info("###注册中心分片处理......");
}
try {
inputStream = response.getEntityInputStream();
StringBuilder stringBuilder = new StringBuilder();
inputStreamReader = new InputStreamReader(inputStream);
bufferedReader = new BufferedReader(inputStreamReader);
String line;
while ((line = bufferedReader.readLine()) != null) {
stringBuilder.append(line);
}
String entityJsonStr = stringBuilder.toString();// 已解压的JSON格式报文
ObjectMapper mapper = new ObjectMapper();
JsonNode node = mapper.readTree(entityJsonStr);
if (node.get("applications") != null) {
int versionsDelta = 1;
Map<String, String> applicationsMap = mapper.readValue(node.get("applications").get("applicationsMap").asText(), Map.class);
if (applicationsMap != null && applicationsMap.size() > 0) {
Set<String> keys = applicationsMap.keySet();
// 用于记录每个分片的注册列表里的hashcode的和值
int up = 0;
int down = 0;
int count = 0;
// 将MAP中的JSON里的application数组全部合并到一个JSON中
JSONObject jsonObject = new JSONObject();
String appsHashcode = "";
for (String key : keys) {
if (logger.isInfoEnabled()) {
logger.info("###注册中心分片的key={}", key);
}
String applicationsGzip = applicationsMap.get(key);
String applicationsStr = GZIPUtils.uncompress(applicationsGzip);
JSONObject applicationsObject = JSONObject.parseObject(applicationsStr);
if (count == 0) {
jsonObject = applicationsObject;
}
String applicationsJson = applicationsObject.getString("applications");
JSONObject applicationObject = JSONObject.parseObject(applicationsJson);
versionsDelta = Integer.parseInt(applicationObject.getString("versions__delta"));
appsHashcode = applicationObject.getString("apps__hashcode");
if (logger.isInfoEnabled()) {
logger.info("###注册中心分片{}的versions__delta={}", key, versionsDelta);
logger.info("###注册中心分片{}的apps__hashcode={}", key, appsHashcode);
}
int tmp = 0;
if (appsHashcode.contains("UP_")) {
if (appsHashcode.contains("_DOWN")) {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("UP_") + 3, appsHashcode.lastIndexOf("_DOWN")));
up = up + tmp;
} else {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("UP_") + 3, appsHashcode.lastIndexOf("_")));
up = up + tmp;
}
} else if (appsHashcode.contains("_UP")) {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("_UP") + 3, appsHashcode.lastIndexOf("_")));
up = up + tmp;
}
if (appsHashcode.contains("DOWN_")) {
if (appsHashcode.contains("_UP")) {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("DOWN_") + 5, appsHashcode.lastIndexOf("_UP")));
down = down + tmp;
} else {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("DOWN_") + 5, appsHashcode.lastIndexOf("_")));
down = down + tmp;
}
} else if (appsHashcode.contains("_DOWN")) {
tmp = Integer.parseInt(appsHashcode.substring(appsHashcode.indexOf("_DOWN") + 5, appsHashcode.lastIndexOf("_")));
down = down + tmp;
}
if (logger.isInfoEnabled()) {
logger.info("###注册中心分片{}的applicationArray.size()={}", key, applicationObject.getJSONArray("application").size());
}
if (count > 0) {
JSONArray applicationArray = applicationObject.getJSONArray("application");
if (applicationArray.size() > 0) {
jsonObject.getJSONObject("applications").getJSONArray("application").addAll(applicationArray);
}
}
count++;
}
if (up != 0 && down == 0) {
appsHashcode = "UP_" + up + "_";
} else if (up == 0 && down != 0) {
appsHashcode = "DOWN_" + down + "_";
} else {
appsHashcode = "DOWN_" + down + "_UP_" + up + "_";
}
if (logger.isInfoEnabled()) {
logger.info("###注册中心分片处理后使用的versions__delta={}", jsonObject.getJSONObject("applications").getString("versions__delta"));
logger.info("###注册中心分片处理后拼接的apps__hashcode={}", appsHashcode);
}
jsonObject.getJSONObject("applications").put("apps__hashcode", appsHashcode);
String applicationsNew = jsonObject.toString();
byteArrayInputStream = new ByteArrayInputStream(applicationsNew.getBytes());
response.setEntityInputStream(byteArrayInputStream);
} else {
logger.error("处理分片的注册列表获取的报文中没有applicationsMap:{}", entityJsonStr);
}
} else {
logger.error("处理分片的注册列表获取的报文中没有applications:{}", entityJsonStr);
}
} catch (IOException e) {
logger.error("处理分片的注册列表操作异常:" + e);
}
} else {
if (logger.isInfoEnabled()) {
logger.info("###注册中心源生处理......");
}
}
// ### END ###
Applications applications = null;
if (response.getStatus() == Status.OK.getStatusCode() && response.hasEntity()) {
applications = response.getEntity(Applications.class);
}
return anEurekaHttpResponse(response.getStatus(), Applications.class)
.headers(headersOf(response))
.entity(applications)
.build();
} finally {
if (logger.isDebugEnabled()) {
logger.debug("Jersey HTTP GET {}/{}?{}; statusCode={}",
serviceUrl, urlPath,
regionsParamValue == null ? "" : "regions=" + regionsParamValue,
response == null ? "N/A" : response.getStatus()
);
}
if (response != null) {
response.close();
}
// ### 新增 TOP ###
if (bufferedReader != null) {
try {
bufferedReader.close();
} catch (IOException e) {
logger.error("关闭bufferedReader异常:{}", e.getMessage());
}
}
if (inputStreamReader != null) {
try {
inputStreamReader.close();
} catch (IOException e) {
logger.error("关闭inputStreamReader异常:{}", e.getMessage());
}
}
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
logger.error("关闭inputStream异常:{}", e.getMessage());
}
}
if (byteArrayInputStream != null) {
try {
byteArrayInputStream.close();
} catch (IOException e) {
logger.error("关闭byteArrayInputStream异常:{}", e.getMessage());
}
}
// ### END ###
}
}
修改后重新编译打包成eureka-client-1.9.13.jar替换本地Maven仓库里同版本的jar包，例如：D:\maven\repository\com\netflix\eureka\eureka-client\1.9.13\eureka-client-1.9.13.jar。再将上面修改后的spring-cloud-netflix-eureka-client（需要引入新打包的eureka-client-1.9.13.jar包）重新编译打包成spring-cloud-netflix-eureka-client-2.2.1.RELEASE.jar，替换本地Maven仓库里同版本的jar包，例如：D:\maven\repository\org\springframework\cloud\spring-cloud-netflix-eureka-client\2.2.1.RELEASE\spring-cloud-netflix-eureka-client-2.2.1.RELEASE.jar
最后在服务工程的pom.xml文件里引入Eureka Client依赖
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
在配置文件application.properties中加入相应的项目名
project.name=APP-1
再配置好注册网关的URL
eureka.client.serviceUrl.defaultZone=http://gatewayIp:port/eureka
经过以上操作后就能实现Eureka多集群分片功能。
—END—
来源：oschina
链接：https://my.oschina.net/u/4377926/blog/4357910