公司自主研发了这样一个缓存系统（名为 UPU），规格如下：
- 同机房间的两个 UPU 互为主备，但同一时刻只有一个 UPU 通过 keepalived 虚地址对外提供服务，主备切换也是通过 keepalived 实现；
- 同一（逻辑）域内的全部 UPU 形成集群，集群间通过 Erlang/OTP 内置的 Mnesia 数据库完成数据同步；
- UPU 集群不存在主的概念，故可以在集群中的任意节点上进行数据更新操作；
- 可以进行多级域的 UPU 级联；
- 业务的数据更新均在本地进行，但数据会向顶级域同步；
- 业务的数据查询均在本地进行，但会根据具体的请求类型决定是否还向顶级域查询；
UPU 服务器部署架构图
UPU 服务器内部结构（Erlang 监督树结构 ）
libupu（客户端库）的结构设计
这套系统存在的部分问题：
- UPU 本身不支持针对缓存消息的超时控制，而是依赖于业务程序自己的控制。在业务自身发生异常的情况下，可能出现数据的不一致或残留（已通过其他方法规避）；
- UPU 的数据同步功能依赖于 Erlang/OTP 内置的 Mnesia 数据库，官方明确说明在发生网络分区时，可能会导致数据不一致，原开发人员本来打算采用环形复制来解决集群数据多写问题，但最终没有实现；
- 在 UPU 级联情况下，级联相关配置是手动写死在配置文件中的，在出现异常时，只能通过修改配置并重启服务的方式进行恢复（应该可以通过 Erlang 的内部方法进行解决）；
- UPU 集群功能借用了 RabbitMQ 中的相应实现，并进行了一定程度的简化，可能存在部分功能缺失或不完备（可以通过对比 RabbitMQ 的源码比较分析并解决）；
- 心跳保活功能存在问题。具体表现为，级联 UPU 之间互发心跳请求和应答时，都以自身状态进行判定，即收到对方发来的请求也不认为对方是活跃的，只有对方回复应答时才认为保活成功，这种实现方式显然不合理；
- libupu（业务使用的客户端库）中判定心跳超时的时间值，与 UPU 服务器侧的判定时间值不同，这应该是开发人员疏忽导致，但该问题会导致业务在实际使用中遇到部分信息被莫名删除的问题；
- 日志输出不够清晰直观。测试以及运维人员无法直接通过日志确认当前的执行流程和结果。日志只有开发人员才能看懂（部分日志不看代码都不明白到底在干啥）；
- 由于 UPU 原开发人员离职，后续的维护人员似乎无法真正领悟其设计实现，相关的功能开发更像是在"打补丁"；
开发人员从这套系统中可以学到什么：
- 监督结构的设计，进程动态起停的控制；
- 有助于对 ETS 和 Mnesia 的使用有更深层次的理解；
- ranch 的使用；
- erlang 控制脚本的编写；
- libevent 的使用；
- 自定义协议的设计；
- 通过 gdb 排查问题；
-=-=-=-=-=- 我是胡歌脑残粉的分隔线 -=-=-=-=-=-=-
libupu 问题排查整理（最难查的一个）
今天终于查出了 libupu 经常会把 CPU 跑到 100% 的原因了。该问题在各种环境中反复出现过不下 10 次，但测试部门只汇报有问题出现，没有安排测试人员配合复现过该问题（不知道是否测试部门太忙？还是对组件的东西不够重视？具我的观察，很大一部分原因是，组件不方便单独进行测试，针对组件库来说，问题尤其严重），而等到与业务集成后的系统测试阶段，测试部门的注意力又不会集中在组件问题上，只有出了问题后，才会想起某某是责任人，派人过去叫你查，查的时候还不忘补一句"我今天还有一大堆业务功能要测，你赶快排查下，不行我先恢复环境测其它业务了"。好吧，开发组件的都不是亲娘生的，还是面对现实吧。
现实是这样的，组件开发和维护人员，要能在业务未集成前，自行设计针对各种场景的模拟，要能根据业务运行时出问题的情况，寻求复现的方法。原因已经说过，没人陪你玩，只能提高自己玩的水平。另外，组件提供给业务使用时，一定要把各种注意事项说清楚，最好白纸黑字写清楚，尤其是否允许跨线程使用啊，锁问题啊，回调数据定义和扩展性啊，等等。否则，任由业务自行发挥，后面可就惨了，甚至残了。
废话啰嗦了这么多，下面进入正题。
问题出现时的表现如下（逐层深入的说明）：
- 某线程 CPU 跑到接近 100% （使用 libupu 库的线程）
- 死循环发生在用户态调用中（libevent 库内部）
- 通过 gdb 发现，发生问题时，定时器小根堆顶的事件状态值存在问题（ev_flags）
由于最初不知道复现问题的方法，所以只能在发生问题后，再通过 gdb 抓取 core 文件进行分析，而此时只能对结果进行分析判断，无法知道程序从正常运行是如何转变成死循环的。这种类型的问题和分析程序崩溃的 core 文件还是不太一样的（个人结论）。
排查问题时遇到的现实问题：
- 多种业务使用同一动态库，每种业务打包相关文件时会将使用的库全打进去，但库的版本有不一致的情况；
- 业务引用动态库的路径各有不同，各业务之间存在交叉引用动态库的情况，导致更新库文件的时候可能对其他业务造成影响；
- 线上环境中的库均为 release 版本，且不是通过剔除 debug 版本符号生成的，故如何正确加载符号也是个小技术点；
在解决了上述问题后，就可以真正分析该问题的情况了。
（下面的跟踪是在能够复现问题的情况下进行的）
[root@upu_1 pas]#
[root@upu_1 pas]# strace -t -s 1024 -p 30732
Process 30732 attached - interrupt to quit
[ Process PID=30732 runs in 32 bit mode. ]
13:59:59 clock_gettime(CLOCK_MONOTONIC, {152346, 824968959}) = 0
13:59:59 gettimeofday({1445666399, 416615}, NULL) = 0
13:59:59 epoll_wait(26, {{EPOLLIN, {u32=29, u64=29}}}, 32, 9120) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152355, 726754428}) = 0
14:00:08 gettimeofday({1445666408, 318367}, NULL) = 0
14:00:08 read(29, "\0", 1024) = 1
14:00:08 read(29, 0x57bffdac, 1024) = -1 EAGAIN (Resource temporarily unavailable)
在 bufferevent-pair 的读端发现 137 字节数据，说明有业务调用了数据发送接口
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:08 write(2, "[libupu][14:00:08]: handle_bussiness_data() => evbuffer_get_length(pair[1]->input) = 137\n", 89) = 89
修正 fd 31 关注可读可写事件
14:00:08 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
将 137 字节数据从 bufferevent-pair 中搬移到网络发送用的 bufferevent 中
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) =
14:00:08 write(2, "[libupu][14:00:08]: handle_bussiness_data() => evbuffer_remove_buffer, move 137 Bytes from pair[1]->input to m_pBufferEvent->output\n", 132) = 132
等待可写事件，触发后，发送该 137 字节数据
14:00:08 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 218) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152355, 727937926}) = 0
14:00:08 gettimeofday({1445666408, 320427}, NULL) = 0
14:00:08 writev(31, [{"\0224Vx\0\0\0\4\0\0\0\23\0\0\0y", 16}, {"{\"callnum\": 0, \"devid\": \"\", \"userdomain\": \"0q0y1aphxubof8ycru6fsgsk\", \"outcallnum\": 0, \"phonecallnum\": 0, \"incallnum\": 0}", 121}], 2) = 137
修正 fd 31 关注可读事件
14:00:08 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
等待可读事件，触发后
14:00:08 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 217) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152355, 730703966}) = 0
14:00:08 gettimeofday({1445666408, 322313}, NULL) = 0
发现 socket 缓冲区中有 16 字节可读数据，读取 16 字节数据
14:00:08 ioctl(31, FIONREAD, [16]) = 0
14:00:08 readv(31, [{"\0224Vx\0\0\0\4\0\0\0\0\0\0\0\0", 16}], 1) = 16
修正 fd 31 不再关注事件（因为不是 EV_PERSIST 类型，所以内部会自动删除）
14:00:08 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
将收到的应答信息回调给业务
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:08 write(2, "[libupu][14:00:08]: handle_upu_response() => ### enter CallBack ### [RESULT] nRspType(0)\n", 90) = 90
14:00:08 gettimeofday({1445666408, 323146}, NULL) = 0
14:00:08 write(6, "\220\6PY", 4) = 4
14:00:08 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\330\350\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(3),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:08 gettimeofday({1445666408, 323701}, NULL) = 0
14:00:08 write(6, "x\4\340V", 4) = 4
14:00:08 send(18, "\3\356\0\0\0\7\0\0\0\0\0\221\0\0\0\0\5\0\217!\350\330\350\0\7\4\335\3[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n", 161, 0) = 161
14:00:08 futex(0x94f48cc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x94f48c8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
14:00:08 futex(0x94f48f8, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:08 futex(0x94f4924, FUTEX_WAIT_PRIVATE, 125, NULL) = 0
14:00:08 futex(0x94f4950, FUTEX_WAKE_PRIVATE, 1) = 0
修正 fd 31 关注可读事件（添加）
14:00:08 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
等待可读事件（此处针对的是 fd 29 和之前不同）
14:00:08 epoll_wait(26, {{EPOLLIN, {u32=29, u64=29}}}, 32, 10000) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152356, 8231631}) = 0
14:00:08 gettimeofday({1445666408, 599857}, NULL) = 0
14:00:08 read(29, "\0", 1024) = 1
14:00:08 read(29, 0x57bffdac, 1024) = -1 EAGAIN (Resource temporarily unavailable)
---- 新一轮数据处理 ----
在 bufferevent-pair 的读端发现 192 字节数据，说明有业务调用了发送接口
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:08 write(2, "[libupu][14:00:08]: handle_bussiness_data() => evbuffer_get_length(pair[1]->input) = 192\n", 89) = 89
14:00:08 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:08 write(2, "[libupu][14:00:08]: handle_bussiness_data() => evbuffer_remove_buffer, move 192 Bytes from pair[1]->input to m_pBufferEvent->output\n", 132) = 132
14:00:08 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 9723) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152356, 9572196}) = 0
14:00:08 gettimeofday({1445666408, 601149}, NULL) = 0
14:00:08 writev(31, [{"\0224Vx\0\0\0\3\0\0\0\f\0\0\0\260", 16}, {"{\"callnum\": 2, \"devid\": \"10103000000002000000000000000002\", \"platformid\": \"6f9bde0c-c069-4af3-946c-78b3a4191bfe\", \"nuaddr\": \"5pys5ZywIzE6MTcyLjE2LjE4NS4xMzUA\", \"nudefroute\": 1}", 176}], 2) = 192
14:00:08 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:08 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 9722) = 1
14:00:08 clock_gettime(CLOCK_MONOTONIC, {152356, 11685126}) = 0
14:00:08 gettimeofday({1445666408, 603291}, NULL) = 0
14:00:08 ioctl(31, FIONREAD, [16]) = 0
14:00:08 readv(31, [{"\0224Vx\0\0\0\3\0\0\0\0\0\0\0\0", 16}], 1) = 16
14:00:08 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:08 time(NULL) = 1445666408
14:00:08 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:08 write(2, "[libupu][14:00:08]: handle_upu_response() => ### enter CallBack ### [RESULT] nRspType(0)\n", 90) = 90
14:00:08 gettimeofday({1445666408, 604082}, NULL) = 0
14:00:08 write(6, "\220\6PY", 4) = 4
14:00:08 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\330\350\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(3),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:08 gettimeofday({1445666408, 604443}, NULL) = 0
14:00:08 write(6, "x\4\340V", 4) = 4
14:00:08 send(18, "\3\356\0\0\0\7\0\0\0\0\0\221\0\0\0\0\5\0\217!\350\330\350\0\7\4\335\3[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n", 161, 0) = 161
14:00:08 futex(0x94f48cc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x94f48c8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
14:00:08 futex(0x94f48f8, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:08 futex(0x94f4924, FUTEX_WAIT_PRIVATE, 127, NULL) = -1 EAGAIN (Resource temporarily unavailable)
14:00:08 futex(0x94f4950, FUTEX_WAKE_PRIVATE, 1) = 0
14:00:08 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
---- 新一轮数据处理 ----
这里出现了 10s 的空等 -- 因为在 10s 内没有可读事件触发
14:00:08 epoll_wait(26, {}, 32, 10000) = 0
14:00:18 clock_gettime(CLOCK_MONOTONIC, {152366, 23210960}) = 0
14:00:18 gettimeofday({1445666418, 614827}, NULL) = 0
这里进行一系列修改，对应代码哪里？
14:00:18 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:18 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:18 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:18 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:18 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
等待可写事件，触发后，发送心跳请求
14:00:18 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 10000) = 1
14:00:18 clock_gettime(CLOCK_MONOTONIC, {152366, 24061012}) = 0
14:00:18 gettimeofday({1445666418, 615704}, NULL) = 0
14:00:18 writev(31, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
等待可读事件，触发后，读取心跳应答
14:00:18 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:18 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 10000) = 1
14:00:18 clock_gettime(CLOCK_MONOTONIC, {152366, 25500295}) = 0
14:00:18 gettimeofday({1445666418, 617107}, NULL) = 0
14:00:18 ioctl(31, FIONREAD, [16]) = 0
14:00:18 readv(31, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
这里对应了回调函数中针对数据 body 为 0 情况的处理（什么动作都没有进行，仅在函数入口和出口处对事件进行了处理）
14:00:18 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:18 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
又等了 10s 没有数据可读
14:00:18 epoll_wait(26, {}, 32, 10000) = 0
14:00:28 clock_gettime(CLOCK_MONOTONIC, {152376, 36211523}) = 0
14:00:28 gettimeofday({1445666428, 627827}, NULL) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
等待可写事件，触发后，发送心跳请求
14:00:28 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 10000) = 1
14:00:28 clock_gettime(CLOCK_MONOTONIC, {152376, 37039083}) = 0
14:00:28 gettimeofday({1445666428, 628705}, NULL) = 0
14:00:28 writev(31, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
等待可读事件，触发后，读取心跳应答
14:00:28 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:28 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 10000) = 1
14:00:28 clock_gettime(CLOCK_MONOTONIC, {152376, 37828723}) = 0
14:00:28 gettimeofday({1445666428, 629429}, NULL) = 0
14:00:28 ioctl(31, FIONREAD, [16]) = 0
14:00:28 readv(31, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
14:00:28 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:28 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
等待可读事件（此处针对的是 fd 29）
14:00:28 epoll_wait(26, {{EPOLLIN, {u32=29, u64=29}}}, 32, 10000) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152385, 814250349}) = 0
14:00:38 gettimeofday({1445666438, 405878}, NULL) = 0
14:00:38 read(29, "\0", 1024) = 1
14:00:38 read(29, 0x57bffdac, 1024) = -1 EAGAIN (Resource temporarily unavailable)
---- 新一轮数据处理 ----
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_bussiness_data() => evbuffer_get_length(pair[1]->input) = 137\n", 89) = 89
14:00:38 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_bussiness_data() => evbuffer_remove_buffer, move 137 Bytes from pair[1]->input to m_pBufferEvent->output\n", 132) = 132
14:00:38 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 224) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152385, 816366473}) = 0
14:00:38 gettimeofday({1445666438, 407970}, NULL) = 0
14:00:38 writev(31, [{"\0224Vx\0\0\0\4\0\0\0\23\0\0\0y", 16}, {"{\"callnum\": 0, \"devid\": \"\", \"userdomain\": \"0q0y1aphxubof8ycru6fsgsk\", \"outcallnum\": 0, \"phonecallnum\": 0, \"incallnum\": 0}", 121}], 2) = 137
14:00:38 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 222) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152385, 817856690}) = 0
14:00:38 gettimeofday({1445666438, 409460}, NULL) = 0
14:00:38 ioctl(31, FIONREAD, [16]) = 0
14:00:38 readv(31, [{"\0224Vx\0\0\0\4\0\0\0\0\0\0\0\0", 16}], 1) = 16
14:00:38 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_upu_response() => ### enter CallBack ### [RESULT] nRspType(0)\n", 90) = 90
14:00:38 gettimeofday({1445666438, 410275}, NULL) = 0
14:00:38 write(6, "\340\4\220Y", 4) = 4
14:00:38 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\331\6\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(3),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:38 gettimeofday({1445666438, 410812}, NULL) = 0
14:00:38 write(6, "x\4\340V", 4) = 4
14:00:38 send(18, "\3\356\0\0\0\7\0\0\0\0\0\221\0\0\0\0\5\0\217!\350\331\6\0\7\4\335\3[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n", 161, 0) = 161
14:00:38 futex(0x94f48cc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x94f48c8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
14:00:38 futex(0x94f48f8, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:38 futex(0x94f4924, FUTEX_WAIT_PRIVATE, 129, NULL) = 0
14:00:38 futex(0x94f4950, FUTEX_WAKE_PRIVATE, 1) = 0
14:00:38 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 epoll_wait(26, {{EPOLLIN, {u32=29, u64=29}}}, 32, 10000) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152386, 116071089}) = 0
14:00:38 gettimeofday({1445666438, 707688}, NULL) = 0
14:00:38 read(29, "\0", 1024) = 1
14:00:38 read(29, 0x57bffdac, 1024) = -1 EAGAIN (Resource temporarily unavailable)
---- 新一轮数据处理 ----
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_bussiness_data() => evbuffer_get_length(pair[1]->input) = 192\n", 89) = 89
14:00:38 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_bussiness_data() => evbuffer_remove_buffer, move 192 Bytes from pair[1]->input to m_pBufferEvent->output\n", 132) = 132
14:00:38 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 9702) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152386, 119894680}) = 0
14:00:38 gettimeofday({1445666438, 711499}, NULL) = 0
14:00:38 writev(31, [{"\0224Vx\0\0\0\3\0\0\0\f\0\0\0\260", 16}, {"{\"callnum\": 2, \"devid\": \"10103000000002000000000000000002\", \"platformid\": \"6f9bde0c-c069-4af3-946c-78b3a4191bfe\", \"nuaddr\": \"5pys5ZywIzE6MTcyLjE2LjE4NS4xMzUA\", \"nudefroute\": 1}", 176}], 2) = 192
14:00:38 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 9698) = 1
14:00:38 clock_gettime(CLOCK_MONOTONIC, {152386, 122371397}) = 0
14:00:38 gettimeofday({1445666438, 713982}, NULL) = 0
14:00:38 ioctl(31, FIONREAD, [16]) = 0
14:00:38 readv(31, [{"\0224Vx\0\0\0\3\0\0\0\0\0\0\0\0", 16}], 1) = 16
14:00:38 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 time(NULL) = 1445666438
14:00:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:38 write(2, "[libupu][14:00:38]: handle_upu_response() => ### enter CallBack ### [RESULT] nRspType(0)\n", 90) = 90
14:00:38 gettimeofday({1445666438, 714792}, NULL) = 0
14:00:38 write(6, "\340\4\220Y", 4) = 4
14:00:38 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\331\6\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(3),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:38 gettimeofday({1445666438, 715315}, NULL) = 0
14:00:38 write(6, "x\4\340V", 4) = 4
14:00:38 send(18, "\3\356\0\0\0\7\0\0\0\0\0\221\0\0\0\0\5\0\217!\350\331\6\0\7\4\335\3[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n", 161, 0) = 161
14:00:38 futex(0x94f48cc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x94f48c8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
14:00:38 futex(0x94f48f8, FUTEX_WAKE_PRIVATE, 1) = 1
等待可读事件，触发后
14:00:38 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:38 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 10000) = 1
14:00:43 clock_gettime(CLOCK_MONOTONIC, {152390, 497764210}) = 0
14:00:43 gettimeofday({1445666443, 89440}, NULL) = 0
发现此时 socket 中可读数据为 0 （此时人为将服务器进程停止）
14:00:43 ioctl(31, FIONREAD, [0]) = 0
读出一些乱七八糟的数据（应该是由于 readv 使用的保存读取结果的 buffer 没有清空的缘故）
14:00:43 readv(31, [{"[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n\0\2\0@\0\320W\371\10\2\0@\0\320W@\0\320Wdows\",\"status\":\"callidle\",\"nuaddr\":\"172.16.185.135\",\"mtaddr\":\"172.16.72.105\",\"userdomain\":\"0q0y1aphxubof8ycru6fsgsk\",\"devid\":\"10103000000002000000000000000002\",\"data\":\"\"}]\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\2\0\0\231\7\2\0\0\0\0\0/localtime\0\0\10\2\0\0\201\7\2\0\0\0\0\0domain\00005\0\0\0 \2\0\0$\0\0\0008\f\320W\0\0\0\0\10\0\0\0\1\0\0\0\300\10\320W\0\0\0\0@\2\0\0,\0\0\0\340\10\320W\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0h\2\0\0,\0\0\0\350\f\320W\0\0\0\0\n\0\0\0h\v\320W\1\0\0\0D\f\320W4\v\320W\n\0\0\0\220\2\0\0004\0\0\0\270\t\320W4\v\320W\344\n\320W\344\n\320W|\n\320W4\n\320WT\t\320WT\t\320W\374\10\320W\374\10\320W\0\0\0\0\201\0\0\0@\0\320W@\0\320W\20\0\0\0,\0\0\0\260\f\320W4\n\320W\204\f\320W\250\n\320W\2\0\0\0prototype\0001\0\0\0\0\0I\0\0\0\200\0\320W\200\0\320W0000002000000000000000002\"\0ff\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0@\3\0\0004\0\0\0\0\0\0\0c52a-deb2-499c-add5-db70f1f62bff\0\0\0\0\0\0\0\0\31\6\2\0@\0\320Wx\6\320W\20\0\0\0004\0\0\0\10\t\320Wc52a-deb2-499c-add5-db70f1f62bff\0\0\0\0\0\0\0\0\331\5\2\0x\6\320W8\t\320W\0\0\0\0\0\0\0\0\0\0\0\0moid\0\0\0\0)\0\0\0H\v\320Wx\6\320W\20\0\0\0\34\0\0\0\200\10\320W110000423\0\0\0H\0\0\0$\0\0\0(\v\320W4\v\320W"..., 4096}], 1) = 0
14:00:43 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
此时回调函数中可以读到 BEV_EVENT_EOF | BEV_EVENT_READING ，errno 为 11 即 EAGAIN
14:00:43 time(NULL) = 1445666443
14:00:43 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:43 write(2, "[libupu][14:00:43]: handle_event() => sEvent(17), recv UPU disconnect(eof), errno:11 error:'Resource temporarily unavailable'\n", 127) = 127
发现代码中存在如下处理逻辑，判定若 errno 为 EAGAIN 则直接 return 。可以看出此时返回，将不会关闭 fd 31 。更进入一步，该逻辑导致没有释放网络通讯所使用的 bufferevent 。
--
if ( EAGAIN == errno )
{
// libevent will sometimes recv again when it's not actually ready,
// this results in a 0 return value, and errno will be set to EAGAIN(try again).
// This does not mean there is a hard socket error, but simple needs to be read again
return ;
}
--
从回调中返回后，代码会从 event_base_dispatch 中退出，因为当前 base 中已经没有其他事件了
但是，需要注意的是，此时在业务结构中包含了 libevent 相关内容并没有进行清理操作，若后续直接被使用，应该会导致问题
14:00:43 time(NULL) = 1445666443
14:00:43 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:43 write(2, "[libupu][14:00:43]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
14:00:43 gettimeofday({1445666443, 91211}, NULL) = 0
14:00:43 write(6, "\340\4\220Y", 4) = 4
14:00:43 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\331\v\0\7\4\335\2[pid]:25002 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94f50f8\n", 126, 0) = 126
业务代码主动挂起 4s
14:00:43 nanosleep({4, 0}, NULL) = 0
对应业务层的处理逻辑如下（这种处理个人觉得很不合理）
--
// 进入线程循环
while( !ptTaskParm->IsTerminal() )
{
ThreadLoop(ptTaskParm);
if (!ptTaskParm->IsTerminal())
{
OspDelay(4*1000);
}
}
--
---- 新建连接处理 ----
4s 挂起后，在业务层重新建立 TCP 连接（注意，不是库内部重连），可以看到，这里直接就使用了 fd 13 ，本次 connect 调用成功
原因就是因为底层没有对 fd 13 进行关闭，重用了之前的 bufferevent 导致
14:00:47 clock_gettime(CLOCK_MONOTONIC, {152394, 500384617}) = 0
14:00:47 gettimeofday({1445666447, 92001}, NULL) = 0
14:00:47 connect(31, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = 0
修正 fd 31 关注可写事件（添加）
14:00:47 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
14:00:47 gettimeofday({1445666447, 92824}, NULL) = 0
14:00:47 write(6, "\340\4\220Y", 4) = 4
14:00:47 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\331\17\0\7\4\335\2[pid]:25002 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94f50f8 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
重新进入 event_base_dispatch 事件循环逻辑
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
等待可写事件，触发后
14:00:47 clock_gettime(CLOCK_MONOTONIC, {152394, 502191015}) = 0
14:00:47 gettimeofday({1445666447, 93795}, NULL) = 0
14:00:47 epoll_wait(26, {{EPOLLOUT, {u32=31, u64=31}}}, 32, 0) = 1
14:00:47 clock_gettime(CLOCK_MONOTONIC, {152394, 502472418}) = 0
14:00:47 gettimeofday({1445666447, 94077}, NULL) = 0
判定 TCP 连接是否出错，此时没有错误发生，认为连接成功
14:00:47 getsockopt(31, SOL_SOCKET, SO_ERROR, [6231809014611574784], [4]) = 0
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: handle_event() => sEvent(128), UPU connect Success! enter Connected state.\n", 95) = 95
回调业务层告知连接成功
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: handle_event() => ### enter CallBack ### [CONNECTED] nRspType(0)\n", 86) = 86
14:00:47 futex(0xda534c, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:47 gettimeofday({1445666447, 95137}, NULL) = 0
14:00:47 write(6, "\340\4\220Y", 4) = 4
14:00:47 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\331\17\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(1),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:47 gettimeofday({1445666447, 95648}, NULL) = 0
14:00:47 write(6, "\340\4\220Y", 4) = 4
14:00:47 send(18, "\3\356\0\0\0\7\0\0\0\0\0y\0\0\0\0\5\0w!\350\331\17\0\7\4\335\3[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu connect succ msg(1) \n", 137, 0) = 137
设置读超时定时器
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: handle_event() => set 10s READ timer for Heartbeat.\n", 72) = 72
修正 fd 31 关注可读可写事件
14:00:47 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN|EPOLLOUT, {u32=31, u64=31}}) = 0
等待可读事件触发，触发后
14:00:47 epoll_ctl(26, EPOLL_CTL_MOD, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
14:00:47 epoll_wait(26, {{EPOLLIN, {u32=31, u64=31}}}, 32, 3) = 1
14:00:47 clock_gettime(CLOCK_MONOTONIC, {152394, 505306960}) = 0
14:00:47 gettimeofday({1445666447, 96911}, NULL) = 0
再次发现此时 socket 中可读数据为 0 （这回就是因为之前没有关闭 fd 31 重用了之前 bufferevent 导致的了）
14:00:47 ioctl(31, FIONREAD, [0]) = 0
14:00:47 readv(31, [{"[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu result msg:3, result type:0, nResultCount:0 \n\0\2\0@\0\320W\371\10\2\0@\0\320W@\0\320Wdows\",\"status\":\"callidle\",\"nuaddr\":\"172.16.185.135\",\"mtaddr\":\"172.16.72.105\",\"userdomain\":\"0q0y1aphxubof8ycru6fsgsk\",\"devid\":\"10103000000002000000000000000002\",\"data\":\"\"}]\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\2\0\0\231\7\2\0\0\0\0\0/localtime\0\0\10\2\0\0\201\7\2\0\0\0\0\0domain\00005\0\0\0 \2\0\0$\0\0\0008\f\320W\0\0\0\0\10\0\0\0\1\0\0\0\300\10\320W\0\0\0\0@\2\0\0,\0\0\0\340\10\320W\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0h\2\0\0,\0\0\0\350\f\320W\0\0\0\0\n\0\0\0h\v\320W\1\0\0\0D\f\320W4\v\320W\n\0\0\0\220\2\0\0004\0\0\0\270\t\320W4\v\320W\344\n\320W\344\n\320W|\n\320W4\n\320WT\t\320WT\t\320W\374\10\320W\374\10\320W\0\0\0\0\201\0\0\0@\0\320W@\0\320W\20\0\0\0,\0\0\0\260\f\320W4\n\320W\204\f\320W\250\n\320W\2\0\0\0prototype\0001\0\0\0\0\0I\0\0\0\200\0\320W\200\0\320W0000002000000000000000002\"\0ff\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0@\3\0\0004\0\0\0\0\0\0\0c52a-deb2-499c-add5-db70f1f62bff\0\0\0\0\0\0\0\0\31\6\2\0@\0\320Wx\6\320W\20\0\0\0004\0\0\0\10\t\320Wc52a-deb2-499c-add5-db70f1f62bff\0\0\0\0\0\0\0\0\331\5\2\0x\6\320W8\t\320W\0\0\0\0\0\0\0\0\0\0\0\0moid\0\0\0\0)\0\0\0H\v\320Wx\6\320W\20\0\0\0\34\0\0\0\200\10\320W110000423\0\0\0H\0\0\0$\0\0\0(\v\320W4\v\320W"..., 4096}], 1) = 0
14:00:47 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLIN, {u32=31, u64=31}}) = 0
此时回调函数中同样会读到 BEV_EVENT_EOF | BEV_EVENT_READING ，errno 为 11 即 EAGAIN
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: handle_event() => sEvent(17), recv UPU disconnect(eof), errno:11 error:'Resource temporarily unavailable'\n", 127) = 127
14:00:47 epoll_wait(26, {}, 32, 1) = 0
再次退出 event_base_dispatch 循环（没有其他事件了）
14:00:47 clock_gettime(CLOCK_MONOTONIC, {152394, 507930524}) = 0
14:00:47 gettimeofday({1445666447, 99538}, NULL) = 0
14:00:47 time(NULL) = 1445666447
14:00:47 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:47 write(2, "[libupu][14:00:47]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
14:00:47 gettimeofday({1445666447, 100023}, NULL) = 0
14:00:47 write(6, "\340\4\220Y", 4) = 4
14:00:47 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\331\17\0\7\4\335\2[pid]:25002 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94f50f8\n", 126, 0) = 126
业务再次挂起 4s
14:00:47 nanosleep({4, 0}, NULL) = 0
---- 新建连接处理 ----
再次使用 fd 31 进行连接，本次 connect 调用失败，errno 为 106 即 EISCONN ，表示在当前 fd 上已经有连接成功建立
14:00:51 clock_gettime(CLOCK_MONOTONIC, {152398, 509179805}) = 0
14:00:51 gettimeofday({1445666451, 100795}, NULL) = 0
14:00:51 connect(31, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EISCONN (Transport endpoint is already connected)
14:00:51 time(NULL) = 1445666451
14:00:51 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:51 write(2, "[libupu][14:00:51]: handle_event() => sEvent(32), connect UPU server failed, errno:106 error:'Transport endpoint is already connected'\n", 136) = 136
在连接出错后，代码中关闭 fd 31
14:00:51 close(31) = 0
回调业务，告知连接断开，其实是发生了错误
14:00:51 time(NULL) = 1445666451
14:00:51 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:51 write(2, "[libupu][14:00:51]: handle_event() => ### enter CallBack ### [DISCONNECT] nRspType(0)\n", 87) = 87
14:00:51 gettimeofday({1445666451, 101988}, NULL) = 0
14:00:51 write(6, "\340\4\220Y", 4) = 4
14:00:51 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\331\23\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(2),resultType:0, nResultCount:0, arg point:0x94f4870 \n", 153, 0) = 153
14:00:51 gettimeofday({1445666451, 102459}, NULL) = 0
14:00:51 write(6, "\340\4\220Y", 4) = 4
14:00:51 send(18, "\3\356\0\0\0\7\0\0\0\0\0w\0\0\0\0\5\0u!\350\331\23\0\7\4\335\2[pid]:25002 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94f50f8 recive upu disconnect msg(2) \n", 135, 0) = 135
14:00:51 futex(0x94f48cc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x94f48c8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
14:00:51 futex(0x94f48f8, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:51 time(NULL) = 1445666451
14:00:51 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:51 write(2, "[libupu][14:00:51]: handle_event() => back to Initial state.\n", 61) = 61
14:00:51 time(NULL) = 1445666451
14:00:51 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:51 futex(0xda5a8c, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:51 write(2, "[libupu][14:00:51]: connect_server() => bufferevent_socket_connect Failed! back to initial state.\n", 98) = 98
14:00:51 gettimeofday({1445666451, 103964}, NULL) = 0
14:00:51 write(6, "\340\4\220Y", 4) = 4
14:00:51 send(18, "\3\356\0\0\0\7\0\0\0\0\0\210\0\0\0\0\5\0\206!\350\331\23\0\7\4\335\2[pid]:25002 [UpuWrapper]: [ThreadLoop] upu_client_connect failed! ClientSeesion:0x94f50f8 IP:172.16.185.136, port:2100 \n", 152, 0) = 152
业务层再次挂起 4s
14:00:51 nanosleep({4, 0}, NULL) = 0
---- 新建连接处理 ----
14:00:55 clock_gettime(CLOCK_MONOTONIC, {152402, 513113075}) = 0
14:00:55 gettimeofday({1445666455, 104729}, NULL) = 0
新建 socket
14:00:55 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 31
14:00:55 fcntl64(31, F_GETFL) = 0x2 (flags O_RDWR)
14:00:55 fcntl64(31, F_SETFL, O_RDWR|O_NONBLOCK) = 0
重建 TCP 连接
14:00:55 connect(31, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
关注可写事件，连接中
14:00:55 epoll_ctl(26, EPOLL_CTL_ADD, 31, {EPOLLOUT, {u32=31, u64=31}}) = 0
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 write(2, "[libupu][14:00:55]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
14:00:55 gettimeofday({1445666455, 105813}, NULL) = 0
14:00:55 write(6, "\220\6PY", 4) = 4
14:00:55 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\331\27\0\7\4\335\2[pid]:25002 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94f50f8 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 futex(0xda5a8c, FUTEX_WAKE_PRIVATE, 1) = 1
进入 event_base_dispatch 循环
14:00:55 write(2, "[libupu][14:00:55]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
14:00:55 futex(0xda534c, FUTEX_WAKE_PRIVATE, 1) = 1
14:00:55 clock_gettime(CLOCK_MONOTONIC, {152402, 515334124}) = 0
14:00:55 gettimeofday({1445666455, 106938}, NULL) = 0
判定 TCP 建链过程是否有错误发生，发现错误（其实是因为服务器尚未启动成功，连接被拒）
14:00:55 epoll_wait(26, {{EPOLLERR|EPOLLHUP, {u32=31, u64=31}}}, 32, 0) = 1
14:00:55 clock_gettime(CLOCK_MONOTONIC, {152402, 515614415}) = 0
14:00:55 gettimeofday({1445666455, 107217}, NULL) = 0
14:00:55 getsockopt(31, SOL_SOCKET, SO_ERROR, [6231809014611574895], [4]) = 0
发现错误，就直接删除 fd 31 的可写事件
14:00:55 epoll_ctl(26, EPOLL_CTL_DEL, 31, {EPOLLOUT, {u32=31, u64=31}}) = 0
连接被拒绝
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 write(2, "[libupu][14:00:55]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
关闭 fd 31
14:00:55 close(31) = 0
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 write(2, "[libupu][14:00:55]: handle_event() => back to Initial state.\n", 61) = 61
此时恰好有业务调用，希望向服务器发送请求
但在回调处理中发现当前连接状态不对，无法进行业务处理
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 write(2, "[libupu][14:00:55]: handle_bussiness_data() => Not Connected! m_nState(0) == UPU_INITIAL_STATE\n", 95) = 95
故直接丢弃当前请求数据
14:00:55 time(NULL) = 1445666455
14:00:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
14:00:55 write(2, "[libupu][14:00:55]: handle_bussiness_data() => evbuffer_drain 233 Bytes from pair[1]->input!\n", 93) = 93
14:00:55 epoll_wait(26, {}, 32, 3) = 0
14:00:55 clock_gettime(CLOCK_MONOTONIC, {152402, 520846158}) = 0
14:00:55 gettimeofday({1445666455, 112450}, NULL) = 0
执行到这里，strace 的输出卡住不动，CPU 进入 100% 状态！
^CProcess 30732 detached
[root@upu_1 pas]#
此时恰好服务器进程启动成功
[root@upu_1 pas]# ps aux|grep upu|grep beam
root 1140 153 0.1 1362136 39444 pts/1 Ssl+ 14:00 0:01 /opt/mcu/upu/erts-6.0/bin/beam.smp -K true -- -root /opt/mcu/upu -progname upu -- -home /opt/mcu/upu/erts-6.0/bin -- -boot /opt/mcu/upu/releases/1/upu -mode embedded -config /opt/mcu/upu/etc/upu.config -mnesia dir '/opt/mcu/upu/data' -sname upu@upu_1 -setcookie upu -- console结果分析：
- 在最初发现服务器异常断开时，没有将相应的 socket 进行关闭，导致重连后，再次使用该 fd ，进而又遇到一次同样的错误信息；
- 第二次重连时，发现当前 fd 上存在成功建立的连接，触发错误处理逻辑，才关闭该 fd ；
- 第三次重连时，新建 socket fd ，并会遇到连接被拒的错误信息，当服务器启动成功后，客户端库触发用户态死循环，说明在业务数据结构部分数据已经出现问题（小根堆顶出现无法删除的事件）。
知道问题原因后，解决办法就很简单了， 去掉 EAGAIN 后就可以了。修改后的输出如下
[root@upu_1 pas]#
[root@upu_1 pas]# strace -t -s 1024 -p 7474
Process 7474 attached - interrupt to quit
[ Process PID=7474 runs in 32 bit mode. ]
16:22:40 clock_gettime(CLOCK_MONOTONIC, {160907, 560286547}) = 0
16:22:40 gettimeofday({1445674960, 151942}, NULL) = 0
16:22:40 epoll_wait(48,
{}, 32, 2016) = 0
16:22:42 clock_gettime(CLOCK_MONOTONIC, {160909, 578207925}) = 0
16:22:42 gettimeofday({1445674962, 169824}, NULL) = 0
16:22:42 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 time(NULL) = 1445674962
16:22:42 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:22:42 write(2, "[libupu][16:22:42]: handle_event() => sEvent(65), Heartbeat timeout 0 times.\n", 77) = 77
16:22:42 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:22:42 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:22:42 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 10000) = 1
16:22:42 clock_gettime(CLOCK_MONOTONIC, {160909, 579617265}) = 0
16:22:42 gettimeofday({1445674962, 171237}, NULL) = 0
16:22:42 writev(59, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
16:22:42 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 epoll_wait(48, {{EPOLLIN, {u32=59, u64=59}}}, 32, 9999) = 1
16:22:42 clock_gettime(CLOCK_MONOTONIC, {160909, 581192194}) = 0
16:22:42 gettimeofday({1445674962, 172802}, NULL) = 0
16:22:42 ioctl(59, FIONREAD, [16]) = 0
16:22:42 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
16:22:42 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:42 epoll_wait(48,
{}, 32, 10000) = 0
16:22:52 clock_gettime(CLOCK_MONOTONIC, {160919, 592222255}) = 0
16:22:52 gettimeofday({1445674972, 183847}, NULL) = 0
16:22:52 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 time(NULL) = 1445674972
16:22:52 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:22:52 write(2, "[libupu][16:22:52]: handle_event() => sEvent(65), Heartbeat timeout 0 times.\n", 77) = 77
16:22:52 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:22:52 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:22:52 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 10000) = 1
16:22:52 clock_gettime(CLOCK_MONOTONIC, {160919, 593696901}) = 0
16:22:52 gettimeofday({1445674972, 185309}, NULL) = 0
16:22:52 writev(59, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
16:22:52 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 epoll_wait(48, {{EPOLLIN, {u32=59, u64=59}}}, 32, 9999) = 1
16:22:52 clock_gettime(CLOCK_MONOTONIC, {160919, 594473398}) = 0
16:22:52 gettimeofday({1445674972, 186085}, NULL) = 0
16:22:52 ioctl(59, FIONREAD, [16]) = 0
16:22:52 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
16:22:52 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:22:52 epoll_wait(48,
{}, 32, 10000) = 0
16:23:02 clock_gettime(CLOCK_MONOTONIC, {160929, 605196847}) = 0
16:23:02 gettimeofday({1445674982, 196807}, NULL) = 0
16:23:02 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 time(NULL) = 1445674982
16:23:02 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:02 write(2, "[libupu][16:23:02]: handle_event() => sEvent(65), Heartbeat timeout 0 times.\n", 77) = 77
16:23:02 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:02 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:02 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 10000) = 1
16:23:02 clock_gettime(CLOCK_MONOTONIC, {160929, 606349531}) = 0
16:23:02 gettimeofday({1445674982, 197945}, NULL) = 0
16:23:02 writev(59, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
16:23:02 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 epoll_wait(48, {{EPOLLIN, {u32=59, u64=59}}}, 32, 9999) = 1
16:23:02 clock_gettime(CLOCK_MONOTONIC, {160929, 607145054}) = 0
16:23:02 gettimeofday({1445674982, 198745}, NULL) = 0
16:23:02 ioctl(59, FIONREAD, [16]) = 0
16:23:02 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
16:23:02 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:02 epoll_wait(48,
{{EPOLLIN, {u32=59, u64=59}}}, 32, 10000) = 1
16:23:05 clock_gettime(CLOCK_MONOTONIC, {160932, 979276748}) = 0
16:23:05 gettimeofday({1445674985, 570870}, NULL) = 0
16:23:05 ioctl(59, FIONREAD, [0]) = 0
16:23:05 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\0\230\6\200Y\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0MsgCallBack] ClientSeesion:0x94da028 recive upu connect succ msg(1) \n\0int:0x\21\t\2\000628 \n\0\2\0\0\0\0\0\1\t\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\36\0\0@\0\200Y@\0\200Y\0\0\0\0\0\0\0\0\0\0\0\0\201\7\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], 1) = 0
16:23:05 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:05 time(NULL) = 1445674985
16:23:05 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:05 write(2, "[libupu][16:23:05]: handle_event() => sEvent(17), recv UPU disconnect(eof), errno:4 error:'Interrupted system call'\n", 117) = 117
16:23:05 close(59) = 0
16:23:05 time(NULL) = 1445674985
16:23:05 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:05 write(2, "[libupu][16:23:05]: handle_event() => ### enter CallBack ### [DISCONNECT] nRspType(0)\n", 87) = 87
16:23:05 gettimeofday({1445674985, 573024}, NULL) = 0
16:23:05 write(6, "\320\4\240V", 4) = 4
16:23:05 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\372i\0\7\4\335\2[pid]:3713 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(2),resultType:0, nResultCount:0, arg point:0x94d4628 \n", 153, 0) = 153
16:23:05 gettimeofday({1445674985, 573516}, NULL) = 0
16:23:05 write(6, "\320\4\240V", 4) = 4
16:23:05 send(18, "\3\356\0\0\0\7\0\0\0\0\0w\0\0\0\0\5\0u!\350\372i\0\7\4\335\2[pid]:3713 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94da028 recive upu disconnect msg(2) \n", 135, 0) = 135
16:23:05 time(NULL) = 1445674985
16:23:05 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:05 write(2, "[libupu][16:23:05]: handle_event() => back to Initial state.\n", 61) = 61
16:23:05 time(NULL) = 1445674985
16:23:05 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:05 write(2, "[libupu][16:23:05]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:05 gettimeofday({1445674985, 574723}, NULL) = 0
16:23:05 write(6, "\320\4\240V", 4) = 4
16:23:05 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372i\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:05 nanosleep({4, 0},
NULL) = 0
16:23:09 clock_gettime(CLOCK_MONOTONIC, {160936, 983810668}) = 0
16:23:09 gettimeofday({1445674989, 575416}, NULL) = 0
16:23:09 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:09 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:09 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:09 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:09 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:09 time(NULL) = 1445674989
16:23:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:09 write(2, "[libupu][16:23:09]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:09 gettimeofday({1445674989, 576481}, NULL) = 0
16:23:09 write(6, "\320\4\240V", 4) = 4
16:23:09 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372m\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:09 time(NULL) = 1445674989
16:23:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:09 write(2, "[libupu][16:23:09]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:09 clock_gettime(CLOCK_MONOTONIC, {160936, 985598386}) = 0
16:23:09 gettimeofday({1445674989, 577239}, NULL) = 0
16:23:09 clock_gettime(CLOCK_MONOTONIC, {160936, 985796661}) = 0
16:23:09 gettimeofday({1445674989, 577402}, NULL) = 0
16:23:09 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 4) = 1
16:23:09 clock_gettime(CLOCK_MONOTONIC, {160936, 986085528}) = 0
16:23:09 gettimeofday({1445674989, 577684}, NULL) = 0
16:23:09 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:09 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:09 time(NULL) = 1445674989
16:23:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:09 write(2, "[libupu][16:23:09]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:09 close(39) = 0
16:23:09 time(NULL) = 1445674989
16:23:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:09 futex(0xda5a8c, FUTEX_WAKE_PRIVATE, 1) = 1
16:23:09 write(2, "[libupu][16:23:09]: handle_event() => back to Initial state.\n", 61) = 61
16:23:09 futex(0xda534c, FUTEX_WAKE_PRIVATE, 1) = 1
16:23:09 epoll_wait(48, {}, 32, 3) = 0
16:23:09 clock_gettime(CLOCK_MONOTONIC, {160936, 990331850}) = 0
16:23:09 gettimeofday({1445674989, 581909}, NULL) = 0
16:23:09 time(NULL) = 1445674989
16:23:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:09 write(2, "[libupu][16:23:09]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:09 gettimeofday({1445674989, 582208}, NULL) = 0
16:23:09 write(6, "\260oM\t", 4) = 4
16:23:09 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372m\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:09 nanosleep({4, 0},
NULL) = 0
16:23:13 clock_gettime(CLOCK_MONOTONIC, {160940, 991197248}) = 0
16:23:13 gettimeofday({1445674993, 582822}, NULL) = 0
16:23:13 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:13 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:13 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:13 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:13 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:13 time(NULL) = 1445674993
16:23:13 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:13 write(2, "[libupu][16:23:13]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:13 gettimeofday({1445674993, 583984}, NULL) = 0
16:23:13 write(6, "\320\4\240V", 4) = 4
16:23:13 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372q\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:13 time(NULL) = 1445674993
16:23:13 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:13 write(2, "[libupu][16:23:13]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:13 clock_gettime(CLOCK_MONOTONIC, {160940, 993432530}) = 0
16:23:13 gettimeofday({1445674993, 585019}, NULL) = 0
16:23:13 clock_gettime(CLOCK_MONOTONIC, {160940, 993575132}) = 0
16:23:13 gettimeofday({1445674993, 585172}, NULL) = 0
16:23:13 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:13 clock_gettime(CLOCK_MONOTONIC, {160940, 993818395}) = 0
16:23:13 gettimeofday({1445674993, 585406}, NULL) = 0
16:23:13 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:13 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:13 time(NULL) = 1445674993
16:23:13 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:13 write(2, "[libupu][16:23:13]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:13 close(39) = 0
16:23:13 time(NULL) = 1445674993
16:23:13 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:13 write(2, "[libupu][16:23:13]: handle_event() => back to Initial state.\n", 61) = 61
16:23:13 epoll_wait(48, {}, 32, 3) = 0
16:23:13 clock_gettime(CLOCK_MONOTONIC, {160940, 998060275}) = 0
16:23:13 gettimeofday({1445674993, 589658}, NULL) = 0
16:23:13 time(NULL) = 1445674993
16:23:13 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:13 write(2, "[libupu][16:23:13]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:13 gettimeofday({1445674993, 590058}, NULL) = 0
16:23:13 write(6, "\260oM\t", 4) = 4
16:23:13 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372q\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:13 nanosleep({4, 0},
NULL) = 0
16:23:17 clock_gettime(CLOCK_MONOTONIC, {160944, 999172495}) = 0
16:23:17 gettimeofday({1445674997, 590787}, NULL) = 0
16:23:17 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:17 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:17 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:17 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:17 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:17 time(NULL) = 1445674997
16:23:17 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:17 write(2, "[libupu][16:23:17]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:17 gettimeofday({1445674997, 591935}, NULL) = 0
16:23:17 write(6, "\260oM\t", 4) = 4
16:23:17 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372u\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:17 time(NULL) = 1445674997
16:23:17 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:17 write(2, "[libupu][16:23:17]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:17 clock_gettime(CLOCK_MONOTONIC, {160945, 1277778}) = 0
16:23:17 gettimeofday({1445674997, 592883}, NULL) = 0
16:23:17 clock_gettime(CLOCK_MONOTONIC, {160945, 1458890}) = 0
16:23:17 gettimeofday({1445674997, 593063}, NULL) = 0
16:23:17 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:17 clock_gettime(CLOCK_MONOTONIC, {160945, 1747907}) = 0
16:23:17 gettimeofday({1445674997, 593352}, NULL) = 0
16:23:17 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:17 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:17 time(NULL) = 1445674997
16:23:17 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:17 write(2, "[libupu][16:23:17]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:17 close(39) = 0
16:23:17 time(NULL) = 1445674997
16:23:17 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:17 write(2, "[libupu][16:23:17]: handle_event() => back to Initial state.\n", 61) = 61
16:23:17 epoll_wait(48, {}, 32, 3) = 0
16:23:17 clock_gettime(CLOCK_MONOTONIC, {160945, 6224704}) = 0
16:23:17 gettimeofday({1445674997, 597831}, NULL) = 0
16:23:17 time(NULL) = 1445674997
16:23:17 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:17 write(2, "[libupu][16:23:17]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:17 gettimeofday({1445674997, 598290}, NULL) = 0
16:23:17 write(6, "\260oM\t", 4) = 4
16:23:17 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372u\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:17 nanosleep({4, 0},
NULL) = 0
16:23:21 clock_gettime(CLOCK_MONOTONIC, {160949, 7473367}) = 0
16:23:21 gettimeofday({1445675001, 599090}, NULL) = 0
16:23:21 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:21 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:21 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:21 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:21 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:21 time(NULL) = 1445675001
16:23:21 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:21 write(2, "[libupu][16:23:21]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:21 gettimeofday({1445675001, 600265}, NULL) = 0
16:23:21 write(6, "\260oM\t", 4) = 4
16:23:21 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372y\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:21 time(NULL) = 1445675001
16:23:21 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:21 write(2, "[libupu][16:23:21]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:21 clock_gettime(CLOCK_MONOTONIC, {160949, 9690310}) = 0
16:23:21 gettimeofday({1445675001, 601298}, NULL) = 0
16:23:21 clock_gettime(CLOCK_MONOTONIC, {160949, 9875450}) = 0
16:23:21 gettimeofday({1445675001, 601506}, NULL) = 0
16:23:21 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:21 clock_gettime(CLOCK_MONOTONIC, {160949, 10229031}) = 0
16:23:21 gettimeofday({1445675001, 601836}, NULL) = 0
16:23:21 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:21 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:21 time(NULL) = 1445675001
16:23:21 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:21 write(2, "[libupu][16:23:21]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:21 close(39) = 0
16:23:21 time(NULL) = 1445675001
16:23:21 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:21 write(2, "[libupu][16:23:21]: handle_event() => back to Initial state.\n", 61) = 61
16:23:21 epoll_wait(48, {}, 32, 3) = 0
16:23:21 clock_gettime(CLOCK_MONOTONIC, {160949, 14762150}) = 0
16:23:21 gettimeofday({1445675001, 606383}, NULL) = 0
16:23:21 time(NULL) = 1445675001
16:23:21 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:21 write(2, "[libupu][16:23:21]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:21 gettimeofday({1445675001, 606918}, NULL) = 0
16:23:21 write(6, "\260oM\t", 4) = 4
16:23:21 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372y\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:21 nanosleep({4, 0},
NULL) = 0
16:23:25 clock_gettime(CLOCK_MONOTONIC, {160953, 16142873}) = 0
16:23:25 gettimeofday({1445675005, 607763}, NULL) = 0
16:23:25 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:25 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:25 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:25 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:25 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:25 time(NULL) = 1445675005
16:23:25 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:25 write(2, "[libupu][16:23:25]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:25 gettimeofday({1445675005, 608919}, NULL) = 0
16:23:25 write(6, "\260oM\t", 4) = 4
16:23:25 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372}\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:25 time(NULL) = 1445675005
16:23:25 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:25 write(2, "[libupu][16:23:25]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:25 clock_gettime(CLOCK_MONOTONIC, {160953, 18286176}) = 0
16:23:25 gettimeofday({1445675005, 609895}, NULL) = 0
16:23:25 clock_gettime(CLOCK_MONOTONIC, {160953, 18472888}) = 0
16:23:25 gettimeofday({1445675005, 610078}, NULL) = 0
16:23:25 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:25 clock_gettime(CLOCK_MONOTONIC, {160953, 18760536}) = 0
16:23:25 gettimeofday({1445675005, 610366}, NULL) = 0
16:23:25 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:25 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:25 time(NULL) = 1445675005
16:23:25 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:25 write(2, "[libupu][16:23:25]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:25 close(39) = 0
16:23:25 time(NULL) = 1445675005
16:23:25 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:25 write(2, "[libupu][16:23:25]: handle_event() => back to Initial state.\n", 61) = 61
16:23:25 epoll_wait(48, {}, 32, 3) = 0
16:23:25 clock_gettime(CLOCK_MONOTONIC, {160953, 23241459}) = 0
16:23:25 gettimeofday({1445675005, 614850}, NULL) = 0
16:23:25 time(NULL) = 1445675005
16:23:25 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:25 write(2, "[libupu][16:23:25]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:25 gettimeofday({1445675005, 615333}, NULL) = 0
16:23:25 write(6, "\260oM\t", 4) = 4
16:23:25 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372}\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:25 nanosleep({4, 0},
NULL) = 0
16:23:29 clock_gettime(CLOCK_MONOTONIC, {160957, 24514610}) = 0
16:23:29 gettimeofday({1445675009, 616132}, NULL) = 0
16:23:29 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:29 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:29 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:29 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:29 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:29 time(NULL) = 1445675009
16:23:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:29 write(2, "[libupu][16:23:29]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:29 gettimeofday({1445675009, 617295}, NULL) = 0
16:23:29 write(6, "\260oM\t", 4) = 4
16:23:29 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372\201\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:29 time(NULL) = 1445675009
16:23:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:29 write(2, "[libupu][16:23:29]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:29 clock_gettime(CLOCK_MONOTONIC, {160957, 26671985}) = 0
16:23:29 gettimeofday({1445675009, 618301}, NULL) = 0
16:23:29 clock_gettime(CLOCK_MONOTONIC, {160957, 26882105}) = 0
16:23:29 gettimeofday({1445675009, 618490}, NULL) = 0
16:23:29 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:29 clock_gettime(CLOCK_MONOTONIC, {160957, 27214855}) = 0
16:23:29 gettimeofday({1445675009, 618826}, NULL) = 0
16:23:29 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:29 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:29 time(NULL) = 1445675009
16:23:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:29 write(2, "[libupu][16:23:29]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:29 close(39) = 0
16:23:29 time(NULL) = 1445675009
16:23:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:29 write(2, "[libupu][16:23:29]: handle_event() => back to Initial state.\n", 61) = 61
16:23:29 epoll_wait(48, {}, 32, 3) = 0
16:23:29 clock_gettime(CLOCK_MONOTONIC, {160957, 31696245}) = 0
16:23:29 gettimeofday({1445675009, 623304}, NULL) = 0
16:23:29 time(NULL) = 1445675009
16:23:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:29 write(2, "[libupu][16:23:29]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:29 gettimeofday({1445675009, 623796}, NULL) = 0
16:23:29 write(6, "\260oM\t", 4) = 4
16:23:29 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372\201\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:29 nanosleep({4, 0},
NULL) = 0
16:23:33 clock_gettime(CLOCK_MONOTONIC, {160961, 32989311}) = 0
16:23:33 gettimeofday({1445675013, 624618}, NULL) = 0
16:23:33 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:33 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:33 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:33 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:33 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:33 time(NULL) = 1445675013
16:23:33 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:33 write(2, "[libupu][16:23:33]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:33 gettimeofday({1445675013, 625662}, NULL) = 0
16:23:33 write(6, "\260oM\t", 4) = 4
16:23:33 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372\205\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:33 time(NULL) = 1445675013
16:23:33 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:33 write(2, "[libupu][16:23:33]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:33 clock_gettime(CLOCK_MONOTONIC, {160961, 34940043}) = 0
16:23:33 gettimeofday({1445675013, 626540}, NULL) = 0
16:23:33 clock_gettime(CLOCK_MONOTONIC, {160961, 35142558}) = 0
16:23:33 gettimeofday({1445675013, 626747}, NULL) = 0
16:23:33 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 4) = 1
16:23:33 clock_gettime(CLOCK_MONOTONIC, {160961, 35424775}) = 0
16:23:33 gettimeofday({1445675013, 627028}, NULL) = 0
16:23:33 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:33 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:33 time(NULL) = 1445675013
16:23:33 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:33 write(2, "[libupu][16:23:33]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:33 close(39) = 0
16:23:33 time(NULL) = 1445675013
16:23:33 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:33 write(2, "[libupu][16:23:33]: handle_event() => back to Initial state.\n", 61) = 61
16:23:33 epoll_wait(48, {}, 32, 3) = 0
16:23:33 clock_gettime(CLOCK_MONOTONIC, {160961, 39835540}) = 0
16:23:33 gettimeofday({1445675013, 631445}, NULL) = 0
16:23:33 time(NULL) = 1445675013
16:23:33 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:33 write(2, "[libupu][16:23:33]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:33 gettimeofday({1445675013, 631926}, NULL) = 0
16:23:33 write(6, "\260oM\t", 4) = 4
16:23:33 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372\205\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:33 nanosleep({4, 0},
NULL) = 0
16:23:37 clock_gettime(CLOCK_MONOTONIC, {160965, 41087982}) = 0
16:23:37 gettimeofday({1445675017, 632710}, NULL) = 0
16:23:37 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:37 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:37 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:37 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:37 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:37 time(NULL) = 1445675017
16:23:37 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:37 write(2, "[libupu][16:23:37]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:37 gettimeofday({1445675017, 633856}, NULL) = 0
16:23:37 write(6, "\320\4\240V", 4) = 4
16:23:37 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372\211\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:37 time(NULL) = 1445675017
16:23:37 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:37 write(2, "[libupu][16:23:37]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:37 clock_gettime(CLOCK_MONOTONIC, {160965, 43210630}) = 0
16:23:37 gettimeofday({1445675017, 634818}, NULL) = 0
16:23:37 clock_gettime(CLOCK_MONOTONIC, {160965, 43395712}) = 0
16:23:37 gettimeofday({1445675017, 635001}, NULL) = 0
16:23:37 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:37 clock_gettime(CLOCK_MONOTONIC, {160965, 43690915}) = 0
16:23:37 gettimeofday({1445675017, 635298}, NULL) = 0
16:23:37 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:37 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:37 time(NULL) = 1445675017
16:23:37 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:37 write(2, "[libupu][16:23:37]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:37 close(39) = 0
16:23:37 time(NULL) = 1445675017
16:23:37 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:37 write(2, "[libupu][16:23:37]: handle_event() => back to Initial state.\n", 61) = 61
16:23:37 epoll_wait(48, {}, 32, 3) = 0
16:23:37 clock_gettime(CLOCK_MONOTONIC, {160965, 48145912}) = 0
16:23:37 gettimeofday({1445675017, 639754}, NULL) = 0
16:23:37 time(NULL) = 1445675017
16:23:37 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:37 write(2, "[libupu][16:23:37]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:37 gettimeofday({1445675017, 640222}, NULL) = 0
16:23:37 write(6, "\320\4\240V", 4) = 4
16:23:37 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372\211\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:37 nanosleep({4, 0},
NULL) = 0
16:23:41 clock_gettime(CLOCK_MONOTONIC, {160969, 49427437}) = 0
16:23:41 gettimeofday({1445675021, 641049}, NULL) = 0
16:23:41 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 39
16:23:41 fcntl64(39, F_GETFL) = 0x2 (flags O_RDWR)
16:23:41 fcntl64(39, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:41 connect(39, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:41 epoll_ctl(48, EPOLL_CTL_ADD, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:41 time(NULL) = 1445675021
16:23:41 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:41 write(2, "[libupu][16:23:41]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:41 gettimeofday({1445675021, 642233}, NULL) = 0
16:23:41 write(6, "\320\4\240V", 4) = 4
16:23:41 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372\215\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:41 time(NULL) = 1445675021
16:23:41 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:41 write(2, "[libupu][16:23:41]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:41 clock_gettime(CLOCK_MONOTONIC, {160969, 51633490}) = 0
16:23:41 gettimeofday({1445675021, 643243}, NULL) = 0
16:23:41 clock_gettime(CLOCK_MONOTONIC, {160969, 51823950}) = 0
16:23:41 gettimeofday({1445675021, 643432}, NULL) = 0
16:23:41 epoll_wait(48, {{EPOLLERR|EPOLLHUP, {u32=39, u64=39}}}, 32, 3) = 1
16:23:41 clock_gettime(CLOCK_MONOTONIC, {160969, 52155839}) = 0
16:23:41 gettimeofday({1445675021, 643768}, NULL) = 0
16:23:41 getsockopt(39, SOL_SOCKET, SO_ERROR, [6222784223170789487], [4]) = 0
16:23:41 epoll_ctl(48, EPOLL_CTL_DEL, 39, {EPOLLOUT, {u32=39, u64=39}}) = 0
16:23:41 time(NULL) = 1445675021
16:23:41 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:41 write(2, "[libupu][16:23:41]: handle_event() => sEvent(32), connect UPU server failed, errno:111 error:'Connection refused'\n", 115) = 115
16:23:41 close(39) = 0
16:23:41 time(NULL) = 1445675021
16:23:41 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:41 write(2, "[libupu][16:23:41]: handle_event() => back to Initial state.\n", 61) = 61
16:23:41 epoll_wait(48, {}, 32, 3) = 0
16:23:41 clock_gettime(CLOCK_MONOTONIC, {160969, 56678893}) = 0
16:23:41 gettimeofday({1445675021, 648291}, NULL) = 0
16:23:41 time(NULL) = 1445675021
16:23:41 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:41 write(2, "[libupu][16:23:41]: upu_client_loop() => event_base_dispatch [out]\n", 67) = 67
16:23:41 gettimeofday({1445675021, 648806}, NULL) = 0
16:23:41 write(6, "\320\4\240V", 4) = 4
16:23:41 send(18, "\3\356\0\0\0\7\0\0\0\0\0n\0\0\0\0\5\0l!\350\372\215\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_loop exit abnormal! ClientSeesion:0x94da028\n", 126, 0) = 126
16:23:41 nanosleep({4, 0},
NULL) = 0
16:23:45 clock_gettime(CLOCK_MONOTONIC, {160973, 58030107}) = 0
16:23:45 gettimeofday({1445675025, 649681}, NULL) = 0
16:23:45 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 59
16:23:45 fcntl64(59, F_GETFL) = 0x2 (flags O_RDWR)
16:23:45 fcntl64(59, F_SETFL, O_RDWR|O_NONBLOCK) = 0
16:23:45 connect(59, {sa_family=AF_INET, sin_port=htons(2100), sin_addr=inet_addr("172.16.185.136")}, 16) = -1 EINPROGRESS (Operation now in progress)
16:23:45 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:45 time(NULL) = 1445675025
16:23:45 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:45 write(2, "[libupu][16:23:45]: connect_server() => bufferevent_socket_connect Success! enter Connecting state.\n", 100) = 100
16:23:45 gettimeofday({1445675025, 650897}, NULL) = 0
16:23:45 write(6, "x\4\340V", 4) = 4
16:23:45 send(18, "\3\356\0\0\0\7\0\0\0\0\0\200\0\0\0\0\5\0~!\350\372\221\0\7\4\335\2[pid]:3713 [UpuWrapper]: [ThreadLoop] upu_client_connect ClientSeesion:0x94da028 IP:172.16.185.136, port:2100 \n", 144, 0) = 144
16:23:45 time(NULL) = 1445675025
16:23:45 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:45 write(2, "[libupu][16:23:45]: upu_client_loop() => event_base_dispatch [in]\n", 66) = 66
16:23:45 clock_gettime(CLOCK_MONOTONIC, {160973, 60319148}) = 0
16:23:45 gettimeofday({1445675025, 651930}, NULL) = 0
16:23:45 clock_gettime(CLOCK_MONOTONIC, {160973, 60510805}) = 0
16:23:45 gettimeofday({1445675025, 652116}, NULL) = 0
16:23:45 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 3) = 1
16:23:45 clock_gettime(CLOCK_MONOTONIC, {160973, 60808691}) = 0
16:23:45 gettimeofday({1445675025, 652415}, NULL) = 0
16:23:45 getsockopt(59, SOL_SOCKET, SO_ERROR, [6222784223170789376], [4]) = 0
16:23:45 time(NULL) = 1445675025
16:23:45 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:45 write(2, "[libupu][16:23:45]: handle_event() => sEvent(128), UPU connect Success! enter Connected state.\n", 95) = 95
16:23:45 time(NULL) = 1445675025
16:23:45 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:45 write(2, "[libupu][16:23:45]: handle_event() => ### enter CallBack ### [CONNECTED] nRspType(0)\n", 86) = 86
16:23:45 gettimeofday({1445675025, 653399}, NULL) = 0
16:23:45 write(6, "x\4\340V", 4) = 4
16:23:45 send(18, "\3\356\0\0\0\7\0\0\0\0\0\211\0\0\0\0\5\0\207!\350\372\221\0\7\4\335\2[pid]:3713 [UpuWrapper]: [UpuMsgCallBack] recive upu callback msg(1),resultType:0, nResultCount:0, arg point:0x94d4628 \n", 153, 0) = 153
16:23:45 gettimeofday({1445675025, 653944}, NULL) = 0
16:23:45 write(6, "x\4\340V", 4) = 4
16:23:45 send(18, "\3\356\0\0\0\7\0\0\0\0\0y\0\0\0\0\5\0w!\350\372\221\0\7\4\335\3[pid]:3713 [UpuWrapper]: [UpuClientMsgCallBack] ClientSeesion:0x94da028 recive upu connect succ msg(1) \n", 137, 0) = 137
16:23:45 time(NULL) = 1445675025
16:23:45 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:45 write(2, "[libupu][16:23:45]: handle_event() => set 10s READ timer for Heartbeat.\n", 72) = 72
16:23:45 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:45 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:45 epoll_wait(48, {}, 32, 3) = 0
16:23:45 clock_gettime(CLOCK_MONOTONIC, {160973, 66717197}) = 0
16:23:45 gettimeofday({1445675025, 658330}, NULL) = 0
16:23:45 epoll_wait(48,
{}, 32, 9995) = 0
16:23:55 clock_gettime(CLOCK_MONOTONIC, {160983, 71186334}) = 0
16:23:55 gettimeofday({1445675035, 662790}, NULL) = 0
16:23:55 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 time(NULL) = 1445675035
16:23:55 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:23:55 write(2, "[libupu][16:23:55]: handle_event() => sEvent(65), Heartbeat timeout 0 times.\n", 77) = 77
16:23:55 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:55 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:23:55 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 10000) = 1
16:23:55 clock_gettime(CLOCK_MONOTONIC, {160983, 72274769}) = 0
16:23:55 gettimeofday({1445675035, 663870}, NULL) = 0
16:23:55 writev(59, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
16:23:55 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 epoll_wait(48, {{EPOLLIN, {u32=59, u64=59}}}, 32, 9999) = 1
16:23:55 clock_gettime(CLOCK_MONOTONIC, {160983, 73025977}) = 0
16:23:55 gettimeofday({1445675035, 664664}, NULL) = 0
16:23:55 ioctl(59, FIONREAD, [16]) = 0
16:23:55 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
16:23:55 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:23:55 epoll_wait(48,
{}, 32, 10000) = 0
16:24:05 clock_gettime(CLOCK_MONOTONIC, {160993, 83205102}) = 0
16:24:05 gettimeofday({1445675045, 674823}, NULL) = 0
16:24:05 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 time(NULL) = 1445675045
16:24:05 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 0
16:24:05 write(2, "[libupu][16:24:05]: handle_event() => sEvent(65), Heartbeat timeout 0 times.\n", 77) = 77
16:24:05 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLOUT, {u32=59, u64=59}}) = 0
16:24:05 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN|EPOLLOUT, {u32=59, u64=59}}) = 0
16:24:05 epoll_wait(48, {{EPOLLOUT, {u32=59, u64=59}}}, 32, 10000) = 1
16:24:05 clock_gettime(CLOCK_MONOTONIC, {160993, 84503406}) = 0
16:24:05 gettimeofday({1445675045, 676115}, NULL) = 0
16:24:05 writev(59, [{"\0224Vx\0\0\0\0\0\0\0\2\0\0\0\0", 16}], 1) = 16
16:24:05 epoll_ctl(48, EPOLL_CTL_MOD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 epoll_wait(48, {{EPOLLIN, {u32=59, u64=59}}}, 32, 9999) = 1
16:24:05 clock_gettime(CLOCK_MONOTONIC, {160993, 85326294}) = 0
16:24:05 gettimeofday({1445675045, 676958}, NULL) = 0
16:24:05 ioctl(59, FIONREAD, [16]) = 0
16:24:05 readv(59, [{"\0224Vx\0\0\0\0\0\0\0\3\0\0\0\0", 16}], 1) = 16
16:24:05 epoll_ctl(48, EPOLL_CTL_DEL, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 epoll_ctl(48, EPOLL_CTL_ADD, 59, {EPOLLIN, {u32=59, u64=59}}) = 0
16:24:05 epoll_wait(48,
^C <unfinished ...>
Process 7474 detached
[root@upu_1 pas]#
来源：oschina
链接：https://my.oschina.net/u/617889/blog/597784