问题
I have installed memsql 5.1.2 in following manner with following resources.
Google cloud server
HDD: 100GB
Machine type: n1-standard-4 (4 vCPUs, 15 GB memory)
Implementation:
2 MEMSQL NODES running on same machine on the following ports
3306 Master Aggregator
3307 Leaf
Resource Utilization:
Memory 14.16 GB / 14.69 GB
Paging 0 B/s
Database size - 10MB
1818 memsql 1.1% 77% /var/lib/memsql/leaf-3307/memsqld --defaults-file=/var/lib/memsql/leaf-3307/memsql.cnf --pid-file=/var/lib/memsql/leaf-3307/data/memsqld.pid --user=memsql
2736 memsql 0.3% 16% /var/lib/memsql/master-3306/memsqld --defaults-file=/var/lib/memsql/master-330
Note: There is no Swap memory implemented in the server.
Database size is taken by running a query on information_schema.TABLES.
All data resides as row store since we have to run queries by considering many relationships among tables.
As soon as the memsql is up the memory goes up to 70% and it keep on increasing and after 2-3 hours memsql gives the following error when try connect with it and connection also can not be done after that.
OperationalError: (1836, "Leaf 'xx.xxx.x.xx':3307 failed while executing this query. Try re-running the query.")
[Mon Mar 27 09:26:31.163455 2017] [:error] [pid 1718] [remote xxx.xxx.xxx.xxx:9956]
The only solution is to restart the server since it has taken up all the memory.
What I can do for this? Is there an issue in the way it's implemented? Any logs should I attach here?
Show status extended; query gives the following result
+-------------------------------------+------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------------------------+------------------------------------------------------------------------+
| Aborted_clients | 48 |
| Aborted_connects | 1 |
| Bytes_received | 85962135 |
| Bytes_sent | 545322701 |
| Connections | 1626 |
| Max_used_connections | 69 |
| Queries | 364793 |
| Questions | 364793 |
| Threads_cached | 19 |
| Threads_connected | 50 |
| Threads_created | 69 |
| Threads_running | 1 |
| Threads_background | 1 |
| Threads_idle | 0 |
| Ready_queue | 0 |
| Idle_queue | 0 |
| Context_switches | 1626 |
| Context_switch_misses | 0 |
| Uptime | 22270 |
| Auto_attach_remaining_seconds | 0 |
| Data_directory | /var/lib/memsql/leaf-3307/data |
| Plancache_directory | /var/lib/memsql/leaf-3307/plancache |
| Transaction_logs_directory | /var/lib/memsql/leaf-3307/data/logs |
| Segments_directory | /var/lib/memsql/leaf-3307/data/columns |
| Snapshots_directory | /var/lib/memsql/leaf-3307/data/snapshots |
| Threads_waiting_for_disk_space | 0 |
| Seconds_until_expiration | -1 |
| License_key | 11111111111111111111111111111111 |
| License_type | community |
| Query_compilations | 62 |
| Query_compilation_failures | 0 |
| GCed_versions_last_sweep | 0 |
| Average_garbage_collection_duration | 21 ms |
| Total_server_memory | 9791.4 MB |
| Alloc_thread_stacks | 70.0 MB |
| Malloc_active_memory | 1254.7 (+0.0) MB |
| Malloc_cumulative_memory | 7315.5 (+0.2) MB |
| Buffer_manager_memory | 1787.8 MB |
| Buffer_manager_cached_memory | 77.2 (-0.1) MB |
| Buffer_manager_unrecycled_memory | 0.0 MB |
| Alloc_skiplist_tower | 263.8 MB |
| Alloc_variable | 501.4 MB |
| Alloc_large_variable | 2.4 MB |
| Alloc_table_primary | 752.6 MB |
| Alloc_deleted_version | 92.9 MB |
| Alloc_internal_key_node | 72.1 MB |
| Alloc_hash_buckets | 459.1 MB |
| Alloc_table_metadata_cache | 1.1 MB |
| Alloc_unit_images | 34.8 MB |
| Alloc_unit_ifn_thunks | 0.6 MB |
| Alloc_object_code_images | 11.6 MB |
| Alloc_compiled_unit_sections | 17.3 MB |
| Alloc_databases_list_entry | 17.9 MB |
| Alloc_plan_cache | 0.1 MB |
| Alloc_replication_large | 232.0 MB |
| Alloc_durability_large | 7239.1 MB |
| Alloc_sharding_partitions | 0.1 MB |
| Alloc_security | 0.1 MB |
| Alloc_log_replay | 0.9 MB |
| Alloc_client_connection | 3.0 MB |
| Alloc_protocol_packet | 6.1 (+0.1) MB |
| Alloc_large_incremental | 0.8 MB |
| Alloc_table_memory | 2144.2 MB |
| Alloc_variable_bucket_16 | allocs:10877846 alloc_MB:166.0 buffer_MB:179.0 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_24 | allocs:4275659 alloc_MB:97.9 buffer_MB:106.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_32 | allocs:2875801 alloc_MB:87.8 buffer_MB:93.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_40 | allocs:724489 alloc_MB:27.6 buffer_MB:31.0 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_48 | allocs:377060 alloc_MB:17.3 buffer_MB:19.8 cached_buffer_MB:0.9 |
| Alloc_variable_bucket_56 | allocs:228720 alloc_MB:12.2 buffer_MB:14.0 cached_buffer_MB:0.8 |
| Alloc_variable_bucket_64 | allocs:150214 alloc_MB:9.2 buffer_MB:10.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_72 | allocs:35264 alloc_MB:2.4 buffer_MB:2.9 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_80 | allocs:14920 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_88 | allocs:5582 alloc_MB:0.5 buffer_MB:0.6 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_104 | allocs:8075 alloc_MB:0.8 buffer_MB:1.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_128 | allocs:8892 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_160 | allocs:17614 alloc_MB:2.7 buffer_MB:3.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_200 | allocs:30454 alloc_MB:5.8 buffer_MB:6.9 cached_buffer_MB:0.6 |
| Alloc_variable_bucket_248 | allocs:4875 alloc_MB:1.2 buffer_MB:1.5 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_312 | allocs:371 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_384 | allocs:30 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_480 | allocs:11 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_600 | allocs:57 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_752 | allocs:62 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_936 | allocs:42 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1168 | allocs:106 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1480 | allocs:126 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1832 | allocs:0 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_2288 | allocs:1 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_2832 | allocs:33 alloc_MB:0.1 buffer_MB:1.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_3528 | allocs:16 alloc_MB:0.1 buffer_MB:0.5 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_4504 | allocs:49 alloc_MB:0.2 buffer_MB:0.8 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_5680 | allocs:66 alloc_MB:0.4 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_6224 | allocs:30 alloc_MB:0.2 buffer_MB:1.0 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_7264 | allocs:94 alloc_MB:0.7 buffer_MB:1.5 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_9344 | allocs:70 alloc_MB:0.6 buffer_MB:2.6 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_11896 | allocs:14 alloc_MB:0.2 buffer_MB:2.4 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_14544 | allocs:7 alloc_MB:0.1 buffer_MB:2.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_18696 | allocs:18 alloc_MB:0.3 buffer_MB:3.2 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_21816 | allocs:4 alloc_MB:0.1 buffer_MB:0.4 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_26184 | allocs:6 alloc_MB:0.1 buffer_MB:0.9 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_32728 | allocs:13 alloc_MB:0.4 buffer_MB:2.4 cached_buffer_MB:1.4 |
| Alloc_variable_bucket_43648 | allocs:12 alloc_MB:0.5 buffer_MB:1.4 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_65472 | allocs:7 alloc_MB:0.4 buffer_MB:2.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_130960 | allocs:3 alloc_MB:0.4 buffer_MB:2.2 cached_buffer_MB:1.9 |
| Alloc_variable_cached_buffers | 21.4 MB |
| Alloc_variable_allocated | 438.7 MB |
| Successful_read_queries | 9048 |
| Successful_write_queries | 19096 |
| Failed_read_queries | 0 |
| Failed_write_queries | 4 |
| Rows_returned_by_reads | 75939 |
| Rows_affected_by_writes | 245 |
| Execution_time_of_reads | 7864 ms |
| Execution_time_of_write | 180311 ms |
| Transaction_buffer_wait_time | 0 ms |
| Transaction_log_flush_wait_time | 0 ms |
| Row_lock_wait_time | 0 ms |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 18446744073709551615 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 20480 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_version | |
| Ssl_session_cache_mode | SERVER |
+-------------------------------------+------------------------------------------------------------------------+
回答1:
From the status output, we can see:
- 10GB total memory on the leaf node
- 7GB Alloc_durability_large
You can see what these variables mean here: https://help.memsql.com/hc/en-us/articles/115001091386-What-Is-Using-Memory-on-My-Leaves-
Most interesting is the large amount in Alloc_durability_large, which is unusual. Do you have a large number of databases and/or partitions? (You can check by counting the number of rows in SHOW DATABASES EXTENDED on the leaf nodoe.) Each will require a fixed amount of transaction buffer memory (default is 64 MB).
来源：https://stackoverflow.com/questions/43038794/memsql-takes-15gb-memory-for-10mb-of-data