简体   繁体   中英

MySQL 5.7: fine tuning for too many connections error

I have an old django 1.9 application connected to a mysql 5.7 database. Sometime I get this error:

[Fri Mar 20 10:27:19.482820 2020] [wsgi:error] [pid 15215] "
[Fri Mar 20 10:28:20.502889 2020] [wsgi:error] [pid 15215] [NXU ts_fw20] [2020-03-20 Fri 10:28:20.1584696500]ERROR   "Exception during GET request (user: AnonymousUser): <QueryDict: {u'udid': [u'B8663A17-C121-4691-A561-E0F698CDD4EA'], u'client_os_version': [u'13.3.1'], u'client_app_version': [u'11.30.1.201910161040'], u'client_app_type': [u'Jukebox'], u'client_app_name': [u'p_ts_w_fw20']}> ->
[Fri Mar 20 10:28:20.502976 2020] [wsgi:error] [pid 15215] Traceback (most recent call last):
[Fri Mar 20 10:28:20.502991 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
[Fri Mar 20 10:28:20.503041 2020] [wsgi:error] [pid 15215]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Mar 20 10:28:20.503101 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/nxlib/core/services/utils/api.py", line 307, in wrapper
[Fri Mar 20 10:28:20.503114 2020] [wsgi:error] [pid 15215]     user = authenticator.check_auth(request)
[Fri Mar 20 10:28:20.503123 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/nxlib/core/networking/auth.py", line 21, in check_auth
[Fri Mar 20 10:28:20.503133 2020] [wsgi:error] [pid 15215]     user = self.authenticate(credentials)
[Fri Mar 20 10:28:20.503142 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/nxlib/core/services/utils/api.py", line 387, in authenticate
[Fri Mar 20 10:28:20.503151 2020] [wsgi:error] [pid 15215]     user = auth.authenticate(username=username, password=password)
[Fri Mar 20 10:28:20.503160 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 74, in authenticate
[Fri Mar 20 10:28:20.503168 2020] [wsgi:error] [pid 15215]     user = backend.authenticate(**credentials)
[Fri Mar 20 10:28:20.503177 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/contrib/auth/backends.py", line 17, in authenticate
[Fri Mar 20 10:28:20.503186 2020] [wsgi:error] [pid 15215]     user = UserModel._default_manager.get_by_natural_key(username)
[Fri Mar 20 10:28:20.503194 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
[Fri Mar 20 10:28:20.503203 2020] [wsgi:error] [pid 15215]     return self.get(**{self.model.USERNAME_FIELD: username})
[Fri Mar 20 10:28:20.503211 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
[Fri Mar 20 10:28:20.503220 2020] [wsgi:error] [pid 15215]     return getattr(self.get_queryset(), name)(*args, **kwargs)
[Fri Mar 20 10:28:20.503228 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/query.py", line 381, in get
[Fri Mar 20 10:28:20.503237 2020] [wsgi:error] [pid 15215]     num = len(clone)
[Fri Mar 20 10:28:20.503244 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/query.py", line 240, in __len__
[Fri Mar 20 10:28:20.503253 2020] [wsgi:error] [pid 15215]     self._fetch_all()
[Fri Mar 20 10:28:20.503261 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
[Fri Mar 20 10:28:20.503269 2020] [wsgi:error] [pid 15215]     self._result_cache = list(self.iterator())
[Fri Mar 20 10:28:20.503278 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/query.py", line 52, in __iter__
[Fri Mar 20 10:28:20.503286 2020] [wsgi:error] [pid 15215]     results = compiler.execute_sql()
[Fri Mar 20 10:28:20.503294 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 846, in execute_sql
[Fri Mar 20 10:28:20.503302 2020] [wsgi:error] [pid 15215]     cursor = self.connection.cursor()
[Fri Mar 20 10:28:20.503311 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/base/base.py", line 233, in cursor
[Fri Mar 20 10:28:20.503319 2020] [wsgi:error] [pid 15215]     cursor = self.make_cursor(self._cursor())
[Fri Mar 20 10:28:20.503327 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/base/base.py", line 204, in _cursor
[Fri Mar 20 10:28:20.503345 2020] [wsgi:error] [pid 15215]     self.ensure_connection()
[Fri Mar 20 10:28:20.503354 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
[Fri Mar 20 10:28:20.503363 2020] [wsgi:error] [pid 15215]     self.connect()
[Fri Mar 20 10:28:20.503372 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
[Fri Mar 20 10:28:20.503380 2020] [wsgi:error] [pid 15215]     six.reraise(dj_exc_type, dj_exc_value, traceback)
[Fri Mar 20 10:28:20.503389 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
[Fri Mar 20 10:28:20.503398 2020] [wsgi:error] [pid 15215]     self.connect()
[Fri Mar 20 10:28:20.503406 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/base/base.py", line 171, in connect
[Fri Mar 20 10:28:20.503415 2020] [wsgi:error] [pid 15215]     self.connection = self.get_new_connection(conn_params)
[Fri Mar 20 10:28:20.503423 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 264, in get_new_connection
[Fri Mar 20 10:28:20.503433 2020] [wsgi:error] [pid 15215]     conn = Database.connect(**conn_params)
[Fri Mar 20 10:28:20.503441 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
[Fri Mar 20 10:28:20.503450 2020] [wsgi:error] [pid 15215]     return Connection(*args, **kwargs)
[Fri Mar 20 10:28:20.503458 2020] [wsgi:error] [pid 15215]   File "/home/nuxie/virtualenv/110-fw20/production/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
[Fri Mar 20 10:28:20.503467 2020] [wsgi:error] [pid 15215]     super(Connection, self).__init__(*args, **kwargs2)
[Fri Mar 20 10:28:20.503475 2020] [wsgi:error] [pid 15215] OperationalError: (1040, 'Too many connections')

My initial MySQL setup:

max_connections                     150

connect_timeout                     10
delayed_insert_timeout              300
have_statement_timeout              YES
innodb_flush_log_at_timeout         1
innodb_lock_wait_timeout            50
innodb_rollback_on_timeout          OFF
interactive_timeout                 28800
lock_wait_timeout                   31536000
net_read_timeout                    30
net_write_timeout                   60
rpl_stop_slave_timeout              31536000
slave_net_timeout                   60
wait_timeout                        28800

bulk_insert_buffer_size             8388608
innodb_buffer_pool_chunk_size       134217728
innodb_buffer_pool_dump_at_shutdown ON
innodb_buffer_pool_dump_now         OFF
innodb_buffer_pool_dump_pct         25
innodb_buffer_pool_filename         ib_buffer_pool
innodb_buffer_pool_instances        1
innodb_buffer_pool_load_abort       OFF
innodb_buffer_pool_load_at_startup  ON
innodb_buffer_pool_load_now         OFF
innodb_buffer_pool_size             134217728
innodb_change_buffer_max_size       25
innodb_change_buffering             all
innodb_log_buffer_size              16777216
innodb_sort_buffer_size             1048576
join_buffer_size                    262144
key_buffer_size                     16777216
myisam_sort_buffer_size             8388608
net_buffer_length                   16384
preload_buffer_size                 32768
read_buffer_size                    131072
read_rnd_buffer_size                262144
sort_buffer_size                    262144
sql_buffer_result                   OFF

max_allowed_packet                  16777216
slave_max_allowed_packet            1073741824

Here also some additional hardware info about the server. I don't know if there is any SSD or NVME devices on MySQL Host server because it's a virtual machine not managed by me, I manage only the django application. Right now of course I would like to avoid also errors from mysql.

$ ps -ef|grep apache2|grep -v grep -c
11

Mysql service status:

$ sudo service mysql status
● mysql.service - MySQL Community Server
  Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
  Active: active (running) since Fri 2020-03-20 10:28:33 CET; 3h 31min ago
 Process: 17847 ExecStartPost=/usr/share/mysql/mysql-systemd-start post (code=exited, status=0/SUCCESS)
 Process: 17834 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 17846 (mysqld)
  Tasks: 166
  Memory: 494.0M
   CPU: 57min 24.058s
  CGroup: /system.slice/mysql.service
      └─17846 /usr/sbin/mysqld
Mar 20 10:28:31 _ systemd[1]: Starting MySQL Community Server...
Mar 20 10:28:33 _ systemd[1]: Started MySQL Community Server.

CPU information:

$ lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2650L v2 @ 1.70GHz
Stepping:              4
CPU MHz:               1700.000
BogoMIPS:              3400.00
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ibrs ibpb stibp kaiser fsgsbase tsc_adjust smep arat arch_capabilities

Free memory:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          32175        2277        1465         369       28432       28317
Swap:           975           2         972

fs.file-max current value:

# sysctl -a |grep file-max
fs.file-max = 3282776

Thanks to this article from Percona I am thinking to change also the wait_timeout variable, but I'm unsure of the value to use.

Any advise about a general fine tuning for my mysql server?


EDIT: I tried also MySQLTuner-perl , but I don't see nothing related to my actual problem:

$ perl mysqltuner.pl --host 127.0.0.1
 >>  MySQLTuner 1.7.19 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[--] Performing tests on 127.0.0.1:3306
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 5.7.25-0ubuntu0.16.04.2
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(744B)
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn't contain any warning.
[!!] /var/log/mysql/error.log contains 2 error(s).
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in MyISAM tables: 1.0K (Tables: 1)
[--] Data in InnoDB tables: 396.9M (Tables: 4650)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 302d 0h 54m 4s (34M q [1.320 qps], 19K conn, TX: 44G, RX: 19G)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory     : 31.4G
[--] Max MySQL memory    : 3.5G
[--] Other process memory: 0B
[--] Total buffers: 192.0M global + 17.1M per thread (201 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 1.4G (4.31% of installed RAM)
[OK] Maximum possible memory usage: 3.5G (11.26% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/34M)
[OK] Highest usage of available connections: 34% (70/201)
[OK] Aborted connections: 0.15%  (30/19766)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 34M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 4% (10K temp sorts / 205K sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 59% (46K on disk / 77K total)
[OK] Thread cache hit rate: 96% (700 created / 19K connections)
[!!] Table cache hit rate: 0% (416 open / 411K opened)
[!!] table_definition_cache(615) is lower than number of tables(4930) 
[OK] Open file limit used: 0% (7/1K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/48.0K
[OK] Read Key buffer hit rate: 99.6% (14K cached / 66 reads)
[OK] Write Key buffer hit rate: 100.0% (121 cached / 121 writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M/396.9M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (18914488493 hits/ 18914570989 total)
[OK] InnoDB Write log efficiency: 96.81% (1249437 hits/ 1290575 total)
[OK] InnoDB log waits: 0.00% (0 waits / 41138 writes)
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control error line(s) into /var/log/mysql/error.log file
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: https://zzz
    Read this before increasing for MariaDB https://mariadb.com/kb/en/library/optimizing-table_open_cache/
    This is MyISAM only table_cache scalability problem, InnoDB not affected.
    See more details here: https://bugs.mysql.com/bug.php?id=49177
    This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
    Beware that open_files_limit (1024) variable 
    should be greater than table_open_cache (431)
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://xxx
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    table_open_cache (> 431)
    table_definition_cache(615) > 4930 or -1 (autosizing if supported)
    innodb_buffer_pool_size (>= 396.9M) if possible.
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

However, I noticied that at beginning I had:

SELECT @@open_files_limit;
1024

To handle this limit I followed (for now within a local Ubuntu installation on my notebook) this answer .

I splitted code blocks to hopefully emprove readibility. On monday I will add also add complete informations about mysql global status, full processlist and iowait stats (added global variables link here ).

The operative system is Ubuntu 16.04.

I am using right now:

  • ulimit -n 24000 (check output from ulimit -a ). This value was 1024
  • fs.file-max = 3282776 (value already set in operative system)
  • select @@open_files_limit; now is 32000 thanks to this answer .
  • max_connections = 311 (now correctly set also after sudo service mysql restart thanks to this answer )

UPDATE 2020-03-30 14:58

Here SHOW GLOBAL STATUS; and actual output for SHOW FULL PROCESSLIST; :

Id,User,Host,db,Command,Time,State,Info
1810,root,localhost:60044,NULL,Sleep,33,,NULL
1811,root,localhost:60046,NULL,Query,0,starting,"show full processlist"

Here full output from ulimit -a , updated at today (2020-03-30 14:42)

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128535
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 24000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 128535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Full output from show global variables; (no changes during last week).

Also with default wait_timeout currently I haven't any major disadvantages.

Thanks in advance.

Rate Per Second = RPS

Suggestions to consider for your my.cnf [mysqld] section

innodb_buffer_pool_size=4G  # from 128M to reduce innodb_buffer_pool_reads RPS of 1
innodb_io_capacity=800  # from 200 to use more of your HDD IOPS
read_rnd_buffer_size=128K  # from 256K to reduce handler_read_rnd_next RPS of 42,434
read_buffer_size=512K  # from 128K to reduce handler_read_next RPS of 5,541
open_files_limit=20000  # from 32000 to allow OS support of other apps handles.

You will find these changes reduce your CPU BUSY significantly.

View my profile, Network profile for contact info and free downloadable Utility Scripts to assist with performance tuning, specifically findfragtables.sql and find-redundant_indexes.sql scripts.

I noticed you SET Open Files handles in ubuntu to 1024 when you ran ulimit -n 1024. Your system needs many more available Open File handles to support your 4650 innodb tables. Suggestion ulimit -n 24000 and then check settings with ulimit -a (to list current limits). I am looking forward to analyzing your data next week from production instance.

Suggestion for your ulimit -a to consider please

ulimit -n 24000 # from 1024 to enable more OS file handles to be active.

The above is dynamic with Linux OS. Stop/Start services would have access to the handles. To make this persistent across OS shutdown/restart, review this url for similar OS instructions. These instructions set 500000 for the file-max, please set your capacity at 24000 for now. ulimit please set to 24000, which will allow MySQL to use requested limits and have spares for other apps

https://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM