top - 22:05:20 up 11 days, 22:11, 1 user, load average: 0.06, 1.16, 6.88
Tasks: 151 total, 2 running, 149 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.3 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1016656 total, 66612 free, 826048 used, 123996 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 41128 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17204 apache 20 0 866468 22012 2936 S 2.0 2.2 0:00.53 php-fpm
17654 root 20 0 780056 5296 800 S 0.3 0.5 32:01.89 AliHids
1 root 20 0 203788 2696 1044 S 0.0 0.3 0:29.33 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:03.64 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14
24 root 20 0 0 0 0 S 0.0 0.0 2:30.44 rcu_sched
25 root 20 0 0 0 0 R 0.0 0.0 5:49.24 rcuos/0
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/1
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/2
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/3
29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/4
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/5
31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuos/6
阿里云ECS,安装的数据库再次无法启动，提示内存不足。可我的网站就我一个人访问啊。。。。
1G内存就剩下70MB左右了，哪去了。很多 rcuos/1 rcuos/2 rcuos/3之类的。这些是什么程序？
/var/log/mariadb/mariadb.log 报错内容：
150629 22:02:44 [Note] /usr/libexec/mysqld: Shutdown complete
150629 22:02:44 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150629 22:20:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150629 22:20:11 InnoDB: The InnoDB memory heap is disabled
150629 22:20:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150629 22:20:11 InnoDB: Compressed tables use zlib 1.2.7
150629 22:20:11 InnoDB: Using Linux native AIO
150629 22:20:11 InnoDB: Initializing buffer pool, size = 128.0M
150629 22:20:11 InnoDB: Completed initialization of buffer pool
150629 22:20:11 InnoDB: highest supported file format is Barracuda.
150629 22:20:11 InnoDB: Waiting for the background threads to start
150629 22:20:12 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150629 22:20:12 [Note] Plugin 'FEEDBACK' is disabled.
150629 22:20:12 [Note] Server socket created on IP: '0.0.0.0'.
150629 22:20:12 [Note] Event Scheduler: Loaded 0 events
150629 22:20:12 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150629 22:20:16 [ERROR] mysqld: Table './new_dalu/tb_info' is marked as crashed and should be repaired
150629 22:20:16 [Warning] Checking table: './new_dalu/tb_info'
150629 22:20:20 [ERROR] mysqld: Table './new_dalu/tb_hit_log' is marked as crashed and should be repaired
150629 22:20:20 [Warning] Checking table: './new_dalu/tb_hit_log'
150629 22:20:21 [ERROR] Got an error from unknown thread, /builddir/build/BUILD/mariadb-5.5.41/storage/myisam/ha_myisam.cc:936
150629 22:20:21 [Warning] Recovering table: './new_dalu/tb_hit_log'
150629 22:20:21 [Note] Found 81732 of 81730 rows when repairing './new_dalu/tb_hit_log'
150629 22:20:39 [ERROR] mysqld: Table './new_dalu/tb_member' is marked as crashed and should be repaired
150629 22:20:39 [ERROR] mysqld: Table './new_dalu/tb_member' is marked as crashed and should be repaired
150629 22:20:39 [Warning] Checking table: './new_dalu/tb_member'
150629 22:24:27 [ERROR] mysqld: Table './new_dalu/tb_info_anon' is marked as crashed and should be repaired
150629 22:24:27 [Warning] Checking table: './new_dalu/tb_info_anon'
Reduce OS Jitter
有以下几个相关的kernel flag
CONFIG_RCU_NOCB_CPU
CONFIG_RCU_NOCB_CPU_ALL
CONFIG_RCU_STALL_COMMON
CONFIG_RCU_USER_QS
这东西是内核线程，每个CPU对应一个rcuob和一个rcuos，几乎不费内存的。
看你的top中有“826048 used”，内存基本已经被用光了，但看不到是哪个进程干的。