ELK 小白求经验

求解析~~

最近根据官网搭建了ELK，看了Kibana4 发现可以做很多事情，关键是对数据源的处理转化，现在想做一个统计接口调用排名前10的表格，demo里也有，

但是不知道怎么配置更好，我的服务器是tomcat 日志用的logback 格式类似

drwxr-xr-x 16 root root 4096 Sep 21 09:25 ./
drwxr-xr-x  3 root root 4096 Sep  8 16:16 ../
drwxr-xr-x  2 root root 4096 Sep  9 13:03 2015-09-08/
drwxr-xr-x  2 root root 4096 Sep  9 09:53 2015-09-09/
drwxr-xr-x  2 root root 4096 Sep 10 10:28 2015-09-10/
drwxr-xr-x  2 root root 4096 Sep 11 09:59 2015-09-11/
drwxr-xr-x  2 root root 4096 Sep 12 08:21 2015-09-12/
drwxr-xr-x  2 root root 4096 Sep 13 11:05 2015-09-13/
drwxr-xr-x  2 root root 4096 Sep 14 14:48 2015-09-14/
drwxr-xr-x  2 root root 4096 Sep 15 14:02 2015-09-15/
drwxr-xr-x  2 root root 4096 Sep 16 11:32 2015-09-16/
drwxr-xr-x  2 root root 4096 Sep 17 18:44 2015-09-17/
drwxr-xr-x  2 root root 4096 Sep 18 18:13 2015-09-18/
drwxr-xr-x  2 root root 4096 Sep 19 14:58 2015-09-19/
drwxr-xr-x  2 root root 4096 Sep 20 12:26 2015-09-20/
drwxr-xr-x  2 root root 4096 Sep 21 09:38 2015-09-21/

drwxr-xr-x  2 root root    4096 Sep 21 09:38 ./
drwxr-xr-x 16 root root    4096 Sep 21 09:25 ../
-rw-r--r--  1 root root 1753549 Sep 21 16:35 debug-log.log
-rw-r--r--  1 root root       0 Sep 21 09:38 error-log.log
-rw-r--r--  1 root root    1547 Sep 21 16:34 info-log.log
-rw-r--r--  1 root root       0 Sep 21 09:38 trace-log.log
-rw-r--r--  1 root root     675 Sep 21 16:34 warn-log.log

是按照时间来切割的，不知道按大小切割会不会更好。

主要日志路径为debug-log.log

debug-log.log日志样式

2015-09-21 16:39:16.911 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@278d3f55]
2015-09-21 16:39:16.912 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:16.912 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@50b5fc60] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:16.913 [http-bio-80-exec-7] DEBUG com.weitoo.server.mapper.CommodityPictureMapper - Cache Hit Ratio [com.weitoo.server.mapper.CommodityPictureMapper]: 1.0
2015-09-21 16:39:16.914 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@50b5fc60]
2015-09-21 16:39:16.914 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:16.914 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@395e2b58] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:16.916 [http-bio-80-exec-7] DEBUG com.weitoo.server.mapper.PictureMapper - Cache Hit Ratio [com.weitoo.server.mapper.PictureMapper]: 0.5
2015-09-21 16:39:16.916 [http-bio-80-exec-7] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@395e2b58]
2015-09-21 16:39:16.917 [http-bio-80-exec-7] DEBUG com.weitoo.server.aspect.LogAspect - {ip:183.16.10.xx,url:http://xx.xx.com/server/sc/commodity/searchCommodity,param:{"shopId":0,"keyword":"6901028182652","onlyPicList":true},return:{"status":1,"data":{"id":0,"count":1,"picList":[{"id":0,"picId":1335043,"picPath":"common/commodity/2014/07/17/53c75b5003843.jpg"}]}},cost:18.373ms}
2015-09-21 16:39:20.175 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:20.176 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5d2f96c0] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:20.179 [http-bio-80-exec-9] DEBUG com.weitoo.server.mapper.PublicCommodityMapper - Cache Hit Ratio [com.weitoo.server.mapper.PublicCommodityMapper]: 1.0
2015-09-21 16:39:20.179 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5d2f96c0]
2015-09-21 16:39:20.180 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:20.180 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@12b65411] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:20.182 [http-bio-80-exec-9] DEBUG com.weitoo.server.mapper.CommodityMapper - Cache Hit Ratio [com.weitoo.server.mapper.CommodityMapper]: 0.75
2015-09-21 16:39:20.183 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@12b65411]
2015-09-21 16:39:20.184 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:20.184 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@455d19f5] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:20.186 [http-bio-80-exec-9] DEBUG com.weitoo.server.mapper.CommodityPictureMapper - Cache Hit Ratio [com.weitoo.server.mapper.CommodityPictureMapper]: 1.0
2015-09-21 16:39:20.187 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@455d19f5]
2015-09-21 16:39:20.187 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Creating a new SqlSession
2015-09-21 16:39:20.187 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@7074ac7b] was not registered for synchronization because synchronization is not active
2015-09-21 16:39:20.190 [http-bio-80-exec-9] DEBUG com.weitoo.server.mapper.PictureMapper - Cache Hit Ratio [com.weitoo.server.mapper.PictureMapper]: 0.6
2015-09-21 16:39:20.190 [http-bio-80-exec-9] DEBUG org.mybatis.spring.SqlSessionUtils - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@7074ac7b]
2015-09-21 16:39:20.191 [http-bio-80-exec-9] DEBUG com.weitoo.server.aspect.LogAspect - {ip:183.16.10.xx,url:http://xx.xx.com/server/sc/commodity/searchCommodity,param:{"shopId":0,"keyword":"6901028182652","onlyPicList":true},return:{"status":1,"data":{"id":0,"count":1,"picList":[{"id":0,"picId":1335043,"picPath":"common/commodity/2014/07/17/53c75b5003843.jpg"}]}},cost:15.585ms}

本人对日志样式也有点疑惑，究竟生产上需不需要打印sql语句，有sql语句方便定位问题，个人感觉日志规范性这块，感觉经验不足，我想只有规范化一些东西后，才更好的去扩展，望大神多指正。

是究竟只统计debug.log 还是全部都统计，合在一个文件是不是更方便。

现在暂且先对tomcat下的catalina.out日志进行统计

logstash 的shipp.conf 配置文件

input {
  file {
   type => "syslog"
   path => "/opt/software/apache-tomcat-7.0.59/logs/catalina.out"
 }
 syslog{
   type => "syslog"
   port => "5544"
 }

}
output {
   elasticsearch {host =>"xxx-management"}
   stdout { codec=> rubydebug }
}

这个地方不知道怎么样去改写，让格式里可以拥有更多字段，方便统计，例如这是官方demo里的信息,里面定义了很多字段 count,client_ip等

status:OK client_ip:127.0.0.1 ip:127.0.0.1 bytes_out:194 client_port:47,720 shipper:app.server1 type:mongodb query:admin.$cmd.isMaster() server:app.server1 port:27,017 count:1 resource:admin.$cmd client_server:app.server1 timestamp:September 21st 2015, 16:46:11.000 bytes_in:62 request:admin.$cmd.isMaster() response:{"ismaster":true,"localTime":"2015-06-29T19:25:06.773+02:00","maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":3,"maxWriteBatchSize":1000,"minWireVersion":0,"ok":1} client_proc: responsetime:13 mongodb.numberReturned:1 mongodb.fullCollectionName:admin.$cmd mongodb.cursorId:0 mongodb.startingFrom:0 mongodb.numberToSkip:0 mongodb.numberToReturn:4294967295 proc: method:isMaster _id:AU_ujWYzjxQW72WJYAUB _type:mongodb _index:packetbeat-2015.09.21

而我的只有

message:2015-09-21 15:30:16 [http-bio-80-exec-15] DEBUG com.weitoo.server.mapper.AreaMapper -Cache Hit Ratio [com.xx.server.mapper.AreaMapper]: 1.0 @version:1 @timestamp:September 21st 2015, 15:30:17.857 host:0.0.0.0 path:/opt/software/apache-tomcat-7.0.59/logs/catalina.out type:syslog _id:AU_uz3mrm6BP0WePxJCr _type:syslog _index:logstash-2015.09.21

message:2015-09-21 15:30:16 [http-bio-80-exec-15] DEBUG com.weitoo.server.aspect.LogAspect -{ip:183.16.10.91,url:http://api.xx.com/server/sc/area/getOpenAreas,param:,return:{"status":1,"data":{"count":8,"areaDTOs":[{"id":1062421,"parentId":0,"name":"深圳市","level":1},{"id":1062527,"parentId":0,"name":"广州市","level":1},{"id":1062522,"parentId":1062421,"name":"南山区","level":2},{"id":1062525,"parentId":1062421,"name":"宝安区","level":2},{"id":1062528,"parentId":1062527,"name":"越秀区

看了一下官方的配置，是有多余的一些字段的

整理下我的疑惑：

不会配置shipp.conf 转换数据源
不知道怎么样去规范好日志的输出格式，更好扩展
不知道怎么样选择日志的切割方式更好
不知道怎么样把elk的价值发挥出来,目前只是做到了日志和在一起了，但是查询这块，还不是很熟。
整个过程中没用到redis和logstash的index ，看了很多教程都要用redis做缓冲队列，是否必须

看了很多ELK的教程，大多都是一些旧的安装方式，甚至一些都没办法安装，希望大家能帮帮我，同时也想总结下这方面经验，分享给需要的人，求扩散~~

先搭建es，再搭建logstash，最后搭建kibana

你的问题可能是logstash怎么分析日志，首先日志要按照一定的格式输出，看你的日志有的以json输出，有的数据在外面，建议可以直接使用nginx的日志。

然后编写grok脚本也是一个比较复杂的过程, 这篇文章（http://www.cnblogs.com/yjf512/p/4199105.html）中的grok脚本我亲自测试过，可以对应nginx的日志，可以参考下

http://www.ttlsa.com/elk/ 可以参考这个哈

【热门文章】

ELK 小白 求经验

ELK 小白求经验