|
| 1 | +# ngx-kasha |
| 2 | + |
| 3 | + |
| 4 | +nginx module for advanced per location logging - aka kasha (🍲) |
| 5 | + |
| 6 | +## Description |
| 7 | + |
| 8 | +This module adds to nginx the ability of advanced JSON logging of HTTP requests per location. |
| 9 | +It's possible to log to a destination ouput any request made to a specific nginx location. |
| 10 | +The output format is configurable. |
| 11 | + |
| 12 | +### Configuration |
| 13 | + |
| 14 | +Each logging configuration is based on a kasha_recipe. (🍲) |
| 15 | + |
| 16 | +A kasha recipe is a ';' separated list of items to include in the logging preparation. |
| 17 | + |
| 18 | +The left hand side part of item will be the JSON Path for the variable name |
| 19 | +The left hand side part can be prefixed with 's:', 'i:' or 'r:', so the JSON encoding type can be controlled. |
| 20 | + |
| 21 | +* 's:' - JSON string ( default ) |
| 22 | +* 'i:' - JSON integer |
| 23 | +* 'r:' - JSON real |
| 24 | + |
| 25 | + |
| 26 | +The right hand side will be the variable's name or literal value. |
| 27 | +For this, known or previously setted variables, can be used by using the '$' before name. |
| 28 | + |
| 29 | +Common HTTP nginx builtin variables like $uri, or any other variable set by other handler modules can be used. |
| 30 | + |
| 31 | +The output is sent to the location specified by the first kasha_recipe argument. |
| 32 | +The possible output locations are: |
| 33 | + |
| 34 | +* "file:" - The logging location will be a local filesystem file. |
| 35 | +* "kafka:" - The logging location will be a Kafka topic. |
| 36 | + |
| 37 | +#### Example Configuration |
| 38 | + |
| 39 | + |
| 40 | +##### A simple configuration example |
| 41 | + |
| 42 | +```yaml |
| 43 | + kasha_recipe file:/tmp/log ' |
| 44 | + src.ip $remote_addr; |
| 45 | + src.port $remote_port; |
| 46 | + dst.ip $server_addr; |
| 47 | + dst.port $server_port; |
| 48 | + _date $time_iso8601; |
| 49 | + r:_real 1.1; |
| 50 | + i:_int 2016; |
| 51 | + i:_status $status; |
| 52 | + _literal root; |
| 53 | + comm.proto http; |
| 54 | + comm.http.method $request_method; |
| 55 | + comm.http.path $uri; |
| 56 | + comm.http.host $host; |
| 57 | + comm.http.server_name $server_name; |
| 58 | + '; |
| 59 | +``` |
| 60 | +
|
| 61 | +This will produce the following JSON line to '/tmp/log' file . |
| 62 | +To ease reading, it's shown here formatted with newlines. |
| 63 | + |
| 64 | +```json |
| 65 | +{ |
| 66 | + "_date": "2016-12-11T18:06:54+00:00", |
| 67 | + "_int": 2016, |
| 68 | + "_literal": "root", |
| 69 | + "_real": 1.1, |
| 70 | + "_status": 200, |
| 71 | + "comm": { |
| 72 | + "http": { |
| 73 | + "host": "localhost", |
| 74 | + "method": "HEAD", |
| 75 | + "path": "/index.html", |
| 76 | + "server_name": "localhost" |
| 77 | + }, |
| 78 | + "proto": "http" |
| 79 | + }, |
| 80 | + "dst": { |
| 81 | + "ip": "127.0.0.1", |
| 82 | + "port": "80" |
| 83 | + }, |
| 84 | + "src": { |
| 85 | + "ip": "127.0.0.1", |
| 86 | + "port": "52136" |
| 87 | + } |
| 88 | +} |
| 89 | +``` |
| 90 | + |
| 91 | +##### A example using perl handler variables. |
| 92 | + |
| 93 | +```yaml |
| 94 | + perl_set $bar ' |
| 95 | + sub { |
| 96 | + my $r = shift; |
| 97 | + my $uri = $r->uri; |
| 98 | +
|
| 99 | + return "yes" if $uri =~ /^\/bar/; |
| 100 | + return ""; |
| 101 | + }'; |
| 102 | +
|
| 103 | + kasha_recipe file:/tmp/log ' |
| 104 | + comm.http.server_name $server_name; |
| 105 | + perl.bar $bar; |
| 106 | + '; |
| 107 | + ``` |
| 108 | + |
| 109 | + A request sent to **/bar** . |
| 110 | + |
| 111 | + ```json |
| 112 | +{ |
| 113 | +
|
| 114 | + "comm": { |
| 115 | + "http": { |
| 116 | + "server_name": "localhost" |
| 117 | + } |
| 118 | + }, |
| 119 | + "perl": { |
| 120 | + "bar": "yes" |
| 121 | + } |
| 122 | +} |
| 123 | +``` |
| 124 | + |
| 125 | +### Directives |
| 126 | + |
| 127 | +--- |
| 128 | +* Syntax: **kasha_recipe** _location_ { _recipe_ }; |
| 129 | +* Default: — |
| 130 | +* Context: http location |
| 131 | + |
| 132 | +###### _location_ ###### |
| 133 | + |
| 134 | +Specifies the location for the output... |
| 135 | + |
| 136 | +The output location type should be prefixed with supported location types. ( **file:** or **kafka:** ) |
| 137 | + |
| 138 | +For a **file:** type the value part will be a local file name. e.g. **file:**/tmp/log |
| 139 | + |
| 140 | +For a **kafka:** type the value part will be the topic name. e.g. **kafka:** topic |
| 141 | + |
| 142 | +The kafka output only happens if a list of brokers is defined by **kasha_kafka_brokers** directive. |
| 143 | + |
| 144 | +###### _recipe_ ###### |
| 145 | + |
| 146 | +See details above. |
| 147 | + |
| 148 | +--- |
| 149 | + |
| 150 | +* Syntax: **"kasha_kafka_partition** _compression_codec_; |
| 151 | +* Default: RD_KAFKA_PARTITION_UA |
| 152 | +* Context: http local |
| 153 | + |
| 154 | +--- |
| 155 | + |
| 156 | +* Syntax: **kasha_kafka_brokers** list of brokers separated by spaces; |
| 157 | +* Default: — |
| 158 | +* Context: http main |
| 159 | + |
| 160 | +--- |
| 161 | + |
| 162 | +* Syntax: **kasha_kafka_client_id** _id_; |
| 163 | +* Default: kasha |
| 164 | +* Context: http main |
| 165 | + |
| 166 | +--- |
| 167 | + |
| 168 | +* Syntax: **"kasha_kafka_compression** _compression_codec_; |
| 169 | +* Default: snappy |
| 170 | +* Context: http main |
| 171 | + |
| 172 | +--- |
| 173 | + |
| 174 | +* Syntax: **"kasha_kafka_log_level** _numeric_log_level_; |
| 175 | +* Default: 6 |
| 176 | +* Context: http main |
| 177 | + |
| 178 | +--- |
| 179 | + |
| 180 | +* Syntax: **"kasha_kafka_max_retries** _numeric_; |
| 181 | +* Default: 0 |
| 182 | +* Context: http main |
| 183 | + |
| 184 | +--- |
| 185 | + |
| 186 | +* Syntax: **"kasha_kafka_buffer_max_messages** _numeric_; |
| 187 | +* Default: 100000 |
| 188 | +* Context: http main |
| 189 | + |
| 190 | +--- |
| 191 | + |
| 192 | +* Syntax: **"kasha_kafka_backoff_ms** _numeric_; |
| 193 | +* Default: 10 |
| 194 | +* Context: http main |
| 195 | + |
| 196 | + |
| 197 | + |
| 198 | +### Build |
| 199 | + |
| 200 | +#### Dependencies |
| 201 | + |
| 202 | +* [libjansson](http://www.digip.org/jansson/) |
| 203 | +* [librdkafka](https://github.com/edenhill/librdkafka) |
| 204 | + |
| 205 | +For Ubuntu or Debian install development packages. |
| 206 | + |
| 207 | +```bash |
| 208 | +$ sudo apt-get install libjansson-dev librdkafka-dev |
| 209 | +
|
| 210 | +``` |
| 211 | + |
| 212 | +Build as a common nginx module. |
| 213 | + |
| 214 | +```bash |
| 215 | +$ ./configure --add-module=/build/ngx-kasha |
| 216 | +$ make && make install |
| 217 | +
|
| 218 | +``` |
| 219 | + |
| 220 | + |
| 221 | + |
| 222 | +### Tests and Fair Warning |
| 223 | + |
| 224 | +**THIS IS NOT PRODUCTION** ready. |
| 225 | + |
| 226 | +This was done over the weekend as a proof of concept, and it also lacks unit tests. |
| 227 | + |
| 228 | +So there's no guarantee of success. It most probably blow up when running in real life scenarios. |
| 229 | + |
0 commit comments