detecting-attacks-web-applications-log-files：

1. Abstract:

Web application log files allow a detailed analysis of a users actions.

Log files have its limits,though. Web server log files contain only a fraction of the fullHTTP request and response。

Knowing those limits, the majority of attacks can be recognized and acted upon to prevent further exploitation.

2. Introduction:

Why should you bother analyzing log files instead of using a network intrusion detection system?

reasons:

The HTTP traffic may be SSL encrypted (HTTPS);
There may be no NIDS (hard to deploy; another zone of
attack);
High traffic load makes it difficult to analyze network
traffic (in real time);
NIDS are designed to work on the TCP/IP level, and thus
they may not be as effective on the HTTP layer;
IDS evasion techniques (HTTP, encoding,
fragmenting, …).

3. Attacks on Web Applications:

4个趋势：

Bots are being used in more than half the attacks against
web applications; 机器人
Attackers are finding flawed web applications using Google and other search tools; 搜索
Directed attacks are growing more sophisticated 复杂
Attackers operating from bases around the world are
getting better at covering their tracks 隐匿

3.1 Web server log files：

standard web servers like Apache and IIS generate logging
messages by default in the Common Log Format (CLF) specification.
The CLF log file contains a separate line for each HTTP request.
A line is composed of several tokens separated by spaces:

3.2 primer to http:

http (short for Hyper Text Transfer Protocol) **is the language
that web servers and browsers speak**.

Universal Resource Locators or URLs:
request和response例子：

3.2.1 HTTP Evasion Techniques:

There are different types of evasions in different places of the
HTTP protocol
They can occur in the request URI portion of the
HTTP protocol, other parts of the HTTP header, the HTTP body, etc
例子：加/

There could be infinite combinations of those evasions. The goal of these techniques is to evade the detection by an IDS but still get executed by the web application.

normalisation method:

URL-decoding
- %XX
Null byte string termination
self-referencing paths
- /./
- =号编码
path back-references
- /../
- =号编码
Mixed case
Excessive use of whitespace
Comment removal
- DELETE/**/FROM
**Conversion of (Windowssupported) backslash characters
into forward slash characters**
Conversion of IISspecific Unicode encoding
- %uXXYY
Decode HTML entities
- c, ", ª
Escaped characters
- \t, \001, \xAA, \uAABB

4.Detecting Attacks

Layer 3/4 Firewall:
Application layer firewall / web application firewall (WAF
web server:
- The web server is the end device of an HTTP request.
- Standard web servers like Apache and IIS are logging by default in the
  Common Log Format (CLF) specification.
Web server logs do not contain any data sent in the HTTP header, like POST parameters.
The HTTP header can contain valuable data, as most forms and their parameters are submitted by POST requests. This comes as a big deficiency for web server log files

4.1 Rulebased Detection (static rules)

Attacks can be detected by two different strategies: rulebased基于规则
and anomalybased. 基于异常
The rulebased strategy defines static rules which have to be defined before the analysis can be made
Static rules are defined once and stay the same during the detection phase.
They have to be defined and specifically crafted for each application.
Static rules make most sense for p**reknown values** like certain input characters, a fixed length of a parameter or an upper limit of a transfer amount.

Static rules can be further divided into two detection
models: the negative and the positive security model.

4.1.1 Negative Security Model:

The negative security model, or the blacklist approach, has a default policy of allow everything. This means that everything is allowed to pass, or everything is considered “normal”.

The policy (the blacklist, or the rulebase) defines what is not allowed.

劣势：
- The biggest disadvantage is that the detection will **only be
  as good as the policy**. 能力取决于规则
- It has to be adopted to new findings and updated to recognize new attack vectors需要根据新的攻击向量更新规则
好处：
- One of the positive points is that it yields very little falsepositives as the rules will usually look for specific, well known attack strings or behaviour.
- 因为是用一些非常明显的特征来作为规则，所以误报率较小

sql注入中的用concat关键字匹配出现过误报：

4.1.2 Positive Security Model：

原则：
- The default policy here is deny all, the policy will then define what is allowed.
- everything else will be considered as malicious.
设置：
- This whitelist **can be defined automatically in a learning phase **or be manually
  defined.
防火墙基于白名单：
- Firewalls are usually configured this way. The default policy will be deny. For every server/service there has to be a new whitelist entry for this specific machine and port
从安全的角度推荐：
- This model is the preferred way from a security standpoint. False negatives can be reduced to a minimum, while false positives help to improve the whitelist.误报率可以降到最低。

4.2 Anomalybased Detection (dynamic rules)

原理：
- Anomalyrules consist of dynamic rules. As the name implies, those rules are** not static** nor are they manually defined. Instead, the rules are defined through a learning phase
定义正常的流量：
- In this learning phase, good traffic is recorded as “normal“. It is of greatest importance, that this traffic is “clean“ and free of attacks, as this will be used as our baseline
识别异常流量：
- The goal of a learning phase is to define how “normal“, accepted traffic looks like to eventually flag anomalous traffic which does not look like “normal“ and raise an alarm.
- Deviations from this ruleset will be flagged as anomalous traffic.

4.3 Detecting the OWASP Top Ten:

全是正则~
粗略了浏览了一遍、暂时不细看了。。。

参考：

Detecting Attacks on Web Applications from Log Files