0
$\begingroup$

I have data coming from web access logs in the following form:

@timestamp  ISP cache_result    client_ip   client_request_host client_request_method   client_ua   client_url  client_user content_type    ... http_response_code  major   os  os_name querystring reply_length_bytes  ts_process_time ts_timestamp    type    ua_name
2018-04-17T08:12:32.000Z    cuaerH c rt,nlEIrnii.cec    TCP_REFRESH_MISS    25.204.184.124  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=2.2.3  25204   321 17/Apr/2018:08:12:32 -0000  testdata    Chrome
2018-04-17T08:12:32.000Z    HeE iclirueIc rat,nrncc.    TCP_REFRESH_MISS    8.157.89.174    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-core/js/min/avada-p...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  2825    177 17/Apr/2018:08:12:32 -0000  testdata    Chrome
2018-04-17T08:12:33.000Z    ,rrnI EnH.ceeiuclcicrat TCP_REFRESH_MISS    37.151.22.36    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  267 275 17/Apr/2018:08:12:33 -0000  testdata    Chrome
2018-04-17T08:12:34.000Z    tn.cHer uE,lecnir aircIc    TCP_REFRESH_MISS    202.165.110.43  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  341 172 17/Apr/2018:08:12:34 -0000  testdata    Chrome
2018-04-17T08:12:34.000Z    rneecHuraci ctInir cl.,E    TCP_REFRESH_MISS    174.201.44.32   testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-builder/assets/js/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  302 180 17/Apr/2018:08:12:34 -0000  testdata    Chrome

What I need is to come up with a model that takes as input such kind of data in batches of 2 minutes data and identify if there exists IPs that are anomalous. So, I am thinking to group the training data by IP and compute some features.

But my question is - how is this possible with batches of 2 minutes, since the anomaly detection needs all the data at once to be able to train it?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Do you have a labeled dataset that specifies which connections are anomalous and which are not? You will need that to train the model.

Yes, you will need all of the individual labeled data rows for training, but you could use batch inference for prediction if you only wanted to check 2-minute batches of connections at a time after the model is deployed.

This is assuming that you are treating each connection as independent of the next: each connection is either anomalous or not by itself, sequencing of connections does not matter. If sequencing does matter, you'll want to consider a temporal network architecture using something like LSTM or a transformer. If this is the case, the assumption that the window of activity is 2 minutes is a strong one (why not 3 or 4 minutes)?

Improve this answer
$\endgroup$
1
  • $\begingroup$ I have only normal access logs and not anomalous. The thing is that I need to identify DDoS attacks with this model. So I have to compute features per IP per 2 minutes, because only in this way you can tell if an IP tries to attack the website. So the first step is to compute the features, like how many times the IP appears in a period of 2 minutes and then train the model. If an IP appears 3 times is fine, if it appears 30 times is anomalous, for example. $\endgroup$
    – Kosmylo
    Commented Jul 9, 2022 at 7:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .