# Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?

Is there a way to use python and AI to compare thousands of files and detect the ones containing "unusual" content?

Those files are supposed to have "homogeneous" configuration types with different parameters, but sometimes some of them might contain "illegal" types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.

Here is one of too many bloc of configuration considered "correct": (the meaning doesn't matter)

vlan XXXX
name  XXX_NAME

interface vxlan 1
vxlan vlan XXX vni YYYY

router bgp YYYY
vlan XXXX
rd A.B.C.D:YYYY
route-target both YYYY:YYYY
redistribute learned


Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example

Interface vlan AAAA


Because it is in only a couple of files ("rare") it is more likely that it is not authorized configuration.

I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.

I am wondering if AI could help with this issue?

• It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?
– nbro
Jan 4 at 15:59
• What makes an illegal configuration illegal? How do you detect that? Jan 5 at 7:03
• Hey, what I call "illegal" is lines different from the most frequent ones (correct). I'll complete the question with an example.
– AJN
Jan 5 at 13:08

Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do interface and Interface mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called "keywords" of your configurations (such as vlan), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.
Then write rules to replace "arbitrary" data (numbers, ips, ...) with a common tag (for example <NUMBER>, <IP>, ...). You should replace all non-keyword words (aka. tokens) with these tags. Use <?> for tokens you don't identify with a regex, or leave them as-is.