I spent some time thinking about it, but I'm aware of only two main meanings. There might be more that aren't coming to me right now though...
In local search problems or sometimes in optimization for machine learning, the word "topology" of a problem is used to refercorresponds to the change in the function you're optimizing as you move between adjacent states. If the change is sharp, you have a "rugged topology". IfIf it's gentle and continuous, you have a "smooth topology". SeeSee on page 2 of An Introduction to Fitness Landscape Analysis and and Cost Models for Local Search for example.
(Note that texture, curvature, and other morphological features lacking a discontinuity are not topology in the canonical mathematical sense. The geological term "topography" may have been a more appropriate choice.)
The other major meaning is with reference to the structure (topology) of a combinatorial graph. Many modern machine learning algorithms are based in the idea of combinatorial graphs, including Bayesian Networks, Sum/Product Networks, and Deep Neural Networks. Here, topology refers to the topological ordering of a directed graph, or more informally, to "how the graph is structured". For example, in a neural network, the depth and width of the network's layers, and the nature of the connections between layers, define the topology of the network.
Additionally, it gets used a lot in the sense of the second meaning in other areas of AI, just because those areas also use graphs. For example, in automated planning, or in probabilistic reasoning, it is also common to represent your problem as a combinatorial graph. You could then talk about the "topology" of the problem.