I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using
x = (state, action) features as an input of a neural network, where the
state is the current webpage fetched and the
action are the out-links of the current page. That is,
x = (state, action) will concatenate the
state and the
action features. The priority queue will use as a metric the Q-value of each out-link.
My main idea is to use a neural network that will use
x = (state, action) as the input and produce for this action the equivalent Q-value. However, in order to create this feature space, I reckon using both word2vec and tf-idf.
For each state, I will use information about this page (text) and its parent pages (relevance frequency and other stats). For each action, I will use information about its anchor text and/or URL text.
- I know that in word2vec each word has two-word representations. Which one should I use, the one that describes it as a center word?
- Should I use TF-IDF or word2vec?
- Can I combine them?
- Can I use an LSTM-RNN with word2vec words as input to output document features?