I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using x = (state, action) features as an input of a neural network, where the state is the current webpage fetched and the action are the out-links of the current page. That is, x = (state, action) will concatenate the state and the action features. The priority queue will use as a metric the Q-value of each out-link.

My main idea is to use a neural network that will use x = (state, action) as the input and produce for this action the equivalent Q-value. However, in order to create this feature space, I reckon using both word2vec and tf-idf.

For each state, I will use information about this page (text) and its parent pages (relevance frequency and other stats). For each action, I will use information about its anchor text and/or URL text.

  1. I know that in word2vec each word has two-word representations. Which one should I use, the one that describes it as a center word?
  2. Should I use TF-IDF or word2vec?
  3. Can I combine them?
  4. Can I use an LSTM-RNN with word2vec words as input to output document features?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.