0
$\begingroup$

I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using x = (state, action) features as an input of a neural network, where the state is the current webpage fetched and the action are the out-links of the current page. That is, x = (state, action) will concatenate the state and the action features. The priority queue will use as a metric the Q-value of each out-link.

My main idea is to use a neural network that will use x = (state, action) as the input and produce for this action the equivalent Q-value. However, in order to create this feature space, I reckon using both word2vec and tf-idf.

For each state, I will use information about this page (text) and its parent pages (relevance frequency and other stats). For each action, I will use information about its anchor text and/or URL text.

  1. I know that in word2vec each word has two-word representations. Which one should I use, the one that describes it as a center word?
  2. Should I use TF-IDF or word2vec?
  3. Can I combine them?
  4. Can I use an LSTM-RNN with word2vec words as input to output document features?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.