TL;DR Here is a beautiful explanation with diagrams: source.
To address:
the cell state is essentially long term memory embedding (correct me if I'm wrong)
The embedding can be long or short term and it is a vector.
To answer:
Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?
Excerpt from source:
The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”
Formally the forget vector is:
$$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$
So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing:
$$C_{t-1}\odot f_t,$$
where $\odot$ is element-wise multiplication.
Thus we see that the forget gate really is acting like a gate: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.
To answer:
How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?
This concatenation takes into account the "rolling" hidden state and the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.
In summary, the forget gate has the primary purpose of forming a vector of values between zero and one. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.
Additional Intuition - More Mathematical
Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of $W_f$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $W_f$ form a sort of "basis" for a "toggle space."
When we concatenate to form $[h_{t-1},x_t]$ and then multiply we see that the term $W_f\cdot[h_{t-1},x_t]$ gives consideration to the hidden state and the input. That is, both $h_{t-1}$ and $x_t$ have there hand in toggling the gate.
The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example.
That is, $W_f\cdot[h_{t-1},x_t]$ will be a vector in the column space of $W_f$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $W_f$. This "toggle" is then transformed into a "gate" in the "gate space."
So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $[h_{t-1},x_t]$. Formally:
$$f_t:[h_{t-1},x_t]\mapsto g_t$$
That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.
Important Note
The LSTM is only one of many types of forget-gate mappings. The main takeaway is that the LSTM works empirically for many applications.
That is a forget gate is an instance of a general "gate mapping":
$$f_t:v_t \mapsto g_t,$$
where $v_t$ could be any number of vectors resulting from any number of concatenations.
Conclusion
Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. One implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.
More Mathematical Intuition
Here is another view of what is happening in the LSTM from a dynamic point of view: video.
