Estimating from an observation is a function, but "really counting" is a process. Feed-forward neural networks can learn arbitrary functions from training examples, but they cannot represent (and therefore cannot learn) processes. They can attempt to estimate the results of completing a process as a function, but that is not the same thing as actually performing the process.
To learn arbitrary processes from examples requires a model with some concept of state and evolution over time in addition to any functions that are required. Recurrent neural networks (RNNs) are suitable models for those kinds of learning problems, but so are other AI learning constructs.
This rabbit hole goes deep, and for any model you could build it is possible to ask:
Is the system really performing a counting task similar to how a human might attempt the same task?
Has the sytem really learned to count, or has it been constructed by the developer so that some parts of the process are inevitable?
Has the system learned to count in general within any sub-component, or are the things that it can count tightly coupled across components?
There are other questions you could ask too, depending on your goals for producing such a model.
It is worth noting that for small quantities and certain patterns, that human "counting" does more closely resemble the estimation process, or perhaps is more akin to an NLP problem. For example, consider how you "count" the pips on a die - although you can use a counting process to confirm what you see, typically you do not do so. Instead, some part of your brain is supplying a very accurate answer quickly and unconsciously:
More generally, counting in humans likely consists of multiple related strategies for converting observations into symbolic and/or sensory representations of quantity. Some may be instinct, some are learned conscious behaviours and some appear to be in-between, perhaps originally learned but turned into subconscious skill through repetition.
Ignoring whether or not you are implementing your AI project for counting items in an image wholly within a neural network for now, I think you need the following components:
A state that represents and tracks the start, progress so far and end of the counting process with respect to the input being processed.
An accumulator that represents quantity counted so far.
A detector that can trigger a counting event when it observes something that needs to be counted.
A strategy or planner for processing input so that detection events are separated and only triggered once for each valid event.
The last component can vary a lot, and humans will use a range of different strategies for counting depending on the difficulty of the task. For instance, you might use working memory (a limited resource in humans) to track a few key points in an image to help segment an image into smaller sections as you work across it. Or you might make visual marks on an image to track each object that had already been counted. For a human, counting strategies can be mixed and matched, and switched between sometimes even during the same counting task.
All of these components could be represented in learnable parts of an AI process, but they do not have to be. When you suggest in your question that
I don't want to use a detector, then count the number of bounding boxes.
then you are most likely saying that you don't want to use a fixed hard-coded strategy. You want to somehow create a neural network that can discover at least one strategy for processing the input.
The trouble you will face is that giving a large RNN model (e.g. LSTM) a dataset of examples with input images and output correct counts will likely be too much of a challenge. Discovery of a robust object counting system from scratch will be too hard.
There are a few things that may help you construct something that "really" learns to count. Here are a couple of ideas:
Curriculum learning, where you start training the neural network with some hand-holding examples, and slowly ramp up the complexity. This is analogous to how we teach children to count.
Designed-in modelling for processing strategy. For instance, you could add a virtual fovea and artificial visual saccades and require that the neural network output a sequence of locations within the image that it picks out to run the detector against. This will be a constraint that allows certain types of human-like counting strategy to work, and could simplify the problem to the point that the network has a chance to learn it.
A paper that uses a "fovea" model for object detection: Object detection through search with a foveated visual system
