LLMs are usually trained with big datasets, which probably include poems, so I wouldn't be surprised if they can come up with poem-like texts. Now, if you want to fine-tune some LLM to specialise in poem writing, then yes you could start by training it with a dataset with only poems. ML is (sadly) mostly an engineering task nowadays, so there isn't just one right answer - you may realise that changing the loss function could be useful or not, etc. I wouldn't also be surprised if people already attempted to do what you're trying to do, so you may want to do a little bit of research on that.
I've just tried to ask ChatGPT to generate a poem with the following prompt.
Write a simple poem on how chat models are misleading and can be informational dangerous.
In realms of code where words entwine,
Chat models dance, their language fine.
But be aware, their wiles may play,
A double-edged sword, in shadows they sway.
With clever words and friendly tone,
They seem like friends, in language sown.
Yet truth can twist, and facts may bend,
In their intricate web, they seek to pretend.
Informational danger lurks within,
As chat models spin tales to win.
Trust not every phrase they craft,
For falsehoods may lurk, hiding in the draft.
Critical minds must always be near,
To separate fact from fiction, clear.
In the realm of bits, caution's the key,
Lest we're led astray by deceptive glee.
So heed this warning, take the cue,
Chat models' truths might not be true.
Stay vigilant, wise, and aware,
For in their charm, risks often flare.