Advice needed on a monster concept. Would this be fun for the party?

I've been sitting on a monster mechanic for a while now and I can't shake the feeling it's a horrible idea. So I would like some of your thoughts and suggestions.

An aberrant spellcaster has somehow successfully cursed all knowledge about itself: It's strength, it's weaknesses, it's manner of spellcasting and even it's favourite colour. Every single bit of knowledge a person acquires about this creature will reduce all their bonuses by 1 when they try to interact with it. So imagine character ACs get penalties, Save DCs are reduced as well as attack bonuses and saves. These stats are reduced with a hard limit on whatever your bonus is from your attributes. So all the things a character excels at is slowly whittled down while aspects to a character they don't excel in are unaffected. So the soft limit on bonus reduction is 5 unless someone has expertise in a skill which sets the limit to 10. Proficiency bonuses are unaffected.

There's 2 things I want to do with this creature to make sure it's playable.

1. I don't want this creature to be much stronger than any individual party member. I will probably take the beholder stat block as a base, remove all the eye stalks and give it very crowd control / disruptive spellcasting such as phantasmal force, mislead (contingency), dimension door, wall of force and such. It's basic attack on a hit will knock enemies prone to make it harder to be approached.

2. In the adventure the adventurers will come upon a goblin servant of the creature who has had its skull removed rather crudely. It's been cursed to forever remember everything it has learned which means it's brain was outgrowing the poor goblins head. The goblin is friendly to the party and tells them everything they wish to know about the creature. Here's the gist of the adventure, this conversation never happens at the table. At the end of the conversation the goblin uses alter memory to wipe all the knowledge about this creature from everyone's mind and instead gives them a sealed scroll containing the entire transcript of their conversation with the goblin. I'm assuming the party will want at least some answers to what's going on with the memory wipe so the goblin will freely tell them anything they want in a one on one conversation and I'm counting on the fact the first person who learns the mechanics of this monster will prevent the rest of the party from asking the goblin any more questions.

So here's my main question. Would you see this work in a short adventure or is this doomed to fail?

For anyone curious about the backstory of this monster I had a concept of a Beholder which was enamoured about the prowess of an Archmage it battled against and it accidentally dreamed a spellcasting, 'humanoid' looking beholder into reality. The combined paranoia of a Beholder mixed with an Archmage had the creature curse all knowledge about itself so it could not be tracked, fought or studied without major consequences. It looks like a figure in a long robe with a large witch's hat but has the unmistakable visage of a beholder hidden under the oversized hat.