For example, the "dungeon" might be a ruined fortress that is now inhabited by goblins and also has a labyrinth of caves below, and a hidden ancient shrine in the sub-basement. This is actually pretty realistic: the Basilica of San Clemente in Rome has no fewer than four layers: at ground level, it's a 12th century church. Go into the basement, and you find a 4th century church that the latter one was built on top of. Another stairs down, and you find the remains of a villa, rebuilt from a mint that was destroyed in a fire in 64, and finally below it all: a labyrinth of tunnels containing a secret cult site of the god Mithras, and a spring that was likely itself a cult site before the mint was even built. It's a site that's been in use - first as a cult location, then as a mint, then as a secret meeting place for Mithras worshippers, then as a church, and finally as another church - for more than two millennia.
The only real difference to an RPG dungeon is that magic isn't real, and you're hence unlikely to wake some slumbering ancient creature in the sub-sub basement.
So if we want to make a really cool dungeon, we want to create it iteratively: layer by layer, both in time and space. There's a bunch of stuff out there that does this or something like it: Dwarf Fortress probably comes closest, with its detailed simulation of history, and ability to abandon, reclaim, and explore fortresses. A pen and paper approach is given by How to Host a Dungeon, a slightly idiosyncratic system for generating a dungeon by simulating a number of "ages". Finally, there's my own Spacegen, which is a sci-fi take on the topic, with planets and aliens instead of rooms and monsters.
Still, these are all kind of limited, so here's my proposed design for a dungeon generator.
The map of the dungeon - or dungeon site, really - is stored as a 3D grid much in the same way as Dwarf Fortress does it. Each grid cube is big enough for a human being to stand in, and can be filled with earth, or rock, or be empty and contain furniture or items or doors, etc.
At the start of the process, a pristine landscape is generated: trees, grass, maybe a river or lake, or some hills. As time passes, someone turns up and starts building or digging, starting some sort of settlement. Eventually, things change: the settlement gets invaded, abandoned, or transformed in some way. The new inhabitants do things differently, or the lack of inhabitants causes a slow decay.
This process repeats, with different inhabitants creating layers of history, repurposing old structures and building new ones upon them. Different inhabitants have different behaviors: Humans build towns and cities with walls, use shallow tunnels, and produce a fair amount of garbage, which in time may raise the street level. Dwarves would dig deep tunnels, mostly ignore surface constructions, and produce produce great stone heaps, again partially burying surface buildings. Elves, on the other hand, would ignore tunnels and build loose cities amongst the trees they plant.
Other inhabitants would be more unpleasant: a necromancer could take up residence building labs and storage rooms, and creating a great tower to rule from. Simple bandits might use ruins as a hideout, not building anything but using intact rooms to to sleep and store their loot, setting up traps to protect themselves. Or the place gets infested by Goblins, who build ramshackle constructions everywhere, repurposing and reshaping with wild abandon, and producing mounds of garbage.
Mixing up this process are disasters, guaranteeing that any inhabitant eventually fails and is replaced by another or a period of no habitation.
As inhabitants gets richer, they become a target for attacks by raiders. As they get more advanced, they eventually do something stupid like creating a servant race, opening portals to other dimensions, trying to bind demons to their will, create powerful magical weapons, or breed monsters. And if a society sticks around for too long, it tends to become more unequal, leading to revolts.
If all else fails, there natural disasters. Earthquakes, meteors, fissures in the earth, and floods are great as they can radically change the layout of the place, making areas inaccessible or newly accessible, perhaps revealing that a peaceful human town has been built atop a giant dwarf-carved labyrinth inhabited by flesh-eating mutants.
Central to the way the generator works is building creation and reuse. A building consists of a set of rooms, and has certain requirements in terms of room size and properties. Any cuboid surrounded by walls can count as a room. When an inhabitant requires a new building, most races first check if there is an empty set of rooms that could be used - also allowing for walls to be knocked down or rooms to be subdivided. If a suitable place is found, it may be cleaned up somewhat, with the previous contents ending up in a garbage heap somewhere, and the relevant furniture and items are installed.
Room-ness is on a sliding scale too. A building with a hole in the roof can be patched up if needed, and goblins in particular are happy to treat the space between two buildings as "a room" that just happens to lack a front and back wall and a ceiling. The addition of some ramshackle walls is sufficient to make it into a valid part of a building. Elves, on the other hand, don't like non-elvish places and are likely to always build new buildings.
Inhabitants are not simulated at the individual level: basically, an inhabiting society - whether a necromancer and his minions or a human town - have a set of buildings they use, and a set of scores like population, wealth and sophistication. As time goes on, they will create new buildings as they can afford them and need them.
Buildings can also be abandoned if they are no longer needed due to population decrease or a shift in behavior, and abandoned buildings will slowly decay.
Finally, conflicts need to be modeled: if there are multiple inhabitant societies, or just one and a bunch of monsters or demons, the fight between them should have a big impact on the shape of the place.
In a conflict, each side has a territory roughly consisting of the buildings it's using. Boundaries are likely to be fortified or trapped. Fighting happens at a basic numerical level: a die roll and some strength values. As a result, territory will shift, and remains from the battles may be placed as items: corpses, equipment, and so on.
All of this produces a 3D dungeon that feels real and has real hazards: difficult, labyrinthine terrain, leftover traps, monsters, treasure, rotten floors, crumbling walls, bricked-up doorways, mysterious dwarven mechanisms. And since it's all at the right granularity, you are at liberty to attack the dungeon in whatever way you wish: hack your way through the monsters, break open doors, rappel down shafts, or scramble over crumbling stonework.
I have to stop myself from coding this for now. I have no game to go with it, and should investigate DF adventurer mode first and see to what degree I just replicated what it is doing anyway. But at some point, this could be a fun thing to put together.