Would you be okay with certain people who knows where you hide a spare emergency door key such as a housekeeper, your child's bestfriend or boyfriend/girlfriend, house sitter, etc, letting themselves into your house without your knowledge beforehand?
Two immediate relatives that I trust implicitly have access to my place and can come and go as they please. That's it.