As far as finding players go....play venues away from your house and identify players that you like hanging out with. Eventually invite them to play at your house.
I started with neighborhood friends, and then "selectively" added players from other groups that I played with at other venues. I say selectively because we have never had much luck bringing in players that enjoy simply playing poker, but don't necessarily like hanging out with others in the group. Playing with us is as much (or more) of a social outing as it is a poker game. Joining the group and becoming a regular requires a 100% yes vote from all other members.
It sounds like you've tried a lot of things, but the thing that I find attracts the most participants is VARIETY. Kicking off with a small tournament (we usually play $20 buy-in with a $5 bounty), and then following that with cash is always popular. We play purely no limit hold-em, and I have had zero luck introducing new games to the mix. But that's okay, we all love hold-em, and that's what the group is comfortable with.
I also find that playing various types of hold-em tournaments is very popular.
- Alpha Shootout - this is a monthly tournament consisting of 12 games total. The winner each month gets 10 points, 2nd gets 9, etc. We keep a cumulative score throughout the year and winner (highest 12 game cumulative score) gets a $10k entry in the WSOP Main Event + $1400 in expenses. The buy-in is $150 per month, 10 players. We pay out $550 to the top 3 each month, and set $950 aside in the bank to cover the entry fee.
- High Roller Tournament - $100 buy-in, $20 bounty. This one is especially popular because it is a deep stack tournament (400k to 500k starting stacks). Payout is 50% to 1st, 30% to 2nd, 20% to 3rd.
- Low Roller Tournament - $20 buy-in, $5 bounty. Payout is 50% to 1st, 30% to 2nd, 20% to 3rd.
- Cash Series Tournament - From Labor Day through Super Bowl Sunday we play a cash game tournament every Friday and keep a running score.
- Scoring is based on your winning and losings each week. If you buy in for $100 and cash out for $200, you get 100 points. If you lose your $100 buy-in, you get -100 points. Allowing negative points keeps people interested that can't play every week.
- The 1st $20 buy in requires a player to contribute $3 to the pool. Every $20 worth of buy in after that requires $1 into the pool.
- At the end of the season, we pay out the top 4 players in the series.
Every tournament is followed by a no limit hold em cash game. The mix of tournament and cash in the same night appeals to a wider variety of players.
I would also "honestly" assess why you think players don't come back. Do they not like particular players? Maybe the game is too serious, or not serious enough? Our game is NOT serious. We have a number of drinkers, and a number of serious players. I let everybody know right up front that this is a "Social Club", and we just happen to play poker
Nothing worse than bringing in a new player that is annoyed by the loud, drunk, stupidly aggressive players.
Good luck, and don't forget...be very selective about the players you invite over. If you wouldn't enjoy hanging out with them away from poker, then probably not a good fit.