I'd like to make a giveaway with a specific rule, but I'm not sure if this is allowed...

if its private its ok

I think your best bet would be to do a steam.losers group giveaway with a contribution requirement. That group only allows people who haven't won yet I think. The site doesn't let you directly use that as a criteria so a group giveaway is your best bet for screening people.

You cant change the rules of public giveaways, it isnt allowed.
Make it private and set your own rules or make it for one group with people with 0 wins only

Your best bet is to raise a support ticket to get the conditions pre-approved. That way, if your winner has already won games, you will be able to re-roll.

Bear in mind that it will still need to be a private or group giveaway (no entry conditions are permitted for public giveaways).

If you want to make it available to the maximum number of entrants, you could make it for a group everyone is a member of (ie Summer Camp), but my advice would be not to do this. I tried it a while back, and most of the people who entered were illiterate lobotomites who didn't even read the conditions in the description, then added me on Steam and bitched/threatened me when the giveaway was re-rolled. From memory, probably less than 5% of entrants paid any attention to the conditions, and it needed seven or eight re-rolls before a winner was found.

I'd suggest a private giveaway, advertised in the forums, would be the way to go. At least that way, you avoid the majority of greedy leeches who just hurtle down the list, clicking indiscriminately, in their fervour to hoover up free shit, without bothering to read such trivia as conditions of entry, what game/dlc they are entering for, etc.

... and yes, it's my time of the month and I'm having an especially misanthropic day...