Strange Validation Scenario, Can't Solve. Any ideas?

I have a web form that needs to validate address field in Sweden. But in Sweden, a lot of addresses don't have addresses so I cannot use RequiredFieldValidator.

I am still required to make this active, to at least give the user an opportunity to type in an address IF they had one.

I then tried a CustomValidator but by design, the CustomValidator event will not fire if the field is left blank so there goes that idea...

I'm kinda stuck at this point. Anyone has any ideas? Changing the business logic is not going to happen.

To recap, form field needs to validate if left blank. However, we are accepting 'blank spaces' as a substitute if no address is available. Oh, and this needs to show up in the validationsummary which I'm using for the rest of the form fields that are legitimately using requiredfieldvalidator.

Thanks!