[BlueOnyx:15259] Re: AOL and DKIM, SPF & Re: SSL change after updates?

[Michael Stauber]

Hi Eric, hi Matt,

I'm combining my reply to your two separate topics regarding wildcards
and SPF. It's a slightly longer rant. :p

SPF:
====

Let me start with SPF, because it's a bit easier to answer. As you know,
SPF records are simply TXT DNS records that are "tacked on". If you know
the right Syntax, you can already go ahead and can use SPF records on
5106R/5107R/5108R as is. Just create a TXT record with the proper
payload via the GUI and you will be fine.

Pages such as http://www.spfwizard.net/ sure are a great help finding
out how the Syntax for your SPF record(s) should look.

However, this usually overlooks some fundamental stuff. What about
emails from localhost? Which would mean system generated emails, mails
from certain scripts and so on. So you *will* need to think your SPF
record layout through thoroughly to really match all usage cases.

Sure, having pages in the BlueOnyx GUI that *assist* with creating
proper SPF records would be nice. But such a wizard is quite complex and
at the end of the day there will be usage cases that it won't and can't
cover. That's where your own brain juice is again needed.

If I wanted to do such a wizard, it would take me four weeks. Minimum.
And once I release it, people will come forward and point out all the
juicy usage cases where it doesn't work right. So we're looking at
another two or three weeks of tweaking it beyond that.

I'm like *THIS* close (imagine me holding thumb and index finger 3mm
apart) of having the new BlueOnyx 520XR GUI done. I *really* don't want
to delay the release (or complicate it unnecessarily) with an SPF
implementation that will be riddled with bugs and limited in capability.

Asking me to implement that now is inviting failure. The new GUI will
already have sufficient bugs and deficiencies that need to be weeded out
through thorough testing before it can be considered stable.

So long reply for a short answer: Yes, one of these days BlueOnyx will
have a basic Wizard to do SPF records. In the meantime (and until that
happens) you can create SPF records by doing normal TXT records in the
DNS GUI.

Wildcards for hostnames:
========================

I just shot my development box for 520XR six ways 'til Sunday *trying*
to get this work. I'll actually have to restore it from a backup before
I can continue development. Aside from loosing time with the restore,
I'll also have to merge changes back into the backup that I did since
the last backup run. Fun and games - half a day wasted.

Let me explain some of the basics of Sausalito here. Not much, but
little enough to understand the problem I'm facing there:

CODB database fields can only take data that matches a certain scheme.
We can define these types of data that a field can take with regular
expressions.

For example:

username:

That field only takes alphanumerical characters plus a limited subset of
other characters. Such as ".", "_" and "-". That's defined in a schema
file like this:

<typedef
	name="alphanum_plus"
	type="re"
	data="^[A-Za-z0-9\\._-]+$"
/>

You can see, the regular expression for this is: ^[A-Za-z0-9\\._-]+$

In a CODB Schema file for a data-field that should contain the username,
we can then simply define that field like this:

    <property
        name="username" type="alphanum_plus" default=""
        writeacl="ruleCapable(modifyUserDefaults)"
    />

And CODB will then make sure that only data is stored into that field
that matched the regular expression we defined for "alphanum_plus".
Likewise the new GUI will also check input with this *while* you type it
into the respective form field. If the entered data doesn't match
validating against "alphanum_plus", you get an error message.

Which is all nice and comfy. It's a real time saver when coding GUI
pages, as you don't have to worry about the user submitting data that
doesn't belong into the respective fields.

Now let me come to "hostname", "domainname" and "fqdn". They are also
defined in Schema files and the regular expressions for that are already
horribly complex.

So I *can* tweak the "hostname" field to accept a wildcard instead.

At the moment the definition for "hostname" looks like this:

<typedef name="hostname"        type="re"
  data="^[A-Za-z0-9][A-Za-z0-9\-]*(\\.[A-Za-z0-9][A-Za-z0-9\-]*)*$"
/>

If I change it to also accept a wildcard, it'll look like this:

<typedef name="hostname" 	type="re"

data="(^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$)|(^\\*$)"
/>

Yes, that is a hell of a lot more complicated, but it does the trick. It
probably can be simplified a bit further, but ... I'm in a hurry.

Ok, so far so good. Now *all* "hostname" fields in *each* and *every*
part of the GUI accept wildcards. Let us think for a moment if that's a
smart idea. Where are we using these?

- Vsite Add: We can create Vsites with a hostname of "*".
- Vsite Mod: We can change the hostname of Vsites to "*".
- DNS Management: We can create DNS records with "*" for the hostname.
- Email aliases: Email server aliases with "*" for hostname.
- Web aliases: Web server aliases with "*" for hostname.
- SSL certificates: Can now contain a wildcard in hostname, too.

So with DNS we're good. We want wildcard DNS. We might also want
wildcard web server aliases.

What we do NOT want is email server aliases *and* Vsites themselves that
have a wildcard in their name.

Which now immediately creates a problem: ALL our "hostname" input fields
in the entire GUI now accept wildcards. Because every GUI page and every
Schema file that deals with hostnames is now hot-wired to accept
wildcards. Even for the parts where we don't want them.

In a lot of places the FQDN is also assembled out of "hostname" and
"domainname", which have separate definitions. And FQDN has a separate
definition of what data is allowed and which not. So if the hostname
*has* a wildcard is merged with the domainname into an FQDN, then the
storage of the data will fail. Because FQDN can't take wildcards in the
hostname part according to our CODB data-type definitions.

But hey, it gets worse: We have Perl handlers which update the email and
webserver configuration. Also proftpd.conf. If I feed those Perl
handlers wildcard hostnames, they blow up. Because they're not designed
to have a wildcard in the data. Wildcards, which could or could not be
(depends on the context) be interpreted as a regular expression itself.

Just three examples:

a.) FQDN contains a wildcard:

Handler base_unique.pl blows up while verifying if another site exists
that already might contain that name. The handler to create the email
server mapfiles triggers after it and blows up, too. Instant hang up of
CODB and further transactions via the GUI aren't possible until the
haning processes are killed and CCEd is restarted.

b.) Email server alias contains a wildcard:

See above. One of the handlers for creating the email server mapfiles
doesn't handle this well.

c.) Web server alias contains a wildcard:

Handlers work fine and the changes go through as intended. Except that
Apache won't restart because the /etc/httpd/conf/vhosts/site* file now
contains a syntax error:

# owned by VirtualHost
NameVirtualHost 208.77.221.197:80

# ServerRoot needs to be set. Otherwise all the vhosts
# need to go in httpd.conf, which could get very large
# since there could be thousands of vhosts:

<VirtualHost 208.77.221.197:80>
ServerName www.basura2.net
ServerAlias basura2.net *.basura2.net
ServerAdmin admin
DocumentRoot /home/.sites/70/site4/web
ErrorDocument 401 /error/401-authorization.html
ErrorDocument 403 /error/403-forbidden.html
ErrorDocument 404 /error/404-file-not-found.html
ErrorDocument 500 /error/500-internal-server-error.html
RewriteEngine on
RewriteCond %{HTTP_HOST}                !^208.77.221.197(:80)?$
RewriteCond %{HTTP_HOST}                !^www.basura2.net(:80)?$ [NC]
RewriteCond %{HTTP_HOST}                !^basura2.net(:80)?$ [NC]
RewriteCond %{HTTP_HOST}                !^*.basura2.net(:80)?$ [NC]
[...]

In the last line above you see the Rewrite-Condition containing the
wildcard. The * in it needs to be escaped like this:

RewriteCond %{HTTP_HOST}                !^\*.basura2.net(:80)?$ [NC]

So that handler needs an extra IF-clause to check web server aliases for
wildcards. And if present, it needs to escape them. Great! \o/

Long story short:

To allow wildcard hostnames we need:

- Updated basetypes.schema with EXTRA definitions for:
	- hostname with and without wildcards
	- FQDN with and without wildcards in the hostname part

- Updated base-dns.mod:
	- DNS.schema needs an update to change the datatype for
	"hostname" to "hostnamePlusWildcard".
	- GUI pages for DNS management (server + Vsite) needs
	changes to change input-validation for "hostname" to
	"hostnamePlusWildcard". That's like 9 GUI pages that
	need changes.

- Updated base-vsite.mod:
	The following GUI pages need input validation for "hostname"
	changed to "hostnamePlusWildcard":
	- "Site Management" / "Web"
	- "Site Management" / "Add Site"
	- "Site Management" / "Site Template"
	- Vsite.schema needs to be updated.
	- Various handlers need to be updated.

- base-ssl.mod:
	Don't get me started on this one. Changes needed?
	- In 5 GUI pages.
	- Probably two schema files
	- Various handlers and constructors
	- PLUS the script that dynamically creates the SSL-containers.

- base-email.mod:
	I'd be damned if I knew. Can't even tell where and where not
	I'd have to make changes. Probably various handlers and
	constructors. Maybe a schema file or two.


My bottom line:
===============

Yesterday I thought I'd be able to finish the new BlueOnyx GUI this
week. With *that* dumped into my lap we're looking at a 4-6 week holdup
plus a lot more potential bugs in critical subsystems.

So I'm offering a compromise here: The new GUI will allow wildcard DNS
records. It will also allow that "web server alias" contains a wildcard
in the hostname (see attached screenshots).

But that's it. Neither FQDN's (for Vsite names) or email server aliases
will take wildcards.

It'll still require a few days of tweaking and testing before I can be
comfortable with it. And this will *NOT* make it into 5106R, 5107R or
5108R. It'll only be in BlueOnyx models with the new GUI (5207R/5208R -
or better).

Good enough?

-- 
With best regards

Michael Stauber
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Bildschirmfoto vom 2014-04-23 15:51:50.png
Type: image/png
Size: 143543 bytes
Desc: not available
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20140423/ba408fa9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Bildschirmfoto vom 2014-04-23 15:51:43.png
Type: image/png
Size: 153151 bytes
Desc: not available
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20140423/ba408fa9/attachment-0001.png>