VA: Simplified HTTP-01 w/ IP address URLs (#3939)

Continued bugs from the custom dialer approach used by the VA for HTTP-01 (most recently #3889) motivated a rewrite.

Instead of using a custom dialer to be able to control DNS resolution for HTTP validation requests we can construct URLs for the IP addresses we resolve and overload the Host header. This avoids having to do address resolution within the dialer and eliminates the complexity of the dialer `addrInfoChan`. The only thing left for our custom dialer now is to shave some time off of the provided context to help us discern timeouts before/after connect.

The existing IP preference & fallback behaviour is preserved: e.g. if a host has both IPv6 and IPv4 addresses we connect to the first IPv6 address. If there is a network error connecting to that address (e.g. an error during "dial"), we try once more with the first IPv4 address. No other retries are done. Matching existing behaviour no fallback is done for HTTP level failures on an IPv6 address (e.g. mismatched webroots, redirect loops, etc). A new Prometheus counter "http01_fallbacks" is used to keep track of the number of fallbacks performed.

As a result of moving the layer at which the retry happens a fallback like described above will now produce two validation records: one for the initial IPv6 connection, and one for the IPv4 connection. Neither will have the "addressesTried" field populated, just "addressesResolved" and "addressUsed". Previously with the dialer doing the retry we would have created just one validation record with an IPv4 "addressUsed" field and both an IPv6 and IPv4 address in the "addressesTried" field.

Because this is a big diff for a key part of the VA the new code is gated by the `SimplifiedVAHTTP` feature flag.

Resolves #3889

Author: cpu

features/featureflag_string.go

@@ -38,6 +38,9 @@ const (
 	ACME13KeyRollover
 	// ProbeCTLogs enables HTTP probes to CT logs from the publisher
 	ProbeCTLogs
+	// SimplifiedVAHTTP enables the simplified VA http-01 rewrite that doesn't use
+	// a custom dialer.
+	SimplifiedVAHTTP
 )
 
 // List of features and their default value, protected by fMu
@@ -62,6 +65,7 @@ var features = map[FeatureFlag]bool{
 	CAAAccountURI:               false,
 	ACME13KeyRollover:           false,
 	ProbeCTLogs:                 false,
+	SimplifiedVAHTTP:            false,
 }
 
 var fMu = new(sync.RWMutex)

features/features.go

@@ -30,7 +30,8 @@
       "RPCHeadroom": true,
       "IPv6First": true,
       "CAAValidationMethods": true,
-      "CAAAccountURI": true
+      "CAAAccountURI": true,
+      "SimplifiedVAHTTP": true
     },
     "accountURIPrefixes": [
       "http://boulder:4000/acme/reg/"

test/config-next/va-remote-a.json

@@ -30,7 +30,8 @@
       "RPCHeadroom": true,
       "IPv6First": true,
       "CAAValidationMethods": true,
-      "CAAAccountURI": true
+      "CAAAccountURI": true,
+      "SimplifiedVAHTTP": true
     },
     "accountURIPrefixes": [
       "http://boulder:4000/acme/reg/"

test/config-next/va-remote-b.json

@@ -36,7 +36,8 @@
       "VAChecksGSB": true,
       "IPv6First": true,
       "CAAValidationMethods": true,
-      "CAAAccountURI": true
+      "CAAAccountURI": true,
+      "SimplifiedVAHTTP": true
     },
     "remoteVAs": [
       {