The waiters now return the AwsErr error "ResourceNotReady: exceeded wait
attempts" when the instance state does not converge after 40 retries.
During longpollWatch() we need to detect this error and continue to
the top of the loop so we can restart the waiters and keep watching for
events.
If CheckApply was called when the instance was pending or stopping, it
would return an error. This patch supresses these errors and tells the
engine that the state can't yet be changed.
This patch makes sure that closeChan is closed as soon as the main loop
returns, so any channel operations are unblocked before we run shutdown.
This ensures that the server's goroutine can return before shutdown
completes and we don't panic by trying to serve the client after
shutdown returns.
This patch creates the cloudwatch rule that detects ec2 instance
state changes, and targets the rule to publish on our sns topic
which, in turn, pushes those event notifications to our endpoint.
This patch replaces the call to Server.ListenAndServe() with
Server.Serve(listener) in order to make sure the listener is up
and running before we subscribe to the topic in a future patch.
If we get an error in the Watch loop, it will send this on awsChan,
which will cause Watch to loop. However, in this scenario it will never
cause closeChan to close, and we will deadlock because we have a
waitGroup in a helper goroutine which is waiting on this channel to
close the context.
Normally this wouldn't be an issue, but since we have more than one
goroutine (with associated waitGroup) it is. It's also good practice to
close all the channels to help avoid this kind of bug.
This patch also moves the waitGroup Wait into a more logical place for
visibility.
