Setup: Does WorkflowGen use encryption when it communicates with the database?

Yes, this feature is provided by SQL Server and IIS server. For more information, see http://technet.microsoft.com/en-us/library/ms189067(SQL.105).aspx.