OAuth tokens used in Service Executor expire after a period of time, causing jobs to fail

[ssaniev]

Service Executor (formerly knowns as pzsvc-exec) uses OAuth tokens in order to authenticate the application with the CF REST API that handles the creation and tracking of CF Tasks. CF Tasks are what run the actual algorithm portion of the application. 
At some point during the lifetime of the Service Executor, these OAuth tokens are expiring which causes all of the Task calls to Cloud Foundry to fail. This means that once the token expires, no new algorithms can be run - and the application will have a feedback loop of attempting to create Tasks that it does not have permission to do - and the console log shows these repeated errors. 
Service Executor leverages the Community GOlang library for delegating the logic for CF Tasks. 
We need to dig into this community library and look at how we can implement some functionality for, perhaps, routinely re-generating the OAuth token, or having some way to detect when the token is going to expire, and ensure we request a new one by the time of expiration. 
The short-term work-around is to restage the apps when the token expires, which causes a new token to be generated.