Redirect loop upon submission of contribution page

Upon submission of a contribution form, we go to CRM_Contribute_Form_Contribution_Main->preProcess(), which takes us to its parent - CRM_Contribute_Form_ContributionBase->preProcess(). In there on line 212 it is looking for an id:

$this->_id = CRM_Utils_Request::retrieve('id', 'Positive', $this);

This is supposed to be the ID of the contribution page, if we trust the comments. If we go inside CRM_Utils_Request::retrieve(), we see that the ID is sought in the $_REQUEST superglobal. If it doesn't find it there, it searches for it in the session. In my case, sometimes there is no ID in the $_REQUEST or in the session. So in those cases retrieve() returns nothing, we enter the if statement and get redirected through $this->controller->invalidKeyRedirect().

Now, even more interestingly, after the redirect we again go to CRM_Contribute_Form_Contribution_Main->preProcess() and then through the same path again. And again there is no ID in $_REQUEST, and so again we get redirected. So now we have started a redirect loop which continues until the browser stops and complains.

Some searching showed that others are experiencing similar problems: https://forum.civicrm.org/index.php?topic=34697.0 , https://forum.civicrm.org/index.php?topic=35261.0 , https://forum.civicrm.org/index.php?topic=34669.0 , http://civicrm.stackexchange.com/questions/6634/wordpress-redirect-loop-with-civicontribute

I can work on a patch, however I am not sure how to fix this. How do you think that we can prevent the redirect loop from occurring?

UPDATE:
Two more pieces of information. I managed to encounter a redirect loop again in my localhost and so looked at it closely in the debugger. Here's what happened:

• The very first redirect did not trigger my breakpoints (which were set on the preprocess() method of the contribution form). The next redirects, however, triggered my breakpoints. To me this suggests a different execution path on the very first redirect.
• There was a qfKey in the session on each iteration of the loop, however there was no id. Because there was no id, Civi could not recover and was stuck in the redirect loop. On each iteration it was searching for an id in the session but there was none. However, there WAS a qfKey in the session. To me this suggests that the session itself doesn't get lost, but somehow the id variable in the session gets lost. When that happens, we enter a redirect loop.

UPDATE 2:
Detailed explanation and steps to reproduce. Please bear with me as I'm also explaining all of this to myself.

On a Wordpress site you can enter a shortcode in your Wordpress page. The shortcode looks like this: [civicrm component="contribution" id="3" mode="test" hijack="0"] It specifies the id of the contribution page, the mode and whether to hijack the whole page. This data gets inserted in $_GET and $_REQUEST when the contribution page is first loaded.

Another piece of the puzzle is that Civi caches the session, removing it from $_SESSION and putting it in the civicrm_cache table.

On the CRM_Contribute_Form_ContributionBase->preprocess() method Civi checks for the id. It checks for it in $_REQUEST and in the session store. If it's not found, it issues a redirect.

We also need to have a qfKey in the session. That is created and managed in CRM_Core_Controller->key(). If it doesn't find a qfKey, it issues a redirect.

Now, it seems that in certain situations a desynchronization happens between the shortcode data and the qfKey. One might be there, but the other missing. In particular, a new qfKey gets created after a redirect, but the shortcode data isn't necessarily there. This leads to a redirect, but after the redirect the situation has not changed and we are again missing the shortcode data, leading to a redirect yet again and again.

Steps to reproduce (on a Wordpress site):
1) Open your browser's debug panel and go to the Network tab
2) Go to the contribution page
3) Delete the last two rows from the civicrm_cache table. They contain the cached session data.
4) Fill out and submit the contribution form. Watch the redirect loop.

I want to emphasize that this is not the only way to make the bug happen. The manual deleting of the cached session is just so that we can force the problem to appear. The redirect loop happens without us needing to delete the cached session when, for example, there is an ajax call after the contribution page has loaded. More details below on what happens during that ajax call.

I imagine there are various circumstances that would lead to the redirect loop. When there is a problem somewhere in the management of the session and a single redirect happens, it could easily lead to a redirect loop. For me it happened with an ajax call as well as during the execution of automated acceptance tests, where the qfKey was being transferred between tests and so it failed validation, leading to a single redirect, which in turn led to a redirect loop.

Relevant pieces of data from the execution cycle:

When the bug appears:
shortcode data (action (i.e. mode) and id) correctly loaded in $_REQUEST | CiviCRM_For_WordPress_Shortcodes->render_single()
a qfKey is created | CRM_Core_Controller->key()
all relevant data is correctly obtained and stored in $_SESSION, together with the qfKey just created | CRM_Contribute_Form_ContributionBase->preProcess()
all that session data is cached correctly in civicrm_cache and it is cleared from $_SESSION
page gets correctly loaded
after the page has loaded, we have an ajax call ()
shortcode data (action (i.e. mode) and id) correctly loaded in $_REQUEST
qfKey not found because it searches for it only in $_REQUEST. but it is neither in $_REQUEST nor in $_SESSION. it appears to be only in civicrm_cache. redirect issued | CRM_Core_Controller->key()
after the redirect, request method is now GET and so after it doesn't find a qfKey, it creates a new one with $key = CRM_Core_Key::get($name, $addSequence); | CRM_Core_Controller->key()
shortcode data (including 'id') is now missing from $_REQUEST. a redirect is issued | CRM_Contribute_Form_ContributionBase->preProcess()
after the redirect, a new qfKey is created | CRM_Core_Controller->key()
shortcode data (including 'id') is now missing from $_REQUEST. a redirect is issued | CRM_Contribute_Form_ContributionBase->preProcess()
and that pattern repeats

When things work correctly:
shortcode data (action (i.e. mode) and id) correctly loaded in $_REQUEST | CiviCRM_For_WordPress_Shortcodes->render_single()
a qfKey is created | CRM_Core_Controller->key()
all relevant data is correctly obtained and stored in $_SESSION, together with the qfKey just created | CRM_Contribute_Form_ContributionBase->preProcess()
all that session data is cached correctly in civicrm_cache and it is cleared from $_SESSION
page gets correctly loaded
no ajax call! we're done

When deleting the last two rows from civicrm_cache and then submitting the form:
qfKey is found
doesn't find id and issues redirect (it would have found the id at this point if I hadn't deleted the rows in civicrm_cache)
after the redirect it creates a qfKey
doesn't find id and issues redirect
after the redirect it creates a qfKey
doesn't find id and issues redirect
ad infinitum

Patch:
The patch I came up with is to load the shortcode data at the point where the qfKey is managed, like this:

$key = CRM_Utils_Array::value('qfKey', $_REQUEST, NULL);
//start hack
$uri = $_SERVER['REQUEST_URI'];
if(strpos($uri, '/donations/research-center/') !== false)

{ $_REQUEST['id'] = 3; }

elseif(strpos($uri, '/donations/motion-picture/') !== false)

{ $_REQUEST['id'] = 2; }

$_REQUEST['action'] = 'preview';
//end hack

I'm doing the if() because I have two contribution pages. It's important to note that this way the shortcode data gets loaded in $_REQUEST regardless of whether a new qfKey gets created or it already exists.

The above patch seems to be working very well, with not a single redirect loop happening for maybe a hundred submissions of the form that I did. Before that the loop happened consistently.

I don't know how to more elegantly load the shortcode data (hence the hacky solution). Does anyone know? Also, please let me know if you have any concerns about fixing this in the proposed way.