CRM-21263 Upgrade select2 to stable version 4.0.4 and fix compatibility issues in Civi

This will supplement https://issues.civicrm.org/jira/browse/CRM-20826 as accessibility improvements made in selectWoo repo is made against select2 (4.0.4). Meanwhile I've cherry-picked all the commits in my select2-4.0.4 copy - https://github.com/select2/select2/compare/master...monishdeb:stable-4.0.4-improve-accessiblity#diff-b30523a5b3b7fb3fa27988e823d3e8ce

Last time I looked at this problem, there was a big breaking change between 3.x and 4.x in that the 3.x version uses a comma-separated string for ajax multiselects, and 4.x uses an array. I think the latter is better, but we've got a ton of not-well-documented form-layer code which depends on the former, so it's going to be a job to find and fix it all.

Thanks, Coleman. We're likely to dig into this strategically important fix to make CiviCRM WCAG compliant and thus more able to be included in bids in various jurisdictions. Monish will start by listing and prioritizing work to be done, aiming for a divide and conquer approach that focuses on major early wins and gives a lower priority to rare single cases.

Mathieu Lutfy do you have any insight into the Almond amd loader used by Select2 v4 for translations and some other stuff https://select2.org/upgrading/new-in-40 ? Should we be concerned about supporting that or just relying on existing ts() infrastructure?

Reference material: https://select2.org/upgrading/migrating-from-35 

I think we should add this to the list started by Tim Otten of 5.0-worthy changes.

So I've had some very bad experiences with doing a lot of work on feature branches that took forever to be merged into master, resulting in lots of extra work on rebasing and unfortunately some errors from that. I don't want to start work on something if 5.0 might mean "we'll probably keep postponing this indefinitely."

I don't care so much about the number as the process and timeline and criteria for merging work if we take this on. Can we plan on some interim deliverables? If not, when would the work be considered done enough to be merged? Monish is going to start by laying out an approach, and various tasks, as we try to better understand at a granular level what the issues are that need to be overcome. I think he'd like to start by understanding better what the wrapper around the v3.5 select2 is doing. What was the previous version of select2? 

I'm a bit sceptical that modifying the wrapper to work with 4.1 (ie 4.0.4 without the special upgrade oriented approaches that are deprecated) will be sufficient to make our widgets accessible, but perhaps some brainstorming could go into avoiding some work with that approach. Alternatively, we could go back and do the work to make the many places using select2 compatible with v3.5, v4.0 and then the wooSelect fork. 

Monish Deb might benefit by searching to see if there are any automated tools for upgrading, at least partially, to select2 v4.

Well Tim Otten and I should discuss this more, but my thinking is that 5.0 is going to be just another increment from 4.7.x and we're not going to maintain the two simultaneously. E.g. 4.7.27, 4.7.28, 4.7.29, 5.0, 5.1, etc.

I don't think there are any huge breaking changes to stick in 5.0. It's mostly about bundling up some extensions like Shoreditch and removing some cruft like case activity revisioning.

Ah, great. Eileen McNaughton suggested elsewhere that the semVer bump to 5.0 should ideally be on a small thing, which I agree with. Because the change to starting semVer will be controversial, I'd really love if it was not on a JMA issue . Nonetheless, we'll proceed with this work on the understanding that when it is ready it will be merged to master.

Right - my current thinking is that our first semVer release should go out in December - we would drop 2 releases 4.7.29 & 5.0.0.0 - they would be exactly the same except one line of code that gives a fatal error if php version is 5.3. That should give us the minimum possible 'oh my god it's a different major version' impact.

Specifically with regards to this ticket & the benefits / risks of making this change - we need some more detailed examples of what breaks, what the fix is, what can be done in preparation e.g can we get more into using wrapper / helper functions so only one place needs to be changed? How will it impact extensions. 

I don't think we should be planning on ever doing a release with 'a whole bunch of breaking changes' - but if managed well this might be able to warrant a major version bump when we are on semver

I like the direction of your thinking, Eileen McNaughton. The link above https://select2.org/upgrading/new-in-40 indicates a bunch of breaking changes in the Select2 v4 rewrite, and I think we will need to go all the way with 4.0, avoiding the special approaches for easing migration which are immediately deprecated in 4.0 and scheduled to be removed in 4.1, before we can easily swap in the accessible wooSelect fork.

Migrating to https://lab.civicrm.org/accessibility/replace-select2/issues/1 . I want to close and am unhappy with current workflows and forced choices of status, amongst which I am selecting Won't Do although that is inaccurate.