Release Management - releasing to a subset of users - how would it work for a public facing website

Question

I read somewhere (sorry don\'t exactly remember the source) that facebook has release Tuesdays. They release the new features to their internal employees first, then to a sm

Answer 1

Option 1:

A (sub)domain for the (semi)(public?) devsite. Certain users are accepted on the devsite based on a manual set cookie (employees etc) < private testing

The main domain which sets cookie for certain users (cookie set when time is between time X and time Y) < depending on your traffic. If you get 1000 (unique)visitors every hour and you want 10% on the dev domain you make sure the delta time is 6 minutes. You can simply delete the cookies when you want users to be redirected to the normal site.(make sure you redirect the whole incoming url to prevent broken bookmarks.

Option 2:

Loadbalancing a certain percentage of the traffic to servers running new app.

Database

1:Interact with live database while developing < regular backups + regular skilled devs = safe

2: look at master slave replication to create a "live" shadow copy of your DB

Answer 2

The simplest approach IMO would be to separate features by sets of pages that use security to determine which page a user gets. Beyond security and other means to ensure a user cannot get to a page directly to which they do not have access (site maps etc.), you could store in the database the list of features and which users or roles have access to that feature:

Create Table Features
    (
    Code varchar(10) not null Primary Key
    , StartPage nvarchar(max) not null
    , Description nvarchar(max) not null
    )

Create Table UserFeatures 
    ( 
    UserId ... not null
    , FeatureCode varchar(10) References Features ( Code )
    )

First, the reason I would use a text code for the feature primary key and not a surrogate key like an IDENTITY column or guid is that only the system will be querying for features. Users will never have the ability to arbitrarily add a feature. Thus, it makes your code much cleaner and easier to read to query for ...Where FeatureCode = 'AdvancedEntry' than ...Where FeatureId = 13.

Second, in this approach, the code of the pages themselves would determine which procedure to call. However, that may mean quite a bit of duplication if the features simply involve additional fields of information.

Thus, if the features are tightly integrated into the existing code base and presentation layer (which by the way is why versioning is so difficult), an alternate approach would be to store the name of the stored procedure that should be used in the Features table. Your code would query join the above tables and return the stored procedure name that should be used. For the arguments, you could store a parameterized call in the database (e.g. exec Schema.Foo @bar, @gamma, @beta) and when executing the query simply check whether that string contains a given parameter and if it does, add that parameter value:

if ( ProcTemplate.Contains( "@bar")
    commandInstance.Parameters.AddWithValue( "@bar", barValue );
if ( ProcTemplate.Contains( "@gamma")
    commandInstance.Parameters.AddWithValue( "@gamma", gammaValue );
...

If you map features to user roles or groups, then you will need to devise "trumping" rules which determine which feature should be used in the case of multiple being returned. In this approach, you would leave existing stored procedures alone barring a change to the schema which required a change to the stored procedure (e.g., the removal of a column). A bonus result of this approach is that you could add a date to the Features table to determine when a new feature should go online.

Answer 3

I really thought this question would have received a lot more attention. Considering what you said and the existing answers, I think your existing solution is the most straight forward and easiest to manage. As you said, there is some "design smell" (I like that phrase) but it makes the most sense.

Perhaps going one step further and combining some slight modifications of the suggestions with your own:

• Keep your existing database version convention
• Release specific candidate versions to either a sub-domain candidate01.yoururl.com or a specific server if you have a server farm
• Use a flag on your user/member table to indicate which production server or sub-domain the user should be directed to
  • Provides the ability to direct some users to the release candidate server(s)
  • Doesn't require the amount of code the per feature coding option mentioned in DK's answer would (you could go either way though depending on how targeted you want to be but I think the least complicated route would be best here and would direct users to specific versions of the app rather than try to turn individual features on/off per user)

Other than that, great question! Oh yes, and when that is all ready you just flip the fluger switch to engage the double buffer mechanism and you are all set.

Answer 4

If I understood you correctly, what you want is to have two mechanisms that use the same live data, but different API versions. Now, assuming you're already working with a double buffer mechanism, I guess that your actual problem is to use live tables during that transition.

The solution is for your tables to include both V1 and V2 columns (e.g., one users table that will include fields from both APIs). Note: All non-common fields must have default values.

In order for this to work seamlessly, you should create views for V1 and V2, exposing only the relevant fields for each API version, and you should develop for views instead of tables (similar to the concept of developing for interfaces instead of implementation).

The same goes for stored procedures - all non-common parameters must have default values, etc...

Answer 5

I was at QCon last year, when a guy from Facebook's Web Team was talking about it this approach. You are 100% correct that there will be code smells in order to implement that. But it takes a lot more than just a code smells (actually they call those code smells Gate Keepers :).

First of all - their way of representing and storing data is a lot more sophisticated, they are not using Foreign keys, or any other "advanced" features of the databases at all :), as far as I remember, their data is not that "relational" as we all want to keep ours :).

What you can do is add gate keepers (if (user is from New Zeland) { use then new version with the new classes } else { stick ot the old one }).

You can probably imagine to what kind of code duplication will this lead in case of structured model (e.g. you will need to have 2 Users, 2 Orders, 2 Order Details). Leaving that aside, I think a lot of regression will also take place if you don't be extremelly careful.

As far as I remember - they are releasing a lot more often than just Tuesdays. I think they have a continous deployment and code goes live every day, just they clean old Gate Keepers / schema changes Tuesdays as they switched all the users to the new functionality by then.

So basically:

1. You add gate keepers everywhere a new functionallity is taking place.
2. You maintain 2 versions of the Schema.
3. You add more and more users to switch to the new one.
4. You clean the old one.
5. You go to point 1 (if another team isn't already there :).

I realize this answer may not be full enough to qualify for an answer, but I heard this from their guy and thought it is worth sharing.

Answer 6

At my company we almost always do any major release in a staggered fashion. We achieve this by adding a flag for every new feature in the users table. By default this flag is set to false; and as we are rolling out the feature to more users we simply switch the flag in the DB table.

This means at the application level, we have to make sure at all places we check for this flag. The code push goes to all servers. DB changes are made; but still only some users see the new feature.

At the database level, we make sure that any changes to SPs are "backward compatible". This is done by following some simple rules:

1. Any new parameters added to the SP have to go to the end of the parameter list.
2. New parameters should have a default value. This is done so existing calls to the SP does not break.
3. If existing parameters to SP has to be changed (or if order of parameters has to change), then obviously all calls to the SP is changed. But the SP is coded in such a way that it supports users that have the feature enabled as well as users that don't have it enabled.
4. Most table changes are related to adding new columns. It is really rare when we have to modify an existing column. All new columns are added with a default value or the allow NULLs.

As for the API, most of our parameters are passed as custom objects(structures). This way we can add a new parameter to the API methods and still prevent existing calls to the API from breaking.

Also, for each user we store the API version they are using. Depending on the version, users go to a different API URL. So basically when users make the first API call to authenticate, we pass a new API URL (based on API version for the user). For all subsequent calls, they are supposed to call the new URL. Salesforce.com also follows this for their API calls.