I have two different systems one with OpenGL 1.4 and one with 3. My program uses Shaders which are part of OpenGL 3 and are only supported as ARB extension in the 1.4 implementa
Like @Nicol Bolas already told you, it's inevitable to create two codepaths for OpenGL-3 core and OpenGL-2. OpenGL-3 core deliberately breaks with compatibility. However stakes are not as bad as it might seem, because most of the time the code will differ only in nuances and both codepaths can be written in a single source file and using methods of conditional compilation.
For example
#ifdef OPENGL3_CORE
glVertexAttribPointer(Attribute::Index[Position], 3, GL_FLOAT, GL_FALSE, attribute.position.stride(), attribute.position.data());
glVertexAttribPointer(Attribute::Index[Normal], 3, GL_FLOAT, GL_FALSE, attribute.position.stride(), attribute.position.data());
#else
glVertexPointer(3, GL_FLOAT, attribute.position.stride(), attribute.position.data());
glNormalPointer(GL_FLOAT, attribute.normal.stride(), attribute.normal.data());
#endif
GLSL shaders can be reused similarily. Use of macros to change orrucances of predefined, but depreceated identifiers or introducing later version keywords e.g.
#ifdef USE_CORE
#define gl_Position position
#else
#define in varying
#define out varying
#define inout varying
vec4f gl_Position;
#endif
Usually you will have a set of standard headers in your program's shader management code to build the final source, passed to OpenGL, of course again depending on the used codepath.