Since the answer is really to long to be written here, I can link you my PAM tutorials:
Write a Linux PAM module and
Linux PAM Configuration tutorial
Before starting writing the module I advise you to read the configuration tutorial first, in which you can learn what does the module do.
To sum up, a module is a shared object loaded by PAM when the application wants to authenticate. Every time the application triggers a "stage" (auth, account, session, password) the correspondent function is called in the module. Therefore, your module should provide the following functions:
PAM_EXTERN int pam_sm_authenticate(pam_handle_t *handle, int flags, int argc, const char **argv){
/* In this function we will ask the username and the password with pam_get_user()
* and pam_get_authtok(). We will then decide if the user is authenticated */
}
PAM_EXTERN int pam_sm_acct_mgmt(pam_handle_t *pamh, int flags, int argc, const char **argv) {
/* In this function we check that the user is allowed in the system. We already know
* that he's authenticated, but we could apply restrictions based on time of the day,
* resources in the system etc. */
}
PAM_EXTERN int pam_sm_setcred(pam_handle_t *pamh, int flags, int argc, const char **argv) {
/* We could have many more information of the user other then password and username.
* These are the credentials. For example, a kerberos ticket. Here we establish those
* and make them visible to the application */
}
PAM_EXTERN int pam_sm_open_session(pam_handle_t *pamh, int flags, int argc, const char **argv) {
/* When the application wants to open a session, this function is called. Here we should
* build the user environment (setting environment variables, mounting directories etc) */
}
PAM_EXTERN int pam_sm_close_session(pam_handle_t *pamh, int flags, int argc, const char **argv) {
/* Here we destroy the environment we have created above */
}
PAM_EXTERN int pam_sm_chauthtok(pam_handle_t *pamh, int flags, int argc, const char **argv){
/* This function is called to change the authentication token. Here we should,
* for example, change the user password with the new password */
}
In this functions you will use PAM functions to retrieve the username and the password from the application. This happens through a conversation function that must be defined in the application (see this tutorial). At the end of every function, you must return a PAM return code that determines the result (for PAM error codes see this and the module writer documentation in general).