We are trying to automate certain data collection using cURL. Unfortunately the source system is protected by siteminder. (Web Access Management Software).
I tried using the normal command
curl -kL -o my_data.xml -u username:password https://example.com/location/of/file
(though the username & password are correct, its displaying error)
Error
HTTP Status 401 -
This request requires HTTP authentication ().
Any idea how to connect and fetch data from a siteminder authenticated page?
Cheers
Use the SiteMinder reference to find the required parameters needed for the login.fcc template:
Here is an example SiteMinder request/response:
http://HostName.example.com:9898/SiteMinderagent/forms/login.fcc?TYPE= 33554433&REALMOID=06-1716e557-15f3-100f-b9a4-835cc8200cb3&GUID=&SMAUTHREASON= 0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4f9R%2bcSa0%2fEgnu6oUQQPMQnUgkU6Zvx5zWZpQ% 3d&TARGET=$SM$http%3a%2f%2fshivalik%2ered%2eiplanet%2ecom%3a9898%2fvalidation% 2findex%2ehtml GET /SiteMinderagent/forms/login.fcc?TYPE=33554433&REALMOID=06-1716e557-15f3- 100f-b9a4-835cc8200cb3&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4 f9R%2bcSa0%2fEgnu6oUQQPMQnUgkU6Zvx5zWZpQ%3d&TARGET=$SM$http%3a%2f%2fshivalik%2 ered%2eiplanet%2ecom%3a9898%2fvalidation%2findex%2ehtml HTTP/1.1 Host: HostName.example.com:9898 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9, text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive HTTP/1.x 200 OK Server: Netscape-Enterprise/6.0 Date: Fri, 01 Feb 2008 23:46:12 GMT Content-Type: text/html; charset=ISO-8859-1 Connection: close ---------------------------------------------------------- http://HostName.example.com:9898/SiteMinderagent/forms/login.fcc?TYPE= 33554433&REALMOID=06-1716e557-15f3-100f-b9a4-835cc8200cb3&GUID=&SMAUTHREASON= 0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4f9R%2bcSa0%2fEgnu6oUQQPMQnUgkU6Zvx5zWZpQ% 3d&TARGET=$SM$http%3a%2f%2fshivalik%2ered%2eiplanet%2ecom%3a9898%2fvalidation% 2findex%2ehtml POST /SiteMinderagent/forms/login.fcc?TYPE=33554433&REALMOID=06-1716e557-15f3- 100f-b9a4-835cc8200cb3&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4 f9R%2bcSa0%2fEgnu6oUQQPMQnUgkU6Zvx5zWZpQ%3d&TARGET=$SM$http%3a%2f%2fshivalik% 2ered%2eiplanet%2ecom%3a9898%2fvalidation%2findex%2ehtml HTTP/1.1 Host: HostName.example.com:9898 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9, text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://HostName.example.com:9898/SiteMinderagent/forms/ login.fcc?TYPE=33554433&REALMOID=06-1716e557-15f3-100f-b9a4-835cc8200cb3& GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4f9R%2bcSa0% 2fEgnu6oUQQPMQnUgkU6Zvx5zWZpQ%3d&TARGET=$SM$http%3a%2f%2fshivalik%2ered% 2eiplanet%2ecom%3a9898%2fvalidation%2findex%2ehtml Content-Type: application/x-www-form-urlencoded Content-Length: 233 SMENC=ISO-8859-1&SMLOCALE=US-EN&USER=test&PASSWORD=test&target=http% 3A%2F%2FHostName.example.com%3A9898%2Fvalidation% 2Findex.html&smauthreason=0&smagentname=sHjbzl4f9R%2BcSa0% 2FEgnu6oUQQPMQnUgkU6Zvx5zWZpQ%3D&postpreservationdata= HTTP/1.x 302 Moved Temporarily Server: Netscape-Enterprise/6.0 Date: Fri, 01 Feb 2008 23:46:18 GMT Content-Type: magnus-internal/fcc Set-Cookie: SMSESSION=2xm2Iw6fTMBcjA6rlK/YUY1CRBudYxwOCkfpCo95YKAp2b4ZzLOPT qi2S14CQ7nRja+fUq53Aj0pmTxDvPKTMcKD1Ql1hGx0gPK7xx2eqMP3IyTAK3qNahRgt7mQRTIB BDEE0rOJcpgrMRtsteC90yMdiJrrEeqfC38utU6mxO9BejwjRuGN2rmf9WM4Odl+4TE0iUOiP/k iCR6sn2r03GBsbBjOi12oSlh/4JAyfOwxsgBJCwDiZVlFXNiKNaKdY1UQr8OcKeO33eNn3w9RW9 ZrjRibQTQcxxmiR+gsvAuM8etEzP6GCFKjc1s8I3DNuSBbDqfyt81YUSYdEYa9UKfvvOJplZOIT BkQajcAEPOq+vTYxQ4BH2RmjdPMVcIxRm2bibM9QtuQD83C9QubTk1lq4j+ywPsvutiYEoGHV+7 6VXws5NsvhK2gH4ZTC0xsd76X2/1no8xMv9c3W4DcSp9cQQ74/7+a7gzT+hxQSpyQFf4mDTnq/D XS5V7tcLS0EyFcf8RwSbvDPnICiebR3vtZgHRL1kEZheEh9ToHmwqIO9cCqz9rJXR7/NL+o/AQr 7M4o+LyA7KxozAueUj0pg8GINteUGVxMLWmR7Xm/Lp0pI9DjM5mfbmP8Ka+w0T6H9LHNlQGaYZA PCkeABAXqLb8q8yJUzPdI0BVlp1awNCx579DereoCIzCZdQ99rVDSQUS77KCQATnYXrHqTxqbXxW beDf6gk9ZCf29XTzO8hBLdScqGOBX1OvDvzdghcjHnupQf1fYltt/3MrZ/Jrxonbpgxg4C5zVgSU PrNqb66RYWQOelZXooh7lTPoFHsMFodVnecsOZmEMXNI8DB08pyo5KhRZJk2Mr4o3rPNtiHPpnXc d+imapuosG3FwF5Sv6flh8jbiE9/MZdIQ06hgWEIiCnUEYdboli4TWgy0/QpCbdJ7OviU275VZiC W6hMTRyrxnEvoQ=; path=/; domain=.red.example.com Cache-Control: no-cache Location: http://HostName.example.com:9898/validation/index.html Connection: close ---------------------------------------------------------- http://HostName.example.com:9898/validation/index.html GET /validation/index.html HTTP/1.1 Host: HostName.example.com:9898 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9, text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://HostName.example.com:9898/SiteMinderagent/forms/ login.fcc?TYPE=33554433&REALMOID=06-1716e557-15f3-100f-b9a4-835cc8200cb3&GUID= &SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=$SM$sHjbzl4f9R%2bcSa0%2fEgnu6oUQQPMQnUg kU6Zvx5zWZpQ%3d&TARGET=$SM$http%3a%2f%2fshivalik%2ered%2eiplanet%2ecom%3a9898% 2fvalidation%2findex%2ehtml Cookie: SMSESSION=2xm2Iw6fTMBcjA6rlK/YUY1CRBudYxwOCkfpCo95YKAp2b4ZzLOPTqi2S14 CQ7nRja+fUq53Aj0pmTxDvPKTMcKD1Ql1hGx0gPK7xx2eqMP3IyTAK3qNahRgt7mQRTIBBDEE0rOJ cpgrMRtsteC90yMdiJrrEeqfC38utU6mxO9BejwjRuGN2rmf9WM4Odl+4TE0iUOiP/kiCR6sn2r03 GBsbBjOi12oSlh/4JAyfOwxsgBJCwDiZVlFXNiKNaKdY1UQr8OcKeO33eNn3w9RW9ZrjRibQTQcxx miR+gsvAuM8etEzP6GCFKjc1s8I3DNuSBbDqfyt81YUSYdEYa9UKfvvOJplZOITBkQajcAEPOq+vT YxQ4BH2RmjdPMVcIxRm2bibM9QtuQD83C9QubTk1lq4j+ywPsvutiYEoGHV+76VXws5NsvhK2gH4Z TC0xsd76X2/1no8xMv9c3W4DcSp9cQQ74/7+a7gzT+hxQSpyQFf4mDTnq/DXS5V7tcLS0EyFcf8Rw SbvDPnICiebR3vtZgHRL1kEZheEh9ToHmwqIO9cCqz9rJXR7/NL+o/AQr7M4o+LyA7KxozAueUj0p g8GINteUGVxMLWmR7Xm/Lp0pI9DjM5mfbmP8Ka+w0T6H9LHNlQGaYZAPCkeABAXqLb8q8yJUzPdI0 BVlp1awNCx579DereoCIzCZdQ99rVDSQUS77KCQATnYXrHqTxqbXxWbeDf6gk9ZCf29XTzO8hBLdS cqGOBX1OvDvzdghcjHnupQf1fYltt/3MrZ/Jrxonbpgxg4C5zVgSUPrNqb66RYWQOelZXooh7lTPo FHsMFodVnecsOZmEMXNI8DB08pyo5KhRZJk2Mr4o3rPNtiHPpnXcd+imapuosG3FwF5Sv6flh8jbi E9/MZdIQ06hgWEIiCnUEYdboli4TWgy0/QpCbdJ7OviU275VZiCW6hMTRyrxnEvoQ= HTTP/1.x 200 OK Server: Netscape-Enterprise/6.0 Date: Fri, 01 Feb 2008 23:46:18 GMT Set-Cookie: SMSESSION=jlO0TgMQfglpU+GHQCJqbnoE2Pevax6fdzPGU7ZAgJuPb/fxTjCbWX1 B1RO6QaLJn6VoVGNK8Sy6IeILAyv+LciS/OMK1E0tSXnL5Uvit3XIuWuiSMuklyDMIlOQ6n3ZSGGr 9sKBUch5YVfGcfGjHQFcBIlzegQxBRrgH/l2rc8aTEHdCrprvBiRHwQlxJbrcWMqfJw7h+HUEtiz9 bQCUkwMbpEW4eBfNyRlZTGov3K5hg4HK4tuoyvOeKdZaewlTB4Lm+QeGWo2qv2mPDP+eVtBiVtRVH HTHGfSthTJYQOOc4rPV2dnl8axpWppGByeUmfmeService Provider9x5hVxDi91iyobTybKpDz0 bltkvnHbqwbLfehUPtJFxS3Z54y9dmiuoQ+B5Kdrs7DNuvrnAI1ZQdDKQEVA4Pt+vA9KO18ah9V1I 7BZ9D/x60uWxfaA3Ty8lRgWhMYqdBulFMD1B29sxboNHWdJ2FaxQJGjMpSEZ5iHB50ovF4YFXRyPP 5Tl7eJxIebLKX02LFrG/osNZ9UKHrMY1MRK5WWHJlYB040ADVcTNrFkc39vcYIA1eGDYhC/NaOd41 2HP5S0UX0/59ADMLBsX/qBjcdODy3li+4eZnK1oHw/9yr3LCjewJ+H9w0k0/dQw99vgwEM2RPFgH5 Y7W6k6h1efp67VKXLBiJ1OZPJe2SCEDAOUla8qsC8fQ0VWTy/TfVhVtqJOaSLZrACX7uhPzbZE1EA Pd8x7UeJquFll3WpdnZYObd0DQLeoWZcF2rPIcfBn+8X8oig5KzvAgQ9R8MR+h7OkYfhmwwBDaQkb KPpIxjpeLNxKpkEVWJ9HoHOpZ/txCQUAHqPV41YjZ6CQfBfUqdOHbfje9O+0pJ1aHMntI4VYZOqdx sA+n9cgKjNQ8ruHOqSKhAQfEgipwcM2fMU3Uqmtr+0/+5bi7Cbs=; path=/; domain=.red.example.com Content-Type: text/html Etag: "dcea10a4-1-0-88" Last-Modified: Thu, 10 Jan 2008 01:42:07 GMT Content-Length: 136 Accept-Ranges: bytes ----------------------------------------------------------
The hidden inputs listed in the following figure are used to hold state for the credential collectors:
An FCC can interpret a number of special name/value pairs (@directives) that invoke nonstandard processing. The special @directives and their meanings follow:
Special Name/Value Pairs
postpreservationdata
Data that a user submits through a post request.
username
Name for the login user name.
password
Password to perform the login.
target
Resource to access after login.
smheaders
Colon separated list of response names to include in the namespace. The colon separated list must contain an entry for each header that you want to include in a transaction. For example, if you want to pass the value of header1 and header2 as part of a transaction, include the following line in your FCC:
@smheaders=header1:header2
smerrorpage
If there is an error on a POST to the custom form, the user browser is redirected to this page. If this special value is not specified in a .fcc file, the system uses the .unauth file that is associated with the .fcc file as the error page.
smretries
Specifies the maximum number of login attempts allowed. If you set this directive to 0, the number of retries is unlimited. If you set the number to 1 or greater, that is the number of retries allowed.
Note: If users log in using a POST to an .fcc form, it may appear that the user is given additional attempts to log in beyond the value of the smretries directive. However, the user is allowed access only if valid credentials are entered in the number of attempts that smretries specifies.
smpasswordfcc
Determines whether data is posted from the Password Services FCC file or from a different FCC file.
Default: 1
Important! We recommend that you use the default value. The SafeWord authentication scheme may not work properly if the default value is changed.
smusrmsg
Text that describes why the user was challenged / failed to login.
smauthreason
Reason code that is associated with a login failure.
smsavecreds
Set to Yes to save user credentials in a persistent cookie on the user browser.
smsave
Colon separated list of names to be saved as persistent cookies.
save
Another name for smsave.
smtransient
Colon separated list of names to be saved as transient cookies.
smagentname
Specifies the agent name that is supplied to the Policy Server when a user enters credentials and submits the form for authentication. If the Agent parameter, FCCCompatMode=NO, specify a value using this directive.
smlogout
Logs a user out of the system, similar to the LogoffUri parameter. By placing @smlogout=true in your .fcc template, the FCC logs a user out and redirect the user to the target. As such, the @smlogout directive is typically used with the @target directive (@target=).
urlencode(name)
Replaced by the URL encoded value of the named variable.
Note: If you expect the additional attributes or the Password to contain special characters (" . & = + ? ; / : @ = , $ %), URL-encode each additional attribute value in the .fcc template file. The template uses US-ASCII encoding.
urldecode(name)
Replaced by the URL decoded value the named variable.
Note: The “sm” prefix for name/value pairs is reserved for additional special names that the system requires. When creating names for your login page do not use the “sm” prefix.
Localization Name/Value Pairs
The .fcc template files include two localization parameters:
smlocale
Used to determine the language used in the HTML forms that collect user information or display status messages.
The value that is paired with smlocale corresponds to part of the name of a localization properties file. The localization properties file contains IDs mapped to text strings in the specified language.
smlocale values have the following format:
COUNTRY-LANGUAGE
For example, the value for smlocale for United States English is:
SMLOCALE=US-EN
smenc
Contains information that tells the browser what language encoding to use. Changing the default value for this variable overrides the encoding set in the following META tag:
At a minimum, an .fcc file must collect the following:
User name
Password
Target
Important! If users will be submitting post requests to a resource protected by an authentication scheme that uses a credential collector (see the following figure), use the postpreservationdata input. Otherwise, data that users attempt to post to the requested resource will be lost.
References
You would need to post your credentials to the .fcc file, and manage the cookies that SM returns (look for SMSESSION cookie)
来源：https://stackoverflow.com/questions/18128906/curl-and-siteminder-authentication