I spent the whole day troubleshooting some authentication/redirection problems in my Rundeck setup, I went to the pub and had a few beers then I went to bed and woke up at 4 AM with one of those productive insomnia feelings so I headed straight to the computer to find the solution for my problems. Let’s go through my setup so I can explain the problem a bit more and hopefully help you avoid the same issue.
The setup
My setup consists of Rundeck running behind a reverse proxy configured in NGINX and the whole box sits behind an Application Load Balancer (ALB). The whole shebang looks more or less like this:
Architecture diagram
The problem
I’m configuring SSL termination on the ALB level, which is way more convenient IMO, however, it was causing some weird problems that took me a while to catch. The main symptom was at the login form. There was no feedback for login attempts (failed or successful) and the login redirection wasn’t working. I was able to log in by clicking on the login button then manually going to the home URL.
Trying the login page with the Developer tools open on Chrome I noticed that it was calling the URL path /j_security_check but the request on Chrome was being shown as canceled in the Networking tab all the times (if you know why it behaves like that leave a comment). Then I decided to manually try the authentication with curl, the command I ran was:
.... #curl verbose stuff here
< HTTP/2 302
< date: Sat, 16 Feb 2019 09:07:48 GMT
< location: https://rundeck.example.com:80/user/error
< server: nginx/1.14.0 (Ubuntu)
< set-cookie: grails_remember_me=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT;Max-Age=0
< expires: Thu, 01 Jan 1970 00:00:00 GMT
< set-cookie: JSESSIONID=node01tl6fp5oi0qoh1x0g3r9kxwu83407.node0;Path=/;Secure;HttpOnly
<
* Connection #0 to host rundeck.example.com left intact
It was taking my standard HTTPS request and responding 302 with a location https://rundeck.example.com:80/user/error. The root cause was that Rundeck wasn’t picking the X-Forwarded-Proto header in the requests although I was forcing it on the NGINX config with the directive proxy_set_header X-Forwarded-Proto 'https';.
Back into de documentation page (which could have a full-text search tool) I discovered that Rundeck isn’t proxy aware by default but there is a way to configure it. Using their own wording:
You can tell Jetty to honor X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Server and X-Forwarded-For headers in two ways:
This solution works for ALB and the Classic Load Balancer (a.k.a ELB). Note that I’m using port 80 so make sure that you set your load balancer health check on the appropriate port.
How about you? Feel free to share a battle story, feelings or suggestions in the comments section.