Koozali.org: home of the SME Server

set reverse proxy to web service

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
set reverse proxy to web service
« on: December 22, 2018, 11:27:01 PM »
I have a web service running on an older SME 8 installation. I have a new SME 9 server running as the primary business server. I would like a specific URL received by SME 9 to be passed to SME 8 for processing. Similar, perhaps, to a load balancing Proxy.

I have tried a number of things without succeeding. I set up a RewriteRule under a Location. The redirect worked but the web service client received a 302 warning. Unfortunately the client cannot handle that and needs a 200 reply code.

I have been trying ProxyPassReverse but haven't hit on the correct set up.

In custom templates for httpd.conf
Code: [Select]
  ProxyPass /ws http://oldserver.mydomain.com/ws
  ProxyPassReverse /ws http://oldserver.mydomain.com/ws

In custom template for httpd.conf, VirtualHosts
Code: [Select]
<Location /ws>
  Order allow, deny
  Allow from all
</Location>

restart httpd and it fails to launch on an error.

Has anyone done this before?
- Mark

Offline mmccarn

  • *
  • 2,626
  • +10/-0

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #2 on: December 24, 2018, 07:17:43 PM »
Thank you. That worked quite well.

Unfortunately for me, something else has fouled the proper function of the web service. The old server is returning a 404 error. At this point I don't know what changed to cause the problem, but I am looking.
- Mark

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #3 on: December 28, 2018, 08:08:23 PM »
Resolved the 404 issue, but still not out of the woods.

I can use a web browser to access the web service admin panel. However, the old web service client still gets a 302 reply code. The old client is expecting a 200.

I think I need to adjust the rewrite log and level to get an idea of what is happening. I think that can be handled in templates-custom, in VirtualHosts, with a file named "10RewriteLogSetting" and the following content:

For Apache 2.4 and higher...
Code: [Select]
    LogLevel alert rewrite:trace3
For Apache 2.2.15, which is what is running on SME 9...
Code: [Select]
    RewriteLog /var/log/httpd/rewrite_log
    RewriteLogLevel 9

That latter bit did cause the log file to be created, no entries were written to it. So, I am kind of stuck trying to find out what is causing the 302 on rewrite.
« Last Edit: December 28, 2018, 11:16:11 PM by Mophilly »
- Mark

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: set reverse proxy to web service
« Reply #4 on: December 29, 2018, 01:58:40 PM »
Just spotted you are trying to reverse proxy a 'ws' url. It won't work as it stands because Apache 2.2 doesn't support it by default.

Daniel has built a ws proxy rpm that will help. It's what I use for Rocket chat.

Please go and have a look at the wiki page and my code.

Can't help much at the minute as on hols and only on a mobile.

Can look later. Note I have got nginx scl running alongside apache to do this as well.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: set reverse proxy to web service
« Reply #5 on: December 29, 2018, 02:23:23 PM »
The instructions for installing and enabling mod_proxy_wstunnel on SME 9.2 are in the WebRTC section of the Rocket Chat wiki page:
https://wiki.contribs.org/Rocket_Chat#WebRTC_configuration

Install requires the FWS repository

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: set reverse proxy to web service
« Reply #6 on: December 29, 2018, 04:52:59 PM »
Thanks Mike.

Note I Rocket running on apache with Daniels rpm and nginx as per here

https://wiki.contribs.org/Nginx

You don't need Daniels rpm for nginx as it has built in ws webservices support unlike apache.

Lesson here is to always fully describe what you are trying to achieve :-)

Please ask if you have more questions. I'm no guru but have it working!! I'll be about a bit on Rocket as well if you want to chat.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #7 on: December 29, 2018, 06:10:16 PM »
Thank you for the links. Very interesting and helpful.

Yesterday I received advise that the http ->> https redirect is likely getting in the way. That is, the old web service client (circa 2008) can handle http but doesn't know what to do with the 302 raise by the https redirect.

So, today I will try to exclude the one URI containing "/ws/" from the https coercion. In the web root .htaccess I have the following condition to exclude the URI from processing by WordPress.
Code: [Select]
RewriteCond %{REQUEST_URI} !^/(ws|ws/.*)$
I am not exactly certain where to focus my attention, but I am going to start with the Location /ws template.
- Mark

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #8 on: December 29, 2018, 09:02:36 PM »
I added a custom template in VirtualHosts, order 26, to add...
Code: [Select]
    RewriteEngine on
    RewriteCond %{REQUEST_URI} ^/ws
    RewriteRule .* - [L]

Expanded the template and restarted httpd. The snipped did appear in the SSL connections area of httpd.conf. However, our tests failed with the old web service client. The request URL is still rewritten to https.

I did notice that if I turn off the "Force Secure Connections" option for the specific iBay, the old web service client works as desired. I am perplexed.

Is the iBay option "Force Secure Connections" necessary if the server redirects to https as a default?

Is it safe to leave that iBay with "Force Secure Connections" disabled?
- Mark

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: set reverse proxy to web service
« Reply #9 on: December 29, 2018, 09:47:11 PM »
I'm little confused.

Are you talking about webservices as in an API call or websocket (normally ws://) ??

Websockets need Daniels rpm.

If it is web services then does the client actually handle https? If not......
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: set reverse proxy to web service
« Reply #10 on: December 29, 2018, 09:48:57 PM »
PS.... as I mentioned please explain exactly what you are trying to do as it isn't clear.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #11 on: December 30, 2018, 01:25:17 AM »
Quote
please explain exactly what you are trying to do

Sorry for the unfocused thread. I do appreciate all the help.

Goal
Support an old web service client in the field with server side adjustments. The old client uses XML-RPC style of invocation. E.g. http://www.newdomain/ws/message/ as a POST with the xml detail in the body.

Problem
1. the web service was hosted on a soon-to-be-retired SME 8 system
2. the old web service clients in the field do not support https, and cannot handle a 302 server reply as modern clients do today.
3. the field installs cannot be upgraded easily if at all.

Recap
At the beginning the task was putting a redirect in place, e.g. http://www.newdomain/ws/message/ is passed to http://www.olddomain/ws/message/. Got that worked out with the help here, but the 302 continued.

In the middle of all this, I thought the problem was the redirect to the old server. 302 was not wrong, exactly, just unwanted. I suspected the server setup might work better if I moved the web service handler to the new machine. Having done that I found the old web service client was still getting a 302, although along the way I had to sort out a 404.

Yesterday a colleague suggested that something other than the redirect and/or location of the web service handler was in play. It was then I remembered the change to have the http ->> https redirect be the default on the system. (doh!)

Today I was trying to direct Apache to leave calls to http://www.newdomain/ws/message/ as is. Don't force to https. My associate, on a hunch, disabled the Force Secure Connections option on the ibay for newdomain.com and the old web service client worked!

Summary
In reflection, the redirect and/or proxypass methods may not have been as significant as I assumed. They worked as designed. Along the way I learned a lot about the SME template system, and the resulting config is cleaner than on the old machine.

What we have now is this:
Using custom templates, the Location /ws is created. This correctly invokes the Apache mod for handling the web service calls. The iBay for www.newdomain.com has Force Secure Connections disabled. It all works with the old clients in the field.

I would like to understand better what is happening with Force Secure Connections on an iBay in SME 9. I am wondering if having the secure connections option off is wise.
- Mark

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: set reverse proxy to web service
« Reply #12 on: December 30, 2018, 05:36:55 AM »
I did this:
1) copy /etc/httpd/conf/httpd.conf to /etc/httpd/conf/httpd.conf.no-force-secure
2) Enable "force secure connection" for my ibay named "wordpress" in server-manager
3) copy /etc/httpd/conf/httpd.conf to /etc/httpd/conf/httpd.conf.force-secure
4) compare the two files
Code: [Select]
# diff -u httpd.conf.no-force-secure httpd.conf.force-secure
--- httpd.conf.no-force-secure 2018-12-29 23:30:36.820411233 -0500
+++ httpd.conf.force-secure 2018-12-29 23:31:23.880290892 -0500
@@ -552,6 +552,8 @@
     ServerAlias ftp.mydomain.us mail.mydomain.us office.mydomain.us proxy.mydomain.us wpad.mydomain.us www.mydomain.us
 
     DocumentRoot         /home/e-smith/files/ibays/wordpress/html
+    RewriteEngine on
+    RewriteRule ^/(.*|$) https://%{HTTP_HOST}/$1 [R,L]
     ScriptAlias /cgi-bin /home/e-smith/files/ibays/wordpress/cgi-bin
     Alias       /files   /home/e-smith/files/ibays/wordpress/files
 
@@ -1120,6 +1122,7 @@
 #------------------------------------------------------------
 
 <Directory /home/e-smith/files/ibays/wordpress/html>
+    SSLRequireSSL
     Options None
     Options +Indexes
     Options +FollowSymLinks

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: set reverse proxy to web service
« Reply #13 on: December 30, 2018, 07:38:07 PM »
I think if you check (and I can't for a few days) the force https adds a redirect section to the port 80 http section.

I think there is a bug on redirection for force ssl because it doesn't really kill http access but just default redirects.

Eg

Host :80
Do this
Do that
Let through letsencrypt if enabled
Etc < your bit here
Else redirect/rewrite to https

You can add a http fragment at about 40acme like letsncrypt that will allow http access. See how it works in letsencrypt for clues.

Ultimately, you already know the real answer is to rewrite your client for https....

What is it actually written in? Is it a compiled app?
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: set reverse proxy to web service
« Reply #14 on: December 30, 2018, 10:06:25 PM »
Ultimately, you already know the real answer is to rewrite your client for https....

Yup. No question.

I wrote the web service in python and the web service client using a RAD tool called Omnis Studio. That work began in 2004. The system is included in desktop applications built and deployed by me and by other developers. In some cases, there are deployments that are no longer actively developed but still in use. I don't want to force the end users to invest in an update for applications that have no other support.

The web service has been updated over the years and there are at least three versions in the wild. So, in a sense, I am doing this as a courtesy.

I will look at lensencrypt and the "40acme" idea. Together with the delta presented by mmccarn, this gives me another option for greater control.

About the bug regarding https redirect, what should a server do beyond forcing the redirect? Is the idea that if force ssl is asserted the server should not redirect but refuse?
- Mark