Koozali.org: home of the SME Server

Solved - letsencrypt challenge not completing

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #45 on: October 13, 2017, 01:04:11 PM »
Thx RequestedDeletion for sharing.

Thinking out loud: would this make any sense?

# config setprop letsencrypt http_proxy=http://192.168.x.x:3128/
and
# config setprop letsencrypt https_proxy=http://192.168.x.x:3128/

Any thoughts, maybe?


Probably not. First I don't think there is any code that would check the letsencrypt key for a proxy setting and secondly the issue seems to be with your firewall proxy mangling stuff, not SME itself.

Remember that SME has it's own http proxy and that is what the settings are for.

As a thought, do you have the squid proxy enabled at all ? That may interfere (guessing here)

Code: [Select]
config show squid
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #46 on: October 13, 2017, 01:25:37 PM »
# config show squid

squid=service
    EnforceSafePorts=no
    SafePorts=21,70,80,81,119,210,443,563,980,1024-65535
    TCPPort=3128
    TCPProxyPort=80:3128
    TransparentPort=3128
    access=private
    status=disabled

Quote
Probably not. First I don't think there is any code that would check the letsencrypt key for a proxy setting and secondly the issue seems to be with your firewall proxy mangling stuff, not SME itself.

Remember that SME has it's own http proxy and that is what the settings are for.

I don't know... But there are settings for clamav and yum for an external proxy. https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#ClamAV_.2F_freshclam

Why not also for letsencrypt? As HF pointed already out obviously letsencrypt should know about an (external) proxy.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #47 on: October 13, 2017, 02:57:39 PM »
# config show squid

squid=service
    access=private
    status=disabled

So squid is disabled. One thing out of the way.

A simple test might be to just try a simple wget for the URL at the Server CLI and see if it throws any errors anywhere either on the server or the firewall proxy e.g.

Code: [Select]
wget https://acme-v01.api.letsencrypt.org/directory
Quote
I don't know... But there are settings for clamav and yum for an external proxy. https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#ClamAV_.2F_freshclam

Why not also for letsencrypt? As HF pointed already out obviously letsencrypt should know about an (external) proxy.

No, HSF pointed out some issues other people had when they were behind a proxy. He never said letsencrypt should know about this.

And why not? Because it was never coded (if it had been then there would have been config settings in the wiki no doubt). We don't all have crystal balls when we code, and don't always know the exact configuration of every single server out there. Of course you could have written the contrib yourself and then you'd have done it right ?

If you had tried to read the dehydrated script you would see that the URLs are hard coded by default. Fortunately I went and did some reading for you:

Code: [Select]
  # Default values
  CA="https://acme-v01.api.letsencrypt.org/directory"

This can be overridden in the config file - here's the template:

Code: [Select]
    if ( $letsencryptStatus eq 'test' ) {

        # Use staging directory for testing
        # Once you are sure you have the settings right then change
        $OUT .= "CA=\"https://acme-staging.api.letsencrypt.org/directory\"\n";
    }

    elsif ( $letsencryptStatus ne 'test' ) {

   # Real server - default setting in the the main file
   # Only use this once you are sure things are OK or you will hit a rate limit.
        $OUT .= "CA=\"https://acme-v01.api.letsencrypt.org/directory\"\n";
    }

So I guess you could try adding the proxy port to the end of the URL. To test, make a copy of the config template and put it in templates-custom, and then add your proxy port to the end of the CA URL.

If that succeeds you could try writing some code to fix it and add a NFR bug for a Proxy key being added to smeserver-letsencrypt. Or ask the developer of dehydrated to add a setting......

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #48 on: October 13, 2017, 10:07:06 PM »
Quote
No, HSF pointed out some issues other people had when they were behind a proxy. He never said letsencrypt should know about this.

Misunderstanding - I didn't ment to say that HF said that letsencrypt should know about a proxy.

I read this: https://community.letsencrypt.org/t/letsencyrpt-via-proxy/4317/4

I understood, that letsencrypt does understand a http_proxy environment variable. Further I read that setting both http_proxy and https_proxy works for chriswheeler for a computer behind a forward proxy.

Did I get this wrong?
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #49 on: October 13, 2017, 10:40:48 PM »
Quote
A simple test might be to just try a simple wget for the URL at the Server CLI and see if it throws any errors anywhere either on the server or the firewall proxy e.g.

Code: [Select]

wget https://acme-v01.api.letsencrypt.org/directory

This brings up:

# wget https://acme-v01.api.letsencrypt.org/directory
--2017-10-13 22:13:45--  https://acme-v01.api.letsencrypt.org/directory
Auflösen des Hostnamen »acme-v01.api.letsencrypt.org«.... 104.122.85.235, 2a02:26f0:6a:293::3d5, 2a02:26f0:6a:280::3d5
Verbindungsaufbau zu acme-v01.api.letsencrypt.org|104.122.85.235|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 561 [application/json]
In »»directory«« speichern.

100%[======================================>] 561         --.-K/s   in 0s     

2017-10-13 22:13:46 (94,4 MB/s) - »»directory«« gespeichert [561/561]

# less directory
{
  "P-9cvapeg7E": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}


Quote
So I guess you could try adding the proxy port to the end of the URL

Can you give an example? At the end just like i.e. "CA=\"https://acme-v01.api.letsencrypt.org/directory:3128\"\n";
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #50 on: October 14, 2017, 12:56:18 AM »
Misunderstanding - I didn't ment to say that HF said that letsencrypt should know about a proxy.

I read this: https://community.letsencrypt.org/t/letsencyrpt-via-proxy/4317/4

I understood, that letsencrypt does understand a http_proxy environment variable. Further I read that setting both http_proxy and https_proxy works for chriswheeler for a computer behind a forward proxy.


Don't confuse letsencrypt with the dehydrated script. Dehydrated is a bash script to generate letsencrypt certs.
Dehyrated doesn't know about env vars as it stands. Not sure how you would set them in bash on SME CLI.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #51 on: October 14, 2017, 01:00:48 AM »
This brings up:
HTTP Anforderung gesendet, warte auf Antwort... 200 OK

My german isn't great but that's 200 OK. Seems you can get the adress ok.

Quote
Can you give an example? At the end just like i.e. "CA=\"https://acme-v01.api.letsencrypt.org/directory:3128\"\n";

Something like that. Add it, expand the template and try it.....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #52 on: October 15, 2017, 10:24:16 AM »
This seems to work:

I added in the script dehydrated

export http_proxy=http://ip-of-the-proxy:port-of-the-proxy, and
export https_proxy=http://ip-of-the-proxy:port-of-the-proxy

i.e.

export http_proxy=http://192.168.92.100:3128 and
export https_proxy=http://192.168.92.100:3128

the result of
# dehydrated -c -x is

# INFO: Using main config file /etc/dehydrated/config
Processing ivbonline.de with alternative names: mail.xxx.de saturn.xxx.de www.xxx.de
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Dec 31 08:26:36 2017 GMT (Longer than 30 days). Ignoring because renew was forced!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for xxx.de...
 + Already validated!
 + Requesting challenge for mail.xxx.de...
 + Already validated!
 + Requesting challenge for saturn.xxx.de...
 + Already validated!
 + Requesting challenge for www.xxx.de...
 + Already validated!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
Set up modSSL db keys
Signal events
All complete
 + Done!

I'll template this and it should be alright.

edit: is there a way to template a bash sript?

regards,
stefan
« Last Edit: October 15, 2017, 12:00:47 PM by SchulzStefan »
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #53 on: October 15, 2017, 02:47:14 PM »
This seems to work:

I added in the script dehydrated

export http_proxy=http://ip-of-the-proxy:port-of-the-proxy, and
export https_proxy=http://ip-of-the-proxy:port-of-the-proxy

i.e.

export http_proxy=http://192.168.92.100:3128 and
export https_proxy=http://192.168.92.100:3128


Well that's good to know.


Quote
I'll template this and it should be alright.

edit: is there a way to template a bash script?

You can template just about anything if you want to. HOWEVER.....

The dehydrated script itself is not templated at all, so if you have added something directly to the script it will get overwritten if the script is updated (which is quite likely at some point)

It would therefore be better if you tried to modify the config file which IS templated. Try adding your export settings in there and see what happens.

Try editing /etc/e-smith/templates/etc/dehydrated/config/10Default and add the proxy port and see what happens - this would be a much easier fix.

Something like this may do it (no guarantees - I am guessing here!) :

$OUT .= "export http_proxy=http://ip-of-the-proxy:port-of-the-proxy"\n";
$OUT .= "export https_proxy=https://ip-of-the-proxy:port-of-the-proxy"\n";

If that works it would be trivial to add a 'Proxy' key to the letsencrypt config key and a bit of code to the template file.

The only thing I am not sure is whether this permanently sets the proxy variables for your server. That may potentially lead to other issues so needs checking.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #54 on: October 20, 2017, 05:12:34 PM »
using wget for debugging was not the best solution as dehydrated make no use of it, but rather curl. wget is also a tool that is used pretty frequently and if ShulzStefan is behind a proxy there is chance he has already a .wgetrc file set to allow him to download :
~/.wgetrc file:
Code: [Select]
use_proxy=on
http_proxy=http://192.168.92.100:3128
https_proxy=http://192.168.92.100:3128


the way I would have tested would rather be :
Code: [Select]
curl https://acme-v01.api.letsencrypt.org/directory
see what happen

then, if it failed try:
Code: [Select]
export http_proxy=http://192.168.92.100:3128
export https_proxy=http://192.168.92.100:3128
curl https://acme-v01.api.letsencrypt.org/directory

if it worked then I would have simply add the two lines to /etc/profile or better  by creating a file in /etc/profile.d/httpproxy.sh that could be templated later
/etc/profile.d/httpproxy.sh file:
Code: [Select]
export http_proxy=http://192.168.92.100:3128
export https_proxy=http://192.168.92.100:3128

then I would logout the user and login again to have our previous temporary  export cleared and have the  profile file used, and finally test again
Code: [Select]
curl https://acme-v01.api.letsencrypt.org/directory
if it works then the direct call of the following command should work too
Code: [Select]
dehydrated -c
However an issue may still remain, because we use cron !
/etc/cron.daily/letsencrypt
Code: [Select]
#!/bin/sh

/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

we should make this script login or load profiles ...

if we were using a regular cron line this would be:
Code: [Select]
3 12 */1 * * bash -l -c '/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null'
or
Code: [Select]
3 12 */1 * * . /etc/profile; /usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

here i think we could simply add to the file the call
Code: [Select]
#!/bin/sh
. /etc/profile
/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

IMHO this should not be a simple quick fix at a contrib level but rather a core work :

there are a lot of work to do on upstream proxy for SME :
- possible configuration : https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration
- yum not using the upstream proxy : https://bugs.contribs.org/show_bug.cgi?id=2407
- yum not using squid : https://bugs.contribs.org/show_bug.cgi?id=542
- make upstream proxy configurable with panel : https://bugs.contribs.org/show_bug.cgi?id=1797

SchulzStefan,
have you ever set the SquidParent config key on this server ? Any issue with using yum behind your proxy ? Have you ever do the things sugegsted here https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration

for me the big change would be to have this SquidParent entry linked to a /etc/profile.d/httpproxy.sh template  and adapt from there.

Code: [Select]
mkdir -p /etc/e-smith/templates-custom/etc/profile.d/httpproxy.sh
echo '
    $OUT = "";
    if (defined $SquidParent && $SquidParent && $squid{"status"} =="disabled" )
    {
        my $port = $SquidParentPort || "3128";
        $OUT .= "export http_proxy=$SquidParent:$port\n";
        $OUT .= "export https_proxy=$SquidParent:$port \n";
    }
' >/etc/e-smith/templates-custom/etc/profile.d/httpproxy.sh

expand-template /etc/profile.d/httpproxy.sh
this would allow when a remote proxy is set and local squid disabled to have all user have it configured and used. This should solve issue with yum , curl wget..... as long as the script is run with a logged in user ... Only a few workaround could have to be added for cron called script. Also we might need to test how scripts called via manager react.


----
other sources :
https://stackoverflow.com/questions/11211705/setting-proxy-in-wget
https://unix.stackexchange.com/questions/27289/how-can-i-run-a-cron-command-with-existing-environmental-variables


Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #55 on: October 21, 2017, 12:17:29 AM »
Jean-Philippe Pialasse,

I'll try to answer your questions:

Quote
there is chance he has already a .wgetrc file set to allow him to download

There's no .wgetrc file in the directory of root. If it should be elsewhere, please advise.

Quote
[the way I would have tested would rather be :
Code: [Select]

curl https://acme-v01.api.letsencrypt.org/directory

# curl https://acme-v01.api.letsencrypt.org/directory

brings up:

{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert",
  "xvh60byUoYE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"

Don't know if this is sucess or a failure?

Quote
SchulzStefan,
have you ever set the SquidParent config key on this server ? Any issue with using yum behind your proxy ? Have you ever do the things sugegsted here https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration

# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    Proxy=http://192.168.100.10:3562/
    RandomDelay=120
    check4updates=daily
    status=enabled

and:

# config show clamav
clamav=service
    ArchiveBlockEncrypted=no
    ArchiveBlockMax=no
    ArchiveMaxCompressionRatio=300
    Checks=24
    DNSDatabaseInfo=current.cvd.clamav.net
    DatabaseMirror=db.local.clamav.net
    Debug=no
    DetectBrokenExecutables=no
    FilesystemScan=weekly
    FilesystemScanExclude=/proc,/sys,/usr/share,/var
    FilesystemScanFilesystems=/home/e-smith/files
    FilesystemScanReportTo=admin
    FilesystemScanUnofficialSigs=no
    Foreground=yes
    HTTPProxyPassword=
    HTTPProxyPort=3562
    HTTPProxyServer=192.168.100.10
    HTTPProxyUsername=
    HeuristicScanPrecedence=no
    IdleTimeout=60
    LeaveTemporaryFiles=no
    LogClean=no
    LogFileUnlock=yes
    LogTime=no
    LogVerbose=yes
    MaxAttempts=6
    MaxConnectionQueueLength=30
    MaxDirectoryRecursion=20
    MaxFileSize=15M
    MaxFiles=1500
    MaxRecursion=8
    MaxThreads=20
    Quarantine=enabled
    QuarantineDirectory=/var/spool/clamav/quarantine
    ReadTimeout=300
    ScanArchive=yes
    ScanHTML=yes
    ScanMail=yes
    ScanOLE2=yes
    ScanPE=yes
    ScanRAR=no
    SelfCheck=1800
    ShowProxySettings=no
    ShowUpdateSettings=no
    SignaturesUpdated=unknown
    UpdateNonOfficeHrs=disabled
    UpdateOfficeHrs=disabled
    UpdateWeekend=disabled
    status=enabled

Without the proxy settings neither yum nor freshclam is working.

# config show SquidParent
SquidParent=

I didn't try this setting yet.

Right now server is working proper with my settings. You see the yum and clamav settings, and the "dirty fix" in the dehydrated script. No errors with yum, clamav and dehydrated. If I can help/test/try let me know.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #56 on: October 21, 2017, 03:36:27 AM »
Stefan,

this confirm that we have some global work to do there.
a keep it simple approach should not be to set a proxy for every service running around  *rather than* every time we step on a new thing not working.

The truth is there is not that many people using a SME behind proxy that I know so I would say leave your server as is as it works according to your need, but if you have some time in the next month I might ask you help to test some new code on a test machine aside if you are ok.

EDIT: added missing words.
« Last Edit: October 23, 2017, 04:51:04 PM by Jean-Philippe Pialasse »

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #57 on: October 23, 2017, 11:31:01 AM »
Jean-Philippe,

just tell me what to do. I'll be happy to assist.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #58 on: February 15, 2018, 12:13:39 PM »
If anyone runs in this issue:

Anacron job 'cron.daily'
/etc/cron.daily/letsencrypt:

  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Details:
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>


This fix might be helpful:

https://github.com/lukas2511/dehydrated/commit/7a0e71c6c2ccc6e98abca5ea1c7de28053e90c02

what was described here:

https://community.letsencrypt.org/t/dehydrated-caused-rate-limits-to-be-reached/52477

IMHO the fix/patch should be implemented in the sme dehydrated rpm.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: letsencrypt challenge not completing
« Reply #59 on: February 15, 2018, 12:25:06 PM »
nice catch.. please raise a bug, thank you