Koozali.org: home of the SME Server

Solved - letsencrypt challenge not completing

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Solved - letsencrypt challenge not completing
« on: June 10, 2017, 07:00:31 PM »
Good Morning,
I have been unsuccessful in getting my serve to complete the challenge for letsencrypt. Here is the output of dehydrated -c

Code: [Select]
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:unauthorized",
    "detail": "Invalid response from http://www.mercyh.org/.well-known/acme-challenge/FMTvp_V7itHCGh1yK9AjZJv2rmGawZYArY-5r3DpTX4: \"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e403 Forbidden\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eForbidden\u003c/h1\u003e\n\u003cp\"",
    "status": 403
  },
  "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/FgNcud1f0aqdQ7akFN1NyrEz6BkKLGFTHN_HlxC7cUE/43197390",
  "token": "FMTvp_V7itHCGh1yK9AjZJv2rmGawZYArY-5r3DpTX4",
  "keyAuthorization": "FMTvp_V7itHCGh1yK9AjZJv2rmGawZYArY-5r3DpTX4.266BsZK-dHl_Lk8qUZQa6lxP_cNRbxz8lP3lFEP_1Rs",
  "validationRecord": [
    {
      "url": "https://www.mercyh.org/.well-known/acme-challenge/FMTvp_V7itHCGh1yK9AjZJv2rmGawZYArY-5r3DpTX4",
      "hostname": "www.mercyh.org",
      "port": "443",
      "addressesResolved": [
        "63.245.178.234"

the httpd error_log file simply gives the following entry:

Code: [Select]
(13)Permission denied: access to /.well-known/acme-challenge/fxiFyqyaNhfd7DQVFHfl2LgUdOQr-AZOyCuC10UCK-Q denied
(13)Permission denied: access to /.well-known/acme-challenge/cVQNMIHnlXVdH0qUUpcMyQ8xlbU-4eMmT-pr6GFE7FE denied
(13)Permission denied: access to /.well-known/acme-challenge/GdfqjRvzhE4viN5pY7xQzdcEJan-veIi9FOvBxkPz68 denied

can anyone give me a pointer toward what I am doing wrong?
« Last Edit: December 10, 2019, 08:53:38 AM by TerryF »

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #1 on: June 10, 2017, 09:03:01 PM »
What are the permissions on /home/e-smith/files/ibays/Primary/html/.well-known and /home/e-smith/files/ibays/Primary/html/.well-known/acme-challenge?
......

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #2 on: June 10, 2017, 11:25:47 PM »
Also are you running the commands as root ? If not try as root.

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #3 on: June 11, 2017, 12:03:57 AM »
I knew it was simple. I had no domains hosted on Primary and somewhere down the line, the permissions on primary got skewed. I never even considered that the Primary Ibay was used here.

correcting the permissions made the challenge work perfectly.

THANKS,

Royce

guest22

Re: letsencrypt challenge not completing
« Reply #4 on: June 11, 2017, 01:26:44 AM »
I never even considered that the Primary Ibay was used here.


In my opinion (and some others) that remains a problem, letsencrypt demanding http access to the Primary domain.

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #5 on: June 11, 2017, 01:29:18 AM »
In my opinion (and some others) that remains a problem, letsencrypt demanding http access to the Primary domain.
If you don't want to give http access to your SME server (HTTPS access is OK, because SME implements a HTTP -> HTTPS redirect), you can always use DNS validation instead--but you'll have to script it yourself.
......

guest22

Re: letsencrypt challenge not completing
« Reply #6 on: June 11, 2017, 01:34:57 AM »
AFAIK and in my experience (see also comments from Stafan) http is required by letsencrypt. e.g. having forced the Primary ibay inot httpS only, letsencrypt will fail.

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #7 on: June 11, 2017, 01:48:42 AM »
AFAIK and in my experience (see also comments from Stafan) http is required by letsencrypt. e.g. having forced the Primary ibay inot httpS only, letsencrypt will fail.
From my own experience, this is incorrect.  When an ibay (including Primary) is set to HTTPS-only on SME, the system implements a HTTP -> HTTPS redirect--if you connect to http://yourhostname, it's redirected to https://yourhostname.  The Let's Encrypt servers will follow this redirect and authenticate without issue.  Now, if you have a firewall blocking access on port 80, the LE servers won't be able to validate, but that isn't an SME setting.  It also isn't generally a good idea--even if you want your server to be SSL-only, people are going to (attempt to) connect from the outside using HTTP.

But, again, if you don't want to open your SME server to web access, you can use DNS validation instead.  You'll need to write some custom template fragments for that, though.  See https://github.com/lukas2511/dehydrated/blob/master/docs/dns-verification.md and https://github.com/lukas2511/dehydrated/wiki/Examples-for-DNS-01-hooks for more information on implementing DNS validation with dehydrated.

Edit:  There's a third option, if the hostname for which you're seeking the cert resolves to a different machine: deploy the challenge file automatically to the appropriate host.  In the simplest case, the contrib can handle the necessary automation.  See https://wiki.contribs.org/Letsencrypt#Obtaining_certificates_for_a_private_SME_Server.  More complicated cases will probably need custom template fragments.
« Last Edit: June 11, 2017, 04:02:34 AM by DanB35 »
......

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #8 on: June 11, 2017, 02:36:15 AM »
I actually set the server to force SSL while I was trying to troubleshoot. The logs showed letsencrypt following the redirect but then failing with the permissions error. I made the assumption that it was following the domain direct on the server to the ibay where the site is hosted. I never considered that it was accessing the primary ibay and the paths in both the letsencrypt reply and the httpd logs were not complete. Thanks again for pointing me in the right direction!!!

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #9 on: June 11, 2017, 02:33:12 PM »
Letsencrypt effectively HAS to follow a http redirect or logically you could not disable http access.

Remember that (if you are sensible) http would be disabled once you have your certs installed, but the auto renew still works....

All my boxes with letsencrypt have http disabled/redirected and I have zero issues.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #10 on: June 11, 2017, 08:55:33 PM »
To be completely clear, every time Let's Encrypt issues a certificate (which will be about every 60 days if you're following their recommendations), they must validate that you control every hostname for which you're seeking a cert.  Their servers support three methods of validation:
  • Look for a specific file under http://$FQDN/.well-known/acme-challenge
  • Look for a specific TLS certificate to be served when connecting to https://$FQDN
  • Look for specific DNS TXT records for each hostname sought to be validated

Dehydrated, the client that @ReetP's contrib works with, supports only the first and third challenges, and the contrib supports only the first--it wouldn't be possible to come up with a consistent set of instructions for the third, since we're all using different DNS hosts.  That means that, any time you're seeking to issue or renew a certificate, the Let's Encrypt servers must be able to connect to your hostname on port 80.  They'll happily follow a redirect, but they must still be able to connect on port 80.  If you don't want port 80 open to the Internet, yes, this is a problem for you.
......

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #11 on: June 11, 2017, 11:05:23 PM »
However HF might have a point here, why put .well-known in the Primary ibay, it could be somewhere else and still work.

We could move it to /var/www/.well-known and make the httpd.conf template fragment to point http://mydomain.com/.well-known to it.
This would avoid customized chmod or chown on the Primary ibay to prevent apache to read the content of .well-known.
Also it would keep it away from the eyes of users not knowing what this is.

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #12 on: June 11, 2017, 11:14:27 PM »
If you have more than domain they may point to different ibays....

I need to check my configs to demonstrate this but I think this proposal may break some stuff for me at least.

I think it is the due to a situation Tony showed me some while back regarding domains but need to check my configs.

Also if we start moving stuff to /var/www then either it should all go in there, or stay the way we are. Splitting locations could confuse some just as easily as helping others
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #13 on: June 11, 2017, 11:26:30 PM »
why put .well-known in the Primary ibay,
Because the Primary ibay is the default web root of the SME Server, and therefore the logical place to put anything that's going to be served out of the web root.  It doesn't have to go there, of course--neither dehydrated nor the Let's Encrypt servers care where on the filesystem the challenge file goes, as long as it's served in response to a request for /.well-known/acme-challenge/whatever--but it does seem like the most logical place to put it.  Why not put it there?
......

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #14 on: June 11, 2017, 11:37:52 PM »
If you have more than domain they may point to different ibays....
if a domain point to an ibay else than Primary, then it is already not pointing tothe Primary and hence there is already a fragment making the url well-known to point to the folder in the ibay, it could then point to somewhere else.


I need to check my configs to demonstrate this but I think this proposal may break some stuff for me at least.
go ahead, indeed i feel some tweaks might, at least with use of the hook script migh break

I think it is the due to a situation Tony showed me some while back regarding domains but need to check my configs.

Also if we start moving stuff to /var/www then either it should all go in there, or stay the way we are. Splitting locations could confuse some just as easily as helping others

well here we have a software indepedant from user data, and with nothing to backup from it, its place should be naturally out of /home/. Another example is horde that sit for a while in /home, but should be moved to a standard path, and has been for 10. Phpmyadmin is not either in /home....

So user accessible web data are kept in /home/e-smith, while software path, not needed to be accessible by the user are out of it.

Because the Primary ibay is the default web root of the SME Server, and therefore the logical place to put anything that's going to be served out of the web root.  It doesn't have to go there, of course--neither dehydrated nor the Let's Encrypt servers care where on the filesystem the challenge file goes, as long as it's served in response to a request for /.well-known/acme-challenge/whatever--but it does seem like the most logical place to put it.  Why not put it there?
Because this does not need to be accessible by the user. It is only for automatized access for cert renewal. I think it was placed in the Primary  ibay by John in the first time only because it was conveniant while developing the contribution and did avoid to play with the handling of virtual path vs absolute server path, but when it was time to handle also domain pointing to other ibays he had to play with alias or other means to point to the current absolute path of .well-known.

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #15 on: June 11, 2017, 11:40:28 PM »
Yeah, either way you'll need an alias to point everything to the right filesystem path.  I don't have strong feelings either way, but "the user doesn't need to see it" doesn't seem like a strong reason to move it somewhere else--especially since all that's there are two empty directories.
......

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #16 on: June 12, 2017, 12:04:53 AM »
I think the scenario I have is liek this:

mydomain.co.uk=domain
    Content=Primary
    Description=Primary domain
    Nameservers=localhost
    Removable=no
    SystemPrimaryDomain=yes
    letsencryptSSLcert=enabled

crm.mydomain.co.uk=domain
    Content=crm
    Description=CRM direct
    Nameservers=localhost
    letsencryptSSLcert=enabled

I think that is done to fool the system to set the crm ibay as the 'Primary' ibay for that domain with the correct doc_root etc so you can go to crm.mydomain.co.uk and not mydomain.co.uk/crm

However, it seems from looking at the config that the port 80 alias for that domain does redirect to 'Primary'

So the suggested change may not affect it.

However, I agree with Dan - I can't see a good enough reason to change it.

Also I don't understand this :

"This would avoid customized chmod or chown on the Primary ibay to prevent apache to read the content of .well-known."

The only 'customisation' is during install (this is in the spec file) and should then not require changing? No one has picked this up before. If it does then it is just a change in the spec file

chmod -R 0775  /home/e-smith/files/ibays/Primary/html/.well-known
chown -R apache:shared /home/e-smith/files/ibays/Primary/html/.well-known

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #17 on: June 12, 2017, 12:52:16 AM »
I guess I am a weird duck... I just installed on a second server at another location. (These are all many generations of SME old and have been through multiple admins and upgrades with restored backups) would you believe I got the same error? Some former website designer had installed WordPress on this one and redirected the primary domain to another ibay. He had either used the primary ibay as a person storage space or for some kind of testing and made himself the owner and used his user as the group on the HTML folder... I think the suggestion of fixing the ownership with the script is ok but might risk breaking someone's non standard usage of that location...

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #18 on: June 12, 2017, 02:09:21 AM »
Yeah, either way you'll need an alias to point everything to the right filesystem path.  I don't have strong feelings either way, but "the user doesn't need to see it" doesn't seem like a strong reason to move it somewhere else--especially since all that's there are two empty directories.
Well it was only a polite way to say to prevent the user to mess with the validation folder, liker here for instance by playing with rights of a parent folder on the filesystem.
Further more this is not a place where a user is supposed to fo anything manually, like for instance the apache icon folder, or the phpmyadmin contrib... so keeping it out of the user sandbox in /home/e-smith/files is a way to avoid a mess and keeping things just working.

It is pretty frequent people will start messing with chown and chmod in ibays....


Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #19 on: June 12, 2017, 02:19:18 AM »
I think the scenario I have is liek this:

mydomain.co.uk=domain
    Content=Primary
    Description=Primary domain
    Nameservers=localhost
    Removable=no
    SystemPrimaryDomain=yes
    letsencryptSSLcert=enabled

crm.mydomain.co.uk=domain
    Content=crm
    Description=CRM direct
    Nameservers=localhost
    letsencryptSSLcert=enabled

I think that is done to fool the system to set the crm ibay as the 'Primary' ibay for that domain with the correct doc_root etc so you can go to crm.mydomain.co.uk and not mydomain.co.uk/crm

However, it seems from looking at the config that the port 80 alias for that domain does redirect to 'Primary'

So the suggested change may not affect it.

However, I agree with Dan - I can't see a good enough reason to change it.

Also I don't understand this :

"This would avoid customized chmod or chown on the Primary ibay to prevent apache to read the content of .well-known."

The only 'customisation' is during install (this is in the spec file) and should then not require changing? No one has picked this up before. If it does then it is just a change in the spec file

chmod -R 0775  /home/e-smith/files/ibays/Primary/html/.well-known
chown -R apache:shared /home/e-smith/files/ibays/Primary/html/.well-known

The thing is that apache will not be able to access to the folder as soon as somebody play with the folder pr any parent folder. Let's say i do not use the primary folder, so i decided to chmod go-rwx /home/e-smith/files/ibays/Primary/html to prevent users to sneak there. Then apache is not able to serve anymore .well-known even if its rights and ownership are right, just because the parent directory does not let him in.

Again this was a very easy and convenient way to put it there when developping the contrib, but this is not the best place, would you put the apache icon folder in Primary? Would you put the folder of phpmyadmin and the one of phpldapadmin? Yea it could work there, but it has no need to be in the way of what should be the primary website. It would quickly start to be messy, and you never know what subtility the webmaster or admin will have the idea to do :
- deleteting the folder
- messing the rights
- messing the parent directory right
- hacking the content if any
-  let your imagination get wild for more possibilities...

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #20 on: June 12, 2017, 06:38:04 AM »
I would like to make this comment about the letsencrypt contrib. This work is very beneficial to the koozali project. I want to give the contrib devs a huge thumbs up and would encourage inclusion into the base code as soon as is reasonable.

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #21 on: June 12, 2017, 11:03:03 AM »
-  let your imagination get wild for more possibilities...

I have no imagination. Regrettably you cannot allow for every user scenario. They often do the strangest of things.

When I wrote the contrib you all had the opportunity to comment on this sort of thing, and no one did.

Please go ahead and do whatever you want - I'm well beyond caring on this.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #22 on: June 12, 2017, 06:19:22 PM »
@ReetP,

Your work goes well beyond my technical abilities. I thank you for allowing me to get away from self signed certificates with an easy install and a few simple settings. I think we have two options here.

#1. change the contrib to relocate the /.well-known paths to a location that is not likely to be tampered with by some ignoramus like myself....

#2. document the issue in the wiki with the exact errors that show and the chown and chmod commands to set all folders in the path to correct ownership and rights.


I am capable of doing the second part but not the first. Would you like me to add to the wiki under the troubleshooting section? Would some of the rest of you be willing to look over my documentation to be sure I do not have it incorrect?

One other issue that would be a possible addition to the wiki is that the following command:

Code: [Select]
config setprop letsencrypt configure all
will likely cause cert generation to fail. When adding a domain, SME automatically adds the ftp. proxy. wpad. hostnames, many folks will not actually point public DNS A records at these names. If so the challenge comes back incomplete and the cert is not issued.


Royce
« Last Edit: June 12, 2017, 06:25:32 PM by mercyh »

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: letsencrypt challenge not completing
« Reply #23 on: June 12, 2017, 06:40:01 PM »
Royce, go for the wiki, thank you

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #24 on: June 12, 2017, 08:19:25 PM »
Royce,

the simple answers are

Always test first
Read comprehensively
Don't make assumptions
Follow the SME way. Use the built in stuff where possible and expect issues if you do non standard stuff. If you take over a server, understand it completely before you dive in, and expect the unexpected!

:-)

I am no coder. I did what I could and it works for me, but comes with no warranty.

It needs a greater mind than mine to do more. It is now in SME CVS so anyone cam jump in.

Rgds
John
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #25 on: June 12, 2017, 08:38:15 PM »
 :cool:

yep, so I am going to give everyone more to read... :)


Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #26 on: June 12, 2017, 09:11:16 PM »
:cool:

yep, so I am going to give everyone more to read... :)

Good man.....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #27 on: June 13, 2017, 06:14:37 AM »
Hey guys,

If you could do a quick proof read of the wiki changes at https://wiki.contribs.org/Letsencrypt#Errors, I would appreciate it. I would hate to steer somebody totally wrong...


Offline gwag

  • 16
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #28 on: June 13, 2017, 07:29:12 PM »
 Enable Test Mode......

If this runs without errors, try to connect to your server-manager page. You should see an error that the security certificate wasn't issued by a trusted certification authority; this is perfectly normal. However, there should be a certificate, it should include all the hostnames you wanted included, and it should be valid for the next ninety days. If this was successful, proceed to production.

does it really make changes to your server manager page in test mode?

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #29 on: June 13, 2017, 07:40:08 PM »
No, but it does issue a certificate to the server that is different from the SME self signed certificate. You can then go into server manager (which forces an https connection) and look at the certificate to verify that a new certificate has been issued that is not self signed by the server.

Once you have the new certificate active, you have verified that everything is working and can then switch over to production and get an actual trusted certificate from them......

In Internet Explorer, from the https://mail.myserver.com/server-manager page, click the padlock icon..

(see attachement)

and go to "view certificate" You should see who the certificate is issued by....

« Last Edit: June 13, 2017, 07:48:35 PM by mercyh »

Offline pcdoc

  • *
  • 56
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #30 on: June 20, 2017, 06:23:49 PM »
Thanks for your work documenting Royce.

However, I have another issue. Everything ran well for first issue of certificates, but the renewal after 60 days (29 days before cert dies) it comes up with a Status 500 error when requesting a new certificate.

Any thoughts on where to look?

Cheers,

Matt
There are 10 types of people in this world,
   Those that know binary, and those who don't!

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #31 on: June 20, 2017, 06:35:52 PM »
Can you post the actual output of the script? I would suggest you start a new thread ad this one might get pretty jumbled if we keep putting new/different problems in it.


Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #32 on: June 20, 2017, 10:09:56 PM »
but the renewal after 60 days (29 days before cert dies) it comes up with a Status 500 error when requesting a new certificate.
Most likely that results from a temporary issue on the Let's Encrypt end.  If it recurs (the renewal should run again tomorrow), I'd agree with @mercyh: start a new thread, and include the exact error output, and we'll see what else we can do to troubleshoot.
......

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #33 on: June 20, 2017, 10:14:29 PM »
If this does complete correctly tomorrow when cron reruns the renew script, be sure to let us know. It would be good to get it in the wiki that if you see this error, you should let the script rerun before digging into the issue.....

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #34 on: June 23, 2017, 08:25:15 PM »
Pcdoc,

Could you update us on the status of your error 500 on certificate regeneration?

Thanks,

Royce

Offline pcdoc

  • *
  • 56
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #35 on: June 29, 2017, 03:31:08 PM »
Sorry I haven't posted back the results. Have been unwell.

Anyway, reporting that I am still getting the Status 500 error every day. Down to 29 days now, so need to get back into it.

I ran >"dehydrated -c" tonight to see how it would go... not well.

Output follows

Quote
+ Requesting challenge for aap.domainname.org.au...
  + ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 429)

Details:
{
  "type": "urn:acme:error:rateLimited",
  "detail": "Error creating new authz :: too many currently pending authorizations",
  "status": 429
}

thoughts?
There are 10 types of people in this world,
   Those that know binary, and those who don't!

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #36 on: June 29, 2017, 04:18:46 PM »
https://letsencrypt.org/docs/rate-limits/

I wonder if you have multiple cron jobs trying to renew the certificate. Where are you seeing the 500 error?

dehydrated is giving you the 429 error which is something entirely different.....

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: letsencrypt challenge not completing
« Reply #37 on: June 29, 2017, 04:19:08 PM »
Rate limits are documented here:

https://letsencrypt.org/docs/rate-limits/

In particular, this looks relevant:

"You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them. Please utilize our staging environment if you’re developing an ACME client'.
« Last Edit: June 29, 2017, 04:20:40 PM by CharlieBrady »

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: letsencrypt challenge not completing
« Reply #38 on: June 29, 2017, 04:27:33 PM »
I am curious about your domain name... I realize you have obfuscated it but....

aap.domainname.org.au

quote from letsencrypt...

Quote
The main limit is Certificates per Registered Domain (20 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk. We use the Public Suffix List to calculate the registered domain.

If you are hosting a subdomain of domainname.org.au and there are many other subdomains in use, you may be hitting this rate limit and it may be out of your control to fix the problem.....

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #39 on: October 01, 2017, 01:47:42 AM »
Not sure if this is relevant to the problem of pcdoc.

Letsencrypt says, they need access to port 80 and do follow re-directions. I do confirm this. As well, as the port 443 has also to be opened. And both ports also in outbound. For me it was not easy to point out, that first choice for letsencrypt is AAAA in the DNS records of the domain and hosts, you want to cert. This point is quite important. If not all hosts you want to cert are set up with an AAAA record dehydrated will fail. The safe way (for me) is to temporarely delete all AAAA records and re-run manually dehydrated -c.

This might depend on my installation as my server is running behind a firewall. If so, there are a few more cavecats. I.e. I have to disable a port-forward rule to the webproxy, for the manually renewal of the certs. The comment in the opnsense forum is: "Status 400 ist ein Syntaxfehler. Vermutlich macht der Client was, was der Proxy nicht versteht... https://forum.opnsense.org/index.php?topic=5201.0" Translation: Syntax error. Probably the client is doing something the proxy does not understand. Worst case would be, that the client does not truely act with the HTTP-protocol. Therefore the proxy rejects the connection ..." I didn't check with i.e. wireshark the packets, so I don't know what really happens.

The interesting thing for me is that the letsencrypt people say, that if .well-known/acme-challenge is accessable through port 80 (I ran several test with different browsers from different IP's to access a file I created in acme-challenge), with and without redirection to ssl, dehydrated should run fine and the renewal should work. Not working in my case. I am able to access my test file through my firewall - but dehydrated fails mostly with a timeout error. All rules on my firewall seems to be correct. Dehydrated is only running through if I bypass the proxy. Besides the IP6 settings in the DNS records. Maybe it's an opnsense issue, I don't know.

One thing is for sure - you need to have full access not only to your server...

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #40 on: October 01, 2017, 03:10:34 AM »
For me it was not easy to point out, that first choice for letsencrypt is AAAA in the DNS records of the domain and hosts, you want to cert. This point is quite important. If not all hosts you want to cert are set up with an AAAA record dehydrated will fail.
Let me clarify/emphasize this point.  If any hostname for which you're seeking a cert has an AAAA record (i.e., an IPv6 record), the Let's Encrypt servers will try to connect to that IPv6 address.  If that connection attempt fails, then the validation process will fail--they won't fall back to IPv4 and try again.

To the best of my knowledge, SME9 doesn't support IPv6 (I don't know if SME10 will or not).  So, if you're trying to validate against an SME9 server, your solution of deleting the AAAA records before getting the cert is a good one.

As to the redirect, I don't know what's happening for you.  In my installation, all http is redirected to https, and I don't have trouble obtaining or renewing certs.
......

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #41 on: October 01, 2017, 09:59:03 PM »
Quote
Letsencrypt says, they need access to port 80 and do follow re-directions. I do confirm this.

Redirection is working, as I said. There seems to be a problem if the SME is behind a firewall and an enabled webproxy. At least for an OPNsense and my configuration. Maybe I misconfigured something, but if so, I don't know what. Everything else is running and behaving as expected. For dehydrated I have to disable port forward from port 80 to the proxy port. Why? This is beyond my knowledge...

Regards,
stefan

And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)


Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #43 on: October 03, 2017, 05:17:31 PM »
Thinking out loud here....

If it is an issue via the Opnsense proxy, could you not get Opnsense to generate all the required certs and then script copying them to the right places ?

Just a thought.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #44 on: October 13, 2017, 12:50:04 PM »
maybe this helps https://community.letsencrypt.org/t/letsencyrpt-via-proxy/4317/5 or this https://community.letsencrypt.org/t/solved-proxy-settings-in-order-to-use-letsencrypt-auto-behind-a-proxy/5465/5

Thx RequestedDeletion for sharing.

#config show letsencrypt brings up:
letsencrypt=service
    ACCEPT_TERMS=yes
    configure=none
    email=admin@xxx.de
    hookScript=disabled
    status=enabled

Thinking out loud: would this make any sense?

# config setprop letsencrypt http_proxy=http://192.168.x.x:3128/
and
# config setprop letsencrypt https_proxy=http://192.168.x.x:3128/


A rush try fails (as expected) with:

/sbin/e-smith/db dbfile setprop key prop1 val1 [prop2 val2] [prop3 val3] ...

Any thoughts, maybe?

regards,
stefan
« Last Edit: October 13, 2017, 12:54:15 PM by SchulzStefan »
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #45 on: October 13, 2017, 01:04:11 PM »
Thx RequestedDeletion for sharing.

Thinking out loud: would this make any sense?

# config setprop letsencrypt http_proxy=http://192.168.x.x:3128/
and
# config setprop letsencrypt https_proxy=http://192.168.x.x:3128/

Any thoughts, maybe?


Probably not. First I don't think there is any code that would check the letsencrypt key for a proxy setting and secondly the issue seems to be with your firewall proxy mangling stuff, not SME itself.

Remember that SME has it's own http proxy and that is what the settings are for.

As a thought, do you have the squid proxy enabled at all ? That may interfere (guessing here)

Code: [Select]
config show squid
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #46 on: October 13, 2017, 01:25:37 PM »
# config show squid

squid=service
    EnforceSafePorts=no
    SafePorts=21,70,80,81,119,210,443,563,980,1024-65535
    TCPPort=3128
    TCPProxyPort=80:3128
    TransparentPort=3128
    access=private
    status=disabled

Quote
Probably not. First I don't think there is any code that would check the letsencrypt key for a proxy setting and secondly the issue seems to be with your firewall proxy mangling stuff, not SME itself.

Remember that SME has it's own http proxy and that is what the settings are for.

I don't know... But there are settings for clamav and yum for an external proxy. https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#ClamAV_.2F_freshclam

Why not also for letsencrypt? As HF pointed already out obviously letsencrypt should know about an (external) proxy.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #47 on: October 13, 2017, 02:57:39 PM »
# config show squid

squid=service
    access=private
    status=disabled

So squid is disabled. One thing out of the way.

A simple test might be to just try a simple wget for the URL at the Server CLI and see if it throws any errors anywhere either on the server or the firewall proxy e.g.

Code: [Select]
wget https://acme-v01.api.letsencrypt.org/directory
Quote
I don't know... But there are settings for clamav and yum for an external proxy. https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#ClamAV_.2F_freshclam

Why not also for letsencrypt? As HF pointed already out obviously letsencrypt should know about an (external) proxy.

No, HSF pointed out some issues other people had when they were behind a proxy. He never said letsencrypt should know about this.

And why not? Because it was never coded (if it had been then there would have been config settings in the wiki no doubt). We don't all have crystal balls when we code, and don't always know the exact configuration of every single server out there. Of course you could have written the contrib yourself and then you'd have done it right ?

If you had tried to read the dehydrated script you would see that the URLs are hard coded by default. Fortunately I went and did some reading for you:

Code: [Select]
  # Default values
  CA="https://acme-v01.api.letsencrypt.org/directory"

This can be overridden in the config file - here's the template:

Code: [Select]
    if ( $letsencryptStatus eq 'test' ) {

        # Use staging directory for testing
        # Once you are sure you have the settings right then change
        $OUT .= "CA=\"https://acme-staging.api.letsencrypt.org/directory\"\n";
    }

    elsif ( $letsencryptStatus ne 'test' ) {

   # Real server - default setting in the the main file
   # Only use this once you are sure things are OK or you will hit a rate limit.
        $OUT .= "CA=\"https://acme-v01.api.letsencrypt.org/directory\"\n";
    }

So I guess you could try adding the proxy port to the end of the URL. To test, make a copy of the config template and put it in templates-custom, and then add your proxy port to the end of the CA URL.

If that succeeds you could try writing some code to fix it and add a NFR bug for a Proxy key being added to smeserver-letsencrypt. Or ask the developer of dehydrated to add a setting......

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #48 on: October 13, 2017, 10:07:06 PM »
Quote
No, HSF pointed out some issues other people had when they were behind a proxy. He never said letsencrypt should know about this.

Misunderstanding - I didn't ment to say that HF said that letsencrypt should know about a proxy.

I read this: https://community.letsencrypt.org/t/letsencyrpt-via-proxy/4317/4

I understood, that letsencrypt does understand a http_proxy environment variable. Further I read that setting both http_proxy and https_proxy works for chriswheeler for a computer behind a forward proxy.

Did I get this wrong?
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #49 on: October 13, 2017, 10:40:48 PM »
Quote
A simple test might be to just try a simple wget for the URL at the Server CLI and see if it throws any errors anywhere either on the server or the firewall proxy e.g.

Code: [Select]

wget https://acme-v01.api.letsencrypt.org/directory

This brings up:

# wget https://acme-v01.api.letsencrypt.org/directory
--2017-10-13 22:13:45--  https://acme-v01.api.letsencrypt.org/directory
Auflösen des Hostnamen »acme-v01.api.letsencrypt.org«.... 104.122.85.235, 2a02:26f0:6a:293::3d5, 2a02:26f0:6a:280::3d5
Verbindungsaufbau zu acme-v01.api.letsencrypt.org|104.122.85.235|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 561 [application/json]
In »»directory«« speichern.

100%[======================================>] 561         --.-K/s   in 0s     

2017-10-13 22:13:46 (94,4 MB/s) - »»directory«« gespeichert [561/561]

# less directory
{
  "P-9cvapeg7E": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}


Quote
So I guess you could try adding the proxy port to the end of the URL

Can you give an example? At the end just like i.e. "CA=\"https://acme-v01.api.letsencrypt.org/directory:3128\"\n";
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #50 on: October 14, 2017, 12:56:18 AM »
Misunderstanding - I didn't ment to say that HF said that letsencrypt should know about a proxy.

I read this: https://community.letsencrypt.org/t/letsencyrpt-via-proxy/4317/4

I understood, that letsencrypt does understand a http_proxy environment variable. Further I read that setting both http_proxy and https_proxy works for chriswheeler for a computer behind a forward proxy.


Don't confuse letsencrypt with the dehydrated script. Dehydrated is a bash script to generate letsencrypt certs.
Dehyrated doesn't know about env vars as it stands. Not sure how you would set them in bash on SME CLI.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #51 on: October 14, 2017, 01:00:48 AM »
This brings up:
HTTP Anforderung gesendet, warte auf Antwort... 200 OK

My german isn't great but that's 200 OK. Seems you can get the adress ok.

Quote
Can you give an example? At the end just like i.e. "CA=\"https://acme-v01.api.letsencrypt.org/directory:3128\"\n";

Something like that. Add it, expand the template and try it.....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #52 on: October 15, 2017, 10:24:16 AM »
This seems to work:

I added in the script dehydrated

export http_proxy=http://ip-of-the-proxy:port-of-the-proxy, and
export https_proxy=http://ip-of-the-proxy:port-of-the-proxy

i.e.

export http_proxy=http://192.168.92.100:3128 and
export https_proxy=http://192.168.92.100:3128

the result of
# dehydrated -c -x is

# INFO: Using main config file /etc/dehydrated/config
Processing ivbonline.de with alternative names: mail.xxx.de saturn.xxx.de www.xxx.de
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Dec 31 08:26:36 2017 GMT (Longer than 30 days). Ignoring because renew was forced!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for xxx.de...
 + Already validated!
 + Requesting challenge for mail.xxx.de...
 + Already validated!
 + Requesting challenge for saturn.xxx.de...
 + Already validated!
 + Requesting challenge for www.xxx.de...
 + Already validated!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
Set up modSSL db keys
Signal events
All complete
 + Done!

I'll template this and it should be alright.

edit: is there a way to template a bash sript?

regards,
stefan
« Last Edit: October 15, 2017, 12:00:47 PM by SchulzStefan »
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #53 on: October 15, 2017, 02:47:14 PM »
This seems to work:

I added in the script dehydrated

export http_proxy=http://ip-of-the-proxy:port-of-the-proxy, and
export https_proxy=http://ip-of-the-proxy:port-of-the-proxy

i.e.

export http_proxy=http://192.168.92.100:3128 and
export https_proxy=http://192.168.92.100:3128


Well that's good to know.


Quote
I'll template this and it should be alright.

edit: is there a way to template a bash script?

You can template just about anything if you want to. HOWEVER.....

The dehydrated script itself is not templated at all, so if you have added something directly to the script it will get overwritten if the script is updated (which is quite likely at some point)

It would therefore be better if you tried to modify the config file which IS templated. Try adding your export settings in there and see what happens.

Try editing /etc/e-smith/templates/etc/dehydrated/config/10Default and add the proxy port and see what happens - this would be a much easier fix.

Something like this may do it (no guarantees - I am guessing here!) :

$OUT .= "export http_proxy=http://ip-of-the-proxy:port-of-the-proxy"\n";
$OUT .= "export https_proxy=https://ip-of-the-proxy:port-of-the-proxy"\n";

If that works it would be trivial to add a 'Proxy' key to the letsencrypt config key and a bit of code to the template file.

The only thing I am not sure is whether this permanently sets the proxy variables for your server. That may potentially lead to other issues so needs checking.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #54 on: October 20, 2017, 05:12:34 PM »
using wget for debugging was not the best solution as dehydrated make no use of it, but rather curl. wget is also a tool that is used pretty frequently and if ShulzStefan is behind a proxy there is chance he has already a .wgetrc file set to allow him to download :
~/.wgetrc file:
Code: [Select]
use_proxy=on
http_proxy=http://192.168.92.100:3128
https_proxy=http://192.168.92.100:3128


the way I would have tested would rather be :
Code: [Select]
curl https://acme-v01.api.letsencrypt.org/directory
see what happen

then, if it failed try:
Code: [Select]
export http_proxy=http://192.168.92.100:3128
export https_proxy=http://192.168.92.100:3128
curl https://acme-v01.api.letsencrypt.org/directory

if it worked then I would have simply add the two lines to /etc/profile or better  by creating a file in /etc/profile.d/httpproxy.sh that could be templated later
/etc/profile.d/httpproxy.sh file:
Code: [Select]
export http_proxy=http://192.168.92.100:3128
export https_proxy=http://192.168.92.100:3128

then I would logout the user and login again to have our previous temporary  export cleared and have the  profile file used, and finally test again
Code: [Select]
curl https://acme-v01.api.letsencrypt.org/directory
if it works then the direct call of the following command should work too
Code: [Select]
dehydrated -c
However an issue may still remain, because we use cron !
/etc/cron.daily/letsencrypt
Code: [Select]
#!/bin/sh

/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

we should make this script login or load profiles ...

if we were using a regular cron line this would be:
Code: [Select]
3 12 */1 * * bash -l -c '/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null'
or
Code: [Select]
3 12 */1 * * . /etc/profile; /usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

here i think we could simply add to the file the call
Code: [Select]
#!/bin/sh
. /etc/profile
/usr/local/bin/letsencrypt.sh -c 2>&1 > /dev/null

IMHO this should not be a simple quick fix at a contrib level but rather a core work :

there are a lot of work to do on upstream proxy for SME :
- possible configuration : https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration
- yum not using the upstream proxy : https://bugs.contribs.org/show_bug.cgi?id=2407
- yum not using squid : https://bugs.contribs.org/show_bug.cgi?id=542
- make upstream proxy configurable with panel : https://bugs.contribs.org/show_bug.cgi?id=1797

SchulzStefan,
have you ever set the SquidParent config key on this server ? Any issue with using yum behind your proxy ? Have you ever do the things sugegsted here https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration

for me the big change would be to have this SquidParent entry linked to a /etc/profile.d/httpproxy.sh template  and adapt from there.

Code: [Select]
mkdir -p /etc/e-smith/templates-custom/etc/profile.d/httpproxy.sh
echo '
    $OUT = "";
    if (defined $SquidParent && $SquidParent && $squid{"status"} =="disabled" )
    {
        my $port = $SquidParentPort || "3128";
        $OUT .= "export http_proxy=$SquidParent:$port\n";
        $OUT .= "export https_proxy=$SquidParent:$port \n";
    }
' >/etc/e-smith/templates-custom/etc/profile.d/httpproxy.sh

expand-template /etc/profile.d/httpproxy.sh
this would allow when a remote proxy is set and local squid disabled to have all user have it configured and used. This should solve issue with yum , curl wget..... as long as the script is run with a logged in user ... Only a few workaround could have to be added for cron called script. Also we might need to test how scripts called via manager react.


----
other sources :
https://stackoverflow.com/questions/11211705/setting-proxy-in-wget
https://unix.stackexchange.com/questions/27289/how-can-i-run-a-cron-command-with-existing-environmental-variables


Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #55 on: October 21, 2017, 12:17:29 AM »
Jean-Philippe Pialasse,

I'll try to answer your questions:

Quote
there is chance he has already a .wgetrc file set to allow him to download

There's no .wgetrc file in the directory of root. If it should be elsewhere, please advise.

Quote
[the way I would have tested would rather be :
Code: [Select]

curl https://acme-v01.api.letsencrypt.org/directory

# curl https://acme-v01.api.letsencrypt.org/directory

brings up:

{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert",
  "xvh60byUoYE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"

Don't know if this is sucess or a failure?

Quote
SchulzStefan,
have you ever set the SquidParent config key on this server ? Any issue with using yum behind your proxy ? Have you ever do the things sugegsted here https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Upstream_proxy_server_configuration

# config show yum
yum=service
    AutoInstallUpdates=disabled
    CheckContribs=enabled
    EnableGroups=no
    GPGCheck=0
    PackageFunctions=disabled
    Proxy=http://192.168.100.10:3562/
    RandomDelay=120
    check4updates=daily
    status=enabled

and:

# config show clamav
clamav=service
    ArchiveBlockEncrypted=no
    ArchiveBlockMax=no
    ArchiveMaxCompressionRatio=300
    Checks=24
    DNSDatabaseInfo=current.cvd.clamav.net
    DatabaseMirror=db.local.clamav.net
    Debug=no
    DetectBrokenExecutables=no
    FilesystemScan=weekly
    FilesystemScanExclude=/proc,/sys,/usr/share,/var
    FilesystemScanFilesystems=/home/e-smith/files
    FilesystemScanReportTo=admin
    FilesystemScanUnofficialSigs=no
    Foreground=yes
    HTTPProxyPassword=
    HTTPProxyPort=3562
    HTTPProxyServer=192.168.100.10
    HTTPProxyUsername=
    HeuristicScanPrecedence=no
    IdleTimeout=60
    LeaveTemporaryFiles=no
    LogClean=no
    LogFileUnlock=yes
    LogTime=no
    LogVerbose=yes
    MaxAttempts=6
    MaxConnectionQueueLength=30
    MaxDirectoryRecursion=20
    MaxFileSize=15M
    MaxFiles=1500
    MaxRecursion=8
    MaxThreads=20
    Quarantine=enabled
    QuarantineDirectory=/var/spool/clamav/quarantine
    ReadTimeout=300
    ScanArchive=yes
    ScanHTML=yes
    ScanMail=yes
    ScanOLE2=yes
    ScanPE=yes
    ScanRAR=no
    SelfCheck=1800
    ShowProxySettings=no
    ShowUpdateSettings=no
    SignaturesUpdated=unknown
    UpdateNonOfficeHrs=disabled
    UpdateOfficeHrs=disabled
    UpdateWeekend=disabled
    status=enabled

Without the proxy settings neither yum nor freshclam is working.

# config show SquidParent
SquidParent=

I didn't try this setting yet.

Right now server is working proper with my settings. You see the yum and clamav settings, and the "dirty fix" in the dehydrated script. No errors with yum, clamav and dehydrated. If I can help/test/try let me know.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #56 on: October 21, 2017, 03:36:27 AM »
Stefan,

this confirm that we have some global work to do there.
a keep it simple approach should not be to set a proxy for every service running around  *rather than* every time we step on a new thing not working.

The truth is there is not that many people using a SME behind proxy that I know so I would say leave your server as is as it works according to your need, but if you have some time in the next month I might ask you help to test some new code on a test machine aside if you are ok.

EDIT: added missing words.
« Last Edit: October 23, 2017, 04:51:04 PM by Jean-Philippe Pialasse »

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #57 on: October 23, 2017, 11:31:01 AM »
Jean-Philippe,

just tell me what to do. I'll be happy to assist.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #58 on: February 15, 2018, 12:13:39 PM »
If anyone runs in this issue:

Anacron job 'cron.daily'
/etc/cron.daily/letsencrypt:

  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Details:
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>


This fix might be helpful:

https://github.com/lukas2511/dehydrated/commit/7a0e71c6c2ccc6e98abca5ea1c7de28053e90c02

what was described here:

https://community.letsencrypt.org/t/dehydrated-caused-rate-limits-to-be-reached/52477

IMHO the fix/patch should be implemented in the sme dehydrated rpm.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: letsencrypt challenge not completing
« Reply #59 on: February 15, 2018, 12:25:06 PM »
nice catch.. please raise a bug, thank you

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #60 on: February 15, 2018, 12:31:13 PM »
Today I've got no more time... tomorrow then.
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #61 on: February 16, 2018, 08:54:13 PM »
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #62 on: April 22, 2018, 08:45:48 PM »
Yum reports an update for dehydated:

dehydrated         noarch      0.5.0-3.el6.sme

# rpm -q dehydrated
dehydrated-0.4.0.20170205.git1163864-1.el6.sme.noarch

I can't find the diffs. Could anybody tell what changes have been made?

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #63 on: April 22, 2018, 11:37:49 PM »
In CVS or check the git diffs (in the rpm name)

The only thing that has changed is the dehydrated script itself.

Check your bug for more:
https://bugs.contribs.org/show_bug.cgi?id=10521#c8

You should be able to update without any issues.

Nothing has been changed in the smeserver-letsencrypt contrib so it doesn't handle multiple certs etc.

There are more changes in the v0.6 dehydrated script which I am testing, but even that seems to work ok, though I need to make a few notes on updating to it as it uses letsencrypt API v2.

Please report any issues in the bug tracker.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #65 on: April 25, 2018, 09:59:20 PM »
In case the server is behind a firewall, it might be important to add this (again) in /usr/bin/dehydrated:

# setting firewall forward proxy
#
export http_proxy=http://192.168.42.1:3128
export https_proxy=http://192.168.42.1:3128

I updated today and will report, if any issues are coming up.

regards,
stefan

And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #66 on: April 25, 2018, 10:33:52 PM »
I think you mean 'using a proxy' hence port 3128

Presume you are running a proxy on your router or something.

All mine behind a firewall with simple port forwarding, or in gateway/server with no firewall, have zero issues.

From.what remember of your previous issues, I think your situation is commonly known as an 'edge case' :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline umbi

  • ***
  • 100
  • +0/-0
letsencrypt issue after sme server reboot
« Reply #67 on: September 30, 2018, 12:15:21 AM »
Hello to all

Im using SME with dehydratet letsencrypt.

i have all ssl domains in domain.txt

domain1
domain2
domain3
(saved wit nano) when i reopen all 3 domains are listed. So it works all perfect, untill i make an sme update and reboot.

when i open again the domain.txt file located in dehydrated directory, there is an old domain.txt file with only domain1 listed.
Domain 2 and 3 are no more in that file.

has someone an idea what i did wrong?

Many thanks in advance for any help.
(if im here in a wrong thread i ask sorry)


Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt issue after sme server reboot
« Reply #68 on: September 30, 2018, 12:20:01 AM »
has someone an idea what i did wrong?
Yes--domains.txt is templated.  If you want your changes to survive a system update, you'll need to make them by creating custom template fragments for domains.txt adding your other domain names.
......

Offline umbi

  • ***
  • 100
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #69 on: September 30, 2018, 12:35:43 AM »
Hi DanB35

Wow havent suspected a so fast answer.
Do you have a help for me, howto

creating custom template fragments for domains.txt

Thx 😊
Greez umbi

Offline DanB35

  • ****
  • 764
  • +0/-0
    • http://www.familybrown.org
Re: letsencrypt challenge not completing
« Reply #70 on: September 30, 2018, 12:37:29 AM »
......

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #71 on: September 30, 2018, 12:58:41 AM »
You should read the letsencrypt page on the wiki on how to add hosts and domains.

https://wiki.contribs.org/Letsencrypt

The templates should be in:

/etc/e-smith/templates/etc/dehydrated/domains.txt

Before making changes copy those templates to:

/etc/e-smith/templates-custom/etc/dehydrated/domains.txt
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline umbi

  • ***
  • 100
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #72 on: September 30, 2018, 01:55:49 AM »
Hi ReetP
Hi DanB35

I apreciate your replys - thanks 4 it.

I was reading the howto links but as im not really stable on english language so i'm a little unsure. I prefer bether ask befor i make something wrong.

Did i understood correct, when i make for each  domain with following command:

db hosts setprop mail.domain1.com
db hosts setprop www.domain1.com
db hosts setprop domain1.com
- then same with domain2+3

  letsencryptSSLcert enabled
  signal-event console-save

It will automatic create the new entrys in the custom template?

greez
Umbi
« Last Edit: September 30, 2018, 02:03:57 AM by umbi »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: letsencrypt challenge not completing
« Reply #73 on: September 30, 2018, 02:31:39 AM »
Yes it should.

Try enabling one host eg www, console-save, and check domains.txt to make sure it is correct.

You can use mode 'test' to make sure the certificates are generated.

Then do this to generate them:

Code: [Select]
dehydrated -c -x
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: letsencrypt challenge not completing
« Reply #74 on: January 01, 2019, 09:02:12 PM »
Best way is not to create custom templates which might be forgotten there an dmigh conflict further updates but to actually use the domain panel to set the domain you need, then set the property to enable the ssl cert as per the lets encrypt page




Hi DanB35

Wow havent suspected a so fast answer.
Do you have a help for me, howto

creating custom template fragments for domains.txt

Thx 😊
Greez umbi

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: letsencrypt challenge not completing
« Reply #75 on: January 05, 2019, 11:01:34 PM »
correcting the permissions made the challenge work perfectly.

Please add "[Solved]" to the Subject.
« Last Edit: December 10, 2019, 08:53:03 AM by TerryF »

Offline SchulzStefan

  • *
  • 620
  • +0/-0
Re: letsencrypt challenge not completing
« Reply #76 on: December 10, 2019, 08:03:38 AM »
In case you receive a http-01 invalid error (type: urn:acme:error:connection) with status 400 on port 80, and you are using fail2ban, check the settings of fail2ban. You may disable fail2ban via server-manager for a test and run dehydrated -c from the cli manually.

regards,
stefan
And then one day you find ten years have got behind you.

Time, 1973
(Mason, Waters, Wright, Gilmour)