Steps to change the keystore password

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Steps to change the keystore password

duncan
Hi

We have been trying to change the truststore passwords from the default, but are running into difficulties and I was hoping someone might be able to advise. The message we are receiving is that the keystore has been tampered with.

We are running Connect 4.2 on Glassfish 3.1.2. These are the steps we followed (ensuring password is consistent for all steps):

Change the Glassfish master password

asadmin change-master-password --savemasterpassword=true

Changing our gateways private keystore password (gateway.jks)


Change the keystore password

keytool -storepasswd -keystore gateway.jks

In domain.xml set the following property with your new gateway keystore password

-Djavax.net.ssl.keyStorePassword=newPassword

In signature.properties, set the new gateway keystore password

org.apache.ws.security.crypto.merlin.keystore.password=newPassword

In saml.properties set the new gateway keystore password

org.apache.ws.security.saml.issuer.key.password=newPassword

Changing the keystore that contains the remote gateways public keys (cacerts.jks)


Change the keystore password

keytool -storepasswd -keystore cacerts.jks

In domain.xml set the following property with your new cacert password

-Djavax.net.ssl.trustStorePassword=newPassword

In truststore.properties set the new ca password

org.apache.ws.security.crypto.merlin.keystore.password=newPassword

This is the first exception we see

javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=89;_ThreadName=Thread-2;|[16/10:51:49:053] FATAL  SAMLIssuerImpl                  Could not initialize SAMLIssuerImpl: class org.apache.ws.security.components.crypto.Merlin cannot create instance
...
 at gov.hhs.fha.nhinc.webserviceproxy.WebServiceProxyHelper.invokeTheMethod(WebServiceProxyHelper.java:273)
        at gov.hhs.fha.nhinc.webserviceproxy.WebServiceProxyHelper.invokePort(WebServiceProxyHelper.java:355)
        at gov.hhs.fha.nhinc.webserviceproxy.WebServiceProxyHelper.invokePortWithRetry(WebServiceProxyHelper.java:402)
        at gov.hhs.fha.nhinc.webserviceproxy.WebServiceProxyHelper.invokePort(WebServiceProxyHelper.java:327)
        at gov.hhs.fha.nhinc.messaging.client.CONNECTBaseClient.invokePort(CONNECTBaseClient.java:54)
        at gov.hhs.fha.nhinc.auditrepository.nhinc.proxy.AuditRepositoryProxyWebServiceSecuredImpl.auditLog(AuditRepositoryProxyWebServiceSecuredImpl.java:79)

Which I believe is caused by this

javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=89;_ThreadName=Thread-2;|rs.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
...
Caused by: org.apache.ws.security.components.crypto.CredentialException: Failed to load credentials.
        at org.apache.ws.security.components.crypto.Merlin.load(Merlin.java:371)
        at org.apache.ws.security.components.crypto.Merlin.loadProperties(Merlin.java:190)
        at org.apache.ws.security.components.crypto.Merlin.<init>(Merlin.java:140)
        ... 121 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
        at java.security.KeyStore.load(KeyStore.java:1214)
        at org.apache.ws.security.components.crypto.Merlin.load(Merlin.java:365)
        ... 123 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed

We have verified that all passwords have been correctly updated and have even updated the internal certificate key password.

Any help appreciated
Reply | Threaded
Open this post in threaded view
|

Re: Steps to change the keystore password

christophermay07
Hi Duncan,

    In addition to the gateway keystore password (signature.properties: org.apache.ws.security.crypto.merlin.keystore.password), the gateway keystore alias password needs to be changed:
       

Change the keystore alias password

keytool -keypasswd -keystore gateway.jks -alias gateway

Update saml.properties

org.apache.ws.security.saml.issuer.key.password=newPassword

Regards,
Chris
Reply | Threaded
Open this post in threaded view
|

Re: Steps to change the keystore password

duncan
Thanks Chris

We had actually done that as well (I was reading that the API meant the keystore and keypass have to be kept in sync), I just forgot to include the steps.

We will try and repeat it again - we must have made a mistake somewhere.

Connect guys, could I perhaps suggest adding these instructions to your FAQ section?

Duncan
Reply | Threaded
Open this post in threaded view
|

Re: Steps to change the keystore password

christophermay07
Hi Duncan,

    Some JREs instantiate both the keystore specified in domain.xml and the default JRE keystore.  If you are still seeing the same exception, try updating the password of the following keystore to match the javax.net.ssl.keyStorePassword in your domain.xml:

    <JAVA_HOME>/jre/lib/security/cacerts.
       
    Let me know if this does not resolve the issue.  Once we determine the root cause of this exception, I will work on documenting these steps.

Regards,
Chris
Reply | Threaded
Open this post in threaded view
|

Re: Steps to change the keystore password

duncan
Thanks Chris, but unfortunately changing the password of the java home cacert didn't resolve the problem. I did see this blog, which suggests setting a property that might prevent this from being necessary http://emelnikov.blogspot.co.nz/2013/04/org.html

We originally tested this on a Windows environment, so decided to attempt it on Linux, but are getting the same result (errors below).

* We are using the same new password consistently.
* I have re-checked that both the gateway.jks and cacerts.jks have been updated and the key password in the gateway.jks updated.
* The signature.properties, saml.properties and truststore.properties all updated. I have searched for all references to changeit and can not find any.

Any other thoughts? Are you guys able to change it?

The error
gov.hhs.fha.nhinc.saml.SAMLIssuerImpl|_ThreadID=102;_ThreadName=Thread-2;|Could not initialize SAMLIssuerImpl: class org.apache.ws.security.components.crypto.Merlin cannot create instance
org.apache.ws.security.WSSecurityException: class org.apache.ws.security.components.crypto.Merlin cannot create instance
        at org.apache.ws.security.components.crypto.CryptoFactory.loadClass(CryptoFactory.java:224)
        at org.apache.ws.security.components.crypto.CryptoFactory.getInstance(CryptoFactory.java:117)
        at org.apache.ws.security.components.crypto.CryptoFactory.getInstance(CryptoFactory.java:169)
        at org.apache.ws.security.components.crypto.CryptoFactory.getInstance(CryptoFactory.java:161)
...
Caused by: org.apache.ws.security.components.crypto.CredentialException: Failed to load credentials.
        at org.apache.ws.security.components.crypto.Merlin.load(Merlin.java:371)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
Reply | Threaded
Open this post in threaded view
|

Re: Steps to change the keystore password

duncan
I believe there is an error with your truststore properties file, but changing it hasn't helped and I really need the involvement of one of the Connect guys.

Looking on the apache WSS4J configuration page http://ws.apache.org/wss4j/config.html I can see that the truststore properties should contain the following property

org.apache.ws.security.crypto.merlin.truststore.password

Their documentation says the default for this property is "changeit" which I think explains the problem I am experiencing (being unable to change the password from changeit).

The trustore.properties that is part of Connects source code / release, has the property org.apache.ws.security.crypto.merlin.keystore.password which is similar but different and i am guessing the cause of the problem.

Is someone able to take a look at this please?

Any help appreciated.

Duncan
Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

matt w

Hi Duncan, what happens if you provide org.apache.ws.security.crypto.merlin.truststore.password and give the password you want to use? We are trying to work in a little bit of time to look at this in depth but we are pushing up against the end of the sprint so we’ll do what we can.

 

From: duncan [via CONNECT Forums] [mailto:ml-node+[hidden email]]
Sent: Tuesday, October 01, 2013 4:16 PM
To: Weaver, Matthew (CGI Federal)
Subject: Re: Steps to change the keystore password

 

I believe there is an error with your truststore properties file, but changing it hasn't helped and I really need the involvement of one of the Connect guys.

Looking on the apache WSS4J configuration page http://ws.apache.org/wss4j/config.html I can see that the truststore properties should contain the following property

org.apache.ws.security.crypto.merlin.truststore.password

Their documentation says the default for this property is "changeit" which I think explains the problem I am experiencing (being unable to change the password from changeit).

The trustore.properties that is part of Connects source code / release, has the property org.apache.ws.security.crypto.merlin.keystore.password which is similar but different and i am guessing the cause of the problem.

Is someone able to take a look at this please?

Any help appreciated.

Duncan


If you reply to this email, your message will be added to the discussion below:

http://forums.connectopensource.org/Steps-to-change-the-keystore-password-tp7578995p7579033.html

To start a new topic under CONNECT Users, email [hidden email]
To unsubscribe from CONNECT Forums, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

duncan
Still the same error.

SEVERE|glassfish3.1.2|gov.hhs.fha.nhinc.saml.SAMLIssuerImpl|_ThreadID=97;_ThreadName=Thread-2;|Could not initialize SAMLIssuerImpl: class org.apache.ws.security.components.crypto.Merlin cannot create instance
org.apache.ws.security.WSSecurityException: class org.apache.ws.security.components.crypto.Merlin cannot create instance
        at org.apache.ws.security.components.crypto.CryptoFactory.loadClass(CryptoFactory.java:224)
        at org.apache.ws.security.components.crypto.CryptoFactory.getInstance(CryptoFactory.java:117)
...
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
---
Caused by: org.apache.ws.security.components.crypto.CredentialException: Failed to load credentials.
        at org.apache.ws.security.components.crypto.Merlin.load(Merlin.java:371)
        at org.apache.ws.security.components.crypto.Merlin.loadProperties(Merlin.java:190)
---
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
---
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

matt w

Hi Duncan, can you check out these issues below, I think there is some similarities, maybe those resolutions will help…

 

https://issues.connectopensource.org/browse/GATEWAY-3306

https://issues.connectopensource.org/browse/GATEWAY-3035

 

 

From: duncan [via CONNECT Forums] [mailto:ml-node+[hidden email]]
Sent: Wednesday, October 02, 2013 3:43 PM
To: Weaver, Matthew (CGI Federal)
Subject: RE: Steps to change the keystore password

 

Still the same error.

SEVERE|glassfish3.1.2|gov.hhs.fha.nhinc.saml.SAMLIssuerImpl|_ThreadID=97;_ThreadName=Thread-2;|Could not initialize SAMLIssuerImpl: class org.apache.ws.security.components.crypto.Merlin cannot create instance
org.apache.ws.security.WSSecurityException: class org.apache.ws.security.components.crypto.Merlin cannot create instance
        at org.apache.ws.security.components.crypto.CryptoFactory.loadClass(CryptoFactory.java:224)
        at org.apache.ws.security.components.crypto.CryptoFactory.getInstance(CryptoFactory.java:117)
...
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
---
Caused by: org.apache.ws.security.components.crypto.CredentialException: Failed to load credentials.
        at org.apache.ws.security.components.crypto.Merlin.load(Merlin.java:371)
        at org.apache.ws.security.components.crypto.Merlin.loadProperties(Merlin.java:190)
---
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
---
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)


If you reply to this email, your message will be added to the discussion below:

http://forums.connectopensource.org/Steps-to-change-the-keystore-password-tp7578995p7579035.html

To start a new topic under CONNECT Users, email [hidden email]
To unsubscribe from CONNECT Forums, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

duncan
Thanks Matt

I think you are right - I would say https://issues.connectopensource.org/browse/GATEWAY-3306 is directly related to the problem we are seeing. I'm not sure I can see an obvious resolution from that though.

There are some test procedures listed on the ticket. I'm not sure if other tests were carried out, but unfortunately the ones documented in the comments don't mention a scenario where the password was successfully changed.
Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

matt w

I have a few minutes, I am trying this out.

 

From: duncan [via CONNECT Forums] [mailto:ml-node+[hidden email]]
Sent: Thursday, October 03, 2013 11:00 PM
To: Weaver, Matthew (CGI Federal)
Subject: RE: Steps to change the keystore password

 

Thanks Matt

I think you are right - I would say https://issues.connectopensource.org/browse/GATEWAY-3306 is directly related to the problem we are seeing. I'm not sure I can see an obvious resolution from that though.

There are some test procedures listed on the ticket. I'm not sure if other tests were carried out, but unfortunately the ones documented in the comments don't mention a scenario where the password was successfully changed.


If you reply to this email, your message will be added to the discussion below:

http://forums.connectopensource.org/Steps-to-change-the-keystore-password-tp7578995p7579041.html

To start a new topic under CONNECT Users, email [hidden email]
To unsubscribe from CONNECT Forums, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

matt w
In reply to this post by duncan

Hi Duncan.

 

I have some good news, I think I found the issue. The Properties*.jar with the default configurations is in the default glassfish ear. This is the highest priority in the classloading so these default values are being loaded as opposed to the ones you provide in the config directory. You can either remove the Properties*.jar from the ear/lib directory, or make the changes within that jar. If you remove the properties file from the ear, you might have to add the config/nhin directory to the jvm classpath (you can do this in the glassfish admin console).

 

Our preliminary testing shows that no modifications are required to the field names in signature/truststore.properties, but if you still see issues there is no harm in putting in the fields names from the wss4j link you sent.

 

Let us know how these work!

Thanks,

Matt

 

From: duncan [via CONNECT Forums] [mailto:ml-node+[hidden email]]
Sent: Thursday, October 03, 2013 11:00 PM
To: Weaver, Matthew (CGI Federal)
Subject: RE: Steps to change the keystore password

 

Thanks Matt

I think you are right - I would say https://issues.connectopensource.org/browse/GATEWAY-3306 is directly related to the problem we are seeing. I'm not sure I can see an obvious resolution from that though.

There are some test procedures listed on the ticket. I'm not sure if other tests were carried out, but unfortunately the ones documented in the comments don't mention a scenario where the password was successfully changed.


If you reply to this email, your message will be added to the discussion below:

http://forums.connectopensource.org/Steps-to-change-the-keystore-password-tp7578995p7579041.html

To start a new topic under CONNECT Users, email [hidden email]
To unsubscribe from CONNECT Forums, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

rjettema
Matt, I just created a new JIRA issue for this as we have seen this in our internal testing within the DIL. https://issues.connectopensource.org/browse/CONN-646 Thanks, Richard
Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

matt w

Thanks Richard, I’ll put the solution in a comment on the ticket, can you try it out and let me know if it works for you?

 

From: rjettema [via CONNECT Forums] [mailto:ml-node+[hidden email]]
Sent: Monday, October 07, 2013 11:19 AM
To: Weaver, Matthew (CGI Federal)
Subject: RE: Steps to change the keystore password

 

Matt, I just created a new JIRA issue for this as we have seen this in our internal testing within the DIL. https://issues.connectopensource.org/browse/CONN-646 Thanks, Richard


If you reply to this email, your message will be added to the discussion below:

http://forums.connectopensource.org/Steps-to-change-the-keystore-password-tp7578995p7579051.html

To start a new topic under CONNECT Users, email [hidden email]
To unsubscribe from CONNECT Forums, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

duncan
Thanks for getting to the bottom of this guys!

I can confirm that we were able to resolve by removing the jar and adding the classpath entry. I'm not sure you can edit the classpath through the admin console on glassfish v3.x. We used the following (to save others the pain):

asadmin set configs.config.server-config.java-config.classpath-prefix=${com.sun.aas.instanceRoot}/config/nhin

Will you be releasing a fix for Connect 4.2? I noticed it was triaged as minor, but the likelihood of a user hitting this is extremely high.
Reply | Threaded
Open this post in threaded view
|

RE: Steps to change the keystore password

Naresh Subramanyan
As per the glassfish documentation (http://docs.oracle.com/cd/E19776-01/820-4507/abhcx/index.html) classpath-suffix, classpath-prefix and system-classpath attributes are Deprecated in 3.1. Please use the below to set the classpath.

command line:-
  asadmin deploy --libraries <Nhin Property directory> <connect ear>

Glassfish Console:-
  Add the <Nhin Property directory> to the Libraries field from the deploy page

Thanks,
Naresh