Recent by Daniel Nashed
Updating autoupdate.nsf with the new template (14.0 08.03.2024) 
By Daniel Nashed | 4/17/24 3:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The new fit & finish work and the new autcat.nsf integration requires template changes.
Please make sure you are getting the template version 14.0 from 08.03.2024 and not the earlier version from 03.11.2023 shipped with Domino 14.
When deploying the container image I noticed an issue with the folder permissions where the container image is getting template updates for Fixpacks.
The directory /opt/hcl/domino/notes/latest/linux/data1_bck/140FP1/localnotesdata contains updated templates.
But the directory can be only accessed by "root" and the container runs with the "notes" user.
This is not new to 14.0 FP1. Also 12.0.2 fixpacks had the same permissions, but nobody noticed the missing updates.
Domino AutoUpdate AUT Catalog integration in action
By Daniel Nashed | 4/17/24 3:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
When the new integration is enabled, client web-kits are just pushed to AUT Catalog.
The push will also happen for existing web-kits once the document is updated with data containing the Metadata XML.
No manual steps needed. The documents and the new view have a button to directly jump into AUT Catalog.
The button on top only shows up for software pushed to AUT Catalog.
AUT Catalog sometimes has multiple documents for the same web-kit.
Adding TOTP to your own application
By Daniel Nashed | 4/16/24 5:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The oathtool is the standard tool on Linux. It comes as a command-line tool or a dynamic and static link lib to be used in your own applications.
You can statically link the code into your application and generate TOTP codes and also validate them.
The homepage contains information about the command line tool "oathtool" and also the lib "liboath".
https://www.nongnu.org/oath-toolkit/
Example how to use it on command-line.
The example used the base32 encoded secret for "test".
oathtool --totp -b ORSXG5AK
Linux - Using Cron to schedule periodic jobs like certificate updates
By Daniel Nashed | 4/11/24 5:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In all the years I have never looked into cron.
But it is really a very straightforward functionality, which is used by Linux itself.
You can either schedule user specific jobs or use /etc/cron.d files or /etc/crontab.
There is a certificate update script --> https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/examples/nginx/cert_upd_nginx.sh
Howto convert cert formats from and to PEM
By Daniel Nashed | 4/11/24 5:09 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
CertMgr uses PEM internally for all operations. The PEM format is the most important format.
But you might get your files from your admin or a CA in different formats.
CertStore can import and export PEM and PKCS12 (PFX, p12).
But this might not always work in the way you expect it because of legacy encryption.
I just wrote a new howto document providing some background and providing OpenSSL command line options.
Domino meets Grafana & Loki
By Daniel Nashed | 4/8/24 12:45 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The latest Sametime version offers a graphical statistics dashboard based on Grafana and Prometheus.
Domino statistics out of the box don't play well with Grafana.
Prometheus needs a pull model and the Domino Stats Package added in Version 10 only supports the push model.
Sametime uses the push gateway, but because the Domino statistic names need to be transformed anyway, I wrote a small servertask to provide the stats to be included into the node_exporter, which already is used to provide Linux system statistics.
Beside statistics I also looked into Grafana Loki to collect logs and make them available over the Grafana interface. The data is collected by promtail.
SNMP with Domino on Docker
By Daniel Nashed | 4/1/24 1:59 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Simple Network Management Protocol (SNMP) is a rarely used functionality in Domino, which has been implemented in Domino in the last century.
But I got a request from a customer to get SNMP working with Domino in a container to monitor the server.
On Kubernetes there are other ways to monitor servers. But for a stand-alone Docker host, SNMP could still make sense and can be implemented.
Important: Domino ID Vault -- Don’t remove old servers if still referenced in user documents
By Daniel Nashed | 3/28/24 4:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
When you migrate to new servers, you have to be aware of the following limitation, which is documented in 12.0.2/14.0 but also affects older servers.
To ensure you can recover all user.IDs make sure the server document is still present and the server is still in the ID vault configuration. See the following warning in help and Kbase document.
This is a recent update in documentation and I just sent it to a customer during a server upgrade/move workshop.
Introducing Domino Borg Backup Integration V2
By Daniel Nashed | 3/19/24 2:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Borg Backup is an interesting backup option for Linux (https://www.borgbackup.org/) and also works inside a Domino container with a local or remote repository.
The first integration with Domino Backup used bash scripts and Borg commands. But this had limitations due to the way Borg handles backups.
Each database was stored in a separate repository. I have been looking for direct integration to avoid this overhead and store all backup data into a single backup.
There is a newer option to import tar data directly into Borg as a stream -> https://borgbackup.readthedocs.io/en/stable/usage/tar.html.
High Domino Backup performance with native ZFS storage on Proxmox
By Daniel Nashed | 3/19/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino 12+ default native backup is a very easy to use option, which also works on Docker containers.
The resulting backup to a file target is always consistent, because delta information is always applied to the backup file.
But a file target raises the challenge that the whole NSF data will be copied to the target file-share or disk. Therefore a de-duplicating target is highly recommended.
I took a look into ZFS in detail in my new local setup to test out performance.
Looking into S3 performance numbers for MinIO -- Is this the right target for backup?
By Daniel Nashed | 3/19/24 2:31 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
I know MinIO for a while and I have been using it for DAOS T2 testing early on. Years later they are now grow up and play in the cloud native storage league.
Still the devil is in the detail and for using it in production environment customers hopefully use the enterprise subscription to get tuning support.
Paying for support this doesn't make it a cheap storage any more if you look at their price tag.
First look at openSUSE Leap 15.6 Beta with Domino 14
By Daniel Nashed | 3/19/24 2:26 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
As some of you know from earlier discussions, the latest currently available SUSE Enterprise and openSUSE Leap 15.5 ships with a too old glibc to work out of the box with Domino 14.
You could still run it on a Docker(or Podman) host, because the container image brings the glibc run-time with it and only uses the kernel from the Docker host.
openSUSE Leap and SUSE Enterprise (SLES) share the repositories and are technically more or less the same.
SUSE Linux 15.6 is scheduled for mid 2024
I have been looking into openSUSE Leap earlier with their Alpha version.
Now the official beta is available for download
As expected Domino 14 works natively with the updated glibc. The requirement is glibc 2.34+. This Linux version will introduce glibc 2.38.
But SUSE also switched again to a new major kernel version with a Service Pack.
This means HCL will have to re-rest SUSE Linux once the final version is released.
It will take some time to have fully tested and support SUSE supported for Domino 14.0.
Important: For Domino SMTP with ECDSA keys for STARTTLS inbound
By Daniel Nashed | 3/19/24 2:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The short version of you don't want to know all the technical details:
If you choose a ECDSA key for your web server, make sure you have also a RSA key for SMTP inbound connections
In case you are interested in the technical details, read on ...
Introducing the Domino native Linux installer and Domino Linux Menu
By Daniel Nashed | 3/19/24 2:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
When I ask a question like "why admins are not moving to Domino on Linux" I might have a plan in my head already.
I cannot solve all the challenges for you at once. But I am helping over years with my Domino Start Script to get Domino on Linux easier to run.
The start script already helps to perform standard operations.
Istallation is and some other operations might be still more complicated at first glance on Linux.
I introduced a build menu into the HCL Domino Community image process recently.
And I took that logic and I am making it available for native installations as well.
This new option also offers automated downloads via the recently released Domino Download script
Domino 14.0 on Linux does not work on very old CPUs
By Daniel Nashed | 3/6/24 4:55 PM | Infrastructure - Notes / Domino | Added by Oliver Busse
Domino 14.0 requires glibc 2.34 because it is build on Redhat Linux 9.1.
Glibc is the Linux C run-time which is a core building blog of Linux and can't be changed.
This means you can't run it on Linux versions which don't have at least glibc 2.34.
Glibc 2.34 itself doesn't work on old CPUs not supporting at least a CPU with microarchitecture level x86-64-v2
Auth0 ODIC OpenID with Domino & Some other interesting findings
By Daniel Nashed | 2/22/24 1:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
We are working on a ODIC setup with a German business partner for a larger German customer.
Auth0 is one of the major providers. We got it working but only with some tricks for now.
It turned out the Auth0 OIDC endpoint has a cache expiration for 15 seconds. This looks like a setting that can't be changed.
The Domino OIDC cache uses the expiration header to invalidate the cache. So our cache on the Domino side was constantly reloading and invalid in some cases.
You really have to have an expiration that is at least a couple of minutes. Better at least 1 hour.
Faking the cache expiration
This has been reported to HCL and the team is working on an enhancement.
Meanwhile I came up with a work-around setting up a Fake provider on a NGINX server to forward the requests.
Domino Backup/Restore with multiple configurations and targets
By Daniel Nashed | 2/22/24 1:14 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino Back/Restore is a flexible framework for native Domino backup.
The dominobackup.nsf plays an important role for backup and restore operation.
It contains the following type of content.
Backup/restore/prune configuration
Inventory documents for restore operations
Restore requests
Backup logs
You could run backup with different excludes defined on command-line.
Or just backup selected databases or incremental backups.
But there cannot be different active configurations nor different backup retention in one dominobackup.nsf
Domino autoupdate.nsf for fast internal software downloads
By Daniel Nashed | 2/22/24 1:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino Autoupdate has been introduced in Domino 14.0. It offers automatic downloads from My HCLSoftware download, which has been on early access in parallel and has been released at the same time.
My Engage session will go into detail about the functionality with tips and tricks and additional information round both features and the new Domino Download script (https://nashcom.github.io/domino-startscript/domdownload/).
But I want already provide some details about options available today with simple integrations.
Easy to use container image providing ICAP support for ClamAV for Domino CScan
By Daniel Nashed | 2/22/24 1:11 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
My friend and fellow Ambassador Roberto Boccadoro submitted an Engage session about Domino CScan with ICAP.
Sadly his session did not make it into the agenda. But he is part of two OpenNTF sessions.
This session idea lead to a new OpenSource project I initiated to help with ICAP support.
Thanks Roberto for pushing me to get this implemented! :-)
The new project provides a simple to build container image, which natively offers ICAP services over TLS with a ClamAV container in the back-end.
The container is ready to be consumed with Domino CScan/ICAP (https://help.hcltechsw.com/domino/14.0.0/admin/conf_scanningattachmentsforviruses.html).
It comes with a docker-compose file which glues the official ClamAV container with this new image.
Running Domino in LXC containers on Proxmox requires a trick
By Daniel Nashed | 2/22/24 1:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Now that VMware might not be everyone's darling any more because of it's new mother ship, I took another look at Proxmox.
I know them for quite a while and they are doing a great job. I rebuilt a Intel NUC with 2 TB NVMe disk with the current version of Proxmox.
Proxmox supports full VMs and also LXC -- which is an interesting option for testing in lab environments.
You can setup a new Linux test machine in minutes from a template. And there are ready to use templates for all major Linux distributions.
I had a post long time ago about Proxmox automation on command-line.
Certificate ASN.1 Decoding online
By Daniel Nashed | 2/15/24 2:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Now that I posted the TLS 1.2 interactive information side today, some of you might also want to get details out of certificates.
Certificates are usually public information. So it should be OK to paste them into the website https://asn1js.eu/.
But there is a GitHub project referenced and you could run it also locally.
The inner guts of certificates are presented in ASN.1. When you ever has looked at OpenSSL C code, you will recognize the structures.
The Illustrated TLS 1.2 Connection -- Every byte explained
By Daniel Nashed | 2/15/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
While debugging a TLS connection issue, I ran into this website -->https://tls12.xargs.org/
It provides more details then most admins ever want to know. But it is a great resource understanding a TLS connection.
Domino Backup customized and centralized logging
By Daniel Nashed | 2/12/24 2:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
This question came up last week in a business partner workshop.
The partner wanted to centralize the logging of all Domino backup instances.
dominobackup.nsf intended per server. You could configure a global configuration database and local instances for the backup inventory.
In theory it could be one database for multiple could be also replicated in smaller environments.
This would not be recommended.
But there is an easier way for a centralized overview of all your Domino backups.
Domino adding Trusted Roots for Java applications
By Daniel Nashed | 2/12/24 2:24 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino has different places to store trusted roots depending on the part of the application.
Beginning with Domino 12.0.2 HCL started to consolidate root certificates into the new domain wide certstore.nsf.
But it will take some time to have all parts of Domino to use the new trusted roots back-end.
New callers like OIDC or CScan/ICAP and the certificate URL heath check already use the new back-end including UI integration.
JVM trusted roots cacerts overwritten by Domino update
Java still uses it's own cacerts file, which is part of the JVM directory. The file is only admin/root writable.
Domino release installers replace the cacerts file with the latest cacerts available.
But this overwrites custom certificates imported into cacarts.
This is a common problem I ran into twice in the last two weeks.
Enable DKIM for Domino
By Daniel Nashed | 2/12/24 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Enable DKIM for Domino
- DKIM inbound is supported starting with Domino 12.0.1
- DKIM outbound is supported starting with Domino 12.0.2
Now the first providers raise the bar for sending mails.
This might not only be relevant for mass mail.
Here is a short write up enabling DKIM for RSA and Ed25519 keys.
New default Let’s Encrypt certificate chain with ISRG Root X1 root
By Daniel Nashed | 2/12/24 2:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Let's Encrypt finally changed their default root certificate from DST Root CA X3 to ISRG Root X1.
The old root expired already 2 1/2 years ago, but was cross signed with the new chain.
Now finally Let's Encrypt uses the new root by default, which results in a shorter chain.
They have been using the older, longer chain to specially support older Android devices, which didn't have the X1 root in their trust store.
When you are using Let's Encrypt ACME and did not specify an alternate chain, there is nothing to change.
The new shorter certificate chain will be automatically used the next time the certificate is renewed.
But in case you set specific settings, you might now have to remove those settings, because they flipped the certificate chains.
The alternate chain is now the older longer certificate chain.
For Domino CertMgr the custom setting is "ACME Alternate Chain Suffix".
The Art of Troubleshooting
By Daniel Nashed | 1/15/24 3:38 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In all the years I am involved in troubleshooting, I still see the same patterns. I am planning to start a new initiative this year.
To start with, I wrote a short abstract end of the year when looking into this.
See this as a beginning of a change from my side to better help on community level and also provide better services as a HCL business partner.
It will also include troubleshooting steps for different kind of problem types like crashes, hangs, memory leaks, performance problems.
Not all of it can be described in howto material. But raising the awareness on all parts of the support process, can significantly help to solve problems faster.
I am in the troubleshooting business for over 25 years, read NSD before breakfast and wrote my own troubleshooting tools.
Important to know about Domino 14 Inbound Mail Disclaimer
By Daniel Nashed | 1/12/24 2:40 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
A partner ask in OpenNTF channel about an issue with the new inbound mail disclaimer in Domino 14.
It turned out I was able to reproduce the problem on one of my lab servers and this is critical to know if you enable disclaimers written into the body.
There are two different ways to setup the feature.
1. Tag the subject line
2. Write HTML to the body of the message with fall back to subject for signed or encrypted messages.
My personal recommendation is always to tag the subject, because modification of the body can lead to all kind of side effects with add-on software and also issues that could happen with incorrect formatted messages.
Running Traveler 14 on Windows 11 for testing
By Daniel Nashed | 1/8/24 3:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The Windows desktop versions are completely untested and unsupported for all Domino related products.
It is not recommended to run any Domino software in production on Windows desktop!
There are still a couple of scenarios where Windows desktop can be relevant.
For example a local notebook installation, a training environment or the Windows Sandbox -- which is also a Windows desktop environment.
Domino 14.0 changed the installer to prompt to confirm the unsupported Windows version.
But the installer continues to work with Windows 11 after you confirmed the message.
As a German partner discovered last week, Traveler 14.0 doesn't install on Windows 11.
InstallAnywhere runs into an error: Windows DLL failed to load during Installation.
Windows 10 works unchanged. The issue starts with Windows 11.
Automatically Updating Ubuntu Linux including mail notification
By Daniel Nashed | 1/8/24 3:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
This was on my list for a while. I looked into it, but the mail part wasn't what I wanted.
Most admins will just install BSD mailx. But this had way too much dependencies for me.
I just wrote a simple tool to replace mailx -> https://github.com/nashcom/nsh-tools/tree/main/nshmailx.
But you could just continue to use the standard package.
It wasn't just an option for me, because I try to not install other mail server components on a Domino server (like sendmail).
The setup for automatic updates are pretty easy and base on scripting.