Systemd services #13

lecris · 2021-05-18T21:39:18-04:00

lecris commented

2021-05-18 21:39:18 -04:00

Here's a start to provide some basic systemd service files. There are a few issues still needing to be addressed, but let's open it up for discussion to see how to address them:

User: Currently there is no standard matrix/bridge users for synapse or dendrite servers. Some concerns are if there is a common user for both the server and bridges, is there a risk of interfering with each other or any security concerns.
line-chrome.target should wait for when line-chrome.service is starting to listen. Currently line.service fails because line-chrome.target is activated too fast. Not sure how to achieve that yet.
Check if erroring and systemctl start/stop interact as they're supposed to:
- line-chrome.service can be start independently
- start line.service triggers start line-chrome.service
- restart line.service does not restart line-chrome.service: Fails
- stop line-chrome.service stops line.service
- restart line-chrome.service restarts line.service
- stop line.service stops line-chrome.service
~~Link to the matrix homeserver service with similar logic as above~~
Confirm that it works on other OS (just need to check executable paths):
- RHEL
- Debian
Add logging?

Discussion:

Should either line.service or line-chrome.service be made into socket? As I understand when we make a socket, it can be automatically started and stopped when required to listen. For the line-chrome that might be unproductive because it would require to re-login unless the status can be saved in between sessions, but also it would not be able to recieve updates.
Maybe these should be templetized and then generated by setup.py. Are there any standard template formats we can use? I like the ones in github.com/smallstep/certificate
Depending on how you want to integrate multiple chrome clients, the line-chrome.service might have to be aliased, i.e. line-chrome@.service, and somehow allowed to be controlled by line.service user via sudoer.

Slight rant:

Currently I don't think there is a stadndard synapse/bridge service infrastructure, and ideally the bridges and plugins should be designed so they can be invoked and editted as something like matrix-bridges@.service, but that's a discussion for the core matrix community later on, e.g. if they can provide a matrix-server.target that we can later link to add dependencies and so on.

Here's a start to provide some basic systemd service files. There are a few issues still needing to be addressed, but let's open it up for discussion to see how to address them: - [ ] `User`: Currently there is no standard matrix/bridge users for synapse or dendrite servers. Some concerns are if there is a common user for both the server and bridges, is there a risk of interfering with each other or any security concerns. - [x] `line-chrome.target` should wait for when `line-chrome.service` is starting to listen. Currently `line.service` fails because `line-chrome.target` is activated too fast. Not sure how to achieve that yet. - [x] Check if erroring and `systemctl start/stop` interact as they're supposed to: - [x] `line-chrome.service` can be start independently - [x] `start line.service` triggers `start line-chrome.service` - [x] `restart line.service` does not restart `line-chrome.service`: Fails - [x] `stop line-chrome.service` stops `line.service` - [x] `restart line-chrome.service` restarts `line.service` - [x] `stop line.service` stops `line-chrome.service` - [ ] ~~Link to the `matrix homeserver` service with similar logic as above~~ - [ ] Confirm that it works on other OS (just need to check executable paths): - [x] RHEL - [ ] Debian - [ ] Add logging? ## Discussion: - Should either `line.service` or `line-chrome.service` be made into socket? As I understand when we make a socket, it can be automatically started and stopped when required to listen. For the `line-chrome` that might be unproductive because it would require to re-login unless the status can be saved in between sessions, but also it would not be able to recieve updates. - Maybe these should be templetized and then generated by `setup.py`. Are there any standard template formats we can use? I like the ones in `github.com/smallstep/certificate` - Depending on how you want to integrate multiple chrome clients, the `line-chrome.service` might have to be aliased, i.e. `line-chrome@.service`, and somehow allowed to be controlled by `line.service` user via `sudoer`. ## Slight rant: Currently I don't think there is a stadndard synapse/bridge service infrastructure, and ideally the bridges and plugins should be designed so they can be invoked and editted as something like `matrix-bridges@.service`, but that's a discussion for the core matrix community later on, e.g. if they can provide a `matrix-server.target` that we can later link to add dependencies and so on.

👍 1

lecris added 1 commit 2021-05-18 21:39:19 -04:00

b5c6d5ea3f Initial Systemd services

fair commented

2021-05-19 20:53:04 -04:00

Thanks for this!
Some comments:

Regarding User: it's probably enough to have a commented-out User (as you already did) to not break default installs, while still suggesting the best-practice of running with a system user.
To handle its dependency on line-chrome.service, maybe line.service can just use Restart=always to make it keep trying to connect to the socket until it succeeds. Or better yet, the Python module itself can poll for a socket connection instead of erroring out on first failure.
Any dependencies on a homeserver unit are IMO out-of-scope for this, given that there's no way to predict which (or how many) homeservers someone has installed, and like you said, there's no standard target/service unit naming scheme for HSs. Plus, this bridge should ideally poll for a HS connection instead of fast-failing, and shouldn't fail or need to restart if the HS crashes (like most bridges), so it may not need a hard dependency on a homeserver at all.
Regarding using a socket unit, I was planning on having the node process manage multiple sockets itself, since it looks like most of the connection-management code can already handle multiple connections. Socket units do sound like a good idea though, so I'll think about it more before deciding anything.

Thanks for this! Some comments: * Regarding `User`: it's probably enough to have a commented-out `User` (as you already did) to not break default installs, while still suggesting the best-practice of running with a system user. * To handle its dependency on `line-chrome.service`, maybe `line.service` can just use `Restart=always` to make it keep trying to connect to the socket until it succeeds. Or better yet, the Python module itself can poll for a socket connection instead of erroring out on first failure. * Any dependencies on a homeserver unit are IMO out-of-scope for this, given that there's no way to predict which (or how many) homeservers someone has installed, and like you said, there's no standard target/service unit naming scheme for HSs. Plus, this bridge should ideally poll for a HS connection instead of fast-failing, and shouldn't fail or need to restart if the HS crashes (like most bridges), so it may not need a hard dependency on a homeserver at all. * Regarding using a socket unit, I was planning on having the node process manage multiple sockets itself, since it looks like most of the connection-management code can already handle multiple connections. Socket units do sound like a good idea though, so I'll think about it more before deciding anything.

👍 1

lecris commented

2021-05-19 21:56:46 -04:00

Or better yet, the Python module itself can poll for a socket connection instead of erroring out on first failure.

That would indeed be preferred. A n time retry with a few seconds time-out should do. If possible I would prefer to avoid Restart=always because it might hide some underlying errors down the line.

Any dependencies on a homeserver unit are IMO out-of-scope for this

Agree, would be nice to have something standardized from the matrix dev team, but until then it is not necessary. Also it should be able to be run with external server in the future like via Dimension or the other pluggable method, so I steadily come to think that it is uneccessary, at least for bridges and such.

Regarding using a socket unit

If node could manage that it would be ideal. One problem with using systemd socket is that it has to be fast to start and persistent. Probably the only place it makes sense to have is on the line.service, but the effect there would be minimal?

Also how about the locations? It would be easy to create these from a template when setting up the database.

> Or better yet, the Python module itself can poll for a socket connection instead of erroring out on first failure. That would indeed be preferred. A n time retry with a few seconds time-out should do. If possible I would prefer to avoid `Restart=always` because it might hide some underlying errors down the line. > Any dependencies on a homeserver unit are IMO out-of-scope for this Agree, would be nice to have something standardized from the matrix dev team, but until then it is not necessary. Also it should be able to be run with external server in the future like via `Dimension` or the other pluggable method, so I steadily come to think that it is uneccessary, at least for bridges and such. > Regarding using a socket unit If node could manage that it would be ideal. One problem with using systemd socket is that it has to be fast to start and persistent. Probably the only place it makes sense to have is on the `line.service`, but the effect there would be minimal? Also how about the locations? It would be easy to create these from a template when setting up the database.

lecris commented

2021-06-03 21:09:09 -04:00

Should check github.com/matrix-org/synapse#9803 for seucrity hardening tips and tools like systemd-analyze security from this blog

Should check [github.com/matrix-org/synapse#9803](https://github.com/matrix-org/synapse/pull/9803) for seucrity hardening tips and tools like `systemd-analyze security` from [this blog](https://www.ctrl.blog/entry/systemd-service-hardening.html)

lecris added 1 commit 2021-06-13 08:09:45 -04:00

a89944a876 Use notify for systemd

lecris commented

2021-06-13 08:12:40 -04:00

Update: with this change, line-chrome is active only when the initialization is finished by node.

There might be a depandancy on systemd is there any check to see if it still works without it? Maybe the sd.notify will just send commands to the void so it might still work.

Update: with this change, `line-chrome` is active only when the initialization is finished by node. There might be a depandancy on `systemd` is there any check to see if it still works without it? Maybe the `sd.notify` will just send commands to the void so it might still work.

lecris added 1 commit 2021-06-13 08:17:25 -04:00

9c09285045 Merge branch 'master' into systemd

lecris force-pushed systemd from 9c09285045 to ea56f82f51

2021-06-13 08:19:55 -04:00

Compare

lecris added 1 commit 2021-06-13 08:35:26 -04:00

471b63c52a Fix ordering and dependence

lecris changed title from ~~WIP: Systemd services~~ to Systemd services

2021-06-13 09:17:43 -04:00

lecris added 1 commit 2021-06-13 19:14:00 -04:00

9d4f2dafac Add `RuntimeDirectory` for sockets

lecris commented

2021-06-13 19:38:16 -04:00

The only other options to set are setting Restart options, which I don't know if it should be added atm or later, and hardening the service files, which is quite a slog to do. Will make a separate file in this branch afterwards.

Should be ok to merge like this, but check the yarn settings on your end as well.

The only other options to set are setting `Restart` options, which I don't know if it should be added atm or later, and hardening the service files, which is quite a slog to do. Will make a separate file in this branch afterwards. Should be ok to merge like this, but check the `yarn` settings on your end as well.

lecris added 5 commits 2021-06-22 21:15:40 -04:00

9ab94be628

Initial Systemd services

eb573a91fd

Use notify for systemd

d93457fcff

Fix ordering and dependence

91885245f1

Add `RuntimeDirectory` for sockets

968832545e Merge remote-tracking branch 'origin/systemd' into systemd

lecris force-pushed systemd from 968832545e to 9d4f2dafac

2021-06-22 21:19:39 -04:00