Add support for DLO automatic cleanup by bastelfreak · Pull Request #278 · puppetlabs/puppetlabs-puppetdb

bastelfreak · 2018-06-01T19:30:11Z

No description provided.

bastelfreak · 2018-06-01T19:30:46Z

This is WIP. Let me know if you're interested in this and I will add tests + README.md updates.

bastelfreak · 2018-06-01T19:33:25Z

+  gem 'rake'
  gem 'puppetlabs_spec_helper', :require => false
-  # Pinning due to bug in newer rspec with Ruby 1.8.7
-  gem 'rspec-core', '3.1.7'


I run into some issues while running the tests locally. For debugging I removed the pinning.

We shouldn't need to support such an old Ruby version anymore, so this should be ok.

I rebased against the release branch to get all this fixed. I hope that was okay.

austb

We are still discussing the details around the customization options for cleanup frequency. I'll get back to this PR as soon as I have more information. Thanks for your contributions!

austb · 2018-07-16T19:39:51Z

+      -> systemd::unit_file{'puppetdb-cleanup.timer':
+        content => epp("${module_name}/puppetdb-DLO-cleanup.timer.epp"),
+      }
+      -> service{'puppetdb-cleanup.timer':


This needs to be a ~> so that the service will be restarted if the puppetdb-cleanup.timer file changes.

As an alternative, it looks like the ~> service block can be omitted entirely with something like this (which I pulled from the README)

systemd::unit_file { 'foo.service': source => "puppet:///modules/${module_name}/foo.service", enable => true, active => true, }

After months of work, this feature is finally released. It wasn't available when when I created this PR. updated the code.

austb · 2018-07-16T19:41:05Z

+    if $facts['systemd'] {
+      # deploy a systemd timer + service to cleanup old reports
+      # https://puppet.com/docs/puppetdb/5.2/maintain_and_tune.html#clean-up-the-dead-letter-office
+      systemd::unit_file{'puppetdb-cleanup.service':


We should rename this to puppetdb-dlo-cleanup.service (and the matching timer one below) in case we want to add another cleanup service in the future.

sounds good, done

austb · 2018-07-16T20:14:57Z

+      }
+      -> service{'puppetdb-cleanup.timer':
+        ensure => 'running',
+        enable => true,


I don't think the enable option is necessary here, but I could be mistaken.

It needs to be enabled. Otherwise the timer is off after the next reboot and the upcoming puppet run would start it.

austb · 2018-07-17T21:45:33Z

+
+[Timer]
+# Daily at 8am
+OnCalendar=*-*-* 08:0:00


We should choose the time of day via fqdn_rand. If there are multiple PuppetDB VMs running on the same physical machine this will prevent them from hammering the disk at the same time.

Also the comment will no longer be accurate, so you can remove it or replace it with a general comment about running once a day with a link to documentation on the systemd timer syntax

You could select the hour with fqdn_rand(24) and the minutes with fqdn_rand(60).

austb · 2018-07-17T21:55:32Z

  }

+  if $automatic_dlo_cleanup {
+    if $facts['systemd'] {


If systemd does not exist, lets try setting up a cron job to clean this up instead. For example Redhat 6 has cron but not systemd.

austb · 2018-07-17T22:03:05Z

@bastelfreak These comments are done and ready for you to take a look at.

austb · 2018-07-17T22:21:48Z

+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/find /opt/puppetlabs/server/data/puppetdb/stockpile/discard/ -type f -mtime +90 -delete


Also, one final note. For security reasons, this service should run as the PuppetDB user

added user and group

bastelfreak · 2018-07-19T20:01:58Z

Personal reminder:

should add tests
Update readme
expose maximum DLO age as variable

austb · 2018-07-19T21:33:04Z

Awesome! This looks good. If you need any help writing tests just let us know and we can help out. I would say these would be nice.

Can be disabled (ie. creates neither the service/timer nor the cronjob)
Creates systemd service/timer, and starts timer
Creates cron job if systemd not available
Actually runs as puppetdb user. ie cannot perform some action the PuppetDB user should not be able to do (we can chat about this if it's unclear what to do, I'll have to think about it as well)

And if you add DLO age as variable, a test that that can be customized.

rbrw

Just a few comments with respect to the travis and sh bits. Thanks.

rbrw · 2018-07-23T16:37:15Z

+  - rm -f Gemfile.lock
+  - gem update --system
+  - gem --version
+  - bundle -v


I'd suggest either moving these to a helper script,

before_install: some/where/prep-tests

or using a different approach, i.e. say with one of the yaml multi-line sytaxes.

For example, using ">":

before_install: > set -ex && bundle -v && rm -f Gemfile.lock && ...

or using "|"

before_install: | set -ex bundle -v rm -f Gemfile.lock ...

(Note that syntax is just approximate -- haven't checked it.)

Otherwise, travis will just keep going iirc. That is, it runs all the bullet points, even if one of them fails, only stopping if there was a failure after it has tried them all.

@rbrw these changes are from the rebase, I'll address them in pr #279

rbrw · 2018-07-23T16:37:19Z

+  - gem --version
+  - bundle -v
+script:
+  - 'bundle exec rake $CHECK'


Hmm, do we need the single quotes, and regardless, wonder if we should quote the CHECK expansion as "$CHECK", just to avoid having to worry about the content.

@rbrw see above, will address in #279

Zak-Kent · 2018-07-24T17:33:18Z

@bastelfreak, sorry but would you be able to rebase this on master again? We ended up making some changes/rebasing the existing pr we had up before it merged. Thanks for all the help!

bastelfreak · 2018-07-24T18:02:38Z

@austb can you review this again please?

There is also a small change to a string that needed to be double quoted.

austb · 2018-07-24T20:16:55Z

Thanks @bastelfreak for the PR!

bastelfreak commented Jun 1, 2018

View reviewed changes

bastelfreak mentioned this pull request Jul 14, 2018

Setup for 7.0.0 release #279

Merged

austb suggested changes Jul 16, 2018

View reviewed changes

austb reviewed Jul 16, 2018

View reviewed changes

austb reviewed Jul 17, 2018

View reviewed changes

bastelfreak force-pushed the systemd branch 2 times, most recently from cd9f333 to 1923209 Compare July 19, 2018 19:55

bastelfreak force-pushed the systemd branch 2 times, most recently from 8b60736 to 7a2bc12 Compare July 20, 2018 20:21

rbrw reviewed Jul 23, 2018

View reviewed changes

bastelfreak force-pushed the systemd branch from 52be38f to 3b8c152 Compare July 24, 2018 17:33

bastelfreak changed the title ~~[WIP] add support for DLO automatic cleanup~~ Add support for DLO automatic cleanup Jul 24, 2018

bastelfreak force-pushed the systemd branch from 3b8c152 to f340561 Compare July 24, 2018 18:00

add support for automatic cleanup

1c25735

bastelfreak force-pushed the systemd branch from f340561 to 1c25735 Compare July 24, 2018 18:02

Add tests for automatic DLO cleanup

df48592

There is also a small change to a string that needed to be double quoted.

austb approved these changes Jul 24, 2018

View reviewed changes

austb merged commit 4e4e81a into puppetlabs:master Jul 24, 2018

bastelfreak deleted the systemd branch July 24, 2018 21:34

smortex added the enhancement label Apr 18, 2024

Uh oh!

Conversation

bastelfreak commented Jun 1, 2018

Uh oh!

bastelfreak commented Jun 1, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

austb left a comment

Choose a reason for hiding this comment

Uh oh!

austb Jul 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

austb Jul 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

austb commented Jul 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bastelfreak commented Jul 19, 2018

Uh oh!

austb commented Jul 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbrw left a comment

Choose a reason for hiding this comment

Uh oh!

rbrw Jul 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zak-Kent commented Jul 24, 2018

Uh oh!

bastelfreak commented Jul 24, 2018

Uh oh!

austb commented Jul 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

austb Jul 16, 2018 •

edited

Loading

austb Jul 16, 2018 •

edited

Loading

austb commented Jul 19, 2018 •

edited

Loading

rbrw Jul 23, 2018 •

edited

Loading