WIP: chore(actions): support cron schedule task by appleboy · Pull Request #22751 · go-gitea/gitea

appleboy · 2023-02-04T08:28:03Z

only support the default branch in the repository setting.
autoload schedule data from the schedule table after starting the service.
support specific syntax like @yearly, @monthly, @weekly, @daily, @hourly

How to use

See the GitHub Actions document for getting more detailed information.

on:
  schedule:
    - cron: '30 5 * * 1,3'
    - cron: '30 5 * * 2,4'

jobs:
  test_schedule:
    runs-on: ubuntu-latest
    steps:
      - name: Not on Monday or Wednesday
        if: github.event.schedule != '30 5 * * 1,3'
        run: echo "This step will be skipped on Monday and Wednesday"
      - name: Every time
        run: echo "This step will always run"

Signed-off-by: Bo-Yi.Wu appleboy.tw@gmail.com

wolfogre · 2023-02-16T08:11:36Z

I have an alternative solution for reference:

Define tables like:

type ActionSchedule struct {
	ID            int64
	Title         string
	Specs         []string
	// and other necessary fields, without EntryIDs
}

type ActionScheduleSpec struct {
	ID         int64
	ScheduleID int64
	Spec       string
}

And the data looks like:

ActionSchedule:

ID	Title	Specs	other fields
1	A	`* 2 * * `, ` 3 * * *`	...
2	B	`* 4 * * *`	...

ActionScheduleSpec:

ID	ScheduleID	Spec
1	1	`* 2 * * *`
2	1	`* 3 * * *`
3	2	`* 4 * * *`

Then get rid of robfig/cron, just use the current cron frame work of Gitea. We need only one cronjob:

It runs once per minute.
Load all rows of ActionScheduleSpec from DB, I think it's fine when there are only 3 fields. Or we can have a cache for them.
Check Spec of ActionScheduleSpec, record theScheduleID if the spec matches the present time.
Load ActionSchedule from DB by the IDs recorded.
Create tasks for them.

So we don't have to do something like resetSchedule or schedule.Remove(row.EntryIDs).

zeripath · 2023-03-11T14:29:16Z

I don't understand why you're using robfig's cron instead of using the inbuilt cron within gitea itself?

appleboy · 2023-03-11T15:46:17Z

@zeripath I will remove the robfig's cron package and refactor the PR, get back to you asap.

codecov-commenter · 2023-03-14T15:00:13Z

Codecov Report

Merging #22751 (a693fe6) into main (f521e88) will decrease coverage by 0.10%.
The diff coverage is 34.12%.

❗ Current head a693fe6 differs from pull request most recent head 95320e8. Consider uploading reports for the commit 95320e8 to get more accurate results

@@            Coverage Diff             @@
##             main   #22751      +/-   ##
==========================================
- Coverage   47.14%   47.04%   -0.10%     
==========================================
  Files        1149     1159      +10     
  Lines      151446   152589    +1143     
==========================================
+ Hits        71397    71789     +392     
- Misses      71611    72325     +714     
- Partials     8438     8475      +37

Impacted Files	Coverage Δ
cmd/dump.go	`0.67% <0.00%> (ø)`
cmd/web.go	`0.00% <0.00%> (ø)`
models/actions/run.go	`1.64% <0.00%> (-0.08%)`	⬇️
models/actions/runner.go	`1.44% <ø> (ø)`
models/actions/schedule_list.go	`0.00% <0.00%> (ø)`
models/actions/schedule_spec_list.go	`0.00% <0.00%> (ø)`
models/packages/package.go	`45.45% <0.00%> (-1.13%)`	⬇️
models/user/search.go	`77.50% <0.00%> (-6.29%)`	⬇️
modules/actions/github.go	`0.00% <0.00%> (ø)`
modules/actions/workflows.go	`0.00% <0.00%> (ø)`
... and 62 more

... and 12 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

appleboy · 2023-03-15T00:59:50Z

@zeripath @wolfogre Done. Please help to review

lunny · 2023-03-15T06:57:14Z

+}
+
+func startTasks(ctx context.Context, opts actions_model.FindSpecOptions) error {
+	specs, count, err := actions_model.FindSpecs(ctx, opts)


For a big instance, how many the records do you think it will contain?

Any better ideas? I think we have to fetch all specs and check them.

Why should we fetch all before checking them?

Or maybe we could get all ids into memory and then fetch by pagination.

wolfogre · 2023-03-17T09:42:13Z

+}
+
+func startTasks(ctx context.Context, opts actions_model.FindSpecOptions) error {
+	specs, count, err := actions_model.FindSpecs(ctx, opts)


Any better ideas? I think we have to fetch all specs and check them.

lunny · 2023-03-24T07:07:35Z

I think this could be refactored after https://gitea.com/gitea/act/pulls/29 merged.

appleboy · 2023-03-31T13:51:26Z

All Done. Please help to review @wolfogre @lunny

techknowlogick · 2023-04-17T15:57:58Z

I've just resolved the conflicts

delvh · 2023-06-12T16:07:42Z

@nephatrine While I feel absolutely the same way about it, 1.20 is out of the picture. We are already in feature freeze, meaning no new features will be accepted for it anymore.

…25716) - cancel running jobs if the event is push - Add a new function `CancelRunningJobs` to cancel all running jobs of a run - Update `FindRunOptions` struct to include `Ref` field and update its condition in `toConds` function - Implement auto cancellation of running jobs in the same workflow in `notify` function related task: #22751 --------- Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com> Signed-off-by: appleboy <appleboy.tw@gmail.com> Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de>

BrunoBernardino · 2023-07-25T05:16:24Z

Sorry for the noise, but I'm so ready for this to be merged, in order to move in a couple of final repos from GitHub into Gitea (I've been following the PR for a few months now)! 😅

Thank you so much for the hard work here @appleboy !

puni9869 · 2023-07-30T01:46:51Z

Could you add some unit test cases.

denyskon · 2023-08-01T08:18:31Z

@appleboy The merged auto-cancellation is only for push events, not for cron, so it won't help us here - or am I wrong?

ghost · 2023-08-10T21:54:00Z

Will this make it into gitea soon, or should I look into woodpecker ci or similar?

silverwind · 2023-08-10T22:38:14Z

Happy to rubber-stamp this once the merge conflicts and CI failures are fixed.

denyskon · 2023-08-13T09:34:35Z

Could somebody clarify the state of this PR? IIRC the merged auto-cancellation of jobs only applies to push events. The blocker for this PR was that a wrongly configured cron task could launch many concurrent jobs. So is this PR still blocked, do we not consider this an issue anymore or is there another solution?

wxiaoguang · 2023-08-13T09:39:35Z

+		e.Limit(opts.PageSize, (opts.Page-1)*opts.PageSize)
+	}
+	var specs SpecList
+	total, err := e.Desc("id").FindAndCount(&specs)


It seems there is only on call to FindSpecs and it doesn't need "count"

wxiaoguang · 2023-08-13T15:00:19Z

 	github.com/go-webauthn/webauthn v0.8.6
 	github.com/gobwas/glob v0.2.3
 	github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f
+	github.com/gogs/cron v0.0.0-20171120032916-9f6c956d3e14


IIRC gogs/cron had been removed, now it is back?

Yes, this is a block change. It needs to be changed to the new cron package or you could write the parser yourself. @appleboy

lunny · 2023-08-22T07:01:51Z

Sorry, I made a mistake. Please follow #26655

Replace #22751 1. only support the default branch in the repository setting. 2. autoload schedule data from the schedule table after starting the service. 3. support specific syntax like `@yearly`, `@monthly`, `@weekly`, `@daily`, `@hourly` ## How to use See the [GitHub Actions document](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule) for getting more detailed information. ```yaml on: schedule: - cron: '30 5 * * 1,3' - cron: '30 5 * * 2,4' jobs: test_schedule: runs-on: ubuntu-latest steps: - name: Not on Monday or Wednesday if: github.event.schedule != '30 5 * * 1,3' run: echo "This step will be skipped on Monday and Wednesday" - name: Every time run: echo "This step will always run" ``` Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com> --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>

yardenshoham added type/enhancement An improvement of existing functionality topic/gitea-actions related to the actions of Gitea labels Feb 4, 2023

appleboy force-pushed the schedule branch from 066128d to 5333fd1 Compare February 5, 2023 02:01

lunny added this to the 1.20.0 milestone Feb 6, 2023

lunny mentioned this pull request Feb 6, 2023

An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins #13539

Closed

20 tasks

wolfogre reviewed Feb 16, 2023

View reviewed changes

GiteaBot added lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 16, 2023

appleboy force-pushed the schedule branch from c61ce25 to 054873b Compare March 10, 2023 10:32

appleboy force-pushed the schedule branch from b1c0ef9 to dc1a6c1 Compare March 14, 2023 09:04

lunny reviewed Mar 15, 2023

View reviewed changes

Comment thread models/actions/schedule.go Outdated

Comment thread models/actions/schedule.go Outdated

Comment thread models/actions/schedule_spec.go Outdated

lunny reviewed Mar 15, 2023

View reviewed changes

Comment thread services/actions/notifier_helper.go

lunny reviewed Mar 15, 2023

View reviewed changes

wolfogre reviewed Mar 17, 2023

View reviewed changes

appleboy commented Mar 17, 2023

View reviewed changes

Comment thread services/actions/schedule_tasks.go Outdated

lunny reviewed Mar 18, 2023

View reviewed changes

Comment thread models/actions/schedule.go Outdated

yardenshoham added type/feature Completely new functionality. Can only be merged if feature freeze is not active. and removed type/enhancement An improvement of existing functionality labels Mar 18, 2023

lunny reviewed Apr 8, 2023

View reviewed changes

Comment thread modules/actions/workflows.go Outdated

lunny reviewed Apr 11, 2023

View reviewed changes

Comment thread services/actions/schedule_tasks.go Outdated

techknowlogick changed the title ~~chore(actions): support cron schedule task.~~ chore(actions): support cron schedule task Apr 17, 2023

telackey mentioned this pull request Apr 17, 2023

Tracking issue for Gitea CI bugs/limitations cerc-io/hosting#25

Closed

7 tasks

silverwind added this to the 1.21.0 milestone Jun 12, 2023

appleboy mentioned this pull request Jul 6, 2023

Implement auto-cancellation of concurrent jobs if the event is push #25716

Merged

silverwind approved these changes Aug 13, 2023

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Aug 13, 2023

wxiaoguang reviewed Aug 13, 2023

View reviewed changes

Comment thread models/actions/schedule_list.go Outdated

wxiaoguang reviewed Aug 13, 2023

View reviewed changes

Comment thread services/actions/schedule_tasks.go Outdated

wxiaoguang reviewed Aug 13, 2023

View reviewed changes

appleboy changed the title ~~chore(actions): support cron schedule task~~ WIP: chore(actions): support cron schedule task Aug 13, 2023

wxiaoguang reviewed Aug 13, 2023

View reviewed changes

pull-request-size Bot added size/XXL and removed size/XL labels Aug 22, 2023

lunny closed this Aug 22, 2023

lunny force-pushed the schedule branch from 6265473 to bd8a253 Compare August 22, 2023 06:49

GiteaBot removed this from the 1.21.0 milestone Aug 22, 2023

pull-request-size Bot added size/XS and removed size/XXL labels Aug 22, 2023

lunny mentioned this pull request Aug 22, 2023

chore(actions): support cron schedule task #26655

Merged

extremelyonline mentioned this pull request Oct 26, 2023

schedule cron task not working #27794

Closed

go-gitea locked as resolved and limited conversation to collaborators Nov 20, 2023

Uh oh!

Conversation

appleboy commented Feb 4, 2023

How to use

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfogre commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeripath commented Mar 11, 2023

Uh oh!

appleboy commented Mar 11, 2023

Uh oh!

codecov-commenter commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

appleboy commented Mar 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lunny commented Mar 24, 2023

Uh oh!

appleboy commented Mar 31, 2023

Uh oh!

Uh oh!

Uh oh!

techknowlogick commented Apr 17, 2023

Uh oh!

delvh commented Jun 12, 2023

Uh oh!

BrunoBernardino commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

puni9869 commented Jul 30, 2023

Uh oh!

denyskon commented Aug 1, 2023

Uh oh!

ghost commented Aug 10, 2023

Uh oh!

silverwind commented Aug 10, 2023

Uh oh!

Uh oh!

denyskon commented Aug 13, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lunny commented Aug 22, 2023

Uh oh!

wolfogre commented Feb 16, 2023 •

edited

Loading

codecov-commenter commented Mar 14, 2023 •

edited

Loading

BrunoBernardino commented Jul 25, 2023 •

edited

Loading