Indexer returns no results for some terms

[gerroon]

• Gitea version (or commit ref): 1.11.0-rc1
• Git version: 2.24.1
• Operating system: Debian testing
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • [X ] Not relevant

Description

I enabled the indexer. It has been running for couple days since then. I am able to search and get some results but some results return no results by the Code search page meanwhile I can get 10s on results for with grep

For the term "tool_set" in the Code search page I get No source code matching your search term found.

Grepping the same code base (eve after deleting the comment lines)

find -type f -name "*.py" -exec grep -i 'tool_set' {} \; |sed '/#/d' |wc -l 44

ini

[indexer]
REPO_INDEXER_ENABLED = true
ISSUE_INDEXER_PATH: indexers/issues.bleve
REPO_INDEXER_PATH: indexers/repos.bleve
UPDATE_BUFFER_LEN: 20
MAX_FILE_SIZE: 1048576

[guillep2k]

The indexer itself can handle your case. I've specifically tested with tool_set and it was indexed correctly when I ran the indexer from scratch. The indexer is having some problems, however, because I'm getting errors in the log I can't pinpoint like:

2020/01/22 21:04:05 ...ndexer/code/queue.go:39:processRepoIndexerOperationQueue() [E] indexer.Index: exit status 1
        /home/gprandi/src/code.gitea.io/gitea/modules/indexer/code/queue.go:39 (0x187f5f7)
                processRepoIndexerOperationQueue: log.Error("indexer.Index: %v", err)
        /home/gprandi/go/src/runtime/asm_amd64.s:1357 (0x46f5d0)
                goexit: BYTE    $0x90   // NOP

Which clogs the indexer queue. If I restart the instance and commit new changes to the repository, the indexer seems to pick them up correctly.

The indexer is expected to take a "long time" to build, but not days. It took a couple of minutes to build from scratch my indexes on 327 MB of repositories.

[gerroon]

That is interesting.Where is a good place to see the indexer having issues? I did grep on he gitea log but not much that I can see

https://paste.debian.net/hidden/3591c5ca/

I also wonder if there is a limit to the size of the indexer db, mine is it at 285mb now and I have many repos in there.

[guillep2k]

My log configuration in app.ini:

[log]
MODE             = file
MAX_DAYS         = 15
LEVEL = Info
ROUTER           = file
ROUTER_LOG_LEVEL = Trace
STACKTRACE_LEVEL = Error
XORM = file
REDIRECT_MACARON_LOG = true

[log.file.xorm]
FILE_NAME = xorm.log

(It's a little redacted, so maybe not all options make sense)

This separates the SQL (XORM) log from the other logs, making everything cleaner. I've also set up a trace to every error, so I know exactly where every log is produced.

To get a meaningful log I stopped Gitea and deleted the repos.bleve directory to force the system to rebuild them when restarted. You'll know it finished when it stops growing (which is not necessarily when the log says it does... in fact my log was not useful about that).

Then I've edited a file using the web UI, and when the indexer attempted to do its thing, it crashed.

(NOTE: your paste doesn't say much, unfortunately)

[lunny]

@guillep2k what's the gitea version?

[guillep2k]

@lunny I've tested on master as of today. (53f9dbf)

[guillep2k]

I also wonder if there is a limit to the size of the indexer db, mine is it at 285mb now and I have many repos in there.

BTW, the indexes of my prod instance are 1.3GB from 1.4GB of repositories (working fine on Gitea 1.10.3).

[gerroon]

@guillep2k I will test with the latest rc2 from today. I will delete the database and force it again.

Btw is there a way to force the indexer while gitea is running?

[guillep2k]

Btw is there a way to force the indexer while gitea is running?

If by force you mean rebuild all, no, there isn't. But files are re-indexed with each commit (only the affected files, the whole file is re-indexed, not just the diff).

[gerroon]

Hmm the latest rc2 fails on me with

2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `name` FROM `user` WHERE `id`=? LIMIT 1 []interface {}{1} - took: 26.358µs
2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `name` FROM `user` WHERE `id`=? LIMIT 1 []interface {}{1} - took: 44.862µs
2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `name` FROM `user` WHERE `id`=? LIMIT 1 []interface {}{1} - took: 34.792µs
2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `name` FROM `user` WHERE `id`=? LIMIT 1 []interface {}{1} - took: 27.177µs
2020/01/22 22:52:07 ...exer/code/indexer.go:54:func2() [I] PID: 3759700 Initializing Repository Indexer at: /opt/gitea/indexers/repos.bleve
2020/01/22 22:52:07 ...er/issues/indexer.go:142:func2() [I] PID 3759700: Initializing Issue Indexer: bleve
2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `pull_request`.`id` FROM `pull_request` WHERE (status=?) []interface {}{1} - took: 158.714µs
2020/01/22 22:52:07 .../xorm/session_raw.go:78:queryRows() [I] [SQL] SELECT `id`, `repo_id`, `hook_id`, `uuid`, `type`, `url`, `signature`, `payload_content`, `http_method`, `content_type`, `event_type`, `is_ssl`, `is_delivered`, `delivered`, `is_succeed`, `request_content`, `response_content` FROM `hook_task` WHERE (is_delivered=?) []interface {}{false} - took: 213.028µs
2020/01/22 22:52:07 routers/init.go:122:GlobalInit() [I] SQLite3 Supported
2020/01/22 22:52:07 routers/init.go:46:checkRunMode() [I] Run Mode: Production
2020/01/22 22:52:07 ...ndexer/code/bleve.go:228:Close() [D] Closing repo indexer
2020/01/22 22:52:07 ...ndexer/code/bleve.go:235:Close() [I] PID: 3759700 Repository Indexer closed
2020/01/22 22:52:07 ...exer/code/indexer.go:63:func2() [F] PID: 3759700 Unable to initialize the Repository Indexer at path: /opt/gitea/indexers/repos.bleve Error: error parsing mapping JSON: unexpected end of JSON input
        mapping contents:

        /go/src/code.gitea.io/gitea/modules/indexer/code/indexer.go:63 (0x124255f)
        /usr/local/go/src/runtime/asm_amd64.s:1357 (0x466c70)

[lunny]

Could you find the file rupture_sharded_meta.json on indexer directory ?

[gerroon]

There is no rupture_sharded_meta.json

find -L -type f|grep -i rupt
./indexers/issues.bleve/rupture_meta.json
./indexers/repos.bleve/rupture_meta.json

[lunny]

@gerroon could you paste the content of that two files?

[gerroon]

cat issues.bleve/rupture_meta.json  repos.bleve/rupture_meta.json 

{"version":1}{"version":4}

[gerroon]

Ok, I deleted the whole indexer thing, installed the latest nightly (v1.11.0-rc2) . The database grew to 3gb

-rw-r--r-- 1 git git   47 Jan 23 00:04 index_meta.json
-rw-r--r-- 1 git git   13 Jan 23 00:04 rupture_meta.json
-rw------- 1 git git 3.0G Jan 23 08:27 store

However it still cant find tool_set

I did another search for builtin. It located about 30 searches in the whole GItea contolled repos. Since I do not have the clones of all the repos, I made a search in the largest one I cloned for builtin It returned and the difference is by huge magnitutes, not even close (30 vs 542).

grep -ir "builtin" *|wc -l
542