[BadAIJsonAnswer] Expecting value: line 1 column 2 (char 1) - Supporting multiple response types from LLM

[gczajkowskiTT]

Hi again! I updated to Gito/4.0.3 and intermittently seeing these errors with gito review

Processing Warnings

• File scripts/add_cg.py was skipped due to error: [BadAIJsonAnswer] Expecting value: line 1 column 2 (char 1).

Should I enable gito --verbose review or is there a way to get more debug information? The previous 3.4.0 version did not have this issue.

[Nayjest]

@gczajkowskiTT

You can check logs in the action output for full details, I believe a record with an LLM answer containing incorrect JSON should be there (one of the "LLM Response" records above the error, not necessarily exactly the latest one, because requests are processed in parallel).

But I wouldn't put a lot of effort into it. LLMs just make mistakes from time to time.
And Gito ignores such cases after N retries to finish the review (Did you receive the final report?)

If error repeats -- maybe there is something in your add_cg.py confusing LLM, for example your app prompts.

If this kind of error happens often for your LLM, you can increase the number of repeats in .gito/config.toml:
https://github.com/Nayjest/Gito/blob/main/gito/config.toml#L136 or slightly increase LLM temperature parameter (carefully) if it produces exactly same output for a file each time.

[Nayjest]

@gczajkowskiTT
Info on verbosity:
https://github.com/Nayjest/Gito/blob/main/gito/cli.py#L76

Use it this way if LLM responses are not present in the output:

gito -v2 review ....rest of arguments here...

[gczajkowskiTT]

hmm.. how can I associate the requests at the bottom of the log

2026-04-06 21:12:15 INFO: get_repo_context: Using local repo...
2026-04-06 21:12:15 INFO: No project config found at �[34m/project/work1/.gito/config.toml�[39m, using defaults
2026-04-06 21:12:15 INFO: Merge base(�[32mbugfix/automation�[39m,�[33morigin_git/main�[39m) --> �[36m2afe87795f905a1ec0339f6c9b2b167c013f6f41�[39m
2026-04-06 21:12:16 INFO: Using merge base: �[36m2afe8779�[39m (fix for main regression lists)
2026-04-06 21:12:16 INFO: Making diff: �[32mINDEX�[39m vs �[33m2afe87795f905a1ec0339f6c9b2b167c013f6f41�[39m
2026-04-06 21:12:16 �[33mWARNING�[39m: �[33mCan't resolve tiktoken encoding for 'anthropic/claude-sonnet-4-6'. Default encoding will be used.�[39m
2026-04-06 21:12:16 �[33mWARNING�[39m: �[33mCan't resolve tiktoken encoding for 'anthropic/claude-sonnet-4-6'. Default encoding will be used.�[39m
2026-04-06 21:12:16 �[33mWARNING�[39m: �[33mCan't resolve tiktoken encoding for 'anthropic/claude-sonnet-4-6'. Default encoding will be used.�[39m
2026-04-06 21:12:16 �[33mWARNING�[39m: �[33mCan't resolve tiktoken encoding for 'anthropic/claude-sonnet-4-6'. Default encoding will be used.�[39m
2026-04-06 21:12:17 INFO: HTTP Request: POST https://proxy.company.com/chat/completions "HTTP/1.1 200 OK"
2026-04-06 21:12:40 INFO: HTTP Request: POST https://proxy.company.com/chat/completions "HTTP/1.1 200 OK"
2026-04-06 21:12:40 �[31mERROR�[39m: �[31mLLM error: Expecting ',' delimiter: line 1 column 3 (char 2)�[39m
2026-04-06 21:12:40 INFO: Retrying... 3 retries left
2026-04-06 21:13:08 INFO: HTTP Request: POST https://proxy.company.com/chat/completions "HTTP/1.1 200 OK"
2026-04-06 21:13:08 �[31mERROR�[39m: �[31mLLM error: Expecting ',' delimiter: line 1 column 3 (char 2)�[39m
2026-04-06 21:13:08 INFO: Retrying... 2 retries left
2026-04-06 21:13:21 INFO: HTTP Request: POST https://proxy.company.com/chat/completions "HTTP/1.1 200 OK"
2026-04-06 21:13:21 �[31mERROR�[39m: �[31mLLM error: Expecting ',' delimiter: line 1 column 3 (char 2)�[39m
2026-04-06 21:13:21 INFO: Retrying... 1 retries left
2026-04-06 21:13:31 INFO: HTTP Request: POST https://proxy.company.com/chat/completions "HTTP/1.1 200 OK"
2026-04-06 21:13:31 �[31mERROR�[39m: �[31mExpecting ',' delimiter: line 1 column 3 (char 2)�[39m
2026-04-06 21:13:31 INFO: Running pipeline... [env: �[33mci�[39m]
2026-04-06 21:13:31 INFO: Running pipeline step: jira

with the top of the log submissions?

Requesting LLM anthropic/claude-sonnet-4-6:
    [User]:
        You are a subsystem of an AI-powered software platform, specifically tasked with performing expert code reviews.
...
Requesting LLM anthropic/claude-sonnet-4-6:
    [User]:
        You are a subsystem of an AI-powered software platform, specifically tasked with performing expert code reviews.
...
LLM Response:
    []
LLM Response:
    Looking at this diff, it's purely a line-ending conversion (CRLF to LF or vice versa) with no actual code changes. The content is identical between old and new. However, I should still review the code itself for real issues.
...

as these can be out of order.

[gczajkowskiTT]

1. It would be very helpful if the submissions and responses were numbered in the log so it they could be correlated

Instead of just

Requesting LLM: anthropic/claude-sonnet-4-6:
   ...
Requesting LLM: anthropic/claude-sonnet-4-6:
   ...
LLM Response:
   []
LLM Response:
   []
...
=== STDERR ===
...
INFO: HTTP Request: POST
INFO: HTTP Request: POST
ERROR: LLM error: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

It would be more along the lines of

Requesting LLM #1: anthropic/claude-sonnet-4-6:
   ...
Requesting LLM #2: anthropic/claude-sonnet-4-6:
   ...
LLM Response #1:
   []
LLM Response #2:
   foo
...
=== STDERR ===
...
INFO: HTTP Request #1: POST
INFO: HTTP Request #2: POST
ERROR: LLM error #2: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

[Nayjest]

Thanks! Great idea, going to implement that

[gczajkowskiTT]

Here is the breakdown of LLM Response content

1. JSON empty list
   LLM Response:
   []

2. JSON correct list
   LLM Response:
   [
   {
   "title": ...
   }
   ]

3. Markdown JSON: triple backtick plus "json" in front (which I cannot show below because it's being interpreted as markdown (this would be correct json if the backticks were stripped).
   LLM Response:
   backtickbacktickbacktickjson
   [
   {
   "title": ...
   }
   ]
   backtickbacktickbacktick

4. Thinking summary and then json
   LLM Response:
   Looking at the diff, I need to analyze the change carefully.
   ...
   []

5.Thinking summary and then Markdown-jsonLLM Response:
Looking at the diff, I need to analyze the change carefully.
...
backtickbacktickbacktickjson
[]
backtickbacktickbacktick

Claude suggests this could be handled

Two files failed completely after all retries:                                                                                                                                                                                      
  - command.py
  - dist.py                                                                                                                                                                                             
                                                                                                                                                                                                       
  Error 1 (lines 28463, 28484, 28490+): Expecting property name enclosed in double quotes: line 1 column 2 (char 1) — LLM returned markdown fences or single-quoted JSON.                                                             
                                                                                                                                                                                                                                      
  Error 2 (line 28480): Extra data: line 1 column 17 (char 16) — LLM returned multiple standalone JSON objects instead of a single array.                                                                                             
                                                                                                                                                                                                                                      
  ---                                                                                                                                                                                                                                 
  Gito Package: LLM Response Parsing                                                                                                                                                                   
                                    
  Key files:
  - core.py:506-523 — invokes mc.llm_parallel(..., parse_json={"validator": _llm_response_validator})                                                                                                                                 
  - core.py:463-473 — _llm_response_validator() validates response is list[dict] matching RawIssue                                                                                                                                    
  - report_struct.py:52-75 — RawIssue dataclass defines expected schema                                                                                                                                                               
                                                                                                                                                                                                                                      
  Expected schema (strict):                                                                                                                                                                            
  [                                                                                                                                                                                                                                   
    {                                                                                                                                                                                                  
      "title": "string",           // REQUIRED                                                                                                                                                                                        
      "details": "string",         // optional                                                                                                                                                         
      "severity": 1-5,             // optional int                                                                                                                                                                                    
      "confidence": 1-4,           // optional int
      "tags": ["string"],                                                                                                                                                                                                             
      "affected_lines": [{"start_line": int, "end_line": int, "proposal": "string"}]                                                                                                                   
    }                                                                               
  ]                                                                                                                                                                                                                                   
   
  Critical Gap: No Markdown Stripping                                                                                                                                                                                                 
                                                                                                                                                                                                                                      
  Gito has no pre-processing to strip markdown fences from LLM responses. It delegates everything to microcore's parse_json handler, which expects raw JSON only. When an LLM wraps output in ```json ``` blocks — which Claude models
   frequently do — parsing fails and retries are consumed.                                                                                                                                                                            
                                                                                                                                                                                                                                      
  Retry Behavior                                                                                                                                                                                       

  - mc.llm_parallel(..., retries=cfg.retries, allow_failures=True) handles retries                                                                                                                                                    
  - On exhaustion, failed responses become [] — the file is silently skipped
  - The log shows 3-4 retries per failure before giving up                                                                                                                                                                            
                                                                                                                                                                                                                                      
  ---                                                                                                                                                                                                                                 
  Root Cause                                                                                                                                                                                                                          
                                                                                                                                                                                                       
  The two permanently failed files (command.py, dist.py) are likely longer/more complex diffs where the LLM consistently wraps its JSON in markdown code fences. Gito has no JSON extraction fallback — it only retries
  the same prompt, so all retries produce the same fenced output and fail identically.                                                                                                                                                
  
  Potential fix: Add a pre-processing step in core.py before the validator to strip ```json ``` fences using a regex like re.search(r'```(?:json)?\s*(\[.*?\])\s*```', response, re.DOTALL).