Metadata crawler improvments #665
Conversation
Andrei-Dolgolev
changed the title
| """ | ||
| retry = 0 | ||
| result = None | ||
| while retry < 3: |
There was a problem hiding this comment.
I sugest to re-write this func to avoid increment of retry var only in exceptions. But would do this by default at the end of loop and if response.status == 200: then write result and break the while loop.
But if this version works well, then probably there are no need any changes.
| f"Start crawling {len(not_updated_tokens)} tokens of address {address}" | ||
| ) | ||
|
|
||
| for requests_chunk in [ |
There was a problem hiding this comment.
Hard to read, probably better to add another variable on top of this for loop.
|
|
||
| if token_uri_data.token_id not in already_parsed: | ||
| metadata = crawl_uri(token_uri_data.token_uri) | ||
| with ThreadPoolExecutor( |
There was a problem hiding this comment.
With thread pool executor better to store results and wait them like in this code https://github.com/bugout-dev/moonstream/blob/e5240d2fd85685383bb40ea4cad741ec90b4fd6d/crawlers/mooncrawl/mooncrawl/blockchain.py#L367
There was a problem hiding this comment.
So this threads which inside another loop from list comprehension range will be under better control))
Changes
How to test these changes?
Related issues