How to evaluate DeepGlobe graph with APLS metric?

[tansor]

Hi, thank you for your response.
I wonder how to evaluate the inferred graph using APLS metric?
What data structure do you use to evaluate, and what's the key attribution that the data structure must have?
The reason why I ask so is that, according to https://github.com/CosmiQ/apls, the evaluation metric seems to be connected with lon/lat coordinate or utm coordinate.
Source codes may help more! Thx!

[anilbatra2185]

Hi @tansor, thanks for the query. I used the simple skeletonization of roads and convert to line-string formats. You can refer Spacenet-Road Challenge 2 winners source code. Evaluate the APLS metrics by APLS Visualizer tool with source code available at link, which is released as part of challenge.

[tansor]

Hi @anilbatra2185 , thanks for your answer. I'm still confused about how to obtain the DeepGlobe gt graph. Because the test method only allows gt graph to be networkx pkl or geoJson.

parser.add_argument('--test_method', default='gt_pkl_prop_pkl', type=str,
                        help="Method for creating ground truth and proposal "
                        + "graphs. Options:"
                        + " gt_pkl_prop_pkl   = ground truth pickle, proposal pickle"
                        + " gt_json_prop_pkl  = ground truth json, proposal pickle"
                        + "gt_json_prop_wkt  = ground truth json, proposal csv in wkt format"
                        + "gt_json_prop_json = ground truth json, proposal json"
                        )

[anilbatra2185]

Hi @tansor, I used the same steps of skeletonization and conversion to line-strings using the DeepGlobe segmentation GT mask. And now it will act as GT graph for deepglobe and you might need to do small changes in APLS visualizer to compare deepglobe GT and predicted graph.

STEPS for DG

• Convert GT mask into skeleton mask.
• Convert to GT graph
• Modify APLS visualizer tool (specifically some paths to read images)
• Evaluate APLS metric for DG using the predicted and GT graph.

Let me know, if you see any issues/concerns.

[tansor]

Thank you for your patience!
You have solved my confusion. This issue can be closed now.

[heeji289]

I also wonder how you evaluate DeepGlobe dataset with APLS metric. I tried to do that via code provided by "https://github.com/CosmiQ/apls", but I failed to use it on DeepGlobe dataset.

Please give me some details.. I read above talks, but I still didn't understand.

[tansor]

Hi, @hjlim0314.
The author did not use the code provided by "https://github.com/CosmiQ/apls", but a java implementation called APLS Visualizer tool in "https://drive.google.com/file/d/1rwbj_o-ELBfruPZuVkCnEQxAX2-Pz5DX/view?usp=sharing", which is released as part of the SpaceNet challenge.

[heeji289]

@tansor Thank you for your tip! It will be very helpful

[antran89]

Hi @hjlim0314, @tansor I just made some modifications for DeepGlobe APLS metric measurement. Please use if needed, https://github.com/antran89/road_visualizer

[heeji289]

Hi @hjlim0314, @tansor I just made some modifications for DeepGlobe APLS metric measurement. Please use if needed, https://github.com/antran89/road_visualizer

Thank you for your code. It works on my deepglobe data set very well.
I appreciate you notice me about that.