Automatic File ordering on Ingest/Update

[DiegoPino]

What is what we want?

Related to esmero/webform_strawberryfield#12 (but here since i want this to be part of src/StrawberryfieldFilePersisterService.php

One of the most common use cases is people uploading images that already have some type of naming convention related to a given order/sequence.

This is a quick first step that requires more code to be fully compliant with esmero/webform_strawberryfield#12 (comment), basically this won't respect a manual reorder, since it will always apply and re-apply this based on given file names. Still, it solves the most common and missing need. Give images a default order based on file name uploaded.

Some ideas on how to move forward:

1.- Only do this if no previous order: Which adds the following problem. What if someone adds during an edit an new File and there is already an order? Do we simply add the new image to the end? What if the order that is there was not manually given but automatic already? How do we do that differentiation?

2.- Have a key that allows overrides. Add an extra key that defines what order was applied. like 'sequence_type': 'natural' or 'manual' ?
If manual is present, don't touch. Still the question persist on what to do when adding a new one. Simply add the highest order +1 ? And let the user then, manually decide what to do? Could be a way

3.- There are sure other ordering issues i have not considered.

To be honest, i don't totally like the idea of another key inside the as:image etc structures. So maybe we can also do the following!. In case of manual order, why don't we add a fully new structure which acts as a ToC for that list of files? That would allow any arbitrary order, but still would allow to have, always, a natural ordered sequence like the one we generate automatically.

@giancarlobi @mitchellkeaney @marlo-longley i would love your opinion, hopefully any of you can in the few days think about use cases, edge cases and UIs to deal with this. Thanks

This is also related to esmero/webform_strawberryfield#12

[marlo-longley]

@DiegoPino this is a nice approach coming directly from use cases.

It seems like you addressed point 2 in your comments for #61 - If the current order, is already totally not in Natural Order, but sequence keys with values are present, when assume its manually ordered. So to force an automatic ordering we just would need a function that removes all sequence keys.- so we can basically know the sequence_type without a key.

The question of new files and where they fall in the order. Yes, if automatic, the filename will dictate position in the sort. If manual, add the end of the sequence and allow user to place. But what is the UI. File naming scheme only, a Textfield for editing file name?. If there are a large number of files to order, a drag/drop will not be a good UI. Scrolling up a page with something you are dragging is bad.

[DiegoPino]

@marlo-longley i guess the questions is where we set the toggle for 'force' reordering in the UI. One way would be to override the managed file upload widget, but that would only apply to things ingested via that way. About the actual ordering UI, i think we already discussed somewhere else how to approach that: esmero/webform_strawberryfield#12 The idea is not to use table drags (bad bad UI) but allow some automatic ordering options and also a UI where you can give a given File (via file name, eg) and an autocomplete, a order and it would push others down.
Still, we have to do it!

[DiegoPino]

@giancarlobi @alliomeria i'm trying to figure out a few things related to automatic ordering and would love to hear your input (nothing fancy, just some ideas)

Fact: right now i we order, always, files based on their filenames. This ordering is happening by adding a sequence key to each structured file data. The order used is natural order which means if the file ends in _1 it will come before of one that ends in _10

As seen here
https://play.archipelago.nyc/do/cf48b5f8-4e4d-4d3d-b3f7-2ec12f0b298d#mode/2up

Problem:

I don't like the re ordering i do every time. because if i need to manually tune the order (let's say i want to move a file that is named Cover.jpeg to -1 because all was already ordered correctly and i uploaded the Cover after the initial creation and i manually (RAW) edit add a -1 to the sequence of that file, Archipelago will simply ignore me and reorder.

Possible solutions:
Be smarter. But how? RAW ideas here

• Order newly added files first. now get the offset (max sequence number of existing files != file count because someone could have modified them manually?). If the 'name' of the first file in the sequence of newly added object is higher in sequence than the filename of the file with highest sequence in the existing ones list, then simply add that number (offset) to new ones. Opposite, push then before. This approach has many many issues. Its assuming either before or after the current block. Not relative internal orders.
• Simpler: ask the user. Do you want to reorder your files?
• Ignore order and keep manual?
• push new files at the end?

The thing is this: if i have the need of changing the order, means automatic will never work out. Means the system/will/not/be able to deduce a better order than myself right? So. If i detect that the automatic order was modified by the user. Do i simply assume no automatic ordering, unless asked for explicitly?

And then all this brings me back to 'ask the user'
How and when?

Any of you see at the bottom of the page some automatic ordering mechanism that could make sense in an cultural heritage institution scenario? https://www.geeksforgeeks.org/sort-array-according-order-defined-another-array/

This whole thing comes also because i uploaded that book (example i pasted) and tiff files were named using _recto and verso at the end, which totally broke my sorting. And then, i found some that ended just in _r and _v.

[giancarlobi]

@DiegoPino my thoughts:

• ordering on file names on first upload is better/simpler/(only?) method
  People are used to file names ordering so with strange file names I think people have to take care to rename correctly them, as probably a lot of people are already doing.
• after first upload, automatic ordering must be disable as IMHO never it will be useful/coherent with my needs (also with AI ... 😄 )
• do we need a procedure/UI/SimpleEditSBFJSONSequenceNumber to manually reordering, right?
  So, we can use it also when we upload a new file to add to an existing list.
  So, upload a new file will put it at the high sequence number free (= to the end).

Sorry for this very basic point of view, I'm following Archipelago philosophy ... simple, simple, simple ...
Have a nice day, amigo

[DiegoPino]

@giancarlobi i totally agree.
So concretely.

• ordering on file names on first upload is better/simpler/(only?)

YES. We do that. We should tell people how file naming goes (documentation)

-after first upload, automatic ordering must be disable as IMHO never it will be useful/coherent with my needs (also with AI ... 😄 )

I agree. AI is not even a real word! hahaha

But i have question is. If we go for never ever touch ordering, are there any exclusions?conditions we want there?. Like what happens if people are playing around remove ordering for certain keys. Should we assume a single missing sequence implies we need to reoder? or do we push the ones without order to the end? LIke in that example. To get the order right i ended renaming the files directly in the JSON.

• do we need a procedure/UI/SimpleEditSBFJSONSequenceNumber to manually reordering, right?

Yes. We need the ordering file tab. Should also be a replace file tab, remove file tab, in general a File/Attachment tab in every Object