Data in be_Prop table

[ViliusRuskys]

I recently bumped into some issues with LineStyles not displaying properly in iModels after exporting them from one iModel to another (imodel-transformer).
After further investigation I found out that the LineStyle elements are extended into the be_Prop sqlite table, i.e. the LineStyle element holds a reference to a row in be_Prop table, where the actual display information for the line style is stored.

To do the export correctly I need to query all data related to a LineStyle and insert it into another iModel. That is where i found out that you cannot query information stored in be_Prop table using ECSql. The LineStyleDefinition.Utils (core-backend) class only contains methods for inserting the display information into the be_Prop table, but nothing for acquiring it.
To manually query the data I would need to use specific names and namespaces (that are hardcoded in itwinjs code), infer the line style type from the type marked by an enum value on the LineStyle element, and then handle the data I receive because styles can reference other styles from within the be_Prop table.

It can get confusing working with something like this because you are inserting data into an iModel which you cannot easily retrieve. Even more so if you are not the one who created the data and are only working with it and you are ignorant on the fact that this particular element is extended into some other DB table.

This already impacts things like:

• imodel-transformer where users are silently losing data
• imodel-diff-tool that is used for integrity checking and when comparing LineStyles false positives are produced because they appear to be identical when they actually might differ.

An even worse thing is that when I was only looking at the line styles, there were additionally more that 50 other data types stored in the be_Prop table (in the iModel I was investigating) so I probably don't even know what data I am missing along with the line styles.

Is there no better way of storing extended element data? Is there documentation somewhere on what is stored in be_Prop table, so that developers that are working with iModels can be aware of these edge cases?

[bbastings]

Is the transformer using?

DgnDbStatus GeometryStreamIO::Import(GeometryStreamR dest, GeometryStreamCR source, DgnImportContext& importer)

DgnImportContext has remap methods for all the ids that can be found in the geometry stream that need to be accounted for when copying between iModels.

I see that DgnImportContext currently has the code to try to copy the line style information over commented out?

DgnStyleId DgnImportContext::_RemapLineStyleId(DgnStyleId sourceId)
{
if (!IsBetweenDbs())
return sourceId;

DgnStyleId dest = FindLineStyleId(sourceId);
if (dest.IsValid())
    return dest;

#if defined(NEEDSWORK_LINESTYLES) // importers are not tested so don't risk passing along bad data.
dest = LineStyleElement::ImportLineStyle(sourceId, *this);
AddLineStyleId(sourceId, dest);
#endif

return dest;
}

If you are calling this, you can see if it works with the NEEDSWORK_LINESTYLES removed, otherwise maybe looking at LineStyleElement::ImportLineStyle is helpful.

Is there no better way of storing extended element data?

Was not someone's best idea. :)

[pmconne]

Importing a line style element (or any other element) should import all the data required by that element. If it does not, it's a bug.

[bbastings]

It was setup to remap ids in the geometry stream. If no element is using the style there wasn't a reason to bring it over. Who is importing the line style when it's not used and why is that useful? The line style elements reference the ids in the props table, so those have to exist before calling insert...you can't just insert elements into a different model without first calling some logic to deep copy so that the element being inserted can reference valid ids for the destination.

[ViliusRuskys]

The transformer clones and imports all definition elements prior to the geometric ones. That of course includes things that are referenced by geometries like GeometryParts, SubCategories, LineStyles etc. In between the import of definition and geometric elements is the state where there are Linestyles that are not yet referenced by any geometry.
Cloning the LineStyle only clones the element itself and inserts the LineStyle into the destinationDb without cloning the be_prop data row.
Even with the experimental code enabled, when we are already exporting the geometry it will detect that the LineStyle element already exists and skip the be_Prop processing.

DgnStyleId dest = FindLineStyleId(sourceId);
if (dest.IsValid())
    return dest;

As I see it now there are 3 options

1. Implement the be_Prop LineStyle data cloning logic on imodel-transformer side
2. Explicitly not import Linestyles, so native does it for us. After that We will need to update these Linestyles because preserving federation guids on cloned elements matters to us.
3. Native imports extended LineStyle data together with the LineStyle (it already does so, but only when importing it while processing another element's geometry stream).
   When processing the geometry all its referenced Elements are treated like an extension to it and all of it is handled (Geom->LineStyle->be_Prop), but it is not handled when these Elements are processed separately.

I'd like to avoid option 1 because someone might hit this issue again in the future and I'd prefer fixing the current commented out logic and going with option 2 or 3.

[bbastings]

Cloning the LineStyle only clones the element itself and inserts the LineStyle into the destinationDb without cloning the be_prop data row.

This means you are currently inserting an element with an invalid component id, because it's still the id from the source model.

For Option 3, if you are referring to implementing _CloneForImport on LineStyleElement, that's fine. As long as you populate the remap table, then like you said geometric elements will find the style id already exists in the destination if you've pre-cloned the definition elements.

One reason not to clone all the definition elements is that it unnecessarily bloats up the destination file with information that isn't needed/used. Geometry parts in particular can add significantly to the file size...

[ViliusRuskys]

You are correct. We currently import Linestyles with dangling component ids.
I was talking in a more generic sense, but _CloneForImport sounds like a nice solution.

[ColinKerr]

Users of the iTwin.js public API should not directly access data in be_Props and our API should handle copying any necessary data if it does exist there.

I created an issue: #7644 to track this bug.

@diegoalexdiaz I think this discussion question also raises a good point about the data stored in be_Props. We would probably store less data there if we were starting over again, maybe some data should be exposed as part of the public API and we could do so via a view only schema so it can be queried with ECSql.

[diegoalexdiaz]

maybe some data should be exposed as part of the public API and we could do so via a view only schema so it can be queried with ECSql

I've been under the assumption that such data could only be exposed via ECSQL functions. I didn't know that data could be also exposed via views (which seems to imply that view-classes can be based on raw SQL). That's neat!

I generally agree that there's data in be_Props that should be accessible via ECSQL (e.g. an iModel's GCS).