[Do not review] Raw resources in the Lake, raw resource separation, history separation, resource id map #4690
Conversation
* History separation V0 * Forgotten resource * replace current by view * TRUNCATE * HardDelete * delete history v0 * missed 64 and removed dead code from SQL query generator * Removed history from current in merge * Triggers * tests and tran * disable/enable indexes * rollback hard delete and merge * rollback delete invisible history * TRUNCATE -> DELETE * line * leftovers * PK check * Cosmetic * WHERE nanes * Get resources with forced indexes * Removed redundant where * Adding feature flag for raw resource dedupping * Enable invisible history by default * parameters * commit * fixed typo * 65 + more realistic diff * adjusted update trigger * right trigger * next iteration * special chars * Added delete and update * \r * Adding script runner * Added verification * cosmetic * HOLDLOCK hint * removed incorrect comments * RT * lock timeout * 180 and remove empty * no changes in update resource search params * blob rewriter tool * Revert "no changes in update resource search params" This reverts commit b2e0c38. * comment * Generic disable indexes and update search params * Dummy resources * fixes * Dummy records based on surr id * adjust test to filter dummy rows * exclude history and current * Adding PerfTest V-1 * Get asyn wrapper without type string to id function * tool * not exists on data copy * packages back * Added comments. * Correct filtered index on resource current * Added redundant IsHistory=0 to index * deduping * Removed dummy records * pp-p * history clause * deduping * testing parameters * get resource by type and surr id range with many versions * skip large databases * Added calls to fhir * added conflict * max retries = 3 * database pings * examples * reverse * start closer * put logic * Create * Move 65 to 84 * Fixes after merge * Adding RawResources table * change capture tweeks * Test fix * Added MI * inline insert * Repeat on updates
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
| private static string _adlsAccountName; | ||
| private static string _adlsAccountKey; | ||
| private static Uri _adlsAccountUri; | ||
| private static string _adlsAccountManagedIdentityClientId; |
Check notice
Code scanning / CodeQL
Missed 'readonly' opportunity Note
| private static string _adlsAccountKey; | ||
| private static Uri _adlsAccountUri; | ||
| private static string _adlsAccountManagedIdentityClientId; | ||
| private static BlobContainerClient _adlsClient; |
Check notice
Code scanning / CodeQL
Missed 'readonly' opportunity Note
| private static Uri _adlsAccountUri; | ||
| private static string _adlsAccountManagedIdentityClientId; | ||
| private static BlobContainerClient _adlsClient; | ||
| private static bool _adlsIsSet; |
Check notice
Code scanning / CodeQL
Missed 'readonly' opportunity Note
| { | ||
| if (!_adlsIsSet) | ||
| { | ||
| _adlsAccountName = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountName"); |
Check notice
Code scanning / CodeQL
Static field written by instance method Note
| if (!_adlsIsSet) | ||
| { | ||
| _adlsAccountName = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountName"); | ||
| _adlsConnectionString = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsConnectionString"); |
Check notice
Code scanning / CodeQL
Static field written by instance method Note
| } | ||
|
|
||
| var db = sqlRetryService.Database.Length < 50 ? sqlRetryService.Database : sqlRetryService.Database.Substring(0, 50); | ||
| _adlsContainer = $"fhir-adls-{db.Replace("_", "-", StringComparison.InvariantCultureIgnoreCase).ToLowerInvariant()}"; |
Check notice
Code scanning / CodeQL
Static field written by instance method Note
| var uriStr = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountUri"); | ||
| if (uriStr != null) | ||
| { | ||
| _adlsAccountUri = new Uri(uriStr); |
Check notice
Code scanning / CodeQL
Static field written by instance method Note
| if (uriStr != null) | ||
| { | ||
| _adlsAccountUri = new Uri(uriStr); | ||
| _adlsAccountManagedIdentityClientId = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountManagedIdentityClientId"); |
Check notice
Code scanning / CodeQL
Static field written by instance method Note
| var bytes = reader.GetSqlBytes(6); | ||
| var transactionId = reader.Read(VLatest.Resource.TransactionId, readRequestMethod ? 10 : 9); | ||
| var offsetInFile = reader.Read(VLatest.Resource.OffsetInFile, readRequestMethod ? 11 : 10); | ||
| var rawResource = ReadRawResource(bytes, decompress, transactionId.Value, offsetInFile); |
Check warning
Code scanning / CodeQL
Dereferenced variable may be null Warning
| { | ||
| rawResource = new Lazy<string>(_invisibleResource); | ||
| // -1 is a temp fix. Remove !!!!! | ||
| rawResource = offsetInFile.Value == -1 ? InvisibleResource : GetRawResourceFromAdls(transactionId.Value, offsetInFile.Value).Value; |
Check warning
Code scanning / CodeQL
Dereferenced variable may be null Warning
No description provided.