Introduction
Hi guys, today let’s learn why the Archival Process of PowerApps Forms is a very important task and how we can achieve it.
Archival Process
We all know that a normal SP list can hold 30 million list items/records in it. Also, we know that a normal SP List View has a Threshold Limit of 5k which can be tuned ton 10k or more by going to Central Admin for On-Prem OR by using PowerShell for Online Environments.
But using PowerApps forms, customizations with Functions have a still lower limit of 2,000 max as shown below. If you try to add more to it, then you will get an error. It's a simple process Goto make.powerapps.com >> Select Apps >> Settings >> Advanced Settings >> Data Row Delegate for Non-Delegate Queries >> Max 2000 can only be entered but not beyond [****PowerApps Limitation]. If you have more than 2k records, say 3k, there will be a difference in the Set ID column value generation whose ‘Default’ section in PowerApp has this formula,
If(SharePointForm1.Mode=New,(Last(<ListName>).ID+1),ThisItem.'Set ID')
The set ID value for a new form will generate 2001, only though we have 3000+ records. To tackle this, we need to come up with an archival process. An archival process is used because the PowerApps function threshold limit is 2k, as verified from my R&D. If we have more than 2k list records with the PowerApps customization, we need to handle it to maintain the Functionality for all the PowerApp functions to run smoothly.
For the above scenario, the only solution is an archival process that includes the below steps:
Create a Cloned Target SP Online List in another Site Collection/Tenant using Modern OOTB features List selection with Site collection selection if in the same Tenant OR use PnP PowerShell scripting.
Follow the Steps for the PowerApps migration process in my previous Blog.
We can use SPMT Tool but it doesn’t allow Online Src List to Online Target List migrations.
SPMT lets you migrate content to SharePoint Online and OneDrive from the following locations:
- SharePoint Server 2013
- SharePoint Foundation 2013
- SharePoint Server 2010
- SharePoint Foundation 2010
- Network and local file shares
You need to use the Export to Excel option in the Source List to save it as a CSV file. Then use it as your source data and connect to your target SP List for SPMT Migration.
The best option is to use a 3
rd Party ShareGate Tool to migrate all the content from the source list to the target list. Please go through this
blog on ShareGate for all steps.
Once all the old records are migrated and validated in the Target list, clean up the Source List by deleting all of the old migrated records programmatically using PnP or SPOnline PowerShell scripting based on some unique ID value like the SET ID column value.
Let me summarize them for a quick overview:
We have 2 types of Lists: PowerApps Customized Modern Lists / Non-PowerApps Customized Lists.
For the former, the Threshold limit is 2k due to the PowerApp functions while the Latter has the ListView Threshold 5k.
For both scenarios, we need an archival solution. Here are the major points:
- To create an Archival Site Collection.
- To create an exact replica of the above Source List columns, look ups, PowerApps functionalities if available [Migrating PowerApp form], SPDesigner WFs and create the final Target SP Lists in the above Archival Site Collection.
- Set the Created Target SP List to READ ONLY.
- Set the Threshold Limit to the same as Source Lists as above.
- Migration using 3rd Party ShareGate Tool.
- Verification of all the old Src List records in the Target List from the Client.
- Approval from the client to clean up old records in the Source List.
Clean up of Src List records programmatically using a Scheduled Jobs through
MSFlows OR PowerShell scripting by checking on a daily basis manually.
***Active WFs related List Items should be carefully ignored for Archival Process.
Challenges to be addressed
Archive list items/documents/pictures based on user-defined data filters (constructed based on SharePoint columns), especially content that is no longer required or actively accessed by users. SharePoint content being voluminous and structured, it is important that you extract only the desired content based on various filter criteria (e.g., modified date, created by, stale libraries/lists, etc.)
Synchronize content in SharePoint libraries and file servers. You must be able to export SharePoint content that has recently changed or have been modified, since the previous export/archive session, in order to avoid exporting full content over and over again.
Export/archive content along with metadata for ready consumption by third-party applications. You need a solution that allows you to export folders/documents along with their associated metadata from SharePoint document libraries. The exported metadata and the content must be easy to manage after export, preferably in human-readable file format/structure. In short, the exported content and metadata must be ready for automation using other tools/applications without the need for manual intervention, such as modifying or cleansing the metadata and/or the content.
Archive content from multiple SharePoint libraries/lists to file server/network shares/local computer and maintain folder hierarchy along with the original version stack. You need to export to various types of file systems as per the original hierarchy and document classification so that it replicates the original SharePoint content as much as possible. Maintain source folder structure and version history when exporting data from the SharePoint list/document library. This is especially important if you need to manage older versions for archival purposes.
Backup SharePoint content in file shares or file servers and migrate contents back to new SharePoint libraries/lists in a new server after de-duplication and metadata cleansing. Duplicate documents are likely to be uploaded to SharePoint over time, despite organizational policies & practices being in place. It is an administrative nightmare for content owners to maintain the document repositories up-to-date with the required content in the respective places.
Please follow the above Archival Process each time, multiple times once the Source List Or Archival List1/List2/List3 also reaches the max threshold values.
Archival list naming should be done based on the time stamp for easy organization, sorting and quick referencing the records/items when tried.
Repeat the above Archival process multiple times as the quantity keeps on increasing to the threshold limit of 2k records on your READ-ONLY based archive power apps list by saving/creating them with a timestamp. Keep a note of them for future related access to your users.
Wait for part 2 for a faster and smarter approach of using MS Flows for archival processing without using the migration tool.
Happy SharePoint Archiving !