hello fellows 🙂
I hope you are all fancy good.
Today we will handle the migration between systems. Like upgrading to a new system, or replacing old system with a new different one.
Checking old system
When you need to upgrade/replace a system, you should check what do you need to import from the old system. Defined the tables, columns, type of the columns in case of a database migration, or files like xmls, folder structures with xml data. You can also use to migrate from SharePoint for instance to your new system.
It’s all the same, just you need to analyze that system and what do you need to import.
Create Middle-system mapping
Second this you need to do is creating a middle-system to be working as a mapping between old system and new system. For example, you are migrating list of a users from “A” to “B”. You need to add that list in a system “C” and to keep the identifier for each item. Also you can add a status for each item, so you can differentiate between the items migrated successfully or pending or if some of them had errors upon migration. This middle system will make the migration process to be easy and straightforward.
Do not update or modify the old system by adding the status or anything mandatory for migration. Add whatever you need in this step, the middle-system mapping.
Prepare the migration tool
The migration tool you are creating or the scripts (sql, non-sql) that you need to run directly if it was just a database migration must be efficient, and can be stopped at any stage. It should contains the below features:
- Can be stopped at anytime
- Can be paged, get 100 or 1000 items every time
- Can process items in parallel (threading)
- Can resume the migration without starting over again
So those are the most common feature for a migration tool. So why the point of getting just 100 items or 1000, this is done for memory management, as we don’t know how much the data is big, and to keep the migration process smooth and easy.
The ability to process items in parallel is very important, let’s consider you are having millions of items. Each item to be migrated needs around 1 sec or 10 secs, so the total estimated time will take days to migrate all the data. So the best practice is to process items in parallel to shorten the cost effect.
The main feature also is to resume the migration process at any time needed. Why we need to run over all the items again so we can continue where we stopped. It’s not a good thing to run over again specially when more than 50% of the data were migrated.
Ways of migration
You can migrate in 2 different ways.
- Export then import data
- Transfer data directly
In the first step, you can export all the data needed to database or file system. Add some xml, or excel sheet to for the metadata, maybe you will choose a JSON object to be as your exported data type. That’s all depends on you. For me, I would use the JSON export type. This step contains more work to do as you need to create two tool one for export and another one for import. But here you can define what data to be exported, like you need to export for just 1 year, or this user etc…
The process here will not be automatic, like the tool is running and you can leave. There are steps to follow.
As for the “Transfer data”, it’s just one tool that read data from old system and copy to the new system. It’s an automatic migration, and no need to followup on it. Btw you can also define here what are the data that you need to migrate in first place, and do it on stages. The transfer data directly is my preferred method for migration.
For both ways, you must update/add the data in the middle-system mapping so can keep track of which data is migrated or not.
Validation
Here comes the final step of migration. After finishing the migration, it’s time to validate the data between old system and new one. So the middle-system mapping is here to help you :).
The middle-system now contains all the data migrated or not, with the identifiers of old system so you can go and check the old system items and do your troubleshooting. You can validate the migrated items, and in case there were any errors of problems it’s easy to get back to the original items in the old system by using the middle-system mapping.
The above are very important things to the migration. Guys let me know if i’ve missed anything and stay tuned 🙂