Data Management

This application deals with large amount of data. Admins are allowed to upload very large sized files (having sizes more than 500MB) in csv, xls or xlsx format. Based on the file headers, the application dynamically creates new temporary tables and inserts data in the application. Admin has the option to map the data in temporary tables to live database and upon approval this data gets replicated in the main database.

There are multiple databases in the application to serve the purpose for the users as well as admin. There is also a json API using which user can send json data and it will be mapped and processed in the same way.  New Relic is used for application performance testing and monitoring.