Repackaging MASSIVE Applications
Hey guys,
I have an application to package that contains about 47,000 files!! It's causing me major delays with compiles and captures... I've read that Wise has a limitation of 15,000 files. I'm starting to use external CABs, but it's not much if at all faster...
Using WPS 5.5, if anyone has experience or advice it would be appreciated.
Cheers,
Dave
I have an application to package that contains about 47,000 files!! It's causing me major delays with compiles and captures... I've read that Wise has a limitation of 15,000 files. I'm starting to use external CABs, but it's not much if at all faster...
Using WPS 5.5, if anyone has experience or advice it would be appreciated.
Cheers,
Dave
0 Comments
[ + ] Show comments
Answers (15)
Please log in to answer
Posted by:
VikingLoki
19 years ago
Brute force. Dual proc workstation, no such thing as too much memory, make sure the wise databases are on a SQL server. Tweak SQL for performance, put SQL in memory, performance monitor your server to look for bottlenecks in the process. Tweak, tweak, tweak.
Also look into ExpressBuild configuration within Wise. There you can set Build Using Multiple Processors, Build Using Multiple Distributed Computers, and Allow My Computer to Build for Others. That way you can get a whole bunch of machines & processors working at your build processes. I haven't played with this yet, but it's on the to-do list.
Wise KB Article #1838 (SQL Performance) and #1996 (What Is ExpressBuild) might help you out:
http://www.wise.com/KBArticle.aspx?articleno=1838&keywords=SQL performance
http://www.wise.com/KBArticle.aspx?articleno=1838&keywords=SQL performance
Also look into ExpressBuild configuration within Wise. There you can set Build Using Multiple Processors, Build Using Multiple Distributed Computers, and Allow My Computer to Build for Others. That way you can get a whole bunch of machines & processors working at your build processes. I haven't played with this yet, but it's on the to-do list.
Wise KB Article #1838 (SQL Performance) and #1996 (What Is ExpressBuild) might help you out:
http://www.wise.com/KBArticle.aspx?articleno=1838&keywords=SQL performance
http://www.wise.com/KBArticle.aspx?articleno=1838&keywords=SQL performance
Posted by:
dm1
19 years ago
OK Guys,
The application I'm trying DESPERATELY to package is driving me insane. I can get past the capture(after about three hours!!) but the compile keeps crashing towards the end...
The app has 47,500 files. Is it possible that some applications are just TOO BIG FOR WISE TO HANDLE??
Any ideas appreciated! [:@]
The application I'm trying DESPERATELY to package is driving me insane. I can get past the capture(after about three hours!!) but the compile keeps crashing towards the end...
The app has 47,500 files. Is it possible that some applications are just TOO BIG FOR WISE TO HANDLE??
Any ideas appreciated! [:@]
Posted by:
VikingLoki
19 years ago
47,500 files! UFF DA!!!! Better you than me, thats 4 sure!
Yeah, you hit the ceiling. By default an .MSI package can only contain 32,767 files because the Sequence column in the File Table is only an integer data type. You run out of Sequence numbers at 32,767. You need to change that data type to double integer, that will raise the ceiling to 2,147,483,647 files.
If you need more files than that, I'll personally show up and slap some sense into you.
Here, do this:
1. Open Microsoft's Windows Installer database editor, Orca.exe. If you do not have Orca, obtain the table editor from the Platform SDK for Windows Installer.
2. Select Open from the File menu and browse for the .MSI file.
---------------
Note:
Select the Windows Application.MSI template (or the template used by your organization), so new installations include the modification to the Sequence column in the File table.
---------------
3. Select Export Tables from the Tables menu. The Export Tables dialog appears.
4. On the Export Tables dialog, browse to the location to export the .IDT file.
5. Mark the File table checkbox and then click OK.
6. Open Notepad and browse to the .IDT file created in step 5.
7. The first row of the file contains the column headings. The second row contains the respective column definitions. Modify the Sequence column definition to "i4". "i4" specifies a double integer data type.
8. Save the .IDT file to preserve changes.
9. Re-open the .MSI file with Orca.
10. Select the Tables menu and then select Import Tables.
11. Browse for the .IDT file.
12. Select the row indicating the File table and choose <<Replace>> for the action and click OK.
13. Save the .MSI with Orca and then close the application.
Along with the File table, the Media and Patch tables are affected by large packages. Use the above steps to change the column definitions for the LastSequence column in the Media table and the Sequence column in the Patch table.
For additional information, please see the topic entitled "Authoring a Large Package" in the Microsoft Windows Installer SDK.
Yeah, you hit the ceiling. By default an .MSI package can only contain 32,767 files because the Sequence column in the File Table is only an integer data type. You run out of Sequence numbers at 32,767. You need to change that data type to double integer, that will raise the ceiling to 2,147,483,647 files.
If you need more files than that, I'll personally show up and slap some sense into you.
Here, do this:
1. Open Microsoft's Windows Installer database editor, Orca.exe. If you do not have Orca, obtain the table editor from the Platform SDK for Windows Installer.
2. Select Open from the File menu and browse for the .MSI file.
---------------
Note:
Select the Windows Application.MSI template (or the template used by your organization), so new installations include the modification to the Sequence column in the File table.
---------------
3. Select Export Tables from the Tables menu. The Export Tables dialog appears.
4. On the Export Tables dialog, browse to the location to export the .IDT file.
5. Mark the File table checkbox and then click OK.
6. Open Notepad and browse to the .IDT file created in step 5.
7. The first row of the file contains the column headings. The second row contains the respective column definitions. Modify the Sequence column definition to "i4". "i4" specifies a double integer data type.
8. Save the .IDT file to preserve changes.
9. Re-open the .MSI file with Orca.
10. Select the Tables menu and then select Import Tables.
11. Browse for the .IDT file.
12. Select the row indicating the File table and choose <<Replace>> for the action and click OK.
13. Save the .MSI with Orca and then close the application.
Along with the File table, the Media and Patch tables are affected by large packages. Use the above steps to change the column definitions for the LastSequence column in the Media table and the Sequence column in the Patch table.
For additional information, please see the topic entitled "Authoring a Large Package" in the Microsoft Windows Installer SDK.
Posted by:
dm1
19 years ago
Thanks VikingLoki for your advice.
Unfortunately when I opened the IDT file that I exported for the file table, the Sequence Column was already set to i4!! I can't understand why this would be because when I open the msi/wsi in Wise the Sequence column is RED after the 32767th file. I've edited the file.idt, media.idt and patch.idt and imported them all back into the MSI in Orca, but to no avail. The column is still red after 32767.
Is there any way around this?? Or is there something I've not done correctly??
Cheers
Unfortunately when I opened the IDT file that I exported for the file table, the Sequence Column was already set to i4!! I can't understand why this would be because when I open the msi/wsi in Wise the Sequence column is RED after the 32767th file. I've edited the file.idt, media.idt and patch.idt and imported them all back into the MSI in Orca, but to no avail. The column is still red after 32767.
Is there any way around this?? Or is there something I've not done correctly??
Cheers
Posted by:
VikingLoki
19 years ago
I'd dive into MSDN or Wise support, it sounds like MS has a few papers on "Authoring a Large Package". It will probably be in the Windows Installer SDK documentation. Also check Wise to see if anything needs to be changed on the wise side. Hopefully you can come up with something there so you're not forced to do a workaround.
If you need a workaround, what are all these files? I assume it must be 10% application 90% data files? Try to break out a bunch of data files so you end up with 2 groups, one all datafiles that don't have registered DLLs/OCXs/EXEs, the other all application files and maybe some data. Both need to be below 32767 files each. Turn it into two MSIs and set it up so that the "application" MSI installs the app and maybe some data, then it launches the "data" MSI which puts the rest of the data down on the machine. It's a PIA, but if you have no solution on the Large MSI Package from Wise or MS, then ya gotta do what ya gotta do.
If you need a workaround, what are all these files? I assume it must be 10% application 90% data files? Try to break out a bunch of data files so you end up with 2 groups, one all datafiles that don't have registered DLLs/OCXs/EXEs, the other all application files and maybe some data. Both need to be below 32767 files each. Turn it into two MSIs and set it up so that the "application" MSI installs the app and maybe some data, then it launches the "data" MSI which puts the rest of the data down on the machine. It's a PIA, but if you have no solution on the Large MSI Package from Wise or MS, then ya gotta do what ya gotta do.
Posted by:
Bladerun
19 years ago
Posted by:
dm1
19 years ago
yeah the app is mostly data files(90%) as far as i can see... It had crossed my mind to split the application into two MSIs, but i thought it would be messy. Do you think I should just copy say 20,000 files and put them into another MSI?
What we do here is package applications by wrapping them in kix scripts for deployment purposes which then handles all of the group permissioning issues and Folder removal on uninstall. The company are not that strict on whether or not we use MSIs or not so i explored installing it silently with setup.exe /s, but this wont work because the app has a seperate installer that doesnt recognise '/s'. On the vendor support website it says to use a -replay command pointing to an installation.properties file if you want to do subsequent silent installs. This installation.properties file is supposedly created when you install the application for the first time, but lo-and-behold it wasn't. So I'm back to square one on this.
Will probably try the splitting into two MSIs.
Cheers!
[:(]
What we do here is package applications by wrapping them in kix scripts for deployment purposes which then handles all of the group permissioning issues and Folder removal on uninstall. The company are not that strict on whether or not we use MSIs or not so i explored installing it silently with setup.exe /s, but this wont work because the app has a seperate installer that doesnt recognise '/s'. On the vendor support website it says to use a -replay command pointing to an installation.properties file if you want to do subsequent silent installs. This installation.properties file is supposedly created when you install the application for the first time, but lo-and-behold it wasn't. So I'm back to square one on this.
Will probably try the splitting into two MSIs.
Cheers!
[:(]
Posted by:
VikingLoki
19 years ago
Posted by:
VikingLoki
19 years ago
On a side note...
I've found that when you're working with a large package (100MB or above), you'll save yourself a lot of time by compiling with external CAB files. That way if you make a modification that doesn't include file changes, the recompile will leave the CAB as-is and will only modify the MSI tables. Much shorter compile since you're not moving an extra 100+ MB of files around. If you don't want external CAB files in the end result, once you have everything just so, switch it to CABs Inside MSI and do one last long recompile to obtain your single large MSI file.
I've found that when you're working with a large package (100MB or above), you'll save yourself a lot of time by compiling with external CAB files. That way if you make a modification that doesn't include file changes, the recompile will leave the CAB as-is and will only modify the MSI tables. Much shorter compile since you're not moving an extra 100+ MB of files around. If you don't want external CAB files in the end result, once you have everything just so, switch it to CABs Inside MSI and do one last long recompile to obtain your single large MSI file.
Posted by:
dm1
19 years ago
yeah thats a good tip, should save people a lot of time. I've been using it myself recently.
I'm still trying to create multiple msis with that app. trouble is, sometimes when i'm copying the files into the .wsi/.msi, wise just crashes. without doubt the most tedious, frustrating package ever!
i've put in a request to deploy the app manually as there's only 9 users! [&:]
I'm still trying to create multiple msis with that app. trouble is, sometimes when i'm copying the files into the .wsi/.msi, wise just crashes. without doubt the most tedious, frustrating package ever!
i've put in a request to deploy the app manually as there's only 9 users! [&:]
Posted by:
glwday
19 years ago
Hi dm1
Your kix scripts way of working sounds remarkably like what we do here. Wonder if it's the same global outsourcing company HQ in CA usa?
Does your mega app have any options you want to include? You could capture the install without the options then capture the options separately. I did this last year with ProEngineer.
HTH
Gazza
Your kix scripts way of working sounds remarkably like what we do here. Wonder if it's the same global outsourcing company HQ in CA usa?
Does your mega app have any options you want to include? You could capture the install without the options then capture the options separately. I did this last year with ProEngineer.
HTH
Gazza
Posted by:
MSIMaker
19 years ago
The other thing you could try is to do a Wininstall capture and get a file set. Find your larger folders and zip the files into a zip file. This way you lessen the amount of total files that Wise has to deal with.
After you get your files into a manageable state you can import them into your Wise package.
During the install you can run custom actions to unzip the files back to their original folders.
After you get your files into a manageable state you can import them into your Wise package.
During the install you can run custom actions to unzip the files back to their original folders.
Posted by:
dm1
19 years ago
Posted by:
pace-support
7 years ago
Rating comments in this legacy AppDeploy message board thread won't reorder them,
so that the conversation will remain readable.
so that the conversation will remain readable.