FPK -vs- Folders and Game Speed

[Tijer]

I have done some tests into the issue of which is faster FPKs or Folders. The truth is the only difference is in the initial map load not in actual game play. At that, the speed difference is minuscule. Once the map is loaded, the FPKs or folders, can be removed from the map folder. What this means is the data from the FPK or Folder are stored in RAM. After initial load all things are the same. I first heard about this in the Forums for the game Port Royal. Since I have also seen mentioned on some of the RRT3 sites. My point is that it may have made more of a difference on slower processors in the past, but as processor speeds have increased, the load speed is less of a factor. Just as a note to mappers, how well the XML are written, ex. vacant lines, are more of a factor. If you leave spaces in the XML from editing, you are more likely to effect all of the game play. Stuttering and eventual crashes can be resolved when the holes are removed.

[Star Ranger4]

Somehow, this information doesn't surprise me at all Tjier. If I were a programmer looking at performance issues, I certainly would prefer to load data into memory once, and keep as much of the actively used data in real ram. Why? pure speed. A memseek to ram to retrieve data is 100 to 1000 times (or more, given todays clock speeds) than accessing physical media, either during IPL or Virtual memory.

This also seems to tie in to a growing but unvoiced feeling I had about the crashes... that it happened during a memory overflow event, such as trying to page memory sections to or from disk. In short, my theory was that because disk access was so much slower, the game thought the data had been lost and just... well, plotzes.

Certainly the new methods being developed by Snoopy and Lowell seem to back this up; by setting the game up so that you only load what is actually used in your scenario, you decrease memory overhead. No need for paging memory to disk would mean 0 chance of a timeout in a disk access operation and much faster execution overall.

Now, if there was only a way to setup the game so that we had all the pieces, but it only loaded the pieces we were actually using for that map, that might be nice.... or is that also part of what Lowell and Snoopy are hoping to achieve in the future?

(edited to correct who made the above post...)

[Tijer]

So long as sloppy XMLs are not an issue, Memory overflow would most likely appear to be the source of crashes. As to Sam or CIC actually curbing the problem is hard to tell. The reason that I say that is both allow the game globals to be loaded first. So long as they exist as they are we will never be able to load only truly needed content. Simply placing a set of GLOBALS on the My Games side of the game as Lowell did in FG, does not automatically mean that the game globals are not still stored in RAM. They maybe over written, but most likely what happens is that the data gets stored twice. Conventional wisdom says that they should be over written, it does not automatically mean that that is the case. The only way to do that is to replace the Game globals with each map install. In previous discussions of this on this forum, is where Snoopy and I where threated with law suits, arrest and all other kinds of nitrates. But it is the only way to have only needed content loaded. That is why Snoopy and I have been so careful with SAM. The objective is to remove as much unneeded content as possible. As to developing SAM2 which would replace the game globals and creating something to install the needed materials on both sides of the game, may happen. However, due to the threats that have been leveled against us, it will not be released here, if anywhere. I do not mean to be selfish, but I do not need the hassle, nor do I need to put my personal property on the line, all the thanks in the world does not pay lawyer's fees.

[snoopy55]

Recent events have caused me to do a little write-up on this subject. It has repeatedly been said that using FPKs will get them lost.......

From a post made Jun 8 in "LongShot New Map":

It would be easy for me to make this map a clean install type map. The only differance with my style is everything is inside one file folder and no loose FPKs to loose later...many people have just one clean RailRoads! folder and swap out the maps one at a time. Loose items placed extra can get lost as time goes by, or if they stick FPKs in the Custom Assets folder they may forget and leave too many in there. All new map content is stored inside the maps folder in a folder called Assets. All that is required is say if the map uses the port from B&R...then you have to dig that models files out and have them in with your Assets. Other "global" files can be added with the scenario files to give players new difficulty money, new goods and/or industries for the game, thus new items can be added much easier.

I would be glad to help build the files if asked.

A post made Mar 28 in "What we are doing here...":

The SAMs run crash free also, and I don't notice a speed difference either way. I test them in folders, then put them into FPKs. No crashes, and, as I said, I notice no speed difference. And unzipping one folder or one folder and one FPK into UserMaps, not a big thing.

I attempted to point all of this out to you last January. Why has it taken so long to see it?

I've tried to make it clear just where the FPK file or files are to go. And yes, anyone can move them to CustomAssets if they so choose, just like they can pull the folder out of a CIC map and put it into CustomAssets if they choose also. As I've also stated before, 'It's their system, they can do as they choose'.

Just for a bit of understanding, let me make a worker out of the game. You choose to play Side to Side. First let's look at a SAM version. You take the map data from the ZIP file and put it into UserMaps. You start the game and choose the map. Well, the game has a job to do, it must gather up all the needed files to play the map. In a SAM it opens up the FPK file included AND the FPK files in the original game. It goes thru and pulls them out of the FPKs and shoves them into memory and, if needed, Virtual Memory. When it has all it needs, it goes on to play the map. It does not have to return to the FPKs to get anything else. In plain English, the FPK files no longer effect the speed of play. Memory speed and HD speed are now in control.

Now let's look at a CIC version. You take the map data from the ZIP file and put it into UserMaps. You start the game and choose the map. Well, the game has a job to do, it must gather up all the needed files to play the map. In a CIC it opens up the folder included AND the FPK files in the original game. It goes thru and pulls them out of the folder and FPKs and shoves them into memory and, if needed, Virtual Memory. When it has all it needs, it goes on to play the map. It does not have to return to the folder or FPKs to get anything else. In plain English, the folder and FPK files no longer effect the speed of play. Memory speed and HD speed are now in control.

So, what IS the difference?? Ah yes, the lost FKP file...... Let's now play out that one....

SAM version. You decide to play a different map. You go into the SMR folder, into the UserMaps folder and delete two things, the map folder and the FPK file. (Yes, they ARE in the same folder, unless you decided to put one of them somewhere else.....) You then unzip the map you want to play and place the files into UserMaps, close it up and play the game....simple, no lost FPK files. (Tijer and I checked, they do NOT grow legs and scurry off to some other folder...)

CIC version. You decide to play a different map. You go into the SMR folder, into the UserMaps folder and delete one thing, the map folder. You then unzip the map you want to play and place the file into UserMaps, close it up and play the game....simple, no lost files. (unless of course the map folders have figured out how to duplicate themselves and run off to another folder....)

So, someone tell me, what is all of this problem someone has with FPKs, stating there is a difference in speed, and why are there people believing it? Computers these day run a terrific speeds! This game is sslllooowwwww compared to UT3, which has much more going on, so how do you notice a speed difference? It is claimed that a map was run using a folder and then using an FPK. So, how do you judge the speed difference? The speed of the train? Train speed is not determined by where the files for the train come from. AND, even if they do, most of the files for Florida Gold come from the Original FPKs in the game. His items use KFM, NIF and a few DDS files. The rest, which include KFMs, NIFs, DDSs, KFs, TGAs and other types, come from those FPK files. And the effects in the map, the smoke, the animals and other things moving, come from the Original FPKs. And the game does NOT go and search for the needed file each time it has to do an animation The thing would never move.

I really do not care if he does CICs, he stated he already has maps made into CICs. Let him post them. Why should I do work that is already done.

And if someone wants a map SAMed, tell me and I'll see what I can do with my time to get it for you. But please do not say things based on illogical statements that cannot be proven. 20 some odd maps, with more being made by our mappers out there, have no Mods, so they would, at the most, need only a 'NewTenders.fpk' file, and many not even that. So, if they need no FPKs and no folders in the map folder, how can they run faster? And a few Mods in FPK files are not going to make that much difference either. The loading of the map is the ONLY place where speed may be a difference, and unless you have EVERY background program shut off, you cannot measure that either.

Attached are two ZIPs, one is Side To Side with FPKs and the other is Side To Side with a folder in it. These zips are the FULL SMR folder, just as it would be loaded. You check them out.

[Lowell]

That side to side map crashes out in smri, it has always crashed out for me and three other people that play the game...standing joke. I think Mid East or whatever crashes out never fixed. Then in the construction set, someone posted that much of it is broken and crashes out...you posted you were too busy to test everything you make.

You tell me...that is your stuff and your words.

That's why I find it hard to believe you sometimes. Then there are these comments that this forum is all about "me" and my maps. I know you "share" the same thoughts with the person who made those remarks. As an Admin for this site you should know the forum rules on personal attacks. Just because some people wanted to help fix a map they like to play...that seems to make you two mad for some strange reason. If people are having fun here leave it alone...you want to scare everyone off? Why if you two have soooo much great knowledge you don't show us the way. Instead you have to be totally rude and trash everything for everyone. I can't tell you how many people are disgusted with the attitudes. Help and be creative or shut up. I'll also have you know, I have always said to all map makers, that I would be glad to help anyway I could...make new trains for them, new buildings etc. it isn't all about me at all, I have a couple maps that I am trying to complete and others have wanted to help. So what's the deal really about.

The FPK issue isn't really about speed. It's about someone leaving a FPK inside custom assets then forgetting and swapping out a sam or cic map. All content must be swaped out. If they place their fpks in usermaps always, then maybe they won't forget to swap it out. But after a couple months, and they try to load the old map, if they forget they fpk and the map crashes, they might not look into that map much longer. The cic maps have all assets inside the one folder. It's no big deal, it's just how any map maker might want to make their files. To each his/her own...no biggie.

[snoopy55]

Point 1....those are not to be played, those are to show that IF there is a speed difference in loading, it is NOT enough to be noticed.

Point 2... If those two maps crashed for you and three others, why did you not post the problem? They tested my 7th grade class for ESP, and I failed.

As to the Building Blocks...at least I didn't have to make 115 plus edits on the same thing, which still isn't finished. If you feel that a map, or something else should be fully working before it's posted, you've just accused many mappers, and yourself, of the same thing. I never said my stuff was picture perfect. If I was perfect, I wouldn't be here, I'd be in the White House.

Personal attacks..... you made those to me by accusing me of editing your posts. K-class did that when he stated that I didn't know what I was talking about after all the time and effort I have spent on the forum. He also stated that the only maps to be played were yours (don't use maps with Installers, SMRI or other maps). And yours are still to be finished.

I never complained about people helping you. I have my thoughts, but I kept them to myself and will continue to.

K-class is the one trying to scare people away by telling them not to load other maps.

As to my knowledge, I spent weeks working with you on that. You condemned it, stated you would NOT do it that way, then came up with this great revolutionary new way. And showing the way???? I seem to recall rewriting Trains XMLs for a couple of people. All those building Blocks which have been seldom used, which, by the way, you stated you were going to fix. All the repairs to map, including helping you with your map. GLOBALS which helped make the 91 maps we have possible. You see, I don't take someone else's idea and redo it and call it my own. And then go and stomp on their way, including stating that it is fine for the little mappers, but MY way is the only one that will work for the BIG maps. We haven't published any maps because we have to make sure they work and work right, and that takes time.

And maybe you are right...since I'm no help on this forum, and since I've never been creative, maybe I should just shut up....but not at your request, or any of your buddies.

[snoopy55]

Lowell, if you have more to say, make another post, don't edit your own, you seem to accuse me of making the changes to them.

[Lowell]

Certainly the new methods being developed by Snoopy and Lowell seem to back this up; by setting the game up so that you only load what is actually used in your scenario, you decrease memory overhead. No need for paging memory to disk would mean 0 chance of a timeout in a disk access operation and much faster execution overall.

Now, if there was only a way to setup the game so that we had all the pieces, but it only loaded the pieces we were actually using for that map, that might be nice.... or is that also part of what Lowell and Snoopy are hoping to achieve in the future?

...yes. In my maps I added a couple of the games global files. One for goods and one for industries. Inside the industries file I only list the game industries I want the game to "turn-on." On the Florida Gold map, there is no coal...so the game doesn't load coal, swine, steel or any other game models and lists. Thus...I load less new stuff than I turn off in the game. This is why all the sudden people are able to play a size 20 map for the first time and it run stable.

The downside is these maps need to be the only ones in the game at a given time. They need swapped in and out when moving to a different map, or have a zillion renamed RailRoads! folders as I have, one for each self contained map. To each his/her own. Many of my friends just have one clean game folder and just swap them in and out of the one game file.

Now, game maps that load and use only the games default stuff, can be left-in a big map pack without any real troubles. It is only when we add new content that it gets a tad tricky.

I need to have more tests done with this new CIC map style. Florida Gold and Holidays_In_Ohio-ToyTrains and soon to be released Mud Skippers, are the first set of maps made this way by me. I think it would be great if others could take my files and do several tests with them just to make sure what I have is needed and works correctly. Each global file was added one at a time and I had to have a reason to add it I do remember. I was hesitant to add any global files into my maps files...but as they unfolded and we beta tested them I found that I had to add several global files for the game to respond correctly. It still would be great for others to verify my work, making sure I didn't overlook anything and that I have everything correct.

FPKs cannot be placed inside a maps game folder and then placed in usermaps...the game will crash out real hard. The FPKs work best when left in custom assets. The fear I had was when swapping out self contained maps, that someone would forget to clean out the custom assets folder and cause a self contained map to fail. Then there is the deal with swapping out so many self contained maps...later as time goes on...people might loose the maps FPKs and then when they swap out the map folder and the map fails and crashes...they would shelf the map and not look farther into it's crash.

So really FPKs are an extra step the map maker doesn't need. Every time you make a simple edit to any of your maps content you have to re-compress everything all over again. It is just an extra step waiting to add to confusion and a crash.

Keep it simple I say...low teck is better teck. Plus for future map makers it is easy for everyone to see the maps content in the assets folder.

However that being said there is nothing wrong with using the FPks if that is what the map maker wishes. "I think the CIC version is much easier for a new map maker to see and learn from." You cannot use FPKs in CIC maps or they crash.

So I don't know where this speed bit came from, but it has more to do with common sense and ease of use than anything else.

[snoopy55]

So I don't know where this speed bit came from, but it has more to do with common sense and ease of use than anything else.

To simply answer your question, from Construction Zone, "What we are doing here, page two:

The difference is the game runs super smooth. No crashes at all...none. When I was testing the maps/game, I minimized the game while it was still running at different stages through out the whole time period of game play. With the maps new items inside the FPK file it ran a tad slower than if the new contents were added to a simple folder inside the maps folder.

As to the rest of the post and the before and after, you can read that for yourself.

[Lowell]

True...it will load a tad quicker as I had said...but the real reason is it is easier to work with...and keeps everything inside one file folder.

I remember I had the map in two forms when that post was made...one with FPKs and one CIC style. The CIC style map always loaded faster.

But the real frustration came from the many edits to game content...and having to re-compress everything all the time. It just seemed like a waste of time.

But like I said, what ever pleases the map maker is what counts. If you like FPKs then use FPKs.

Just remember that people will always gravitate to the simpler solution...one file folder. It's what people asked for...a single file folder that they could just drop into usermaps, having nothing to copy and paste to their default games files. That is what was created...what the people wanted.

[Jancsika]

Lowell you are missing the point.
Most of the people on this board (including me) have no idea and really don’t care if the game is loading slower or faster, would not even notice unless somebody stands by with a stopwatch.
Both your CIC games and Snoopy’s SAM-s are stable once they are loaded.

There are a few of us who like make new scenarios cobbled together from other peoples materials. We use parts from the basic Firaxis game and pick up morsels of innovation from here and there.
This is where the Building Blocks come in play. I have made couple of scenarios using the basic SAM structure but adding other bits of units from what's available on the bord and they are very stable.

To make CIC game is beyond my and most of the people’s knowledge, but to make a SAM all you need to be able to handle a text editor.

Lowell you cold be a tremendous help to us ignorant gamers if you cold make some of your industries into a freestanding FPK file with the corresponding XML. Rodea did this with his Bank and Limestone.

Remember; there is more than one way to skin a cat..
Jancsika

[Lowell]

Jancsika I just got through saying this speed deal isn't what CICs are about. It is about keeping everything a map needs to run inside One File Folder. I could care less how fast the game loads unless it is excessive. It's about maps that have to have a ton of files to run...people had posted on my map thread a question...could I get all my maps files and content into a single file folder that people can drag and drop and play. That means they were asking if FPKs can be gotten rid of or hide them inside the map folder so they only have One file folder to mess with.

A single file folder system is what we are talking about...not speed.